Evals

A test tells you whether a function returns the right value. An eval tells you whether a prompt change made the answers better or worse. ghola needs both. Half of it is code, and the other half asks a model to judge something.

ghola writes no eval runner. The stock eval worker already runs durable same-model A/B evaluations, alternates variant order to reduce order bias, persists its reports, and injects a console page. What it leaves open is the one thing it cannot know: whether a particular answer was any good.

Running them

make eval                                 every case
make eval CASE=prove-cites-evidence       one case
make eval RESULT=eval_abc123              what it found

Evaluations are durable and run in the background. They cost money, so nothing runs them on a timer. Run them before you ship a prompt change. That is the moment they earn their price.

sequenceDiagram
    autonumber
    actor You
    participant W as the eval worker
    participant M as the model
    participant C as contracts.read
    participant G as the grader
    participant R as the report

    You->>W: make eval CASE=...
    Note over W: one case changes exactly one dimension
    loop each run, variant order alternated
        W->>M: the control prompt
        M-->>W: an answer
        W->>M: the treatment prompt
        M-->>W: an answer
    end
    W->>C: parse both answers
    C-->>W: the verdict, after any downgrade
    W->>G: output, metrics, arguments, role
    G-->>W: passed, and why
    W->>R: pass counts, the delta, eligible
    You->>R: make eval RESULT=eval_abc123
    

What a case looks like

Each file in evals/ is an eval::start request:

{
  "dimension": "prompt",
  "model": {"model": "claude-sonnet-5", "provider": "anthropic"},
  "control":   {"label": "names a file and line",  "prompt": "..."},
  "treatment": {"label": "objects without naming", "prompt": "..."},
  "evaluator": {
    "function_id": "ghola::eval::verdict-is",
    "arguments": {"contract": "verdict", "equals": "concerns"}
  }
}

One case changes exactly one dimension. An A/B that changed two things tells you something moved. It never tells you which thing moved it. The worker enforces this, and so does make test.

A case that cannot fail is not evidence

Write the failing half first. A suite where every case passes tells you the suite runs, and nothing else.

Both shipped cases discriminate against a real model. verdict-is passes a review naming app.py:9. It fails one that says "something about this feels wrong", which is the behavior the contract exists to catch. Control scores 1/1, treatment scores 0/1, and the pass-rate delta is -1.0.

eligible is the worker's own judgment and it is strict on purpose: every treatment run must pass, and the pass count must not regress. A candidate that is no worse is not evidence that it is better.

The four graders ghola ships

These are the same checks the pipeline runs. contracts.read grades prove and review in production, and the graders reach it rather than reimplementing it. So an eval measures the thing that ships instead of a second copy that drifts.

verdict-is grades after the contract's downgrades, not before. A PROVEN: yes with no evidence parses as unproven, so a case expecting yes fails on it. Grading the raw claim would reward the behavior the contract exists to catch.

cites-evidence is the most useful one on a prove phase. A model that stops running things and starts asserting them still produces output that reads exactly like a proof.

Your own cases, without a fork

Point settings/evals.yaml at a directory anywhere:

suites:
  - ~/code/our-harness-evals/cases

ghola reads it alongside evals/ rather than instead of it. Your suite lives in your repository. It is versioned with the prompts it grades, and it goes through review like anything else.

A suite that is not where it said it would be gets reported before anything runs. That matters. A run of no cases prints exactly like a run that passed.

Your own graders

A grader is fifteen lines on any worker, in any language:

worker.register_function("acme::eval::mentions-a-ticket", grade)

It receives (output, metrics, arguments, role) and returns {passed, score?, reason?, details?}. Name it in a case file's evaluator, and list it under graders: in settings/evals.yaml so make config can show it.

Two rules, and both come from how the worker delivers:

It must be deterministic and idempotent. Durable delivery is at-least-once. A grader that answered differently on a redelivery would make the report a coin toss.

A grader that cannot run must fail, never pass. An unknown name, a raised exception, a contract nobody defined: each one is a failed case. Report any of them as a pass and you have put a green tick on a case nobody graded.

flowchart LR
    g{"the grader"}
    g -- "it ran, and it passed" --> pass["pass"]
    g -- "it ran, and it failed" --> fail["fail"]
    g -- "unknown name" --> fail
    g -- "raised an exception" --> fail
    g -- "no such contract" --> fail
    fail --> why["a green tick on a case
nobody graded is worse
than a red one"]

    classDef gate fill:#3A2112,stroke:#E7C982,color:#E7C982
    classDef ok fill:#1A100A,stroke:#A4470F,color:#E8A33D
    classDef bad fill:#23150C,stroke:#6E2E13,color:#E7C982
    classDef aside fill:#0F0A06,stroke:#3B2515,color:#AB9070
    class g gate
    class pass ok
    class fail bad
    class why aside
    

What is not covered

Six phases produce something. Two of them have cases. plan, run, refine and improve have none. A change to one of those prompts ships with nothing here telling you whether you made it worse.