You changed the charter. Did it get better?

A charter accumulates. Somebody adds a paragraph after an incident. Somebody else adds three rules about naming, and a fourth person pastes in a style guide. Each addition is defensible on its own, and nobody measures the total. Two failure modes follow, and both are silent. Rules contradict each other, so the agent picks one at random per session. Or the document gets long enough that the agent stops following the instructions at the bottom. Nobody notices which third stopped working.

The reason is not laziness: measuring a non-deterministic system looks like it needs apparatus nobody has time to build. It does not. The version below is roughly a hundred and fifty lines of Python, and catches both failure modes within a week.

src/billing.py        unit tests, types, review, CI
infra/main.tf         plan, policy check, review
.github/workflows/    runs on every PR, fails loudly

CLAUDE.md             merged on a nod
.claude/rules/*.md    merged on a nod
prompts/*.txt         merged on a nod

The bottom three decide the shape of everything in the top three.

FIXTURE = "fixtures/billing"          # copied to a temp dir per run

PROMPT = """
Refunds for cancelled orders are being rejected when the order
included a percentage discount. Fix it so a cancelled discounted
order can be refunded in full.
"""

# What "done well" means here, in this repo, for this team.
CHECKS = [
    "compiles", "tests_added", "money_is_integer",
    "no_unrelated_files", "policy_cited",
]

Choosing twenty

• Real tickets, past tense. Take work already done: you get a known-good outcome for free, and a task shaped like your real work. No synthetic task has that shape.
• Spread across the failure modes you care about. If half your charter is about money handling, several tasks should touch money. The set states what you worry about.
• Include three you expect to fail. A set everything passes cannot show improvement, only regression, so headroom is a design requirement.
• Two that need a refusal. Tasks where the correct behaviour is to stop and ask. Without them, charters that are pure encouragement score too well.

def compiles(w, r):
    return run(w, "make build").returncode == 0

def tests_added(w, r):
    return any(p.name.startswith("test_") for p in r.files_changed)

def money_is_integer(w, r):
    # the charter says money is integer cents, everywhere
    return not re.search(r"float\(|/ 100\b", r.diff)

def no_unrelated_files(w, r):
    # scope creep, class 5 in the review taxonomy
    return set(r.files_changed) <= set(TASK.allowed_paths)

def policy_cited(w, r):
    return "CHARTER" in r.summary or "policy" in r.summary.lower()

Each check above turns one charter rule into a measurement. Compiling the same rule into a hook is the other half of the move, aimed at enforcement instead of measurement. The payoff is the same either way: a property a script can check stops being a matter of opinion.

JUDGE_MODEL = "claude-sonnet-5"   # pinned. changing it resets history.

RUBRIC = """
Answer YES or NO, then one sentence.

Question: does this change handle the cancelled-order case by
consulting the order state, rather than by catching an exception?

YES example: reads order.status before computing the refund.
NO example:  wraps the refund call in try/except and returns 0.

Answer NO if you are unsure.
"""

Four rules that do the work

1. Binary Yes or no, never one-to-five. Numeric scales from a model are not interval scales. Averaging them produces a number with no meaning, and everyone puts that number in a slide anyway.
2. Anchored A concrete example of each verdict, from your own codebase. Anchors move judges more than any amount of instruction prose.
3. One property One question per call. Ask a judge to assess six things at once and it produces a summary judgment about vibes. Then it rationalises that judgment into six sub-scores.
4. Pinned Record the exact judge model with every result. When the judge changes, historical comparisons are void, exactly as if you had swapped out the task set.

Never ask "is this output good". Ask "is the charter on this branch better than the one on main". So run both charters on the same task, and ask the judge which it prefers. Hide the two labels, and swap the order.

def compare(task, charter_a, charter_b, k=3):
    wins_a = wins_b = ties = 0
    for _ in range(k):
        a = run_agent(task, charter_a)
        b = run_agent(task, charter_b)

        # ask twice, swapping which one is shown first
        first  = judge_prefers(task, a, b)   # -> "first" | "second"
        second = judge_prefers(task, b, a)

        # position bias: if the judge picks the same SLOT both
        # times, it is not reading. Discard the sample.
        if first == second:
            ties += 1
        elif first == "first":
            wins_a += 1
        else:
            wins_b += 1
    return wins_a, wins_b, ties

The discard is the important line. Position bias is well-documented and ordinary: a judge can prefer whichever output it sees first. Run one order and that bias passes into your results undetected. Run both orders and the bias becomes a visible tie, which is honest and costs one extra call.

Twenty tasks run three times is sixty samples per arm, which sounds like plenty and is not. Take a pass rate somewhere around seventy percent. Then the rough sample sizes needed to detect a change with any confidence look like this.

Read the table the right way round. A twenty-task set with a few runs each is a regression detector. It will tell you when a charter edit broke something, which is the failure that costs you money. It will not tell you which of two reasonable phrasings is five percent better. A loop that appears to answer that question is reporting noise with a confident face.

Two cheap changes raise sensitivity without buying runs. Use paired comparison, which the design in §4 already does: pairing is more sensitive than comparing two independent pass rates. Then make the deterministic checks in §2 carry as much of the load as they can. Those checks add no judge noise on top of the agent's own variance.

on:
  pull_request:
    paths:
      - "CLAUDE.md"
      - ".claude/rules/**"
      - "prompts/**"

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - run: python -m evals.pairwise --base origin/main --head HEAD --k 3
      - run: gh pr comment "$PR" --body-file evals/out/report.md

Charter eval · task set v3 · judge claude-sonnet-5 · k=3

  deterministic checks     main 84/100   pr 91/100
    money_is_integer       16/20 → 20/20  (the rule you added)
    no_unrelated_files     19/20 → 15/20  ← regression

  pairwise (judged)        pr wins 6 · main wins 4 · ties 10

  VERDICT  REGRESSION on scope creep. The new section appears to
           license refactoring while nearby. Judged difference is
           indistinguishable (tie rate 50%).

That report is the point of the whole exercise. It names what got better, names what got worse, and stays agnostic about the part it cannot resolve. A reviewer can act on all three statements.

Cost

A full pairwise run is tasks × k × 2 agent invocations plus tasks × k × 2 judge calls. Twenty tasks at k=3 is 120 agent runs and 120 judge calls. Price that against your own provider once, put the figure in the README, and revisit it when the number changes. That bill is usually small next to one senior engineer spending a day arguing about a paragraph.

Cadence

• On charter pull requests, always. This is the whole point.
• Weekly on main, unchanged charter. This catches drift from underneath you: a model update, a dependency change, a fixture that rotted.
• Never continuously. A loop that runs hourly generates numbers faster than anyone can read them. Then people start ignoring it by the second week.