You changed the charter. Did it get better?

A charter accumulates. Somebody adds a paragraph after an incident, somebody else adds three rules about naming, a fourth person pastes in a style guide. Each addition is individually defensible and nobody ever measures the total. Two failure modes follow, and both are silent. Rules contradict each other, so the agent picks one at random per session. Or the document gets long enough that the instructions at the bottom stop being followed, and nobody notices which third stopped working.

The reason this persists is not laziness. It is that measuring a non-deterministic system feels like it requires apparatus nobody has time to build. It does not. The version below is roughly a hundred and fifty lines of Python and it will catch the two failure modes above within a week.

src/billing.py        unit tests, types, review, CI
infra/main.tf         plan, policy check, review
.github/workflows/    runs on every PR, fails loudly

CLAUDE.md             merged on a nod
.claude/rules/*.md    merged on a nod
prompts/*.txt         merged on a nod

The bottom three decide the shape of everything in the top three.

FIXTURE = "fixtures/billing"          # copied to a temp dir per run

PROMPT = """
Refunds for cancelled orders are being rejected when the order
included a percentage discount. Fix it so a cancelled discounted
order can be refunded in full.
"""

# What "done well" means here, in this repo, for this team.
CHECKS = [
    "compiles", "tests_added", "money_is_integer",
    "no_unrelated_files", "policy_cited",
]

Choosing twenty

• Real tickets, past tense. Take work already done. You get a known-good outcome for free and the task is genuinely shaped like your work, which a synthetic task never is.
• Spread across the failure modes you care about. If half your charter is about money handling, several tasks should touch money. The set is a statement about what you are worried about.
• Include three you expect to fail. A set everything passes cannot show improvement, only regression. Headroom is a design requirement.
• Two that need a refusal. Tasks where the correct behaviour is to stop and ask. Charters that are pure encouragement score suspiciously well until you add these.

def compiles(w, r):
    return run(w, "make build").returncode == 0

def tests_added(w, r):
    return any(p.name.startswith("test_") for p in r.files_changed)

def money_is_integer(w, r):
    # the charter says money is integer cents, everywhere
    return not re.search(r"float\(|/ 100\b", r.diff)

def no_unrelated_files(w, r):
    # scope creep, class 5 in the review taxonomy
    return set(r.files_changed) <= set(TASK.allowed_paths)

def policy_cited(w, r):
    return "CHARTER" in r.summary or "policy" in r.summary.lower()

Each of these is a rule from the charter, compiled into a sensor. That is the same move as turning a rule into a hook, pointed at measurement instead of enforcement, and it has the same payoff: a property that is checked mechanically stops being a matter of opinion.

JUDGE_MODEL = "claude-sonnet-5"   # pinned. changing it resets history.

RUBRIC = """
Answer YES or NO, then one sentence.

Question: does this change handle the cancelled-order case by
consulting the order state, rather than by catching an exception?

YES example: reads order.status before computing the refund.
NO example:  wraps the refund call in try/except and returns 0.

Answer NO if you are unsure.
"""

Four rules that do the work

1. Binary Yes or no, never one-to-five. Numeric scales from a model are not interval scales, and averaging them produces a number with no meaning that everyone will nonetheless put in a slide.
2. Anchored A concrete example of each verdict, from your own codebase. Anchors move judges more than any amount of instruction prose.
3. One property One question per call. A judge asked to assess six things at once produces a summary judgment about vibes and then rationalises it into six sub-scores.
4. Pinned Record the exact judge model with every result. When the judge changes, historical comparisons are void, exactly as if you had swapped out the task set.

The question is never "is this output good". It is "is the charter on this branch better than the one on main". So run both, on the same task, and ask the judge which it prefers, with the two labels hidden and the order swapped.

def compare(task, charter_a, charter_b, k=3):
    wins_a = wins_b = ties = 0
    for _ in range(k):
        a = run_agent(task, charter_a)
        b = run_agent(task, charter_b)

        # ask twice, swapping which one is shown first
        first  = judge_prefers(task, a, b)   # -> "first" | "second"
        second = judge_prefers(task, b, a)

        # position bias: if the judge picks the same SLOT both
        # times, it is not reading. Discard the sample.
        if first == second:
            ties += 1
        elif first == "first":
            wins_a += 1
        else:
            wins_b += 1
    return wins_a, wins_b, ties

The discard is the important line. A judge that prefers whichever output it was shown first is a well-documented and entirely ordinary failure, and running only one order lets it pass silently into your results. Running both orders converts the bias from an invisible skew into a visible tie, which is honest and costs one extra call.

Twenty tasks run three times is sixty samples per arm, which sounds like plenty and is not. For a pass rate somewhere around seventy percent, the rough sample sizes needed to detect a change with any confidence look like this.

Read the table the right way round. A twenty-task set with a few runs each is a regression detector. It will tell you loudly when a charter edit broke something, which is the failure that actually costs you money. It will not tell you which of two reasonable phrasings is five percent better, and any loop that appears to answer that question is reporting noise with a confident face.

Two cheap ways to buy sensitivity without buying runs: use paired comparison, which the design in §4 already does and which is substantially more sensitive than comparing two independent pass rates, and make the deterministic checks in §2 carry as much of the load as possible, because they have no judge noise on top of the agent's own variance.

on:
  pull_request:
    paths:
      - "CLAUDE.md"
      - ".claude/rules/**"
      - "prompts/**"

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - run: python -m evals.pairwise --base origin/main --head HEAD --k 3
      - run: gh pr comment "$PR" --body-file evals/out/report.md

Charter eval · task set v3 · judge claude-sonnet-5 · k=3

  deterministic checks     main 84/100   pr 91/100
    money_is_integer       16/20 → 20/20  (the rule you added)
    no_unrelated_files     19/20 → 15/20  ← regression

  pairwise (judged)        pr wins 6 · main wins 4 · ties 10

  VERDICT  REGRESSION on scope creep. The new section appears to
           license refactoring while nearby. Judged difference is
           indistinguishable (tie rate 50%).

That report is the point of the whole exercise. It is specific about the thing that got better, specific about the thing that got worse, and explicitly agnostic about the part it cannot resolve. A reviewer can act on all three statements.

Cost

A full pairwise run is tasks × k × 2 agent invocations plus tasks × k × 2 judge calls. Twenty tasks at k=3 is 120 agent runs and 120 judge calls. Price that against your own provider once, put the figure in the README, and revisit it when the number changes. It is usually small next to one senior engineer spending a day arguing about a paragraph.

Cadence

• On charter pull requests, always. This is the whole point.
• Weekly on main, unchanged charter. This catches drift from underneath you: a model update, a dependency change, a fixture that rotted.
• Never continuously. A loop that runs hourly generates numbers faster than anyone can interpret them, and people start ignoring it by the second week.