Harness Engineering / Reverse-review / 30 minutes, weekly

The junior reviews the agent. The senior reviews the review.

Code review was built for a world where a human wrote the code and the scarce thing was correctness. The agent writes the code now, and the scarce thing is judgment. So point the review the other way: the junior reviews the agent's diff, then defends that review, and the senior grades the review rather than the code. It costs a senior fifteen minutes instead of an hour, and the fifteen minutes land on the only thing that actually transfers.

seeded diffs and answer keys · github.com/tacoda/fulcorum-reverse-review
who it's for · early-career engineers, and the senior who has thirty minutes a week, not five hours
what you need · one agent-written diff, one review card, one timer
the bar it trains for · the Comprehension Standard, L2 and up
The diff
What the agent produced. Plausible by construction, because that is what the model is good at.
The catch
A defect named, classified, and located. A named class transfers to the next diff. A hunch does not.
The proof
A test that goes red. Without it you have a suspicion, and suspicions do not survive a disagreement.
0 The idea

Turn the review around

One diagram, and the rest of the page is how to run it.

where the senior's attention goesthe whole idea
NORMAL REVIEW
  junior writes  ──▶  senior reviews the code
                      ~1 hour, spent on one diff, mostly on defects

REVERSE-REVIEW
  agent writes   ──▶  junior reviews the code
                 ──▶  senior reviews the review
                      ~15 min, spent on judgment, which is the part that transfers

The agent already writes acceptable code most of the time, so a senior line-editing that code is spending scarce attention on the abundant thing. What is scarce is somebody who can look at a fluent, passing, confident change and say "wait." That is not knowledge you can be handed. It is trained by being wrong, repeatedly, in front of someone who can tell you why. Reverse-review is a container for exactly that, with a fixed cost.

There is a second effect worth naming, because it is most of why this works in practice. The junior is not defending their own code, so nothing about the exercise threatens them. Being wrong about a diff the agent wrote is cheap and unembarrassing, and people look harder when looking harder cannot backfire. Normal review has the opposite property, which is why so much of it turns into negotiation.

The trade

You give up some line-by-line senior scrutiny of the code. You get a junior who can supply that scrutiny themselves within a couple of quarters. On any timescale longer than a sprint, that is a good trade.

1 The loop

Thirty minutes, three roles

Thirty minutes is the whole budget, and the timeboxes are not decorative. An exercise that quietly grows to an hour is an exercise that gets cancelled in week five.

dealer picks the diff reviewer the junior grader the senior
  1. Before The dealer picks a diff. Either real agent output from this week's work, or a seeded diff from section 8. Under about 60 lines. The dealer does not say what is wrong with it, or whether anything is.
  2. 0:00 → 0:05 Read cold. No agent, no ticket summary, no running the tests. Just the diff and the surrounding code. Five minutes is deliberately not enough time to be thorough, which is the same condition a real review happens under.
  3. 0:05 → 0:20 Review. Fill the card (section 2). Write the prediction line before running anything. Then run the tests, read the callers, use the agent as a reference if you want. Fifteen minutes, then pens down whether or not you are finished.
  4. 0:20 → 0:30 Defend. The junior states the verdict first, in one sentence. The senior asks questions, does not lecture, and finishes by scoring the review against the rubric (section 4) and naming one thing to do differently next week.
Pens down means pens down

A review that ran out of clock is a legitimate outcome, and worth saying out loud: "I did not get to the migration, so I am not confident about the data path." That sentence is a senior-shaped sentence. Knowing the edge of what you checked is a large part of the skill, and an exercise with an unlimited clock never teaches it.

2 The artifact

The review card

One page, filled by hand, kept afterward. The top half is the comprehension block from the standard. The bottom half is what makes it a review instead of a summary.

the review cardprint it, fill it, keep it
REVIEW CARD          diff: __________  reviewer: __________  date: ______

THE CLAIMS  (what the author should have been able to say)
  does:      ________________________________________________
  seam:      __________________ -> __________________
  policy:    __________________________________  | CHARTER-GAP
  falsifier: ________________________________________________
             covered by: _____________________   | NONE

THE REVIEW
  prediction:   what I expect to be wrong, written BEFORE I run anything
                ______________________________________________
  verdict:      ACCEPT  /  ACCEPT WITH CHANGE  /  REJECT
  defect class: ______________________________  | NONE FOUND
  proof:        failing test __________________ | unproved suspicion
  durable fix:  fix the code / write the rule / add the hook / nothing
                because: ____________________________________
  1. prediction Written first, before the tests run, before the agent is consulted. Being wrong on paper is free, and it is the only way to find out whether your instincts are calibrated. A reviewer who never predicts never learns that they are consistently wrong about the same class of thing.
  2. verdict Committed before the discussion, in one sentence. A verdict you can revise mid-conversation is not a verdict, it is agreement with whoever spoke last.
  3. defect class Named from the taxonomy in section 3. The class is what carries to the next diff. "There is a bug on line 40" teaches you about line 40; "this is a policy inversion" teaches you where to look forever.
  4. proof A test that goes red, or a concrete input and the wrong output it produces. Anything less is written down honestly as unproved suspicion, which is a perfectly respectable thing to hand a senior. It is not a catch.
  5. durable fix Which surface stops this from recurring: fix the code, write the rule the agent reads, add the hook that refuses it, or deliberately do nothing because the defect is not worth the constraint. Saying "nothing, because" is a real answer and often the right one.
3 The taxonomy

Seven ways agent code is plausible and wrong

"Look harder" is not a technique. A named list of failure shapes is. These seven cover most of what shows up in practice, and every one of them can ship with a green pipeline.

the plausible-but-wrong taxonomy, v1
ClassWhat it looks likeHow you catch it
1 · Policy inversion Textbook-correct behavior that is not this business's behavior. Rounds to the nearest nickel when the policy is round down, retries when the policy is fail fast. Ask what the business rule is, not what the algorithm is. If nothing in the repo answers, you have found a charter gap, which is its own valuable result.
2 · Silent representation change Integer cents become a float, a decimal becomes a string, a set becomes a list, a UTC timestamp becomes local. Diff the types, not the logic. Scan for a new cast, a division, a bare numeric literal, a format call.
3 · A test that cannot fail The assertion re-derives the implementation, or asserts against a mock the test itself configured. Reads like coverage, proves nothing. Break the implementation on purpose. If the test stays green, it was never a test.
4 · Swallowed error except: pass, .catch(() => null), a default value that hides a failed lookup, a log line where a raise belongs. Find every place in the diff where a failure stops travelling. Ask what the caller now believes happened.
5 · Scope creep The requested one-line fix arrives with an unrequested rename, reformat, or refactor riding along. Read the ticket, then the diff. Anything not traceable to the ticket is a separate decision that nobody approved and nobody reviewed as a decision.
6 · Invariant drift The code changed; the docstring, type hint, comment, or error message still promises the old contract. Read the prose next to the code. When code and prose disagree, one of them just shipped a lie, and the prose is what the next reader will trust.
7 · Imitation of the wrong pattern The change faithfully copies the dominant pattern in the codebase, and the dominant pattern violates the rule. Compare the diff against the rule, never against its neighbors. The more non-compliant code exists, the more confidently the agent reproduces it.
Why the suite does not save you

Every one of these seven can pass a full test run. Some pass because the case was never tested (1, 2, 6), some because the failure was caught and hidden (4), one because the test is a decoration (3), and two because the code is doing exactly what the codebase told it to (5, 7). Green means the tests that exist passed. It has never meant the code is correct, and the agent has made the gap between those two statements much wider.

Seven is not a complete list and should not be treated as one. It is a starting vocabulary. When your team catches something that does not fit, name it, write the row, and add it to your fork. A team with its own eighth and ninth class is a team that has been paying attention.

4 The rubric

Score the review, not the diff

Four points. The score itself does not matter much. Which point somebody loses, week after week, is the entire diagnostic value.

reverse-review rubric, 4 points
PointEarned whenLost when
Found Named a real defect, or correctly concluded there was not one. Missed a planted defect, or raised one that is not there. A false positive costs exactly what a miss costs.
Proved A failing test, or a specific input and the specific wrong output it produces. "It feels off." Real instinct, zero points. Instinct is where a catch starts, not where it ends.
Classified Named the class from the taxonomy. Described the symptom only. The class is the part that generalizes to next week's diff.
Durable fix Proposed the change that stops the class, and said which surface it belongs on. Patched this instance and stopped. Or reached for a hook when a two-line code fix was the honest answer.

A perfect four is rare and is not the goal. The useful signal is the pattern: a reviewer who always loses proved needs practice writing failing tests, not more diffs. One who always loses durable fix is reviewing code but not yet reviewing the system. Those are different coaching problems and the rubric tells them apart in about three weeks.

Deal clean diffs, and do not announce it

If every diff in the exercise contains a planted defect, you are not training reviewers, you are training a false-positive machine that will find a bug in anything because it has learned that finding one is how you pass. Roughly one in four should be genuinely clean. Correctly accepting a clean diff, with reasons, is a full four points and should be praised exactly as loudly as a catch.

5 Facilitation

The senior's run sheet

Ten minutes of talking, and most of the skill is in what you do not say. The failure mode is a well-meaning senior turning the defend phase into a lecture about the diff.

  1. Don't Give the answer first. Ask instead: "what would have to be true for this to be wrong?" It is the question that generates the skill, and handing over the answer spends the whole exercise to save four minutes.
  2. Don't React to the diff in front of them. The moment you frown at line 40, the search is over. Keep your face out of it until the verdict is committed.
  3. Don't Let it become a code review. If you find yourself explaining how you would have written it, the exercise has reverted and the hour is gone. The diff is the material, not the subject.
  4. Do Say what you distrusted, and why, after the verdict. Seniors flinch at certain diffs before they can articulate the reason. Articulating it afterward is the single most valuable thing you have to give, and it is exactly what never comes up in normal review.
  5. Do Treat a CHARTER-GAP as your bug. If "why is it written this way" is not answered anywhere in the repo, the team owes them that answer, and it should be written down that day. This is the exercise paying rent back to the codebase.
  6. Do End with one sentence. Which rubric point they lost and what to do differently next week. One. A list of five improvements is a list nobody acts on.
Fifteen minutes, protected

Ten minutes in the room, five minutes picking the diff. If you are routinely spending an hour, the exercise has drifted back into normal review, and the economics that made it worth doing are gone.

6 No senior

Running it without a senior

The senior is the scarce input, and plenty of teams do not have one with time to spare. Here are the substitutes, in descending order of how well they work, and what none of them replace.

  1. Best Two juniors, same diff, separate cards. Fill them independently, then compare cards before comparing either to any answer key. Where you disagree is the lesson, and arguing a verdict with a peer is most of what the defend phase does anyway.
  2. Workable Seeded diffs with a written key. Grade yourself against section 8. Weaker, because you know in advance that a defect exists, which is the single biggest thing the exercise is supposed to withhold. It still teaches the taxonomy, which is worth a lot on its own.
  3. Workable The agent as adversary. Ask it to make a small change, then open a fresh session to review it. Never the same session: it defends its own work, agrees with its own reasoning, and will happily talk you out of a correct catch.
  4. Not a substitute Someone who knows what correct means here. Policy inversion (class 1) is undetectable without knowing the business rule. That knowledge is local, undocumented more often than not, and lives in a person. No model has it, and no exercise manufactures it.

Which is worth saying plainly, because it is the honest limit of this page: reverse-review makes a senior's time go about four times further. It does not create one out of nothing. If your team has nobody who can say what the business rule actually is, that is a much larger problem than your review process, and it will not be solved by an exercise.

7 Cadence

Weekly, and measured in catches

One card a week beats a heroic month and then nothing. Keep the cards; the stack is the record of somebody becoming senior, and it is far better evidence than a ticket count.

A catch is a named class plus a proof. Not a comment, not a suspicion, not a style objection. Counting catches instead of tickets changes what people optimize for, which is the only reason to count anything.

  1. 4 weeks Fills the card without prompting. Catches the loud classes: swallowed errors, scope creep, drifted docstrings. Still needs a nudge to write the proof.
  2. 12 weeks Catches the quiet classes: policy inversion, imitation of the wrong pattern. Proves them with a failing test unprompted. Correctly accepts clean diffs, which is the harder half.
  3. 6 months Reviews the diff and the charter at the same time. Files the CHARTER-GAP that stops the class for everyone, and can argue for doing nothing when a constraint would cost more than the defect.

Those timelines are what this has looked like in practice on small teams, not a measured curve, and someone doing one card a week in a codebase they know well will move faster than the numbers suggest. Treat them as a shape, not a schedule.

The habit is the deliverable

No single catch matters. What matters is that thirty minutes a week goes to the one skill the agent cannot supply, for long enough that it becomes how somebody reads code.

8 Seeded diffs

The seeded diffs

One diff per defect class, plus clean ones, applied to the cash-register seed from the junior-to-senior workshop. If you did that workshop you are already set up.

seed fulcorum-junior-to-senior-workshop/seed diffs fulcorum-reverse-review/diffs/
deal one diffgreen, and that proves nothing
git clone https://github.com/tacoda/fulcorum-junior-to-senior-workshop.git
git clone https://github.com/tacoda/fulcorum-reverse-review.git

cd fulcorum-junior-to-senior-workshop/seed
pytest                        # 3 passed, this is your baseline

# the dealer picks one and does not say which class it is
git apply ../../fulcorum-reverse-review/diffs/03-test-that-cannot-fail.diff
pytest                        # still green. start the clock.

# after the defend phase
git apply -R ../../fulcorum-reverse-review/diffs/03-test-that-cannot-fail.diff

What ships in the diffs folder

  • Seven defective diffs, one per class in section 3, all of which leave the suite green.
  • Three clean diffs that make a real, correct change. Deal these regularly and never flag them.
  • Answer keys in a separate folder so the dealer can hand out a diff without reading its solution first. A dealer who knows the answer telegraphs it.

The filenames give away the class, which is fine for self-study and useless for a real session. Rename them to a.diff through j.diff before you deal them, or have the dealer apply the diff for the reviewer.

Better than the seed: your own repo

Seeded diffs teach the taxonomy. Your own codebase teaches the thing that actually matters, because class 1 and class 7 only exist relative to a real policy and a real prevailing pattern. Once the vocabulary is in place, switch to reviewing actual agent output from last week's work and never go back.

Where this sits

The Comprehension Standard sets the bar this exercise trains for. The junior-to-senior workshop is the one-hour lab that introduces the comprehension card, a green test suite, and a customer quietly overcharged a nickel.