Harness Engineering / The Mentor's Playbook / the senior's half

Grading a review is a different skill from doing one.

Reverse-review documents what the junior does and gives the senior ten minutes of facilitation advice. That advice is enough for week one and runs out somewhere around week five, when the questions get harder: what makes a planted defect worth missing, why two mentors score the same card two points apart, and what it means when someone earns the same three rubric points for eight weeks running. This is the other half.

who it's for · the senior running reverse-review, and the lead running four of them at once
prerequisite · the exercise itself. Its §5 covers the ten minutes in the room; this page does not repeat it
what you need · a diff pool the reviewer cannot see, a score sheet, and twelve weeks
pairs with · the Seniority Ladder, where this is most of the work of R3
The job
Calibrating somebody else's confidence. Not finding defects: you already can, and that is not what is being transferred.
The drift
Every failure mode here ends the same way, with the exercise quietly reverting to a normal code review.
The stop
The point where you change something: the diffs, the cadence, the rubric, or the conversation.
0 The premise

You are not grading correctness

You already know whether the diff is wrong. The thing being measured is whether their confidence tracked the evidence, and that is a separate question with a separate answer.

The natural instinct in the defend phase is to compare their verdict to yours and score the distance. That instinct is why most attempts at this drift back into normal review by week five. A reviewer can arrive at your exact verdict by luck, by pattern-matching the rubric, or by having seen the diff earlier in the week, and none of those are the skill.

What you are actually watching is the relationship between two things: how sure they said they were, and how much they had. A confident catch backed by a red test and a confident catch backed by a feeling look identical on the card and are opposite results. Most of the technique on this page exists to pull those two apart.

the four cells you are sorting intowhat to praise, what to press
                    HAD EVIDENCE        HAD NONE

  WAS SURE        the goal            the dangerous cell
                    praise it            press hardest here, even
                                         when the verdict is right

  WAS UNSURE      underclaiming        honest, and normal
                    teach them to        in weeks one to four
                    commit to what
                    they proved

The top-right cell is the one that matters, and it is the one a correctness-focused mentor never sees, because when the verdict is right there is nothing to correct. Press it anyway. A junior who is right for no reason will be wrong for no reason next month, with the same delivery.

1 The dealer

Seeding a defect worth missing

A good planted defect is one a competent person misses for a reason. A bad one is merely hidden, and hiding things teaches nothing.

rule one defect per diff rule must survive the suite size under 60 lines

Five conditions for a usable seed

  1. One only Exactly one defect. Two in one diff means finding either can be reported as success, and neither miss can be diagnosed.
  2. Visible Detectable from the diff plus the surrounding code, with no knowledge the reviewer could not have had. A defect requiring the ticket tests memory, not judgment.
  3. Survives green The suite must pass. If your pipeline catches it, you have tested your CI and wasted the reviewer's thirty minutes.
  4. Well dressed The surrounding code is clean and idiomatic. Defects nested in obviously bad code teach "distrust bad code", which everyone already does.
  5. Near The correct version should be a line or two different. The reviewer's near-miss (right line, wrong class) is the most useful thing the exercise produces, and it only happens when correct and incorrect are close together.

Seeds that waste a week

  • The trick question. An off-by-one buried inside an unfamiliar API. It measures exposure to that API, which is not on the rubric and never will be.
  • The two-defect diff. Usually authored when the first defect felt too easy. Split it into two weeks instead.
  • The language edge case. Mutable default arguments, integer caching, truthiness of an empty collection. Fun, memorable, and almost never the shape real agent defects take.
  • The one you find clever. Cleverness in a seed correlates strongly with unteachability. If explaining the answer takes you four minutes, it was the wrong seed.
The best source, by a wide margin

Last year's real incidents, re-planted. They are already the right shape, they already survived a real pipeline, and the debrief comes with a story about what it cost. A team that converts each postmortem into one seeded diff has a training set nobody can buy.

2 Calibration

Two mentors, one card, two scores

If two seniors score the same review differently, the rubric is measuring the mentor. Find out once a quarter, in twenty minutes.

Any team with more than one mentor has this problem and almost none of them measure it. The junior experiences it as arbitrary: the same quality of work scores three one week and one the next, depending on who was in the room. That is the fastest way to convince somebody the whole exercise is theatre.

The drill

  1. Pick three Three completed cards from previous weeks, ideally spanning a good one, a poor one, and a genuinely ambiguous one.
  2. Score blind Both mentors score all three independently, in writing, before speaking. Same anchoring problem as any calibration: whoever talks first sets the number.
  3. Compare Target is agreement within one point on eight of ten cards over time. Wider than that and the rubric is not shared, whatever everyone believes.
  4. Localise Find which rubric point the disagreement concentrates in. Write your team's local definition of that point into your fork, with an example on each side.
It is almost always "durable fix"

One mentor accepts a regression test as a durable fix; another demands a lint rule or a gate. Both are defensible and the reviewer cannot read minds. The resolution is not to argue it in the abstract but to write down your team's altitude rule: which classes of defect earn a sentence in the charter, which earn a check, and which earn a gate that refuses. That document is useful far beyond this exercise.

Second most common is "proved", where the disagreement is whether a specific input plus its wrong output counts, or whether only a committed failing test does. Pick one, write it down, and stop relitigating it weekly.

3 The trend

Reading twelve weeks of scores

One card tells you almost nothing. Twelve tell you which of six quite different coaching problems you have.

Keep a four-column sheet, one row per week: found, proved, classified, durable fix. Do not total it. The total is the least informative number on the page, and writing it down invites everyone to optimise it. What you want is which column is flat.

trajectories and what each one actually is
Pattern over 6+ weeksWhat it usually isThe move
Found rising, proved flat Catching by instinct, cannot evidence it. Will fold under any pushback. Failing test before they say a word, every week for a month.
Classified always earned, found often lost Learned the vocabulary, not the search. Naming classes from memory. Deal two clean diffs in a row. See §4.
Durable fix always lost Reviewing the code, not the system. A very common ceiling. Pair them on one charter rule. This is the on-ramp to R4.
All four earned by week six Either exceptional, or your diffs are too easy. The second is more likely. Raise difficulty once. If it holds, move to live diffs early.
Found rising, false positives rising Trained to produce findings. Your clean-diff ratio is too low. Go to one clean diff in three and say nothing about it.
Everything flat for eight weeks Not a rubric problem. See §8 before concluding anything about the person. Audit your own inputs first.

The two flagged rows are the ones that get misread as effort problems. Both are usually produced by the mentor: the first by a diff pool that is always defective, the second by material that is too hard, too easy, or too samey to generate any signal at all.

4 Gaming

Four ways a rubric gets played

Not cheating. Ordinary, mostly unconscious optimisation toward the thing being measured, and the reason it needs naming is that it is invisible in the score.

  1. Class-stuffing A class gets named for every finding, because naming one is worth a point. The classes are increasingly decorative. Counter: require the class to predict a second place in this repo where the same class could occur. Memorised vocabulary cannot do that; understood vocabulary does it easily.
  2. The hedge "Possibly a policy inversion, or maybe invariant drift." Covering the space so that something lands. Counter: one class per finding, committed in writing before the defend phase opens. A hedge after commitment is fine and honest. A hedge instead of commitment is the game.
  3. Test theatre A failing test that fails for a different reason than the defect. Looks like proof, scores like proof. Counter: revert only the defective line and ask whether their test goes green. If it stays red, it was never testing the defect.
  4. The pre-read They saw the diff in the queue on Tuesday. Not dishonest, and it silently invalidates the rep. Counter: deal from a pool they have no access to. When that is impossible, ask up front, and discount the card rather than pretending.
Do not punish any of these

All four are evidence that the rubric is being taken seriously, which is better than the alternative. Name the game, apply the counter, move on. A mentor who treats optimisation as dishonesty gets a reviewer who optimises more carefully and stops telling you things.

5 Week 8+

The pushback drill, run safely

Arguing the wrong side on purpose is the only way to find out whether a correct finding survives disagreement. It is also the fastest way to damage the whole exercise if run badly.

never before week 8 never on their own code always reveal, same session

Roughly one review in three, push back on a finding you know is correct. Calmly, with a plausible reason, no tell. What you are measuring is whether their position is attached to evidence or to your face.

Rules for the mentor

  1. Reasons only Argue with an actual argument. Never with authority, never with tenure, never with "I've seen this a hundred times." Authority tests compliance, and compliance is what you are trying to remove.
  2. Two rounds Push twice, then stop. Beyond that you are no longer testing whether the position holds, you are testing whether they will endure you.
  3. If they fold Do not say "actually you were right." Say: "walk me back through why you wrote it down." Make them rebuild the reasoning under their own power first, then reveal. The rebuild is the whole lesson.
  4. Always reveal Every time, in the same session, before anyone leaves the room. A drill discovered later, secondhand, reads as a trick and costs you the next six months of honest answers.
Two hard boundaries

Never run this on a review of their own code, where the social cost of holding is real rather than simulated. And never run it before week eight. It requires an existing relationship to be a drill instead of an ambush, and there is no way to shortcut that.

The target failure mode is worth stating plainly, because it is what justifies the risk. An engineer who cannot find defects is early in their career. An engineer who finds them and abandons them whenever a confident person disagrees will still be doing that at eight years in, and by then nobody is checking whether they were right.

6 45 minutes

Four juniors, one senior, forty-five minutes

The answer to "we have six early-career engineers and one person with time." Cost per reviewer drops to about eleven minutes.

  1. 0:00 → 0:05 Same diff to everyone. Read cold, no agent, no ticket, no tests.
  2. 0:05 → 0:20 Independent cards. Silence in the room. No screen sharing, no thinking aloud.
  3. 0:20 Simultaneous reveal. All cards visible at once. One confident person speaking first would erase three independent results, which is the only real risk of the group format.
  4. 0:20 → 0:40 They compare cards to each other before anyone compares to the key. Disagreements between peers do most of the teaching, and the mentor stays quiet through this part.
  5. 0:40 → 0:45 Mentor adjudicates only what the group could not settle, then gives each person one sentence. Four sentences, not one lecture.

The group format is better than the one-to-one version in one specific way: hearing three peers reach three different verdicts on the same lines is a more convincing argument for humility than anything a senior can say. It is worse in one way too, which is that quiet people can hide. Rotate who presents their card first, and read the written cards rather than trusting the discussion.

Scaling honestly

Four is the working maximum. At six, the compare phase stops converging and the mentor reverts to lecturing to save time. Two groups of four beats one group of eight, even when the mentor is the same person on the same afternoon.

7 Exits

Three exits, including the good one

The exercise is supposed to end. A weekly ritual that runs forever has become furniture.

  1. Seeded → live Four reps completed with at least two correct classifications. The miss rate will jump immediately and that is expected, because live diffs mostly contain nothing wrong.
  2. Live → self-directed Three catches in live diffs, and one correct position held under deliberate pushback with evidence. At this point they are the last check on someone else's change, which is R3 on the ladder.
  3. The inversion When they catch something you missed, twice running, swap the roles. They deal and grade; you review. It costs nothing to try and it is the most honest graduation ceremony available.

That last exit deserves defending, because it feels uncomfortable and gets skipped. A mentor who has never been graded by the person they trained has no evidence the training worked, only an impression. The swap produces evidence in one session, and if the result is embarrassing, that is a fact you needed and did not have.

8 The hard one

Twelve weeks, no movement

Audit your own inputs before you conclude anything about the person. Three of the four common causes are on the mentor's side of the table.

In this order

  1. The diffs Were they all the same class? All too hard? All defective? A pool with no variance produces a flat sheet regardless of who is reading it.
  2. The cadence Did it actually run twelve times, or six times with gaps? Fifteen minutes weekly beats an hour monthly by a wide margin, and a skipped week is usually the mentor's diary rather than the junior's.
  3. The load Somebody delivering forty hours a week of feature work has no attention left for a training exercise, and will correctly deprioritise it. That is a staffing decision showing up in a score sheet.
  4. Then, the person Only after the first three are ruled out. And then say it specifically.
Say the specific thing

"You are not improving" is unusable and lands as a verdict on character. "You have lost 'proved' eleven weeks running, we tried writing the test first and it did not move, and I do not have another idea yet" is usable, honest about your own uncertainty, and gives them something to push against.

Then distinguish three cases, because they need different responses. Not yet is the most common and deserves more time and a different approach. Cannot is rare, hard to establish, and should never be concluded from one exercise in isolation. Will not is not a mentoring problem at all: it is a management conversation about whether the person wants this job, and running more reps will not surface it any faster.

9 Your side

Five tells that the mentor is the problem

Every one of them is comfortable, defensible in the moment, and ends with the exercise becoming a code review again.

  1. Grading the diff Your feedback is mostly about the code. Reverse-review §5 warns about this for a single session; the trend version is worse, because it is invisible week to week and total by week ten.
  2. Never losing You have never conceded a point in a defend phase. Either you are exceptional or you are using authority, and the base rate strongly favours the second.
  3. Clever seeds Your planted defects are the kind you would tell a friend about. Cleverness and teachability are close to inversely correlated here.
  4. Unarticulated taste You cannot say why you distrusted a diff, only that you did. Fine as an instinct, useless as instruction, and it is the single most valuable thing you own. Getting it into words is what turns it into a charter rule.
  5. Skipped weeks The cadence is the intervention. An exercise that runs when the calendar is kind teaches that it is optional, which is precisely the lesson you are trying not to teach.
Where to start

If you take one thing from this page, take the fourth tell. Spend the last two minutes of every session saying out loud what made you uneasy and why, including the times you were wrong. It is the part of your expertise that never surfaces in normal review, it costs nothing, and it is the only thing here that transfers directly.