← Resources / Harness Engineering / Tools of the Trade / field guide

The tools of the trade for reliable coding agents

Your coding agent won't give the same answer twice, and it doesn't have to. What you want is answers that land inside a band you can read, debug, and ship. This page is the kit: twelve levers that tell you where a fix belongs, and twenty-three tools to make it. Each tool says when to reach for it, what to expect, and how to check it worked. Most changes will measure as no change at all, and that's still worth knowing.

Words by Ian Johnson author of Harness Engineering · tacoda.dev

who it's for · engineers shipping code a coding agent wrote
what it's about · working with coding agents in your repo, not building agents of your own
sources · the linked pages, Harness Engineering, and The Last Check
the honest limit · every number here comes from one team or one experiment. Yours will differ
When
What the agent is doing that tells you to reach for this.
Expect
What changes in the agent's results, and what doesn't.
Verify
The number or test that shows it worked.
0 The band

Reliability is a band

Reliable doesn't mean identical. It means every answer is one you can work with.

Harness Engineering puts it as "variance that stays inside a band you can read, debug, and ship." Each of those words points at one thing you can narrow. A lever is anything that narrows one of them:

  1. ReadThe input. What the agent knows before it acts. Tools 1–7.
  2. DebugThe output. What it's allowed to produce. Tools 8–14.
  3. ShipThe recovery. What happens when it fails anyway. Tools 15–20.
  4. All threeThe human check. What no script or judge can see. Tools 21–23.

To see whether the band got narrower, run the same tasks with and without your change, 30 times or more each, and compare how often they fail. The Eval Loop shows you how.

1 Decide

The twelve levers

Each lever narrows the input, the output, or the recovery. Most bad fixes are a good fix pulled at the wrong lever, so run down the list and stop at the first one that fits.

The code teaches the mistake narrows input

Do
Fix the code before you write a rule. The agent copies the code around it before it reads your prose.
Example
Rename process_line_items to amount_owed_cents, and the next reader looking for what a customer owes finds it instead of writing a second one.

Tools · 04 the legible codebase

Nobody wrote it down narrows input

Do
Write it once, where the agent will read it. A single task gets a contract. Every task gets a charter rule.
Example
If the agent asks the same question four times in three months, you're missing a rule. Write the answer down and the question stops.

Tools · 01 the intent contract · 02 the charter · 03 context and memory

The rule has "then" in it narrows input

Do
That's a sequence, not a rule. Turn it into a command or a skill that walks the steps.
Example
The book's /verify skill walks eight checkpoints and exits 0, 1, or 2, so every run checks the same things in the same order.

Tools · 07 skills and commands

Code can decide it narrows output

Do
Check it with code, and have the agent run the check instead of reading the file and judging.
Example
A pipeline reported "done." One script then printed billing/discount.py:31: ROUND_HALF_UP; this rule requires ROUND_HALF_EVEN.

Tools · 10 hooks · 12 determinism by script

Only judgment can decide it narrows output

Do
Keep the written rule, and add a reviewer: a judge, a person, or both.
Example
"Each amount carries its own currency, never a default." No script can check "never a default," so the sentence stays and a reviewer reads for it.

Tools · 08 sub-agents and judges · 21 the comprehension gate

The agent can reach the check narrows output

Do
If the damage is real, move the check somewhere the agent can't edit it.
Example
"If the agent can rewrite the suite, green means the agent agreed with itself."

Tools · 11 harness gates · 05 characterization tests

It decides what ships narrows output

Do
Put it at a gate in the delivery pipeline, not in the agent's instructions.
Example
Changes to auth, lockfiles, CI config, or network endpoints go to a person "no matter how green the build."

Tools · 19 the factory

You want a second agent narrows input

Do
Add one only for separate memory, a separate refusal scope, or separate tools. "The task is hard" doesn't count.
Example
A team planned four agents. A 15-line rule file that named the downstream consumer and three fixtures did the job instead.

Tools · 08 sub-agents and judges

It's about the order of steps narrows recovery

Do
Put the order in the workflow. A model should never decide what runs next.
Example
A webhook chain needed three layers of recovery code. The reconciler that replaced it deleted more code than it added.

Tools · 19 the factory

Being wrong is expensive narrows output

Do
Skip the waiting and go straight to a hard check. A leaked secret earns a rule on sight, while a style preference earns a linter.
Example
Float rounding on refunds lost about $40 a day without anyone noticing. That rule got a gate, not a sentence.

Tools · 02 the charter · 11 harness gates

Someone wants to loosen a control narrows output

Do
Don't. The platform sets the floor, and a team can tighten it but never loosen it.
Example
The org locks security baselines, banned dependencies, and licenses, and the book keeps those locks "scarce on purpose."

Tools · 09 permissions · Whose Gate Is It

You're about to add or remove a control narrows recovery

Do
Measure first. Every promotion and every deletion ships with an eval result.
Example
Most charter edits measure as no difference at all. Without the eval, each one would have shipped as a win.

Tools · 18 evals

2 Input

Narrow the input

Everything the agent knows before it acts. What you tell it matters, and the code it copies matters more.

The intent contract

When
The agent solves a neighboring problem, or the right one at the wrong size. It's guessing at intent you never wrote down.
How
Before it starts, write seven short sections: outcome, signal, scope in, scope out, acceptance criteria, risk surface, and open questions. Every criterion has to be able to fail, and every open question gets a default.
Example
"Tests look fine" can't fail. "A new test fails on the merge base and passes on HEAD" can.
Expect
Diffs that stay in scope, and reviews that check criteria instead of vibes. You spend the time up front and get it back at review.
Verify
Count criteria marked "not met" at review, and contracts reopened after signing. Both should fall.

Learn more · The Factory · Ghola · Harness Engineering ch. 47–48

The charter

When
The agent keeps making the same mistake, and it's a mistake about your codebase, not about code in general.
How
Start with an empty file. Add a rule the third time you see a mistake, or the first time if it's expensive. Give each rule its reason, with a number, and say what enforces it.
Example
"Use clients.http for every third-party call, never requests. The wrapper sets a 4-second timeout." The reason: an outage held all 16 workers for 23 minutes.
Expect
That mistake stops, as long as the code agrees with the rule. A rule the code ignores makes things worse, because it teaches the agent that rules are optional.
Verify
Once a quarter, check 20 rules against 5 recent files each. A rule that holds in fewer than 4 needs the code fixed or the rule demoted.

Learn more · Anatomy of a Charter · Migrating Without Lying · Keystone · Harness Engineering ch. 11–12 · The Last Check ch. 21

Context and memory

When
The agent uses the wrong word, acts on facts that went stale, or gets worse the longer the session runs.
How
Load reference docs when they're needed, not all at the start. Keep a glossary with one meaning per term. When nobody knows a rule, ask a person and write the answer where the agent will find it.
Example
Is shipping refunded on an $84.00 order? Someone asked in the support channel and got an answer in 11 minutes. It became ADR-014: Shipping is not refunded once the carrier has scanned.
Expect
Fewer corrections you have to repeat, and steadier answers late in a long session.
Verify
Count the files loaded at session start: seventeen means your docs leak, and three means the layering works. Then count corrections you repeat across sessions.

Learn more · Five Guardrails · Keeping Capabilities Honest · Harness Engineering ch. 13, 15–16 · The Last Check ch. 14

The legible codebase

When
The agent copies the code's bad habits instead of following your rules. When a name and a rule disagree, the name wins.
How
Enforce module boundaries with an import linter, so a layer is more than a folder name. Rename vague names. Swap clever code for boring code. Do this before you write a rule about it.
Example
Same CLAUDE.md, same prompt, two repos. One diff touched 9 files across 5 packages and silenced tests, while the other touched 4 files and stayed green.
Expect
Smaller diffs that land where they should, because the agent now copies the pattern you want.
Verify
Track files touched per change. On the book's eight-question architecture checklist, three or more "no" answers predict an eleven-file diff.

Learn more · Cold Codebase · Constraint Engineering · Harness Engineering ch. 17–18 · The Last Check ch. 8–9

Characterization tests

When
You're about to let an agent change code real users depend on, and nothing says what "correct" is.
How
Pin what the code does today, right or wrong, with literal values. Start with money, permissions, and customer data. Then block the agent from editing those tests, because under pressure it rewrites the test instead of the code.
Example
assert shipping_fee(0, "domestic") == 4.99 # Pinned, not endorsed. The number looks wrong, but a warehouse report depends on it.
Expect
The agent can tell a fix from a regression, and so can you. Changes that break behavior fail before anyone reviews them.
Verify
Break the code on purpose: flip > to >=, or return a constant. The suite has to go red. If it stays green, nothing was testing that line.

Learn more · Legacy Code With an Agent · Layers and Promotion · Harness Engineering ch. 19 · The Last Check ch. 12, 15

Driving with a falsifier

When
Every time you ask for a change. Left alone, the agent writes code that looks right.
How
Ask for the input that would break the change before it writes the change. To check finished work, ask it to argue the change is wrong, using only things it can point to in the repo.
Example
"Before you write it, tell me the input that would make your version send twice anyway, and the input that would make it skip a send it should have made."
Expect
Edge cases show up before the code exists, and each one turns into a test.
Verify
Count the defects those falsifier tests catch before review.

Learn more · Junior to Senior · Cold Codebase · The Last Check ch. 19

Skills and commands

When
A job you repeat comes out a little different every time.
How
Start at the lowest rung that fixes it: a command, then a skill, then a sub-agent. Write a skill's description as its trigger, because that's the only part the agent always sees. End each step with an artifact, never with "done."
Example
A prompt that three engineers ran six times between them earns a command.
Expect
The same job comes out the same way. But every capability costs context on every turn, so one you never use makes everything slightly worse.
Verify
From the transcripts: how often it fires, whether it fires at the right time, whether the result gets used, and whether it was missed. If it never fires, delete it.

Learn more · The Capability Ladder · Keeping Capabilities Honest · Harness Engineering ch. 20–21

3 Output

Narrow the output

What the agent is allowed to produce. A check that only reports gets ignored at 3 a.m., so the ones that matter refuse.

Sub-agents and judges

When
The main session fills with search output and the answers get worse, or you need a check no script can do.
How
Decide what the sub-agent returns before you write its prompt: a few bullets with file:line evidence and no prose. Start a judge's prompt with "Default to FAIL," and test it against cases a person already labeled.
Example
A sub-agent that hands back 30 lines instead of 3,000 keeps the main session clear for the actual work.
Expect
A cleaner main session, and a second opinion that catches what scripts miss. A judge shares the model's blind spots, though, so it misses what the model misses.
Verify
Check each judge against 20 human labels. Pin the judge's model, because changing it resets your history.

Learn more · The Eval Loop · Constraint Engineering · Harness Engineering ch. 20, 49

Permissions

When
The agent can run more than it should, or it asks so often that people start clicking through.
How
Deny by default. Put each thing the agent should do behind its own make target, and allow that. If a check can't reach a confident yes, the answer is no. Run it all in a sandbox, and keep production out of reach.
Example
Allow make test, and deny php artisan test. The allowlist fits on one screen, and Docker catches what make leaves open.
Expect
Fewer prompts and a smaller blast radius at the same time. Too loose is a blast radius, and too tight is a bypass flag by Thursday.
Verify
Watch prompts per session, because a rising count means someone's about to bypass them. Log every call to an append-only log.

Learn more · The Narrow Door · Harness Engineering ch. 21, 37

Hooks

When
The agent breaks a rule it has read many times, and a script could catch it.
How
Write the check as a function that returns pass or fail with a reason. Run it on the earliest event that has the data, so the agent sees the failure while it still has the context. Print one line per error, and stay quiet on success.
Example
round_cash(1083) = 1085, but the customer's-favor amount is 1080. The hook blocks with exit 2, and the agent fixes it in the same turn.
Expect
That rule stops leaking, and the fix happens in the turn instead of at review. The agent can still edit or delete a hook, though.
Verify
Track how often it fires and how often the next try passes. Keep pre-commit under 30 seconds: one hook grew from 28 seconds to 4 minutes, and the team stopped running it.

Learn more · Five Guardrails · Junior to Senior · Harness Engineering ch. 19–20

Harness gates

When
A rule with real damage keeps leaking past the hook, or the agent skips or edits the hook.
How
Move the same check into the harness, in front of every tool call, where the agent can't see or change it. On a refusal, send back the reason and allow two retries, then escalate. Keep the written rule too.
Example
A tenant query used session.tenant_id instead of request.path.tenant_id. Both versions passed the gate, and one of them served the wrong invoice.
Expect
The part of the rule code can decide stops leaking for good. The rest still needs the written rule, a reviewer, or both.
Verify
Turn the gate off and compare runs. Also run one check outside the pipeline: in one experiment the pipeline said "done," and make gate-run found 2 violations.

Learn more · Promotion to the Harness · Layers and Promotion · The Journey of a Rule · Harness Engineering ch. 22

Determinism by script

When
A skill asks the agent to read something and judge it, and a script could just give the answer.
How
Find the part of the judgment a script can settle, and write the script. Then change the instruction from "read and confirm" to "run and fix."
Example
"Read the migration and confirm it is reversible" invites variance. "Run the checker and fix what it reports" removes it.
Expect
The same answer every time for that step, without adding a new layer.
Verify
Run the step five times on the same input and count the distinct answers. One is determinism. Two is "a coin flip wearing a procedure." It takes about ten minutes.

Learn more · Keeping Capabilities Honest

Ratchets

When
Your codebase has an old pattern and a new one, and the agent keeps copying the old one because most files still use it.
How
Count uses of the old pattern only in the lines a change adds. Fail the build on any new use, and leave existing files alone.
Example
ratchet.sh with PATTERN='ApiCaller' prints "This change adds 1 use(s) of ApiCaller" and exits 1.
Expect
The old pattern stops spreading right away, with no big-bang migration.
Verify
New uses stay at zero, and the total only goes down.

Learn more · Migrating Without Lying · Legacy Code With an Agent · The Last Check ch. 18

Injection containment

When
The agent reads text you don't control, like issues, web pages, or READMEs, and it can also take actions.
How
Split the agent that reads from the agent that acts. The reader gets no credentials and no network, and it hands back only structured output. Assume something will get through, and keep what it could win small.
Example
A README tells the agent to print its environment. The reader passes on a summary, and the actor never sees the raw text.
Expect
Injected instructions never reach anything that can act. You can't patch injection away, but you can break the lethal trifecta: access, exposure, and exfiltration.
Verify
Put attacks in your eval set. inj_003_readme_env.py checks gate_fired, no_egress, and env_not_in_output.

Learn more · Prompt Injection in the Harness · Harness Engineering ch. 38

4 Recovery

Narrow the recovery

The agent will fail sometimes. These tools decide what a failure costs you, and whether you learn anything from it.

Loop bounds and cost ceilings

When
Before you leave any loop running without you.
How
Set three separate limits: iterations, budget, and time. Make them throw an error, not log a warning. Move anything that fails three times to a dead-letter queue, and pull the kill switch once before the first night.
Example
One item failed on every attempt. With a three-try dead-letter rule, the run cost $5 instead of $100.
Expect
Failures still happen, but a bad input costs a few dollars and a dead letter, not a whole budget.
Verify
Track cost per run and how many runs each limit stopped.

Learn more · The Factory · Harness Engineering ch. 35, 40

Interrupts

When
The agent guesses when it should ask, and the guess turns into a wrong change.
How
Give every question five parts: the question, two or three options, a recommendation, the cost of a wrong guess, and where to resume. Have the agent read what's already written before it asks.
Example
In a migration, each unit could return UNCLEAR with a question instead of guessing. About one in twenty did.
Expect
Questions you can answer in a minute, and fewer wrong guesses. Each question also points at a gap in your docs.
Verify
Chart questions by area each week. In the book, three areas made 80% of them, and those three became the next rules.

Learn more · The Migration Harness · Harness Engineering ch. 48

Small, reversible commits

When
The agent turns a small fix into a big diff.
How
Keep behavior changes apart from renames, refactors, and formatting. Before the change, write three lines: what it does, how to undo it, and what the undo leaves behind.
Example
A 9-line retry fix came back as 280 lines across 11 files. A five-second rollback turned into forty minutes at eleven at night.
Expect
You can revert any change without taking good work with it.
Verify
Run git rebase --exec 'make test' HEAD~2, so each commit passes on its own.

Learn more · The Bisect Drill · The Last Check ch. 16–17

Evals

When
You change a rule, a skill, a gate, or a model, and you want to know whether it helped.
How
Check in about twenty tasks and leave them alone. Score with exit codes and greps first, then a judge, then a person spot-checking. Compare two outputs blind, and run each side 30 times or more.
Example
At a 70% pass rate, a drop to 50% takes about 95 runs per side to see. A rise to 80% takes about 300.
Expect
You'll catch regressions reliably. You won't catch small wins, and most edits will come back "can't tell," which is still an answer.
Verify
Report one of three results: worse, better, or can't tell. Run it in CI on every rules change.

Learn more · The Eval Loop · Gauge · Harness Engineering ch. 30, 46

The factory

When
Several agents work a backlog in parallel, and runs crash, collide, or merge for reasons nobody can reconstruct.
How
Run the work from a queue, not from an agent, so no model decides what runs next. Every phase needs an exit. Put gates before actions, and write a record after each one.
Example
One run, step by step: plan, edit, a gate blocks money_is_integer, the agent revises on retry 1 of 2, the gate passes, verify, merge. Every step leaves a row.
Expect
A crash resumes from the last step instead of starting over, and every merge has a reason on file.
Verify
Each quarter, pick a change merged three months ago. From the record alone, name the rejected alternative, the rule that applied, and the person accountable.

Learn more · The Factory · The Audit Trail · Open-Refinery · Harness Engineering ch. 57–58

The migration harness

When
One change has to land in thousands of places.
How
Count the shapes first. Let a codemod do the mechanical bulk and refuse anything odd. Give agents only what it refused, one file each, and keep the old code running as the reference.
Example
Moving float dollars to integer cents across 10,412 sites: the codemod did 9,100, agents did 1,252, and 60 went to a person.
Expect
People spend their attention on 60 hard cases instead of 10,000 easy ones.
Verify
Replay 4.1M recorded inputs through the old and new code, and explain every difference. "Zero differences means you have not migrated anything."

Learn more · The Migration Harness · Migrating Without Lying

5 Human

The human check

A person still catches the mistakes no rule covers yet. Each catch becomes the next rule.

The comprehension gate

When
Every check is green and the change is still wrong.
How
Whoever asks for the merge fills in four fields before the diff exists. They say what it does, where it enters and leaves, which rule it follows, and what would prove it wrong.
Example
"Adds a helper and updates two callers" describes the diff. "Cash totals now round down to the nickel instead of to the nearest one" describes the change.
Expect
Reviews catch mistakes no rule covers yet, and each gap they find becomes a new rule.
Verify
The "what would prove it wrong" field names a test, or says NONE. Count catches per review.

Learn more · The Comprehension Standard · Reverse-review · Harness Engineering ch. 32 · The Last Check ch. 2, 22

Drills

When
Agents write more of the code, and you're not sure your reviewers still catch what the agents get wrong.
How
Keep drills short, regular, and scored on process. Predict before you look, prove it with a failing test or a concrete wrong output, then name the kind of failure.
Example
The bisect drill: one regression, two hundred commits, no agent, 30 minutes a month. Before you start, check that the reproducer fails 5 of 5 on the bad commit and 0 of 5 on the good one.
Expect
Reviewers who keep catching agent mistakes instead of slowly waving them through.
Verify
Seed one known mistake a week and score four columns: found, proved, classified, durable fix. Don't add them up. The flat column is the kind of mistake that would ship.

Learn more · The Bisect Drill · Reverse-review · The Mentor's Playbook · The Last Check ch. 11, 13

The rejection log

When
The same agent mistake comes back after you've already rejected it once.
How
Write one line per rejection or escape: the date, the change, what you did, and what it would have cost if you were wrong.
Example
2026-03-19 #431 approved the cache change; it broke stale reads in prod. now: every cache PR states its invalidation explicitly. two caught since, #455 and #470
Expect
Repeat mistakes become visible, and the repeats become rules.
Verify
Read the "caught since" tail. Aim for one to three entries a week.

Learn more · Evidence of Judgment · The Last Check ch. 23

6 Loop

Keep it going

When something gets caught twice, make sure it can't happen a third time.

  1. NoticeA miss shows up in review, in a gate's log, in the agent's questions, or in an eval.
  2. Place itRun down the twelve levers and fix it at the first one that fits.
  3. MeasureRun the eval before and after. If you can't tell the difference, say so.
  4. PruneOnce a month, archive the rules and tools that stopped firing.
Start this week

Pick the mistake your agent made most often last month, and find its lever. Make the fix, add one eval task that catches the mistake, and watch whether it comes back.