Harness Engineering · A Talk

The journey of a rule

A constraint is born in a code review, and it needs somewhere to live. So it tries six homes. Each one takes it in, but each one makes it leave something at the door. In the end it stops looking for a single home.

Who's talking

Ian Johnson

  • Staff engineer at Parento. Agents against a production codebase, daily.
  • Founder of Fulcorum. Video courses on shipping AI-generated code.
  • Author of Harness Engineering, and of ghola, where the ladder in this talk ships as a worker.
Months of this, not years. So I will show you the runs, and name the parts that are still opinion.
The next thirty minutes

A fable, a measurement, a moral.

One · the fable

One constraint goes looking for a home. It is invented for this talk, and I will say so again where it matters.

Two · the instrument

A harness enforcing one real rule at six rungs, with a switch for each. Twelve runs. Two findings were bugs in my own report.

Three · the moral

Three questions for every constraint you own. Which rung carries it, how strict it is, and who can still reach it.

One · meet the constraint

The whole constraint. All five words of it

No query crosses a tenant.

Five words. Everyone agrees with it on sight, and nobody can hand you a test that proves the codebase obeys it. Break it and nothing complains: tests pass, latency is flat, the page renders, and someone else's invoice renders on it. That silence is why it has to go looking.

The constraint is invented and the tenant is imaginary. The search for a home is the real part.

Six doors

Six homes, three borders between them.

CONTEXT is it in the turn? ARGUMENT can it be reinterpreted? REACH who can edit the enforcer? a head tribal knowledge a document it gets written down the charter it gets heard a hook it gets hands a judge it gets understood a predicate it gets obeyed
What to watch at each door

Two things move, and they move against each other.

Force

How hard the constraint is to ignore. People ignore a rumor by accident. Nobody can ignore a predicate in CI, not even you at 6pm on a Friday.

Meaning

How much of the sentence got through the door. Prose carries the whole constraint, judgment included. A predicate carries what a machine can settle.

Force rises as it travels. Meaning is what each door asks it to leave outside.
Homes 1 & 2 a head, then a document

It starts in a head, then gets an address, and an address is not a summons.

The houses
The reviewer who has seen this bug before. Then a design doc, a wiki page, an architecture decision record.
They ask
A head asks nothing at all. A document asks that somebody write it down and keep it current.
Left at the door
Nothing, which is the problem. The document paid rent and bought no force.
Why it leaves
The reviewer changes teams, and because nothing was written down, nothing visibly goes missing. Nobody reads the page, so the code drifts while the page does not, and a year later it is wrong and still authoritative.
Force1/5
Meaning5/5
Your agent cannot overhear anything, and nothing ever goes looking. Both homes are invisible to it.
Home 3 the charter · context border

It gets heard, and joins every conversation.

The house
The charter. CLAUDE.md and .claude/.
It asks
Tokens on every turn, forever, competing with the other forty things you wrote.
Left at the door
Nothing yet. The whole sentence fits through, judgment and all.
Why it leaves
It gets outvoted. One paragraph stands against a codebase, so the model imitates whatever the code shows.
Force2/5
Meaning5/5
The last door the whole constraint fits through. Remember that when it comes back.
Home 4 a hook · argument border

It gets hands, and can finally stop something.

The house
A hook. A script the harness runs at a fixed moment in the turn.
It asks
That the constraint become a pattern, and that somebody maintain it.
Left at the door
Most of its meaning. The sentence it came from is not a pattern.
Why it leaves
The house is not safe. The hook config sits in the charter, so anything that writes there can switch it off.
Force4/5
Meaning2/5
The cheapest room in town. Also the only mechanical one inside its own blast radius.
The house it moved into

An enforcer inside the blast radius is not an enforcer.

  • The hook config is a file, and the agent has an edit tool. Nothing sits between those two facts.
  • Test files have the same shape. If the agent can rewrite the suite, green means the agent agreed with itself.
  • This is not about a scheming model. Even a careless human, a bad merge, or an instruction hidden in a fetched page reaches the same file.
  • So never ask whether it would. Ask what at this layer can.
Anything the agent can edit is not a constraint on the agent. It is a suggestion that fails later with a stack trace.
Home 5 a judge · reach border

It gets understood again, and out of everyone's reach.

The house
A check that spends a model call. A second model reads the diff and returns a verdict.
It asks
Money, and its own evals. A check that grades work has to be graded.
Left at the door
Certainty. The same diff can pass Tuesday and fail Thursday.
Why it leaves
It does not have to. This is the first house that fits, and most of the sentence gets through.
Force4/5
Meaning4/5
The one door where meaning goes back up, which is why it does not belong last.
Home 6 a predicate

It gets obeyed, and barely recognizes itself.

The house
Forty lines over a syntax tree, mounted in CI.
It asks
That the constraint be decidable. Everything else stays outside.
Left at the door
The judgment. It can ask whether a tenant column appears. It cannot ask where the value came from.
Why it leaves
It does not leave. It just is not the constraint anymore, and nothing in the report will tell you that.
Force5/5
Meaning1/5
Maximum force, minimum meaning. It has never been stronger and has never held less.
What got left at that door

Both pass the gate. One serves the wrong invoice.

Obeys the constraint
row = db.one(
  "SELECT * FROM invoices "
  "WHERE id=? AND tenant_id=?",
  invoice_id,
  session.tenant_id,
)
Breaks the constraint
row = db.one(
  "SELECT * FROM invoices "
  "WHERE id=? AND tenant_id=?",
  invoice_id,
  request.path.tenant_id,
)

Both filter on tenant_id, so every line satisfies the predicate. One identifier decides whether the caller or the system chooses whose data comes back. No check over query text can tell them apart, because the constraint was never about query text.

One · how the fable ends

Every door asked it to leave something outside.

REACH BORDER force meaning a head a document the charter a hook a judge a predicate meaning dips at the hook, returns at the judge, collapses at the predicate
Promotion is not moving a constraint up. It is splitting one, and housing every piece where it fits.
Two · the instrument

So I built the thing that measures this.

  • A coding agent enforcing one rule at six rungs. Each rung is its own process with its own switch.
  • Switching a rung off is not a feature flag. It is one of six processes that does not launch, so the other five cannot be disturbed.
  • Twelve runs, Opus driving the loop, against a live engine.
  • Everything after this is a mechanism, not a rate. Twelve runs show that something happens, never how often.
My first version of this measured nothing and said so confidently. That is finding zero, and it is why I trust the rest.
Finding one

Prose is worth its distance from the surrounding code.

  • My first rule restated what the seed code already showed. So charter on and charter off gave identical results, and I nearly published "prose is worthless".
  • That measured the imitation surface, not the charter. So I wrote one rule the code had no example of, and one every example contradicted.
  • With the code showing one thing and the charter demanding another, the charter won.
Audit your charter against the codebase, not your intentions. Delete a line and ask what changes.
Finding two

Catch counts misrank rungs, and the report recommends a mistake.

  • With a hook installed, the hook fires and the rung above never does. My report showed it at zero catches and appeared to recommend deleting it.
  • It should not be deleted. That rung is what still exists after something edits the hook config.
  • Another rung records zero catches permanently, by construction. It is the one that surfaced a deadlock in my harness.
  • What matters is what reaches the tree if this rung is gone. That costs a switch-off, not a counter.
A rung idle because a deletable rung above it fired has told you nothing.
Finding three

The pipeline reported done, over a tree with two violations.

# every mechanical rung switched off
pipeline: status done, rungs fired: []

# the same tree, checked outside the pipeline
$ make gate-run JOB=<id>
  billing/discount.py:31: ROUND_HALF_UP; this rule requires ROUND_HALF_EVEN
  billing/invoice.py:29:  half-up rounding; this rule requires half-to-even
  2 finding(s)

A self-report from the system that did the work is a hypothesis. Only a check with independent access to the artifact turns it into a fact. CI is the boring version, and here it was the one that was right.

Finding four · the uncomfortable one

My report invented three incidents, and they flattered my thesis.

  • It counted runs where no hook ever existed as the agent deleting its own guard. Three fabricated instances of the scariest behavior in the project.
  • The cause was mundane. Nothing recorded whether a hook existed, so absent and deleted looked the same.
  • Now the planner records it, and the gate marks what it cannot establish as unknown instead of claiming it.
If your harness report has never told you something inconvenient, that is information about the report.
Before the moral, the limits

What twelve runs cannot tell you.

n is small

Twelve runs, one or two arms per cell. Recurrence is one of the three tests for promoting a rule, and nothing here establishes it.

One model, one seed, one rule family

All of it is money arithmetic in a forty-line package. A legacy repo has a far louder imitation surface, and I have not tested one.

Non-determinism is real

The same arm produced two different wrong idioms on two runs. So the mode was wrong both times, and a single run is still not a result.

I designed the rules to show the effect

I picked rules whose imitation surfaces were absent and opposed. That makes the effect visible. It does not make it typical.

Three · moral one, which home

Choose by what the constraint is, not how badly you want it obeyed.

# the ladder, as it ships in ghola
0  prose        stated, and nothing enforces it
1  tool grant   the phase never held the tool
2  hook         the repo's own hook refuses
3  in-turn      refused before the target runs
4  stage gate   over the finished diff
5  CI           out of reach of agent and factory
  • Three tests decide eligibility. Consequential, recurrent, decidable. The third kills most candidates, and should.
  • "No raw SQL outside the repository layer" is settleable. "Model the domain honestly" is not, and promotion will not make it so.
  • Eligible is not due. Promote when violations are silent, or when you catch yourself restating something you already wrote down.
  • Rung 1 is the home the fable skipped. Withhold the tool and there is nothing to enforce against.
Moral two · how strict

Strictness is a dial. One position never moves.

refuse

The call does not happen. It is not queued and not warned about. The refusal names the constraint and the sanctioned way around it.

ask

A person answers. Right when the exception is legitimate often enough that refusing would teach people to route around you.

record

It happens, and it is written down. Where a new constraint starts, so you learn how often it fires before handing it teeth.

Unattended, ask degrades to refuse, never to allow. An unanswered question has not been answered yes. Get this wrong and every "ask" in your config becomes permission at 3am.

Moral three · whose reach

Reach is a property of the layer, not a judgment about the agent.

  • Ask it mechanically. What can write to the thing that enforces this? Then list every actor at that layer, not the one you had in mind.
  • At the charter layer that list holds the agent, an injection in a fetched page, a careless merge, and whatever lane proposes improvements.
  • So put the first mechanical check inside the turn, and the last one outside the pipeline.
  • The two do different jobs. An in-turn refusal teaches while the work moves, but an outside check is what makes done mean anything.
Nothing that removes enforcement should travel the same path as ordinary work.
Where it actually lives

Three addresses, one for each boundary you decline to trust.

The sentence

prose, in the charter

the undecidable part
no predicate can hold it

The judgment

one refusal in the turn

teaches while the agent
can still act on it

The fragment

one check in CI

independent access
to the tree

Two homes differing only in strictness are redundancy, and the redundancy wrecks your ability to tell which one works. Two homes differing in what can reach them are not redundant at all. The common error is four rungs on the one rule somebody got burned by. The twenty rules nobody has been burned by yet get prose alone.

One last way this fails

A home can be furnished, documented, and empty.

  • ghola pins its harness version, and the pin is load-bearing. On one version the harness honors a hook's deny. On the version before, it ignored the deny and the call ran.
  • A ladder on the wrong version looks wired in every config file you can read, and refuses nothing.
  • So ask of every rung: what would I see if this stopped working? If the answer is the same green checkmark, it is decoration.
Test that your refusals refuse. Until you watch a guard fire, you do not know it works.
Close

A constraint does not have a home. It has three, and a different form for each.

Pick one constraint this week. Then write the whole sentence in the charter, judgment included. Next, put one refusal inside the turn and one check outside the pipeline. Finally, find out what at that layer can still edit them, because that is the number deciding whether any of it holds.

Contact

Find me, take the code.

The starter kit
tacoda.dev/ghola

the ladder, as a shipped worker

The experiment
tacoda.dev/layers-and-promotion

all twelve runs, and the bugs

The design standard
tacoda.dev/constraint-engineering

rung, policy, layer, verdict

Fulcorum
fulcorum.com
Email
ian@fulcorum.com
Site · LinkedIn · GitHub
tacoda.dev
linkedin.com/in/tacoda
github.com/tacoda
head · document
the charter
a hook
a judge
a predicate
01 / 27
← Harness Engineering
← → move · T theme · F full