Harness Engineering · A Talk

Making AI reliable and governable

Fold the processes, policies, and regulation you already run into the harness. Then wrap the model in deterministic layers it cannot talk its way past : gates you enforce, and an audit trail you can read.

soft : rules the model reads hard : gates the harness enforces stop : fail-closed
Who's Talking

Ian Johnson

  • Staff software engineer at Parento, running agents against a real production codebase, daily.
  • Founder of Fulcorum · fulcorum.com, on-demand video courses about agentic engineering.
  • Author of Harness Engineering and the open-source tool Keystone.
The model is the least controllable part of the system. So put the control everywhere else.
The Reliability Problem

Same prompt. Same repo. Two days apart.

Tuesday

Added a handler in routes/health.ts. One happy-path test. Stopped. Ten lines.

Wednesday

Noticed an old status.ts. Unified the two. Refactored both. Broke a test, marked it flaky. Opened a PR claiming done. Four hundred lines.

Nobody had touched the model, and nobody had touched the rules. So the variance was real: it came from the substrate and it was going to keep happening.Harness Engineering · Ch.3
Determinism Is the Wrong Goal

The model is non-deterministic. That part is not a defect.

  • Temperature zero narrows the distribution, but it does not collapse it.
  • The prompt and model are identical, yet the output differs, because the inputs are never identical at the byte level.
You cannot pin a single output. So stop trying. Narrow the band the outputs fall in instead.
The Reliability Band

You can't flatten the signal. But you can narrow the band.

agent output : the signal drifts the rails you build narrow the band
Measurable, Not Aspirational

The band is a number a skeptic can check.

A repository carrying a charter with the properties this book describes measurably narrows its agent-output band. The same repository without a charter, model held constant, produces a wider band.Harness Engineering · Ch.3 : the falsifiable claim
The properties it must have
  • Accreted, never authored in advance.
  • Every rule names its incident, dated and costed.
  • Every rule names its altitude : sentence, check, or gate.
  • Every rule names its expiry.
  • Adding one requires removing one.
How you prove it
  • Two arms : charter absent vs present, nothing else touched.
  • Same task set, same model, n ≥ 30 per arm.
  • Score every run blind against the intent contract.
  • Compare failure rate and band width.
Why the band matters · the amplification thesis

The agent is a multiplier on the discipline it finds.

one repo reliable band narrows entropy band widens volume →
The agent does not change your standards. It enforces them. So the lower the standards, the more enforcement you get.Harness Engineering · Ch.2

Volume is the variable that did not exist before, and volume picks whichever side the existing code points at.

Three Levers

Reliability is the joint product of three narrowings.

raw intent acceptable INPUT feedforward : the contract OUTPUT gates : enforced, observed RECOVERY known response
The Reframe

You don't need a new AI policy. Instead, codify the one you already have.

You already run the processes that make AI governable. Change control. Code review. Approvals. Separation of duties.

SOC 2ISO 27001HIPAAGDPR
A policy that lives only in a runbook is aspirational. It earns its keep by compiling down into a primitive the agent actually meets.Harness Engineering · Ch.62
The ladder · from tribal knowledge to settled delivery

Push each policy as far down as it can honestly go.

soft : the model can ignore it hard : delivery enforces it → Tribal in a senior's head Inferential a rule the model reads Check a sensor with teeth Judgment an LLM-as-judge scores Workflow enforced by the run Stage settled by the pipeline CHARTER what you author HARNESS how work is done FACTORY how tasks complete
Governance · what goes where

Every control you own has an address.

CONTRACT what work needs to get done e.g. add idempotency keys to the refund endpoint ORG / TEAM CHARTER org and team standards e.g. money is integer cents, never float REPO CHARTER project standards e.g. refunds only via RefundService e.g. no direct writes to the ledger standards : what good means FACTORY how tasks get completed e.g. no merge until the eval suite is green HARNESS how work gets implemented e.g. a test that fails when reverted e.g. an LLM-as-judge scores the diff process : how it happens
Governance states the policy. But the repo is where it becomes a wall.Harness Engineering · Ch.62

Standards say what good means. Process says how it happens. A component in the wrong box is the common defect, because the rule lands wherever the person who noticed it had commit access.

Govern before, observe after

Prevent what you can. Detect the rest. Feed one into the other.

Prevent policy at the invoke seam the bad action never runs the cheaper half Detect observability + a cadence a pattern becomes a policy the counterweight gaps slip through caught twice → prevented the third time
Prevention · at the invoke seam

A guardrail runs whether the model cooperates or not.

input SCREEN in model untrusted output VALIDATE out ship ambiguous → BLOCK fail-closed by default
The guardrail that waits

Human-in-the-loop stops before the action, not after.

A guardrail that pauses the run and escalates on a high-risk or low-confidence action. Not a review after the fact : the loop stops and waits before the action.Harness Engineering · Ch.37
  • Second signer on a payments deploy.
  • The harness holds a tool call because the diff touches a sensitive surface.
  • The pause is a row in the record : accountable judgment where the regime requires it.
Wrap the Model · Ch.43

Control flow lives in the graph, not the model's head.

start agent loop contract → diff gate done retry ( < cap ) stop @ cap
The agent loop is one node. The graph around it decides, in code, whether a failing gate means retry or stop.
The Orchestrator Is a Queue, Not an Agent

Sequencing is never a model decision.

The model appears in exactly one place : inside a step, doing the judgment the step exists for. Everything around it is deterministic code over a durable store.
What determinism buys the audit
  • Cheap : no tokens spent on sequencing.
  • Reproducible : replay the data, replay the run.
  • Auditable : every decision is a row, not a rationale.

A metric you cannot re-derive from the event trail is an assertion.

Attestations · the gate with teeth

The loop will not open a step until its checks are green.

  • A step declares the checks required before entry.
  • The loop refuses to enter until they attest green.
  • A missing or failed attestation raises an error, and the run stops.

A violated design preference is a finding a sensor records. But a leaked credential or an unmet attestation is a stop.

Enforcement has a dial
audit

Allow by default. Deny what a rule names.

strict

Deny by default. Allow only a whitelist. Instead of allowing a policy gap silently, the harness raises a visible refusal.

Observability · the audit trail your regime consumes

Everything is an event. The record is the source of truth.

Retention

Audit records carry a purge policy. Both "Keep forever" and "keep ninety days" are choices you write down, not accidents of disk space.

Residency

You host the record yourself. It lives in a jurisdiction you chose, inside the same compliance boundary as the code it describes.

What observability catches twice, governance prevents the third time.

The six phases · policy-governed touchpoints

Every boundary is a checkpoint the orchestrator will not skip.

1 Intake 2 Investigation 3 Implementation 4 Review 5 Merge 6 Learn contract approval plan diff change approval change pipeline no boundary crossed without a verdict : every verdict leaves a document
human touchpoint
The discipline that holds it all up

A deterministic system that does the wrong thing is a reliable defect.

The charter's rules might not change the agent's behavior at all. The harness's graph might route perfectly and still produce the wrong answer. So measure rather than assert.Harness Engineering · Ch.61
Same instrument at both levels
  • A corpus of tasks.
  • A scorer that runs blind.
  • A pass rate, computed on every change.

You do not know either artifact works until you have measured it.

What to do Monday

One turn of the crank at each layer.

  • Write one intent contract for the next real task : scope, falsifiable criteria, risk.
  • Harden one policy you already have into a gate : a hook or a CI check at the seam.
  • Add one attestation a step must pass before it opens, and one audit event so the trail exists.
  • Measure the band on that area : n ≥ 30, blind.
Close

Don't make the model deterministic. Wrap it in things that are.

Narrow the band. Compile the policy you already run down the ladder until the agent meets it as a wall, one you can enforce, observe, and audit. The amplifier will multiply whatever it finds, so point it at your discipline.

Contact

Find me.

Fulcorum
fulcorum.com
Email
ian@fulcorum.com
Site
tacoda.dev
Book
leanpub.com/harness-engineering
LinkedIn · GitHub
linkedin.com/in/tacoda
github.com/tacoda
Open source · github.com/tacoda
keystone: charter layer
sigma: harness layer
open-refinery: factory layer
01 / 24
Harness Engineering
← → move · T theme · F full