Harness Engineering · A Talk

Making AI reliable and governable

Fold the processes, policies, and regulation you already run into the harness. Wrap the model in deterministic layers it cannot talk its way past — gates you enforce, and an audit trail you can read.

soft — rules the model reads hard — gates the harness enforces stop — fail-closed
Who's Talking

Ian Johnson

  • Staff software engineer at Parento — agents against a real production codebase, daily.
  • Founder of Fulcorum · fulcorum.com — on-demand video courses on shipping AI-generated code that's production-grade.
  • Author of Harness Engineering and the open-source tool Keystone.
The model is the least controllable part of the system. So put the control everywhere else.
The Reliability Problem

Same prompt. Same repo. Two days apart.

Tuesday

Added a handler in routes/health.ts. One happy-path test. Stopped. Ten lines.

Wednesday

Noticed an old status.ts. Unified the two. Refactored both. Broke a test — marked it flaky. Opened a PR claiming done. Four hundred lines.

Nobody had touched the model. Nobody had touched the rules. The variance was real, it was the substrate, and it was going to keep happening.Harness Engineering · Ch.3
Determinism Is the Wrong Goal

The model is non-deterministic. That part is not a defect.

  • Temperature zero narrows the distribution — it does not collapse it.
  • Sampling shifts with tool-result order, file timestamps, how the last turn tokenized, the order of a directory listing.
  • Identical prompt, identical model — different output, because the inputs are never identical at the byte level.
You cannot pin a single output. So stop trying. Narrow the band the outputs fall in instead.
The Reliability Band

You can't flatten the signal. You can narrow the band.

agent output — the signal drifts the rails you build narrow the band
Measurable, Not Aspirational

The band is a number a skeptic can check.

A repository carrying a charter with the properties this book describes produces a measurably narrower agent-output band than the same repository without one, holding the model constant.Harness Engineering · Ch.3 — the falsifiable claim
How you prove it
  • Two arms — charter absent vs present, nothing else touched.
  • Same task set, same model, n ≥ 30 per arm.
  • Score every run blind against the intent contract.
  • Compare failure rate and band width.
Why The Band Matters · The Amplification Thesis

The agent is a multiplier on the discipline it finds.

one repo reliable band narrows entropy band widens volume →
The agent does not change your standards. It enforces them. The lower the standards, the more enforcement you get.Harness Engineering · Ch.2

Volume is the variable that did not exist before — and volume picks whichever side the existing code points at.

Three Levers

Reliability is the joint product of three narrowings.

raw intent shippable INPUT feedforward — the contract OUTPUT gates — enforced, observed RECOVERY known response
The Reframe

You don't need a new AI policy. Codify the one you already have.

You already run the processes that make AI governable. Change control. Code review. Approvals. Separation of duties.

SOC 2ISO 27001HIPAAGDPR
A policy that lives only in a runbook is aspirational. It earns its keep by compiling down into a primitive the agent actually meets.Harness Engineering · Ch.62
The Ladder · From Tribal Knowledge to Deterministic Workflow

Push each policy as far down as it can honestly go.

soft — the model can ignore it hard — the run enforces it → Tribal in a senior's head Inferential a rule the model reads Check a sensor with teeth Workflow enforced by the run itself
Governance · The Same Move At Business Altitude

The org charter is business governance, compiled.

BUSINESS GOVERNANCE SOC 2 · HIPAA · GDPR · legal · risk ORG CHARTER read · enforced · queried REPO CHARTER + GATES this project only a hook · a CI check at the seam
Governance states the policy. The repo is where it becomes a wall.Harness Engineering · Ch.62

The charter need not stay a document. The same layer can enforce itself as a harness — gates for the irreversible policies — or serve org knowledge the agent queries on demand, as a RAG index or an MCP server.

Govern Before, Observe After

Prevent what you can. Detect the rest. Feed one into the other.

Prevent policy at the invoke seam the bad action never runs the cheaper half Detect observability + a cadence a pattern becomes a policy the counterweight gaps slip through caught twice → prevented the third time
Prevention · At the Invoke Seam

A guardrail runs whether the model cooperates or not.

input SCREEN in model untrusted output VALIDATE out ship ambiguous → BLOCK fail-closed by default
The Guardrail That Waits

Human-in-the-loop stops before the action — not after.

A guardrail that pauses the run and escalates on a high-risk or low-confidence action. Not a review after the fact — the loop stops and waits before the action.Harness Engineering · Ch.37
  • Second signer on a payments deploy.
  • Tool call held because the diff touches a sensitive surface.
  • The pause is a row in the record — accountable judgment where the regime requires it.
Wrap the Model · Ch.43

Control flow lives in the graph, not the model's head.

start agent loop contract → diff gate done retry ( < cap ) stop @ cap
The agent loop is one node. The graph around it decides, in code, whether a failing gate means retry or stop.
The Orchestrator Is a Queue, Not an Agent

Sequencing is never a model decision.

The model appears in exactly one place — inside a step, doing the judgment the step exists for. Everything around it is deterministic code over a durable store.
What determinism buys the audit
  • Cheap — no tokens spent on sequencing.
  • Reproducible — replay the data, replay the run.
  • Auditable — every decision is a row, not a rationale.

A metric you cannot re-derive from the event trail is an assertion.

Attestations · The Gate With Teeth

A step will not open until its checks are green.

  • A step declares the checks required before entry.
  • The loop refuses to enter until they attest green.
  • A missing or failed attestation raises — it does not proceed.

A violated design preference is a finding a sensor records. A leaked credential or an unmet attestation is a stop.

Enforcement has a dial
audit

Allow by default. Deny what a rule names.

strict

Deny by default. Allow only a whitelist. Every policy gap becomes a visible refusal, not a silent allow.

Observability · The Audit Trail Your Regime Consumes

Everything is an event. The record is the source of truth.

Retention

Audit records carry a purge policy. "Keep forever" and "keep ninety days" are both choices you write down — not accidents of disk space.

Residency

Self-hosted. The record lives in a jurisdiction you chose, inside the same compliance boundary as the code it describes.

Today's detection is tomorrow's prevention. What observability catches twice, governance prevents the third time.

The Six Phases · Policy-Governed Touchpoints

Every boundary is a checkpoint the orchestrator will not skip.

1Intake 2Investigation 3Implementation 4Review 5Merge 6Learn ↑ touchpoints — no boundary crossed without a verdict
Concrete · The Org Rule Charter

Three tiers. An owner. A cadence.

# Org rule charter · Owner: head of platform · Cadence: quarterly

## Universal principles  # every codebase we own
  Separation of concerns   → names the failure it prevents
  Fail-fast at boundaries  → ...

## Non-negotiables       # hold across every team
  No execution without an approved contract.
  Three touchpoints maximum per contract.
  Every closed contract is independently releasable.

## Identity statements   # what we refuse to ship
The seam, made real: business governance compiled into lines an agent reads — and the load-bearing ones harden into gates.
The Discipline That Holds It All Up

A deterministic system that does the wrong thing is a reliable defect.

The charter's rules might not change the agent's behavior at all. The harness's graph might route perfectly and produce the wrong answer. Measure rather than assert.Harness Engineering · Ch.61
Same instrument at both levels
  • A corpus of tasks.
  • A scorer that runs blind.
  • A pass rate, computed on every change.

You do not know either artifact works until you have measured it.

What To Do Monday

One turn of the crank at each layer.

  • Write one intent contract for the next real task — scope, falsifiable criteria, risk.
  • Harden one policy you already have into a gate — a hook or a CI check at the seam.
  • Add one attestation a step must pass before it opens, and one audit event so the trail exists.
  • Measure the band on that area — n ≥ 30, blind.
Close

Don't make the model deterministic. Wrap it in things that are.

Narrow the band. Compile the policy you already run down the ladder until the agent meets it as a wall — one you can enforce, observe, and audit. The amplifier will multiply whatever it finds. Point it at your discipline.

Contact

Find me.

Fulcorum
fulcorum.com
Email
ian@fulcorum.com
Site
tacoda.dev
Book
leanpub.com/harness-engineering
LinkedIn · GitHub
linkedin.com/in/tacoda
github.com/tacoda
Open source · github.com/tacoda
keystone — charter layer
sigma — harness layer
open-refinery — factory layer
01 / 24
Harness Engineering
← → move · T theme · F full