Harness Engineering / A lights-on factory / Anatomy of a running system

A harness by definition, a factory by operation, a queue by design

A lights-on factory: it takes a tracker card and returns a merged pull request, and the lights are on in two distinct senses — governance gates raise a decision to a human before an action is taken, and observability and provenance record what happened after it. Autonomy is a separate dial. It is about 5,300 lines of Python: three runtime dependencies, one model dependency, sixteen job handlers, seven seams, 193 tests. Nothing in it knows what the code it delivers is for — no product vocabulary, no business rules, no domain types. That absence is a design property, not an omission: the factory is the machinery, and everything domain-specific lives in the target repo's charter, where it can be authored and changed without touching the engine. The five rungs below ascend the way the system was actually built: one dependency, one pure function, one durable queue, one writer per card, one lit loop.

harness surface · ~5,300 lines of Python · 3 runtime deps (fastapi · uvicorn · pyyaml) · 1 model dep · 16 job handlers · 7 seams · 193 tests · state is files
substitutions · make = your task runner · PROJ-000 = your issue key · tracker and code host = whatever you already run
Deterministic
A pure function or a gate decides. No inference, no spend, unit-testable.
Model turn
The expensive, non-deterministic move. Spent only where inference is required.
Durable
A persisted artifact — a file the queue can resume from and an auditor can read.
1 One dependency

The smallest model dependency that does the job

This harness reaches for exactly one library to run an agent: the Claude Agent SDK. Not because frameworks are bad — because the SDK is Claude Code, and Claude Code is already the harness this shop prefers. Point it at a directory and the target repo's charter, hooks, skills and MCP servers load natively. Everything a framework would add on top is either already there or not needed yet.

dep claude-agent-sdk>=0.1 seam providers/base.py · ModelProvider impl providers/claude_sdk.py impl providers/stub.py config FACTORY_MODEL · FACTORY_MODEL_API_KEY

The whole dependency list is short enough to print. Three packages run the daemon and its board; one optional extra runs the model. There is no workflow engine, no graph library, no agent framework, and no vector store.

pyproject.toml — the entire dependency surfaceone model dep
dependencies = [
  "fastapi>=0.115",   # the board's API
  "uvicorn>=0.30",    # serves it
  "pyyaml>=6.0",      # charter frontmatter
]

[project.optional-dependencies]
# the model seam: claude-agent-sdk runs Claude Code (drives the `claude` CLI,
# which the image installs). `anthropic` is only for the eval llm_judge — a
# single Messages-API completion, no agent.
agent = ["claude-agent-sdk>=0.1", "anthropic>=0.40"]

The reason one dependency is enough is that the SDK is not a thin HTTP client — it runs Claude Code in-process. Pointed at cwd=<the task's worktree>, it loads that repo's CLAUDE.md, its glob-scoped .claude/rules, its skills, its .mcp.json servers and its settings.json hooks — natively, with no reproduction on the harness side. A framework-based harness has to rebuild all of that: context assembly, tool definitions, permission modes, hook dispatch. Here the harness injects only the one layer Claude cannot know about — cross-repo governance policy — and gets the rest for free.

providers/base.py — the seam, in full~30 lines
@dataclass
class ModelResult:
    text: str                      # final assistant text
    cost_usd: float = 0.0
    tokens: int = 0
    ok: bool = True
    error: str | None = None
    data: dict | None = None       # parsed structured output


class ModelProvider(ABC):
    @abstractmethod
    def run(self, prompt: str, *, cwd: Path | None = None, extra_dirs: list[Path] | None = None,
            timeout: float = 3600, schema: dict | None = None, phase: str = "",
            on_event=None) -> ModelResult:
        """Run one agent turn. The harness NEVER shells out to a model
        directly; it goes through here."""

Subscription auth, and why it decides the architecture

One provider covers both auth modes, and the default is the interesting one. If an API key is configured it is passed through to the SDK; if it is not, the SDK falls back to the claude CLI's own ambient authentication — the subscription already being paid for. No token to mint, rotate, scope or leak, and no per-token invoice for a factory whose normal state is four long agent turns running at once.

providers/claude_sdk.py — the option kwargskey optional, by design
self.api_key = api_key or None       # "" → None: no key means ambient CLI auth

kwargs = {
    "cwd": str(root),
    "permission_mode": "bypassPermissions",   # headless: no interactive approval
    "disallowed_tools": self._disallowed(phase),
}
if self.model:
    kwargs["model"] = self.model
if self.api_key:                     # ONLY when explicitly configured
    kwargs["env"] = {"ANTHROPIC_API_KEY": self.api_key}

# Two behaviors are set explicitly because SDK defaults differ from `claude -p`:
# the Claude Code system prompt (SDK apps default to none) and permission bypass.
# `setting_sources` is left at the SDK default — all sources, i.e. CLI parity.
return ClaudeAgentOptions(system_prompt={"type": "preset", "preset": "claude_code"}, **kwargs)

Cost is still measured. The SDK's terminal ResultMessage carries total_cost_usd and a token count, so every job records what the turn would have cost whether or not anything was metered — which is what makes the per-card price in rung 5 possible without an invoice.

When to reach past the SDK

This is a threshold, not a preference. The SDK is the right tool while the flow you need is the flow a stock harness already runs. The moment you need something a stock harness cannot give you — a genuinely custom control flow, or the ability to swap the model underneath — that is when a framework earns its place, and Pydantic AI, LangChain, LangGraph and deepagents are the right answers to that question.

  1. Stock flow Run an agent inside a repo, under that repo's own charter, and get structured output back. The SDK does this natively — schema= plus a JSON-extraction helper is the entire structured-output layer here.
  2. Orchestration Retries, leases, dead letters, checkpoints, backpressure. A durable queue, ~200 lines of it (rung 3) — not a workflow engine. The queue is the orchestration.
  3. Custom flow A control flow no stock harness will run for you: a graph with joins and conditional fan-in, per-node memory policies, a human-approval node mid-graph, cross-process resumption of a partial graph. Reach for the framework.
  4. Model choice Swapping models or providers per step — a cheap model to classify, an expensive one to implement, a third to judge. A provider-agnostic framework is the honest way to do that; a single-vendor SDK is not.
  5. Not yet Neither threshold has been hit here, so neither dependency exists here. If one arrives, the change is one class behind ModelProvider — which is the whole reason the seam was written before it was needed.
Without it

Start with a framework and you pay twice. Once in reproduction: the framework's runtime re-implements charter loading, tool policy, hook dispatch and permission modes that Claude Code already does, and your versions drift from the ones your engineers use interactively. And once at the meter: framework paths authenticate with a provider API key, so an unattended factory bills per token from the first run, before you know whether the flow is even right.

For the book

Take the highest rung of the ladder that holds. If your preferred harness is Claude Code, its SDK is your agent framework — reaching past it buys custom flow and model portability, and until you need those two things it buys only surface area. The discipline that makes this safe is not the choice, it is the seam: thirty lines of abstract class means the decision stays reversible, and a stub implementation of the same interface is what lets the whole factory run in tests with no model at all.

2 Pure function

Derive, don't remember

The daemon holds no opinion about where a card is. Every tick it reads current external state, hands a snapshot to a pure function, and gets back at most one action to enqueue. The controller model, not the event model: if the daemon dies mid-run, external state is untouched and the next tick resumes from it.

pure harness/derive.py · derive_action pure derive.py · route_feedback loop harness/reconciler.py · tick seam Tracker · CodeHost guardrail per-issue at-most-one-active guardrail poison guard

derive.py has no imports that touch the network, the disk or the clock. It maps one issue snapshot to one decision, and the pipeline it encodes is a table anyone can read: Backlog → Needs Details → To Do → In Progress → In Review → In Staging → Done, over four phases (Discover → Define → Dispatch → Verify).

harness/derive.py — the state machine, no I/Odeterministic
@dataclass(frozen=True)
class Decision:
    action: str | None       # job fn to enqueue, or None (waiting / terminal)
    reason: str

def derive_action(issue: Issue) -> Decision:
    s = issue.status
    if s == BACKLOG:
        # nothing rendered yet, or new feedback since the last render
        return Decision("draft", "in Backlog → draft the contract") if _needs_refine(issue) else NOTHING
    if s == NEEDS_DETAILS:
        if issue.has_marker(M_APPROVE):
            return Decision("approve", "contract approved → move to To Do")
        return Decision("draft", "draft the contract") if _needs_refine(issue) else NOTHING
    if s == TO_DO:      return Decision("dispatch", "approved and ready → dispatch executor chain")
    if s == IN_PROGRESS: return Decision("execute", "in progress → run/continue executor chain")
    # In Review (verdict), In Staging / Done (terminal), Backlog (awaiting triage)
    return NOTHING

Note what isn't here. No model call decides where a card goes next. Routing feedback is the same story — whether a comment means "revise the open PR" or "file a follow-up card" is decided by the card's status, in one line, for free:

derive.py — routing without inferenceone line, testable
def route_feedback(status: str) -> str:
    """In-flight work is revised on the same PR; shipped or not-yet-started work
    is triaged as a follow-up card, never touching a frozen PR."""
    return "revise" if status in (IN_PROGRESS, IN_REVIEW) else "triage"

The reconciler is the impure half, and it is deliberately dull: read the tracker, call derive_action, enqueue, write a snapshot for the board. All the subtlety is in not enqueueing — three guards, each answering a different failure.

harness/reconciler.py — the three guards on enqueuerefuses duplicates
def _has_active(self, key, action):     # already queued or leased → skip
    return any(j.status in (QUEUED, LEASED) and j.fn == action
               and j.payload.get("issue_key") == key for j in self.store.all())

def _is_poisoned(self, key, action):
    # A dead-lettered OR operator-cancelled (key, action) blocks re-enqueue —
    # otherwise a failing action re-derives EVERY TICK, forever. The operator
    # retries (resume) or drops (cancel) it from the job page.
    return any(j.status in (DEAD, CANCELLED) and j.fn == action
               and j.payload.get("issue_key") == key for j in self.store.all())

# a drop or a from-scratch restart in flight pre-empts other work for the card
_PREEMPT = ("close", "restart")

def _retire_poison(self, key, action):
    # Newer operator feedback SUPERSEDES a failed prior attempt, so it stops
    # counting as poison and a fresh job runs. Every feedback yields a NEW job —
    # we never revive an old one.
    ...

The tracker and the code host are ports too

The reconciler reads two systems it does not own — the ticketing system the cards live in and the code host the branches and pull requests live on — and neither belongs in derive_action. The same discipline that put ModelProvider in front of the agent puts a port in front of each of these: the pure function sees an Issue snapshot and returns a decision, and the vendor's REST shapes, field ids, status names and rate limits stay on the far side of an adapter.

a sketch, not shipped code — two more ports, one adapter per vendorthe shape it wants
class Tracker(Protocol):              # the ticketing system: cards in, decisions mirrored back
    def issues(self) -> list[Issue]: ...
    def transition(self, key: str, status: str) -> None: ...
    def comment(self, key: str, body: str) -> None: ...

class CodeHost(Protocol):             # branches, pull requests, checks
    def open_pr(self, branch: str, title: str, body: str) -> PR: ...
    def checks_green(self, pr: PR) -> bool: ...
    def merge(self, pr: PR) -> None: ...

TRACKERS   = {"jira": JiraTracker, "linear": LinearTracker, "gh-issues": GhIssuesTracker}
CODE_HOSTS = {"github": GitHubHost, "gitlab": GitLabHost}
FAKES      = {"tracker": InMemoryTracker, "host": InMemoryHost}   # the 193 tests run on these

What each adapter absorbs is the vendor-shaped part: how a status is named and transitioned, whether a merge is a squash or a rebase, which webhook or poll tells you a check went red, and how the credential is scoped. What none of them touch is the pipeline. Swapping Jira for Linear, or GitHub for GitLab, is a class and its fake — the state machine, the guards and the queue do not know a substitution happened. It is also what makes tracker and code host = whatever you already run a claim rather than a hope, and what lets the whole delivery loop run in tests against in-memory doubles: a controller that derives from external state is only as testable as the seam it reads that state through.

What the controller model buys

  1. Crash safety The daemon can be killed at any instant. Nothing is lost because nothing was held: state lives in the tracker, the code host, git and the queue's files. The next tick reads the world as it is and continues.
  2. Testability The interesting logic is a pure function over a dataclass, so the entire pipeline — every transition, every route — is covered by fast tests with no daemon, no network and no model.
  3. Cost Routing decisions cost nothing. Inference is spent on drafting a contract, implementing a change and reviewing a diff — the three places judgment is actually required.
  4. Idle discipline A card mid-approval derives NOTHING rather than re-rendering. _needs_refine re-drafts only when new feedback arrives after the last render, so a waiting card is not silently re-spent every tick.
  5. Free observability The same pass writes pipeline.json — cards grouped by stage, each with its next action — which is the board's entire data source. The projection is a by-product of deciding.
Without it

Go event-driven with state in the process and two failure modes arrive together. A missed or duplicated webhook wedges a card in a state nobody can name, and a deploy mid-run loses the only copy of "where were we". Then, because in-process state is unreliable, the natural fix is to ask the model where the card is — which is expensive, non-deterministic, and untestable, three properties you do not want in a routing decision.

For the book

Deterministic decisions stay in the harness; a model turn is spent only where inference is required. The test for a rule is whether you can write it as a pure function over external state — if you can, it does not belong in a prompt. Derive, don't remember is the same discipline at the system level that "read the file, don't recall it" is at the agent level.

3 Durable queue

The queue is the design, not the plumbing

Every property this factory needs turns out to be a queue property. Resume after a crash? A lease. Survive a transient failure? Bounded retries with backoff. Stop a wedged card from burning money forever? A dead-letter. Know what a card cost? A field on the job. Limit how many agents run at once? The size of the pool. Get all of it in about 200 lines of Python over files on disk — no broker, no database, no daemon but your own.

seam queue/store.py · Queue impl queue/file_store.py · FileStore model queue/job.py · Job · JobState pool queue/worker.py · WorkerPool ctx worker.py · Context.save / bill registry queue/registry.py

The unit of work is a job, and the job record is the whole contract between the loop and the world. Everything the factory later wants to know — what ran, how many times, what it cost, what it had gotten through before it died — is a field here.

queue/job.py — the unit of workone file per job on disk
@dataclass
class Job:
    id: int
    fn: str                                  # handler name in the registry
    payload: dict = field(default_factory=dict)
    status: JobState = JobState.QUEUED
    attempts: int = 0
    max_attempts: int = 3
    idempotency_key: str | None = None
    lease_until: float = 0.0                 # visibility timeout while leased
    run_after: float = 0.0                   # backoff / delayed scheduling
    parent_id: int | None = None             # chained job trees
    error: str | None = None
    result: Any = None
    checkpoint: dict = field(default_factory=dict)   # durable mid-job state
    cost_usd: float = 0.0
    tokens: int = 0
    created_at: float = field(default_factory=now)
    updated_at: float = field(default_factory=now)
The state machine of a job — who moves it, and what it means
StateSet byMeaningLeasable
queuedenqueue · failready, or waiting out a backoff (run_after)yes
leasedleasea worker holds it until lease_untilon expiry
donecompletesucceeded; result, cost and tokens recordedno — final
deadlease (attempts exhausted)dead-letter: off the main queue, kept on diskvia resume
pausedoperator control fileheld by a human; not leasable until resumedvia resume
cancelledoperator control fileterminated by a humanno — final
supersedednewer feedbackretired, kept for history, no longer counts as poisonno — final

superseded is the state worth stealing. Without it, a dead-lettered job is permanent poison: the reconciler refuses to re-enqueue that (card, action) pair, so a card that failed once can never be retried, and the operator's button does nothing. With it, newer human feedback retires the corpse and a fresh job runs — history intact, no zombie revived.

Durability: write-through, not write-later

The store keeps an in-memory cache for speed and a file per job for truth. Every mutation updates the cache and immediately flushes JSON to disk; startup rehydrates the cache from the files. There is no migration, no schema and nothing to back up beyond a directory — and the contents are readable with cat at three in the morning, which is a feature.

queue/file_store.py — durability in six linesstate is files
"""Write-through queue store: in-memory cache for speed AND a file per job for
durability. Every mutation updates the cache then flushes to disk. On startup we
rehydrate the cache from the files, so a crash resumes where it left off.

Concurrency: a single re-entrant lock serializes writes. Correct and simple at
this scale. ponytail: global lock, shard by job-id if throughput ever matters."""

def _rehydrate(self):
    for f in sorted(self.jobs_dir.glob("*.json")):
        job = Job.from_dict(json.loads(f.read_text()))
        self._cache[job.id] = job
        if job.idempotency_key:
            self._idem[job.idempotency_key] = job.id
        self._seq = max(self._seq, job.id)

def _flush(self, job):
    job.updated_at = self._now()
    (self.jobs_dir / f"{job.id}.json").write_text(json.dumps(job.to_dict(), indent=2))

Leasing: crash recovery without a heartbeat

lease() is where four mechanisms meet in twenty lines. It hands out one ready job, reclaims leases that expired because a worker died, increments the attempt counter, and dead-letters anything that has burned its budget. Nothing else in the system needs to know that a worker crashed.

file_store.py — lease, retry and dead-letter in one placeexhausted → dead
def lease(self, lease_seconds):
    with self._lock:
        self.apply_controls()                # operator pause/cancel first
        t = self._now()
        while True:
            candidate = self._next_ready(t)
            if candidate is None:
                return None
            candidate.attempts += 1
            if candidate.attempts > candidate.max_attempts:
                candidate.status = JobState.DEAD    # dead-letter, off the queue
                self._flush(candidate)
                self.trace(candidate.id, "error", "dead_letter", {"fn": candidate.fn})
                continue                     # … and look at the next candidate
            candidate.status = JobState.LEASED
            candidate.lease_until = t + lease_seconds
            self._flush(candidate)
            return candidate

A one-hour visibility timeout is right for agent work and wrong for a restart, so startup has its own recovery: at that moment no worker holds a lease, therefore every leased job is orphaned and can be requeued immediately instead of waiting the timeout out.

file_store.py — orphan recovery at bootno waiting out an hour
def recover_orphaned(self) -> int:
    """Requeue jobs left LEASED by a previous process (crash/restart). At startup
    no workers hold leases, so a leased job is orphaned."""
    for job in self._cache.values():
        if job.status == JobState.LEASED:
            job.status = JobState.QUEUED
            job.run_after = self._now()
            self._flush(job)
            self.trace(job.id, "warn", "recovered_orphan", {"fn": job.fn})

Checkpoints: the difference between retry and restart

A retried agent job is not a pure function — its first attempt may already have brought up a container stack, created a branch or opened a pull request. The checkpoint is what keeps a retry from redoing the expensive, side-effecting parts. Handlers get a Context whose save() writes durable mid-job state and whose bill() accrues spend, and the pattern in practice is a single guard:

the checkpoint pattern — do the costly thing onceretry-safe
# orchestrator._dispatch — bring the stack up ONCE, not per retry
if self.setup_cmd and not ctx.checkpoint.get("stack_up"):
    ok, tail = _run_cmd(self.setup_cmd, path, env=_compose_env(key))
    ctx.save(stack_up=True)              # flushed to the job file immediately
    ctx.trace("stack_setup", {"cmd": self.setup_cmd, "ok": ok, "tail": tail})

# worker.Context — the only sanctioned way to persist mid-job state
def save(self, **state):
    self.job.checkpoint.update(state)
    self.store.checkpoint(self.job.id, self.job.checkpoint)

def bill(self, cost_usd, tokens=0):          # worker bills this on complete AND on fail
    self.cost_usd += cost_usd
    self.tokens += tokens

Billing on failure is the small decision that makes cost data trustworthy. A failed attempt spent real money; if only successful jobs recorded their spend, the factory's own numbers would flatter it. So fail() takes cost_usd too, and the job's total is the sum of everything it ever burned.

worker.py — every failure routes through the queuehandlers never crash the loop
def backoff_for(attempt, base, cap):
    return min(cap, base * (2 ** max(0, attempt - 1)))   # exponential, capped

try:
    result = handler(ctx, job.payload)
    self.store.complete(job.id, result, cost_usd=ctx.cost_usd, tokens=ctx.tokens)
except Exception as exc:  # noqa: BLE001 — route every failure through the queue
    self.store.fail(job.id, f"{type(exc).__name__}: {exc}",
                    backoff=backoff_for(job.attempts, self.backoff_base, self.backoff_cap),
                    cost_usd=ctx.cost_usd, tokens=ctx.tokens)   # bill the failed attempt

What each mechanism actually buys

  1. Idempotent enqueue A repeated idempotency_key returns the original job id and traces a dedupe. Duplicate ticks, retried webhooks and an operator double-click are all harmless by construction.
  2. Lease + timeout At-least-once delivery with crash recovery and no heartbeat protocol. A dead worker's job becomes leasable again on expiry; a restart shortcuts that with orphan recovery.
  3. Backoff Capped exponential (60s base, 30min cap). A rate limit or a flaky remote costs waiting, not a tight retry loop against someone else's service.
  4. Dead-letter Three attempts and the job leaves the queue instead of retrying forever. Combined with the reconciler's poison guard, a wedged card stops spending — visibly, in a state a human can see and act on.
  5. Checkpoint Resume rather than restart: the branch, the worktree path and the "stack is already up" flag survive a crash, so attempt two continues instead of re-paying.
  6. Append-only trace Every enqueue, lease, checkpoint, failure and completion appends one JSON line to traces.jsonl. That file is the audit log — the dashboard's database is a projection of it, rebuildable at any time.
Without it

Run the same loop with in-process state and a for loop over cards, and the failure is not that it breaks — it is that it breaks invisibly and expensively. A retry re-runs a handler whose first attempt already opened a pull request. A restart loses the fact that a container stack is up, and the next attempt collides with it. A card that fails deterministically retries forever, and you find out from the bill. None of these need a distributed system to appear; they appear on one laptop, on the first bad afternoon.

For the book

Choose the durable unit first and the rest of the system hangs off it. Because the job record carries cost, attempts, checkpoint and result, the factory gets pricing, observability, operator control and crash recovery without a second mechanism for any of them. Agentic orchestration is a queue problem wearing a new hat — and the boring, decades-old answers (idempotency keys, visibility timeouts, dead-letters, capped backoff) are the ones that hold when the worker is a non-deterministic agent that runs for twenty minutes.

4 Concurrency

One writer per card, parallel everywhere else

Four agents run at once here, and none of them can collide, because the serialization point is in the leasing function rather than in anyone's prompt. Per-card single-writer in the queue, a git worktree per card, a container namespace per card, and a bounded pool that is the backpressure. The operator's controls arrive as files, because the board and the daemon are different processes.

gate file_store._next_ready policy policies/single-writer.md seam harness/workspace.py · GitWorkspace config FACTORY_WORKERS=4 control queue/control.py · ControlStore switch state/STOP

The single-writer guarantee is eight lines inside the function that picks the next job. A job carrying an issue_key is skipped while another job for that same key holds a live lease. Two agents therefore never write one worktree — not because they were told not to, but because the second one is never handed the work.

file_store._next_ready — the single-writer guaranteeskips a busy card
def _next_ready(self, t):
    # Per-issue serialization: at most one job per issue_key runs at a time, so two
    # jobs never write the same worktree concurrently. Jobs without an issue_key
    # are not serialized.
    active_keys = {
        self._issue_key(j) for j in self._cache.values()
        if j.status == JobState.LEASED and j.lease_until > t and self._issue_key(j)
    }
    ready = []
    for j in self._cache.values():
        if j.status == JobState.QUEUED and j.run_after <= t:
            if self._issue_key(j) and self._issue_key(j) in active_keys:
                continue                          # another job for this card is in flight
            ready.append(j)
        elif j.status == JobState.LEASED and j.lease_until < t:
            ready.append(j)                       # expired lease → reclaimable
    return min(ready, key=lambda j: j.id) if ready else None   # FIFO by id

Serializing per card is only half of it; the other half is making sure two different cards cannot collide either. Each card gets its own git worktree off the source repo — shared .git, so it is fast and there is no full clone per card — and its own container project namespace, so make up and make down refer to the same stack and to nobody else's.

isolation, twice: the tree and the stackno shared mutable ground
# harness/workspace.py — a worktree per card, idempotent (resume returns the existing one)
def prepare(self, issue_key, branch, base="main") -> Path:
    dest = self._dir(issue_key)
    if dest.exists():
        return dest                                   # resume: worktree already there
    _git(["worktree", "prune"], cwd=self.source)      # clear stale registrations
    if self._branch_exists(branch):
        _git(["worktree", "add", "--force", str(dest), branch], cwd=self.source)
    else:
        _git(["worktree", "add", "--force", "-b", branch, str(dest), base], cwd=self.source)
    return dest

# orchestrator._compose_env — a container namespace per card
def _compose_env(issue_key: str) -> dict:
    # So N concurrent jobs don't collide on container/network/volume names — and so
    # `make down` tears down the same stack `make up` brought up.
    # ponytail: names are isolated; if the stack binds FIXED HOST PORTS, the
    # consumer repo's compose must make those per-project too.
    slug = re.sub(r"[^a-z0-9]+", "-", issue_key.lower()).strip("-")
    return {"COMPOSE_PROJECT_NAME": f"factory-{slug}"}

Backpressure needs no separate mechanism: the pool is bounded, so the number of agents in flight is the number of workers, and the default (four) is set by the ceiling the target repo can actually run. Stopping needs no protocol either — a file on disk, checked by every worker between jobs, external to the loop's own logic so it works even when the loop is confused.

worker.py — bounded pool, file kill switchstop is a file
"""Bounded pool = backpressure (N agents in flight, no more). A kill switch (a
file on disk) stops the loop cleanly mid-run, external to the loop's own logic."""

def killed(self) -> bool:
    return self._stop.is_set() or (self.kill_switch is not None and self.kill_switch.exists())

def _loop(self, worker_id):
    while not self.killed():
        if not self.run_one():
            time.sleep(self.idle_sleep)      # nothing ready — don't spin hot

# halt the whole factory, from anywhere, with no signal handling:  touch state/STOP

Cross-process control: the board writes files, the daemon owns state

The dashboard and the daemon are separate processes with separate caches of the same job files. If the dashboard mutated a job directly it would race the daemon's write-through flush. So the board never writes job state — it drops a control file, and the daemon applies it on the next lease, owning every status transition. Same mechanism as the kill switch, for the same reason.

queue/control.py + apply_controls — one writer, againfile-based IPC
# The dashboard and the daemon are separate processes with separate FileStore
# caches, so the dashboard must NOT mutate job files directly. It drops a control
# file here; the daemon's FileStore applies it (owning all status transitions).

if   action == PAUSE  and job.status == QUEUED: job.status = PAUSED
elif action == RESUME and job.status == PAUSED: job.status = QUEUED
elif action == RESUME and job.status == DEAD:
    job.status, job.attempts = QUEUED, 0         # retry with a fresh attempt budget
elif action == CANCEL and job.status in (QUEUED, PAUSED, DEAD):
    job.status = CANCELLED                       # … and the card's worktree gets reset
elif action == CANCEL and job.status == LEASED:
    self.trace(job_id, "warn", "cancel_pending_running", {})
    # can't preempt a running subprocess; leave pending for a requeue

Five collisions, five answers

  1. Same card twice Two jobs for one card — a resume arriving while an execute is mid-flight. Prevented in _next_ready: the second is not leasable until the first's lease ends.
  2. Two cards, one tree Prevented by a worktree per card. Isolated checkouts on their own branches, sharing one .git, cleaned up on merge or drop.
  3. Two stacks, one name Prevented by a per-card compose project name. The known ceiling is stated in the code: fixed host ports still need the target repo's compose to be per-project.
  4. Too many agents Prevented by the pool size — the only throttle, and it is one integer. Four is the target repo's port-slot cap, not a guess about model limits.
  5. Cancel mid-run A leased job cannot be preempted, so the cancel is traced and left pending; when it lands, a reset job stops the stack and hard-resets the worktree to base, leaving the card cleanly restartable.
Without it

Two agents in one worktree produce interleaved edits that no review can untangle and no test can localise — the diff is a merge of two intentions. Two container stacks under one project name mean one card's make down silently kills another card's database. And an unbounded pool discovers its own limit the expensive way: rate limits, a saturated laptop, and a set of half-finished branches whose runs died together.

For the book

Serialize where writes land; parallelize everything else. The important part is where the serialization lives: in the queue, as a property of leasing, not as an instruction in a prompt an agent may reason its way around. A concurrency rule an agent has to remember is not a concurrency rule — and the corollary is that giving each unit of work its own ground (its own tree, its own stack, its own namespace) removes more coordination than any amount of locking adds.

5 Lit loop

Lights on: gates before the action, the record after it

Lights-on is not "nobody is watching" — it is the opposite, and it has two halves. Before an action: governance gates raise the decision to a human — an approval touchpoint, an interrupt on ambiguity, a merge gate that refuses a red pull request. After it: observability and provenance — a trace line per completed step, a persisted artifact per step, a cost per card, every decision mirrored where the work lives. Autonomy is a separate dial, and it moves without disturbing either half.

audit state/traces.jsonl readmodel observability/ · sqlite (rebuildable) interrupt harness/interrupts.py dial FACTORY_OVERSIGHT gate harness/gates.py · merge_ready loop harness/improve.py

One card, start to finish

  1. Draft A model turn drafts the intent contract — outcome, acceptance, scope in and out, open questions — against a JSON schema, so the next step gets fields rather than prose. The rendered contract is written back to the card and the card moves to await approval.
  2. Dispatch No model. Prepare the worktree, checkpoint the branch and path, bring the stack up once, assign the card, move it to In Progress. Every one of those is durable, so a crash here costs seconds.
  3. Execute The agent runs inside the worktree with the contract inlined — the card is the source of truth, so the executor is handed it rather than left to fetch it. Then commit, count added lines, and verify.
  4. Verify The harness dispatches the target repo's own Stop hooks — its authored make pre-commit and test commands — and falls back to a configured test command only if the repo has none. Commands belong to the charter; dispatch belongs to the harness.
  5. Review Push, open the pull request, move to In Review, and chain an automated review job that returns structured findings and a suggested verdict — recorded as an artifact and mirrored as a comment.
  6. Merge merge_ready is a hard precondition even on a human accept: a red or closed pull request cannot be squashed. Merge, move to staging, then tear down — stop the stack, remove the worktree.

Two things make that sequence auditable rather than merely automatic. Every step ends with a hard, persisted artifact — which is exactly what lets the queue resume it and an auditor replay it. And the audit log is the queue's own append-only traces.jsonl: the dashboard's sqlite database is a projection of that file, rebuildable at will, never the source of truth. Files are the store; the tracker and the pull request are mirrors, kept current so decisions are visible where the work lives.

The two halves, mechanism by mechanism

  1. Before · gate Approval touchpoints. Two moves in the whole pipeline can require a recorded human approval before they happen — promoting a drafted contract, and merging a reviewed pull request. Which of them need one is the dial's only job.
  2. Before · gate Interrupts. The executor that hits a real decision raises a question instead of choosing. The card blocks — no further action is taken on it — until a human answers on the board.
  3. Before · gate Deterministic preconditions. merge_ready refuses a red or closed pull request even when a human has already said accept, and at lights-out settings a blocking review finding holds the card instead of auto-accepting it.
  4. After · record Traces. Every enqueue, lease, checkpoint, model turn, failure and completion appends a line to traces.jsonl once the step has happened — including the agent's own step-by-step output, flushed into the job's log as each turn lands.
  5. After · record Artifacts and cost. Each phase persists its structured result, and each job carries the dollars and tokens it burned — including on the attempts that failed.
  6. After · record Provenance. Contracts, interrupt answers, verdicts and review findings are mirrored onto the card and the pull request, so the decision trail sits where the work lives rather than only in the harness's own files.

Ambiguity becomes a question, not a guess

An agent that hits a real decision point has two bad options — pick something, or stop — and one good one. The execute schema carries a needs_input flag; when it comes back set, the harness files an interrupt, the card blocks, and the question appears on the board for a human. The answer is written back, the reconciler enqueues a resume carrying the feedback, and the same answer is mirrored to the card and the pull request so the decision is recorded where the work is.

orchestrator._execute — the interrupt pathblocks, does not guess
data = res.data or {}
# structured signal: the model needs a human decision → raise an interrupt
# (resolved on the board), mirrored to the tracker for visibility.
if data.get("needs_input"):
    question = data.get("question") or "The executor needs a decision to proceed."
    self.interrupts.raise_(key, question, job_id=ctx.job.id)
    self.contracts.comment(key, f"factory:interrupt {question}")
    return "interrupted"          # stays In Progress; the reconciler blocks the card

# … and on the way back, in the reconciler:
if intr["status"] == RESOLVED:
    self._retire_poison(full.key, "resume")   # a prior failed resume must not wedge this one
    self.store.enqueue("resume", {"issue_key": full.key, "feedback": intr["feedback"]})

The dial

Autonomy is a setting, not a rewrite. Every level runs the same handlers and writes the same audit trail; they differ only in which transitions need a recorded human approval first. There are exactly two human touchpoints to place on that scale — approving a drafted contract, and merging a reviewed pull request — and at the middle setting the default gate is just the merge.

FACTORY_OVERSIGHT — the same factory, five amounts of human contact
LevelApproval requiredWho proposesAudit trail
manualevery transitiona human drivesfull
assisted (default)every transitionAI proposes, human acceptsfull
supervisedgated steps only — merge by defaultAI proposesfull
autonomousnone; humans notified out of bandAIfull
darknone — fully lights-outAIfull

Lights-out is not the same as ungated. At the two highest settings a card in review auto-accepts only if the automated review's findings pass a deterministic gate; a blocking finding holds the card and marks it on the board for a human, who can still accept it in the app. The dial removes the human's obligation to touch every move — it does not remove the gates.

The loop that changes the rules — and the asymmetry

The factory can mine its own failures, propose a change, validate it, and adopt it. Which raises the obvious question: adopt it where? The answer is the governor. A charter change is authored text, cheap to review and trivial to revert, so a validated proposal is applied. A harness change is code, so a proposal is only ever recorded — it becomes a human-reviewed edit like any other.

orchestrator._improve — apply the charter, only record the harnessasymmetric by design
"""Self-improvement at two levels:
  - level 'charter' (scoped to a job/card): mine that card's failures, propose a
    charter guide, validate, and APPLY it (charter is authored, reversible, cheap).
  - level 'harness' (whole system): mine across all jobs, propose changes, but
    only RECORD them — a harness/code change is a human-reviewed edit."""

for p in proposals:
    valid = loop.validate(p)                       # validated against evals/
    if valid and level == "charter":
        loop.accept(p)                             # applied to the repo's .claude/rules
    out.append({"id": p.primitive_id, "accepted": p.accepted,
                "rationale": p.rationale, "validation": p.validation, "level": level})

The same split runs through the whole system, and it is the one line worth carrying away from this page: authored → charter; applied → harness. Behaviour specifications — rules, gates, skills, the commands that verify a change — live in the target repo, where the people who own that code can author them. The machinery that runs specifications — the loop, the queue, dispatch, isolation, the dial — lives in the harness, and knows nothing about what it is delivering. That is why there is no domain vocabulary anywhere in these 5,300 lines, and why the factory can be pointed at a different repo on a Tuesday without a rewrite.

Without it

Skip the dial and autonomy becomes all-or-nothing: a system nobody trusts enough to leave running, or one running unattended before anyone has earned the right to. Skip the asymmetry and it is worse — a harness that edits its own code from mined failures is a system whose behaviour drifts without a diff anyone reviewed. And skip the persisted artifact at each step and none of the rest is even measurable: a reported success whose write is invisible in external state did not happen.

For the book

A harness by definition, a factory by operation, a queue by design — and the three are not separate claims. It is a harness because it applies a charter it does not author, which is what keeps domain knowledge out of the engine. It is a lights-on factory because a gate raises the decision to a human before an action and provenance records what happened after it — so autonomy becomes a dial you can turn rather than a leap you take. And it is a queue because durability, retry, isolation, cost and audit are all properties of a job record — which is what makes the other two claims survive contact with a crash at 3am.