Harness Engineering / A lights-on factory / Anatomy of a running system

A harness by definition, a factory by operation, a queue by design

A lights-on factory: it takes a tracker card and returns a merged pull request. The lights are on in two distinct senses. Gates raise a decision to a human before the factory acts. Then observability and provenance record what happened after it. Autonomy is a separate dial. It is about 5,300 lines of Python: three runtime dependencies, one model dependency, sixteen job handlers, seven seams, 193 tests. Nothing in it knows what the code it delivers is for. No product vocabulary, no business rules, no domain types. That absence is a design property, not an omission: the factory is the machinery. Everything domain-specific lives in the target repo's charter, where its owners change it. The five rungs below ascend in build order: one dependency, one pure function, one durable queue, one writer per card, one lit loop.

harness surface · ~5,300 lines of Python · 3 runtime deps (fastapi · uvicorn · pyyaml) · 1 model dep · 16 job handlers · 7 seams · 193 tests · state is files
substitutions · make = your task runner · PROJ-000 = your issue key · tracker and code host = whatever you already run
Deterministic
A pure function or a gate decides. No inference, no spend, unit-testable.
Model turn
The expensive, non-deterministic move. Spent only where the work needs inference.
Durable
A persisted artifact: a file the queue can resume from and an auditor can read.
1 One dependency

The smallest model dependency that does the job

This harness reaches for exactly one library to run an agent: the Claude Agent SDK. Not because frameworks are bad, but because the SDK is Claude Code, the harness this shop already prefers. Point it at a directory and the target repo's charter, hooks, skills and MCP servers load natively. Everything a framework would add on top is either already there or not needed yet.

dep claude-agent-sdk>=0.1 seam providers/base.py · ModelProvider impl providers/claude_sdk.py impl providers/stub.py config FACTORY_MODEL · FACTORY_MODEL_API_KEY

The whole dependency list is short enough to print. Three packages run the daemon and its board; one optional extra runs the model. There is no workflow engine, no graph library, no agent framework, and no vector store.

pyproject.toml: the entire dependency surfaceone model dep
dependencies = [
  "fastapi>=0.115",   # the board's API
  "uvicorn>=0.30",    # serves it
  "pyyaml>=6.0",      # charter frontmatter
]

[project.optional-dependencies]
# the model seam: claude-agent-sdk runs Claude Code (drives the `claude` CLI,
# which the image installs). `anthropic` is only for the eval llm_judge — a
# single Messages-API completion, no agent.
agent = ["claude-agent-sdk>=0.1", "anthropic>=0.40"]

One dependency is enough because the SDK is not a thin HTTP client: it runs Claude Code in-process. Point it at cwd=<the task's worktree> and it loads that repo's CLAUDE.md, its glob-scoped .claude/rules, its skills, its .mcp.json servers and its settings.json hooks. It loads them natively, so the harness reproduces none of it. A framework-based harness has to rebuild all of that: context assembly, tool definitions, permission modes, hook dispatch. This harness injects only the one layer Claude cannot know about, cross-repo governance policy, and gets the rest free.

providers/base.py: the seam, in full~30 lines
@dataclass
class ModelResult:
    text: str                      # final assistant text
    cost_usd: float = 0.0
    tokens: int = 0
    ok: bool = True
    error: str | None = None
    data: dict | None = None       # parsed structured output


class ModelProvider(ABC):
    @abstractmethod
    def run(self, prompt: str, *, cwd: Path | None = None, extra_dirs: list[Path] | None = None,
            timeout: float = 3600, schema: dict | None = None, phase: str = "",
            on_event=None) -> ModelResult:
        """Run one agent turn. The harness NEVER shells out to a model
        directly; it goes through here."""

Subscription auth, and why it decides the architecture

One provider covers both auth modes, and the default is the interesting one. Configure an API key and the provider passes it through to the SDK. Leave it unset and the SDK falls back to ambient claude CLI authentication: the subscription this shop already pays for. That removes the token to mint, rotate, scope or leak. It also removes the per-token invoice for a factory whose normal state is four long agent turns at once.

providers/claude_sdk.py: the option kwargskey optional, by design
self.api_key = api_key or None       # "" → None: no key means ambient CLI auth

kwargs = {
    "cwd": str(root),
    "permission_mode": "bypassPermissions",   # headless: no interactive approval
    "disallowed_tools": self._disallowed(phase),
}
if self.model:
    kwargs["model"] = self.model
if self.api_key:                     # ONLY when explicitly configured
    kwargs["env"] = {"ANTHROPIC_API_KEY": self.api_key}

# Two behaviors are set explicitly because SDK defaults differ from `claude -p`:
# the Claude Code system prompt (SDK apps default to none) and permission bypass.
# `setting_sources` is left at the SDK default — all sources, i.e. CLI parity.
return ClaudeAgentOptions(system_prompt={"type": "preset", "preset": "claude_code"}, **kwargs)

The harness still measures cost. The SDK's terminal ResultMessage carries total_cost_usd and a token count, so every job records what the turn would have cost. That record holds whether or not the vendor metered anything. So rung 5 can price a card without an invoice.

When to reach past the SDK

This is a threshold, not a preference. The SDK is the right tool while the flow you need is the flow a stock harness already runs. A framework earns its place the moment you need something a stock harness cannot give you. Two cases qualify: a genuinely custom control flow, or the ability to swap the model underneath. Pydantic AI, LangChain, LangGraph and deepagents are the right answers to that need.

  1. Stock flow Run an agent inside a repo, under that repo's own charter, and get structured output back. The SDK does this natively: schema= plus a JSON-extraction helper is the entire structured-output layer here.
  2. Orchestration Retries, leases, dead letters, checkpoints, backpressure. A durable queue, ~200 lines of it (rung 3), not a workflow engine. The queue is the orchestration.
  3. Custom flow A control flow no stock harness will run for you. Four shapes qualify: a graph with joins and conditional fan-in, per-node memory policies, a human-approval node mid-graph, cross-process resumption of a partial graph. Reach for the framework.
  4. Model choice Swapping models or providers per step: a cheap model to classify, an expensive one to implement, a third to judge. A provider-agnostic framework is the honest way to do that; a single-vendor SDK is not.
  5. Not yet Neither threshold has arrived here, so neither dependency exists here. If one arrives, the change is one class behind ModelProvider: the whole reason the seam went in early.
Without it

Start with a framework and you pay twice. Once in reproduction: the framework's runtime re-implements charter loading, tool policy, hook dispatch and permission modes Claude Code already does. Your versions then drift from the ones your engineers use interactively. And once at the meter: framework paths authenticate with a provider API key. So an unattended factory bills per token from the first run, before you know whether the flow is right at all.

For the book

Take the highest rung of the ladder that holds. If your preferred harness is Claude Code, its SDK is your agent framework. Reaching past it buys custom flow and model portability. Until you need those two, it buys only surface area. The discipline that makes this safe is not the choice, it is the seam. Thirty lines of abstract class keep the decision reversible. And a stub implementation of the same interface runs the whole factory in tests with no model at all.

2 Pure function

Derive, don't remember

The daemon holds no opinion about where a card is. Every tick it reads current external state and hands a snapshot to a pure function. That function returns at most one action to enqueue. The controller model, not the event model: if the daemon dies mid-run, external state stays as it was and the next tick resumes from it.

pure harness/derive.py · derive_action pure derive.py · route_feedback loop harness/reconciler.py · tick seam Tracker · CodeHost guardrail per-issue at-most-one-active guardrail poison guard

derive.py has no imports that touch the network, the disk or the clock. It maps one issue snapshot to one decision, and the pipeline it encodes is a table anyone can read. Backlog → Needs Details → To Do → In Progress → In Review → In Staging → Done. Those seven states span four phases: Discover → Define → Dispatch → Verify.

harness/derive.py: the state machine, no I/Odeterministic
@dataclass(frozen=True)
class Decision:
    action: str | None       # job fn to enqueue, or None (waiting / terminal)
    reason: str

def derive_action(issue: Issue) -> Decision:
    s = issue.status
    if s == BACKLOG:
        # nothing rendered yet, or new feedback since the last render
        return Decision("draft", "in Backlog → draft the contract") if _needs_refine(issue) else NOTHING
    if s == NEEDS_DETAILS:
        if issue.has_marker(M_APPROVE):
            return Decision("approve", "contract approved → move to To Do")
        return Decision("draft", "draft the contract") if _needs_refine(issue) else NOTHING
    if s == TO_DO:      return Decision("dispatch", "approved and ready → dispatch executor chain")
    if s == IN_PROGRESS: return Decision("execute", "in progress → run/continue executor chain")
    # In Review (verdict), In Staging / Done (terminal), Backlog (awaiting triage)
    return NOTHING

Note what isn't here. No model call decides where a card goes next. Routing feedback is the same story. The card's status decides whether a comment means "revise the open PR" or "file a follow-up card", in one line, for free:

derive.py: routing without inferenceone line, testable
def route_feedback(status: str) -> str:
    """In-flight work is revised on the same PR; shipped or not-yet-started work
    is triaged as a follow-up card, never touching a frozen PR."""
    return "revise" if status in (IN_PROGRESS, IN_REVIEW) else "triage"

The reconciler is the impure half, and it is deliberately dull: read the tracker, call derive_action, enqueue, write a snapshot for the board. All the subtlety is in not enqueueing: three guards, each answering a different failure.

harness/reconciler.py: the three guards on enqueuerefuses duplicates
def _has_active(self, key, action):     # already queued or leased → skip
    return any(j.status in (QUEUED, LEASED) and j.fn == action
               and j.payload.get("issue_key") == key for j in self.store.all())

def _is_poisoned(self, key, action):
    # A dead-lettered OR operator-cancelled (key, action) blocks re-enqueue —
    # otherwise a failing action re-derives EVERY TICK, forever. The operator
    # retries (resume) or drops (cancel) it from the job page.
    return any(j.status in (DEAD, CANCELLED) and j.fn == action
               and j.payload.get("issue_key") == key for j in self.store.all())

# a drop or a from-scratch restart in flight pre-empts other work for the card
_PREEMPT = ("close", "restart")

def _retire_poison(self, key, action):
    # Newer operator feedback SUPERSEDES a failed prior attempt, so it stops
    # counting as poison and a fresh job runs. Every feedback yields a NEW job —
    # we never revive an old one.
    ...

The tracker and the code host are ports too

The reconciler reads two systems it does not own. The cards live in the tracker, and the branches and pull requests live on the code host. Neither belongs in derive_action. The same discipline that put ModelProvider in front of the agent puts a port in front of each. The pure function sees an Issue snapshot and returns a decision. Meanwhile the vendor's REST shapes, field ids, status names and rate limits stay on the far side of an adapter.

a sketch, not shipped code: two more ports, one adapter per vendorthe shape it wants
class Tracker(Protocol):              # the ticketing system: cards in, decisions mirrored back
    def issues(self) -> list[Issue]: ...
    def transition(self, key: str, status: str) -> None: ...
    def comment(self, key: str, body: str) -> None: ...

class CodeHost(Protocol):             # branches, pull requests, checks
    def open_pr(self, branch: str, title: str, body: str) -> PR: ...
    def checks_green(self, pr: PR) -> bool: ...
    def merge(self, pr: PR) -> None: ...

TRACKERS   = {"jira": JiraTracker, "linear": LinearTracker, "gh-issues": GhIssuesTracker}
CODE_HOSTS = {"github": GitHubHost, "gitlab": GitLabHost}
FAKES      = {"tracker": InMemoryTracker, "host": InMemoryHost}   # the 193 tests run on these

Each adapter absorbs the vendor-shaped part: how the vendor names and transitions a status, and whether a merge is a squash or a rebase. The same adapter owns which webhook or poll tells you a check went red, and how the vendor scopes the credential. None of them touch the pipeline. Swapping Jira for Linear, or GitHub for GitLab, is a class and its fake. The state machine, the guards and the queue do not know a substitution happened. That seam also makes tracker and code host = whatever you already run a claim rather than a hope. It lets the whole delivery loop run in tests against in-memory doubles. A controller that derives from external state is only as testable as the seam it reads that state through.

What the controller model buys

  1. Crash safety Kill the daemon at any instant and it loses nothing, because it held nothing. State lives in the tracker, the code host, git and the queue's files. The next tick reads the world as it is and continues.
  2. Testability The interesting logic is a pure function over a dataclass. Fast tests cover the entire pipeline, every transition and every route, with no daemon, no network and no model.
  3. Cost Routing decisions cost nothing. The factory spends inference on drafting a contract, implementing a change and reviewing a diff: the three places needing judgment.
  4. Idle discipline A card mid-approval derives NOTHING rather than re-rendering. _needs_refine re-drafts only when new feedback arrives after the last render, so a waiting card is not re-spent every tick.
  5. Free observability The same pass writes pipeline.json: cards grouped by stage, each with its next action. That file is the board's entire data source, and the projection is a by-product of deciding.
Without it

Go event-driven with state in the process and two failure modes arrive together. A missed or duplicated webhook wedges a card in a state nobody can name. And a deploy mid-run loses the only copy of "where were we". Then, because in-process state is unreliable, the natural fix is to ask the model where the card is. That fix is expensive, non-deterministic and untestable: three properties you do not want in a routing decision.

For the book

Deterministic decisions stay in the harness, and a model turn goes only where the work needs inference. The test for a rule is whether you can write it as a pure function over external state. If you can, it does not belong in a prompt. Derive, don't remember is the same discipline at the system level that "read the file, don't recall it" is at the agent level.

3 Durable queue

The queue is the design, not the plumbing

Every property this factory needs turns out to be a queue property. Resume after a crash? A lease. Survive a transient failure? Bounded retries with backoff. Stop a wedged card from burning money forever? A dead-letter. Know what a card cost? A field on the job. Limit how many agents run at once? The size of the pool. Get all of it in about 200 lines of Python over files on disk: no broker, no database, no daemon but your own.

seam queue/store.py · Queue impl queue/file_store.py · FileStore model queue/job.py · Job · JobState pool queue/worker.py · WorkerPool ctx worker.py · Context.save / bill registry queue/registry.py

The unit of work is a job, and the job record is the whole contract between loop and world. Every later question is a field on that record: what ran, how many times, what it cost, what it had gotten through before it died.

queue/job.py: the unit of workone file per job on disk
@dataclass
class Job:
    id: int
    fn: str                                  # handler name in the registry
    payload: dict = field(default_factory=dict)
    status: JobState = JobState.QUEUED
    attempts: int = 0
    max_attempts: int = 3
    idempotency_key: str | None = None
    lease_until: float = 0.0                 # visibility timeout while leased
    run_after: float = 0.0                   # backoff / delayed scheduling
    parent_id: int | None = None             # chained job trees
    error: str | None = None
    result: Any = None
    checkpoint: dict = field(default_factory=dict)   # durable mid-job state
    cost_usd: float = 0.0
    tokens: int = 0
    created_at: float = field(default_factory=now)
    updated_at: float = field(default_factory=now)
The state machine of a job: who moves it, and what it means
StateSet byMeaningLeasable
queuedenqueue · failready, or waiting out a backoff (run_after)yes
leasedleasea worker holds it until lease_untilon expiry
donecompletesucceeded; result, cost and tokens recordedno (final)
deadlease (attempts exhausted)dead-letter: off the main queue, kept on diskvia resume
pausedoperator control fileheld by a human; not leasable until resumedvia resume
cancelledoperator control fileterminated by a humanno (final)
supersedednewer feedbackretired, kept for history, no longer counts as poisonno (final)

superseded is the state worth stealing. Without it, a dead-lettered job is permanent poison. The reconciler refuses to re-enqueue that (card, action) pair, so nothing can retry a card that failed once. The operator's button does nothing. With it, newer human feedback retires the corpse and a fresh job runs, history intact, no zombie revived.

Durability: write-through, not write-later

The store keeps an in-memory cache for speed and a file per job for truth. Every mutation updates the cache and immediately flushes JSON to disk; startup rehydrates the cache from the files. There is no migration, no schema and nothing to back up beyond a directory. You can read the contents with cat at three in the morning, which is a feature.

queue/file_store.py: durability in six linesstate is files
"""Write-through queue store: in-memory cache for speed AND a file per job for
durability. Every mutation updates the cache then flushes to disk. On startup we
rehydrate the cache from the files, so a crash resumes where it left off.

Concurrency: a single re-entrant lock serializes writes. Correct and simple at
this scale. ponytail: global lock, shard by job-id if throughput ever matters."""

def _rehydrate(self):
    for f in sorted(self.jobs_dir.glob("*.json")):
        job = Job.from_dict(json.loads(f.read_text()))
        self._cache[job.id] = job
        if job.idempotency_key:
            self._idem[job.idempotency_key] = job.id
        self._seq = max(self._seq, job.id)

def _flush(self, job):
    job.updated_at = self._now()
    (self.jobs_dir / f"{job.id}.json").write_text(json.dumps(job.to_dict(), indent=2))

Leasing: crash recovery without a heartbeat

lease() is where four mechanisms meet in twenty lines. It hands out one ready job and reclaims leases that expired because a worker died. It also increments the attempt counter and dead-letters anything that has burned its budget. Nothing else in the system needs to know that a worker crashed.

file_store.py: lease, retry and dead-letter in one placeexhausted → dead
def lease(self, lease_seconds):
    with self._lock:
        self.apply_controls()                # operator pause/cancel first
        t = self._now()
        while True:
            candidate = self._next_ready(t)
            if candidate is None:
                return None
            candidate.attempts += 1
            if candidate.attempts > candidate.max_attempts:
                candidate.status = JobState.DEAD    # dead-letter, off the queue
                self._flush(candidate)
                self.trace(candidate.id, "error", "dead_letter", {"fn": candidate.fn})
                continue                     # … and look at the next candidate
            candidate.status = JobState.LEASED
            candidate.lease_until = t + lease_seconds
            self._flush(candidate)
            return candidate

A one-hour visibility timeout fits agent work and not a restart, so startup has its own recovery. At that moment no worker holds a lease, so every leased job is an orphan. Startup requeues those orphans at once instead of waiting the timeout out.

file_store.py: orphan recovery at bootno waiting out an hour
def recover_orphaned(self) -> int:
    """Requeue jobs left LEASED by a previous process (crash/restart). At startup
    no workers hold leases, so a leased job is orphaned."""
    for job in self._cache.values():
        if job.status == JobState.LEASED:
            job.status = JobState.QUEUED
            job.run_after = self._now()
            self._flush(job)
            self.trace(job.id, "warn", "recovered_orphan", {"fn": job.fn})

Checkpoints: the difference between retry and restart

A retried agent job is not a pure function. Its first attempt may already have brought up a container stack, created a branch or opened a pull request. The checkpoint is what keeps a retry from redoing the expensive, side-effecting parts. Handlers get a Context whose save() writes durable mid-job state and whose bill() accrues spend, and the pattern in practice is a single guard:

the checkpoint pattern: do the costly thing onceretry-safe
# orchestrator._dispatch: bring the stack up ONCE, not per retry
if self.setup_cmd and not ctx.checkpoint.get("stack_up"):
    ok, tail = _run_cmd(self.setup_cmd, path, env=_compose_env(key))
    ctx.save(stack_up=True)              # flushed to the job file immediately
    ctx.trace("stack_setup", {"cmd": self.setup_cmd, "ok": ok, "tail": tail})

# worker.Context — the only sanctioned way to persist mid-job state
def save(self, **state):
    self.job.checkpoint.update(state)
    self.store.checkpoint(self.job.id, self.job.checkpoint)

def bill(self, cost_usd, tokens=0):          # worker bills this on complete AND on fail
    self.cost_usd += cost_usd
    self.tokens += tokens

Billing on failure is the small decision that makes cost data trustworthy. A failed attempt spent real money; if only successful jobs recorded their spend, the factory's own numbers would flatter it. So fail() takes cost_usd too, and the job's total is the sum of everything it ever burned.

worker.py: every failure routes through the queuehandlers never crash the loop
def backoff_for(attempt, base, cap):
    return min(cap, base * (2 ** max(0, attempt - 1)))   # exponential, capped

try:
    result = handler(ctx, job.payload)
    self.store.complete(job.id, result, cost_usd=ctx.cost_usd, tokens=ctx.tokens)
except Exception as exc:  # noqa: BLE001 — route every failure through the queue
    self.store.fail(job.id, f"{type(exc).__name__}: {exc}",
                    backoff=backoff_for(job.attempts, self.backoff_base, self.backoff_cap),
                    cost_usd=ctx.cost_usd, tokens=ctx.tokens)   # bill the failed attempt

What each mechanism actually buys

  1. Idempotent enqueue A repeated idempotency_key returns the original job id and traces a dedupe. Duplicate ticks, retried webhooks and an operator double-click are all harmless by construction.
  2. Lease + timeout At-least-once delivery with crash recovery and no heartbeat protocol. A dead worker's job becomes leasable again on expiry; a restart shortcuts that with orphan recovery.
  3. Backoff Capped exponential (60s base, 30min cap). A rate limit or a flaky remote costs waiting, not a tight retry loop against someone else's service.
  4. Dead-letter Three attempts and the job leaves the queue instead of retrying forever. With the reconciler's poison guard, a wedged card stops spending in a state a human can see and act on.
  5. Checkpoint Resume rather than restart. The branch, worktree path and "stack is already up" flag survive a crash, so attempt two continues instead of re-paying.
  6. Append-only trace Every enqueue, lease, checkpoint, failure and completion appends one JSON line to traces.jsonl. That file is the audit log. The dashboard's database is a projection of it, rebuildable at any time.
Without it

Run the same loop with in-process state and a for loop over cards. The failure is not that it breaks. It is that it breaks invisibly and expensively. A retry re-runs a handler whose first attempt already opened a pull request. A restart loses the fact that a container stack is up, and the next attempt collides with it. A card that fails deterministically retries forever, and you find out from the bill. None of these need a distributed system to appear; they appear on one laptop, on the first bad afternoon.

For the book

Choose the durable unit first and the rest of the system hangs off it. The job record carries cost, attempts, checkpoint and result. That gives the factory pricing, observability, operator control and crash recovery without a second mechanism for any of them. Agentic orchestration is a queue problem wearing a new hat. The boring, decades-old answers are idempotency keys, visibility timeouts, dead-letters and capped backoff. Those four hold even when the worker is a non-deterministic agent that runs for twenty minutes.

4 Concurrency

One writer per card, parallel everywhere else

Four agents run at once here, and none of them can collide. That is because the serialization point is in the leasing function, not in anyone's prompt. Per-card single-writer in the queue, a git worktree per card, a container namespace per card, and a bounded pool that is the backpressure. The operator's controls arrive as files, because the board and the daemon are different processes.

gate file_store._next_ready policy policies/single-writer.md seam harness/workspace.py · GitWorkspace config FACTORY_WORKERS=4 control queue/control.py · ControlStore switch state/STOP

The single-writer guarantee is eight lines inside the function that picks the next job. That function skips a job carrying an issue_key while another job for that same key holds a live lease. Two agents therefore never write one worktree, and not because a prompt told them not to: the queue never hands the second one the work.

file_store._next_ready: the single-writer guaranteeskips a busy card
def _next_ready(self, t):
    # Per-issue serialization: at most one job per issue_key runs at a time, so two
    # jobs never write the same worktree concurrently. Jobs without an issue_key
    # are not serialized.
    active_keys = {
        self._issue_key(j) for j in self._cache.values()
        if j.status == JobState.LEASED and j.lease_until > t and self._issue_key(j)
    }
    ready = []
    for j in self._cache.values():
        if j.status == JobState.QUEUED and j.run_after <= t:
            if self._issue_key(j) and self._issue_key(j) in active_keys:
                continue                          # another job for this card is in flight
            ready.append(j)
        elif j.status == JobState.LEASED and j.lease_until < t:
            ready.append(j)                       # expired lease → reclaimable
    return min(ready, key=lambda j: j.id) if ready else None   # FIFO by id

Serializing per card is only half of it. The other half stops two different cards from colliding. Each card gets its own git worktree off the source repo. It shares .git, so it is fast and there is no full clone per card. Each card also gets its own container project namespace. So make up and make down refer to the same stack and to nobody else's.

isolation, twice: the tree and the stackno shared mutable ground
# harness/workspace.py: a worktree per card, idempotent (resume returns the existing one)
def prepare(self, issue_key, branch, base="main") -> Path:
    dest = self._dir(issue_key)
    if dest.exists():
        return dest                                   # resume: worktree already there
    _git(["worktree", "prune"], cwd=self.source)      # clear stale registrations
    if self._branch_exists(branch):
        _git(["worktree", "add", "--force", str(dest), branch], cwd=self.source)
    else:
        _git(["worktree", "add", "--force", "-b", branch, str(dest), base], cwd=self.source)
    return dest

# orchestrator._compose_env — a container namespace per card
def _compose_env(issue_key: str) -> dict:
    # So N concurrent jobs don't collide on container/network/volume names — and so
    # `make down` tears down the same stack `make up` brought up.
    # ponytail: names are isolated; if the stack binds FIXED HOST PORTS, the
    # consumer repo's compose must make those per-project too.
    slug = re.sub(r"[^a-z0-9]+", "-", issue_key.lower()).strip("-")
    return {"COMPOSE_PROJECT_NAME": f"factory-{slug}"}

Backpressure needs no separate mechanism. A bounded pool means the number of agents in flight is the number of workers. The ceiling the target repo can run sets the default, which is four. Stopping needs no protocol either: a file on disk, which every worker checks between jobs. That file sits outside the loop's own logic, so it works even when the loop has lost track.

worker.py: bounded pool, file kill switchstop is a file
"""Bounded pool = backpressure (N agents in flight, no more). A kill switch (a
file on disk) stops the loop cleanly mid-run, external to the loop's own logic."""

def killed(self) -> bool:
    return self._stop.is_set() or (self.kill_switch is not None and self.kill_switch.exists())

def _loop(self, worker_id):
    while not self.killed():
        if not self.run_one():
            time.sleep(self.idle_sleep)      # nothing ready — don't spin hot

# halt the whole factory, from anywhere, with no signal handling:  touch state/STOP

Cross-process control: the board writes files, the daemon owns state

The dashboard and the daemon are separate processes with separate caches of the same job files. If the dashboard mutated a job directly it would race the daemon's write-through flush. So the board never writes job state. It drops a control file, and the daemon applies it on the next lease, owning every status transition. Same mechanism as the kill switch, for the same reason.

queue/control.py + apply_controls: one writer, againfile-based IPC
# The dashboard and the daemon are separate processes with separate FileStore
# caches, so the dashboard must NOT mutate job files directly. It drops a control
# file here; the daemon's FileStore applies it (owning all status transitions).

if   action == PAUSE  and job.status == QUEUED: job.status = PAUSED
elif action == RESUME and job.status == PAUSED: job.status = QUEUED
elif action == RESUME and job.status == DEAD:
    job.status, job.attempts = QUEUED, 0         # retry with a fresh attempt budget
elif action == CANCEL and job.status in (QUEUED, PAUSED, DEAD):
    job.status = CANCELLED                       # … and the card's worktree gets reset
elif action == CANCEL and job.status == LEASED:
    self.trace(job_id, "warn", "cancel_pending_running", {})
    # can't preempt a running subprocess; leave pending for a requeue

Five collisions, five answers

  1. Same card twice Two jobs for one card: a resume arriving while an execute is mid-flight. Prevented in _next_ready: the second is not leasable until the first's lease ends.
  2. Two cards, one tree Prevented by a worktree per card. Isolated checkouts on their own branches, sharing one .git, cleaned up on merge or drop.
  3. Two stacks, one name Prevented by a per-card compose project name. The code states the known ceiling: fixed host ports still need the target repo's compose to be per-project.
  4. Too many agents Prevented by the pool size, the only throttle, and it is one integer. Four is the target repo's port-slot cap, not a guess about model limits.
  5. Cancel mid-run Nothing can preempt a leased job, so the queue traces the cancel and leaves it pending. When the cancel lands, a reset job stops the stack and hard-resets the worktree to base. The card is then cleanly restartable.
Without it

Two agents in one worktree produce interleaved edits: no review can untangle them and no test can localise them. The diff is a merge of two intentions. Two container stacks under one project name mean one card's make down silently kills another card's database. And an unbounded pool finds its own limit the expensive way: rate limits, a saturated laptop, dead half-finished branches.

For the book

Serialize where writes land; parallelize everything else. The important part is where the serialization lives. It lives in the queue, as a property of leasing, not as a prompt instruction an agent may reason around. A concurrency rule an agent has to remember is not a concurrency rule. The corollary: give each unit of work its own ground, meaning its own tree, its own stack, its own namespace. That isolation removes more coordination than any amount of locking adds.

5 Lit loop

Lights on: gates before the action, the record after it

Lights-on is not "nobody is watching". It is the opposite, and it has two halves. Before an action, gates raise the decision to a human: an approval touchpoint, an interrupt on ambiguity, a gate that refuses a red pull request. After it, observability and provenance record what happened: a trace line per completed step, a persisted artifact per step, a cost per card. And each decision gets mirrored where the work lives. Autonomy is a separate dial, and it moves without disturbing either half.

audit state/traces.jsonl readmodel observability/ · sqlite (rebuildable) interrupt harness/interrupts.py dial FACTORY_OVERSIGHT gate harness/gates.py · merge_ready loop harness/improve.py

One card, start to finish

  1. Draft A model turn drafts the intent contract against a JSON schema, so the next step gets fields, not prose. The contract carries outcome, acceptance, scope in and out, and open questions. The harness writes the rendered contract back to the card, and the card moves to await approval.
  2. Dispatch No model. Prepare the worktree, checkpoint the branch and path, bring the stack up once, assign the card, move to In Progress. Every one of those is durable, so a crash here costs seconds.
  3. Execute The agent runs inside the worktree with the contract inlined. The card is the source of truth, so the harness hands it to the executor rather than leaving it to fetch. Then commit, count added lines, and verify.
  4. Verify The harness dispatches the target repo's own Stop hooks, plus its authored make pre-commit and test commands. If the repo has none, the harness falls back to a configured test command. Commands belong to the charter; dispatch belongs to the harness.
  5. Review Push, open the pull request, and move to In Review. Then chain an automated review job. It returns structured findings and a suggested verdict, recorded as an artifact and mirrored as a comment.
  6. Merge merge_ready is a hard precondition even on a human accept: nothing squashes a red or closed pull request. Merge, move to staging, then tear down: stop the stack, remove the worktree.

Two things make that sequence auditable rather than merely automatic. Every step ends with a hard, persisted artifact: exactly what lets the queue resume it and an auditor replay it. And the audit log is the queue's own append-only traces.jsonl. The dashboard's sqlite database is a projection of that file, rebuildable at will, never the source of truth. Files are the store. The tracker and the pull request are mirrors, kept current so decisions are visible where the work lives.

The two halves, mechanism by mechanism

  1. Before · gate Approval touchpoints. Two moves in the whole pipeline can require a recorded human approval before they happen: promoting a drafted contract, and merging a reviewed pull request. Which of them need one is the dial's only job.
  2. Before · gate Interrupts. The executor that hits a real decision raises a question instead of choosing. The card blocks, and nothing else happens on it, until a human answers on the board.
  3. Before · gate Deterministic preconditions. merge_ready refuses a red or closed pull request even when a human has already said accept. And at lights-out settings, a blocking review finding holds the card instead of auto-accepting it.
  4. After · record Traces. Every enqueue, lease, checkpoint, model turn, failure and completion appends a line to traces.jsonl once the step has happened. The trace includes the agent's own step-by-step output, which flushes into the job's log as each turn lands.
  5. After · record Artifacts and cost. Each phase persists its structured result. Each job carries the dollars and tokens it burned, including on failed attempts.
  6. After · record Provenance. The harness mirrors contracts, interrupt answers, verdicts and review findings onto the card and the pull request. So the decision trail sits where the work lives, not only in the harness's own files.

Ambiguity becomes a question, not a guess

An agent that hits a real decision point has two bad options, pick something or stop, and one good one. The execute schema carries a needs_input flag. When it comes back set, the harness files an interrupt, the card blocks, and the question goes to the board. A human writes the answer back. The reconciler then enqueues a resume carrying the feedback, and mirrors the answer to the card and the pull request. The decision therefore lands where the work is.

orchestrator._execute: the interrupt pathblocks, does not guess
data = res.data or {}
# structured signal: the model needs a human decision → raise an interrupt
# (resolved on the board), mirrored to the tracker for visibility.
if data.get("needs_input"):
    question = data.get("question") or "The executor needs a decision to proceed."
    self.interrupts.raise_(key, question, job_id=ctx.job.id)
    self.contracts.comment(key, f"factory:interrupt {question}")
    return "interrupted"          # stays In Progress; the reconciler blocks the card

# … and on the way back, in the reconciler:
if intr["status"] == RESOLVED:
    self._retire_poison(full.key, "resume")   # a prior failed resume must not wedge this one
    self.store.enqueue("resume", {"issue_key": full.key, "feedback": intr["feedback"]})

The dial

Autonomy is a setting, not a rewrite. Every level runs the same handlers and writes the same audit trail. They differ only in which transitions need a recorded human approval first. Exactly two human touchpoints sit on that scale: approving a drafted contract, and merging a reviewed pull request. At the middle setting the default gate is just the merge.

FACTORY_OVERSIGHT: the same factory, five amounts of human contact
LevelApproval requiredWho proposesAudit trail
manualevery transitiona human drivesfull
assisted (default)every transitionAI proposes, human acceptsfull
supervisedgated steps only (merge by default)AI proposesfull
autonomousnone; humans notified out of bandAIfull
darknone (fully lights-out)AIfull

Lights-out is not the same as ungated. At the two highest settings, a card in review auto-accepts only if the automated review's findings pass a deterministic gate. A blocking finding holds the card and marks it on the board. A human can still accept it in the app. The dial removes the human's obligation to touch every move, but it does not remove the gates.

The loop that changes the rules: and the asymmetry

The factory can mine its own failures, propose a change, validate it, and adopt it. Which raises the obvious question: adopt it where? The answer is the governor. A charter change is authored text, cheap to review and trivial to revert, so the loop applies a validated proposal. A harness change is code, so the loop only records the proposal, which becomes a human-reviewed edit like any other.

orchestrator._improve: apply the charter, only record the harnessasymmetric by design
"""Self-improvement at two levels:
  - level 'charter' (scoped to a job/card): mine that card's failures, propose a
    charter guide, validate, and APPLY it (charter is authored, reversible, cheap).
  - level 'harness' (whole system): mine across all jobs, propose changes, but
    only RECORD them — a harness/code change is a human-reviewed edit."""

for p in proposals:
    valid = loop.validate(p)                       # validated against evals/
    if valid and level == "charter":
        loop.accept(p)                             # applied to the repo's .claude/rules
    out.append({"id": p.primitive_id, "accepted": p.accepted,
                "rationale": p.rationale, "validation": p.validation, "level": level})

The same split runs through the whole system. Carry that away if nothing else: authored → charter; applied → harness. Behaviour specifications live in the target repo: rules, gates, skills, and the commands that verify a change. The people who own that code author them. The machinery that runs specifications (the loop, the queue, dispatch, isolation, the dial) lives in the harness, and knows nothing about what it is delivering. That is why there is no domain vocabulary anywhere in these 5,300 lines. It is also why you can point the factory at a different repo on a Tuesday without a rewrite.

Without it

Skip the dial and autonomy becomes all-or-nothing. Instead you get a system nobody trusts enough to leave running, or one running unattended before anyone earned the right. Skip the asymmetry and it is worse, because a harness that edits its own code from mined failures drifts unreviewed. And skip the persisted artifact at each step and nothing else here is measurable. A reported success whose write is invisible in external state did not happen.

For the book

A harness by definition, a factory by operation, a queue by design, and the three are not separate claims. It is a harness because it applies a charter it does not author. That keeps domain knowledge out of the engine. It is a lights-on factory because a gate raises the decision to a human before an action. Provenance records what happened after it. So autonomy becomes a dial you can turn rather than a leap you take. And it is a queue because durability, retry, isolation, cost and audit are all properties of a job record. That record is what makes the other two claims survive contact with a crash at 3am.