Harness Engineering / One workflow, three architectures / 1 of 3 · Actors & supervision

Half of OTP does not port. It dissolves.

If you have written Elixir, you carry a supervision tree in your head: processes hold state, die, and get restarted. But a durable-queue system models the same territory differently. State outlives the thing computing it, so failure belongs to a message rather than to a process. Most of OTP survives that translation as pure configuration. The parts that do not survive are more interesting, because they answered a question the other architecture never asks.

companion repo · github.com/tacoda/iii-otp
built twice · once as iii configuration, once from scratch in stdlib Python
the workload · the same incident analysis all three repos run : extract claims, propose an explanation, score it, accept
verification · every iii command shown was executed against a live engine (0.22.0) before it was written down
tests · 10 passed, offline and deterministic
Ports as config
The OTP concept has a direct iii equivalent, reachable with a YAML entry or a trigger binding.
Dissolves
The concept has no referent here. Not a missing feature: the problem it solved does not arise.
Genuine gap
OTP gives you something real that this architecture does not, and no configuration closes it.
0 Two lenses

Who owns this state, versus what happened to it

Actors and durable queues are two answers to the same question. They diverge on one decision, and everything else follows from it.

The actor model puts state inside a process. The process is the unit of identity, the unit of concurrency, and the unit of failure, all at once. That bundling is what makes OTP elegant: because the three coincide, one mechanism, supervision, handles all three.

A durable-queue system unbundles them: identity is a key, concurrency is a queue lane, and failure is a message delivery. Because they no longer coincide, no single mechanism covers all three. But each one gets a mechanism better suited to it than a supervision tree was.

the same system, asked two different questions
 Actor lensDurable-queue lens
Primary questionWho owns this state and who restarts it?What happened, and can I replay it?
IdentityA pid, ephemeralA key, permanent
State livesIn RAM, inside the processIn a store, outside anything
Unit of failureThe processThe message
RecoveryRestart and rebuildRetry; the state never left
OrderingGuaranteed between two processesBest effort; breaks under retry
DeliveryAt-most-once, never duplicatedAt-least-once; be idempotent
Back-pressureNone (mailboxes grow until the node dies)Queue depth, concurrency caps, DLQ
Audit trailWhatever you loggedThe log is the system
Read the last three rows together

This is not a scoreboard where one column wins. BEAM gives you ordering and exactly-once local delivery, and those are real guarantees you will miss. The queue gives you back-pressure and durable history, and OTP practitioners have hand-rolled both more than once. The trade is legible, which is the most you can ask of an architectural choice.

1 On iii

Three declarations, no source file

Building a GenServer this way is not the surprise. The callback module is the surprise: the thing you would assume has to be code turns out to be data.

Start with the mailbox: a named FIFO queue whose message_group_field is the actor id. Messages that share a group value run in order, and different groups run concurrently. That one field is the difference between a mailbox and a work pool.

the mailboxports as config
iii trigger queue::define queue=otp-genserver config='{
  "type": "fifo",
  "message_group_field": "actor_id",
  "concurrency": 10,
  "max_retries": 3,
  "backoff_ms": 1000
}'

{ "changed": true, "queue": "otp-genserver" }

Now the callback. state::update takes an ordered list of atomic operations and applies them to a (scope, key). It returns both the old value and the new one. That pair is handle_call/3: a state transition and a reply, written as data rather than as a function body.

GenServer.cast/2durable · serialized · retried
iii trigger engine::queue::enqueue \
  queue=otp-genserver function_id=state::update \
  messageReceiptId=msg-001 \
  data='{ "actor_id": "c1",
          "scope": "otp:counter", "key": "c1",
          "ops": [{"type":"increment","path":"n","by":100}] }'

$ iii trigger state::get scope=otp:counter key=c1
{ "n": 106 }

That one command is a durable, per-actor-serialized, retry-backed, idempotency-keyed message send into a persistent state machine. Still, no source file appears anywhere in it. The supervisor is a worker-compose.yaml dependency graph, monitors are trigger bindings, and restart intensity is max_retries plus a dead-letter queue.

Three functions that are in no README

state::compare-and-set exists and works, returning {"swapped": false, "current": …} on conflict. The correction matters, because the workflow worker's own documentation states there is no CAS. It tells you to file an engine feature request instead. That guidance is out of date as of state 0.22.2. state::barrier (a fan-in condition for trigger bindings) and state::claim-namespace (singleton election over a function prefix) are likewise undocumented. Enumerate engine::functions::list before believing any prose about what is missing, including this page.

2 The translation

The whole mapping, with the honest column

Most rows are unremarkable. Read the last six.

OTP to iii, and what the translation costs
OTPiiiStatus
GenServer process(scope, key) in stateconfig
Process stateThe value at that keyconfig
handle_caststate::update op-listconfig
handle_call reply{old_value, new_value}config
MailboxFIFO queue, message_group_fieldconfig
GenServer.castengine::queue::enqueueconfig
call timeouttimeout_ms, default 30 minconfig
Optimistic concurrencystate::compare-and-setconfig
Supervisor treeworker-compose.yaml depends_onconfig
init/1{:ok, _}Readiness = registration on the engineconfig
rest_for_oneCascading stop, reverse dependency orderconfig
max_restartsmax_retries + backoff → DLQconfig
send_intervalcron bindingconfig
monitorstate trigger on the scopeconfig
pg groupspubsub topicsconfig
ETSstate scopesconfig
which_childrenstate::list_keysconfig
Barrier / joinstate::barrier as a conditionconfig
Hot code upgradeCallbacks are config; config hot-reloadsfree
Restarting a crashed actordissolves
Restart intensity windowsdissolves
Process registry (name → pid)dissolves
Links / EXIT propagationdissolves
Mailbox ordering under retrygap
Delayed / one-shot sendgap

Note the row that says free. OTP is famous for making hot code upgrade possible and notorious for making it painful: code_change/3, appups, relups. Here the callback is a list of operations in a config entry, and config hot-reloads. You get the capability by accident, because you gave up expressing the callback as code.

3 From scratch

The supervisor works perfectly and the work never finishes

Writing OTP by hand makes the "dissolves" column legible. It also surfaces the one property the translation genuinely costs you.

The from-scratch half is three classes: a Mailbox (strict FIFO), a GenServer (state plus a serialized loop), and a Supervisor. That supervisor carries one_for_one, rest_for_one, and a real restart-intensity window. Restart intensity is about fifteen lines, and it is the thing everyone forgets. Without that window, a crash loop looks the same as a busy system.

scratch/otp.pywhy "restart" dissolves on iii
def restart(self) -> None:
    self.state = self._init()   # state is gone; this is the cost
    self.mailbox = Mailbox()    # so is anything still queued
    self.starts += 1

Once you have written that line, the mapping table stops being a claim and starts being obvious. On iii there is no such line to write, because the state was never in the process. So nothing needs rebuilding.

The demo runs the shared workflow with one injected provider failure. The supervisor does its job:

one transient model failureeverything healthy, nothing delivered
supervisor log:
  score crashed: provider error
  score restarted (start #2)
  accept restarted (start #2)          ← rest_for_one cascade

final state per child:
  extract   starts=1  {"claims": [...]}
  propose   starts=1  {"hypothesis": "..."}
  score     starts=2  {}
  accept    starts=2  {"accepted": null}          ← the work never completed

Every process is healthy, and the cascade was correct. Still, the run produced nothing, because the restart cleared the mailbox for score. That mailbox held the message that caused the crash, so the message went with it. Supervision restores the process, never the message.

This is not a bug in the implementation. It is OTP.

In BEAM the answer is that the sender retries. That sender is usually another supervised process a microsecond away, so nobody notices the cost. With an agentic stage, the sender is a model call you already paid for. Then losing the in-flight message costs a few cents and several seconds to replay. So move the unit of failure off the process and onto the message. That move is the strongest practical argument for the durable-queue translation.

4 What dissolves

Four features that were answers to one question

OTP's machinery cascades from a single fact. Remove the fact and the cascade has nothing to hold onto.

The fact is this: state lives in volatile memory inside something that can die. Supervision exists to restart that thing, and restart intensity exists to stop supervision from looping forever. Links exist to stop collaborators from acting on state that died with its owner. The registry exists for one more reason: pids are ephemeral and unroutable.

the cascade, and where it stopsdissolves
flowchart LR
  ram["state in volatile RAM"] --> die["process can die with state"]
  ram --> pid["pid is ephemeral"]
  die --> sup["supervision to restart it"]
  sup --> intensity["restart intensity to bound the loop"]
  die --> links["links so peers don't read stale state"]
  pid --> registry["registry for stable names"]
  durable["state in a durable store"] --> subject["nothing above has a subject"]
  class sup,intensity,links,registry pass
  class durable,subject dim
  • Restart. No process owns the state, so there is nothing to restart: the state is already there. The unit of failure moved to the message, and retries already handle message failure.
  • Restart intensity. Three retries with exponential backoff, then dead-letter. The dead-letter queue does the same job better: it keeps the poison message and lets you inspect it. Redrive it with iii::queue::redrive once you fix the cause. OTP escalates instead and loses the message.
  • Registry. A registry maps a stable name onto an unstable pid. Here the address is the name. (scope, key) never goes stale.
  • Links and EXIT. Links protect collaborators from acting on state that died. But no stale memory exists here to protect against. What survives is cascade for genuine data dependencies, which the compose daemon already does.
The reframe worth keeping

It is tempting to score this as "iii is missing supervision." That reading is backwards. Supervision is a solution, and solutions only look like requirements until you change the problem. The right question for any substrate is not whether it has a supervisor. Ask what it does with state, and what follows from that answer.

5 What doesn't

Two things you will actually miss

One is inherent to the architecture. The other is a missing feature wearing an actor-model costume.

Ordering under retry. A BEAM mailbox keeps order: two messages sent from A to B arrive in send order, always. The queue worker documents its own divergence: a failed FIFO message goes back on the queue via nack. That nack keeps the poller running, so a message enqueued after the failure can arrive before the retried one.

the reordering windowgenuine gap
sequenceDiagram
  participant P as producer
  participant Q as queue
  participant W as worker
  P->>Q: enqueue A
  Q->>W: deliver A
  W--xW: A fails
  W->>Q: nack, backoff 1000ms
  P->>Q: enqueue B
  Q->>W: deliver B
  Note over W: B lands first
  Q-->>Q: retry A
  Q->>W: deliver A
  Note over P,W: BEAM: A then B, always.
  Note over P,W: queue: A then B, unless A stumbles.

No configuration closes that gap. Handlers have to be order-independent, or carry sequence numbers and park what arrives early. If your domain has genuine causal chains, that constraint shapes your design. It is also the strongest argument for staying on BEAM.

Delayed delivery. iii has no Process.send_after. The queue accepts delaySeconds and visibilityTimeout "for parity" and ignores both. But send_after does four jobs, and three of them already have cover. Call timeouts go to timeout_ms, periodic work goes to cron, and the queue handles retry backoff natively. That leaves business-time scheduling: a scheduling feature, not a hole in the actor model.

One landmine that is neither

Grouped FIFO only applies to named queues created through queue::define. A plain durable:subscriber binding stays globally serial: one actor at a time across the entire system. Same code, same config file, silently 1x instead of 10x. Assert your engine version at boot, the way the harness worker does. That worker refuses to start unless it can ensure its queue.

6 Takeaway

Pseudo-OTP is the point, not the consolation prize

You can build most of OTP out of queue definitions and state operations. You should probably build less of it than you can.

The translation is real but partial, and the partiality is informative. A GenServer falls out of three declarations, and a supervision tree falls out of a depends_on graph. Then monitors, timers, process groups, ETS and barriers are all trigger bindings or state scopes. But restart, restart intensity, the registry and links have no referent at all. That leaves ordering under retry and delayed delivery, the two places you give something up.

The mailbox is what you lose. Going in, the interesting gap looked like supervision, and supervision dissolves. The property worth mourning is strict FIFO between two endpoints. The durable-queue translation cannot give it back, because its own retry mechanism breaks the order.

Then ask whether actors were the right lens at all. Supervision assumes failure is crash-stop and detectable. But a confident wrong answer exits zero and returns a well-formed payload. Nothing in the from-scratch implementation would notice it, and nothing in the iii translation would either. OTP offers superb machinery for the failure mode agents mostly do not have.

That last point is why this is a series rather than an essay. The same incident-analysis workflow runs over a blackboard, where control is opportunistic and partial results are a legitimate outcome. It also runs over an event log, where history is the system and replay is the debugging surface. Both answer pressures that agent systems really do have. Read them next.