Harness Engineering / One workflow, three architectures / 1 of 3 · Actors & supervision

Half of OTP does not port. It dissolves.

If you have written Elixir, you carry a mental model of distributed systems shaped like a supervision tree: processes that hold state, die, and get restarted. Durable-queue systems model the same territory differently — state outlives the thing computing it, and failure is a property of a message rather than a process. Most of OTP survives that translation as pure configuration. The parts that don't survive are more interesting than the parts that do, because they were answers to a question the other architecture never asks.

companion repo · github.com/tacoda/iii-otp
built twice · once as iii configuration, once from scratch in stdlib Python
the workload · the same incident analysis all three repos run — extract claims, propose an explanation, score it, accept
verification · every iii command shown was executed against a live engine (0.22.0) before it was written down
tests · 10 passed, offline and deterministic
Ports as config
The OTP concept has a direct iii equivalent, reachable with a YAML entry or a trigger binding.
Dissolves
The concept has no referent here. Not a missing feature — the problem it solved does not arise.
Genuine gap
OTP gives you something real that this architecture does not, and no configuration closes it.
0 Two lenses

Who owns this state, versus what happened to it

Actors and durable queues are two answers to the same question. They diverge on one decision, and everything else follows from it.

The actor model puts state inside a process. The process is the unit of identity, the unit of concurrency, and the unit of failure, all at once. That bundling is what makes OTP elegant: because the three coincide, one mechanism — supervision — handles all three.

A durable-queue system unbundles them. Identity is a key. Concurrency is a queue lane. Failure is a message delivery. Because they no longer coincide, no single mechanism covers all three — but each one gets a mechanism better suited to it than a supervision tree was.

the same system, asked two different questions
 Actor lensDurable-queue lens
Primary questionWho owns this state and who restarts it?What happened, and can I replay it?
IdentityA pid, ephemeralA key, permanent
State livesIn RAM, inside the processIn a store, outside anything
Unit of failureThe processThe message
RecoveryRestart and rebuildRetry; the state never left
OrderingGuaranteed between two processesBest effort; breaks under retry
DeliveryAt-most-once, never duplicatedAt-least-once; be idempotent
Back-pressureNone — mailboxes grow until the node diesQueue depth, concurrency caps, DLQ
Audit trailWhatever you loggedThe log is the system
Read the last three rows together

This is not a scoreboard where one column wins. BEAM gives you ordering and exactly-once local delivery, and those are real guarantees you will miss. The queue gives you back-pressure and durable history, and OTP practitioners have hand-rolled both more than once. The trade is legible, which is the most you can ask of an architectural choice.

1 On iii

Three declarations, no source file

The interesting part is not that this is possible. It is that the callback module — the thing you would assume has to be code — turns out to be data.

Start with the mailbox. A named FIFO queue whose message_group_field is the actor id: messages sharing a group value run in order, different groups run concurrently. That one field is the difference between a mailbox and a work pool.

the mailboxports as config
iii trigger queue::define queue=otp-genserver config='{
  "type": "fifo",
  "message_group_field": "actor_id",
  "concurrency": 10,
  "max_retries": 3,
  "backoff_ms": 1000
}'

{ "changed": true, "queue": "otp-genserver" }

Now the callback. state::update takes an ordered list of atomic operations, applies them to a (scope, key), and returns both the old and new value. That is handle_call/3: a state transition and a reply, expressed as data rather than a function body.

GenServer.cast/2durable · serialized · retried
iii trigger engine::queue::enqueue \
  queue=otp-genserver function_id=state::update \
  messageReceiptId=msg-001 \
  data='{ "actor_id": "c1",
          "scope": "otp:counter", "key": "c1",
          "ops": [{"type":"increment","path":"n","by":100}] }'

$ iii trigger state::get scope=otp:counter key=c1
{ "n": 106 }

That is a durable, per-actor-serialized, retry-backed, idempotency-keyed message send into a persistent state machine, and there is no source file anywhere in it. The supervisor is a worker-compose.yaml dependency graph. Monitors are trigger bindings. Restart intensity is max_retries and a dead-letter queue.

Three functions that are in no README

state::compare-and-set exists and works, returning {"swapped": false, "current": …} on conflict. This matters because the workflow worker's own documentation states there is no CAS and instructs you to file an engine feature request — that guidance is out of date as of state 0.22.2. state::barrier (a fan-in condition for trigger bindings) and state::claim-namespace (singleton election over a function prefix) are likewise undocumented. Enumerate engine::functions::list before believing any prose about what is missing, including this page.

2 The translation

The whole mapping, with the honest column

Most rows are unremarkable. Read the last six.

OTP to iii, and what the translation costs
OTPiiiStatus
GenServer process(scope, key) in stateconfig
Process stateThe value at that keyconfig
handle_caststate::update op-listconfig
handle_call reply{old_value, new_value}config
MailboxFIFO queue, message_group_fieldconfig
GenServer.castengine::queue::enqueueconfig
call timeouttimeout_ms, default 30 minconfig
Optimistic concurrencystate::compare-and-setconfig
Supervisor treeworker-compose.yaml depends_onconfig
init/1{:ok, _}Readiness = registration on the engineconfig
rest_for_oneCascading stop, reverse dependency orderconfig
max_restartsmax_retries + backoff → DLQconfig
send_intervalcron bindingconfig
monitorstate trigger on the scopeconfig
pg groupspubsub topicsconfig
ETSstate scopesconfig
which_childrenstate::list_keysconfig
Barrier / joinstate::barrier as a conditionconfig
Hot code upgradeCallbacks are config; config hot-reloadsfree
Restarting a crashed actordissolves
Restart intensity windowsdissolves
Process registry (name → pid)dissolves
Links / EXIT propagationdissolves
Mailbox ordering under retrygap
Delayed / one-shot sendgap

Note the row that says free. Hot code upgrade is the thing OTP is famous for making possible and notorious for making painful — code_change/3, appups, relups. Here the callback is a list of operations in a config entry, and config hot-reloads. You get the capability by accident, because you gave up expressing the callback as code.

3 From scratch

The supervisor works perfectly and the work never finishes

Writing OTP by hand is what makes the "dissolves" column legible — and it surfaces the one property the translation genuinely costs you.

The from-scratch half is a Mailbox (strict FIFO), a GenServer (state plus a serialized loop), and a Supervisor with one_for_one / rest_for_one and a real restart-intensity window. Restart intensity — the thing everyone forgets, and without which a crash loop is indistinguishable from a busy system — is about fifteen lines.

scratch/otp.pywhy "restart" dissolves on iii
def restart(self) -> None:
    self.state = self._init()   # state is gone; this is the cost
    self.mailbox = Mailbox()    # so is anything still queued
    self.starts += 1

Once you have written that line, the mapping table stops being a claim and starts being obvious. On iii there is no such line to write, because the state was never in the process. There is nothing to rebuild.

The demo runs the shared workflow with one injected provider failure. The supervisor does its job flawlessly:

one transient model failureeverything healthy, nothing delivered
supervisor log:
  score crashed: provider error
  score restarted (start #2)
  accept restarted (start #2)          ← rest_for_one cascade

final state per child:
  extract   starts=1  {"claims": [...]}
  propose   starts=1  {"hypothesis": "..."}
  score     starts=2  {}
  accept    starts=2  {"accepted": null}          ← the work never completed

Every process is healthy. The cascade was correct. And the run produced nothing, because the restart cleared score's mailbox and the message that caused the crash is gone. Supervision restores the process, never the message.

This is not a bug in the implementation. It is OTP.

In BEAM the answer is that the sender retries — and the sender is usually another supervised process a microsecond away, so nobody notices the cost. With an agentic stage, the sender is a model call you already paid for. Losing the in-flight message on restart is a very different proposition when replaying it costs a few cents and several seconds. That is the strongest practical argument for moving the unit of failure from the process to the message — which is exactly what the durable-queue translation does.

4 What dissolves

Four features that were answers to one question

OTP's machinery cascades from a single fact. Remove the fact and the cascade has nothing to hold onto.

The fact is this: state lives in volatile memory inside something that can die. Supervision exists to restart that thing. Restart intensity exists to stop supervision from looping forever. Links exist to stop collaborators from acting on state that died with its owner. The registry exists because pids are ephemeral and unroutable.

the cascade, and where it stopsdissolves
state in volatile RAM
  └─► process can die with state
        └─► supervision to restart it
              └─► restart intensity to bound the loop
        └─► links so peers don't read stale state
  └─► pid is ephemeral
        └─► registry for stable names

state in a durable store
  └─► nothing above has a subject
  • Restart. No process owns the state, so there is nothing to restart — the state is already there. The unit of failure moved to the message, and message failure is already handled by retries.
  • Restart intensity. Three retries with exponential backoff, then dead-letter. Same job done better: the poison message is preserved, inspectable, and replayable with iii::queue::redrive once you fix the cause. OTP escalates and the message is gone.
  • Registry. A registry maps a stable name onto an unstable pid. Here the address is the name. (scope, key) never goes stale.
  • Links and EXIT. Links protect collaborators from acting on state that died. There is no stale memory to protect against. What survives is cascade for genuine data dependencies, which the compose daemon already does.
The reframe worth keeping

It is tempting to score this as "iii is missing supervision." That reading is backwards. Supervision is a solution, and solutions only look like requirements until you change the problem. The right question for any substrate is not does it have a supervisor but what does it do with state, and what follows from that.

5 What doesn't

Two things you will actually miss

One is inherent to the architecture. The other is a missing feature wearing an actor-model costume.

Ordering under retry. A BEAM mailbox is a mailbox precisely because it is ordered: two messages sent from A to B arrive in send order, always. The queue worker documents its own divergence — a failed FIFO message is re-queued via nack rather than blocking the poller, so a message enqueued after the failure can be delivered before the retried one catches up.

the reordering windowgenuine gap
enqueue A ──► deliver A ──► A fails ──► nack, backoff 1000ms
enqueue B ─────────────────► deliver B  ◄── B lands first
                             retry A ──► deliver A

BEAM: A then B, always.
queue: A then B, unless A stumbles.

No configuration closes this. Handlers have to be order-independent, or carry sequence numbers and park what arrives early. If your domain has genuine causal chains, this is the constraint that shapes your design, and it is the strongest argument for staying on BEAM.

Delayed delivery. There is no Process.send_after. The queue accepts delaySeconds and visibilityTimeout "for parity" and ignores both. But look at what send_after is actually used for: call timeouts (timeout_ms covers it), periodic work (cron), retry backoff (the queue does it natively), and business-time scheduling. Only the fourth is uncovered — a scheduling feature, not a hole in the actor model.

One landmine that is neither

Grouped FIFO only applies to named queues created through queue::define. A plain durable:subscriber binding stays globally serial — one actor at a time across the entire system. Same code, same config file, silently 1x instead of 10x. Assert your engine version at boot the way the harness worker does; it refuses to start if its queue cannot be ensured.

6 Takeaway

Pseudo-OTP is the point, not the consolation prize

You can build most of OTP out of queue definitions and state operations. You should probably build less of it than you can.

The translation is real but partial, and the partiality is informative. A GenServer falls out of three declarations. A supervision tree falls out of a depends_on graph. Monitors, timers, process groups, ETS and barriers are all trigger bindings or state scopes. Restart, restart intensity, the registry and links have no referent at all. Ordering under retry and delayed delivery are the two places you give something up.

The mailbox is what you actually lose. Going in, the interesting gap looked like supervision — and supervision dissolves. The property worth mourning is strict FIFO between two endpoints, and the durable-queue translation cannot give it back, because its retry mechanism is what breaks the order.

Then ask whether actors were the right lens at all. Supervision assumes failure is crash-stop and detectable. A confident wrong answer exits zero and returns a well-formed payload — nothing in the from-scratch implementation would notice, and nothing in the iii translation would either. OTP offers superb machinery for the failure mode agents mostly do not have.

That last point is why this is a series rather than an essay. The same incident-analysis workflow runs over a blackboard, where control is opportunistic and partial results are a legitimate outcome, and over an event log, where history is the system and replay is the debugging surface. Both answer pressures that agent systems actually have. Read them next.