If you have written Elixir, you carry a supervision tree in your head: processes hold state, die, and get restarted. But a durable-queue system models the same territory differently. State outlives the thing computing it, so failure belongs to a message rather than to a process. Most of OTP survives that translation as pure configuration. The parts that do not survive are more interesting, because they answered a question the other architecture never asks.
github.com/tacoda/iii-otp10 passed, offline and deterministic
Actors and durable queues are two answers to the same question. They diverge on one decision, and everything else follows from it.
The actor model puts state inside a process. The process is the unit of identity, the unit of concurrency, and the unit of failure, all at once. That bundling is what makes OTP elegant: because the three coincide, one mechanism, supervision, handles all three.
A durable-queue system unbundles them: identity is a key, concurrency is a queue lane, and failure is a message delivery. Because they no longer coincide, no single mechanism covers all three. But each one gets a mechanism better suited to it than a supervision tree was.
| Actor lens | Durable-queue lens | |
|---|---|---|
| Primary question | Who owns this state and who restarts it? | What happened, and can I replay it? |
| Identity | A pid, ephemeral | A key, permanent |
| State lives | In RAM, inside the process | In a store, outside anything |
| Unit of failure | The process | The message |
| Recovery | Restart and rebuild | Retry; the state never left |
| Ordering | Guaranteed between two processes | Best effort; breaks under retry |
| Delivery | At-most-once, never duplicated | At-least-once; be idempotent |
| Back-pressure | None (mailboxes grow until the node dies) | Queue depth, concurrency caps, DLQ |
| Audit trail | Whatever you logged | The log is the system |
This is not a scoreboard where one column wins. BEAM gives you ordering and exactly-once local delivery, and those are real guarantees you will miss. The queue gives you back-pressure and durable history, and OTP practitioners have hand-rolled both more than once. The trade is legible, which is the most you can ask of an architectural choice.
Building a GenServer this way is not the surprise. The callback module is the surprise: the thing you would assume has to be code turns out to be data.
Start with the mailbox: a named FIFO queue whose message_group_field is the
actor id. Messages that share a group value run in order, and different groups run
concurrently. That one field is the difference between a mailbox and a work pool.
iii trigger queue::define queue=otp-genserver config='{
"type": "fifo",
"message_group_field": "actor_id",
"concurrency": 10,
"max_retries": 3,
"backoff_ms": 1000
}'
{ "changed": true, "queue": "otp-genserver" }
Now the callback. state::update takes an ordered list of atomic operations and
applies them to a (scope, key). It returns both the old value and the new one.
That pair is handle_call/3: a state transition and a reply, written as data
rather than as a function body.
iii trigger engine::queue::enqueue \
queue=otp-genserver function_id=state::update \
messageReceiptId=msg-001 \
data='{ "actor_id": "c1",
"scope": "otp:counter", "key": "c1",
"ops": [{"type":"increment","path":"n","by":100}] }'
$ iii trigger state::get scope=otp:counter key=c1
{ "n": 106 }
That one command is a durable, per-actor-serialized, retry-backed, idempotency-keyed
message send into a persistent state machine. Still, no source file appears anywhere in it.
The supervisor is a worker-compose.yaml dependency graph, monitors are trigger
bindings, and restart intensity is max_retries plus a dead-letter queue.
state::compare-and-set exists and works, returning
{"swapped": false, "current": …} on conflict. The correction matters, because
the workflow worker's own documentation states there is no CAS. It tells you
to file an engine feature request instead. That guidance is out of date as of state 0.22.2.
state::barrier (a fan-in condition for trigger bindings) and
state::claim-namespace (singleton election over a function prefix) are
likewise undocumented. Enumerate engine::functions::list before believing any
prose about what is missing, including this page.
Most rows are unremarkable. Read the last six.
| OTP | iii | Status |
|---|---|---|
| GenServer process | (scope, key) in state | config |
| Process state | The value at that key | config |
handle_cast | state::update op-list | config |
handle_call reply | {old_value, new_value} | config |
| Mailbox | FIFO queue, message_group_field | config |
GenServer.cast | engine::queue::enqueue | config |
call timeout | timeout_ms, default 30 min | config |
| Optimistic concurrency | state::compare-and-set | config |
| Supervisor tree | worker-compose.yaml depends_on | config |
init/1 → {:ok, _} | Readiness = registration on the engine | config |
rest_for_one | Cascading stop, reverse dependency order | config |
max_restarts | max_retries + backoff → DLQ | config |
send_interval | cron binding | config |
monitor | state trigger on the scope | config |
pg groups | pubsub topics | config |
| ETS | state scopes | config |
which_children | state::list_keys | config |
| Barrier / join | state::barrier as a condition | config |
| Hot code upgrade | Callbacks are config; config hot-reloads | free |
| Restarting a crashed actor | — | dissolves |
| Restart intensity windows | — | dissolves |
| Process registry (name → pid) | — | dissolves |
| Links / EXIT propagation | — | dissolves |
| Mailbox ordering under retry | — | gap |
| Delayed / one-shot send | — | gap |
Note the row that says free. OTP is famous for making hot code upgrade possible and
notorious for making it painful: code_change/3, appups, relups. Here the
callback is a list of operations in a config entry, and config hot-reloads. You get the
capability by accident, because you gave up expressing the callback as code.
Writing OTP by hand makes the "dissolves" column legible. It also surfaces the one property the translation genuinely costs you.
The from-scratch half is three classes: a Mailbox (strict FIFO), a
GenServer (state plus a serialized loop), and a Supervisor. That
supervisor carries one_for_one, rest_for_one, and a real
restart-intensity window. Restart intensity is about fifteen lines, and it is the thing
everyone forgets. Without that window, a crash loop looks the same as a busy system.
def restart(self) -> None:
self.state = self._init() # state is gone; this is the cost
self.mailbox = Mailbox() # so is anything still queued
self.starts += 1
Once you have written that line, the mapping table stops being a claim and starts being obvious. On iii there is no such line to write, because the state was never in the process. So nothing needs rebuilding.
The demo runs the shared workflow with one injected provider failure. The supervisor does its job:
supervisor log:
score crashed: provider error
score restarted (start #2)
accept restarted (start #2) ← rest_for_one cascade
final state per child:
extract starts=1 {"claims": [...]}
propose starts=1 {"hypothesis": "..."}
score starts=2 {}
accept starts=2 {"accepted": null} ← the work never completed
Every process is healthy, and the cascade was correct. Still, the run produced nothing,
because the restart cleared the mailbox for score. That mailbox held the
message that caused the crash, so the message went with it.
Supervision restores the process, never the message.
In BEAM the answer is that the sender retries. That sender is usually another supervised process a microsecond away, so nobody notices the cost. With an agentic stage, the sender is a model call you already paid for. Then losing the in-flight message costs a few cents and several seconds to replay. So move the unit of failure off the process and onto the message. That move is the strongest practical argument for the durable-queue translation.
OTP's machinery cascades from a single fact. Remove the fact and the cascade has nothing to hold onto.
The fact is this: state lives in volatile memory inside something that can die. Supervision exists to restart that thing, and restart intensity exists to stop supervision from looping forever. Links exist to stop collaborators from acting on state that died with its owner. The registry exists for one more reason: pids are ephemeral and unroutable.
flowchart LR ram["state in volatile RAM"] --> die["process can die with state"] ram --> pid["pid is ephemeral"] die --> sup["supervision to restart it"] sup --> intensity["restart intensity to bound the loop"] die --> links["links so peers don't read stale state"] pid --> registry["registry for stable names"] durable["state in a durable store"] --> subject["nothing above has a subject"] class sup,intensity,links,registry pass class durable,subject dim
iii::queue::redrive once you fix the cause.
OTP escalates instead and loses the message.
(scope, key) never goes stale.
It is tempting to score this as "iii is missing supervision." That reading is backwards. Supervision is a solution, and solutions only look like requirements until you change the problem. The right question for any substrate is not whether it has a supervisor. Ask what it does with state, and what follows from that answer.
One is inherent to the architecture. The other is a missing feature wearing an actor-model costume.
Ordering under retry. A BEAM mailbox keeps order: two messages sent from A
to B arrive in send order, always. The queue worker documents its own divergence: a failed
FIFO message goes back on the queue via nack. That nack keeps the poller
running, so a message enqueued after the failure can arrive before the retried one.
sequenceDiagram participant P as producer participant Q as queue participant W as worker P->>Q: enqueue A Q->>W: deliver A W--xW: A fails W->>Q: nack, backoff 1000ms P->>Q: enqueue B Q->>W: deliver B Note over W: B lands first Q-->>Q: retry A Q->>W: deliver A Note over P,W: BEAM: A then B, always. Note over P,W: queue: A then B, unless A stumbles.
No configuration closes that gap. Handlers have to be order-independent, or carry sequence numbers and park what arrives early. If your domain has genuine causal chains, that constraint shapes your design. It is also the strongest argument for staying on BEAM.
Delayed delivery. iii has no Process.send_after. The queue
accepts delaySeconds and visibilityTimeout "for parity" and
ignores both. But send_after does four jobs, and three of them already have
cover. Call timeouts go to timeout_ms, periodic work goes to
cron, and the queue handles retry backoff natively. That leaves business-time
scheduling: a scheduling feature, not a hole in the actor model.
Grouped FIFO only applies to named queues created through
queue::define. A plain durable:subscriber binding stays globally
serial: one actor at a time across the entire system. Same code, same config file,
silently 1x instead of 10x. Assert your engine version at boot, the way the harness worker
does. That worker refuses to start unless it can ensure its queue.
You can build most of OTP out of queue definitions and state operations. You should probably build less of it than you can.
The translation is real but partial, and the partiality is informative.
A GenServer falls out of three declarations, and a supervision tree falls out of a
depends_on graph. Then monitors, timers, process groups, ETS and barriers are
all trigger bindings or state scopes. But restart, restart intensity, the registry and
links have no referent at all. That leaves ordering under retry and delayed delivery, the
two places you give something up.
The mailbox is what you lose. Going in, the interesting gap looked like supervision, and supervision dissolves. The property worth mourning is strict FIFO between two endpoints. The durable-queue translation cannot give it back, because its own retry mechanism breaks the order.
Then ask whether actors were the right lens at all. Supervision assumes failure is crash-stop and detectable. But a confident wrong answer exits zero and returns a well-formed payload. Nothing in the from-scratch implementation would notice it, and nothing in the iii translation would either. OTP offers superb machinery for the failure mode agents mostly do not have.
That last point is why this is a series rather than an essay. The same incident-analysis workflow runs over a blackboard, where control is opportunistic and partial results are a legitimate outcome. It also runs over an event log, where history is the system and replay is the debugging surface. Both answer pressures that agent systems really do have. Read them next.