If you have written Elixir, you carry a mental model of distributed systems shaped like a supervision tree: processes that hold state, die, and get restarted. Durable-queue systems model the same territory differently — state outlives the thing computing it, and failure is a property of a message rather than a process. Most of OTP survives that translation as pure configuration. The parts that don't survive are more interesting than the parts that do, because they were answers to a question the other architecture never asks.
github.com/tacoda/iii-otp10 passed, offline and deterministic
Actors and durable queues are two answers to the same question. They diverge on one decision, and everything else follows from it.
The actor model puts state inside a process. The process is the unit of identity, the unit of concurrency, and the unit of failure, all at once. That bundling is what makes OTP elegant: because the three coincide, one mechanism — supervision — handles all three.
A durable-queue system unbundles them. Identity is a key. Concurrency is a queue lane. Failure is a message delivery. Because they no longer coincide, no single mechanism covers all three — but each one gets a mechanism better suited to it than a supervision tree was.
| Actor lens | Durable-queue lens | |
|---|---|---|
| Primary question | Who owns this state and who restarts it? | What happened, and can I replay it? |
| Identity | A pid, ephemeral | A key, permanent |
| State lives | In RAM, inside the process | In a store, outside anything |
| Unit of failure | The process | The message |
| Recovery | Restart and rebuild | Retry; the state never left |
| Ordering | Guaranteed between two processes | Best effort; breaks under retry |
| Delivery | At-most-once, never duplicated | At-least-once; be idempotent |
| Back-pressure | None — mailboxes grow until the node dies | Queue depth, concurrency caps, DLQ |
| Audit trail | Whatever you logged | The log is the system |
This is not a scoreboard where one column wins. BEAM gives you ordering and exactly-once local delivery, and those are real guarantees you will miss. The queue gives you back-pressure and durable history, and OTP practitioners have hand-rolled both more than once. The trade is legible, which is the most you can ask of an architectural choice.
The interesting part is not that this is possible. It is that the callback module — the thing you would assume has to be code — turns out to be data.
Start with the mailbox. A named FIFO queue whose message_group_field is the
actor id: messages sharing a group value run in order, different groups run concurrently.
That one field is the difference between a mailbox and a work pool.
iii trigger queue::define queue=otp-genserver config='{
"type": "fifo",
"message_group_field": "actor_id",
"concurrency": 10,
"max_retries": 3,
"backoff_ms": 1000
}'
{ "changed": true, "queue": "otp-genserver" }
Now the callback. state::update takes an ordered list of atomic operations,
applies them to a (scope, key), and returns both the old and new value. That
is handle_call/3: a state transition and a reply, expressed as data rather
than a function body.
iii trigger engine::queue::enqueue \
queue=otp-genserver function_id=state::update \
messageReceiptId=msg-001 \
data='{ "actor_id": "c1",
"scope": "otp:counter", "key": "c1",
"ops": [{"type":"increment","path":"n","by":100}] }'
$ iii trigger state::get scope=otp:counter key=c1
{ "n": 106 }
That is a durable, per-actor-serialized, retry-backed, idempotency-keyed message send into
a persistent state machine, and there is no source file anywhere in it. The supervisor is a
worker-compose.yaml dependency graph. Monitors are trigger bindings. Restart
intensity is max_retries and a dead-letter queue.
state::compare-and-set exists and works, returning
{"swapped": false, "current": …} on conflict. This matters because the
workflow worker's own documentation states there is no CAS and instructs you
to file an engine feature request — that guidance is out of date as of state 0.22.2.
state::barrier (a fan-in condition for trigger bindings) and
state::claim-namespace (singleton election over a function prefix) are
likewise undocumented. Enumerate engine::functions::list before believing any
prose about what is missing, including this page.
Most rows are unremarkable. Read the last six.
| OTP | iii | Status |
|---|---|---|
| GenServer process | (scope, key) in state | config |
| Process state | The value at that key | config |
handle_cast | state::update op-list | config |
handle_call reply | {old_value, new_value} | config |
| Mailbox | FIFO queue, message_group_field | config |
GenServer.cast | engine::queue::enqueue | config |
call timeout | timeout_ms, default 30 min | config |
| Optimistic concurrency | state::compare-and-set | config |
| Supervisor tree | worker-compose.yaml depends_on | config |
init/1 → {:ok, _} | Readiness = registration on the engine | config |
rest_for_one | Cascading stop, reverse dependency order | config |
max_restarts | max_retries + backoff → DLQ | config |
send_interval | cron binding | config |
monitor | state trigger on the scope | config |
pg groups | pubsub topics | config |
| ETS | state scopes | config |
which_children | state::list_keys | config |
| Barrier / join | state::barrier as a condition | config |
| Hot code upgrade | Callbacks are config; config hot-reloads | free |
| Restarting a crashed actor | — | dissolves |
| Restart intensity windows | — | dissolves |
| Process registry (name → pid) | — | dissolves |
| Links / EXIT propagation | — | dissolves |
| Mailbox ordering under retry | — | gap |
| Delayed / one-shot send | — | gap |
Note the row that says free. Hot code upgrade is the thing OTP is famous for
making possible and notorious for making painful — code_change/3, appups,
relups. Here the callback is a list of operations in a config entry, and config
hot-reloads. You get the capability by accident, because you gave up expressing the
callback as code.
Writing OTP by hand is what makes the "dissolves" column legible — and it surfaces the one property the translation genuinely costs you.
The from-scratch half is a Mailbox (strict FIFO), a GenServer
(state plus a serialized loop), and a Supervisor with
one_for_one / rest_for_one and a real restart-intensity window.
Restart intensity — the thing everyone forgets, and without which a crash loop is
indistinguishable from a busy system — is about fifteen lines.
def restart(self) -> None:
self.state = self._init() # state is gone; this is the cost
self.mailbox = Mailbox() # so is anything still queued
self.starts += 1
Once you have written that line, the mapping table stops being a claim and starts being obvious. On iii there is no such line to write, because the state was never in the process. There is nothing to rebuild.
The demo runs the shared workflow with one injected provider failure. The supervisor does its job flawlessly:
supervisor log:
score crashed: provider error
score restarted (start #2)
accept restarted (start #2) ← rest_for_one cascade
final state per child:
extract starts=1 {"claims": [...]}
propose starts=1 {"hypothesis": "..."}
score starts=2 {}
accept starts=2 {"accepted": null} ← the work never completed
Every process is healthy. The cascade was correct. And the run produced nothing, because
the restart cleared score's mailbox and the message that caused the crash is
gone. Supervision restores the process, never the message.
In BEAM the answer is that the sender retries — and the sender is usually another supervised process a microsecond away, so nobody notices the cost. With an agentic stage, the sender is a model call you already paid for. Losing the in-flight message on restart is a very different proposition when replaying it costs a few cents and several seconds. That is the strongest practical argument for moving the unit of failure from the process to the message — which is exactly what the durable-queue translation does.
OTP's machinery cascades from a single fact. Remove the fact and the cascade has nothing to hold onto.
The fact is this: state lives in volatile memory inside something that can die. Supervision exists to restart that thing. Restart intensity exists to stop supervision from looping forever. Links exist to stop collaborators from acting on state that died with its owner. The registry exists because pids are ephemeral and unroutable.
state in volatile RAM
└─► process can die with state
└─► supervision to restart it
└─► restart intensity to bound the loop
└─► links so peers don't read stale state
└─► pid is ephemeral
└─► registry for stable names
state in a durable store
└─► nothing above has a subject
iii::queue::redrive once you fix the cause. OTP
escalates and the message is gone.
(scope, key) never goes stale.
It is tempting to score this as "iii is missing supervision." That reading is backwards. Supervision is a solution, and solutions only look like requirements until you change the problem. The right question for any substrate is not does it have a supervisor but what does it do with state, and what follows from that.
One is inherent to the architecture. The other is a missing feature wearing an actor-model costume.
Ordering under retry. A BEAM mailbox is a mailbox precisely because it is
ordered: two messages sent from A to B arrive in send order, always. The queue worker
documents its own divergence — a failed FIFO message is re-queued via nack
rather than blocking the poller, so a message enqueued after the failure can be
delivered before the retried one catches up.
enqueue A ──► deliver A ──► A fails ──► nack, backoff 1000ms
enqueue B ─────────────────► deliver B ◄── B lands first
retry A ──► deliver A
BEAM: A then B, always.
queue: A then B, unless A stumbles.
No configuration closes this. Handlers have to be order-independent, or carry sequence numbers and park what arrives early. If your domain has genuine causal chains, this is the constraint that shapes your design, and it is the strongest argument for staying on BEAM.
Delayed delivery. There is no Process.send_after. The queue
accepts delaySeconds and visibilityTimeout "for parity" and
ignores both. But look at what send_after is actually used for: call timeouts
(timeout_ms covers it), periodic work (cron), retry backoff (the
queue does it natively), and business-time scheduling. Only the fourth is uncovered — a
scheduling feature, not a hole in the actor model.
Grouped FIFO only applies to named queues created through
queue::define. A plain durable:subscriber binding stays globally
serial — one actor at a time across the entire system. Same code, same config file,
silently 1x instead of 10x. Assert your engine version at boot the way the harness worker
does; it refuses to start if its queue cannot be ensured.
You can build most of OTP out of queue definitions and state operations. You should probably build less of it than you can.
The translation is real but partial, and the partiality is informative.
A GenServer falls out of three declarations. A supervision tree falls out of a
depends_on graph. Monitors, timers, process groups, ETS and barriers are all
trigger bindings or state scopes. Restart, restart intensity, the registry and links have
no referent at all. Ordering under retry and delayed delivery are the two places you give
something up.
The mailbox is what you actually lose. Going in, the interesting gap looked like supervision — and supervision dissolves. The property worth mourning is strict FIFO between two endpoints, and the durable-queue translation cannot give it back, because its retry mechanism is what breaks the order.
Then ask whether actors were the right lens at all. Supervision assumes failure is crash-stop and detectable. A confident wrong answer exits zero and returns a well-formed payload — nothing in the from-scratch implementation would notice, and nothing in the iii translation would either. OTP offers superb machinery for the failure mode agents mostly do not have.
That last point is why this is a series rather than an essay. The same incident-analysis workflow runs over a blackboard, where control is opportunistic and partial results are a legitimate outcome, and over an event log, where history is the system and replay is the debugging surface. Both answer pressures that agent systems actually have. Read them next.