Harness Engineering / One workflow, three architectures / 3 of 3 · Event sourcing

Your agent transcript is already an event log

Event sourcing derives state instead of storing it. The authoritative record is an append-only log of what happened, and every read model is a fold over that log. An agent system produces the log whether or not it means to. So the choice is not whether to have one, but what you get out of it. Commit to the log and you get replay, projections and an audit trail. Ignore it and you keep a chat history you grep when something breaks.

companion repo · github.com/tacoda/iii-event-sourcing
built twice · once as iii configuration, once from scratch in stdlib Python
the workload · the same incident analysis all three repos run : extract claims, propose an explanation, score it, accept
verification · every iii command shown was executed against a live engine (0.22.0) before it was written down
tests · 8 passed, offline and deterministic
The log
Append-only, ordered by an explicit parent chain. The only authoritative record.
Projection
A read model: a pure fold over the log. Disposable, rebuildable, never authoritative.
Notification
At-least-once and unordered. A doorbell, not data, and the source of the hazard below.
0 The shape

State as a fold, not a cell

"What is the state" and "how did it get that way" stop being two questions with two levels of support.

A write never updates a value in place. It appends HypothesisScored{confidence: 0.86} to a stream, and the current confidence is whatever the fold over that stream returns. Every read model derives from the log, which makes it disposable and rebuildable. If a projection has a bug, fix the fold and replay instead of migrating corrupted rows.

what changes when the log is authoritative
 State-in-placeEvent-sourced
Current valueRead a rowFold a stream (or read a projection)
HistoryWhatever you remembered to logThe system itself
Fixing a bad read modelData migrationFix the fold, replay
"Why is it like this?"ForensicsA query
StorageBounded by state sizeGrows forever
Cost of a schema changeOne migrationOld events are immutable (the fold carries every version, forever)
Why this suits agents in particular

With deterministic code, replay is a debugging convenience, because you can always run the code again. With a model in the loop you cannot: a second run gives you a different run. Because of that, the log is the only record of what the agent saw and decided. That record moves replay from a convenience to the debugging surface you depend on. So the row above about unbounded growth is a price worth paying here, and often not elsewhere.

1 On iii

An event store wearing a chat-shaped label

The docs call session-manager a conversation store. Without modification, it is also a correct event log.

A stream is a session. An event is a custom entry whose custom_type names the event type. That field is a real discriminator slot, not a type you smuggle into the payload. Ordering is an explicit parent_id chain, and the engine returns it on every append. So the log is a linked list rather than a timestamp-sorted bag.

appending a domain eventcustom_type is the event type
iii trigger session::append session_id=$SID \
  message='{"role":"custom",
            "custom_type":"ClaimRecorded",
            "content":[{"type":"text","text":"{\"claim\":\"one consumer attached\"}"}],
            "timestamp":1717800000000}'

{ "entry_id": "e_45d958d6…", "parent_id": "e_5e3d1086…", "timestamp": … }
event sourcing, mapped and verified
ConceptiiiStatus
Streama session; aggregate identity in metadataconfig
Event typecustom_type on the entryconfig
Orderingexplicit parent_id chainconfig
Read the logsession::messages, already in chain orderconfig
Projectionsession::message-added binding → state scopeconfig
Per-aggregate filtermetadata subset match on the bindingconfig
Fork / what-if replaysession::forkconfig
Aggregate version guardstate::compare-and-set , not in the logconfig
Optimistic concurrency on appendabsent
2 The hazard

Two writers, two success responses, one surviving event

The dangerous kind of missing feature is the one with a convincing near-miss rather than the one that errors.

session::append has no conditional write. Pass expected_revision and the engine accepts the parameter, ignores it, and appends anyway. No error tells you that your concurrency control does nothing:

the near-missaccepted. ignored.
$ iii trigger session::append session_id=$SID expected_revision=3 message=…
{"entry_id": "e_399c8b87…", "parent_id": "e_20ec1d48…"}   ← appended regardless

parent_id, meanwhile, is real, and the engine does validate it: an unknown parent returns session/parent_not_found. So it looks exactly like the conditional-append primitive. It isn't. Two writers appending at the same parent both succeed:

the corruptionboth writers got a 200
writer A -> e_fd3ce7e1ffe6   (parent e_9da6…)   ok
writer B -> e_cad909ed44b3   (parent e_9da6…)   ok

$ iii trigger session::messages session_id=$SID
  … e_9da67bbfd6c1  {"confidence":0.86}
  … e_cad909ed44b3  {"claim":"writer B"}
                                       ← A is not here

$ iii trigger session::get-message entry_id=e_fd3ce7e1ffe6
  {"id":"e_fd3ce7e1ffe6", "message":{…"writer A"…}, "revision": 0}

A's event is durable, and you can fetch it by id, but it sits on a sibling branch. session::messages does not walk that branch, so no projection reading the ordinary way will ever see A's event. Still, A's writer received a success response, and nothing anywhere reports a problem.

The operational consequence

The version guard has to live in state, and the order is not negotiable: compare-and-set first, append second. Append first and you have written an event you cannot un-append. The log is append-only, which is the point, so a concurrency mistake is permanent. There is also no branch enumeration: session::leaves does not exist, and the engine ignores include_branches. So you can recover an orphaned entry only if you already know its entry_id.

3 From scratch

Four lines that upstream doesn't have

Writing the missing feature by hand settles how missing it is.

scratch/eventstore.pyreal optimistic concurrency
def append(self, stream, type, payload, *, expected_version=None) -> Event:
    log = self._streams.setdefault(stream, [])
    if expected_version is not None and expected_version != len(log):
        raise Conflict(f"{stream}: expected {expected_version}, found {len(log)}")
    event = Event(seq=len(log) + 1, type=type, payload=payload, stream=stream)
    log.append(event)
    self._notify(event)
    return event

Four lines. Which is what makes the absence notable: this is not a hard feature to write. The convincing near-miss is what makes it dangerous rather than just missing.

The more valuable half of the from-scratch build is the UnreliableBus. That bus is a notification channel that is at-least-once and unordered on purpose, because session-manager documents that contract for itself. It exists so one test can prove the claim below, which is the single most transferable finding in this series:

the headline testsame events, same fold, different answers
_seed(store)                       # 4 events, appended in order
bus.drain(lambda e: (naive_handler(e), doorbell_handler(e)))

truth = Projection("truth", incident_fold).rebuild(store.read(STREAM))

assert doorbell.state == truth    # re-reads the ordered log. correct.
assert naive.state    != truth    # folds in arrival order. corrupted.
assert len(naive.state["claims"]) != len(truth["claims"])

Identical events. Identical fold function. One projection is right and one is silently wrong. The only difference is whether it trusted the delivery order. Neither case raises an exception, and the wrong projection just reports a wrong number.

4 What it taught

Four findings, one of which applies everywhere

The ordering finding is about every event-sourced system with an asynchronous notification channel, not about iii.

  • An ordered log does not give you ordered projections. iii orders its log by the parent chain. The docs call its notifications "at-least-once and unordered". Those two facts live in different documents, and the gap between them is where wrong dashboards come from. Check both guarantees in whatever you are using, because almost nobody states them in the same place.
  • Treat the notification as a doorbell, not as data. Re-read the ordered log on every notification, and you are immune to reordering, duplication and gaps. That immunity costs one read per event. Fold the delivered event directly and you are fast and wrong. Both implementations arrived at the same rule independently.
  • A near-miss is worse than an absence. A missing feature makes you build something. But a feature that accepts your parameter and then ignores it makes you believe you have a guard. The parameter next to it does validate, which is what makes the pair convincing. Test that your concurrency control rejects a conflicting write.
  • Replay is worth more with a model in the loop. Fork the stream just before a model call, then replay it with a different prompt. That loop beats re-running and hoping, because a re-run does not reproduce the run.
5 Takeaway

You already have the log. Decide whether it counts.

The work is committing to the log being authoritative and getting the ordering discipline right, not producing events.

Agent systems are event-sourced whether or not they mean to be. The transcript is an append-only record of what happened. So event sourcing is the decision to treat that transcript as the source of truth rather than as a log file. It also means deriving read models from the log instead of maintaining them alongside it.

On iii, the store is already there and the concurrency control is not. session-manager gives you streams, typed events, chain ordering, projections via trigger bindings, and fork-for-replay. It does not give you conditional append, it accepts expected_revision and ignores it, and concurrent appends branch silently. Put the version in state, compare-and-set before you append, and never the other way round.

Write the ordering test before you write the projection. A projection that folds in arrival order is correct in development and correct under light load. Then it is wrong the first time anything retries, and it never raises. So you find out only by delivering events out of order and asserting that the projection still matches the log.

That ends the series: the same incident-analysis workflow over actors and supervision, a blackboard, and an event log. Three architectures, one task, two implementations each. The pattern across all three is that iii supplies more of each model than expected. Its gaps are consistently the parts that fail without an error, rather than with one.