Understanding Durable Execution

The worker executor runs WASM agents whose progress survives the loss of the process running them. Everything below is verified against golem-worker-executor/src/ and tests/; when code and guide disagree, the code wins and the guide needs a fix. The scoped AGENTS.md files under golem-worker-executor/ state the mandatory rules; this skill explains the mechanisms.

Deeper material lives in reference/: timelines.md (worked oplog timelines), crash-matrix.md (what recovery does for each crash window), testing-patterns.md (tests that fail under a wrong model), streams.md (durable streams and streaming invocations), tools.md (tool invocations and entity bodies), retries.md (in-function versus trap-based retries), and filesystem-inspection.md (exact-path live reads, shared scheduling and generation-pinned production).

Three axioms

1. Replay is deterministic by construction. The guest is a deterministic function of its guest-observable host inputs. The executor records every nondeterministic host interaction in the oplog and feeds the recorded result back during replay. If a replaying guest asks for a different host call than the one recorded next, the executor has a bug (missing recording, wrong reconstruction, wrong completion association). It is never "normal nondeterminism". The host side is not deterministic — completions are tokio futures and finish in varying order — but replay reproduces the guest-visible completion order from the recorded CompletionDelivered markers, so the guest sees the same inputs in the same order. Owner: durable_host/replay_state/claims.rs (a missing Start match while replay is active is a strict divergence error, except for entries deleted by a Jump). The axiom is about replaying the same execution contract; replaying old history against a changed component during an Automatic update may legitimately diverge, and that is reported as FailedUpdate, not tolerated.

2. The resident runtime is disposable. The Wasmtime Store, the worker task, the executor process, sockets, channels and subscriptions can vanish while work is pending. Suspension, eviction, restart and crash must all be recoverable the same way, on the same executor: throw the instance away, build a new Store, replay the oplog, continue. Lifecycle hints (Suspend, Interrupted, Restart) change status and scheduling policy, not the recovery mechanism. Losing the shard does not reconstruct in place: a durable agent's oplog asserts this executor's shard epoch whenever it appends or deletes entries in storage, and once another executor holds the shard, the write is refused (OplogError::Fenced) instead of accepted. The agent is then retired for its lost shard (InterruptKind::ShardLost): that generation is stopped, dropped from this executor and never restarted, and the executor that now holds the shard builds a new Store and replays — the new owner, or this executor again if the shard came back to it at a higher epoch. See "Resharding, revocation and the oplog epoch fence" below and crash-matrix.md. Owner: worker/invocation_loop.rs::run (outer loop: create instance → recover → run → suspend/retry) and durable_host/mod.rs::prepare_instance.

3. Durable agent RPC is exactly-once at the logical callee-invocation level. Dispatch attempts (replay, transport retry, atomic rollback) share a durable key. An executing target persists one invocation and result; read-only cache hits/followers persist neither. Attempts are not executions. Owner: durable_host/wasm_rpc/mod.rs (derive_idempotency_key(begin_index)), worker/mod.rs::enqueue_worker_invocation_with_effect (lookup_invocation_result dedupe before appending PendingAgentInvocation).

┌──────────────┐ host call ┌──────────────────┐ append ┌──────────────┐
│  live guest  │──────────▶│ durable_host fn  │───────▶│    oplog     │
│  (Store #1)  │◀──────────│ Start…End…Deliver│        │ (persistent) │
└──────────────┘  result   └──────────────────┘        └──────┬───────┘
        ✕ Store #1 disappears (suspend / evict / crash / restart)  │
┌──────────────┐ host call ┌──────────────────┐  read   │
│ replay guest │──────────▶│ claim Start,     │◀────────┘
│  (Store #2)  │◀──────────│ resolve End,     │
└──────────────┘ recorded  │ deliver at marker│
                           └────────┬─────────┘
                    cursor exhausted│ + reconstruction validated
                                    ▼
                           publish Live → new work appends again

Durable versus resident state

Durable (authoritative)	Resident (disposable, derived)
Oplog entries and their payloads (golem-common/src/base_model/oplog/mod.rs)	Wasmtime Store, instance, linear memory
PendingAgentInvocation / AgentInvocationStarted / AgentInvocationFinished	Worker.queue: VecDeque<QueuedWorkerInvocation>, event subscriptions
Idempotency keys and recorded invocation results	hydrated_invocation_results cache, read-only cache
Durable-call Start/End/Cancelled and CompletionDelivered/CompletionDiscarded markers	DurableCallSession, ReplayableOneshot, spawned store tasks
PendingUpdate / SuccessfulUpdate / FailedUpdate, Snapshot hints, snapshot blobs	In-flight update decision, loaded snapshot bytes
BeginAtomicRegion/EndAtomicRegion/Jump entries (they mark skipped history; existing entries keep their indices)	Atomic-region logical idempotency counter (rebuilt from those entries during replay)
Agent status folded from the oplog (worker/status.rs)	Status cache / checkpoints (baselines only)
Persisted promises and scheduler actions (WakeupScheduler), RunningWorkers recovery index	Tokio timers, in-memory wake channels

Anything in the right column may be rebuilt from the left column at any time; a design that needs something from the right column to survive a restart is wrong. Sockets and other OS resources are recreated, not preserved (durable_host/sockets, durable_host/http); the application protocol must tolerate reconnect. Durable liveness also needs rediscovery: a timed wait schedules its wakeup as a persisted scheduler action first (durable_host/suspendable_wait.rs, WakeupScheduler::sleep_until), and the shard-keyed RunningWorkers index is updated synchronously so a crash/reshard can enumerate workers with pending work (worker/status_flusher.rs). The status blob cache is an asynchronously flushed baseline only. Status, clean-checkpoint, invocation-result-index, and durable-stream-index cache namespaces include the current Create.instance_id (AgentFingerprint). Readers select them only after resolving the authoritative Create entry, while resident workers carry that fingerprint directly. A delayed writer from a deleted incarnation therefore cannot mix fields into its replacement's derived state. These namespaces have a renewable expiry and are rebuilt from the oplog after expiry or cache loss; metadata compare-and-mutate operations make each multi-field publication atomic. The flat cached agent mode only chooses which oplog namespace to probe first and never proves existence or identity. RunningWorkers entries also include the fingerprint, so shard recovery admits only the recorded incarnation and removes only an exact stale member.

Two persistence steps matter for every crash window: append puts an entry in the oplog buffer; commit makes it recoverable (commit_oplog_and_update_state(CommitLevel)). The buffer is a performance trade-off (no storage round trip per append). CommitLevel (services/oplog/mod.rs) says how strict a commit is: Always waits for durable storage; Deferred still waits for durable oplogs but lets an ephemeral oplog return after ordered writer handoff (bounded queue backpressure may wait); DurableOnly waits only for durable agents and remains a no-op for ephemeral agents. Explicit protocol barriers and invocation acceptance still use their existing storage guarantees; this is not a global weakening of commit semantics.

Successful invocation completion has a narrower contract. It appends AgentInvocationFinished, then waits for the state actor's commit receipt before notifying waiters. Durable completion uses Always, so that receipt follows storage. Ephemeral completion uses Deferred, so its receipt proves ordered writer handoff, not storage. Neither mode waits for the status fold: the actor retains that fold after acknowledging completion. A later actor-FIFO status read that requires freshness waits behind the queued fold; the separately persisted status cache remains asynchronous. Admission and authority jobs continue to await their complete folds, and the RunningWorkers recovery index remains synchronously flushed by the status actor. The unchanged synchronous Create write preserves an ephemeral agent's identity before execution; after executor loss, that identity reconstructs an observation-only owner, never a fresh execution.

A commit can also be refused. A durable agent's primary oplog, opened while this executor holds the agent's shard, asserts that shard epoch whenever it appends or deletes entries in storage — an explicit commit, a threshold flush, a deletion — inside the storage transaction. Storage refuses the write with OplogError::Fenced unless the record it holds names that epoch: a newer epoch means another executor took over, and no record at all refuses too. An append only buffers, so it meets the check when the buffer is committed. commit_oplog_and_update_state and add_and_commit_oplog surface the refusal rather than swallowing it, so a refused PendingAgentInvocation commit is not acknowledged as accepted and a refused AgentInvocationFinished commit is not published to waiters. The first refusal latches: every later add or commit on that handle is refused locally, without reaching storage, and the agent is retired for its lost shard instead of retried. The refusal travels as a typed error (OplogError::Fenced, StreamStoreError::Fenced, SessionError::Fenced, WorkerExecutorError::OplogFenced) to whoever acts on it; nothing re-reads the latch to reinterpret a failure that already happened.

The check sits at the commit, not at the effect. A call whose Start is only buffered when its effect runs — an idempotent WriteRemote, which opens no committed scope — can still run on an executor that has just lost the shard; its commit is then refused and the new owner runs it again, the same window as a crash before that commit. Non-idempotent, batched and transactional calls commit their scope Start first, so a refusal stops them before the effect. Oplogs that assert no epoch are never refused: ephemeral agents' oplogs (their archive layer appends without one), a handle opened before this executor has an assignment, fork stages and their publication, and compressed archive chunks. Nor do two writes a primary oplog makes outside its entries: the prefix it drops once the archive transfer has copied it (drop_prefix takes no epoch, and a latched fence does not stop the transfer), and blob uploads of large payloads.

worker/state_actor.rs::commit_and_update_state samples the appended tip before its explicit commit and ignores receipt entries already folded into the published status. Primary/ephemeral threshold flushes and replica waits can commit outside the status actor, so even an empty receipt may hide a committed suffix. Ephemeral threshold flushes hand batches to a bounded asynchronous writer; read snapshots include both handed-off entries and the buffered tail after the persisted writer watermark. Up to 32 batch receipts are retained for status folding. If that cap creates a receipt gap, exceptional catch-up forces storage only after the completion receipt has already acknowledged the caller, then reads the missing range. Otherwise, unless the remaining receipt is exactly the contiguous suffix after the last published index, the status actor catches up with status::try_fold_status_from: committed storage is read in bounded chunks, external StreamSession payloads are hydrated, and the result is published once. This avoids retaining an unbounded auto-flushed tail and adds neither oplog entries nor a protocol change. Gap recovery conservatively invalidates authority snapshots after the fold. This is status reconstruction, not replay tolerance.

Component map

Area	Files	Responsibility
Oplog model	golem-common/src/base_model/oplog/{mod.rs,oplog_macro.rs}	Entry kinds, is_hint(), Start/End pairing rules
Replay cursor	durable_host/replay_state/{mod.rs,cursor.rs,claims.rs}	Single cursor lock, speculative read vs commit, identity claims, Replaying→Settling→Live
Durable calls	durable_host/concurrent/{mod.rs,call.rs,delivery.rs,replay.rs}	DurableCallSession state machine, drop policies, completion delivery tokens
Durability guard	durable_host/durability.rs	begin_durable_function/end_durable_function, DurableFunctionType, in-function retry
Host context	durable_host/mod.rs	prepare_instance, resume_replay, switch_to_live, PendingReplayToLive, idempotency-key derivation, snapshot boundary checks
Tail work	durable_host/tail_work.rs	Keeps store tasks running until no durable work is active before AgentInvocationFinished
RPC	durable_host/wasm_rpc/mod.rs	Key derivation, first dispatch vs MayExist, replay claims, ephemeral phantom identity
Worker	worker/{mod.rs,invocation_loop.rs,lifecycle.rs,instance.rs,status.rs}	Queue persistence, dedupe, interrupt/resume/update decisions, eviction, retry decisions
Cut points	worker/cut_point.rs	Preserves atomic/transaction outcomes; ordinary calls recover from the exact retained prefix
Durable streams	durable_host/{durable_stream/mod.rs,durable_session/mod.rs,stream_session.rs,stream_bus.rs,stream_transport.rs,schema_value_stream.rs}	Producer-oplog stream records, consumer session journal, exactly-once item delivery, protocol terminals
Tool invocations	durable_host/tool/{mod.rs,streams.rs,boundary.rs,operation/mod.rs,attachment.rs}, durable_host/entity.rs, worker/{entity_slot.rs,owner_lane.rs,entity_invocation.rs,instance.rs}	Outer-surface authorization, pinned middleware plans, typed boundaries, owner-oplog entity bodies, shared replay state, operation settlement

Worker lifecycle and reconstruction

Worker (worker/mod.rs) receives invocations, persists them, and processes them in order. The run loop (worker/invocation_loop.rs::run) is:

┌─────────────────────────────────────────────────────────────────────┐
│ outer loop (one iteration = one resident instance)                  │
│  create Store + instance ──▶ prepare_instance                       │
│    durable agent:  handle PendingUpdate ▸ try snapshot ▸ resume_replay│
│    ephemeral:      replay typed initialization only, append Restart │
│  ──▶ inner loop: pop durable queue, run invocation, persist Finished │
│  ──▶ instance ends (idle / interrupt / trap / suspend)               │
│  ──▶ RetryDecision: Immediate | Delayed | ReacquirePermits | None | TryStop │
└─────────────────────────────────────────────────────────────────────┘

resume_replay (durable_host/mod.rs) loops get_oplog_entry_agent_invocation_started, replays each recorded invocation in InvocationMode::Replay, and when there is no further AgentInvocationStarted it switches to live. Ephemeral owners replay at most one recorded AgentInitialization, and only when their resolved owner is a typed agent. Component-baseline tool owners instantiate the deployed component but never queue or replay an agent constructor. Replay starts from the chosen snapshot baseline (see Snapshots and updates), not necessarily from OplogIndex::INITIAL. Interruption kinds (Worker::set_interrupting): Interrupt stays interrupted, Restart is a simulated crash with automatic recovery, Suspend unloads and resumes on demand — each of these three reconstructs on this same executor. ShardLost does not: it means this executor lost the agent's shard (a revoked/reassigned shard, or an oplog write refused on the shard epoch), and instead of reconstructing, that generation is retired — stopped without writing its status, dropped from this executor and never restarted — and the executor that now holds the shard reconstructs it.

Eviction (EvictionClass::{LoadedIdle, WarmRunnable}) never unloads a worker that is executing or holds non-durable in-memory work. Ephemeral agents are fail-stop: reconstructed_ephemeral rebuilds only for observation and result lookup, "but the instance must never be started again" (worker/mod.rs, INACTIVE_EPHEMERAL_AGENT_ERROR).

Explicit interruption retires the cached owner and fences its replacement startup. Automatic shard-assignment recovery leaves Interrupted workers stopped, even with queued invocations or updates; an executor restart or shard move is not a request to resume them. If the worker has already unloaded (for example during OOM backoff), the retiring owner must commit an unclaimed terminal interrupt and notify invocation waiters before removal; no Store remains to do it. A terminal interrupt already claimed by the invocation loop is not recorded again, and a completed or failed invocation is not overwritten. Test: tests/scalability.rs::interrupt_during_oom_backoff_is_durable_before_restart.

recover_immediately selects Restart for Running, Suspended and Retrying workers. It never turns a simulated crash of a parked worker into a permanent interruption. If no invocation loop remains, the existing promise, scheduler or permit wakeup starts reconstruction; the queued restart does not fail the invocation waiter or append Interrupted.

Environment and application deletion invalidate component metadata, environment state and agent type caches before awaiting owner retirement. New metadata lookups then observe deletion instead of admitting requests against a retiring cached owner.

External metadata observation uses a fallible FIFO status read. If the actor has stopped, the lookup takes the cold oplog lifecycle guard, verifies retirement, joins writer completion, and reconstructs from persisted metadata and the oplog. A failed deletion can therefore leave a cached but stopped worker observable until removal is retried. Missing storage means absence; failed reconstruction means an error, never a stale cached status or permission to restart work.

Ephemeral response leases delay only normal archival, not Store unloading or explicit retirement. The shared gRPC owner lookup acquires the lease before reading session metadata or accepting work. If normal archival already fenced the owner, lookup joins archival through cache removal, then resolves an observation-only owner from storage. Archive failure or cancellation rejects the lookup; it never grants access to the old poisoned producer or restarts the ephemeral invocation.

Normal archival and explicit interruption share Worker::quiesce_for_owner_retirement under the owner-cleanup lock: stop execution, drain stream retirement and lifecycle/forwarding work, commit, then stop status writers. Only explicit interruption records a pending terminal interrupt. Archival moves the oplog before removing the cached worker. The open oplog generation and its forwarding wrapper may be reused by the next worker, so ordinary retirement does not close their task admission or forwarding. Deletion claims ownership under the same owner-cleanup lock, then drains the resident stream producer before running maintenance on a private producer. Failed maintenance is drained before a retry; only deletion closes the oplog generation permanently. Archive transfers append and verify the destination before dropping the source. Archive storage failures therefore fail only that maintenance attempt: they are logged, never fail the agent, the authoritative source remains intact, and threshold, scheduled or sweep maintenance retries with per-agent backoff. A scheduled retry remains valid when later commits advance the oplog tip. An archive read needed for replay still fails recovery rather than being treated as absent data.

Producer-targeted attachment controls (consumer-side finalization, activation, refresh) reach the producer's executor through Rpc::control_durable_stream_attachment, which returns DurableStreamRemoteError like slot reads. A producer that is recovering, fenced, or being deleted answers Unavailable; the caller retries with backoff instead of failing. A producer that no longer exists answers not-found, and consumer-side finalization treats that as success because the producer's own deletion cascade already dropped the attachment. Consumer deletion bounds each producer finalization with a timeout so a producer that never becomes reachable fails the deletion attempt instead of hanging it; the attempt can be retried. Producer deletion never waits for consumer cooperation.

Cold acquisition reserves one unresolved Worker in ActiveAgents. initialize_with owns one shared attempt independently of request cancellation. finish_construction prepares resolved data privately; failure drains and joins attempt-owned work before returning to Unresolved, without deleting persisted data. Existing waiters receive that attempt's error; later explicit demand retries by reloading persisted identity and pending initialization. Local success publishes the resolved data and Unloaded state. Remote topology recovery and dependent finished-session recovery run together in the post-publication reconciler, preserving the deletion gate: attachment RPCs can acquire mutually referring cold workers on different executors, so awaiting them before publication would create a cycle. Local readiness does not authorize a merely prepared stream attachment. The reconciler folds only through the committed last_known_status.oplog_idx. Once its topology cache has no unfinished recovery, it parks on committed stream-state notifications even while attachments are active. First producer load wakes a worker with no prior stream history; committed terminals wake deferred session completion after the status fold. Only failed or unfinished recovery arms a retry deadline. Stale producer attachments are reconciled lazily, including before deletion decides which dependencies remain. A source read repairs a missing producer activation only after checking the exact committed Active consumer topology; healthy reads do not enter the mutation lane. Each reconciler uses a child of the executor shutdown token, so graph shutdown stops new recovery passes and wakes parked reconcilers. Explicit owner retirement, revert, and deletion cancel and join the reconciler's in-flight pass; TTL cache retirement does not itself cancel or join the reconciler. Tests: tests/worker_initialization.rs exercises shared failure, real actor completion, cancellation, existing-only acquisition, reciprocal cold topologies, and reconciler graph shutdown.

Lifecycle operations acquire the cached or persisted Worker through an existing-only path, so interrupt, delete, resume, update, revert, and plugin changes never create an absent agent. Delete is owned by that worker: concurrent callers share its retained attempt result, a later call retries only unfinished cleanup stages after failure, and successful cleanup retires active-worker and open-oplog cache entries only for the generation being deleted. A stale Arc<Worker> therefore cannot continue deletion against, or evict cache state belonging to, a replacement with the same AgentId.