State Snapshots vs Event Replay for Durable Workflows

Two different answers to the same crash

After a process dies, an engine needs enough durable information to continue. Replay stores an ordered history and runs workflow code against it to reconstruct state. Snapshots store the accepted state at an execution boundary and continue from that record.

Neither model removes side-effect safety. A payment or email that may have succeeded still needs an idempotency key and reconciliation path.

Run the decision experiment

Publish the setup and raw measurements if you turn this into a benchmark. A result without engine versions, storage configuration, workload, and failure point is marketing, not evidence.

• Start a workflow on version A and stop it after an external effect.
• Deploy version B with a renamed step and changed branch.
• Restart the worker and observe whether the old run continues safely.
• Measure recovery time, storage growth, and the steps an operator must understand.
• Repeat with a lost response from the external provider.

Choose the burden your team can operate

Replay fits teams that value a detailed execution history and can enforce deterministic workflow rules. Snapshots fit teams that want explicit state boundaries and conventional application code, while accepting schema migration work.

Orch8 uses persisted execution state rather than rebuilding a run by replaying workflow code. That narrows the recovery model, but it does not make migrations automatic. Keep old definitions available while old runs exist and test upgrades against copied production state.

What happens during recovery in each model?

In an event-replay model, the durable service retains a history of workflow events such as task scheduling, activity results, timers, and signals. A worker re-executes workflow logic against that history to reconstruct the same decisions. Calls that can vary—wall-clock time, random values, network requests, or unordered iteration—must go through deterministic workflow APIs or activities so replay reaches the recorded path.

In a snapshot model, the engine commits an explicit execution state after accepted boundaries. Recovery loads the stored definition version, current position, context, accepted outputs, timers, and ownership data, then schedules the next eligible work. The handler does not re-run simply to reconstruct control flow. Schema compatibility moves to the stored snapshot and definition contract.

Both models still need transaction boundaries. A crash between accepting a step result and persisting the next state must not lose or duplicate the transition. Both also need a separate contract for external effects because neither replay nor snapshots can know what a remote provider did after a lost response without idempotency or reconciliation.

How should a snapshot-versus-replay benchmark be designed?

Publish a repository that fixes engine versions, database versions, machine shape, network topology, durability settings, and workload generator. Use equivalent business workflows: a short linear run, a branch-heavy run, a long history with timers, and a payload-growth case. Separate cold recovery, steady-state transition latency, storage growth, and operator actions; one combined throughput number hides the trade-offs.

For recovery, start a known number of runs, stop workers or the service at controlled boundaries, restart, and measure time until all eligible work resumes. Record how much history is replayed or state is loaded, database reads and writes, CPU, memory, and any continuation or compaction behavior. Include an ambiguous external call, but measure it as a correctness exercise rather than pretending engine recovery resolves provider uncertainty.

Run enough repetitions to show distributions, not one best result. Report median and tail values, raw samples, warmup method, and failures. State whether clients, engine, workers, and database share a host. A benchmark on one laptop can still teach mechanics if it is labeled as a lab result and not generalized to production scale.

engine_version:
database_version:
hardware_and_os:
workflow_shape_and_history_length:
payload_size:
durability_settings:
failure_injection_boundary:
repetitions_and_warmup:
raw_results_location:

Where do upgrades fail?

Replay upgrades fail when new workflow code produces decisions that conflict with recorded history. Safe techniques include patch markers, pinned worker builds, replay testing over sampled histories, and continuing old runs on old code. A harmless-looking refactor can matter if it changes command order, branching, or nondeterministic values.

Snapshot upgrades fail when new code cannot interpret stored state or when a definition changes the meaning of the current position. Safe techniques include versioned state schemas, additive fields with defaults, pinned definitions, explicit transforms, and migration previews over copied production state. Renaming a step is not merely cosmetic if active snapshots refer to its ID.

Create a compatibility inventory before every release: oldest active workflow version, stored schema versions, worker capabilities, delayed timers, retryable failures, and retention. Test the candidate against representatives of each state. Promotion should route new runs separately from the decision to migrate old ones.

• Keep old definitions and workers until dependent runs close or migrate.
• Archive migration code with the release that introduced it.
• Reject an unrecognized state version instead of guessing a default path.
• Preserve effect identities through every state transformation.
• Test rollback after the candidate has processed real canary runs.

Which model gives operators the evidence they need?

Operators need a current state, a causal timeline, external effect identities, ownership, and supported controls. Replay history can provide a detailed event sequence, but long histories require navigation and domain labels. A snapshot view can make the current boundary direct, but it needs an append-only transition or audit log if teams must explain how the state arrived there.

Evaluate ordinary tasks: identify why a run is waiting, find the worker that owns it, inspect the last accepted output without leaking secrets, reconcile an external call, retry or resume through policy, compare the workflow version with the current release, and export evidence for an incident. Count operator steps and required specialist knowledge.

The best recovery model is the one whose constraints the development and operations teams enforce consistently. A theoretically rich history does not help if nobody can interpret it, and a simple snapshot does not help if migrations overwrite meaning.

What evidence should a prototype produce?

Create equivalent workflows in both models and stop them at a timer, after an accepted activity, during a version change, and after an ambiguous external response. Record history size or snapshot size, recovery latency, worker work performed during recovery, external request count, and operator steps. Use production-shaped state rather than a toy counter.

Then corrupt or reject one stored value deliberately. The system should identify the incompatible event or snapshot version and preserve the original evidence. A prototype that only demonstrates the happy path cannot reveal whether replay determinism or snapshot migration will dominate operational work. Keep the fixtures as regression tests for upgrades.

• Measure recovery time as histories or snapshots grow.
• Verify completed external effects never repeat during reconstruction.
• Test old-state compatibility with the candidate release.
• Restore from backup and resume one controlled execution.