Skip to content
orch8.io

Demo · crash recovery

Crash the engine mid-run. Watch the run come back.

This is a scripted replay of the real orch8 demo crash-recovery command. It leaves a workflow instance mid-flight, simulates the engine dying by dropping its only database connection, then restarts against the same database file and reclaims the orphaned work.

orch8 demo crash-recovery

$

$ orch8 demo crash-recovery Crash recovery completed instance: 01a0dbca-ff4c-757a-b05c-28ab274ac2ad state: running -> scheduled reclaimed: 1 $ orch8 demo crash-recovery --output json { "database": "/var/folders/nx/mmm7msjj4mn84m1wlsk0p3ch0000gn/T/.tmpRcy27t/crash-recovery.db", "instance_id": "01a0dbcb-001d-77ae-84a2-583e0c2df5ae", "before_restart": "running", "recovered": 1, "after_restart": "scheduled" }

  1. 1

    Create a real SQLite database

    A fresh file-backed database in a temp directory, with a sequence and one workflow instance for tenant crash-demo.

  2. 2

    Leave the instance mid-flight

    The instance is written in the running state with its last update ten minutes ago, as if a worker had claimed it and never reported back.

  3. 3

    Simulate the crash

    The only storage connection pool is dropped without any cleanup, which simulates the engine process disappearing mid-run. No real process is killed.

  4. 4

    Restart and recover

    A new storage handle opens the same file, as a restarted engine would, and runs the engine's stale-instance recovery with a 60-second threshold.

  5. 5

    Check the invariant

    The command exits with an error unless exactly one instance was reclaimed and it is back in scheduled, ready to be claimed again.

The output was captured verbatim from real runs of orch8 0.7.1 built from the engine main branch on September 26, 2026. Instance IDs and temp paths change on every run. The animation respects your reduced-motion setting.

Run it yourself

The demo needs no server, database, or network. It creates a throwaway SQLite file in a temp directory.

The demo crash-recovery subcommand is on the engine's main branch and is not in a tagged release yet (latest: v0.7.1), so build the CLI from source for now. Once it ships, brew install orch8 or the other installers will include it.

It exits non-zero if the recovery invariant fails, so you can also use it as a smoke test in CI.

Read the demo source (demo.rs) ↗
build from main
git clone https://github.com/orch8-io/engine.git
cd engine
cargo run -p orch8-cli --bin orch8 -- demo crash-recovery
with an installed CLI (once released)
orch8 demo crash-recovery

# machine-readable evidence
orch8 demo crash-recovery --output json

What this proves, and what it doesn't

It exercises the recovery routine the engine runs at startup and on a periodic reaper: a running instance whose owner vanished is detected as stale and moved back to scheduled, so the next worker can claim it. It uses the real SQLite storage adapter and the engine's own recovery function against a real database file.

It is deliberately small. There are no workers, no HTTP server, and no external side effects. Protecting a side effect that was in flight at crash time is a separate mechanism: stable idempotency keys and the effect ledger, which records whether an effect was uncommitted, committed, unknown, or compensated.

Resilience docs →Interactive playground →Benchmark methodology →Compare engines →

Now try it with your own workflow

Start a local project, add a worker, and kill it mid-step. The quickstart walks through it in a few minutes.

Run the quickstart