How to Deploy While Long-Running Workflows Are Active

Separate definition, worker, and data versions

A run that waits for days will cross deployments. Record the workflow definition version when the run starts. Route its next task only to a worker that understands that version, or provide an explicit migration.

Database compatibility needs the same discipline. Expand schemas first, deploy compatible readers and writers, then contract only after old workers and runs no longer depend on the old shape.

Drain before the process exits

Set the shutdown grace period from real task duration percentiles. A default copied from a web server is often too short for model calls, exports, or human-adjacent processing.

• Fail readiness so the instance stops receiving new traffic.
• Stop polling for new workflow tasks.
• Continue heartbeats for work already owned.
• Finish or checkpoint before the termination deadline.
• Release or let leases expire; never forge completion after ownership is lost.

Use a run-aware release checklist

What must remain compatible during a deployment?

Long-running executions preserve assumptions from the version that started them. Inventory the definition identifier, step IDs, handler names, serialized context, accepted outputs, timers, signal names, retry policy, and effect identities. A release is compatible only if the new workers can interpret every active state they may receive or routing keeps those states on an older worker set.

Classify changes before rollout. Additive optional fields and new paths for new runs are usually lower risk. Renaming a completed step, changing the meaning of a stored value, removing a handler, or reordering an external effect requires versioning or migration. A TypeScript compile does not validate persisted JSON created by a previous release.

Publish a compatibility matrix that names which worker versions accept which workflow versions. Validate it against actual active-state counts, not a sample from the happy path. Unknown versions should fail closed into a visible operational state instead of being interpreted as current data.

How should old and new runs be routed?

Separate admission from execution. A deterministic gate assigns new runs to a definition version, while active runs continue on the version recorded at creation. Do not derive behavior from whichever container happens to poll next. Store the assignment and expose it in search, metrics, and incident views.

Use compatible worker pools or build explicit version branches. With pools, old workers stop taking new admissions but remain available until their run count and delayed work reach zero. With branches, one artifact contains guarded logic for recorded versions. Either approach needs an end date, evidence for retirement, and protection against an old signal or retry arriving after apparent drain.

Canary by a stable dimension such as tenant or intent hash. Measure task failures, migration rejects, stale lease completions, latency, external-effect conflicts, and manual interventions against the control cohort. Pause admission automatically when a safety threshold is crossed; avoid broad rollback that also restarts already accepted work.

on_start: persist workflow_version = assignment(intent_id)
on_task: route by persisted workflow_version
on_signal: validate signal schema for that version
on_rollback: stop new admission; preserve accepted state

When is state migration appropriate?

Migrate only when the target version has an unambiguous equivalent for the source state. Define preconditions, transformation, preserved effect identities, rejected shapes, and the point after which rollback is impossible. Execute migration with a compare-and-swap on state version so a concurrent task or signal cannot be overwritten.

Dry-run against a production-shaped export with secrets removed. Report counts by source version and state, every validation failure, and before-and-after digests. A successful migration means semantic invariants hold: completed effects remain completed, pending timers retain meaning, and no run gains or loses a required branch merely because a field defaulted.

Keep the migration code and source schema for the retention period. Back up before bulk transformation and perform a restore exercise. If only a handful of runs are incompatible, finishing them on old workers is often safer than migrating everything to simplify the deployment diagram.

• Declare source and target versions.
• Lock or compare state versions during transformation.
• Preserve business intent and provider receipt identifiers.
• Validate timers, signals, children, and retry counters.
• Record operator, tool version, timestamp, and before/after digest.
• Prove restore and define the rollback boundary.

What evidence closes the release?

Before rollout, start representative workflows on the current version and stop them at waiting, retrying, timer, human-approval, and external-effect boundaries. Deploy the candidate, deliver signals, expire timers, and force one worker termination. Assert each run resumes from its accepted state without repeating completed effects.

During rollout, retain dashboards for active runs by version, task failure and retry rates, unknown effects, migration rejects, worker compatibility, and queue age. Link release annotations to the artifact digest and definition version. A clean application error rate can hide an old-version queue that no worker polls.

Retire old workers only when no active run, delayed retry, timer, signal route, child, dead-letter entry, or operator control depends on them. Observe for one maximum retry and retention window. Store the definition, schemas, worker artifact, migration tools, and runbook so historical execution records remain intelligible.

A pre-deploy review that can stop the release

Have the release owner query active executions by definition version and state, then reconcile that inventory with available worker versions. Review state migrations, effect-boundary changes, admission rules, dashboards, rollback conditions, and who has authority to pause the rollout. Record the exact artifact digests and configuration rather than relying on a branch name.

Require a demonstrated failure case before approval: terminate a canary worker after a committed boundary, then verify the replacement resumes with no repeated provider effect. If the team cannot identify the authoritative state or route the old version, delay the release. This gate turns compatibility from an optimistic code review comment into observable operational evidence.

Record the result as release evidence: tested workflow versions and states, injected crash point, provider request count, recovery time, and the person who reviewed the outcome. Keep the evidence beside the release so an incident responder can distinguish what was proven from what was assumed. Repeat the check whenever routing or persistence behavior changes.

• Active states have compatible workers.
• New admissions are deterministic and reversible.
• Migration rejects are visible and recoverable.
• Rollback preserves accepted progress.
• Old-version retirement has a zero-dependency query.