By Orch8 Engineering · Editorial policy
Versioning Workflows That Run for Weeks
Pin each workflow run to the definition contract it started with. Deploy additive worker and schema changes first, route old and new runs to compatible handlers, and migrate active state only through a tested explicit transformation. Retire a version after no active or retryable run depends on it.
What is the practical answer?
Long-running workflow versioning must preserve the contract each execution started with. Store the definition version on the run and account for every related contract: payload schemas, handler names, credential scopes, external API behavior, and database columns. Prefer coexistence before migration. Keep old workers for runs that will finish soon, use additive handlers when both versions can share behavior, and transform stored state only through an explicit, tested migration with rollback. Database changes should remain backward compatible until no old worker or active run needs the prior shape. Before retiring a version, count active, waiting, failed, and retryable runs, including scheduled retries and delayed signals. Archive the definition and migration code with the release and verify rollback after real canary executions. A successful deployment only proves the new code started; it does not prove the previous workflow contract is unused. Remove old capability based on execution evidence, not elapsed time or deployment confidence.
Evidence: Temporal: worker versioning · Martin Fowler: parallel change
Version every contract the run touches
The workflow graph is only one contract. Payload schemas, handler names, external API behavior, credential scopes, and database columns may all change while a run is waiting.
Store the definition version and validate payloads at each boundary. If a handler changes meaning, give it a new versioned name or route by capability instead of silently replacing behavior.
Choose coexistence before migration
| Strategy | Use when | Main cost |
|---|---|---|
| Keep old workers | Old runs will finish soon | Parallel capacity |
| Compatible handler | Change is additive | Longer compatibility code |
| Explicit state migration | Old path cannot remain | Testing and rollback |
| Terminate and restart | Business permits it | Lost progress or repeated effects |
Retire based on evidence
Orch8 versions sequence definitions so active runs can retain their contract. That helps coexistence; teams still need a retirement query, compatible workers, and an operational decision for abandoned runs.
- Count active, waiting, failed, and retryable runs by version.
- Include delayed signals and scheduled retries.
- Archive definitions and migration code with releases.
- Test rollback after the new version processes real canary runs.
- Remove old capability only when the count is zero and retention policy permits.
What must be inventoried before changing a workflow?
List every active definition version and count runs by state: executing, waiting on a timer, waiting on a signal, failed but retryable, paused, and retained for operator action. Include child workflows and schedules that can start another run. A version with no currently executing tasks may still be required tomorrow when a timer fires.
Map each definition to worker capabilities, payload schema, stored state schema, credentials, external API versions, and database assumptions. A workflow version is not just a JSON file or function body. If version one calls handler charge:v1 and reads customer.currency as optional, those contracts must remain available until all dependent runs close or migrate.
Automate the inventory and attach it to the release decision. Human memory is unreliable across workflows that last weeks. The release should fail closed when an active version has no eligible worker or when a proposed database contraction removes a field still read by an active contract.
| Inventory item | Question | Retirement evidence |
|---|---|---|
| Definitions | Which versions have open runs? | all dependent counts zero |
| Workers | Which builds serve each version? | no pinned routing remains |
| State schemas | Which shapes are stored? | migration or retention complete |
| External APIs | Which provider versions are called? | old calls and retries exhausted |
How should a change be classified?
Additive changes introduce an optional field, new branch for new runs, or handler capability that old contracts ignore. Compatible changes preserve existing IDs and meaning while improving implementation. Migrating changes alter stored state or routing through an explicit transformation. Breaking changes remove or reinterpret something an active run may still reference.
Classify the workflow, payload, state, handler, and database changes separately. A new optional payload field may be additive while renaming the step that consumes it is breaking for snapshots. A replay system may accept a database change while rejecting control-flow reorder during history replay. The most restrictive classification controls the release path.
Record why the classification is safe and which tests prove it. Static diff tools can flag removed IDs or schema changes, but domain meaning still needs review. Changing approved to accepted may be technically compatible and legally significant.
- Additive: old readers ignore the new field.
- Compatible: implementation changes, contract meaning stays stable.
- Migrating: stored state changes through a versioned transform.
- Breaking: an active run can no longer interpret or execute its contract.
- Unknown: evidence is insufficient, so block automatic promotion.
What makes a state migration safe?
A migration must declare source and target versions, preconditions, transformation, rejected states, effect-identity preservation, and rollback boundary. Run it first as a preview over a production-sized copy. Report counts and hashes for unchanged, transformed, and rejected records. A migration that silently defaults an unknown branch is not safe.
Make application repeat idempotent. Store the migration ID and resulting state version with each record so a retry can return the prior result. Process bounded batches and expose progress. Pause new transitions for the affected records or use compare-and-set so a workflow cannot advance between read and write.
Rollback may stop being possible after the target version dispatches a new external effect or writes data the source contract cannot interpret. Define that point explicitly, require approval before crossing it, and retain the source snapshot or transformation evidence according to policy.
source_version: 3
target_version: 4
precondition: waiting_at_review
transform: review_id -> approval_id
preserve: effect_intent_ids
reject: unknown_step_id
rollback_until: first_v4_effect_dispatchHow do you retire an old workflow version?
Stop assigning new runs to the old version and wait through the maximum schedule and retry delay. Query open and retained states, delayed signals, child workflows, and dead-letter entries. Confirm that old workers have no owned leases and that no operator control can resume an old run after the worker disappears.
Archive the definition, worker artifact digest, schemas, migration code, and runbook. Remove routing only after the evidence gate reaches zero. Observe for one retention cycle before deleting compatibility code or database fields. If storage cost requires earlier cleanup, terminate or migrate the remaining runs through an approved business decision rather than pretending they do not exist.
After retirement, run a restore test from a backup taken before the change. Historical records may be needed for an audit or incident even when they cannot resume. Document which artifacts are required to read them and how long those artifacts are retained.
How do you rehearse a versioning incident?
Start runs on every supported definition version and pause them at timers, approvals, retries, child waits, and external-effect boundaries. Deploy the candidate, remove one compatible worker intentionally, deliver delayed signals, and exercise rollback. The expected outcome is visible routing failure with preserved state—not silent interpretation by the wrong code.
Have an operator identify affected runs, stop new admission, restore compatible capacity, and verify effect receipts without editing the database. Measure detection and safe recovery time. Archive the drill's state samples and assertions so future schema, SDK, and routing changes face the same evidence gate.
Include an execution whose state predates the current retention dashboard and a delayed message carrying an old schema. These cases catch retirement queries that count visible running rows but omit dormant control paths. The drill closes only when both are routed, rejected safely, or migrated through a documented decision.
- Include delayed and dead-letter work in active-version counts.
- Validate signal schemas against the recorded version.
- Reject unknown step identifiers explicitly.
- Prove the old artifact can still be built or retrieved.
Sources and further reading
Official references support technical claims; community discussions are used only as problem signals.