Verdict
Verdict
A green run is not proof of a correct result. Production automation needs explicit outcome checks, durable state, idempotent effects, and an owner for every failure path.
The dangerous failure is a plausible success
A hard crash is obvious. A workflow that reports success while creating a blank PDF, skipping three rows, or emailing the wrong attachment is worse. The scheduler moves on, the dashboard stays green, and a customer finds the defect hours later.
Test runs rarely reproduce production timing, expired credentials, partial API responses, concurrent executions, large payloads, or a worker restart between two side effects. Those are normal operating conditions, not edge cases.
- Transport succeeded, but the business outcome failed
- A retry repeated a non-idempotent side effect
- A webhook arrived twice or out of order
- The run stalled after a worker disappeared
- A downstream API returned a valid but incomplete response
Separate execution health from outcome correctness
Track two contracts. The execution contract asks whether every scheduled step reached a terminal state. The business contract asks whether the expected artifact exists and passes validation.
For a form-to-PDF workflow, an HTTP 200 response is execution evidence. A non-empty PDF, stored under the expected record, with required fields present is outcome evidence. Alert on both.
- Assert required output fields before completion
- Record external object IDs beside the workflow run
- Reconcile created artifacts against source records
- Treat empty or partial output as a failure
- Keep a searchable reason code for every terminal path
Make retries safe before enabling them
Retries improve availability only when repeating a step cannot repeat the business effect. Give each external write a stable idempotency key derived from the workflow instance and logical step. Store the returned provider ID before moving forward.
If the provider lacks idempotency support, add a local effect ledger. Check the ledger before the call, write the result atomically afterward, and reconcile uncertain timeouts instead of guessing.
- Payment: workflow ID plus invoice ID
- Email: campaign ID plus recipient plus touch number
- PDF: source record ID plus template version
- CRM update: entity ID plus intended state version
Design for interruption
A worker can die after an API accepted a request but before the engine recorded success. That tiny gap causes most duplicate-effect bugs. Persist progress around side effects and expose stuck-run detection based on heartbeats or leases.
Orch8 persists sequence state in PostgreSQL or SQLite and lets workers remain ordinary HTTP services. That makes crash recovery visible, but handlers still need idempotency because no workflow engine can erase ambiguity inside a remote API.
- Set timeouts shorter than the business deadline
- Renew leases only while useful work is happening
- Resume from the last durable boundary
- Route exhausted retries to a named owner
- Preserve inputs and outputs needed for replay
Build alerts around actionability
An alert that says ‘workflow failed’ creates another investigation queue. Include the workflow, instance, last successful step, failure class, affected customer or record, and the safest next action.
Page immediately for widespread corruption, money movement, or security impact. Send lower-risk failures to a queue with an SLA. Alert fatigue is a reliability defect too.
- Failure rate by sequence and version
- Age of oldest running instance
- Retry exhaustion count
- Duplicate-effect prevention count
- Outcome validation failures
- Time from failure to acknowledgement
A practical production test
Do not stop at the happy path. Kill a worker during the external call, replay the trigger, expire a credential, return malformed JSON, delay the downstream API, and send the same webhook twice. Then confirm that the run either completes once or stops with enough evidence to repair it.
The real acceptance criterion is blunt: someone on call at 02:00 can understand what happened without reading application source.
Implementation checklist
- 01Define execution and business outcome contracts
- 02Add stable idempotency keys to every side effect
- 03Validate outputs before marking the run complete
- 04Test duplicate, timeout, restart, and partial-response cases
- 05Assign an owner and SLA to every terminal failure
- 06Measure stuck runs and retry exhaustion
- 07Run a reconciliation job for high-value outputs