Exactly-Once Workflow Execution: What the Guarantee Really Means

Name the guarantee at each layer

Treat an unknown outcome as a state

The hardest case is a timeout after dispatch. Retrying may duplicate the effect; failing may hide a success. Persist the request identity before dispatch, then query the provider using that identity before deciding.

Do not generate a new key for each retry. The key belongs to the invoice charge or recipient notification, not to the worker attempt.

prepared -> confirmed
prepared -> unknown -> reconciled
prepared -> rejected
confirmed -> compensated

Prove the business outcome

Orch8 can reject stale task completion and retain the workflow boundary. Your handler and provider integration still determine whether the external result is safe. Document both sides of the contract.

• Kill the worker after the provider accepts the call.
• Deliver the same task to two workers.
• Expire a lease while the first worker is still running.
• Retry after the provider's idempotency window.
• Verify one external effect and one durable receipt.

Which delivery and execution terms should a design use?

At-most-once delivery means the system will not deliberately deliver a task more than once, but loss is possible. At-least-once delivery means the system retries until acknowledgement or policy exhaustion, so duplicates are possible. Exactly-once processing is meaningful only inside a named transactional boundary. Exactly-once effect across an engine, network, and unrelated provider generally requires cooperation from every participant and should not be claimed as one blanket property.

Separate four questions in documentation: can the engine commit a state transition once, can a task be delivered again, can the handler run again, and can the external effect repeat? A stale lease token can protect the first question while leaving the other three dependent on acknowledgement timing and provider contracts.

Use the phrase effectively once for a business outcome only when duplicates are detected or neutralized through a stable identity, ledger, and reconciliation. State the time window and failure modes. If a provider forgets idempotency keys or has no lookup, the guarantee weakens after that boundary.

When does a transactional outbox help?

An outbox atomically stores an outgoing command in the same database transaction as the business state that requires it. A separate dispatcher reads unsent rows and delivers them at least once. This closes the gap where application state commits but a process dies before enqueueing the message. It does not, by itself, close the gap where the consumer or external provider acts but acknowledgement is lost.

Give each outbox record a command ID derived from the business intent. Consumers store processed command IDs in the same transaction as their local changes. For external providers, pass the ID as the provider idempotency key and retain the receipt. Mark the outbox delivered only after the downstream acknowledgement is durably recorded; retries reuse the same ID.

Partition and index the outbox for polling, cap batch size, and monitor oldest unsent age. Archive or compact delivered rows without removing deduplication evidence before its retention window. If multiple dispatchers compete, claim rows with transactional locking and accept that delivery can still repeat after a crash.

transaction:
  update business_state
  insert outbox(command_id, payload_digest, pending)

dispatcher:
  claim command_id
  send with command_id as idempotency key
  persist receipt and delivered status

What can deduplication not guarantee?

A consumer inbox can prevent applying the same message ID twice to one local database. It cannot detect two different IDs that represent the same business command, so producers must preserve intent identity. It also cannot undo an external effect performed before the inbox transaction. Place the deduplication check before local mutation and use a provider key for the remote operation.

Retention matters. If inbox records expire after seven days and a dead-letter message can be replayed after 30 days, the old duplicate becomes new again. Align retention with queue redelivery, backfill, dispute, and audit windows. For unbounded histories, store a compact domain marker such as the latest billing cycle processed rather than every transient attempt where the domain permits it.

Payload mismatches must fail closed. When the same ID arrives with a different digest, do not return the earlier result silently. Log a conflict, quarantine the message, and investigate whether an ID generator was reused or a caller mutated a command during retry.

• Deduplicate by business command ID, not delivery attempt ID.
• Persist the accepted payload digest.
• Make the check and local mutation one transaction.
• Align evidence retention with maximum replay windows.
• Treat duplicate ID plus different payload as an integrity error.

An architecture review for consequential effects

For each payment, email, entitlement, or publication step, document the intent key, system of record, atomic boundary, delivery contract, provider idempotency behavior, ambiguous-response lookup, compensation, evidence retention, and operator control. Draw the crash points on the sequence diagram and assign the durable state expected after each one.

Then test concurrency. Deliver the same task twice, let two workers race, expire one lease, delay acknowledgement, and retry after a deployment. Assert provider request count, accepted state version, ledger status, and user-visible outcome. A green workflow status is insufficient evidence.

Review product copy with the same rigor. Replace universal exactly-once statements with the exact layer and condition: commit guards select one dispatch winner for a declared effect identity; provider idempotency and reconciliation handle remote uncertainty. This wording is less dramatic and far more useful during an incident.

How should the guarantee be documented for users and operators?

Write a guarantee table for every consequential handler. Name the internal transaction boundary, delivery mode, stable business identity, provider behavior, maximum deduplication window, reconciliation method, and terminal states. Include one example that succeeds after a lost acknowledgement and one case that remains unknown. This gives application engineers a contract they can test rather than a slogan they must interpret.

Expose the same distinctions in operational screens. Show accepted workflow state separately from task delivery and provider effect status. A workflow can be retrying while its payment intent is already confirmed, or internally complete while a downstream notification remains unknown. Operators should not need to infer those facts from an undifferentiated success flag.

Review the contract when a queue, provider, SDK, timeout, retry window, or retention policy changes. A provider that shortens idempotency retention can invalidate an application promise without changing your workflow code. Add the stated guarantee and its failure-injection cases to release review so marketing, documentation, implementation, and evidence remain aligned.