Idempotency for Workflow Side Effects: Prevent Double Charges and Duplicate Sends

The dangerous crash window

The worst failure is not a clean error. It is a timeout after the request left your process. The payment provider may have charged the card. The email API may have accepted the message. Your worker never received the response, so its local record still says running.

Retry blindly and you may repeat the effect. Mark it complete and you may hide a real failure. This ambiguity exists whether the caller is a cron script, a queue worker, or a durable workflow.

Give the business intent a stable identity

A retry attempt is not a new business action. Derive the key from the intent—order, invoice, recipient, and notification type—not from the process ID or attempt number. Every attempt for that intent must carry the same key.

Provider behavior varies. Check how long the key remains valid, whether parameters must match, and how you retrieve the original result. If the API has no idempotency support, add an application-side effect ledger with a unique constraint and a reconciliation state for unknown outcomes.

const intentKey = `invoice:${invoice.id}:collect-v1`;

const existing = await effects.find(intentKey);
if (existing?.status === "confirmed") return existing.result;

const charge = await payments.charge({
  amount: invoice.total,
  idempotencyKey: intentKey,
});

await effects.confirm(intentKey, charge.id, charge);

Use a recovery protocol, not another retry flag

Orch8 can keep the workflow boundary and effect identity durable across a restart. The provider still owns the external truth. Completed internal boundaries can stay completed, while uncertain external writes remain explicit instead of being disguised as success or failure.

• Prepared before dispatch: intent key, provider, operation, and request digest.
• Confirmed after response: provider result ID, response digest, and completion time.
• Unknown after timeout or worker loss: block automatic failover and reconcile first.
• Rejected on a definitive provider error: apply the documented retry or failure policy.
• Compensated through a separate, idempotent business action—not by deleting history.

Test the ugly path on purpose

Kill the worker after the provider accepts the request but before the local receipt is saved. Restart it. The expected result is one external effect, one reconciled receipt, and one completed workflow. If your test produces two emails or charges, retries are still controlling your business semantics.

Watch counts for prepared, confirmed, unknown, reconciled, and compensated effects. Alert on unknown outcomes that exceed the provider's normal response window. A generic failed-job counter cannot tell an operator whether a customer was charged.

How should an idempotency key be designed?

Start with the business command, not the transport attempt. A useful key answers which customer-visible action this is, which object it belongs to, and which semantic version of the action is intended. For an invoice collection, a key such as invoice:inv_42:collect:v1 is stable across timeouts, worker restarts, and queue redelivery. A random key generated inside each attempt is unique but not idempotent because the provider sees every retry as a new action.

Keep the key narrow enough that a genuinely new command can use another identity. A second installment, a corrected invoice, or an explicitly approved resend should not collide with the first command. Store a digest of the request parameters beside the key and reject reuse with different parameters. This prevents a stale caller from silently applying a previous receipt to a different amount or recipient.

Document the provider boundary as well. Record its idempotency retention window, parameter-matching behavior, lookup endpoint, and response for concurrent requests. If the provider forgets keys after 24 hours while your workflow retries for seven days, your application ledger must prevent a late dispatch or require operator reconciliation before the provider window expires.

What belongs in an effect ledger?

An effect ledger is a durable record of what the application intended to do outside its own transaction. Write the prepared record before network dispatch in the same transaction that advances the workflow into the effect boundary. At minimum, retain the intent key, workflow and step IDs, provider, operation, request digest, status, attempt count, and timestamps. When a response is confirmed, append the provider object ID and a response digest rather than storing secrets or unnecessary personal data.

Model unknown separately from failure. A declined card, invalid recipient, or provider validation error is definitive: no effect occurred. A connection reset, deadline, or worker termination after dispatch is not definitive. Unknown should prevent automatic provider failover and route the run to a lookup or reconciliation handler. That handler can query by idempotency key, provider object ID, or domain attributes, then atomically record confirmed or rejected.

Use a unique database constraint on tenant plus intent key, and make every state change conditional on the prior state. Two workers may prepare or reconcile concurrently. Only one should create the command, and both should observe the same terminal receipt. Preserve the ledger after workflow completion for at least the provider dispute and retry window; deleting it with transient task data removes the evidence needed to explain a later duplicate report.

create table effect_intents (
  tenant_id text not null,
  intent_key text not null,
  request_digest text not null,
  status text not null,
  provider_result_id text,
  updated_at timestamptz not null,
  primary key (tenant_id, intent_key)
);

How does an operator reconcile an unknown effect?

The runbook should be executable without database editing. Provide a read view for evidence, a provider lookup action, and guarded controls to accept a confirmed receipt, mark a definitive rejection, or launch compensation. Log the operator identity, evidence reference, and reason for every resolution. For high-impact payments, require a second approver above a threshold.

Measure unknown-effect age, reconciliation success by lookup method, duplicate-prevention conflicts, and compensation rate. A rising unknown rate often signals network deadlines that are shorter than provider processing, a missing provider request ID, or workers losing leases during shutdown. Fixing those causes reduces manual work without weakening the safety boundary.

• Freeze automatic redispatch for the intent key and show the workflow, customer object, provider, amount or recipient, request digest, and last attempt time.
• Query the provider using its idempotency lookup first. If unavailable, search by provider request ID or a narrow set of domain attributes and time bounds.
• When one matching effect exists, record its provider ID and response digest, then resume the workflow from confirmed without calling the provider again.
• When the provider proves no effect occurred, record rejected or safe-to-retry according to the provider contract; reuse the original intent key if it is retained.
• When evidence remains ambiguous, keep the intent unknown and escalate. Do not turn uncertainty into success merely to unblock the queue.
• If the business chooses compensation, issue it as a new idempotent command with its own identity and link it to the original effect.

Which failure tests prove the integration?

Run tests against a provider sandbox or a deterministic fake that records every request. Inject termination before dispatch, immediately after dispatch, after provider acceptance, after response receipt, and during local receipt persistence. Repeat with two workers, a delayed response beyond the client deadline, provider 429 and 500 responses, and a retry after the advertised idempotency window.

For every case, assert both sides: one durable workflow outcome and the expected number of provider effects. Inspect the ledger state and provider request log, not only the final HTTP response. A test that ends in confirmed while the provider received two creates is a failure even if the application returned success once. Keep these tests in the release gate for provider SDK, timeout, worker, and database changes.