By Orch8 Engineering · Editorial policy
Webhook Deduplication for Durable Workflows: An Intake Pattern
Verify the provider signature against the raw request body, store the delivery identity and payload before acknowledging it, and start processing asynchronously. Enforce a unique event key so provider retries reuse one intake record. Treat a second business event about the same object separately, because delivery deduplication does not resolve out-of-order state changes or prevent duplicate downstream effects.
What is the practical answer?
A reliable webhook intake verifies the provider signature against the raw request body, persists the accepted event under a unique delivery key, and only then returns success. The business workflow runs asynchronously from that durable record, so a slow step does not cause the provider request to time out and retry. If the same event arrives again, a database uniqueness constraint maps it to the existing intake row and workflow rather than starting another run. A second event about the same object is a separate decision: provider event IDs protect delivery deduplication, while object ID plus event type or an authoritative resource read may be needed for business state. Stripe says events can be duplicated and delivered out of order. Track accepted-event age and dead letters after the 2xx response, because a healthy webhook response rate does not prove downstream processing completed.
Evidence: Stripe: receive webhook events · GitHub: webhook best practices · Orch8 Engine: webhook triggers
Acknowledge receipt, not completion of the whole workflow
A webhook provider wants a quick response. The workflow may take minutes, wait for a person, or survive a deployment. Running that work inside the HTTP request couples provider delivery to the slowest downstream step. If the request times out after the work started, the provider retries and can start another run.
The receiver should verify and durably record the event, then acknowledge it. A worker or workflow trigger processes the retained record after the response. The acknowledgment now means 'we own this event,' not 'every business effect succeeded.' Stripe recommends a quick 2xx and asynchronous handling; GitHub also recommends a timely response and queueing longer work.
| Receiver outcome | HTTP response | What happens next |
|---|---|---|
| Signature invalid | 4xx | No event is accepted |
| Event persisted | 2xx | Async processor starts or resumes work |
| Storage unavailable | 5xx | Provider can retry delivery |
| Known duplicate | 2xx | Return the prior accepted identity |
Verify the raw delivery before parsing business fields
Verify the provider's signature against the exact raw bytes and the endpoint secret. A framework that normalizes JSON before signature verification can change the signed bytes. Check the timestamp or replay window defined by that provider, use the intended secret for the endpoint, and reject oversized requests before expensive parsing. Keep secret values out of logs.
Do not invent one universal header contract. Stripe uses Stripe-Signature and requires the raw body; GitHub has its own signature and X-GitHub-Delivery identifier. Record the provider, endpoint, event ID, event type, and receipt time so a support engineer can find the original delivery without storing credentials in the event row.
Separate delivery deduplication from business deduplication
The same provider event can arrive twice. Give that delivery one unique key, normally provider plus account or endpoint plus event ID. The database must enforce uniqueness so concurrent requests cannot both start a workflow. A duplicate request returns success after finding the accepted record; it does not repeat processing.
Two distinct events can describe the same business object. Stripe notes that some duplicate business notifications have separate Event objects; the object ID plus event type may be the right second-level key for that situation. Decide the key from the operation's meaning. 'invoice.paid' and 'invoice.refunded' must not collapse into one invoice ID, while two deliveries of the same invoice.paid event should not create two fulfillment runs.
CREATE UNIQUE INDEX webhook_delivery_once
ON webhook_inbox (provider, account_id, event_id);
-- Insert the verified delivery and commit before replying 2xx.
-- On a uniqueness conflict, load the existing intake row.Never infer current state from arrival order
Providers may deliver events out of order. Stripe explicitly does not guarantee generation order in delivery and recommends retrieving missing objects where needed. An older subscription.updated event arriving after invoice.paid should not overwrite a more recent paid state merely because the webhook arrived last.
Make the workflow fetch authoritative current state when the event is a notification rather than a complete command. If applying deltas, keep a provider version or monotonic sequence and reject stale transitions. If neither is available, queue dependent events for reconciliation instead of guessing from timestamps. Record which event produced a state change so later replay is explainable.
Test the receiver and the worker as separate failure paths
Replay one signed delivery twice and concurrently. Both requests should acknowledge the same intake identity and only one workflow should start. Kill the receiver after the durable insert but before the response; the provider retry should return the already-accepted result. Then kill the worker after an external effect but before local completion and use the effect's own idempotency key to reconcile it.
Monitor invalid signatures, storage failures, accepted-event age, duplicate deliveries, processing retries, and dead-letter age separately. A 2xx rate can be healthy while the async processor is stalled. The operational promise is that every accepted event reaches a visible outcome: completed, waiting, retrying, or parked for review.
- Repeat the exact event ID.
- Send distinct events for one object out of order.
- Stop the process before and after the acknowledgment.
- Confirm one accepted run and one provider effect.
Sources and further reading
Official references support technical claims; community discussions are used only as problem signals.