Design webhook handling as an at-least-once, potentially out-of-order process: record each event, make its effect safe to repeat, and reconcile state after failures. This avoids relying on a sender to deliver each notification exactly once or in sequence. The right safeguards depend on whether an event carries a complete state snapshot or only announces that something changed.
A webhook is a notification, not a guaranteed sequence
Treat a webhook endpoint as a receiver of potentially repeated, delayed, or missed notifications—not as a perfectly ordered feed. The sender may retry after a timeout even if your system completed the work, and an outage can delay delivery. Your integration should identify events, apply them safely, and provide a way to detect and repair gaps.
The key design question is what each event means. Some events report a specific change; others say that a record now has a particular state. That distinction determines whether a delayed event can overwrite newer information or whether your system can simply fetch the latest state.
Separate receiving from business processing
Keep the public endpoint’s first job small: validate the request using the sender’s supported authentication method, check the payload shape, and durably record the event. Then return the acknowledgement expected by that sender. Process the recorded event separately so a slow database operation or downstream API does not unnecessarily hold open the request.
Persist an event identifier, receipt time, relevant resource identifier, payload or secure reference to it, and processing status. Store only what the workflow needs and protect it appropriately. Acknowledging before the event is safely recorded risks losing work; waiting for every downstream task can trigger avoidable retries.
Make duplicate delivery harmless
A sender may retry because it did not receive an acknowledgement, even when your application already acted. Use the sender’s stable event ID, where available, to detect a replay. Enforce uniqueness in durable storage rather than relying only on an in-memory check, which can fail when requests arrive simultaneously or reach different application instances.
Deduplication alone is not enough if processing can stop halfway through. Design the business operation to be idempotent: repeating it leads to the same intended state rather than creating a second order, payment action, or notification. For example, record that a particular event has been applied as part of a transaction with the related update, when the database and workflow allow it.
Protect current state from out-of-order events
Do not assume arrival order is the same as change order. If an event includes a trustworthy sequence number, version, or update timestamp, compare it with the value already stored and reject stale updates. Check the sender’s documentation to understand what that field guarantees; timestamps can have limited precision or represent different moments in different systems.
If the event only identifies a changed resource, fetch that resource’s current state from the source system before updating your own copy. This trades additional API calls and possible delay for less dependence on event ordering. For business-critical records, also define how conflicts are resolved and which system owns each field.
Retry transient failures without hiding permanent ones
A temporary network failure or unavailable dependency may justify a retry. A malformed payload, revoked permission, or invalid business value usually needs investigation or correction instead. Classify failures, record the reason, and use bounded retries with increasing delays rather than retrying continuously. Confirm the sender’s retry policy and acknowledgement rules because those vary by service.
Keep events that exhaust retries in a visible failure queue or equivalent work list. Provide an authorized way to inspect, correct, and replay them without bypassing deduplication safeguards. Monitor the age of the oldest unprocessed event and recurring failure categories, not just whether the endpoint is reachable.
Plan recovery for events that never arrive
Retries address known delivery failures; they cannot recover an event the sender never delivered. Where the source supports it, run periodic reconciliation: compare a bounded set of recently changed source records with local state, or request changes since a saved checkpoint. Define a time window that accounts for delayed updates, and make reconciliation safe to repeat.
Before release, test duplicate requests, reversed event order, a process crash after recording but before applying, a downstream outage, and recovery after the sender stops retrying. Check that records are neither silently skipped nor applied twice, permissions protect any replay mechanism, and logs let an operator trace an event from receipt through completion. If events are noncritical and the source can be queried later, a simple periodic sync may be more appropriate than a complex event pipeline.
Webhook reliability checklist
- Assign each event a stable identifier and record it before processing.
- Define whether events represent a change or a complete current-state snapshot.
- Make each business update safe to repeat, or detect duplicates before applying it.
- Reconcile event timestamps or versions against the state already stored.
- Keep failed events visible, retry transient failures, and alert on exhausted retries.
- Test duplicate, delayed, out-of-order, and outage-recovery scenarios before release.
Questions about this approach
Should every webhook handler fetch the source record?
Not necessarily. If an event identifies a resource but does not contain reliable current state, fetching the latest record can prevent a delayed notification from restoring old values. If events include trusted versions and complete snapshots, version checks may be enough. Follow the sender’s documented semantics and account for API limits and temporary read failures.
Is checking for a duplicate event ID enough?
Use the event ID for delivery deduplication, but also make the actual business change safe to repeat. A process may fail after changing a record but before marking the event complete. Store progress and related changes transactionally where practical, or use a stable business-operation key so retrying cannot create a second effect.
What if the sender does not retry missed events?
Keep a durable record of receipt, processing status, and failure reason; alert on persistent failures or growing delay. If the sender cannot replay missed events, use a source query or periodic reconciliation when available. Confirm that reconciliation can run repeatedly without overwriting newer local work.
For help applying this approach, explore our API Integration & Custom API Development services.
Discuss your project with Full Blown Studio
Tell us what needs to work better and what you have tried. Discuss your project, email bob@fullblown.com, or call (661) 429-0940.
