How it works
A consumer registers an HTTPS endpoint and subscribes to event types, such as payment.succeeded. Store the subscription, tenant, endpoint, signing secret, and enabled status; define whether events contain a resource snapshot or an ID the consumer must fetch.
Create a durable event when the business change commits. For a payment update stored in a relational database, write an outbox record in the same transaction, then publish it asynchronously so a crash cannot leave a committed payment with no notification event.
Resolve subscriptions and create durable delivery records for each event-endpoint pair. Workers send HTTP POST requests containing a stable event ID, event type, creation time, schema version, and payload; track attempts separately from event identity.
Treat an agreed success response, usually HTTP 2xx, as acknowledgment of receipt, not proof that downstream business work finished. Receivers should verify the request, durably accept it, and respond quickly before doing expensive processing.
Retry transient failures and timeouts with exponential backoff and jitter, honoring Retry-After where applicable. A timeout is ambiguous: the receiver may already have accepted the event, so retries keep the same event ID and receivers deduplicate it atomically with their local state change.
Authenticate requests with HTTPS and a signature over the timestamp and raw body, using a per-endpoint secret. Enforce timestamp freshness and support secret rotation; protect outbound workers against SSRF by validating destinations and resolved addresses at connection time and disabling redirects or validating every redirect target.
A worked example
A payment webhook with a lost acknowledgment
Suppose a payment service produces 100 events per second, each matching three merchant endpoints: 300 initial deliveries per second before retries. It commits payment p42 and outbox event e17 together. A worker posts e17 to a merchant, which atomically records e17 and updates its order before returning 200. The response is lost, so the sender retries e17 after a backoff. The merchant finds the recorded ID and returns 200 without updating the order again. A different endpoint remains unavailable; its retries use a bounded concurrency budget so healthy merchants keep receiving events. After the retry window expires, the sender retains a failed delivery for inspection and replay using the same event ID.
Trade-offs to say out loud
- Less polling, more delivery machinery
- Push avoids repeated empty reads, but the sender now owns queues, retry scheduling, endpoint health, and delivery history.
- Duplicate-tolerant delivery, not exactly-once effects
- Retries improve delivery chances within a bounded retry window, but duplicate requests remain possible and external side effects require their own idempotency controls.
- Ordering versus throughput
- Concurrent sends and retries can reorder events, while serializing delivery per resource limits throughput and lets one failed event block later updates.
- Recovery versus resource cost
- Long retry windows and retained payloads help consumers recover from outages but increase storage, bandwidth, and sensitive-data exposure.
- Shared capacity versus tenant isolation
- A shared worker pool is efficient, but per-endpoint concurrency limits and tenant quotas are needed to keep slow consumers from delaying everyone else.
- Rich payloads versus coupling
- Snapshots reduce follow-up API calls but require schema versioning and may be stale by arrival, while ID-only events add read load and depend on resource availability.
When to reach for it
- An external service needs event notifications and can expose an HTTPS endpoint.
- Payment, payout, or provider outcomes arrive asynchronously after the original request has returned.
- Consumers need relatively prompt updates without repeatedly polling an API.
- A notification service must deliver events to customer-controlled endpoints that have independent latency and availability.
How to bring it up in the interview
- Start with the contract: event types, payload semantics, acknowledgment meaning, retry window, ordering scope, and replay behavior.
- Draw the durable path from business transaction to outbox, delivery records, workers, and receiver; expect the interviewer to ask what happens at each crash boundary.
- Explain the lost-response case explicitly: the receiver commits, its response disappears, and the sender retries; defend deduplication with a stable event ID and an atomic local transaction.
- Describe per-endpoint timeouts, concurrency limits, backoff, and a dead-letter or failed-delivery store; expect questions about a customer endpoint being down for a day.
- Include security and operations: signature verification, SSRF defenses, secret rotation, delivery-age metrics, attempt logs, and controlled replay after an outage.
Common mistakes
- Sending the webhook inside the business request, making the transaction depend on a customer endpoint's latency or availability.
- Treating a timeout as proof of failure or promising exactly-once delivery over HTTP.
- Returning 2xx before durable acceptance, allowing a receiver crash to lose an acknowledged event.
- Assuming arrival order matches business order instead of using resource versions, valid state transitions, or reconciliation.
- Accepting arbitrary destination URLs without SSRF controls, or checking signatures against reserialized JSON instead of the raw request body.
Where it comes up
Interview questions where webhooks is part of a strong answer, and why.
- Design a notification systemWebhook delivery is an external notification channel that needs durable dispatch, endpoint isolation, retries, and duplicate handling.
- Design a payment systemAsynchronous payment-provider callbacks must be authenticated, deduplicated, and reconciled with payment state after uncertain outcomes.
- Design a hotel booking systemA delayed or duplicated payment callback can race with hold expiration, requiring explicit booking transitions and compensation rather than blindly confirming inventory.
Questions people ask
What is the difference between a webhook and an API?
A webhook uses an HTTP API endpoint, but reverses who initiates communication: the event producer calls the consumer when something happens. With polling, the consumer repeatedly calls the producer to discover changes.
How do you make webhooks reliable?
Persist events and delivery state, close the database-to-queue gap with an outbox or equivalent mechanism, and retry transient failures with backoff. Receivers must durably accept events before acknowledgment and handle duplicates. Provide failed-delivery inspection, replay, and reconciliation because a bounded retry policy cannot guarantee eventual delivery to an unavailable endpoint.
How do you handle duplicate or out-of-order webhooks?
Deduplicate using a stable event ID scoped to the sender or subscription, recording it atomically with the local business update. Use resource versions or validated state transitions to avoid applying stale updates. If an event exposes a gap or ambiguous state, fetch authoritative state or run reconciliation rather than trusting arrival order.