Guide

Webhooks explained: signatures, retries, and idempotency

Your payment processor knows a charge succeeded before your database does. A GitHub push lands in a repo before your CI runner notices. Instead of asking every few seconds "anything new yet?", modern platforms push events to you — that push is a webhook. This guide explains how webhook delivery works, how to verify that a request really came from the sender, and how to write handlers that survive retries without shipping the same order twice or crediting a wallet twice.

Polling vs webhooks: two ways to learn about events

In a polling model, your server repeatedly calls an API: GET /orders?since=timestamp every 30 seconds, or getSignatureStatuses on a Solana RPC until a transaction finalizes. Polling is simple to reason about — you control the schedule — but it wastes requests when nothing changed and adds latency when something did.

A webhook inverts the direction. When an event occurs (payment captured, pull request opened, block finalized), the provider sends an HTTP POST to a URL you register. Your endpoint receives a JSON payload describing what happened. You respond with 2xx to acknowledge receipt; anything else tells the sender to try again later.

Webhooks shine when events are infrequent but time-sensitive: confirming an on-chain deposit, unlocking a digital good, or kicking off a deploy pipeline. They are less ideal when you need a guaranteed chronological stream of every state change — providers can drop, reorder, or duplicate deliveries. Production systems almost always combine webhooks with occasional reconciliation polling as a safety net.

Anatomy of a webhook request

Although every vendor formats payloads differently, most webhook POSTs share the same skeleton:

HTTP method — almost always POST.
Content-Type — typically application/json.
Body — a JSON object with an event type, resource IDs, and metadata.
Signature header — proves the body was not tampered with in transit.
Delivery ID — a unique identifier for this specific delivery attempt.

Stripe sends Stripe-Signature with a timestamp and HMAC. GitHub sends X-Hub-Signature-256. Shopify, Twilio, and Slack each have their own header names, but the underlying idea is the same: you recompute a hash of the raw request body with a shared secret and compare it to the header value using a constant-time comparison function.

// Pseudocode: verify an HMAC webhook signature
const rawBody = request.rawBytes;  // must be exact bytes received
const expected = hmacSha256(WEBHOOK_SECRET, rawBody);
if (!timingSafeEqual(expected, request.header('X-Signature'))) {
  return response(401, 'invalid signature');
}

Two common mistakes here: parsing JSON before verifying (whitespace changes break the hash), and comparing strings with === instead of a timing-safe compare, which can leak information to an attacker byte by byte.

Registering and securing your endpoint

You register a webhook URL in the provider's dashboard: something like https://api.example.com/webhooks/stripe. That URL must be reachable over the public internet (or through a tunnel like ngrok during development) and must terminate TLS — providers refuse plain HTTP in production.

Secrets and rotation

The provider gives you a signing secret when you create the endpoint. Store it in environment variables or a secrets manager, not in source control. When you rotate secrets, support both old and new during a overlap window so in-flight deliveries still verify.

Replay protection

A captured webhook could be replayed by an attacker. Stripe embeds a timestamp in the signature; reject requests older than five minutes. Track seen delivery IDs in a short-TTL cache (Redis works well) and drop duplicates before your business logic runs.

Firewall and IP allowlists

Signature verification is the primary defense. IP allowlists are a secondary layer — useful but brittle, because provider IP ranges change. Never skip signature checks just because the request came from a "known" address.

Delivery guarantees: at-least-once, not exactly-once

Webhook systems are at-least-once delivery. If your server returns 500, times out, or is down during maintenance, the sender retries — often with exponential backoff over hours or days. The same payment_intent.succeeded event can arrive two, three, or ten times.

That means your handler must be idempotent: processing the same event twice must leave the system in the same state as processing it once. The standard pattern:

Extract a stable event ID from the payload (e.g. evt_abc123).
Begin a database transaction.
Insert the event ID into a processed_webhooks table with a unique constraint.
If the insert conflicts, commit nothing and return 200 — you already handled it.
Otherwise, run business logic (fulfill order, credit balance) and commit.

Respond with 2xx only after the event is durably recorded. If you acknowledge before writing to the database and the process crashes, you lose the event forever. If you write first and crash before responding, the sender retries — which is fine because idempotency catches the duplicate.

Retries, timeouts, and backpressure

Senders expect a response within a few seconds. Heavy work — generating a PDF, calling a slow third-party API, minting an NFT — should not run inline in the webhook handler. Instead:

Verify signature and persist the raw event.
Enqueue a background job (Sidekiq, BullMQ, SQS, etc.).
Return 200 immediately.
Let workers process at a sustainable rate.

This pattern also protects you from retry storms. If processing takes 45 seconds and the sender times out at 30, every attempt looks like a failure and retries multiply. Fast acknowledgement plus async processing is the norm at scale.

When your queue backs up, resist the temptation to return 503 to slow down the sender unless you have no other option — most providers keep retrying aggressively. Prefer absorbing events into durable storage and draining the queue when capacity returns. If you must shed load, use documented pause/disable controls in the provider dashboard rather than returning errors that trigger exponential retry ladders.

Webhooks in crypto and on-chain payments

Blockchains do not natively "call your server" when a transaction confirms. You bridge the gap in one of three ways:

Indexer webhooks — services like Helius or QuickNode can POST when an address receives funds or a program emits a log.
Self-hosted listeners — your process subscribes to WebSocket accountSubscribe or polls RPC, then fans out internal events.
Payment processor webhooks — fiat on-ramps and custodial APIs notify you when a user-funded wallet is ready.

On-chain confirmation has its own timing semantics: processed, confirmed, and finalized commitment levels mean different risk profiles. A webhook fired at processed can still be reverted in a minority fork. For anything irreversible — shipping goods, unlocking paid features — wait for finalized or run your own reconciliation against the canonical chain state.

Memo fields, unique payment amounts, and reference public keys are common ways to match an incoming transfer to an order without trusting client-side claims. Treat the webhook as a hint to investigate, not as proof by itself — always verify the transaction on-chain before fulfilling.

Testing and debugging webhooks locally

Local development needs a public URL. Tools like ngrok, Cloudflare Tunnel, or stripe listen --forward-to localhost:3000/webhooks proxy external POSTs to your laptop. Log the raw body before parsing so you can replay events in unit tests.

Most providers offer a "send test event" button and a delivery log showing HTTP status codes, response bodies, and retry schedules. When a webhook fails in production, that log is your first stop — not your application logs.

Checklist before going live

Signature verification on the raw body, with timing-safe compare.
Idempotency table or cache keyed by event/delivery ID.
Fast 2xx after durable write; heavy work in a queue.
Reconciliation job that polls for missed events (daily is enough for many apps).
Alerting when delivery failure rate spikes in the provider dashboard.

Webhooks vs WebSockets and SSE

Webhooks are server-to-server: the provider's infrastructure calls yours. They work when your endpoint has a stable public URL and you do not need sub-second delivery to a browser tab.

WebSockets and server-sent events (SSE) are for server-to-browser (or server-to-server persistent) streams — live price ticks, chat messages, game state. They complement webhooks: the webhook updates your database; the WebSocket pushes the fresh state to connected clients. Choosing between them is a question of who initiates the connection and whether the consumer is always online.