How idempotency keys and safe retries work

Idempotency in one sentence

In mathematics, applying a function twice equals applying it once: f(f(x)) = f(x). In distributed systems, an operation is idempotent if repeating it after a successful commit does not create additional durable effects — charging twice, shipping two boxes, or inserting duplicate rows.

Retries are not optional. TCP reconnects, HTTP clients time out, load balancers reset idle connections, and message brokers redeliver when consumers crash mid-ack. The question is never whether a request will be attempted twice; it is whether the second attempt is harmless. For HTTP verb semantics and API surface design, see our REST API design guide; here we focus on the reliability mechanics underneath.

Natural idempotency vs manufactured idempotency

Some operations are idempotent by definition:

• GET, HEAD, OPTIONS — read-only; repeating does not mutate server state (ignoring side effects like audit logs).
• PUT with a full replacement — setting /users/42 to the same JSON twice leaves the resource in the same final state.
• DELETE — deleting an already-deleted resource should return success (often 204 or 404) without error storms.

POST is the troublemaker. It creates new resources and triggers side effects — charges, emails, on-chain transfers. POST is not idempotent unless you add machinery. That machinery usually takes one of two shapes:

• Client-supplied idempotency keys — a unique token per logical operation, sent on every attempt of that operation.
• Natural keys from the domain — payment intent IDs, order numbers, blockchain transaction signatures, or webhook event IDs that uniquely identify the business action.

Natural keys are preferable when the domain already has a stable identifier. Manufactured keys (Idempotency-Key: uuid) are necessary when the client generates a new resource ID only after the server responds.

The Idempotency-Key header pattern

Stripe popularized the pattern: on POST requests that create charges or customers, the client sends Idempotency-Key: <opaque-string>. The server maintains a deduplication table keyed by (account_id, idempotency_key) with a TTL (often 24 hours).

On each request:

1. Lookup — if the key exists and processing finished, return the stored HTTP status and response body verbatim (including the same resource IDs).
2. In-flight lock — if another worker is processing the same key, return 409 Conflict or block until completion (Stripe returns the cached response once ready).
3. First attempt — insert the key in a processing state inside a transaction, execute the side effect, commit the result, mark the key completed, store the serialized response.

Critical detail: the key must be scoped to the authenticated principal. Reusing the same UUID across different API keys or user accounts must not suppress someone else's operation. Keys should be unguessable (UUID v4 or similar) — predictable keys enable cross-user collision attacks on poorly scoped stores.

Store enough to replay accurately: status code, headers that affect clients (especially Location), and JSON body. Partial failures are the hard case: if the charge succeeded but the response write crashed, the retry must return the charge ID, not create a second charge.

At-least-once delivery and idempotent consumers

Message queues — Kafka, RabbitMQ, SQS — typically guarantee at-least-once delivery: a message may arrive more than once, but it will not silently disappear if the consumer crashes before acknowledging. "Exactly-once" end-to-end is marketing unless you narrow the scope to a single broker with transactional reads and writes in one product (and even then, cross-service effects break the illusion).

The engineering goal is effectively-once: duplicates may arrive, but processing them twice does not double the business outcome. Patterns:

• Idempotent consumer table — before handling event evt_abc, insert evt_abc into a processed_events table with a unique constraint. The second insert fails; skip work.
• Upsert with monotonic version — apply updates only if incoming_version > stored_version.
• Outbox + single writer — one service owns the aggregate; events are projections of state that can be replayed safely.

For webhook delivery specifically — HMAC signatures, retry schedules, and handler design — see our webhooks guide. Webhook providers include a unique event ID in every payload; your handler should treat that ID as the natural idempotency key.

Database-level idempotency

Relational databases offer primitives that pair well with retries:

• Unique constraints — UNIQUE(order_id) turns a duplicate insert into a deterministic error you can catch and map to success.
• INSERT ... ON CONFLICT DO NOTHING (Postgres) or equivalent upsert — second insert is a no-op.
• Transactions — atomicity ensures you do not mark an event processed without committing the side effect. For isolation pitfalls under concurrency, see ACID transactions explained.

Combine application-level keys with database constraints. The dedup table catches fast replays; the unique constraint catches races where two workers process the same key simultaneously before either commits.

Payments, blockchains, and external idempotency

Card networks and crypto rails add their own layers:

• Payment intents — create an intent ID client-side or server-side first; all capture attempts reference that ID. Processors dedupe on their side too, but you cannot rely on that alone.
• On-chain transfers — a Solana transaction signature is a natural idempotency token. Submitting the same signed transaction twice does not move funds twice; the chain rejects duplicate signatures. Server-side fulfillment should still record tx_signature uniquely before shipping digital goods.
• External API calls — wrap third-party POSTs in your own idempotency record. If Stripe already dedupes but your inventory service does not, you can still oversell.

Time windows matter: idempotency keys expire. A retry three days later may legitimately be a new purchase — document TTL behavior and return 422 when a stale key is reused with a different request body.

Retries without amplification

Idempotency solves duplicate effects; backoff solves duplicate load. Pair safe handlers with disciplined retry policy:

• Retry only idempotent operations or requests carrying idempotency keys.
• Use exponential backoff with jitter on 429 and 503.
• Cap maximum attempts; surface failure to the user instead of infinite loops.
• Do not retry 400 validation errors — the body will not magically become valid.

When dependencies are unhealthy, raw retries become storms. Circuit breakers stop hammering sick downstream services; idempotency keys ensure the requests that do get through are safe to repeat. Rate limits (see rate limiting algorithms) protect capacity; idempotency protects correctness.

Common pitfalls

• Key reuse across different payloads — same key, different amount or recipient must be rejected, not silently deduped to the wrong operation.
• Non-atomic check-and-act — read key, then charge, without locking allows two workers to both pass the check. Use transactions or compare-and-set.
• Storing only success — if the first attempt failed validation, retries with the same key should return the same error, not hit a clean slate.
• Side effects outside the transaction — sending email inside the charge handler without dedup means retries spam users even when billing is idempotent.
• Assuming GET is always safe — rare APIs use GET for mutations (legacy payment gateways). Do not copy that pattern.

Practical checklist

• Identify every POST or async handler that moves money, inventory, or irreversible state.
• Accept Idempotency-Key (or use domain natural keys) on those paths.
• Persist key, request fingerprint, status, and response with a scoped unique index.
• Make message consumers dedupe on event ID before side effects.
• Add database unique constraints as a backstop against races.
• Document TTL, error replay behavior, and retry guidance for API clients.
• Load-test concurrent duplicate requests — the race window is where bugs hide.

Related on Solana Garden: REST API design guide, Webhooks guide, ACID transactions explained, Circuit breakers explained, Explainers hub.