Transactional outbox pattern explained

The dual-write failure modes

"Write to DB, then publish to queue" looks simple in a tutorial. In production it is not atomic. Four common failure shapes:

• DB commits, broker fails — order exists, warehouse never decrements stock, customer gets no confirmation email.
• Broker succeeds, DB rolls back — consumers ship products for an order that was never persisted.
• Process crashes between steps — partial state with no clear recovery path unless you build compensating logic by hand.
• Retries duplicate publishes — the HTTP client retries the broker call after a timeout; consumers see two identical events unless they deduplicate.

Distributed two-phase commit across Postgres and Kafka is theoretically possible but operationally rare — latency, operational complexity, and vendor support limits push most teams toward application-level patterns. The outbox is the most widely adopted: it leans on the database you already trust for ACID and treats the message broker as an eventually consistent fan-out layer.

How the transactional outbox works

Instead of calling the broker inside your request handler, you insert a row into an outbox_events table alongside your domain mutation — both in one SQL transaction. The outbox row stores everything a downstream consumer needs: event type, aggregate ID, JSON payload, and metadata for routing.

Typical outbox schema

A minimal table might include:

• id — monotonic primary key or UUID for deduplication
• aggregate_type / aggregate_id — which entity changed
• event_type — e.g. order.placed
• payload — JSON blob with the public event body
• created_at — insertion timestamp
• published_at — NULL until the relay marks it sent (optional but useful for monitoring lag)

Because the insert shares a transaction with INSERT INTO orders, either both land or neither does. The HTTP response can return success knowing the event is durably queued for delivery — not yet delivered, but guaranteed to be relayed unless the entire database is lost.

The relay worker

A separate process — the message relay or outbox publisher — reads unpublished rows and pushes them to Kafka, RabbitMQ, or SQS. After the broker acknowledges the publish, the relay sets published_at or deletes the row. If the relay crashes mid-batch, unpublished rows remain and the next poll retries them.

This introduces at-least-once delivery to the broker: a relay crash after publish but before marking the row can cause a duplicate message. That is expected — consumers must be idempotent or use deduplication keys on the event ID.

Polling relay vs change-data capture

Two mainstream ways to move rows from the outbox table to the broker:

Polling publisher

The relay runs SELECT * FROM outbox_events WHERE published_at IS NULL ORDER BY id LIMIT 100 FOR UPDATE SKIP LOCKED on a timer. SKIP LOCKED lets multiple relay instances work in parallel without double-processing the same row. After publishing, it updates or deletes the row in the same relay transaction.

Polling is simple, works on any SQL database, and is easy to reason about in staging. Downsides: added read load, publish latency tied to poll interval (often 100 ms–1 s), and you must index published_at or equivalent to keep scans cheap as the table grows.

Log-based CDC (Debezium and friends)

Change data capture tails the database write-ahead log (WAL in Postgres, binlog in MySQL) and streams row inserts to Kafka directly — no polling loop. Debezium connectors treat the outbox table like any other source table; a downstream Kafka Connect sink or custom transformer maps rows to topic messages.

CDC reduces latency to milliseconds and offloads read pressure from the application database, but adds operational machinery: replication slots, connector monitoring, schema registry compatibility, and careful handling of tombstone deletes if you remove rows after publish. Teams already running Kafka Connect for analytics often adopt CDC; smaller services usually start with polling and graduate when lag or load demands it.

Ordering, partitioning, and event contracts

Message brokers usually guarantee order within a partition, not globally. Route outbox events by aggregate_id as the partition key so all events for one order or one user arrive in sequence. Cross-aggregate ordering is generally unnecessary and expensive to enforce.

Version your event payloads. Consumers on an older schema should tolerate unknown fields; producers should not rename fields without a migration window. Store enough context in the outbox payload that consumers do not need to call back into the writer service for every field — but avoid dumping entire internal entity graphs that leak private columns.

For read models built from events, pair the outbox with event-driven architecture discipline: explicit event names, documented schemas, and contract tests that fail CI when a producer breaks a consumer assumption.

Outbox vs saga vs inbox

These patterns solve different layers of the same problem space:

• Transactional outbox — reliable publish of events from a single service after a local commit. One database, one transaction boundary.
• Saga pattern — coordinates multi-service workflows with compensating steps when a remote call fails. Each participating service may use its own outbox to announce local state changes.
• Inbox pattern — the consumer-side mirror: deduplicate incoming messages by storing processed message IDs before handling side effects. Outbox + inbox together give end-to-end exactly-once effect when both sides cooperate.

Do not confuse the outbox with two-phase commit across services. The outbox does not hold locks on remote databases; it only guarantees that this service's state change and this service's intent to notify others are atomic. Remote services still see events eventually and must handle duplicates and out-of-order edge cases within their partition guarantees.

Failure modes and operations

Production teams should monitor and runbook these scenarios:

• Relay lag — gap between created_at and published_at. Alert when p99 exceeds your SLA (e.g. 5 s for email, 30 s for analytics).
• Poison outbox rows — malformed JSON or oversized payloads that the broker rejects forever. Move to a quarantine table or dead-letter queue after N attempts; page on-call rather than infinite retry.
• Table bloat — if you delete rows after publish, vacuum/archival is cheap. If you retain for audit, partition by month and archive cold partitions.
• Schema migrations — adding a NOT NULL column to outbox_events during traffic requires expand-contract discipline, same as any hot table (see database migration strategies).
• Exactly-once illusion — marketing "exactly-once" without idempotent consumers is a bug waiting to happen. Document at-least-once on the wire and exactly-once processing in the consumer via inbox dedup.

Implementation sketch (PostgreSQL + polling)

Pseudocode for the write path inside your API handler:

BEGIN;
  INSERT INTO orders (id, customer_id, total) VALUES (...);
  INSERT INTO outbox_events (aggregate_id, event_type, payload)
    VALUES (order_id, 'order.placed', '{"orderId":"...","total":99.00}');
COMMIT;
-- HTTP 201 returned; relay will publish asynchronously

The relay loop (simplified):

rows = db.query("SELECT id, payload, event_type FROM outbox_events
                   WHERE published_at IS NULL
                   ORDER BY id LIMIT 50 FOR UPDATE SKIP LOCKED");
FOR row IN rows:
  kafka.publish(topic_for(row.event_type), key=row.aggregate_id, body=row.payload);
  db.execute("UPDATE outbox_events SET published_at = now() WHERE id = ?", row.id);

Wrap each row's publish + mark in a try/catch; on broker failure, leave published_at null so the next poll retries. Use exponential backoff on repeated broker outages to avoid hammering a degraded cluster.

Production checklist

1. Never call the message broker inside the same request without an outbox or inbox safety net.
2. Insert outbox rows in the same transaction as domain mutations — verify with integration tests that rollbacks remove both.
3. Assign stable event IDs; consumers deduplicate on that ID.
4. Partition by aggregate ID to preserve per-entity ordering.
5. Monitor relay lag, outbox table depth, and publish error rates.
6. Define a max-retry policy and DLQ path for permanently failing rows.
7. Index (published_at, id) or partial index on unpublished rows.
8. Load-test relay throughput before Black Friday, not after.
9. Document which events are public contracts vs internal signals.
10. Re-evaluate CDC when polling latency or DB read load becomes a bottleneck.

Key takeaways

• Dual writes are not atomic — separate DB and broker calls can diverge on any failure.
• The outbox makes publish intent durable — same transaction as your business data.
• Delivery is at-least-once — design idempotent consumers and consider the inbox pattern on the receiving side.
• Polling is fine to start; CDC scales further — match complexity to traffic and team maturity.
• Outbox is one piece of the puzzle — sagas coordinate cross-service flows; DLQs handle poison messages.

Related reading

• Message queues explained — Kafka, RabbitMQ, and SQS delivery guarantees that the outbox feeds
• Saga pattern explained — multi-service workflows where each step may write its own outbox events
• Idempotency explained — deduplication keys and safe retries when relays publish duplicates
• Dead letter queues explained — quarantine poison outbox publishes and failed consumer handling