Transactional Outbox Pattern: solving the Dual Write problem when saving to a DB and publishing eventsNote · digest for November 3
Clear infographic diagram of the Transactional Outbox pattern showing a local ACID transaction and reliable delivery to a message broker.
Note2 sections≈ 4 min

Transactional Outbox Pattern: solving the Dual Write problem when saving to a DB and publishing events

If you design microservices or distributed systems, you've almost certainly written code like this: inside an HTTP request handler, a service first saves an order entity to a relational database and then publishes an OrderCreated notification to an Apache Kafka topic or RabbitMQ broker.

On a developer's local machine, that naive pairing works flawlessly. In a real distributed environment, however, you inevitably run straight into the classic Dual Write dilemma.

Networks between services are unreliable by nature:

  • Scenario A (database first, broker second): the database transaction commits cleanly, but when trying to send the message to Kafka, the network blips or the container crashes. The result is catastrophic: the order exists in the database, yet downstream inventory, billing, and shipping services will never learn about it. Data is hopelessly out of sync.
  • Scenario B (broker first, database second): the service publishes the event to the bus, then fails to commit in the DB due to a foreign key violation or serialization conflict. The result: the broker has already notified the entire system about a nonexistent order (a phantom event), inventory reserved the item, and there is no record in the database.

Distributed two-phase commits (2PC / XA) across heterogeneous systems are either unsupported by modern brokers or destroy service throughput. The standard solution to this problem is the Transactional Outbox Pattern.

A local ACID transaction instead of distributed unreliability

The idea behind the Outbox pattern is brilliant in its simplicity: since we can't guarantee atomicity between a database and an external broker, let's move saving the event inside that very same relational database!

A utility outbox table is added to the schema:

-- DDL for the outbox events utility table in PostgreSQL
CREATE TABLE outbox (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    aggregate_type VARCHAR(64) NOT NULL,
    aggregate_id VARCHAR(64) NOT NULL,
    event_type VARCHAR(64) NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    processed_at TIMESTAMP WITH TIME ZONE
);

CREATE INDEX idx_outbox_unprocessed ON outbox (created_at) WHERE processed_at IS NULL;

Now writing the business entity and recording the event happen inside a single local transaction:

BEGIN;

-- 1. Persist the business entity
INSERT INTO orders (id, user_id, amount, status)
VALUES ('ord_12345', 'usr_999', 1500.00, 'PAID');

-- 2. Atomically record the outgoing event in the same transaction
INSERT INTO outbox (aggregate_type, aggregate_id, event_type, payload)
VALUES (
    'Order',
    'ord_12345',
    'OrderPaid',
    jsonb_build_object('orderId', 'ord_12345', 'amount', 1500.00, 'userId', 'usr_999')
);

COMMIT;

Thanks to the ACID guarantees of a relational DBMS, desynchronization is physically impossible: either both the order and the event row appear in the database, or both writes roll back.

Delivering events: polling vs. reading the WAL with Debezium

All that remains is delivering accumulated rows from outbox to the external broker. There are two paths here:

  1. Polling Publisher: a background worker runs every 200–500 milliseconds, executes SELECT ... FROM outbox WHERE processed_at IS NULL ORDER BY created_at FOR UPDATE SKIP LOCKED LIMIT 100, sends the batch to the broker, and marks it processed. The SKIP LOCKED pattern lets you run multiple parallel workers without lock contention. Under heavy throughput, however, constant polling creates parasitic load on the disk subsystem.
  2. Transaction Log Tailing (Change Data Capture / CDC): a more mature solution built on specialized platforms like Debezium. The Debezium connector reads the Write-Ahead Log (WAL in PostgreSQL) directly and asynchronously streams events into Kafka topics with sub-millisecond latency and zero load on the SQL engine.

Combined with practices for building distributed queues on Apache Kafka, which we covered earlier, this pipeline guarantees at-least-once delivery.

Mandatory consumer idempotence

Because a worker can resend a message during network failures, consumers on the receiving side must check the unique event_id (for instance, in Redis or via a unique index on an inbox events table). Without idempotence on the receiver side, at-least-once delivery will inevitably lead to duplicate actions.

Adopting Transactional Outbox requires schema design discipline, but it permanently rids a microservice architecture of phantom records and lost transactions.

Want a deep diveVotes help us choose what to break down in detail. This material has 0 votes.
Sources · 3