Transactional Outbox: Close the Database-Queue Gap

A transactional outbox addresses a specific failure: your application commits a database change but crashes before publishing the corresponding event. Reversing the order creates the opposite problem, where an event describes a change that never commits.

Suppose an order service saves an order and tells a fulfillment worker to prepare it. I would begin by writing down which state is authoritative, what must happen after commit, and how long downstream processing may lag. That contract is more useful than starting with a message broker diagram.

Put the intent beside the business change

The transactional outbox pattern stores the outgoing message in the same database transaction as the business update. A separate relay publishes it. The pattern's original description also calls out duplicate publication and the need for idempotent consumers. See Transactional outbox.

For a proposed order service, an illustrative transaction is:

BEGIN;
INSERT INTO orders (id, customer_id, status)
VALUES ($1, $2, 'accepted');

INSERT INTO outbox (event_id, aggregate_id, event_type, payload)
VALUES ($3, $1, 'order.accepted', $4);
COMMIT;

The example omits schema definitions, validation, and application parameter binding. Its point is the commit boundary: either the order and publication intent are both committed, or neither is.

Keep the outbox payload intentionally small. I would include the event identity, order identity, event schema version, and fields the consumer actually needs. Copying an entire customer record into every event creates unnecessary retention and compatibility obligations.

The relay introduces another recoverable boundary

A polling relay might claim pending rows, publish them, and record successful publication. A change-data-capture relay can instead observe committed database changes. Debezium's Outbox Event Router documents an outbox convention using event IDs, aggregate fields, and payload routing.

Neither choice removes the need to explain acknowledgement. Imagine the broker accepts an event, but the relay crashes before saving its published marker. On restart, it may publish again. Marking the row before publication merely trades that duplicate possibility for potential loss.

My proposed contract would permit duplicate transport while requiring the fulfillment consumer to deduplicate by event ID and protect the business action by order state. Those are two related protections: one handles repeated messages, the other handles distinct messages that would otherwise repeat fulfillment.

Make the relay's state machine explicit

For a polling design, I would use states such as pending, claimed, and published, with an expiring claim or lease. A crashed worker must not hold an event forever. The claim also needs an owner identity so an expired worker cannot blindly mark work completed after a replacement has taken over.

This is application design guidance, not a full implementation recipe. Locking, lease renewal, broker acknowledgements, and transaction boundaries need to be specified together. A few SQL lines cannot prove a correct distributed relay.

A useful rehearsal is:

1. Commit five orders while the broker is unavailable.
2. Verify that five publication intents remain recoverable.
3. Restore the broker and stop the relay after an acknowledgement.
4. Restart it and inspect duplicate handling.
5. Confirm that each order reaches one intended fulfillment state.

The experiment should validate business outcomes and audit records, not merely count consumed messages. A consumer can acknowledge a message without completing the intended work.

Define ordering at the narrowest useful scope

For an order, “accepted” should not be interpreted after a later “cancelled” event in a way that reactivates fulfillment. I would assign an explicit aggregate version and have consumers reject or reconcile impossible transitions.

Global ordering across every order is often a stronger requirement than the product needs. For this example, ordering per order is enough. That scope should inform broker partitioning or routing, while the consumer still handles duplicates and delayed delivery.

If the relay processes rows concurrently, a creation timestamp alone is an unconvincing ordering guarantee. Document how related events are sequenced and what happens when one event repeatedly fails. Otherwise a poison event can either block unrelated work or be skipped without anyone realizing that the sequence is incomplete.

Treat backlog as a user-visible delay

An outbox that grows indefinitely is a failing system even if every API request returns success. I would monitor the age of the oldest pending event, publish attempts, consumer lag where available, and the number of events requiring intervention.

Choose alerts from product expectations. If an order must appear in fulfillment within an agreed interval, backlog age matters more than raw row count. Ten old events can be more urgent than thousands of recent events during a legitimate burst.

Plan cleanup after publication with enough retention for audit and replay needs. Keep replay behavior version-aware: an old payload may outlive the code that originally produced it. A schema version field helps route that decision but does not implement compatibility by itself.

Use the webhook crash-window review when designing the consumer side of the workflow. Account for relay database use in the connection-pool budget so reliable delivery does not monopolize application capacity.

I would adopt an outbox when losing the publication intent is unacceptable and the team can own the relay, consumer deduplication, and backlog recovery. The extra table is the easy part; the recoverable workflow is the feature.