The Outbox Pattern in .NET Microservices: Publishing Events Without Losing Consistency With the Database

A service that processes an order usually has to do two things: save the change to its own database, and tell the rest of the system that change happened, by publishing an event to a queue or message bus. The problem shows up the moment those two steps are written as separate operations, because there is no transaction tying a SaveChangesAsync call to a PublishAsync call. If the process saves the order and then crashes, restarts, or simply loses its connection to the broker before publishing, the order is saved but the rest of the system never finds out. If the event is published first and the database write fails afterward, the event goes out describing something that was never actually persisted. Either way, the service's own state and the state the rest of the system sees drift apart, and no broker-side retry fixes that, because from the broker's point of view the message was delivered just fine.

Why a distributed transaction doesn't solve this

The obvious fix — wrapping both operations in a distributed transaction, with a two-phase commit coordinator spanning the database and the broker — is almost never workable in practice. Most of the message brokers we use in .NET projects (RabbitMQ, Azure Service Bus in its standard tier, SQS) don't participate in a two-phase protocol with SQL Server or PostgreSQL, and even if they did, the operational cost of keeping a distributed transaction coordinator alive and reliable far outweighs the problem it solves. The outbox pattern sidesteps that coordinator entirely by turning the event publication into just another write inside the same local transaction that already saves the order.

The outbox table

The core idea is simple: instead of publishing the event straight to the broker, you insert a row describing that event into an Outbox table in the same database, inside the same transaction that modifies the business aggregate. If the transaction commits, the event row commits along with the order, inside the same atomic unit the database engine already guarantees. If the transaction rolls back, the event never existed, exactly like the order.

public class OutboxMessage
{
    public Guid Id { get; set; }
    public string Type { get; set; } = default!;
    public string Payload { get; set; } = default!;
    public DateTime OccurredOnUtc { get; set; }
    public DateTime? ProcessedOnUtc { get; set; }
}

public async Task CreateOrderAsync(Order order, CancellationToken ct)
{
    using var transaction = await _context.Database.BeginTransactionAsync(ct);

    _context.Orders.Add(order);

    _context.OutboxMessages.Add(new OutboxMessage
    {
        Id = Guid.NewGuid(),
        Type = nameof(OrderCreated),
        Payload = JsonSerializer.Serialize(new OrderCreated(order.Id, order.CustomerId, order.Total)),
        OccurredOnUtc = DateTime.UtcNow
    });

    await _context.SaveChangesAsync(ct);
    await transaction.CommitAsync(ct);
}

With EF Core, you don't even need to open the transaction explicitly in most cases: a single SaveChangesAsync call that includes both the Order and the OutboxMessage in the same DbContext already runs inside one implicit transaction. What makes this work isn't the framework — it's the decision to treat "save the order" and "record that an event is pending" as the same database operation, never as two separate calls to two separate systems.

The background publisher

Once the event lives as a row in the database, you need a separate process to read it and publish it to the real broker. The simplest way to implement this in .NET is a BackgroundService that polls the table at short intervals, publishes the pending messages, and marks them as processed:

public class OutboxPublisher : BackgroundService
{
    private readonly IServiceScopeFactory _scopeFactory;
    private readonly IMessageBus _bus;

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            using var scope = _scopeFactory.CreateScope();
            var context = scope.ServiceProvider.GetRequiredService<AppDbContext>();

            var pending = await context.OutboxMessages
                .Where(m => m.ProcessedOnUtc == null)
                .OrderBy(m => m.OccurredOnUtc)
                .Take(50)
                .ToListAsync(stoppingToken);

            foreach (var message in pending)
            {
                await _bus.PublishAsync(message.Type, message.Payload, stoppingToken);
                message.ProcessedOnUtc = DateTime.UtcNow;
            }

            await context.SaveChangesAsync(stoppingToken);
            await Task.Delay(TimeSpan.FromSeconds(2), stoppingToken);
        }
    }
}

This publisher can fail halfway through a batch — the process restarts after publishing ten of fifty messages, before saving ProcessedOnUtc — and that's acceptable by design: the next pass will publish those ten messages again. The outbox pattern gives you at-least-once delivery, not exactly-once, which means every consumer of the event has to be idempotent against duplicates, typically by storing the id of each message it already processed and discarding repeats. Anyone who adopts outbox assuming it solves duplicates on the other end has only solved half the problem.

What didn't work: the publisher that fell behind

In a service with a notably higher write volume than the rest, the polling publisher started accumulating unprocessed messages faster than it could drain them. The symptom wasn't an error: events kept reaching consumers, but with a delay that grew steadily throughout the day, because each pass of the BackgroundService took longer than the Task.Delay interval allowed for, and passes effectively started overlapping in practice even though the code never ran them in parallel. The Take(50) that had looked like a reasonable safety measure in development turned out to be the bottleneck in production: with thousands of messages piled up, publishing in batches of fifty every two seconds was never going to keep pace with how fast events were being generated.

The fix wasn't to shorten the polling interval, which would only have added load on the table without addressing the underlying imbalance. It was two separate changes: raising the batch size to a number sized against the service's actual events-per-minute volume, and adding an explicit lag metric — the difference between DateTime.UtcNow and the OccurredOnUtc of the oldest unprocessed message — exposed as a gauge, to catch the publisher falling behind before some external consumer asked why an event had taken so long to arrive. Without that metric, the lag only surfaced when someone outside the team reported it, which is the worst possible time to find out.

Purging the table instead of letting it grow forever

A related mistake, less dramatic but just as real, is treating OutboxMessages as if it could grow forever. Once a message's publication is confirmed, it contributes nothing beyond serving as an audit trail for a bounded period. We kept a separate job, running far less often than the publisher, that deletes in batches any message whose ProcessedOnUtc is older than a fixed retention window. Without that job, the outbox table ends up as the largest table in the database, and every SELECT the publisher runs — even with the right index on ProcessedOnUtc — gets progressively more expensive simply because of the table's physical size.

What we check before signing off on an outbox implementation

Before considering an outbox implementation in a .NET service done, we verify that the insert into the outbox table happens inside the same transaction as the write to the business aggregate, never in a later call; that the publisher only marks a message as processed after confirming the publish, never before; that every consumer of the event is idempotent against retries, since the real guarantee is at-least-once; that there's a lag metric on the oldest unprocessed message, visible before any external consumer notices it; and that there's a purge job running independently of the publisher. When those conditions hold, a crash or a restart mid-batch stops being a message lost somewhere between two systems, and becomes exactly what the pattern promises: a message that will get published, sooner or later, because it never stopped being there.