Dead-Letter Queues in SQS: How We Design Retries Before Losing a Message

When we wire SQS to Lambda for asynchronous event processing, the first version almost always works fine in testing and fails in a very specific way once it hits production: a message that can't be processed — a malformed payload, a downstream service that's down, an exception nobody accounted for — doesn't just disappear. SQS redelivers it, the function fails again, and the cycle repeats indefinitely. On a FIFO queue that message can block the entire group behind it; on a standard queue it won't block ordering, but it keeps eating invocations, concurrency, and retry budget that healthy messages should be using instead. A dead-letter queue (DLQ) doesn't stop a message from failing. What it does is stop a message that fails permanently from competing forever with the ones that can actually be processed.

The setting you have to sync first: visibility timeout

Before touching the DLQ at all, there's an earlier setting that, if misconfigured, makes everything downstream fail in confusing ways: the queue's visibility timeout. When Lambda picks up a message, SQS hides it from other consumers for that window, waiting for the function to confirm processing by deleting it. If the visibility timeout is shorter than the time the function actually needs, SQS makes the message visible again before the original run finishes, and a second invocation can pick up the same message while the first one is still executing. AWS recommends setting the queue's visibility timeout to at least six times the Lambda function's configured timeout, to leave room for the event source mapping's internal retries without overlapping invocations on the same message.

const queue = new sqs.Queue(this, 'OrdersQueue', {
  visibilityTimeout: cdk.Duration.seconds(180), // Lambda timeout: 30s
  deadLetterQueue: {
    queue: dlq,
    maxReceiveCount: 5,
  },
});

maxReceiveCount decides when a retry stops being a retry and becomes a problem

The redrive policy that routes messages to the DLQ is defined by maxReceiveCount: how many times SQS can deliver a message without it being deleted before moving it to the secondary queue automatically. Picking that number is not a minor detail. Set it too low (one or two) and you send messages to the DLQ that would have succeeded on the third attempt — say, when the failure is a transient timeout against an external API that recovers on its own within seconds. Set it too high (past ten) and broken messages keep circulating in the main queue for a long time, firing the same error alarm over and over for the same message.

The distinction we use to pick the number is whether the failure is transient or deterministic. A network timeout, a downstream rate limit, a service that's mid-restart — these usually resolve on their own and justify several retries with backoff. A payload that doesn't match the expected schema, an ID that doesn't exist in the database, a plain programming exception — these don't get fixed by retrying, so insisting many times just delays the inevitable and makes it harder to find the root cause once the message finally lands in the DLQ surrounded by a hundred identical attempts.

batchItemFailures: don't let one bad message take down the whole batch

When Lambda's event source mapping uses a batch size greater than one, an unhandled error on any message in the batch makes Lambda report the entire batch as failed, and SQS retries every message in it — including the ones that were processed correctly. Reporting partial failures avoids that:

def handler(event, context):
    failures = []
    for record in event['Records']:
        try:
            process(record['body'])
        except Exception:
            failures.append({'itemIdentifier': record['messageId']})
    return {'batchItemFailures': failures}

With functionResponseType set to ReportBatchItemFailures on the event source mapping, SQS only redelivers the messages whose messageId shows up in failures. Everything else is considered handled. Without this, one problematic message inside a batch of ten causes the other nine to be reprocessed as well, risking duplicate side effects if those nine aren't idempotent.

Idempotency isn't optional, even when nothing is failing

Standard SQS delivers messages at least once, not exactly once. Even with zero application errors, the same message can still be processed twice: if the function runs close to the visibility timeout limit, or if there's a network hiccup between Lambda and SQS right after processing but before the delete confirmation goes through. Designing the handler so that processing the same message twice doesn't duplicate the effect — by recording the message identifier before running the business operation, or by using a domain-level idempotency key when SQS's messageId isn't enough because the producer resends the same event under a different messageId — stops being an optimization and becomes a requirement, because the queue's delivery guarantee already comes with that condition baked in.

What we monitor on the DLQ

An unmonitored DLQ is a lost message with a different name: nobody finds out until a customer reports that something never happened. The bare minimum we set up on every DLQ is a CloudWatch alarm on ApproximateNumberOfMessagesVisible with a threshold above zero, because on a properly isolated DLQ any message present is already a signal that something needs human review, not a value you'd expect to grow with normal traffic. That alarm fires a notification to SNS that reaches the on-call channel, not a dashboard someone has to remember to check.

Once we've found the cause and fixed the bug or the downstream service is back, we use SQS's redrive feature to move messages from the DLQ back to the main queue without writing a manual migration script, specifying how many messages to move and from what point in time — which is safer than draining the whole DLQ at once when only part of the backlog corresponds to the bug that's actually been fixed.

What didn't work: manual backoff retries inside the handler

In an early integration we tried to sidestep the complexity of the visibility timeout by building our own backoff inside the function: if the call to the downstream service failed, the handler would wait a few seconds and retry internally before returning control to Lambda. The idea was to get fine-grained control over backoff without depending on the queue's configuration. It didn't work, for two reasons. The function kept its concurrency slot occupied for the entire wait, so under a burst of downstream failures the manual backoff multiplied the billable execution time of each invocation instead of freeing up the slot for other messages. And because the retry lived inside the same invocation, a cold restart or a function timeout during the wait lost the retry's state without SQS ever knowing it needed to reschedule it — unlike what happens when the queue itself owns the retry. We went back to relying on SQS's visibility timeout and maxReceiveCount for backoff between retries, and kept the handler's own logic limited to deciding whether a failure is retryable or should fail straight to the DLQ.

The question we ask before closing out an integration

Before calling an SQS-to-Lambda integration done, we check four things: that the visibility timeout leaves enough margin over the function's timeout, that maxReceiveCount reflects whether the expected failures are transient or deterministic, that the handler reports partial batch failures instead of all-or-nothing, and that there's an alarm on the DLQ that reaches a person, not just a log line. Once all four are in place, a failing message stops being a silent-loss risk and becomes, at worst, an alert someone acts on — with the message still sitting there, ready to be reprocessed.