Dead-Letter Queue Keeps Growing: Where to Start

The dead-letter queue alarm fires, or someone notices it holds 40,000 messages instead of the usual dozen. The instinct is to hit “redrive” and move on. That usually sends the same messages back through the same failure and fills the DLQ again within the hour. This guide is the order of operations: what to look at first, how to find out why messages failed, and when a redrive is actually safe.

The steps below apply to Amazon SQS, RabbitMQ, and Kafka dead-letter topics. The mechanics differ, and they matter, because each system decides differently what counts as “failed” and records different amounts of information about why.

What does a growing dead-letter queue mean?

A dead-letter queue (DLQ) is a separate queue or topic that holds messages the consumer could not process after a set number of attempts, so they stop blocking or looping on the main queue. A DLQ that grows means the rate of failed messages exceeds the rate at which anyone resolves them. That is always a symptom. The DLQ tells you that messages failed; it rarely tells you why.

Before investigating, be precise about what your system counts as a failure:

Kafka Connect settings: errors.deadletterqueue.topic.name and errors.deadletterqueue.context.headers.enable=true.

SQS needs a closer look, because its rule surprises people. SQS does not count failures. It counts receives. Every time a consumer receives a message and does not delete it before the visibility timeout expires, the message becomes visible again with a higher receive count. When that count passes maxReceiveCount, SQS moves it to the DLQ. The SQS dead-letter queue documentation covers the redrive policy in full.

{
  "deadLetterTargetArn": "arn:aws:sqs:us-east-1:123456789012:orders-dlq",
  "maxReceiveCount": "5"
}

Is it a step, a ramp, or a burst?

Start with the shape of the growth, because it narrows the cause before you read a single message. Plot DLQ depth and the age of the oldest message over the last few days, then line them up with deploys, config changes, and dependency incidents. For SQS, the CloudWatch metrics are ApproximateNumberOfMessagesVisible and ApproximateAgeOfOldestMessage on the DLQ. Watch depth rather than NumberOfMessagesSent: AWS documents that messages moved to a DLQ by the redrive policy are not counted in the DLQ’s NumberOfMessagesSent.

How do you inspect DLQ messages without making it worse?

Inspect a sample of messages without deleting anything, and join each one to the error that sent it there. Two mistakes are common here: consuming the DLQ with a script that deletes as it reads, and assuming the DLQ itself knows why the message failed.

In SQS, reading a message is a receive. It hides the message for the visibility timeout and increases its receive count, and the console’s “poll for messages” is also a receive. That is harmless on a DLQ with no redrive policy of its own, but do not point an inspection script at the source queue during an incident, and never delete during sampling. A small read-only sampler is enough:

import boto3, collections, json

sqs = boto3.client("sqs")
seen, groups = {}, collections.Counter()

for _ in range(20):
    resp = sqs.receive_message(
        QueueUrl=DLQ_URL, MaxNumberOfMessages=10, WaitTimeSeconds=1,
        VisibilityTimeout=600,                 # hide sampled messages so we don't re-read them
        MessageSystemAttributeNames=["All"], MessageAttributeNames=["All"],
    )
    for m in resp.get("Messages", []):
        body = json.loads(m["Body"])
        seen[m["MessageId"]] = body.get("event_id")     # ids to look up in logs
        groups[(body.get("type"), body.get("schema_version"))] += 1
        # no delete_message: the DLQ stays intact

for key, n in groups.most_common():
    print(n, key)

For a Kafka dead-letter topic, read with a new consumer group or kafka-console-consumer.sh --from-beginning --property print.headers=true. Reading does not remove records, and the Spring headers such as kafka_dlt-exception-fqcn, kafka_dlt-exception-message, and kafka_dlt-original-offset give you the reason directly. For RabbitMQ, the x-death header on each dead-lettered message shows the reason: rejected, expired, maxlen, or delivery_limit.

Join each message to its failure

With SQS, the reason lives in the consumer’s logs and traces, not in the queue. Search for the sampled ids around each message’s receive attempts and collect the exception and stack trace. This only works if the consumer logs a message id or business id with every failure, which is the first thing to fix if it does not. How to Correlate Logs, Traces, and a Code Change covers doing this join quickly under pressure.

Then group by error signature: the exception type plus the top application frame, with ids and values stripped out. In most incidents, one or two signatures account for almost all of the backlog. Fix those, and treat the long tail separately.

Which kind of failure is it?

Every dead-lettered message falls into one of five classes, and each class needs a different action before any redrive. Classify by evidence, not by the loudest error in the logs.

The lease class deserves attention because it hides. When processing gets slower, for example after a dependency slows down or a batch size increases, messages that were going to succeed are received again and again until SQS dead-letters them. Some of them have already been processed, possibly more than once. Redriving those messages processes them yet again, so it is only safe with an idempotent consumer. SQS or Kafka Message Processed Twice: Designing Safe Consumers covers how to build one.

When is it safe to redrive?

Redrive is safe when four things are true: the cause is fixed and deployed, the consumer is idempotent, the messages are still valid to apply, and the replay rate will not overload the consumer or its dependencies. Check each one explicitly.

Gate 3 is the one teams skip. A message that sat in the DLQ for three days may describe a state that is no longer true: a price that has since changed, an address that was updated, a cancellation that arrived after the original order. Replaying it can overwrite newer data. Consumers that apply events with a version or timestamp check (“only update if this event is newer than what we have”) are safe here; consumers that blindly set values are not.

For SQS, use the console’s DLQ redrive or the StartMessageMoveTask API. Without a destination ARN, messages go back to the queue they came from. Set a rate:

# Current backlog (queue attributes, not CloudWatch metric names)
aws sqs get-queue-attributes --queue-url "$DLQ_URL" \
  --attribute-names ApproximateNumberOfMessages ApproximateNumberOfMessagesNotVisible

# Move messages back to their source queue at 10 per second
aws sqs start-message-move-task \
  --source-arn arn:aws:sqs:us-east-1:123456789012:orders-dlq \
  --max-number-of-messages-per-second 10

# Watch progress, or stop it if errors return
aws sqs list-message-move-tasks --source-arn arn:aws:sqs:us-east-1:123456789012:orders-dlq
aws sqs cancel-message-move-task --task-handle "$TASK_HANDLE"

For a canary, move a small number of messages first. With SQS you can run a short move task and cancel it, or receive a handful from the DLQ with a script and send them to the source queue. For Kafka, there is no built-in redrive: a small replay consumer reads the dead-letter topic and republishes to the original topic, or processes the records directly, with its own offsets so you can stop and resume.

How do you stop the DLQ from growing again?

Most DLQs that grow unnoticed have three gaps: no alarm, no owner, and no distinction between errors worth retrying and errors that never will succeed. Close all three.

• Alarm on depth and age. For most queues, one message in the DLQ is worth a ticket, and a rising oldest-message age is worth a page before retention runs out.
• Separate retryable from non-retryable errors. Validation and parse failures will not improve with retries. Send them straight to the DLQ with the reason attached, and retry only transient errors with backoff. Blanket retries on everything are how a small bug becomes a retry storm.
• Carry the reason with the message. Spring and RabbitMQ do this with headers. For SQS, a consumer that dead-letters a message itself can add the error as a message attribute.
• Size the lease and the retry count together. A visibility timeout above worst-case processing, and a maxReceiveCount that tolerates a short blip.
• Give the DLQ an owner and a runbook, including a redrive procedure that is safe to run twice. The Production Readiness Checklist for a Backend Service lists this with the other runbook items.

@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
    // Publishes failed records to <topic>.DLT (same partition) with exception headers
    var recoverer = new DeadLetterPublishingRecoverer(template);
    var handler = new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3L));
    handler.addNotRetryableExceptions(ValidationException.class, JsonParseException.class);
    return handler;
}

Because the default recoverer writes to the same partition number on the .DLT topic, create the dead-letter topic with at least as many partitions as the source. The same pattern, separating the error classes and recording why, applies to any framework.

How Tomosu helps

Tomosu analyzes a repository and its pull requests for production reliability risks. Most of what makes a DLQ grow is visible in code before it is visible in a dashboard: how a consumer classifies errors, what it retries, where it acknowledges, and what it logs when it fails. For this problem, Tomosu:

• Maps consumers and their failure paths: which exceptions are retried, which are dead-lettered, and which are swallowed.
• Flags handlers that will not be safe to redrive, such as non-idempotent side effects or state updates with no version check.
• Points at changes that shift the failure rate: new validation, a changed payload contract, a slower downstream call inside the consumer, or a reduced timeout.
• Surfaces missing failure context, such as error logging without a message or business id, which makes the join in this guide impossible.

These signals feed the Production Reliability Index. To try it on your consumers, follow the repository scan guide.

Scan your repository with Tomosu →

Key takeaways

• A dead-letter queue that keeps growing is a symptom; do not redrive before you know the cause.
• Read the growth shape first: a step at a deploy, a steady ramp, a burst during an outage, or growth that tracks traffic.
• SQS counts receives, not failures, and stores no reason; join messages to consumer logs by id.
• Sample without deleting, then group failures by error signature.
• Classify each group as poison, bug, outage, lease settings, or missing prerequisite, and fix accordingly.
• Redrive only through four gates: cause fixed, consumer idempotent, messages still valid, canary batch clean; then rate-limit.
• Prevent the next backlog with depth and age alarms, non-retryable error routing, failure context, and an owner.

Frequently asked questions

Every message in a dead-letter queue is a failure that already happened. Tomosu helps you see the consumer paths that will put the next ones there, before they ship. Assess your repository →