Webhook Dead Letter Queue (DLQ) Architecture with AWS SQS: Production Blueprint

Why Webhooks Without DLQs Cause Data Loss

In asynchronous architectures, webhooks trigger critical downstream operations: provisioning customer accounts, updating order statuses, or syncing inventory. When third-party consumers throw unhandled exceptions (e.g. schema changes, null pointer bugs, expired auth tokens), naive retry queues either drop the messages or cycle forever in an infinite loop.

A properly architected Dead Letter Queue acts as an immutable safety buffer. Instead of discarding messages after maxReceiveCount is exceeded, the message is atomically routed to a quarantine queue with full request headers and payload intact.

AWS SQS Redrive Policy Configuration (Terraform)

Here is the production Terraform definition connecting a primary webhook ingestion queue to a secure dead letter queue:

# 1. Dead Letter Queue with 14-day retention
resource "aws_sqs_queue" "webhook_dlq" {
  name                      = "webhook-events-dlq"
  message_retention_seconds = 1209600 # 14 days
  sqs_managed_sse_enabled   = true
}

# 2. Primary Webhook Dispatch Queue
resource "aws_sqs_queue" "webhook_primary" {
  name                       = "webhook-events-primary"
  visibility_timeout_seconds = 60
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.webhook_dlq.arn
    maxReceiveCount     = 5
  })
}

# 3. Redrive Allow Policy (Restricts who can route into the DLQ)
resource "aws_sqs_queue_redrive_allow_policy" "dlq_allow" {
  queue_url = aws_sqs_queue.webhook_dlq.id
  redrive_allow_policy = jsonencode({
    redrivePermission = "byQueue"
    sourceQueueArns   = [aws_sqs_queue.webhook_primary.arn]
  })
}

DLQ Inspection & Automated Alerting

Messages in a DLQ require immediate operational visibility. Configure an AWS CloudWatch Alarm triggered when ApproximateNumberOfMessagesVisible > 0:

• Alert Channel: Route CloudWatch SNS notifications directly to your team's Slack or PagerDuty on-call roster.
• Audit Log: Store message payload, source IP, failure timestamp, and exception stack trace in AWS DynamoDB or Datadog for root cause analysis.
• Automated Redrive: Once your team deploys a patch for the root bug, initiate the SQS StartMessageMoveTask API to replay quarantined messages back to the primary queue with zero manual scripting.

Python Redrive Automation Script (Boto3)

import boto3

sqs = boto3.client('sqs', region_name='us-east-1')

def redrive_dlq_to_source(source_arn: str, dlq_arn: str):
    """
    Initiates native AWS managed redrive task from DLQ back to primary queue.
    """
    response = sqs.start_message_move_task(
        SourceArn=dlq_arn,
        DestinationArn=source_arn,
        MaxNumberOfMessagesPerSecond=100
    )
    task_handle = response.get('TaskHandle')
    print(f"Redrive initiated successfully. TaskHandle: {task_handle}")
    return task_handle