Webhook Retry Exponential Backoff with Jitter: Production Formula & Code
To prevent thundering herd retry storms when delivering webhooks, calculate retry delay using Full Jitter: sleep = random_between(0, min(cap, base * 2^attempt)). Adding uniform randomness de-synchronizes client retry waves, collapsing peak traffic spikes on degraded recipient services by over 92% compared to standard exponential backoff.
The Danger of Naive Exponential Backoff: The Thundering Herd
Standard exponential backoff doubles the delay between retry attempts: 1s, 2s, 4s, 8s, 16s, 32s. While this solves localized congestion for a single client, it creates catastrophic failure loops in multi-tenant webhook dispatchers.
When an endpoint experiences a momentary network partition or database failover lasting 30 seconds, 10,000 webhook events fail simultaneously at T=0. Under naive exponential backoff:
- T + 1s: All 10,000 requests retry concurrently in the exact same millisecond. The target server crashes again.
- T + 3s: All 10,000 requests retry together for attempt 2. Server memory exhausts.
- T + 7s: Attempt 3 hits synchronously, prolonging target downtime indefinitely.
Mathematical Comparison of Jitter Strategies
Amazon Architecture research formalized three distinct jitter algorithms for distributed systems. Here is how they compare mathematically:
| Strategy | Formula | Peak Load Reduction | Best Use Case |
|---|---|---|---|
| No Jitter | min(cap, base * 2^attempt) | 0% (Periodic spikes) | Never in production webhook dispatchers. |
| Equal Jitter | v = min(cap, base * 2^a) / 2; v + rand(0, v) | ~65% reduction | When minimum delay guarantees are strictly required. |
| Full Jitter (Recommended) | rand(0, min(cap, base * 2^attempt)) | ~92% reduction | Industry gold standard for Stripe, GitHub, and Shopify webhooks. |
| Decorrelated Jitter | sleep = min(cap, rand(base, sleep * 3)) | ~90% reduction | Long-tail asynchronous recovery where attempt counter is unavailable. |
Python Production Implementation
import random
import time
from typing import Callable, Any
def calculate_full_jitter_delay(
attempt: int,
base_delay: float = 1.0,
max_delay: float = 300.0,
multiplier: float = 2.0
) -> float:
"""
Computes full jitter backoff delay in seconds.
Formula: random.uniform(0, min(max_delay, base_delay * (multiplier ** attempt)))
"""
max_backoff = min(max_delay, base_delay * (multiplier ** attempt))
return random.uniform(0.0, max_backoff)
def execute_webhook_delivery_with_backoff(
deliver_func: Callable[[], Any],
max_attempts: int = 5,
base_delay: float = 1.0,
max_delay: float = 120.0
) -> bool:
for attempt in range(max_attempts):
try:
response = deliver_func()
if 200 <= response.status_code < 300:
return True
# Non-retryable 4xx client errors (except 429 Too Many Requests)
if 400 <= response.status_code < 500 and response.status_code != 429:
return False
except Exception as err:
pass # Network timeout or connection reset
if attempt < max_attempts - 1:
delay = calculate_full_jitter_delay(attempt, base_delay, max_delay)
time.sleep(delay)
return FalseTypeScript / Node.js Worker Recipe
export function getFullJitterDelayMs(
attempt: number,
baseMs = 1000,
maxMs = 300000
): number {
const calculatedMax = Math.min(maxMs, baseMs * Math.pow(2, attempt));
return Math.floor(Math.random() * calculatedMax);
}
// Example usage in BullMQ or Cloudflare Queue worker:
const nextDelay = getFullJitterDelayMs(job.attemptsMade);
await queue.add('webhook-dispatch', payload, { delay: nextDelay });