webhook-delivery-service

Live chaos demo

Watch a webhook pipeline survive an outage.

Press the button and 1,000 events go through the real API, queue and worker toward an endpoint that is about to crash, flap, then recover. Retries, the circuit breaker and dead letters play out live. Nothing is simulated in your browser: every number below is read from Postgres and Redis.

Scenario control

Connecting…

0:00 / 2:20

Waiting

One run takes 2 min 20 s. Anyone visiting the page watches the same run.

Delivered

0/ 1000

Pending

0

queued or waiting for a retry

Dead

0

rejected with 4xx, not retried

HTTP attempts

0

0 postponed by the breaker

Circuit breaker

closed

failure streak 0 / 5

Event status over time

Sampled every second. Background bands are the endpoint's behaviour.

  • delivered
  • pending
  • dead
  • breaker open

No run yet. Press Run outage scenario to start one.

Delivery attempts

The worker's audit log, newest first.

teventtryhttpmsoutcome

Attempts will stream here.

Receiver side

What the customer endpoint saw.

Webhooks received
0
Signature verified
0
Bad signature
0
Events accepted by the API
0 / 0
  • 2xx 0
  • 5xx 0
  • 4xx 0

Every request carries webhook-id, webhook-timestamp and an HMAC-SHA256 webhook-signature. The endpoint verifies it in constant time before answering, exactly like a production consumer should.

What you are looking at

  1. Browser
  2. API Fastify
  3. Queue BullMQ · Redis
  4. Worker
  5. Endpoint

1 Idempotent ingestion

POST /events requires an Idempotency-Key. The API stores the event in Postgres, enqueues it and answers 202 immediately: delivery is asynchronous.

ingestion.ts

2 Signed delivery

The worker signs id.timestamp.body with the endpoint's secret (stored AES-256-GCM encrypted) and logs every attempt, success or not.

deliver.ts

3 Retries with backoff

5xx, timeouts and network errors are retried: 1 s, 10 s, 1 min, 10 min, 1 h, 6 h, with ±20% jitter. 429 Retry-After is honoured.

backoff.ts

4 Circuit breaker

5 failures in a row open the breaker for 30 s (doubling on relapse). Deliveries are postponed, not attempted. Then one probe goes through (half_open). State lives in Redis, updated by an atomic Lua script shared by all workers.

record_outcome.lua

5 Dead letters

A 4xx means the endpoint is alive but refuses this payload: retrying won't help. The event goes straight to dead, and can be replayed manually later. 1 event in 25 here is built to be rejected.

classify.ts