Webhooks push delivery events to your server as they happen: message.sent, message.delivered, message.failed. They are the cheapest way to keep your own database in sync with every message, and they fail for boring reasons: a deploy, an expired certificate, a firewall rule, a handler that became slow. This article covers what happens when they fail and how to recover cleanly.
Why deliveries fail
An attempt counts as failed when your endpoint does not answer with a 2xx status within 5 seconds. In practice the causes are:
- A deploy or restart that leaves the endpoint down for a few minutes.
- An expired or misconfigured TLS certificate.
- A firewall or WAF rule that starts blocking our requests.
- A handler that does real work before answering and becomes slower than the timeout under load.
The retry schedule
Failed attempts are retried 6 attempts over 24 hours, with growing delays: one minute after the first failure, then five minutes, thirty minutes, two hours and twelve hours. That covers a deploy, a restart and most outages of an afternoon. After the last attempt, the event is marked undelivered.
Undelivered events and replays
Each endpoint in the panel has a list of undelivered events, kept for 30 days, with the status code or error of the last attempt. Once the endpoint is fixed, replay events one by one or select them in bulk. A replayed event keeps its original event id and gets a new timestamp and signature, so a handler that deduplicates on the event id processes it exactly once even if an earlier attempt had in fact reached you.
Endpoint health
The same page shows each endpoint's success rate, last failure and average response time. A handler that takes four seconds to answer will show it there before it starts timing out.
A handler that survives all of this
- Verify first. Check the HMAC-SHA256 signature in the
X-Meteor-Signatureheader against the raw body and theX-Meteor-Timestampheader, and refuse old timestamps. The signature proves the event came from us; the timestamp stops replays by anyone else. - Acknowledge fast. Store the event and answer 200 immediately; do the real work in a background job.
- Deduplicate. Keep the event ids you have processed and ignore repeats. Retries and replays are normal, not errors.
- Order by time, not arrival. A replayed
message.sentcan arrive after themessage.deliveredof the same message. Order bycreated_atand by status precedence, and never move a message back from delivered to sent. - Monitor. Alert on your own error rate and check the undelivered list after every incident.
The guide to webhooks in production has code for each step.
Test the failure path
With a test key, webhooks fire exactly like live ones, signed, without sending anything. Use it to check that your handler verifies signatures, survives duplicates and keeps statuses in order. Then stop your endpoint for a few minutes while sending test messages and bring it back: the retries should deliver the missed events, and your database should end in the right state. Replays from the undelivered list behave the same way, with the original event ids.


