Webhook Reliability for Social Data: Delivery Guarantees, Ordering and Retries

Webhooks look simple from the outside. A platform posts to your endpoint, you return 200, done. The reliability work is all in what happens when that does not go smoothly, and it goes not smoothly more often than people expect.

The useful mental model is that a webhook is an unreliable network call with a retry policy attached. It gives you fewer guarantees than a queue and more than nothing, and the gap between those is where your receiver has to do work.

Here is what you actually get, what varies between platforms, and how to build a receiver that holds up. For which platforms offer webhooks at all, and the cost comparison against polling, see webhooks versus polling.

What do webhooks guarantee?

3 things, and 2 of them are the absence of a guarantee people assume they have.

That third row is worth stating plainly because the industry language is confusing. No platform in this category offers exactly-once delivery, and HubSpot says so in its own documentation. What you can build is a receiver where processing an event twice has the same result as processing it once, which gets you the same outcome by a different route.

How long do you have to respond?

Less time than most handlers take, and the budget is set by whichever platform you support with the tightest limit.

A 6x spread, so a handler sized for a 30 second budget will time out on a 5 second platform. And a timeout is not a neutral event: the platform records a failed delivery and queues a retry, so a slow handler manufactures its own extra traffic.

That is the feedback loop to avoid. Slow handler, timeout, retry, more load, slower handler, more retries. It is the same amplification we flagged on refresh jobs at scale, and it has the same fix: do almost nothing in the request.

Before building receivers for each platform, check which events are actually available. Per-platform coverage is public. See the coverage list

What should the handler do?

Verify, queue, respond. Everything else belongs in a worker.

# Capture the RAW body. A JSON parser that re-serialises will
# change the bytes and your signature check will fail.
app.use(express.raw({ type: "application/json", limit: "2mb" }))

def handler(request):
    # 1. Signature first, before trusting any field.
    if not valid_signature(request.raw_body, request.headers):
        return 401                 # do NOT queue unverified payloads

    # 2. Reject stale deliveries to block replay attacks.
    ts = int(request.headers["X-Request-Timestamp"])
    if abs(now() - ts) > 300:      # 5 minute window
        return 401

    # 3. Queue and acknowledge. Nothing else.
    queue.put(request.raw_body)
    return 200

# Typical time in handler: a few milliseconds.
# Whatever the platform timeout is, you are inside it.

The raw body detail causes more signature failures than anything else. Most frameworks parse JSON by default, and a re-serialised body has different bytes from the one the platform signed.

The timestamp check is cheap replay protection. Several platforms send a timestamp header for exactly this purpose, and without it a captured payload can be replayed later with a valid signature.

How do you handle duplicates?

With an idempotency key and a store, and with some care about what makes a good key.

def process(event):
    key = dedupe_key(event)

    # SETNX: only the first worker gets True. No race.
    if not seen.set_if_absent(key, ttl=7*86400):
        return                      # already handled

    apply(event)

def dedupe_key(event):
    # An event id is the obvious key, and not always sufficient.
    # HubSpot documents that its eventId is NOT guaranteed unique.
    # Combine fields until the key is unique in practice.
    return f"{event.source}:{event.id}:{event.object_id}:{event.occurred_at}"

# TTL matters. Too short and a late retry gets processed twice.
# Platforms retry for up to 7 days, so size the window to the
# longest retry policy you support.

2 common mistakes: a read-then-write instead of an atomic set, which races when 2 workers take the same delivery, and a 1 hour dedupe window on a platform that retries for days.

How do you handle out of order events?

By trusting the payload timestamp rather than the arrival time, and by refusing to let an older event overwrite newer data.

def apply(event):
    current = store.get(event.object_id)

    # Arrival order means nothing. Event time is the truth.
    if current and current.updated_at >= event.occurred_at:
        return                      # stale. Drop it.

    store.upsert(event.object_id, event.payload, event.occurred_at)

This matters most for sequences. An update arriving before its create is common, and a naive handler either errors or creates a half-formed record. Treating the store as last-writer-wins by event time handles both cases without special casing.

The field name varies by platform, as field names do generally, so normalise it in your adapter layer. Keeping platform differences in 1 place is the same pattern that makes deprecations survivable.

What happens when retries run out?

It depends on the platform, and the range is wide enough that you cannot assume a safety net.

After retries are exhausted, a good platform parks the event somewhere you can inspect and replay. Many do not. Either way, the reconciliation sweep from the webhooks versus polling post is what catches whatever was lost, and its hit rate is the only honest measure of how well your webhooks are really doing.

Can your endpoint get switched off?

Yes, and faster than you might expect. Platforms track endpoint health and stop delivering to one that keeps failing.

Published thresholds include marking an endpoint as failing after 95% non-200 responses over 15 minutes, failed after 95% over 72 hours, and disabling entirely after 5 days of total failure. Once disabled you receive nothing until someone reactivates it, and events during that window are usually gone.

So a deploy that breaks your handler for an afternoon is not only an outage, it can be the start of a longer one. Worth having an alert on your own non-200 rate rather than waiting to notice the silence.

What should you measure?

1. Time spent in the handler, as a percentile. If p99 approaches the tightest platform timeout, retries are coming.
2. Duplicate rate. Deliveries dropped by your dedupe check. A rising rate means retries are climbing, which usually means you got slower.
3. Your own non-200 rate per platform. This is what endpoint health is judged on.
4. Longest gap with no events per subscription. Webhooks fail silently, so absence is the alarm, much like a revoked token.
5. Reconciliation hits. Events your polling sweep found that no webhook delivered. This is your real miss rate and nothing else measures it.

Summary

Treat webhooks as at-least-once delivery with no ordering, because that is what they are. Build a handler that verifies the signature on the raw body, queues the event and returns 200 in a few milliseconds, then does the real work in a worker.

Dedupe on a key you have confirmed is actually unique, with a window as long as the longest retry policy you support. Order by event time rather than arrival. And keep a slow polling sweep running, because on platforms like TikTok it is the only path, and elsewhere it is how you learn what you never received.

Want 1 webhook endpoint across every platform, with the retries and dedupe already handled? Get a demo