Webhook retry behavior on failed deliveries

Seeing inconsistent delivery on our webhook endpoint. Some events arrive 3-4 times. Need to understand:

  • What's the retry policy? Exponential backoff or fixed intervals?
  • How many attempts before FieldPulse gives up?
  • Is there a dead letter queue or visibility into failed deliveries?

Currently returning 500 on purpose for testing — seeing retries come in at weird intervals. No docs on this.

Also: what's the idempotency key situation? Need to dedupe on our end.

  • Carlos — yeah, the retry behavior isn't super well documented yet (we're fixing that). Here's what's happening:

    Current retry policy:

    • Exponential backoff: 1s, 2s, 4s, 8s, 16s, then every 60s
    • Max attempts: 24 hours worth of retries, then the event is dropped
    • No dead letter queue today — we just log and move on

    The "weird intervals" you're seeing are probably the exponential backoff plus some jitter we add to prevent thundering herd. The jitter is ±20% of the calculated delay.

    Idempotency: Each delivery attempt has a unique X-FieldPulse-Delivery-Id header, but that's per-attempt, not per-event. The event itself has event.id in the payload — that's what you want to dedupe on.

    Example payload structure:

    {
      "event": {
        "id": "evt_abc123",
        "type": "work_order.status_changed",
        "created_at": "2025-08-18T10:20:00Z"
      },
      "data": { ... }
    }

    Store evt_abc123 with a TTL and ignore duplicates. We don't guarantee exactly-once delivery.

    Worth noting: if your endpoint is consistently failing (5xx or timeouts > 30s), we do start backing off the connection pool for your endpoint, which can delay initial delivery attempts. So that 500 test might be affecting observed behavior.

    We're tracking a proper delivery dashboard internally — no ETA but it's on the roadmap. For now, email me your webhook ID if you want me to pull recent delivery attempts from our logs.

  • Heads up on the dedupe approach — we learned this the hard way. The event.id is stable across retries for the same event, but if a webhook subscription gets deleted and recreated (even with the same URL), you'll see new event IDs for the same underlying change. Rare edge case but burned us during a migration.

    What we ended up doing: composite key of event.id + hash of the payload's data object. Extra protection against accidental double-processing if someone clones a subscription.

    YMMV but saving you the headache we had.

  • @eli.torres — thanks. The 30s timeout is good to know. We're seeing occasional 28-29s response times during DB maintenance windows, probably hitting that.

    Is that timeout configurable per-webhook? Or account-wide?

  • Not configurable today — fixed at 30s for everyone. We've discussed making it flexible but it's lower priority than some other webhook improvements (signed payloads, event filtering, etc).

    If you're regularly hitting 25s+ you might want to ack early and process async, or we can look at whether something's slow on our serialization side. DM me your account ID and I can check.

  • Related docs:

    • Setting Up Webhooks
    • API Rate Limits and Best Practices (includes webhook section)

    Note: event.id is guaranteed unique per event stream but not globally unique across different webhook subscriptions. If you have multiple subscriptions pointing to the same endpoint, each gets its own event ID.