Workaround: API Pagination for Large Data Exports

Yeah this one tripped me up too when I first looked at it. You're hitting 429s halfway through a big export and losing data, which is... not ideal. The API's pagination at the time of writing has this quirk where the offset parameter interacts badly with rate limiting if you're moving through large datasets quickly.

What you're seeing

Your export script runs fine for a few hundred records, then starts getting 429 responses. You back off, retry, but now your offset math is off because some pages returned partial data before the rate limit kicked in. Been there.

The workaround: cursor-based pagination with a local checkpoint

Instead of trusting offset and limit alone, use the created_at or id field as a cursor. This one's idempotent — if you hit a 429, you can retry the same cursor without drifting.

GET /v2/work-orders?created_after={cursor}&limit=100&sort=created_at:asc

Store the last id or created_at you successfully processed. If the request fails, retry with exponential backoff starting at 2s. The key is never advancing your cursor until you've confirmed the page is fully written to your destination.

Sample flow (pseudocode, but you get it)

cursor = "2024-01-01T00:00:00Z"
while True:
    try:
        page = fetch(f"/work-orders?created_after={cursor}&limit=100")
        if not page.items:
            break
        write_to_destination(page.items)
        cursor = page.items[-1].created_at  # last item becomes next cursor
    except RateLimit:
        sleep(backoff)
        backoff *= 2  # yeah this can get long, but it works

Edge case worth mentioning: if you have multiple records with the same created_at at a page boundary, you might get duplicates. I usually dedupe on id at the destination, or switch to id cursor if your data has tight timestamps.

The catch

This is slower than raw offset pagination. You're looking at maybe 2-3x the wall time for a full export. But it's the only reliable way I've found to get complete data when you're in the 10k+ record range.

Related: if you're hitting this because you're trying to build a data warehouse sync, heads up that webhook delivery is generally more reliable for that use case. See this thread on warehouse exports for a different approach.

Also worth noting: the engineering team's aware of the offset+rate-limit interaction. No ETA on a proper fix, hence this workaround.