Best Pattern for Handling Duplicate and Out-of-Order GitHub Webhook Events? #211830
Replies: 6 comments 1 reply
|
💬 Your Product Feedback Has Been Submitted 🎉 Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users. Here's what you can expect moving forward ⏩
Where to look to see what's shipping 👀
What you can do in the meantime 💻
As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities. Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐ |
|
Your design is solid, and the delivery ID is the right key for deduplication. It just can't handle ordering, so I'd add a second check when you update each record. An internal key of repo + event type + resource ID won't work as a deduplication key on its own, because one pull request or run legitimately sends many events. It would drop real updates. 1. Receive the event
2. Skip repeat deliveries CREATE TABLE webhook_deliveries (
delivery_id text PRIMARY KEY,
status text NOT NULL DEFAULT 'queued',
received_at timestamptz NOT NULL DEFAULT now()
);
INSERT INTO webhook_deliveries (delivery_id)
VALUES ($1)
ON CONFLICT (delivery_id) DO NOTHING;If the insert affects 0 rows, return a 2XX and don't queue it again, unless that delivery is marked as failed. A redelivery keeps the same 3. Never let an older event overwrite newer data Give each resource a version you can compare, and only update when the incoming version is higher. For INSERT INTO workflow_runs (run_id, run_attempt, status_rank, status, conclusion)
VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (run_id) DO UPDATE SET
run_attempt = EXCLUDED.run_attempt,
status_rank = EXCLUDED.status_rank,
status = EXCLUDED.status,
conclusion = EXCLUDED.conclusion
WHERE (EXCLUDED.run_attempt, EXCLUDED.status_rank)
> (workflow_runs.run_attempt, workflow_runs.status_rank);A late 4. Choose a version for each event type
5. Refetch when timestamps can't settle it Timestamps only have one-second precision, so two pull request events can tie. For fields that matter, treat the webhook as a signal that something changed, then get the current state from the REST API. You'll use more API requests, but you won't have to work out the order yourself. 6. Plan for missed events GitHub doesn't automatically redeliver failed deliveries, so run a scheduled job that lists recent deliveries through the REST API and redelivers the failed ones. Your dead-letter queue covers events that fail on your side, and this job covers events that never reached you. Keep delivery IDs for at least as long as you might replay them. Drafted with AI assistance. |
|
I'd recommend using the GitHub delivery ID (X-GitHub-Delivery) as the primary deduplication key. However, I wouldn't rely on it alone for maintaining data consistency. A good approach would be:
My suggestion: Use delivery IDs for duplicate delivery detection and resource IDs for maintaining state consistency. They solve two different problems. |
|
One fault-injection case worth adding to the canonical-GET approach above: reverse the order the GET responses reach your database.
I ran this with two SQLite connections on Linux. With an unconditional UPSERT, the slow response changed Keep the status ranking specific to the workflow states you support. For PR or deployment fields without a suitable version, serialize per resource or fence each refresh so an older in-flight result cannot commit after a newer one. A fresh GET on its own is not a write fence. |
|
3d4df872b9dd0e0316e9f3bb9a2a35d2362fad54 |
|
3d4df872b9dd0e0316e9f3bb9a2a35d2362fad54 |
|
Handling duplicate and out-of-order webhook events is a classic distributed systems challenge. Here is the industry-standard architectural pattern to solve both issues reliably: 1. Deduplication (Idempotency Key via
|
Uh oh!
There was an error while loading. Please reload this page.
🏷️ Discussion Type
Bug
💬 Feature/Topic Area
API
Body
We are building an internal integration that uses GitHub webhooks to update deployment and project-status records in another system.
During testing, we noticed that the same webhook can occasionally be delivered more than once. Events may also arrive late or in a different order from the actions shown in GitHub. The integration must process valid events without creating duplicate records or overwriting newer data.
Our development team at AppVerticals, a mobile app company, is considering this approach:
Store each GitHub delivery ID for idempotency.
Verify the webhook signature before processing.
Add accepted events to a queue.
Retry temporary failures with exponential backoff.
Record event timestamps before applying updates.
Send repeatedly failing events to a dead-letter queue.
For teams operating GitHub Apps at scale, is the delivery ID sufficient for deduplication, or should we create an internal idempotency key using the repository, event type, and resource ID?
I would also appreciate guidance on handling out-of-order events, particularly for pull-request, workflow-run, and deployment-status webhooks.
All reactions