September 13, 2026
WEBHOOK AND AUTOMATION FAILURE CHECKLIST

Use this checklist when a webhook-driven workflow misbehaves — duplicate access grants, missing confirmations, silent failures — or before promoting any flow to production. It turns Lesson 88.1–88.4's rules into a diagnostic walk. Work top to bottom; most incidents resolve in the first two sections.
1. Was it ever verified? (signature first)
- [ ] Reproduce with the logged raw body and signature. Does verification pass with the current secret? Check against Stripe webhooks signature semantics if Stripe is the sender.
- [ ] Confirm the code verifies the raw body, not re-serialized JSON — re-serialization breaks signatures silently.
- [ ] Confirm failures return 4xx with no side effects. Any grant/email/CRM write on unverified delivery is a stop-ship bug.
- [ ] If the secret recently rotated, check which secret version each delivery used. Keep old-secret acceptance briefly during rotation, then close it.
2. Was it processed exactly once? (idempotency)
- [ ] Look up the event ID in the idempotency store. Processed twice with two side effects means the lookup is missing or after the action — move it before.
- [ ] Confirm the endpoint returns 200 fast and defers slow work to a background job. Timeouts breed retries that look like duplicates.
- [ ] For platform flows, check execution history in n8n, Make incomplete executions, or Zapier help replay — then confirm the dedupe key survived replays.
3. Was the payload valid? (malformed input)
- [ ] Validate against the expected schema: required fields, types, enum values. Log the rejection reason with the event ID.
- [ ] Unknown event types should acknowledge + log, never crash the handler.
- [ ] If the sender changed payload shape, pin the version you accept and add the new shape deliberately — don't loosen validation to "make errors stop."
4. Did a dependency fail? (vendor delay)
- [ ] Check each downstream call's timeout and attempt count. No timeout anywhere is the root cause of most "frozen" workflows.
- [ ] Confirm backoff (e.g. 1 / 5 / 30 min) with a max-attempt budget, then parking in the review queue — not infinite retry.
- [ ] Verify queue aging: items older than the SLA escalated to the named owner, not sitting unread.
5. Did a human need to step in? (unexpected + takeover)
- [ ] Unexpected replies routed with full context (payload + attempts + history) to the owner within SLA? The workflow never improvised?
- [ ] Stop switch reachable and tested — who pulled it, what halted safely, what finished? Resume procedure followed and logged?
6. Close-out
Record the incident in the decision log (Lesson 89.5): cause class (one of the five), fix (code, scope, threshold, or checklist change), and the regression check that proves it stays fixed. If the same class recurs within 30 days, the corrective action failed — narrow the workflow or tighten the gate rather than retrying optimism. Promote to production only when the forged-event, duplicate-event, malformed, vendor-down, and manual-pause simulations all land where this checklist says they should.
ARTICLE DISCUSSION
JOIN THE
CONVERSATION.
Got a question, a take, or a better way to do this? Log in and leave a comment.
