6 JSON Webhook Mistakes That Lose or Duplicate Events

Quick answer: A reliable JSON webhook receiver preserves signed request bytes, acknowledges durable acceptance, claims duplicates atomically, handles delayed events without trusting arrival order, validates the event contract, and provides a tested recovery path. These six controls address different failure boundaries: authentic JSON can still be duplicated, stale, unsupported, or lost after a premature success response.

Your payment dashboard says an event was delivered. Your application says the customer is still unpaid. A successful HTTP response proves that the sender received an acknowledgement; it does not prove your business update completed. The useful debugging question is where responsibility for that event changed hands.

A webhook is an event-triggered callback mechanism. This guide focuses on receiving JSON events, rather than designing a publisher. For the broader request/response context, start with JSON in REST APIs. The examples below are small local demonstrations, not a complete payment integration.

1. Are you verifying a reformatted body instead of the received bytes?

Verify a JSON webhook against the original request bytes using the provider's documented signature scheme. Do not reconstruct the signed input by stringifying a parsed object. Formatting, escaping, whitespace, and numeric representation can change those bytes without changing the apparent data, causing genuine deliveries to fail authentication or unsafe debugging workarounds to emerge.

Think of the body as two separate artifacts: evidence used by the verifier, and structured data used by your handler. Save the first before middleware transforms it. Decode the second only through a path that preserves the relationship to the verified body. A log showing the same fields is not evidence of byte equality.

The GitHub signature documentation specifies HMAC-SHA256 and the X-Hub-Signature-256 header. HMAC means hash-based message authentication code. GitHub's scheme signs the body; other providers use different inputs and headers. Prefer maintained provider verification libraries where available instead of treating this scheme as universal.

This runnable Node.js demonstration shows why equivalent JSON can fail a byte-based signature:

import { createHmac, timingSafeEqual } from "node:crypto";
import assert from "node:assert/strict";

function signature(body, secret) {
  return "sha256=" + createHmac("sha256", secret).update(body).digest("hex");
}

function verify(body, header, secret) {
  if (!Buffer.isBuffer(body) || typeof secret !== "string" || !secret) return false;
  if (typeof header !== "string" || !/^sha256=[0-9a-f]{64}$/i.test(header)) return false;
  const supplied = Buffer.from(header.slice(7), "hex");
  const expected = createHmac("sha256", secret).update(body).digest();
  return supplied.length === expected.length && timingSafeEqual(supplied, expected);
}

const secret = "local-test-secret-not-for-production";
const raw = Buffer.from('{ "id": "evt_demo", "amount": 2900.0 }', "utf8");
const header = signature(raw, secret);
const reformatted = Buffer.from(JSON.stringify(JSON.parse(raw.toString("utf8"))));
assert.equal(verify(raw, header, secret), true);
assert.equal(verify(reformatted, header, secret), false);
assert.equal(verify(raw, undefined, secret), false);
assert.equal(verify(raw, "sha256=ab", secret), false);
assert.equal(verify(raw, header, "wrong-secret"), false);
assert.equal(verify(raw, header, ""), false);
console.log("Signature checks passed");

The Node.js crypto reference requires equal byte lengths for timingSafeEqual and warns that surrounding code still matters. The format check keeps malformed header values out of the comparison. This helper demonstrates GitHub-style body authentication; it does not implement timestamp-based replay protection or every provider's rotation behavior.

For inspection, use a sanitized copy in the JSON Viewer to expand nested event fields and copy paths. Keep the original verification input separate. A viewer does not authenticate events, and its formatted output must not replace the received body.

2. Are you sending success before the event is durably accepted?

Return a successful webhook response only after authenticated data has been durably accepted for processing or its required work has safely completed. For slower workflows, persist an inbox record and let a worker continue. An immediate acknowledgement without durable ownership creates a gap where a crash loses the event and suppresses the sender's recovery mechanism.

Picture a handler that schedules an in-memory promise, responds with success, and then restarts during deployment. The promise disappears. The provider has already been told delivery worked. Moving the task to a queue helps only when the queue acceptance itself is durable and your application checks the result.

The acceptance boundary should be explicit in code review. A useful receiver sequence is: enforce transport limits, authenticate, validate the supported envelope, commit an inbox record, then acknowledge. The worker owns subsequent failures. If the inbox write fails, return a failure that fits the provider's delivery contract rather than pretending ownership transferred.

GitHub's webhook best practices document its response deadline and recommend asynchronous processing. Treat that deadline as provider-specific. Choose an internal budget with room for storage latency and response transfer, then measure it under load. Increasing an application timeout cannot change the sender's deadline.

Store enough state for another worker to resume: provider, account scope, event identity, payload reference, processing status, attempts, and last failure category. Separate receipt time from business-event time. For a design review, draw the point where a crash becomes recoverable without another delivery from the publisher. If no durable record exists at that point, acknowledgement is too early.

The related JSON API timeout guide explains caller deadlines. A webhook adds a second obligation: accepting responsibility also requires your own recovery mechanism after the caller stops waiting.

3. Can two workers both pass your duplicate check?

Deduplicate webhook deliveries with an atomic operation backed by durable storage, not a separate lookup followed by an insert. Scope event identity to the provider and account where necessary, and distinguish receipt from completion. A duplicate claim should prevent parallel processing while still allowing failed or abandoned work to resume through controlled retries.

The classic race looks harmless in a unit test. Worker A asks whether an event exists and sees nothing. Worker B does the same. Both send the welcome email before either inserts its completion record. A unique constraint added after the email is sent cannot undo that external effect.

An inbox table can make the acceptance claim indivisible. Here is a PostgreSQL example for a provider that supplies stable event IDs in verified data:

CREATE TABLE webhook_inbox (
  provider text NOT NULL,
  account_id text NOT NULL,
  event_id text NOT NULL,
  payload jsonb NOT NULL,
  status text NOT NULL DEFAULT 'received',
  PRIMARY KEY (provider, account_id, event_id)
);

INSERT INTO webhook_inbox (provider, account_id, event_id, payload)
VALUES ('demo', 'acct_demo', 'evt_demo', '{"type":"order.paid"}'::jsonb)
ON CONFLICT (provider, account_id, event_id) DO NOTHING
RETURNING event_id;

The PostgreSQL INSERT reference defines ON CONFLICT and RETURNING. A returned row means this statement inserted the event; no returned row means it did not. This SQL was checked against the documented syntax, but it was not executed against a database in this article's local validation. It is an acceptance example, not a complete worker-locking implementation.

A worker needs a separate claim or lease policy, retries, and abandoned-work recovery. Commit local domain changes and completion state in one transaction when feasible. For external calls, carry a stable business-operation key if the destination supports idempotency. Otherwise document the remaining failure window and provide reconciliation.

Identity also has layers. Two different provider events can describe the same business transition. Event-ID deduplication alone will not prevent two shipment requests for an already shipped order. Read JSON API idempotency mistakes for the downstream distinction between repeated delivery and repeated business effects.

4. Does a delayed event overwrite newer business state?

Do not treat webhook arrival order as business order unless the provider explicitly guarantees it. Decide whether each event is a historical fact or a notification to refresh current state. Use documented sequence guarantees when available; otherwise retrieve authoritative state, apply domain transition rules, and test delayed events against already updated records.

A cancellation notification can arrive before an older subscription update. Blindly applying both snapshots can restore access after cancellation. Removing duplicate IDs does not solve this problem because the two events are distinct. You need a state policy, not another seen-event cache.

For a current-state projection, an event can trigger retrieval of the resource's latest state. Serialize or version the local update so two workers cannot apply older fetch results after newer ones. Fetching current state reduces reliance on event snapshots; it does not automatically eliminate a race between fetch and commit.

For accounting facts, preserve the historical event instead. A payment received yesterday remains a fact even if today's subscription status is canceled. Separate the ledger of events from the current entitlement view. Using one mutable status field for both makes delayed events unnecessarily dangerous.

Stripe's event-delivery documentation says delivery order is not guaranteed and warns against using snapshot created timestamps to determine order. Equal timestamps and provider-specific semantics matter. A generic instruction to ignore anything with an older timestamp can silently discard legitimate work.

Test the permutations that matter to your domain: update then cancel, cancel then update, duplicate cancel, and a missing event recovered later. Define the expected outcome before writing the handler. When an account can legitimately reactivate, your transition rules must distinguish a real reactivation from an old active snapshot rather than permanently rejecting every active state after cancellation.

5. Are you accepting any valid JSON as a supported event?

Successful JSON parsing proves syntax, not that an authenticated event has a supported type, compatible shape, correct account context, or acceptable resource size. Validate the envelope and the fields required by the selected handler. Keep unknown events separate from malformed events, and make version changes visible before unsupported payloads reach business logic.

A top-level array is valid JSON. So is null. Neither necessarily matches your provider's event contract. Likewise, a string amount can parse successfully while breaking arithmetic. Use an event-type allowlist and validate only against the actual documented contract, including nullable fields and legitimate variation.

This small guard illustrates envelope checks for an invented demonstration contract:

import assert from "node:assert/strict";

function isDemoEvent(value) {
  return value !== null && typeof value === "object" && !Array.isArray(value)
    && typeof value.id === "string" && value.id.length > 0
    && value.type === "order.paid"
    && value.data !== null && typeof value.data === "object" && !Array.isArray(value.data)
    && typeof value.data.orderId === "string" && value.data.orderId.length > 0;
}

assert.equal(isDemoEvent({ id: "evt_demo", type: "order.paid", data: { orderId: "ord_demo" } }), true);
for (const value of [null, [], {}, { id: "evt_demo", type: "unknown", data: {} }]) {
  assert.equal(isDemoEvent(value), false);
}
console.log("Envelope checks passed");

This is not a schema for Stripe or GitHub. Production validation should follow your provider's event definition and your handler's required fields. The JSON Schema validation guide covers traps such as optional properties and validation settings. A syntax formatter cannot replace these contract checks.

Apply request-size limits before buffering an unlimited body, while preserving allowed bodies exactly for authentication. Validate account routing from verified information rather than trusting an arbitrary query parameter. Handle unknown but authentic event types according to the provider's contract; logging and acknowledging an intentionally ignored type can be correct, whereas silently dropping a newly required event is an integration defect.

The OWASP webhook security guidance brings together payload limits, authentication, replay controls, and secret-safe logging. Keep each control's purpose clear: a valid signature does not make unlimited resource consumption or every field value acceptable.

6. Can you recover a failed event without repeating its side effects?

Give JSON webhook processing a durable retry path, a visible failure state, and replay tests that exercise the same business safeguards as live traffic. Record event identity and processing outcomes without leaking secrets. Recovery should distinguish unaccepted deliveries, queued work, completed effects, and uncertain external results so operators do not blindly repeat irreversible actions.

A dashboard showing received events is useful, but it is not a recovery plan. Track worker attempts, queue age, last successful processing time, and failure category. A receiver returning success while its queue grows indefinitely deserves attention even if the publisher reports healthy delivery.

Use a failed-event area or dead-letter queue for work that has exhausted normal retries. Retain the reason and enough protected context to diagnose it. Avoid copying full production bodies into tickets: customer data and signed headers can turn an incident report into another security problem. Link a controlled event record instead.

Replay should use the same event identity and domain checks. A special repair script that bypasses deduplication can create precisely the duplicates the normal handler prevents. If the outcome of an external payment or email call is uncertain, query the destination or reconcile records before retrying. An exception after a remote success is not proof that nothing happened.

Use the following failure matrix during release verification:

Both JavaScript demonstrations in this article were executed locally, including rejected signatures and unsupported envelopes. The matrix describes integration tests to implement; it does not claim a live provider endpoint was tested. Add provider-generated signed fixtures, real middleware, storage concurrency, and deployment interruption tests before relying on your receiver in production.

Keep a short operational note with the endpoint's owner, retry policy, retention policy, and recovery command. That turns a webhook incident from a search through raw logs into a controlled decision about which accepted event still needs work.