Backend From First Principles

Chapter 29. Webhooks: How the Server Calls You

In traditional client-server architectures, the client initiates all communication. However, when building modern distributed systems, we frequently encounter scenarios where an external service (like Stripe, GitHub, or Clerk) needs to notify our system immediately when an asynchronous event occurs.

This chapter explores Webhooks—the architectural pattern that reverses this flow, allowing external servers to call your application. We will dissect the mechanics of webhook delivery, security strategies to prevent spoofing and replay attacks, reliability patterns like idempotency and exponential backoff, and how to build a production-grade webhook dispatching engine from scratch.

1. What a Webhook Is: Polling vs. Pushing

At its core, a webhook is a user-defined HTTP callback. It is an architectural pattern where an event source (the provider) makes an HTTP POST request to a URL configured by the event consumer (the receiver) to deliver real-time data.

To understand why webhooks are essential, we must contrast them with polling:

Real-Life Analogy

• Polling is like constantly calling a delivery office every 10 minutes to ask, "Is my package here yet?" It wastes your time, clogs their phone lines, and is highly inefficient.
• Webhooks are like giving the delivery office your home address and phone number. You go about your day, and they ring your doorbell (an HTTP POST request) the moment your package arrives.

Comparison: Polling vs. Webhooks

2. One Delivery End-to-End: The Receiver and GitHub's Record

Let us trace a single webhook delivery from a provider (e.g., GitHub) to a receiver (your local application).

When a developer pushes code to a GitHub repository, GitHub's internal event system triggers a webhook. GitHub looks up the configured webhook URL for that repository and dispatches an HTTP POST request containing a JSON payload describing the push event.

GitHub maintains a Delivery Log for every webhook configured. This log is an invaluable debugging tool, recording:

• The unique Delivery ID (e.g., X-GitHub-Delivery).
• The exact HTTP headers sent.
• The raw JSON payload.
• The response status code, headers, and body returned by your server.

3. The Anatomy of a Delivery and Timeouts

A webhook delivery is a standard HTTP request, but it is highly constrained by timeouts.

Because webhook providers dispatch millions of events daily, they cannot afford to keep TCP connections open waiting for slow receiver servers. If your server takes too long to process a request, the provider will forcefully terminate the connection and mark the delivery as a failure.

Common Provider Timeout Limits

• GitHub: 10 seconds
• Stripe: 30 seconds
• Svix / Standard Webhooks: 15 seconds

The Golden Rule of Webhook Receivers

Never perform heavy business logic synchronously inside a webhook handler.

If your handler needs to write to multiple databases, send emails, or call third-party APIs, you must decouple ingestion from processing:

4. Handshakes and Tunnels to Your Laptop

When developing webhooks locally, your application runs on localhost (e.g., http://localhost:8080). External providers like Stripe or GitHub cannot route HTTP requests to your private local network.

To bypass this, developers use tunneling tools to expose local ports to the public internet securely.

GitHub Server (Public Internet)
       |
       | POST https://xyz.ngrok-free.app/webhook
       v
 ngrok Cloud Service
       |
       | Secure Tunnel (WebSocket/TCP)
       v
 ngrok Agent (Running on your laptop)
       |
       | forwards to localhost:8080
       v
Your Local App (localhost:8080/webhook)

Popular Tunneling Tools

• ngrok: The industry standard, offering secure tunnels, request inspection, and replay capabilities.
• Localtunnel: A free, open-source alternative.
• Cloudflare Tunnels: Highly secure, enterprise-grade tunneling.
• Provider-Specific CLIs: Stripe and Shopify provide built-in CLI commands (e.g., stripe listen --forward-to localhost:8080/webhook) that poll their servers for events and forward them locally, bypassing the need for public ingress entirely during development.

5. Five Ways to Prove Who Sent It (and HMAC)

Because webhook endpoints are public HTTP URLs, anyone on the internet can send a fake POST request to your server. You must verify that incoming requests originate from the trusted provider.

Here are five common strategies used to prove sender identity, ordered from weakest to strongest:

Strategy 1: IP Whitelisting (Static IP Ranges)

The provider publishes a list of static IP addresses they send webhooks from. Your firewall or application discards requests from any other IP.

• Downside: IPs change, requiring manual maintenance. It is also vulnerable to IP spoofing in certain network configurations.

Strategy 2: Shared Secret in Query Parameters or Headers

You configure a secret token on the provider's dashboard, and the provider includes it in a header (e.g., Authorization: Bearer my-secret-token).

• Downside: If an attacker intercepts the token (or if it is logged in plain text by a reverse proxy), they can spoof requests indefinitely.

Strategy 3: Mutual TLS (mTLS)

Both the provider and receiver present cryptographic certificates to establish a secure, verified TLS connection.

• Downside: Highly secure, but extremely complex to configure, rotate, and maintain across diverse cloud environments.

Strategy 4: Asymmetric Cryptography (Public/Private Keys)

The provider signs the payload using their private key. You verify the signature using their public key.

• Downside: Computationally expensive and complex to implement correctly.

Strategy 5: HMAC (Hash-based Message Authentication Code) — The Industry Standard

The provider and receiver share a symmetric secret key. The provider hashes the request payload using this secret and sends the resulting signature in a header (e.g., X-Hub-Signature-256 or Stripe-Signature). The receiver recalculates the hash using the same secret and compares it to the incoming signature.

Provider Side:
Signature = HMAC-SHA256(Payload, SharedSecret)

Receiver Side:
CalculatedSignature = HMAC-SHA256(RawPayload, SharedSecret)
IsVerified = ConstantTimeCompare(CalculatedSignature, IncomingSignature)

6. What Exactly Is Signed, and the "Raw-Body" Trap

When implementing HMAC verification, developers frequently fall into the Raw-Body Trap.

Most modern web frameworks automatically parse incoming JSON payloads into native objects or dictionaries before they reach your controller code. However, JSON parsing is non-deterministic: keys can be reordered, spacing can change, and floats can be formatted differently depending on the parser.

If you attempt to serialize a parsed object back to a string to calculate the HMAC, the string will not match the exact byte-for-byte payload sent by the provider, causing signature verification to fail.

Node.js/Express Example: Preserving the Raw Body

To prevent this, you must capture the raw, unparsed request body buffer specifically for your webhook routes:

const express = require('express');
const crypto = require('crypto');
const app = express();

// Middleware to capture raw body for webhook route
app.post('/webhook', express.raw({ type: 'application/json' }), (req, res) => {
    const signature = req.headers['x-signature'];
    const secret = process.env.WEBHOOK_SECRET;

    // Calculate HMAC using the raw buffer
    const hmac = crypto.createHmac('sha256', secret);
    hmac.update(req.body); // req.body is a Buffer containing raw bytes
    const expectedSignature = hmac.digest('hex');

    if (crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expectedSignature))) {
        // Signature is valid; parse JSON now
        const payload = JSON.parse(req.body.toString());
        console.log("Verified Payload:", payload);
        res.sendStatus(200);
    } else {
        res.sendStatus(401); // Unauthorized
    }
});

7. Replays, Constant-Time Comparison, and SSRF

Securing webhooks requires defending against three advanced attack vectors:

1. Replay Attacks

If an attacker intercepts a valid webhook request and its signature, they can replay the exact same request to your server. Even though the signature is valid, the action (e.g., provisioning a subscription) will be executed twice.

• Defense: Providers include a timestamp in the signature header (e.g., t=1672531199,v1=signature). Your server must parse this timestamp and reject requests older than a specific threshold (e.g., 5 minutes). The timestamp must be included in the HMAC signature calculation so an attacker cannot modify it.

2. Timing Attacks

If you compare signatures using standard string comparison (if (sigA === sigB)), the comparison engine returns false as soon as it finds the first mismatched character. Attackers can measure execution times down to nanoseconds to guess the signature character-by-character.

• Defense: Always use a constant-time comparison function (like crypto.timingSafeEqual in Node.js or hmac.Equal in Go) which compares every byte regardless of where a mismatch occurs.

3. Server-Side Request Forgery (SSRF)

If you are building a system that sends webhooks, users will provide URLs for your system to call. A malicious user could input an internal IP address (e.g., http://127.0.0.1:8080/admin or http://169.254.169.254/latest/meta-data/) to force your server to attack your own internal infrastructure.

• Defense: Use an egress proxy like Stripe's open-source Smokescreen to resolve and block private, loopback, and link-local IP addresses before dispatching requests.

8. At-Least-Once Delivery: Idempotency and Ordering

In distributed systems, network partitions are inevitable. Webhook providers prioritize reliability, guaranteeing At-Least-Once Delivery. This means a provider will retry sending an event until they receive a successful response, which occasionally results in your server receiving the exact same event multiple times.

Idempotency: The Ultimate Shield

Your webhook receiver must be idempotent—processing the same event multiple times must yield the same system state as processing it once.

-- Schema design for idempotent processing
CREATE TABLE processed_webhooks (
    event_id VARCHAR(255) PRIMARY KEY,
    processed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

When an event arrives:

1. Start a database transaction.
2. Attempt to insert the event's unique ID (e.g., evt_123) into processed_webhooks.
3. If the insert fails due to a unique constraint violation, abort immediately and return HTTP 200 OK (the event was already processed).
4. If the insert succeeds, process the business logic and commit the transaction.

Out-of-Order Delivery

Webhooks can arrive out of order due to network routing anomalies or retries. If a user updates their profile twice in rapid succession, Update #2 might arrive before Update #1.

• Defense: Include a version number or an updated timestamp in the payload. Your database updates should conditionally apply only if the incoming timestamp is greater than the existing record's timestamp:
  SQL

  Copy

  UPDATE users SET email = :email, updated_at = :incoming_ts 
  WHERE id = :id AND updated_at < :incoming_ts;
  

9. Retries, Backoff, Jitter, and Load Management

When sending webhooks, your dispatching engine will encounter failing receiver servers. To handle this gracefully, you must implement an intelligent retry strategy.

Exponential Backoff

Instead of retrying immediately (which can DDoS a struggling receiver), increase the delay exponentially between attempts:

Jitter (Random Noise)

If a major network outage occurs, thousands of receivers might fail simultaneously. If they all retry at the exact same calculated intervals, they will create massive traffic spikes (thundering herd problem). Adding Jitter randomizes the delay slightly to smooth out the load.

Standard Retry Schedule (e.g., Svix)

• Attempt 1: Immediate
• Attempt 2: 5 seconds
• Attempt 3: 5 minutes
• Attempt 4: 30 minutes
• Attempt 5: 2 hours
• Attempt 6: 5 hours (then move to Dead-Letter Queue / disable webhook)

10. Building Your Own Webhook System: The Outbox and Dispatcher

If you are building a platform that needs to send webhooks to your customers, you must design a highly resilient dispatching architecture. A naive implementation that makes HTTP requests directly inside your API controllers will degrade API performance and lose events if your server crashes.

The Outbox Pattern Architecture

To guarantee that events are never lost, use the Transactional Outbox Pattern. This ensures that your core business state changes and your webhook events are saved within the same atomic database transaction.

Sample Outbox Schema (PostgreSQL)

CREATE TABLE webhook_events (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    target_url TEXT NOT NULL,
    payload JSONB NOT NULL,
    status VARCHAR(50) DEFAULT 'PENDING',
    attempts INT DEFAULT 0,
    next_attempt_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

11. When a Webhook Is the Wrong Tool: The Event Log

While webhooks are excellent for real-time notifications, they are not a silver bullet. They introduce significant operational complexity when handling massive high-throughput streams.

When to Avoid Webhooks

• High-Frequency Events: If your system generates thousands of events per second (e.g., clickstream tracking, IoT sensor data), sending an individual HTTP request for every single event will overwhelm both your dispatchers and your customers' servers.
• Strict Ordering Requirements: If events must be processed in absolute chronological order with zero exceptions, webhooks are too fragile due to network retries and concurrent processing.

The Alternative: The Event Log (Pull-Based Streams)

For high-volume or highly sensitive data synchronization, expose an Event Log (e.g., via Apache Kafka, AWS Kinesis, or a custom cursor-based HTTP API) instead of pushing webhooks.

With an Event Log, the customer pulls batches of events using a cursor (e.g., GET /events?after_id=1042). This shifts the burden of rate-limiting, backpressure, and state tracking entirely to the consumer, guaranteeing perfect ordering and eliminating delivery timeouts.

Summary and Key Takeaways

• Webhooks are push-based HTTP callbacks that allow servers to notify external systems of events in real-time, eliminating the waste of polling.
• Always process webhooks asynchronously. Validate the signature, persist the payload to a queue, and immediately return a 200 OK or 202 Accepted to avoid provider timeouts.
• Never trust incoming webhook payloads without verification. Use HMAC-SHA256 signatures with a shared secret to verify the sender's identity.
• Avoid the Raw-Body Trap. Always perform HMAC calculations on the raw, unparsed byte buffer of the incoming HTTP request body.
• Defend against replay attacks by validating timestamp headers included in the signature, and prevent timing attacks using constant-time string comparison.
• Design for At-Least-Once delivery. Ensure your webhook handlers are strictly idempotent by tracking processed event IDs in your database.
• Implement exponential backoff with jitter when dispatching webhooks to prevent overwhelming failing receiver servers.
• Use the Transactional Outbox Pattern when building a webhook provider to guarantee that events are never lost during system failures.
• Know when to use an Event Log. For high-frequency data streams, pull-based event logs are far more resilient and scalable than push-based webhooks.