Episode 16: Preventing Zombie Queries Using Context in Go

In distributed client-server architectures, handling long-running or stalled operations gracefully is critical to maintaining backend resilience. When a client cancels an HTTP request, backend database queries triggered by that request often continue to run in the background if cancellation signals are not propagated, creating unmonitored "zombie queries" that consume database CPU, memory, and connection pool resources. Go's standard context.Context package provides a standardized mechanism for cancellation propagation, deadlines, and request-scoped metadata, allowing developers to immediately terminate orphaned database operations and prevent systemic resource exhaustion.

1. The Production Reality: Happy Path vs. Zombie Queries

In local development and simple happy-path scenarios, backends and databases respond almost instantaneously. A client issues a request, the backend queries the database, the database quickly yields results, and the backend returns an HTTP response. In this ideal cycle, performance bottlenecks and connection leaks remain concealed.

Client
   |
   | 1. Sends HTTP Request (GET /listings)
   v
Go Backend API
   |
   | 2. Executes SQL Query
   v
PostgreSQL Database
   |
   | 3. Returns Result Set (Fast)
   v
Go Backend API
   |
   | 4. Returns HTTP 200 OK
   v
Client

Production environments, however, inevitably diverge from the happy path:

• Network jitter, packet drops, or cross-region latency introduce unpredictable overhead.
• Database tables scale to millions of rows, causing unindexed or poorly optimized queries to slow down significantly.
• High concurrent load causes lock contention, forcing queries to queue up.

Consider a scenario where an unoptimized query takes 10 to 20 seconds to finish. Real-world users rarely wait 20 seconds; after 2 to 4 seconds, the user typically closes the browser tab, navigates to another route, or cancels the request.

When the HTTP connection drops, the naive backend server remains completely unaware of the cancellation. It allows the database query to run its full course for the remaining 16 to 18 seconds, returning data to a server that no longer has a client to receive it. These orphaned, background-executing statements are known as Zombie Queries.

2. Anatomy of a Zombie Query: Resource Exhaustion and Connection Pools

A zombie query continues to consume hardware and operational capacity long after its original caller has abandoned the operation.

The compounding damage of zombie queries manifests across several vectors:

1. Database CPU and Memory Saturation: Complex joins, table scans, and aggregate operations consume execution threads and buffer cache on the database engine.
2. Database Connection Pool Depletion: Applications communicate with databases through a managed connection pool of fixed size (e.g., 25, 50, or 100 connections). Each zombie query monopolizes a connection slot for its entire duration.
3. Cascading Service Starvation: When 20 or 30 slow queries run concurrently and clients abandon them, all available connection pool slots become trapped by abandoned operations. Incoming legitimate, fast queries from active users are blocked waiting for an open connection, eventually leading to HTTP 504 Gateway Timeouts and complete backend failure.

3. Real-World Engineering Failure Modes: The Jenkins and Third-Party Hang Incident

Zombie workloads are not exclusive to relational databases; they occur whenever an application communicates with external systems over network boundaries without cancellation boundaries.

Production Post-Mortem Example: The Jenkins CI/CD Freeze

In a production engineering system, an automated orchestration service triggered CI/CD pipeline builds programmatically via the Jenkins REST API. During an incident, the remote Jenkins instance stopped responding to status polling queries, leaving HTTP connections hanging indefinitely.

Because the Go client did not pass a cancellable context or an explicit deadline with the HTTP client requests:

• Each spawned goroutine hung waiting on a socket read that never resolved.
• Hundreds of incoming pipeline events triggered new goroutines, which joined the queue of blocked routines.
• Memory usage climbed as goroutine stacks and socket buffers accumulated over an hour.
• The application server ran out of file descriptors and memory, eventually crashing.

A similar failure occurs when calling third-party services such as Large Language Model (LLM) APIs, payment gateways, or external microservices. Without context-driven timeouts and client cancellation propagation, any upstream delay ripples backwards, locking downstream worker threads and connection pools.

4. The Three Pillars of Go's Context Package

In Go, the standard library package context provides the context.Context interface, which is the foundational primitive for managing lifecycles across API boundaries and concurrent goroutines.

type Context interface {
    Deadline() (deadline time.Time, ok bool)
    Done() <-chan struct{}
    Err() error
    Value(key any) any
}

The Context type fulfills three distinct responsibilities in production architectures:

1. Cancellation Signals

Allows parent processes to broadcast cancellation events to child operations. When an HTTP client aborts a connection, the web server closes the context's internal channel (Done()), notifying listening database drivers, external API clients, and child goroutines to halt immediately.

2. Deadlines and Timeouts

Enforces strict ceilings on how long an operation is permitted to run. Functions such as context.WithTimeout(parent, duration) or context.WithDeadline(parent, time) attach an automated timer to the context. If the database or network fails to return a response before the timer elapses, the context triggers cancellation automatically.

3. Request-Scoped Values

Carries metadata across execution layers within the lifecycle of a single request via context.WithValue(parent, key, val). This is commonly used in authentication and observability middleware to inject authenticated User IDs, permission scopes, correlation IDs, and distributed tracing spans into downstream business logic.

5. Empirical Reproduction: Simulating Slow Queries with PostgreSQL pg_sleep

To observe a zombie query directly, we can deliberately introduce an artificial delay into an existing SQL query using PostgreSQL's native pg_sleep(seconds) function.

The Targeted Listing Query

Consider a query fetching real estate listings:

SELECT 
    id, 
    title, 
    description, 
    price, 
    created_at,
    pg_sleep(20)
FROM listings;

Note on Row-by-Row Execution: When pg_sleep(t) is placed in the SELECT column list of a query, PostgreSQL executes the sleep function for each matching row in the result set. If the table contains 2 rows, pg_sleep(20) keeps the query active for $2 \times 20 = 40$ seconds.

The Naive Query Handler (Vulnerable to Zombie Queries)

Prior to implementing context, the Go handler executes queries using the standard db.Query() method:

// Naive approach: Does not accept a Context
func (s *ListingStore) GetAllListings() ([]Listing, error) {
    query := `
        SELECT id, title, description, price, created_at, pg_sleep(20)
        FROM listings
    `
    // db.Query has no mechanism to receive cancellation signals
    rows, err := s.db.Query(query)
    if err != nil {
        return nil, err
    }
    defer rows.Close()

    var listings []Listing
    for rows.Next() {
        var l Listing
        var sleepVal any
        if err := rows.Scan(&l.ID, &l.Title, &l.Description, &l.Price, &l.CreatedAt, &sleepVal); err != nil {
            return nil, err
        }
        listings = append(listings, l)
    }

    return listings, nil
}

6. Diagnosing Running Queries in PostgreSQL

To verify whether a database is running zombie queries after an HTTP client disconnects, we can inspect PostgreSQL's process activity catalog using the pg_stat_activity system view.

Diagnostic SQL Script

SELECT 
    pid, 
    usename, 
    state, 
    query, 
    query_start, 
    now() - query_start AS duration
FROM pg_stat_activity
WHERE state = 'active'
  AND query NOT LIKE '%pg_stat_activity%'
ORDER BY duration DESC;

The Reproduction Experiment Walkthrough

Because s.db.Query(...) was executed without a context binding, the database engine remains unaware of the client's disconnection. The PostgreSQL process runs uninterrupted for the full 40 seconds, wasting hardware resources and locking an open connection pool slot.

7. The Solution: Propagating Request-Scoped Contexts to the Database

To terminate queries the moment a client disconnects, the HTTP server must pass its request-scoped context directly down into the database driver.

Every incoming standard Go HTTP request (*http.Request) already includes a managed context accessible via r.Context().

ctx := r.Context()

How r.Context() Handles Cancellation

The standard library net/http server manages this context throughout the request lifecycle:

• It returns an active context that defaults to context.Background() under the hood.
• When an incoming client terminates the connection prematurely, the server automatically signals cancellation to r.Context().
• Passing this context to the database driver allows it to abort executing statements on the database server.

8. Code Evolution: Transitioning from Naive Querying to Context-Aware Operations

To implement full cancellation propagation, we replace all uncontextualized database operations (Query, QueryRow, Exec) with their context-aware counterparts (QueryContext, QueryRowContext, ExecContext).

Refactoring the Store Layer

Before (Naive, Uncontextualized):

// Vulnerable to zombie execution
func (s *ListingStore) GetAll() ([]Listing, error) {
    query := `SELECT id, title, price, created_at FROM listings`
    rows, err := s.db.Query(query)
    // ...
}

After (Context-Aware, Production-Grade):

package store

import (
    "context"
    "database/sql"
)

type Listing struct {
    ID          string  `json:"id"`
    Title       string  `json:"title"`
    Description string  `json:"description"`
    Price       float64 `json:"price"`
}

type ListingStore struct {
    db *sql.DB
}

// GetAllListings accepts context as its first parameter by Go convention
func (s *ListingStore) GetAllListings(ctx context.Context) ([]Listing, error) {
    query := `
        SELECT id, title, description, price
        FROM listings
    `
    
    // QueryContext binds query execution to the lifecycle of ctx
    rows, err := s.db.QueryContext(ctx, query)
    if err != nil {
        return nil, err
    }
    defer rows.Close()

    var listings []Listing
    for rows.Next() {
        var l Listing
        if err := rows.Scan(&l.ID, &l.Title, &l.Description, &l.Price); err != nil {
            return nil, err
        }
        listings = append(listings, l)
    }

    if err := rows.Err(); err != nil {
        return nil, err
    }

    return listings, nil
}

// DeleteListing demonstrates context propagation for write operations
func (s *ListingStore) DeleteListing(ctx context.Context, id string) error {
    query := `DELETE FROM listings WHERE id = $1`
    
    // ExecContext cancels the ongoing delete operation if ctx is aborted
    _, err := s.db.ExecContext(ctx, query, id)
    if err != nil {
        return err
    }
    
    return nil
}

Refactoring the Controller / Handler Layer

In the HTTP handler, extract the request context and forward it as the first parameter to the storage layer:

package handler

import (
    "encoding/json"
    "log"
    "net/http"
)

type ListingHandler struct {
    store ListingStoreInterface
}

func (h *ListingHandler) HandleGetListings(w http.ResponseWriter, r *http.Request) {
    // 1. Extract the request-scoped context
    ctx := r.Context()

    // 2. Propagate context down to the database store
    listings, err := h.store.GetAllListings(ctx)
    if err != nil {
        // If the client canceled, ctx.Err() will evaluate to context.Canceled
        if ctx.Err() == context.Canceled {
            log.Printf("Request canceled by client: %v", err)
            return
        }
        
        log.Printf("Failed to retrieve listings: %v", err)
        http.Error(w, "Internal Server Error", http.StatusInternalServerError)
        return
    }

    // 3. Serialize and return response
    w.Header().Set("Content-Type", "application/json")
    if err := json.NewEncoder(w).Encode(listings); err != nil {
        log.Printf("Failed to encode response: %v", err)
    }
}

Compiler Note on log.Printf: In Go, avoid using the %w format specifier inside log.Printf(). %w is only valid within fmt.Errorf() for error wrapping. Use %v or %s for general logging to avoid compiler or linter diagnostics.

9. Verification: Validating Real-Time Query Cancellation

With QueryContext(ctx, ...) wired to r.Context(), we can re-run our diagnostic reproduction experiment.

1. Send Request: Trigger the endpoint configured with pg_sleep(20).
2. Inspect Activity: Run the diagnostic SQL query in the PostgreSQL console. The query appears as state = 'active'.
3. Cancel Client Request: Click "Cancel" in Insomnia or close the client connection.
4. Re-Inspect Activity: Immediately re-run the diagnostic query in the PostgreSQL console.

Result: The slow query disappears from pg_stat_activity immediately upon client cancellation. The database stops executing pg_sleep, releases the CPU thread, and returns the pooled connection to the idle pool.

10. Architectural Comparison: Database Query Execution Methods

The database/sql package provides parallel APIs for all query operations. In production systems, the context-aware variants should always be preferred.

11. Summary and Key Takeaways

• The Problem of Zombie Queries: When an HTTP client aborts a request, unmanaged backend operations continue running on the database. Over time, these orphaned queries drain CPU, exhaust memory, and monopolize connection pool resources.
• Connection Pool Exhaustion: Connection pools have strict upper limits. Stalled zombie queries retain their acquired connections, starving incoming requests and causing cascading system-wide outages.
• Wider Distributed Applicability: Stalled operations occur across all external network boundaries, including third-party HTTP integrations, CI/CD runners (like Jenkins), and microservice calls.
• Go Context Architecture: The context.Context interface coordinates three core concerns across application layers: cancellation signals, execution deadlines/timeouts, and request-scoped metadata.
• Leveraging r.Context(): Standard library HTTP requests automatically supply a context that detects client-side disconnections via the underlying TCP connection state.
• Always Use Context-Aware DB Methods: Replace legacy db.Query(), db.QueryRow(), and db.Exec() calls with db.QueryContext(), db.QueryRowContext(), and db.ExecContext().
• First Parameter Idiom: In Go, contexts should always be explicitly passed as the first parameter of a function (ctx context.Context), never nested inside a struct.
• Immediate Production Gain: Propagating request contexts to database drivers ensures that when a client abandons an HTTP request, all associated database statements are terminated immediately, preserving system stability.