Golang Monolith Mastery: System Design & Architecture

This chapter establishes the end-to-end system design for a production-grade classifieds application (analogous to OLX) built as a modular Go monolith. It covers the end-to-end journey from defining functional and non-functional requirements to designing secure media upload pipelines, reliable background processing, transactional integrity, and edge content delivery.

1. Problem Space & Product Mental Model

A classifieds platform connects two distinct user personas with fundamentally asymmetrical workflows:

• The Seller: Needs low friction when publishing listings and uploading high-resolution product photographs. Once created, sellers require full administrative control over their listings (editing price, updating descriptions, deleting sold inventory).
• The Buyer: Primarily consumes data. Buyers query, search, and browse listings across categories (such as Electronics or Home Appliances) with sub-second latency and minimal data overhead.

Rather than fragmenting this platform into premature microservices, the system is designed as a unified, modular Go monolith. A single binary hosts both the user-facing HTTP APIs and the internal asynchronous worker pipelines, maximizing operational simplicity while maintaining clear internal domain boundaries.

2. Functional Requirements Breakdown

The functional surface area defines the operations users can execute on the platform:

1. Authentication & Identity:

   • Users must be able to register (sign up) and authenticate (sign in).
   • Identity context must propagate across all state-mutating requests.

2. Listing Lifecycle (CRUD):

   • Sellers must be able to create, read, update, and soft-delete listings.
   • Listings contain metadata such as title, description, price, condition, category, and associated image references.

3. Product Media Management:

   • Sellers must attach multiple images to any listing.
   • Media processing must occur out-of-band without blocking HTTP request threads.

4. Hierarchical Categorization:

   • System-managed taxonomy (e.g., Electronics, Vehicles, Furniture, Appliances) to organize listings and power search filters.

5. Resource-Level Authorization:

   • A strict ownership model: only the verified creator of a listing has the permission to edit or delete it.

3. Non-Functional Requirements & Engineering Guardrails

Non-functional requirements (NFRs) dictate how the system behaves under load, failure, and security constraints.

Critical NFR Pillars

• Data Security: Credential protection via slow, salted hashing (bcrypt). Secure external media intake using cryptographic pre-signed URLs.
• User Privacy (EXIF Scrubbing): Modern smartphones embed precise exchangeable image file format (EXIF) metadata—including camera model, timestamp, and accurate GPS coordinates—into every photo. Exposing raw images would leak sellers' private home locations. The system must strip all GPS metadata before publishing.
• Worker Reliability & Job Reclamation: Background jobs running in memory can fail if a process crashes mid-operation. The system must implement job leases, crash recovery, and exponential backoff retry schedules.
• Edge Delivery Performance: Browsing listings is read-heavy. Images must be cached at regional edge nodes via a Content Delivery Network (CDN) to ensure low Time to First Byte (TTFB).
• Storage Portability: The system must adhere to standard S3-compatible APIs, allowing zero-code-change switching between object storage providers (e.g., Cloudflare R2 and AWS S3).
• Maintainability & Observability: Clean layered architecture (Handler $\to$ Service $\to$ Repository), version-controlled schema migrations, structured JSON logging, and correlation IDs injected into every HTTP request.

4. Media Storage Architecture: Server Disk vs. Direct-to-Object Storage

A common early architectural pitfall is routing file uploads directly through the web application server to be saved onto local disk storage.

The Anti-Pattern: Local Disk Storage

Why This Fails in Production

1. Horizontal Scaling Blockers: If Server A handles the upload, Server B behind a round-robin load balancer cannot access the image unless a shared, error-prone network file system (NFS) is mounted.
2. Resource Exhaustion: Concurrent uploads of 10–20 MB images consume HTTP worker threads, exhaust server RAM, saturate ingress network bandwidth, and tie up compute resources that should be servicing API requests.

The Production Pattern: Direct Upload via Pre-signed URLs

The scalable solution delegates file intake entirely to external object storage (such as Cloudflare R2 or AWS S3), completely bypassing the Go application server.

• Zero Memory Footprint: The Go monolith never buffers raw image bytes in memory.
• Bandwidth Optimization: Cloud ingress bandwidth is offloaded to the storage provider's global endpoints.
• Strict Expiration Windows: Pre-signed URLs expire after a short duration (e.g., 5–15 minutes) and are constrained to specific MIME types (image/jpeg, image/webp).

5. Media Processing Pipeline & Privacy Protections

Once a raw image is uploaded to object storage, it cannot be served directly to consumers:

1. Privacy Risk: Raw smartphone photos contain EXIF metadata tags:
   • GPSLatitude, GPSLongitude, GPSAltitude
   • Camera serial numbers, device timestamps, software versions
2. Bandwidth Inefficiency: Raw camera captures often exceed 10–15 MB. Serving these directly degrades client performance and escalates bandwidth overhead.

Asynchronous Processing Pipeline

6. Background Worker Architecture: Outbox Pattern & Reliability

To process images asynchronously inside a monolith, the system must trigger workers without tight coupling to the incoming HTTP request.

Ingestion via Transactional Outbox Pattern

Relying on an in-memory channel to notify workers means any unexpected process termination results in lost jobs. Instead, jobs are persisted transactionally alongside the listing entity in PostgreSQL.

-- Conceptual Outbox Schema
CREATE TABLE outbox_jobs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    job_type VARCHAR(64) NOT NULL,
    payload JSONB NOT NULL,
    status VARCHAR(32) NOT NULL DEFAULT 'PENDING',
    retry_count INT NOT NULL DEFAULT 0,
    locked_until TIMESTAMPTZ,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

Because the listing and the background job are committed in the same atomic database transaction, it is mathematically impossible for a listing to exist without its corresponding processing job.

Worker Loop & Deadlock-Free Polling

Workers run as persistent Go routines within the monolith process, polling the outbox_jobs table using row-level locking:

// Conceptual Go Polling Worker Loop
func (w *Worker) Start(ctx context.Context) {
    ticker := time.NewTicker(2 * time.Second)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            w.processNextJob(ctx)
        }
    }
}

To fetch jobs safely across multiple horizontally scaled worker instances without collisions, PostgreSQL's SKIP LOCKED feature is utilized:

-- Atomic job acquisition with lease lock
UPDATE outbox_jobs
SET status = 'PROCESSING',
    locked_until = NOW() + INTERVAL '5 minutes'
WHERE id = (
    SELECT id
    FROM outbox_jobs
    WHERE status = 'PENDING' 
       OR (status = 'PROCESSING' AND locked_until < NOW())
    ORDER BY created_at ASC
    LIMIT 1
    FOR UPDATE SKIP LOCKED
)
RETURNING id, payload, retry_count;

Failure Resilience: Job Reclamation & Exponential Backoff

If a worker node crashes mid-execution (e.g., OOM kill, hardware fault), the job remains marked as PROCESSING.

1. Job Reclamation: The subquery checks (status = 'PROCESSING' AND locked_until < NOW()). If the worker fails to renew its lease before locked_until elapses, the job automatically becomes available for other workers to reclaim.
2. Exponential Backoff: When an external dependency fails (such as an S3 API rate limit or transient network timeout), retrying immediately risks compounding the failure. Retries back off exponentially:

Where $B$ is the base delay interval (e.g., 2 minutes):

• Attempt 1 Failure: Wait $2 \times 2^0 = 2\text{ minutes}$
• Attempt 2 Failure: Wait $2 \times 2^1 = 4\text{ minutes}$
• Attempt 3 Failure: Wait $2 \times 2^2 = 8\text{ minutes}$

If the job reaches maximum retry threshold (e.g., 5 attempts), it transitions to FAILED (Dead Letter Queue state) for manual operator review.

7. Edge Delivery & Regional Caching via Content Delivery Networks (CDN)

Directly fetching images from an object storage bucket on every page view introduces two core bottlenecks:

1. Latency: If the storage bucket is in US-East and a buyer requests an image from Delhi, high Round Trip Times (RTT) degrade user experience.
2. Cost: Object storage providers frequently bill for egress data transfer and GET requests.

CDN Edge Caching Topology

• Latency Reduction: After the initial cache miss, all subsequent users in that geographic region receive the asset directly from edge memory/local NVMe cache.
• Origin Shielding: The storage bucket is protected from traffic spikes, minimizing object storage read costs.

8. Database Architecture: Why PostgreSQL?

The application uses PostgreSQL as its single source of truth.

9. End-to-End High-Level System Architecture

The following diagram unifies the ingestion, storage, processing, and delivery pipelines within the Go monolithic architecture:

Complete End-to-End Sequence Walkthrough

1. Client requests upload authorization:
   POST /api/v1/images/upload-ticket
   Go API generates pre-signed PUT URL using S3-compatible credentials.

2. Client streams raw binary directly to Cloudflare R2:
   PUT https://r2.cloudflarestorage.com/.../raw/item_42.jpg
   R2 stores the raw object and returns 200 OK.

3. Client publishes listing:
   POST /api/v1/listings
   Body includes title, category, price, and the raw image path.

4. Go API writes transaction to PostgreSQL:
   BEGIN;
     INSERT INTO listings ...;
     INSERT INTO listing_images ... (status = 'PENDING');
     INSERT INTO outbox_jobs ... (job_type = 'PROCESS_IMAGE');
   COMMIT;

5. Background Worker Goroutine claims the outbox job:
   Queries outbox via 'FOR UPDATE SKIP LOCKED'.
   Locks job until NOW() + 5 minutes.

6. Worker fetches, transforms, and stores image:
   - Downloads /raw/item_42.jpg from R2.
   - Strips EXIF GPS metadata.
   - Encodes image to optimized .webp format.
   - Uploads result to /processed/item_42.webp in R2.

7. Worker updates PostgreSQL state:
   - Sets listing_images.status = 'READY'.
   - Sets outbox_jobs.status = 'COMPLETED'.

8. Buyer accesses the listing:
   - Fetches metadata via GET /api/v1/listings/42.
   - Loads image via CDN: https://cdn.domain.com/processed/item_42.webp.
   - Cloudflare CDN edge serves image directly from regional cache.

10. Architectural Decision Records (ADRs) & Trade-offs

11. Summary and Key Takeaways

• Monolithic Simplicity: Building a classifieds platform does not warrant microservices at the start. A modular Go monolith containing internal background workers maximizes velocity, operational simplicity, and deployment ease.
• Direct-to-Storage Uploads: Never route raw binary media uploads through web application servers. Issue short-lived pre-signed URLs to stream files directly from the client browser to object storage.
• Mandatory Privacy Scrubbing: Smartphone cameras bake personal GPS coordinates into EXIF tags. Media pipelines must strip all metadata out-of-band before publishing images to the public.
• Transactional Job Outbox: Prevent lost jobs by committing listing records and worker jobs within the same ACID database transaction in PostgreSQL.
• Crash-Resilient Workers: Avoid in-memory background dispatchers that lose state on crashes. Use PostgreSQL's FOR UPDATE SKIP LOCKED with explicit lease timeouts (locked_until) to allow automated job reclamation.
• Exponential Backoff: Always back off exponentially ($2 \to 4 \to 8\text{ minutes}$) when retrying transient failures to prevent hammering struggling upstream dependencies.
• Edge Acceleration: Shield storage origins and minimize buyer latency by terminating media requests at regional CDN edge nodes.