System Design: Design Distributed Logging System (Splunk / Logstash)

A distributed logging platform is an online system that collects, parses, normalizes, and indexes logs from millions of distributed microservices across the globe. Industry-standard equivalents include Splunk, OpenObserve, and Logstash.

1. Why Do We Need a Distributed Logging System?

In a monolithic architecture, debugging is straightforward because all logs reside in a single local file. However, in a modern microservices architecture running across multiple clusters and data centers, application logs are scattered across thousands of ephemeral containers.

Without a centralized logging platform:

• Developers must SSH into individual servers or containers to inspect logs.
• Log aggregation during high-throughput traffic spikes becomes nearly impossible.
• Correlating a single user transaction across ten downstream microservices requires manually piecing together disjointed text files.

2. Core Mental Model & Internal Log Flow

Understanding how logs travel from a running application to a centralized database is crucial for designing the ingestion pipeline. Logs do not stream directly from application code to enterprise search engines; instead, they follow a rigorous multi-step pipeline.

Application (Java / Log4j)
   |
   | writes log string
   v
Standard I/O Streams (stdout / stderr)
   |
   | captured by container runtime
   v
Node Container Log Files (/var/log/raw.logs)
   |
   | read by node agent (Fluentbit / Custom Agent)
   v
Ingestion Service (Distributed Logging Backend)

Real-Life Analogy: The Postal Sorting Facility

Think of microservices as individual post offices scattered across the country generating millions of package slips (logs) every hour.

• The Local Agent (Fluentbit) acts as the local mail carrier who picks up raw slips from each post office bin.
• The Ingestion Service & Flink Engine act as the central postal sorting facility that opens packages, inspects them for contraband (validation), standardizes the address formats (normalization), and files them into appropriate filing cabinets (Cassandra) or display boards (Elasticsearch).

3. Functional Requirements

The system must satisfy four primary functional requirements during an interview:

• 1. Multi-Source Log Ingestion: Ingest logs seamlessly from millions of microservice instances distributed globally.
• 2. Dual Ingestion Modes:
  • Near Real-Time Ingestion: Logs transmitted instantly as they are generated.
  • Offline / Batch Submission: For offline or legacy devices that store logs locally and flush them in batches once internet connectivity is restored.
• 3. Validation, Parsing, and Normalization: Sanitize incoming logs (checking for malicious content), parse unstructured text, and normalize them into a uniform schema before persistence.
• 4. Interactive Dashboard & Search UI: Provide a search interface allowing developers and support engineers to query logs by keywords, trace IDs, and log levels.

4. Non-Functional Requirements & Scale Estimations

• Scale: Millions of log events per hour (translating to thousands of requests per second).
• Latency: Low latency for near real-time ingestion and search retrieval.
• CAP Theorem Priority: Availability (A) over Consistency (C). If a logging node drops temporarily, microservices must never block or crash; log ingestion must remain continuously available, even if logs take a few extra seconds to appear.
• Reliability & Data Retention (TTL): Zero data loss during ingestion, coupled with a strict Time-To-Live (TTL) policy (e.g., keeping hot logs for 14 days, warm logs for 45 days, and cold archive storage indefinitely).

5. Core Entities

6. API Endpoint Design

While interviewers prefer focusing on high-level architecture, having a clean REST API specification demonstrates production-readiness.

7. High-Level Architecture (HLD)

8. Low-Level Design (LLD) & Component Deep Dive

To prevent bottlenecks and handle millions of events per hour, a naive direct-to-database architecture must be replaced with an asynchronous, decoupled streaming pipeline.

1. Ingestion & Streaming Buffer (Kafka)

Directly writing millions of events per minute from injection services into a database causes severe lock contention and crashes.

• Solution: Incoming logs are dumped immediately into an Apache Kafka cluster (raw-logs topic). This acts as a resilient buffer, decoupling ingestion throughput from downstream processing speeds.

2. Stream Processing with Apache Flink

Instead of writing custom microservices for validation, parsing, normalization, enrichment, and deduplication, we utilize Apache Flink.

• Flink consumes raw log events from Kafka in real time.
• It sanitizes payloads, drops malicious strings, normalizes formats into a standard JSON schema, and routes clean data downstream.

3. Tiered Storage Strategy

Storing all logs in high-performance search indices for years is cost-prohibitive. We implement a three-tier storage model:

9. Schema Design

Client Metadata DB (PostgreSQL)

Stores organizational client registrations and authentication tokens.

CREATE TABLE client_metadata (
    client_id VARCHAR(64) PRIMARY KEY,
    client_name VARCHAR(255) NOT NULL,
    token_id VARCHAR(255) UNIQUE NOT NULL,
    token_ttl TIMESTAMP NOT NULL,
    environment VARCHAR(32) NOT NULL -- DEV, STAGING, PROD, QA
);

Log Storage Schema (Cassandra DB)

Cassandra is right-optimized and uses composite primary keys tailored around common query access patterns (such as querying by service name and log level).

CREATE KEYSPACE distributed_logs 
WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 3};

CREATE TABLE distributed_logs.logs_by_service (
    service_name text,
    log_level text,
    bucket_date text, -- e.g., '2023-10-25'
    timestamp timestamp,
    trace_id text,
    message text,
    metadata map<text, text>,
    PRIMARY KEY ((service_name, log_level, bucket_date), timestamp, trace_id)
) WITH CLUSTERING ORDER BY (timestamp DESC, trace_id ASC);

10. Advanced Follow-Up: Automated Alerting System

Interviewers often ask how to trigger real-time alerts (e.g., PagerDuty emails or SMS) when critical error logs spike.

• Error Detection: Flink identifies whether an incoming log level is an ERROR or fatal exception.
• Alert Topic & Service: Critical errors are duplicated into an alert-topic in Kafka. An independent Alert Service consumes these messages.
• Rule Engine & Preferences: The Alert Service checks rules stored in an Alert DB (e.g., trigger an alert only if the checkout service encounters > 10 errors in 1 minutes). It then inspects the client's notification preference (Pager, Email, SMS) stored in Postgres and dispatches the alert via third-party providers.

Summary and Key Takeaways

• Distributed Logging architecture bridges microservices and centralized monitoring tools like Splunk and OpenObserve.
• Agent-based collection (using tools like Fluentbit) tails raw container log files (/var/log/raw.logs) and forwards them asynchronously.
• Decoupling via Kafka ensures ingestion services never crash under sudden traffic spikes or database latency.
• Apache Flink serves as a powerful stream processor for real-time validation, parsing, deduplication, and normalization.
• Tiered storage optimizes costs by splitting logs across Elasticsearch (Hot, 14 days), Cassandra (Warm, 45 days), and Amazon S3 (Cold, archival).
• Cassandra schema design must optimize around query access patterns using composite partition keys (service_name, log_level, bucket_date).
• Real-time alerting workflows leverage separate Kafka topics, rule evaluation engines, and configurable notification sinks (Pager, Email, SMS).