Observability Zero to Hero: OpenSearch Stack & Managed Log Analytics

This chapter explores the fundamentals of log aggregation and search engines, focusing on OpenSearch as a powerful open-source alternative to Elasticsearch. It covers the architectural differences between self-hosted and managed search engines, provides a step-by-step guide to provisioning a managed OpenSearch instance using Aiven, and demonstrates how to collect and route Kubernetes cluster logs to OpenSearch using Fluent Bit.

1. The Problem: The Log Explosion and the Need for Search Engines

Modern software architectures rely heavily on distributed systems, microservices, and containerized environments like Kubernetes. To maintain visibility, organizations configure logging at multiple layers of their infrastructure:

• Application Logs: Runtime exceptions, transaction traces, and business logic events.
• Database Logs: Slow queries, connection pool exhaustion, and transaction states.
• Infrastructure Logs: System metrics, kernel events, and network traffic.
• Orchestration Logs: Kubernetes API server events, kubelet logs, and container lifecycles.

While these logs are invaluable for debugging, the sheer volume of data generated at scale is overwhelming. During an outage, a Site Reliability Engineer (SRE) or DevOps engineer cannot manually log into dozens of servers or containers to inspect individual log files.

Without a centralized search engine, identifying the root cause of an issue is highly inefficient. Organizations need a way to aggregate logs (collect them from various sources) and store them in a centralized log storage service equipped with advanced, real-time search capabilities.

2. Real-World Analogy: The E-Commerce 502 Gateway Crisis

The Scenario

Imagine you are a Site Reliability Engineer (SRE) managing a high-traffic e-commerce application. During a peak shopping event, customers begin reporting 502 Bad Gateway errors at checkout.

The Manual Approach (No Search Engine)

To troubleshoot, you must:

1. Identify which microservices are failing.
2. SSH into multiple running containers or run kubectl logs across dozens of pods (Checkout, Payment, Inventory, Cart).
3. Manually parse thousands of lines of raw text logs.
4. Correlate timestamps across different servers to find where the failure originated.

This manual process delays recovery, increases Mean Time to Resolution (MTTR), and directly impacts business revenue.

The Search Engine Approach

With a centralized search engine like OpenSearch in place:

1. All logs from all microservices, databases, and gateways are continuously streamed to a single location.
2. The SRE runs a single query for the term "502" or "Error" across the entire cluster.
3. Within milliseconds, the search engine processes gigabytes of log data and returns a visual timeline showing exactly which service started throwing 502 errors first, along with the corresponding database connection timeout log.

3. The Architecture of Log Aggregation

A standard production log aggregation pipeline consists of three main layers: the Producer, the Shipper/Collector, and the Search/Storage Engine.

Kubernetes Pods (Application Logs)
       |
       | (Writes logs to stdout/stderr on host node)
       v
Fluent Bit (DaemonSet Collector)
       |
       | (Filters, parses, and aggregates logs)
       | (Pushes logs via HTTPS)
       v
Aiven for OpenSearch (Managed Search Engine)
       |
       | (Indexes and stores log documents)
       v
OpenSearch Dashboards (Visualization UI)
       |
       | (SRE queries and visualizes logs)
       v
SRE / DevOps Engineer

Algorithmic Efficiency of Search Engines

Traditional text search using tools like grep on raw files scales linearly with the size of the log data, resulting in a time complexity of $\mathcal{O}(N)$, where $N$ is the total number of log lines.

Search engines like OpenSearch solve this by parsing incoming log strings and building an inverted index. This data structure maps terms (words) to their exact locations in the database. Consequently, searching for a specific error code or keyword is optimized from a linear scan to a highly efficient lookup, scaling at $\mathcal{O}(1)$ or $\mathcal{O}(\log N)$ depending on the query complexity.

4. OpenSearch vs. Elasticsearch: The Licensing Shift

OpenSearch and Elasticsearch share a common history. To understand why OpenSearch exists, it is necessary to examine the licensing changes that occurred in the open-source ecosystem.

The History of the Fork

Elasticsearch was originally released under the permissive Apache 2.0 open-source license. It became the industry standard for log analytics, often deployed as part of the ELK (Elasticsearch, Logstash, Kibana) stack.

In 2021, the parent company (Elastic) transitioned Elasticsearch from the Apache 2.0 license to a dual-licensing model under the Server Side Public License (SSPL) and the Elastic License. This change restricted how other companies could use the software.

Key Differences and Licensing Impacts

5. Self-Hosted vs. Managed OpenSearch (The Case for Aiven)

While OpenSearch is completely free and open-source, running it at scale presents significant operational challenges.

The Challenges of Self-Hosting

1. Infrastructure Management: OpenSearch is a distributed system. Operating a production-grade cluster requires configuring master nodes, data nodes, sharding strategies, and replica counts.
2. Upgrades and Patching: Keeping up with security patches, operating system updates, and minor/major version upgrades requires dedicated engineering hours and risks data loss if executed incorrectly.
3. Scaling and Storage: Managing disk space, scaling up nodes during traffic spikes, and configuring cold/warm storage tiers to optimize costs is operationally complex.

The Managed Service Alternative (Aiven)

Managed service providers like Aiven abstract away the operational overhead of running open-source data technologies. Aiven allows teams to deploy fully managed instances of OpenSearch, Apache Kafka, PostgreSQL, ClickHouse, and Valkey across multiple cloud providers (AWS, GCP, Azure, DigitalOcean) through a unified interface.

6. Step-by-Step Evolution: Provisioning Managed OpenSearch on Aiven

Aiven organizes resources using a clean, hierarchical structure:

Step 1: Create an Account and Project

1. Sign up at Aiven (no credit card required; includes $300 in free trial credits).
2. Create a new project named payments-dev-environment-demo.

Step 2: Provision the OpenSearch Service

1. Select OpenSearch from the list of available services.
2. Choose a cloud provider (e.g., AWS) and select a region close to your workload.
3. Select a service plan. For development and testing, choose the Startup-4 plan (single-node deployment). For production, multi-node plans with dedicated master nodes are recommended.
4. Click Create Service. The instance will provision and transition to the Running state within a few minutes.

Step 3: Retrieve Connection Details

Once the service is running, navigate to the Overview tab to retrieve the connection parameters:

• Service URI: The full connection string.
• Host: The hostname of the OpenSearch cluster.
• Port: The connection port (e.g., 11937).
• User: The administrative user (default: avnadmin).
• Password: The auto-generated secure password.

7. Setting Up the Kubernetes Cluster and Log Generator

To demonstrate real-world log aggregation, we will spin up a local Kubernetes cluster using kind (Kubernetes in Docker) and deploy a synthetic log generator application.

Step 1: Create the Kubernetes Cluster

Run the following command to create a lightweight cluster named payments-open-search-demo:

kind create cluster --name payments-open-search-demo

Step 2: Define the Log Generator Application

Create a file named log-generator.yaml. This application uses a lightweight busybox image to continuously write structured log messages to standard output (stdout) every 5 seconds.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: log-generator
  labels:
    app: log-generator
spec:
  replicas: 1
  selector:
    matchLabels:
      app: log-generator
  template:
    metadata:
      labels:
        app: log-generator
    spec:
      containers:
      - name: log-generator
        image: busybox
        args:
        - /bin/sh
        - -c
        - >
          while true; do
            echo "$(date) [INFO] Application started successfully.";
            sleep 5;
            echo "$(date) [DEBUG] Debugging application logic - connection pool active.";
            sleep 5;
            echo "$(date) [ERROR] An error occurred: Connection timed out.";
            sleep 5;
          done

Step 3: Deploy the Application

Apply the manifest to your cluster and verify that logs are being generated:

# Deploy the application
kubectl apply -f log-generator.yaml

# Verify the pod is running
kubectl get pods

# Check the generated logs
kubectl logs deployment/log-generator

8. Configuring Fluent Bit as the Log Shipper

Fluent Bit is a lightweight, high-performance log processor and forwarder. It runs as a DaemonSet on Kubernetes, meaning one instance of Fluent Bit runs on every node of the cluster to collect logs from all running pods.

Step 1: Create the Logging Namespace

Fluent Bit should be isolated in its own namespace:

kubectl create namespace logging

Step 2: Define the Fluent Bit ConfigMap

Create a file named fluent-bit-config.yaml. This configuration instructs Fluent Bit to read logs from the standard host path /var/log/containers/* (Input) and route them to our managed OpenSearch instance (Output).

Note: Replace the Host, Port, HTTP_User, and HTTP_Password placeholders with your actual Aiven OpenSearch connection details.

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: Server, Input, Filters and Output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-opensearch.conf

  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        Keep_Log            Off
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On

  output-opensearch.conf: |
    [OUTPUT]
        Name            opensearch
        Match           *
        Host            <YOUR_AIVEN_OPENSEARCH_HOST>
        Port            <YOUR_AIVEN_OPENSEARCH_PORT>
        HTTP_User       avnadmin
        HTTP_Password   <YOUR_AIVEN_OPENSEARCH_PASSWORD>
        TLS             On
        Index           kubernetes
        Type            _doc
        Suppress_Type_Name On

  parsers.conf: |
    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

Apply the ConfigMap:

kubectl apply -f fluent-bit-config.yaml

Step 3: Define the Fluent Bit DaemonSet

Create a file named fluent-bit-daemonset.yaml. This manifest deploys Fluent Bit to all nodes, mounts the configuration from the ConfigMap, and sets up the necessary ServiceAccount and Role-Based Access Control (RBAC) permissions so Fluent Bit can query the Kubernetes API for pod metadata.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit
  namespace: logging
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:3.0.4
        imagePullPolicy: Always
        ports:
          - containerPort: 2020
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config

Apply the DaemonSet:

kubectl apply -f fluent-bit-daemonset.yaml

9. Integrating Kubernetes with OpenSearch

Once both components are deployed, verify that Fluent Bit is running and successfully communicating with the managed OpenSearch instance.

Step 1: Verify Fluent Bit Pod Status

Ensure the Fluent Bit pod is running without errors:

kubectl get pods -n logging

Step 2: Inspect Fluent Bit Logs

Check the logs of the Fluent Bit pod to confirm it has successfully connected to the OpenSearch endpoint and is shipping data:

kubectl logs -n logging -l k8s-app=fluent-bit-logging --tail=50

A successful connection will show initialization logs and no connection timeout or authentication errors.

Step 3: Verify Index Creation in Aiven Console

1. Log in to your Aiven Console.
2. Navigate to your OpenSearch service.
3. Click on the Indexes tab.
4. You should see a newly created index named kubernetes (or kubernetes-<date>).
5. Note the document count and index size increasing, which confirms that logs are actively streaming from your local cluster to the cloud.

10. Visualizing and Querying Logs in OpenSearch Dashboards

OpenSearch Dashboards provides a web-based user interface for searching, visualizing, and analyzing log data.

Step 1: Access OpenSearch Dashboards

1. In the Aiven Console, locate the OpenSearch Dashboards connection information.
2. Click on the Service URI link.
3. Enter the administrative username (avnadmin) and the secure password provided in the Aiven Console.

Step 2: Create an Index Pattern

Before you can search your logs, you must define an index pattern to tell Dashboards which indexes to query:

1. Click the menu icon (three horizontal lines) in the top-left corner and navigate to Stack Management > Index Patterns.
2. Click Create index pattern.
3. Enter kubernetes* as the index pattern name. The interface will automatically match the active index streamed by Fluent Bit.
4. Click Next step.
5. Select I don't want to use the time filter (or select @timestamp if your logs include structured timestamps).
6. Click Create index pattern.

Step 3: Discover and Query Logs

1. Navigate back to the main menu and click Discover.
2. Select your kubernetes* index pattern from the dropdown.
3. You will see a real-time stream of logs from your Kubernetes cluster.
4. Use the search bar to run full-text search queries. For example, search for the string:
   TEXT

   Copy

   "An error occurred: Connection timed out"
   

5. OpenSearch will instantly filter the logs, showing exactly when and how many times this error occurred.

11. Production Considerations: Alerting, Monitoring, and Backups

When running OpenSearch in production, logging is only the first step. You must also configure proactive monitoring and disaster recovery.

Automated Backups

With Aiven for OpenSearch, backups are fully automated. The service automatically takes snapshots of your data every 60 minutes and stores them securely. This ensures that you can restore your cluster to a healthy state in the event of data corruption or accidental deletion.

Proactive Alerting

Instead of waiting for users to report issues (like the e-commerce 502 gateway crisis), you can configure OpenSearch to monitor logs automatically:

1. Monitors: Set up a monitor in OpenSearch Dashboards that runs a query (e.g., counting the frequency of 502 or 500 HTTP status codes) at regular intervals (e.g., every 1 minute).
2. Triggers: Define a threshold trigger (e.g., if the count of 502 errors is greater than 10 within a 5-minute window).
3. Destinations: Configure notifications to alert your engineering team immediately via integrations like Slack, PagerDuty, custom webhooks, or email.

12. Summary and Key Takeaways

• Log Aggregation is Essential: Centralized log storage and indexing are critical for debugging modern, distributed microservice architectures at scale.
• Search Engine Efficiency: OpenSearch uses an inverted index to optimize full-text search queries, reducing search time complexity from a slow linear scan $\mathcal{O}(N)$ to an efficient $\mathcal{O}(1)$ or $\mathcal{O}(\log N)$ lookup.
• The OpenSearch Fork: OpenSearch was created as a community-driven, Apache 2.0-licensed fork of Elasticsearch after Elastic transitioned to a restrictive, proprietary licensing model.
• Fully Open-Source Security: Unlike Elasticsearch, which locks advanced security and RBAC features behind a paid tier, OpenSearch provides all enterprise security features completely free of charge.
• Managed Services Reduce Overhead: Utilizing a managed service provider like Aiven eliminates the operational complexity of provisioning, scaling, patching, and backing up distributed OpenSearch clusters.
• Fluent Bit as a DaemonSet: Deploying Fluent Bit as a Kubernetes DaemonSet ensures that logs are automatically collected from every node and container in the cluster.
• End-to-End Visibility: Integrating Kubernetes with OpenSearch and OpenSearch Dashboards allows SREs to run real-time queries, build visual dashboards, and configure proactive alerts to resolve production issues before they impact customers.