Observability Zero to Hero | Day-5: Logging with EFK Stack (Elasticsearch, FluentBit, Kibana)

This chapter transitions from the first pillar of observability (metrics) to the second pillar: logging. It covers the fundamental role of logs in distributed systems, the architectural design of the EFK (Elasticsearch, FluentBit, Kibana) stack, and provides a step-by-step guide to deploying and configuring this centralized logging solution on a Kubernetes cluster.

1. Introduction to Logging & The "Why"

In observability, metrics tell you what is failing by tracking system states and performance indicators over time. However, metrics alone cannot explain why a failure occurred. This is where logs come in.

Logs are discrete, time-stamped text messages written by developers within application code to record execution milestones, errors, and system states. They provide the deep context required to debug, troubleshoot, and perform root-cause analysis when an application behaves unexpectedly.

The Three Pillars of Observability Relationship

2. Real-World Analogy: Code Execution With and Without Logs

To understand the value of logging, consider a simple program designed to add two numbers.

Scenario A: The Naive Approach (No Logging)

A developer writes a script that expects two command-line arguments, adds them, and prints the result.

#!/bin/bash
# Naive addition script without logs
read a
read b
c=$((a + b))
echo $c

When an end-user runs this script, they are presented with a blank, blinking cursor. The user has no context:

• Is the program waiting for a string, an integer, or a boolean?
• Has the program hung, or is it actively processing?
• If it fails, did it fail because of bad input, a syntax error, or system resource exhaustion?

Scenario B: The Better Approach (With Logging)

The developer enriches the script with clear log messages indicating the application's state at each critical step.

#!/bin/bash
# Production-ready script with logs
echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') - Starting the addition process..."
read -p "Enter first number: " a
echo "[DEBUG] $(date '+%Y-%m-%d %H:%M:%S') - Received first input: $a"
read -p "Enter second number: " b
echo "[DEBUG] $(date '+%Y-%m-%d %H:%M:%S') - Received second input: $b"

if [[ ! $a =~ ^[0-9]+$ ]] || [[ ! $b =~ ^[0-9]+$ ]]; then
    echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') - Invalid input. Both arguments must be integers." >&2
    exit 1
fi

c=$((a + b))
echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') - Addition process complete successfully."
echo "Result: $c"

Scaling to Production

While debugging a 5-line script is trivial, modern enterprise applications contain tens of thousands of lines of code. If a production application crashes, logs are the only way to pinpoint the exact line of failure (e.g., an unhandled IndexOutOfBoundsException at line 5,042) without attaching an interactive debugger to a live production environment.

3. The Challenge of Distributed Systems & Centralized Logging

In a microservices architecture running on Kubernetes, an organization might deploy hundreds of containerized services across multiple namespaces.

The Problem: Log Fragmentation

Imagine a scenario where a database connection timeout occurs intermittently.

• The Naive Troubleshooting Method: A platform engineer must manually run kubectl logs <pod-name> -n <namespace> across dozens of pods to locate the error. This is highly inefficient, time-consuming, and fails if pods have restarted (as local container logs are ephemeral and destroyed along with the container).
• The Centralized Logging Solution: All container logs are continuously collected, shipped, and aggregated into a single, highly searchable database.

Business and Security Impact

During critical security incidents (such as the Log4j vulnerability), engineers have very little time to react. A centralized logging system allows security teams to query the entire infrastructure instantly to identify vulnerable services and active exploitation attempts.

4. The EFK Stack Architecture & Mental Model

The EFK stack is a popular open-source centralized logging solution. Its architecture closely mirrors the metrics monitoring architecture used by Prometheus and Grafana.

Architectural Mapping: Metrics vs. Logs

The EFK Log Pipeline Flow

5. Architectural Comparison: EFK vs. ELK

A common architectural decision is choosing between the EFK stack and the ELK stack. The only difference lies in the log collection agent: FluentBit versus Logstash.

Note: For most Kubernetes environments, FluentBit is highly preferred over Logstash because running a heavy JVM-based agent on every single node consumes valuable compute resources that should otherwise be allocated to application workloads.

6. Kubernetes Storage & AWS Integration (EKS, IRSA, and EBS CSI Driver)

Because Elasticsearch is a database, it must store logs persistently. In a production environment, log volume scales linearly with transaction volume.

Where $M$ is the number of active microservices. This volume can easily reach tens or hundreds of gigabytes per day.

To prevent data loss when Elasticsearch pods reschedule, we deploy Elasticsearch as a StatefulSet backed by AWS Elastic Block Store (EBS) volumes.

The Storage Communication Challenge

Elasticsearch runs inside the Amazon EKS cluster, while the EBS volumes exist outside the EKS cluster in the AWS cloud infrastructure. To allow EKS to dynamically provision and attach EBS volumes, we must configure:

1. IAM Roles for Service Accounts (IRSA): Maps a Kubernetes Service Account to an AWS IAM Role with permissions to manage EBS volumes.
2. EBS CSI (Container Storage Interface) Driver: The controller that listens to Kubernetes PersistentVolumeClaims (PVCs) and calls AWS APIs to provision physical EBS volumes.

7. Step-by-Step EFK Stack Deployment Guide

This section details the commands and configurations required to deploy the EFK stack on an Amazon EKS cluster.

Step 1: Create the IAM Role for the EBS CSI Driver Service Account

Run the following command to associate an IAM policy with a Kubernetes service account in the kube-system namespace. Replace observability-cluster with your actual cluster name.

eksctl create iamserviceaccount \
  --name ebs-csi-controller-sa \
  --namespace kube-system \
  --cluster observability-cluster \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --approve \
  --role-only \
  --role-name AmazonEKS_EBS_CSI_DriverRole

Step 2: Retrieve the IAM Role ARN

Retrieve the ARN of the newly created role and store it in an environment variable:

export EBS_CSI_ROLE_ARN=$(aws iam get-role --role-name AmazonEKS_EBS_CSI_DriverRole --query "Role.Arn" --output text)
echo $EBS_CSI_ROLE_ARN

Step 3: Enable the EBS CSI Driver Add-on

Deploy the EBS CSI driver to your EKS cluster using the service account role:

eksctl create addon \
  --name aws-ebs-csi-driver \
  --cluster observability-cluster \
  --service-account-role-arn $EBS_CSI_ROLE_ARN \
  --force

Step 4: Create the Logging Namespace

Isolate all logging infrastructure components inside a dedicated namespace:

kubectl create namespace logging

Step 5: Deploy Elasticsearch via Helm

Add the Elastic Helm repository and install Elasticsearch. We configure the storage class to use gp2 (EBS).

helm repo add elastic https://helm.elastic.co
helm repo update

helm install elasticsearch elastic/elasticsearch \
  --namespace logging \
  --set volumeClaimTemplate.storageClassName=gp2 \
  --set volumeClaimTemplate.resources.requests.storage=10Gi

Step 6: Retrieve Elasticsearch Credentials

Elasticsearch generates a default administrative password during installation. Retrieve and decode it:

# Get Username (Default: elastic)
echo "Username: elastic"

# Get Password
kubectl get secret --namespace logging elasticsearch-master-credentials -o jsonpath="{.data.password}" | base64 --decode; echo

Keep this password secure; it is required to configure FluentBit and log into Kibana.

Step 7: Deploy Kibana via Helm

Install Kibana and expose it using an AWS LoadBalancer to allow external access to the web interface:

helm install kibana elastic/kibana \
  --namespace logging \
  --set service.type=LoadBalancer

8. Configuring FluentBit (values.yaml Deep Dive)

FluentBit is configured using four primary blocks: Service, Input, Filter, and Output.

FluentBit Configuration Architecture

Customizing values.yaml for FluentBit

To allow FluentBit to securely authenticate and write logs to Elasticsearch, you must update the output configuration in your custom values.yaml file.

Below is a production-grade snippet of the FluentBit configuration:

config:
  inputs: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

  filters: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Keep_Log            Off
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On

  outputs: |
    [OUTPUT]
        Name            es
        Match           *
        Host            elasticsearch-master.logging.svc.cluster.local
        Port            9200
        HTTP_User       elastic
        HTTP_Passwd     <YOUR_DECODED_ELASTICSEARCH_PASSWORD>
        TLS             On
        TLS.verify      Off
        Logstash_Format On
        Logstash_Prefix k8s-logs
        Retry_Limit     False

Warning: Ensure that the HTTP_Passwd field is updated with the plaintext password retrieved in Step 6. If your cluster uses internal TLS, keep TLS On and adjust verification settings based on your certificate authority setup.

Deploying FluentBit

Add the FluentBit Helm repository and install the chart using your customized values.yaml:

helm repo add fluent https://fluent.github.io/helm-charts
helm repo update

helm install fluent-bit fluent/fluent-bit \
  --namespace logging \
  -f values.yaml

9. Deploying a Test Application & Verifying Log Flow

To verify that the logging pipeline is functioning end-to-end, we deploy a sample application in a separate namespace and observe the log flow.

Step 1: Deploy Sample Applications

Create a dev namespace and deploy two microservices (service-a and service-b):

kubectl create namespace dev

# Deploy test services
kubectl apply -f https://raw.githubusercontent.com/iam-veeramalla/observability-zero-to-hero/main/day-4/service-a.yaml -n dev
kubectl apply -f https://raw.githubusercontent.com/iam-veeramalla/observability-zero-to-hero/main/day-4/service-b.yaml -n dev

Step 2: Verify Pod Status

Ensure all logging and application pods are running successfully:

kubectl get pods -n logging
kubectl get pods -n dev

Since FluentBit is a DaemonSet, you will see exactly one FluentBit pod running for every node in your cluster:

NAME                             READY   STATUS    RESTARTS   AGE
elasticsearch-master-0           1/1     Running   0          15m
fluent-bit-2g5hx                 1/1     Running   0          10m
fluent-bit-z8r9w                 1/1     Running   0          10m
kibana-7c8bf9699f-gq2xw          1/1     Running   0          12m

Step 3: Access Kibana and Configure Data Views

1. Retrieve the external URL of the Kibana LoadBalancer:
   BASH

   Copy

   kubectl get svc kibana -n logging
   

2. Open the external IP in your web browser on port 5601 (e.g., http://<External-IP>:5601).
3. Log in using the username elastic and your retrieved password.
4. Navigate to Management -> Stack Management -> Data Views (under Kibana).
5. Click Create data view.
6. Set the Index pattern to k8s-logs-* (matching the Logstash_Prefix defined in the FluentBit output configuration).
7. Save the Data View.

Step 4: Querying Logs in Kibana Discover

Navigate to the Discover tab in Kibana. You can now view real-time logs from your Kubernetes cluster.

To isolate logs from your newly deployed test application, use the Kibana Query Language (KQL) in the search bar:

kubernetes.namespace_name : "dev" and kubernetes.pod_name : "service-a"

This query filters out system noise (such as kube-proxy or aws-node logs) and displays only the logs generated by service-a in the dev namespace.

10. Summary and Key Takeaways

• The Purpose of Logging: While metrics indicate system health and state changes, logs provide the detailed context necessary to understand why an application failed.
• Centralized Logging Necessity: In distributed Kubernetes environments, manual log inspection via kubectl is highly inefficient and vulnerable to data loss due to ephemeral pod lifecycles. Centralized logging aggregates all logs into a single searchable store.
• The EFK Stack Components:
  • Elasticsearch acts as the highly scalable document database.
  • FluentBit runs as a lightweight DaemonSet on every node to collect and forward logs.
  • Kibana provides the visualization and query interface for end-users.
• FluentBit vs. Logstash: FluentBit is written in C and has a minimal resource footprint, making it ideal for containerized edge environments compared to the resource-heavy, JVM-based Logstash.
• EKS Storage Integration: Stateful workloads like Elasticsearch require persistent storage. This is achieved in AWS EKS by configuring the EBS CSI Driver alongside IAM Roles for Service Accounts (IRSA).
• FluentBit Pipeline Architecture: Configuration is modularly split into Service (global settings), Input (log sources), Filter (metadata enrichment and namespace exclusion), and Output (destination endpoints like Elasticsearch).
• Vendor Neutrality: FluentBit is vendor-neutral. If an organization decides to migrate from Elasticsearch to another platform (e.g., Splunk or Datadog), the collection mechanism remains identical; only the Output block of the configuration needs to be modified.