Observability Zero to Hero: Distributed Tracing with Jaeger (Day 6)

Distributed tracing is the third pillar of observability, alongside metrics and logs. This guide covers the core concepts of distributed tracing, the architecture of Jaeger, how developers instrument applications using OpenTelemetry, and how DevOps/SRE teams deploy and configure Jaeger on a production-grade Kubernetes cluster (EKS) using Helm and Elasticsearch.

1. Why Do We Need Distributed Tracing?

In a monolithic application, tracking a request is relatively straightforward because the execution flow occurs within a single process. However, in a modern microservices architecture, a single user request can trigger a cascade of downstream API calls across dozens of independent services, network proxies, load balancers, and databases.

When a user experiences latency (e.g., a checkout page taking 4 seconds instead of the expected 1 second), metrics and logs alone are often insufficient:

• Metrics can tell you that a service is slow (e.g., high P99 latency on the payment service), but they cannot show the exact path of the slow request.
• Logs can show errors or timestamps within individual services, but correlating logs across multiple asynchronous services requires complex, manual trace-ID parsing.

Distributed tracing solves this by tracking the lifecycle of a request as it flows through the entire distributed system, pinpointing exactly where delays or errors occur.

2. The Core Mental Model: The Travel Itinerary Analogy

To understand distributed tracing, consider a real-world travel analogy.

Real-Life Analogy: Traveling from Hyderabad to Boston

Imagine you are traveling from Hyderabad, India, to visit a friend in Boston, USA, for the first time. You map out a detailed travel itinerary:

Home (Hyderabad)
   |
   | (Cab - 45 mins)
   v
Hyderabad International Airport
   |
   | (Flight 1 - 3.5 hours)
   v
Dubai / Abu Dhabi Airport (Layover)
   |
   | (Flight 2 - 12 hours)
   v
Boston Logan International Airport
   |
   | (Cab - 2.5 hours)
   v
Friend's House (Boston)

The entire journey takes 25 hours. When you arrive, your friend notes that the journey typically only takes 23 hours. To find the source of the 2-hour delay, you analyze the duration of each leg (or "span") of your itinerary:

• Expected Cab Time (Boston Airport to House): 30 minutes.
• Actual Cab Time: 2.5 hours.

By tracing the journey step-by-step, you isolate the bottleneck: you boarded the wrong cab at the Boston airport, causing a 2-hour delay.

In software engineering:

• The Travel Itinerary is the Trace.
• Each Leg of the Journey is a Span.
• The Delay Analysis is Latency Profiling.

3. Microservices Latency: Naive vs. Traced Approach

Consider an e-commerce application where a user initiates a payment.

The Naive Approach (Without Tracing)

The user experiences a 4-second delay.

Without tracing, the SRE or developer must:

1. Check the API Gateway logs to see when the request entered.
2. Log into the database or search Kibana for logs across four different services.
3. Attempt to manually correlate timestamps, which may be out of sync across servers.

The Traced Approach (With Distributed Tracing)

With distributed tracing implemented, the request is assigned a unique Trace ID at the entry point. As the request hops from service to service, metadata (headers) containing this Trace ID is propagated. Each service records its start and end times for its portion of the work (called a Span).

The visual trace immediately reveals that Service B (the Discount Engine) consumed 3 seconds of the total 4-second response time, allowing developers to focus their optimization efforts precisely where they are needed.

4. The Two Pillars of Tracing Implementation

Implementing distributed tracing requires a coordinated effort between two distinct engineering roles:

A. Developer Responsibility: Application Instrumentation

Developers must write code within the application to generate and propagate trace data.

• Manual/SDK Instrumentation: Developers import tracing libraries (such as OpenTelemetry) to initialize a tracer, create spans, and inject/extract trace contexts across HTTP/gRPC boundaries.
• OpenTelemetry (OTel): The modern industry standard. It is vendor-neutral, meaning you can instrument your application once and send the trace data to Jaeger, Zipkin, Datadog, or Dynatrace without changing your application code.

B. DevOps & SRE Responsibility: Infrastructure Deployment

DevOps and Site Reliability Engineers are responsible for setting up the backend infrastructure to collect, store, and visualize these traces.

• Deploying tracing collectors and agents (e.g., Jaeger Agent and Collector).
• Provisioning high-performance, persistent storage backends (e.g., Elasticsearch or Cassandra) to index trace data.
• Exposing the query user interface (Jaeger UI) to developers for debugging.

5. Jaeger Architecture & Comparison with Prometheus/EFK

Jaeger is an open-source, end-to-end distributed tracing system originally built by Uber and hosted by the Cloud Native Computing Foundation (CNCF).

Jaeger Core Components

1. Jaeger Agent: A network daemon that listens for spans sent over UDP. It is typically deployed as a Kubernetes DaemonSet or a sidecar container. It batches the trace data and routes it to the collector.
2. Jaeger Collector: Receives traces from the agents, runs them through a processing pipeline, validates them, indexes them, and writes them to the storage backend.
3. Storage Backend: Jaeger is non-opinionated about storage. While it supports in-memory storage for development, production environments require persistent databases like Elasticsearch (highly recommended for search speed) or Cassandra.
4. Jaeger Query & UI: A query service that retrieves traces from storage and hosts a clean, minimalistic user interface to visualize trace timelines and system dependency graphs.

Architectural Comparison: Prometheus vs. EFK vs. Jaeger

6. Complete Observability Architecture Integration

In a production-grade Kubernetes cluster, all three pillars of observability run side-by-side, sharing resources and storage backends where appropriate.

Note on Storage Persistence: To ensure that logs and traces survive pod restarts, Elasticsearch must be backed by persistent storage. In AWS EKS, this is achieved by deploying the AWS EBS CSI (Container Storage Interface) Driver. The driver uses an IAM Role mapped to a Kubernetes Service Account via OIDC (OpenID Connect) to dynamically provision Amazon Elastic Block Store (EBS) volumes.

7. Step-by-Step Kubernetes & Jaeger Deployment Guide

This section outlines how to provision the infrastructure required for distributed tracing on an AWS EKS cluster.

Step 1: Configure AWS EBS CSI Driver (IAM & Service Account)

To allow Elasticsearch to provision EBS volumes, create an IAM-associated service account:

# Create the IAM Service Account with the AmazonEBSCSIDriverPolicy
eksctl create iamserviceaccount \
  --name ebs-csi-controller-sa \
  --namespace kube-system \
  --cluster your-eks-cluster-name \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --approve \
  --role-only \
  --role-name AmazonEKS_EBS_CSI_DriverRole

Retrieve the ARN of the newly created IAM role and deploy the EBS CSI driver:

# Deploy the EBS CSI Driver Addon
eksctl create addon \
  --name aws-ebs-csi-driver \
  --cluster your-eks-cluster-name \
  --service-account-role-arn arn:aws:iam::<YOUR_AWS_ACCOUNT_ID>:role/AmazonEKS_EBS_CSI_DriverRole \
  --force

Step 2: Deploy Elasticsearch (Storage Backend)

Create the logging namespace and deploy Elasticsearch using the official Elastic Helm chart:

# Create namespace
kubectl create namespace logging

# Add Elastic Helm repository
helm repo add elastic https://helm.elastic.co
helm repo update

# Install Elasticsearch
helm install elasticsearch elastic/elasticsearch \
  --namespace logging \
  --set antiAffinity=soft \
  --set replicas=1

Extract and save the auto-generated elastic user password:

# Extract password from Kubernetes secret
kubectl get secrets --namespace logging elasticsearch-master-credentials -o jsonpath="{.data.password}" | base64 --decode; echo

Extract the CA Certificate for secure TLS communication:

# Extract CA Certificate
kubectl get secrets --namespace logging elasticsearch-master-certs -o jsonpath="{.data.ca\.crt}" | base64 --decode > es-ca.crt

Step 3: Configure and Deploy Jaeger

Create the tracing namespace and store the Elasticsearch credentials and CA certificate in Kubernetes ConfigMaps and Secrets:

# Create namespace
kubectl create namespace tracing

# Create ConfigMap for the ES CA Certificate
kubectl create configmap es-ca-cert --from-file=ca.crt=es-ca.crt -n tracing

# Create Secret for Elasticsearch credentials
kubectl create secret generic es-credentials \
  --from-literal=username=elastic \
  --from-literal=password=<YOUR_ELASTICSEARCH_PASSWORD> \
  -n tracing

Create a custom Helm values file named jaeger-values.yaml to configure Jaeger to use Elasticsearch as its storage engine:

# jaeger-values.yaml
storage:
  type: elasticsearch
  elasticsearch:
    hosts: ["https://elasticsearch-master.logging.svc.cluster.local:9200"]
    secretName: es-credentials
    caConfigMap: es-ca-cert
    caConfigMapKey: ca.crt

Deploy Jaeger using the Jaeger Helm chart:

# Add Jaeger Helm repository
helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm repo update

# Install Jaeger with custom values
helm install jaeger jaegertracing/jaeger \
  --namespace tracing \
  -f jaeger-values.yaml

Verify that all Jaeger components are running:

kubectl get pods -n tracing

8. Application Instrumentation & Trace Analysis Walkthrough

A. Code Instrumentation (Node.js & OpenTelemetry)

Below is a conceptual example of how developers instrument an application (tracing.js) using OpenTelemetry to export spans to the Jaeger Agent:

// tracing.js
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');

// Configure the exporter to send spans to the Jaeger Agent/Collector
const traceExporter = new OTLPTraceExporter({
  url: 'http://jaeger-collector.tracing.svc.cluster.local:4317', // gRPC endpoint
});

const sdk = new NodeSDK({
  traceExporter,
  instrumentations: [getNodeAutoInstrumentations()],
  serviceName: 'service-a',
});

sdk.start();
console.log('Tracing initialized successfully.');

B. Deploying the Demo Application

Deploy the pre-instrumented demo applications (service-a and service-b) into the dev namespace:

# Create dev namespace
kubectl create namespace dev

# Deploy applications using Kustomize or standard manifests
kubectl apply -k github.com/iam-veeramalla/observability-zero-to-hero/day4/manifests/

Get the external URL of service-a (LoadBalancer):

kubectl get svc -n dev

C. Generating and Analyzing Traces

1. Generate Single-Service Trace: Access the /healthy endpoint of service-a:

   BASH

   Copy

   curl http://<LOAD_BALANCER_IP>/healthy
   

2. Generate Multi-Service Trace: Access the /call-service-b endpoint, which forces service-a to make an HTTP call to service-b:

   BASH

   Copy

   curl http://<LOAD_BALANCER_IP>/call-service-b
   

3. Access Jaeger UI: Port-forward the Jaeger Query service to your local machine:

   BASH

   Copy

   kubectl port-forward svc/jaeger-query 16686:16686 -n tracing
   

   (Note: If running on a remote EC2 instance, append --address 0.0.0.0 to the port-forward command and ensure port 16686 is open in your AWS Security Group).

4. Analyze Spans on Jaeger UI:

   • Open http://localhost:16686 in your browser.
   • Select service-a from the Service dropdown and click Find Traces.
   • Click on the trace for /call-service-b. You will see a nested timeline showing 12 spans spanning across both service-a and service-b, detailing the exact execution time of express middlewares, logging functions, and network requests.

9. Real-World Troubleshooting & Interview Scenarios

Scenario A: Jaeger Query Pod in CrashLoopBackOff

• The Problem: During deployment, the jaeger-query pod fails its liveness and readiness probes and enters a crash loop.
• The Root Cause: This is frequently caused by stale configuration maps or secrets from a previous Elasticsearch installation. Jaeger attempts to connect to Elasticsearch using outdated TLS certificates or incorrect credentials, causing the query service to crash on startup.
• The Fix:
  1. Inspect the pod logs: kubectl logs -n tracing <jaeger-query-pod-name>.
  2. If certificate errors are present, delete the existing configmap and secret:
     BASH

     Copy

     kubectl delete configmap es-ca-cert -n tracing
     kubectl delete secret es-credentials -n tracing
     

  3. Re-extract the active CA certificate from the running Elasticsearch instance and recreate the Kubernetes resources.

Scenario B: "How have you used distributed tracing to solve a production issue?" (Interview Question)

• The Answer:

  "In our microservices environment, users were reporting intermittent 3-to-4 second delays during checkout. We had metrics showing high latency on the gateway, but logs were clean.

  After instrumenting our Node.js services with OpenTelemetry and routing traces to Jaeger, we analyzed the slow traces. We discovered that a downstream discount-validation service was executing sequential database queries inside a loop instead of a single batched query.

  The Jaeger UI clearly showed a single span (validateDiscounts) containing over 50 nested database spans, each taking 50–80 milliseconds. We shared this trace visualization with the development team, who refactored the code to batch the queries. This reduced the overall request latency from 4 seconds to under 200 milliseconds."

10. Summary and Key Takeaways

• The Third Pillar: Distributed tracing provides end-to-end visibility into the lifecycle of a request across distributed boundaries, complementing metrics (what is broken) and logs (why it broke).
• Spans vs. Traces: A Trace represents the entire journey of a request, while a Span represents a single, bounded unit of work (e.g., an HTTP call, database query, or function execution) within that journey.
• OpenTelemetry Standard: Industry best practice dictates using OpenTelemetry for application instrumentation to avoid vendor lock-in, allowing seamless migration between backends like Jaeger, Datadog, or Zipkin.
• Jaeger Architecture: Jaeger relies on lightweight Agents (collecting spans), Collectors (processing and indexing), a Storage Backend (Elasticsearch/Cassandra for persistence), and a Query UI (visualization).
• EBS CSI Driver Integration: For production deployments on AWS EKS, Elasticsearch requires persistent volumes managed via the EBS CSI driver, configured using IAM Roles for Service Accounts (IRSA).
• Actionable Debugging: Jaeger's minimalistic UI allows SREs and developers to quickly identify latency bottlenecks, network delays, and failing downstream dependencies by visually inspecting nested span timelines.