End-to-End Observability Project: Beginner-Level Guide for Complex Microservices

Deploying, instrumenting, and observing a multi-language microservices architecture using OpenTelemetry, Jaeger, Prometheus, and Grafana.

1. Why Do We Need an End-to-End Observability Demo?

Modern cloud-native systems consist of dozens—sometimes hundreds—of distributed microservices written in different programming languages. When an error occurs or latency spikes in production, identifying the root cause across a distributed network of services is exceptionally difficult.

To learn or evaluate observability tooling without setting up a synthetic business application from scratch, engineers rely on open-source distributed demo applications.

• Industry Collaboration: Maintained by top observability companies like DataDog, Dynatrace, Microsoft, Alibaba, and Grafana Labs.
• Polyglot Architecture: Written across multiple languages (Python, Go, C#, Java, Node.js), demonstrating how different stacks handle telemetry instrumentation.
• Production-Grade Simulation: Includes built-in fake load generators to simulate real-world e-commerce traffic patterns, generating live traces, metrics, and logs.

2. Core Mental Model: Observability vs. Instrumentation

To establish a robust observability pipeline, we must cleanly separate Instrumentation from Collection & Exporting.

• Instrumentation (The Developer's Responsibility): Modifying application source code using OpenTelemetry APIs and SDKs to emit telemetry data (metrics, logs, traces) directly from critical application functions.
• Implementation & Operations (The DevOps/SRE Responsibility): Provisioning infrastructure (Kubernetes clusters via Helm/Terraform), deploying collectors, and managing backend storage (Elasticsearch, Prometheus TSDB, Grafana dashboards).

3. The Vendor Lock-In Problem: Why OpenTelemetry?

The Naive Approach: Hardcoded Proprietary SDKs

In older architectures, development teams imported vendor-specific monitoring clients directly into their application source code (e.g., Prometheus client libraries, Datadog dogstatsd clients, or Nagios agents).

# Naive Approach: Hardcoded Prometheus Client in Application Code
from prometheus_client import Counter

REQUEST_COUNTER = Counter('http_requests_total', 'Total HTTP Requests')

def handle_request():
    REQUEST_COUNTER.inc()
    # Business logic here...

The Production Pain Point: If an organization decides to switch observability vendors (e.g., migrating from Prometheus to a proprietary SaaS tool, or from Datadog to Dynatrace), developers must rewrite instrumentation logic across hundreds of microservices.

The Modern Standard: OpenTelemetry (OTel)

OpenTelemetry provides a single, vendor-agnostic standard governed by the CNCF. Developers instrument their code once using OTel APIs and SDKs. Routing telemetry data to different backends is handled entirely through external configuration files (exporters).

# Better Approach: OpenTelemetry API (Vendor Agnostic)
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def handle_request():
    with tracer.start_as_current_span("handle_request"):
        # Business logic here...

4. OpenTelemetry Architecture & Data Flow

The OpenTelemetry Collector processes telemetry data through three distinct stages configured in its pipeline:

1. Receivers: Accept telemetry data from applications (supports OTLP, Jaeger, Prometheus, Zipkin formats).
2. Processors: Modify, filter, batch, or enrich telemetry data (e.g., adding resource attributes like Kubernetes pod names or cluster regions).
3. Exporters: Transmit processed telemetry data to designated backend storage systems or analysis platforms.

5. Polyglot Microservices Layout of the Demo Application

The reference e-commerce application is split into specialized microservices, each demonstrating instrumentation in a different programming language:

6. Step-by-Step Deployment on Kubernetes (EKS / Kind)

Step 1: Provision the Kubernetes Cluster

Ensure your target cluster (eks, kind, or minikube) is active and accessible via kubectl:

# Verify cluster connection
kubectl get nodes

(Note: Complete cluster creation scripts, including eksctl configurations and OIDC association steps, are available in the repository's day2 folder).

Step 2: Add the OpenTelemetry Helm Repository

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

Step 3: Deploy the Complete Demo Application via Helm

Deploy the entire multi-service mesh into your Kubernetes cluster with a single Helm command:

helm install my-otel-demo open-telemetry/opentelemetry-demo

Step 4: Verify Running Pods

Check that all microservice pods and observability components have successfully initialized:

kubectl get pods

(Expect upwards of 15 to 20 pods running concurrently, including the fake load generator).

7. Exposing and Accessing Observability UIs

Port Forwarding the Frontend Application

To access the e-commerce web shop on your local machine:

kubectl port-forward svc/frontend-proxy 8080:8080

• Local Access: Open http://localhost:8080 in your browser.
• Remote VM / Cloud Instance Access: Bind to all network interfaces by passing the address flag:
  BASH

  Copy

  kubectl port-forward svc/frontend-proxy 8080:8080 --address 0.0.0.0
  

  Access via your EC2/VM public IP: http://<PUBLIC_IP>:8080.

Accessing Jaeger UI for Distributed Traces

Jaeger is deployed as part of the Helm chart for trace collection. Port-forward or access the Jaeger query service UI to examine request lifecycles:

• Navigate to the Jaeger UI endpoint (or port-forward the Jaeger query service port 16686):
  BASH

  Copy

  kubectl port-forward svc/my-otel-demo-jaeger-query 16686:16686
  

• Open http://localhost:16686 to inspect service spans, latency breakdowns, and context propagation.

8. Analyzing Distributed Traces in Jaeger

When you execute actions in the e-commerce application (adding items to a cart, changing currencies, checking out), requests propagate across multiple language boundaries via Context Propagation (passing W3C Trace Context HTTP headers).

Anatomy of a Distributed Trace Span

When inspecting a trace for the cart-service or checkout-service:

1. Trace ID & Span ID: Unique identifiers linking all related child operations back to the root HTTP request.
2. Span Duration: Microsecond-level execution times for specific functions (e.g., get_cart, process_payment).
3. Metadata (Attributes): Container IDs, cloud regions, gRPC status codes, and Kubernetes pod details attached directly to the span.
4. Context Propagation Chain:
   Structure & Concept Flow

   Copy DiagramCopy

   Browser -> Frontend Proxy (Node.js) -> Checkout Service -> gRPC Call -> Cart Service (Go)

    

9. Querying Metrics in Prometheus & Grafana

Accessing Grafana Dashboards

Grafana is pre-configured in the Helm deployment with anonymous authentication enabled (no username/password required by default).

kubectl port-forward svc/my-otel-demo-grafana 3000:80

Open http://localhost:3000 to view the pre-built OpenTelemetry Demo Dashboard, which visualizes frontend latency, error rates, and throughput.

Writing PromQL Queries in Prometheus

To query custom application metrics directly against the Prometheus data source:

• HTTP Server Request Duration Histogram:
  PROMQL

  Copy

  http_server_duration_seconds_bucket
  

• Application Counter Metrics (e.g., Recommendations Counter):
  PROMQL

  Copy

  recommendation_requests_total
  

10. Real-World Production Pitfalls & Missing Infrastructure

Why Node-Level Metrics Are Missing in Default Demos

If you attempt to query node CPU usage, memory consumption, or Kubernetes cluster-level metrics in Prometheus (node_cpu_seconds_total or kube_pod_status_phase), you will notice that Prometheus returns empty results.

Root Cause: The default application Helm chart instruments application-level telemetry (OTel spans and custom counters), but does not deploy infrastructure-level scraping agents.

Resolution: To capture infrastructure metrics, you must install complementary monitoring agents:

• Node Exporter: Collects underlying Linux OS and hardware metrics from every worker node.
• Kube-State-Metrics: Listens to the Kubernetes API server and generates metrics about object states (deployments, pods, persistent volumes). (Detailed installation steps for the complete Prometheus operator stack are located in the repository's day3 and day4 folders).

Summary and Key Takeaways

• Observability is a Collaborative Effort: Requires tight alignment between developers (who instrument application code with OTel SDKs) and SREs/DevOps engineers (who provision collectors and backends).
• OpenTelemetry Standardizes Telemetry: Eliminates vendor lock-in by decoupling application instrumentation from backend exporters (Jaeger, Prometheus, Datadog).
• Polyglot Microservices Architecture: Real-world systems use multiple languages; OTel provides uniform libraries across Go, Python, Java, and C#.
• Context Propagation Links Distributed Calls: W3C Trace Context headers allow distributed systems to stitch together end-to-end request flows across gRPC and HTTP boundaries.
• Exporters Route Data Dynamically: A single OTel collector configuration file determines whether traces go to Jaeger, Prometheus, or commercial APM platforms without modifying code.
• Application vs. Infrastructure Metrics: OTel demo applications capture application telemetry, but complete cluster observability requires additional infrastructure agents like Node Exporter and Kube-State-Metrics.
• Practical Learning via Helm: Complex multi-service environments can be deployed instantly to Kubernetes clusters (EKS/Kind) using Helm charts for rapid hands-on experimentation.