Observability Zero to Hero: Metrics, Monitoring, and Prometheus Basics

This chapter explores the first core pillar of observability: Metrics. By establishing a foundational understanding of metrics and monitoring through real-world analogies, we bridge the gap between raw telemetry data and actionable system insights, culminating in the architectural design, installation, and configuration of Prometheus and Grafana on a production-grade Kubernetes cluster.

1. The Core Mental Model: The Patient and the Nurse Analogy

To understand why we need metrics and monitoring, consider a real-world medical scenario.

Real-Life Analogy: Hospital Patient Care

Imagine a patient admitted to an Intensive Care Unit (ICU). To ensure the patient's safety and recovery, a nurse is assigned to monitor their health.

• The Manual Process (Raw Metrics): Every 15 minutes, the nurse visits the patient to measure and record vital signs: heart rate, blood pressure, and body temperature. The nurse writes these numbers down in a paper logbook:

  • 10:00 AM — Heart Rate: $76\text{ bpm}$
  • 10:15 AM — Heart Rate: $81\text{ bpm}$
  • 10:30 AM — Heart Rate: $62\text{ bpm}$

• The Analytical Need: At the end of the day, the attending physician reviews this logbook. By analyzing these historical, periodic data points, the doctor determines whether the patient's condition is stable, improving, or deteriorating. If a sudden drop occurred after administering a specific medication, the doctor can correlate the event and adjust the treatment plan.

• The Automation Upgrade (Monitoring & Alerting): If the patient is critical, a nurse cannot manually measure heart rate every 30 seconds. Instead, the hospital connects the patient to an electronic bedside monitor. This machine continuously reads the patient's vitals (scraping metrics), displays them as real-time waveforms on a screen (visualization/dashboards), and sounds a loud alarm if the heart rate exceeds $110\text{ bpm}$ or drops below $50\text{ bpm}$ (alerting).

In this analogy:

• The Vital Signs are the Metrics (raw, historical, periodic data points).
• The Bedside Monitor is the Monitoring System (ingesting, visualizing, and alerting on those metrics).

2. Defining Metrics vs. Monitoring

Many engineers conflate metrics with monitoring. However, they represent distinct stages in the observability lifecycle.

Metrics: The Raw Telemetry

Metrics are numeric, historical data points recorded over time to measure the state of a system. Mathematically, a single metric data point can be represented as a tuple containing a timestamp and a value:

Where:

• $t$ is the epoch timestamp when the measurement occurred.
• $v$ is the numeric value of the measurement (e.g., CPU percentage, request count).

A time series is a sequence of these data points ordered chronologically:

Metrics are inherently raw, unstructured, and difficult to interpret in bulk when stored in flat files or spreadsheets.

Monitoring: The Actionable Superstructure

Monitoring is the active process of collecting, aggregating, visualizing, and alerting on metrics. It is the system that turns raw numbers into operational intelligence.

3. Translating the Analogy to IT Infrastructure

In modern software engineering, your application is the patient, and your infrastructure is the hospital environment. To understand the health of a distributed system, you must collect metrics across multiple layers.

By collecting these metrics periodically, you can diagnose complex production issues. For example, if your application experiences a sudden spike in latency, you can correlate it with a corresponding spike in CPU utilization or a database connection pool exhaustion event.

4. Architectural Deep Dive: How Prometheus Works

Prometheus is the industry-standard, open-source monitoring and alerting toolkit graduated by the Cloud Native Computing Foundation (CNCF). It is designed specifically for dynamic, containerized environments like Kubernetes.

The Prometheus Architecture

Core Components of Prometheus

1. Retrieval Engine (Scraper): Prometheus is fundamentally a pull-based monitoring system. Instead of applications constantly pushing metrics to a central database (which can overwhelm the database during traffic spikes), the Prometheus Retrieval Engine periodically queries configured targets via HTTP GET requests to fetch their current metric states.

2. Time Series Database (TSDB): Once retrieved, metrics are stored in a custom, highly optimized Time Series Database. The TSDB structures data on disk as sequential blocks over time, allowing for extremely fast writes and efficient queries across specific time ranges.

3. HTTP Server & PromQL: Prometheus exposes an HTTP API that allows external tools to query its internal TSDB. It uses PromQL (Prometheus Query Language), a powerful functional query language designed specifically for selecting and aggregating real-time time-series data.

4. Service Discovery: In dynamic environments like Kubernetes, Pods are ephemeral—they are created and destroyed constantly, changing their IP addresses. Prometheus uses Service Discovery to automatically detect new scrape targets by communicating directly with the Kubernetes API server.

5. Alertmanager: Prometheus evaluates alerting rules defined in its configuration. If a metric violates a rule (e.g., CPU usage $> 80\%$ for more than 5 minutes), Prometheus forwards the alert to the Alertmanager. The Alertmanager deduplicates, groups, and routes the alerts to notification channels such as Slack, PagerDuty, Email, or custom webhooks.

5. The Role of Grafana in the Monitoring Stack

While Prometheus includes a basic web interface for executing PromQL queries and viewing simple graphs, it is not designed for rich, production-grade visualization.

• Prometheus acts as the Data Source. It excels at scraping, storing, and querying time-series data.
• Grafana acts as the Presentation Layer. It connects to Prometheus, executes PromQL queries behind the scenes, and renders beautiful, highly customizable, real-time dashboards.

Together, Prometheus and Grafana form the standard monitoring stack for modern cloud-native engineering teams.

6. Step-by-Step Installation of Prometheus and Grafana on AWS EKS

To demonstrate Prometheus in a production-like environment, we will deploy it to an Amazon Elastic Kubernetes Service (EKS) cluster using Helm, the package manager for Kubernetes.

Prerequisites

• A running Kubernetes cluster (EKS, Minikube, or Kind).
• kubectl configured to communicate with your cluster.
• helm installed on your local machine.

Step 1: Add the Prometheus Community Helm Repository

First, add the official Prometheus community Helm chart repository to your local Helm client and update the charts:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Step 2: Create a Dedicated Namespace

Isolate all monitoring infrastructure inside a dedicated namespace named monitoring:

kubectl create namespace monitoring

Step 3: Clone the Repository and Inspect Configurations

Clone the course repository to access custom configuration files:

git clone https://github.com/iam-veeramalla/observability-zero-to-hero.git
cd observability-zero-to-hero/day-2

In this directory, you will find a custom configuration file (e.g., values.yaml) that enables specific sub-components, such as the Alertmanager:

# Example snippet of custom values enabling Alertmanager
alertmanager:
  enabled: true

Step 4: Install the Prometheus Stack

Deploy the kube-prometheus-stack Helm chart. This comprehensive chart installs the Prometheus Server, Alertmanager, Grafana, Node Exporter, and Kube-State-Metrics:

helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  -f values.yaml

Step 5: Verify the Deployment

Check the status of the deployed pods in the monitoring namespace:

kubectl get pods -n monitoring

You should see output similar to this:

NAME                                                     READY   STATUS    RESTARTS   AGE
prometheus-grafana-76bf4c76b7-abcde                      3/3     Running   0          2m
prometheus-kube-prometheus-alertmanager-0                2/2     Running   0          2m
prometheus-kube-prometheus-operator-84b5876c-fghij       1/1     Running   0          2m
prometheus-kube-state-metrics-58497c6c-klmno             1/1     Running   0          2m
prometheus-prometheus-node-exporter-pqrst                1/1     Running   0          2m
prometheus-prometheus-kube-prometheus-prometheus-0       2/2     Running   0          2m

7. Accessing the Monitoring Stack via Port Forwarding

In a production environment, you would expose these services securely using an Ingress Controller and SSL certificates. For initial setup and local validation, we will use Kubernetes port-forwarding to access the user interfaces.

Port-Forwarding Commands

Open three separate terminal windows or run these commands in the background:

1. Access Prometheus UI (Port 9090):

   BASH

   Copy

   kubectl port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090 -n monitoring
   

   Open your browser and navigate to: http://localhost:9090

2. Access Grafana UI (Port 3000):

   BASH

   Copy

   kubectl port-forward svc/prometheus-grafana 3000:80 -n monitoring
   

   Open your browser and navigate to: http://localhost:3000

3. Access Alertmanager UI (Port 9093):

   BASH

   Copy

   kubectl port-forward svc/prometheus-kube-prometheus-alertmanager 9093:9093 -n monitoring
   

   Open your browser and navigate to: http://localhost:9093

Logging into Grafana

• Default Username: admin
• Default Password: prom-operator

Once logged in, you can configure Prometheus as a data source by navigating to Connections > Data Sources > Add Data Source, selecting Prometheus, and entering the internal Kubernetes service URL: http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090.

8. How Prometheus Gathers Metrics: The Three Pillars of Scraping

Prometheus cannot magically read metrics from your infrastructure or application code without helper agents. It relies on three primary mechanisms to collect metrics across your entire stack.

Summary of Scraping Mechanisms

Deep Dive: Node Exporter

Operating systems do not expose metrics in a format Prometheus understands. The Node Exporter is a daemon running on every virtual machine (node) in your cluster. It reads raw system files from the Linux /proc and /sys directories, formats them into Prometheus-compatible plaintext metrics, and exposes them over an HTTP endpoint (typically port 9100).

Example raw metric exposed by Node Exporter:

node_cpu_seconds_total{cpu="0",mode="idle"} 284719.23

Deep Dive: Kube-State-Metrics (KSM)

While Node Exporter monitors the physical/virtual servers, it knows nothing about Kubernetes concepts like Pods, Services, or Deployments. Kube-State-Metrics is an agent that listens to the Kubernetes API server. It translates the state of Kubernetes objects into Prometheus metrics.

Example raw metric exposed by KSM:

kube_pod_status_phase{namespace="default",pod="payment-service-xyz",phase="Running"} 1

Deep Dive: Application-Level Metrics

To monitor application-specific logic (e.g., how many payments succeeded, how long a database query took), developers write code using Prometheus client libraries (available in Java, Go, Python, Node.js, etc.). These libraries expose a /metrics HTTP endpoint on the application itself.

// Go example exposing a custom HTTP request counter
import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "net/http"
)

var pingCounter = prometheus.NewCounter(
    prometheus.CounterOpts{
        Name: "ping_request_count",
        Help: "No of ping requests processed",
    },
)

func main() {
    prometheus.MustRegister(pingCounter)
    http.Handle("/metrics", promhttp.Handler())
    http.HandleFunc("/ping", func(w http.ResponseWriter, r *http.Request) {
        pingCounter.Inc()
        w.Write([]byte("pong"))
    })
    http.ListenAndServe(":8080", nil)
}

Prometheus uses its Service Discovery configuration to find these application pods and scrape their /metrics endpoints automatically.

9. Industry Alternatives and the CNCF Ecosystem

While Prometheus is the dominant player in the Kubernetes ecosystem, other tools exist, each with its own trade-offs.

• Nagios: An industry pioneer. Nagios is primarily host-based and relies on active/passive checks. It is difficult to scale in dynamic, ephemeral container environments where IPs change constantly.
• InfluxDB: A highly performant time-series database. Unlike Prometheus, which is a complete monitoring system (scraper + database + alert engine), InfluxDB is primarily a database. It is often used in push-based architectures.
• Graphite: A push-based metrics collection and graphing tool. It lacks the built-in service discovery and rich querying capabilities (PromQL) that make Prometheus ideal for Kubernetes.

Why Prometheus Dominates

Prometheus is built from the ground up for cloud-native environments. Its native integration with Kubernetes, powerful PromQL query engine, pull-based architecture (which protects applications from performance degradation), and massive CNCF community support make it the default choice for modern DevOps and Site Reliability Engineering (SRE) teams.

10. Summary and Key Takeaways

• Metrics vs. Monitoring: Metrics are raw, historical, numeric data points ($d = (t, v)$). Monitoring is the active system that scrapes, visualizes (dashboards), and alerts on those metrics.
• The Pull Model: Prometheus uses a pull-based architecture, periodically scraping metrics from targets via HTTP GET requests, which prevents applications from overloading the monitoring server.
• Time Series Database (TSDB): Prometheus stores metrics in a highly optimized TSDB, tracking values chronologically alongside unique key-value pairs (labels).
• Node Exporter: A daemon that translates host-level OS metrics (CPU, memory, disk) from the /proc and /sys directories into Prometheus format.
• Kube-State-Metrics: An agent that queries the Kubernetes API server to generate metrics representing the state of cluster objects (Pods, Deployments, ReplicaSets).
• Application Metrics: Custom metrics exposed directly by application code on a /metrics endpoint using Prometheus client libraries.
• Grafana Integration: Grafana acts as the visualization layer, executing PromQL queries against Prometheus to render rich, interactive dashboards.
• CNCF Standard: Prometheus is a graduated CNCF project, second only to Kubernetes in adoption and community strength, making it the industry standard for cloud-native observability.