Observability Zero to Hero: Deep-Dive Prometheus Architecture, Exporters, and PromQL

This chapter provides a practical, hands-on exploration of the Prometheus monitoring ecosystem. We will examine how Prometheus scrapes metrics from infrastructure and cluster-state exporters, write queries using Prometheus Query Language (PromQL), analyze the architectural differences between Node Exporter and Cube State Metrics, and build custom dashboards in Grafana.

1. Why Do We Need Exporters? (The Problem of Metric Collection)

Prometheus is fundamentally a pull-based monitoring system. It does not automatically know how to read the memory of an operating system, the CPU utilization of a virtual machine, or the status of a Kubernetes deployment.

For Prometheus to collect metrics, target systems must expose their internal state in a format that Prometheus understands. This format is a plain-text representation of metrics exposed over an HTTP endpoint, typically /metrics.

Because many systems (such as Linux kernels, databases, or third-party applications) cannot be modified to natively expose Prometheus metrics, we use exporters. Exporters act as translators: they query the target system's internal APIs or system files, translate that data into the Prometheus metric format, and serve it over HTTP.

2. Core Mental Model: The Pull-Based Scraping Architecture

To understand how metrics flow from a physical node or a Kubernetes resource into a dashboard, consider the following vertical workflow:

Kubernetes API Server / OS Kernel
   |
   | Exposes raw state data
   v
Exporters (Node Exporter / Cube State Metrics)
   |
   | Translates and hosts data on http://<IP>:<PORT>/metrics
   v
Prometheus Server (Scrape Engine)
   |
   | Periodically pulls metrics and writes to Time Series Database (TSDB)
   v
Prometheus HTTP Server
   |
   | Evaluates PromQL queries
   v
Grafana / Prometheus Web UI

Real-Life Analogy: The Restaurant Order Workflow

Imagine a busy restaurant kitchen (the Kubernetes Cluster). The head chef (Prometheus) needs to know the status of every table and ingredient.

• Instead of the chef running around asking every single line cook, there are dedicated station managers (Exporters) who keep a running tally of their station's status on a clipboard (the /metrics endpoint).
• At regular intervals (the scrape interval), the head chef walks around the kitchen, reads the clipboards (pulls the metrics), and writes the numbers down in a master ledger (the Time Series Database).
• If the chef wants to know how many salads were made over the last hour, they look at the ledger and calculate the rate (PromQL).

3. Deep Dive: Node Exporter vs. Cube State Metrics

In a Kubernetes environment, Prometheus relies on two primary exporters to gather system and cluster-level data. Understanding the structural differences between them is critical for cluster administration.

4. Step-by-Step Evolution: From Raw Endpoint to TSDB to PromQL

When Prometheus scrapes an exporter, it stores the collected data in a Time Series Database (TSDB).

The Structure of a Time Series

A traditional relational database stores the current state of an object. A TSDB, however, records how a value changes over time. Every entry in a TSDB consists of:

1. Metric Name: The identifier of the feature being measured (e.g., kube_pod_container_status_restarts_total).
2. Labels (Dimensions): Key-value pairs that categorize the metric (e.g., namespace="default", pod="busybox-crash").
3. Timestamp: The exact time the metric was scraped.
4. Value: A float64 numeric value recorded at that timestamp.

Mathematically, a time series can be represented as a sequence of tuples:

Where $t_i$ represents the timestamp and $v_i$ represents the metric value at that time.

5. Practical Hands-on: Accessing Raw Metrics via Curl

To demystify how exporters present data, we can log directly into a Kubernetes node and query the exporters using curl.

Step 1: Verify the Exporter Pods are Running

Run the following command to list the pods in your monitoring namespace:

kubectl get pods -n monitoring

You will see the Prometheus stack components, including the Node Exporter DaemonSet and the Cube State Metrics Deployment:

NAME                                                     READY   STATUS    RESTARTS   AGE
prometheus-grafana-76bf4556cc-abcde                      3/3     Running   0          2d
prometheus-kube-prometheus-stack-prometheus-0            2/2     Running   0          2d
prometheus-kube-state-metrics-5d76d57b88-fghij           1/1     Running   0          2d
prometheus-prometheus-node-exporter-klmno                1/1     Running   0          2d
prometheus-prometheus-node-exporter-pqrst                1/1     Running   0          2d

Step 2: Retrieve the Service IP Addresses

Find the ClusterIP services for the exporters:

kubectl get svc -n monitoring

Look for the ports and internal IPs:

• Node Exporter: Port 9100
• Cube State Metrics: Port 8080

Step 3: Query the Raw Endpoints

Log into one of your Kubernetes nodes (via AWS Systems Manager Session Manager, or by running minikube ssh if using a local cluster) and execute a curl request against the exporter's internal IP.

Querying Node Exporter (Port 9100):

curl http://10.100.12.34:9100/metrics

Example Output:

# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 1.04857e+06
node_cpu_seconds_total{cpu="0",mode="system"} 1245.82
node_cpu_seconds_total{cpu="0",mode="user"} 4582.11
# HELP node_memory_Active_bytes Memory information field Active_bytes.
# TYPE node_memory_Active_bytes gauge
node_memory_Active_bytes 1.23498e+09

Querying Cube State Metrics (Port 8080):

curl http://10.100.12.56:8080/metrics

Example Output:

# HELP kube_pod_container_status_restarts_total The number of times the container was restarted.
# TYPE kube_pod_container_status_restarts_total counter
kube_pod_container_status_restarts_total{container="main",namespace="default",pod="app-pod-xyz"} 0

6. Simulating a Production Failure: The Crash Loop Back-Off Demo

To observe how Prometheus and Cube State Metrics capture real-time failures, we will deploy a pod designed to crash repeatedly.

Step 1: Deploy a Crashing Pod

Run a pod that immediately exits with an error code (exit 1). Kubernetes will attempt to restart it, causing it to enter a CrashLoopBackOff state.

kubectl run busybox-crash --image=busybox --restart=Always -- /bin/sh -c "exit 1"

Step 2: Watch the Restarts Increase

Monitor the pod status in the default namespace:

kubectl get pods --watch

NAME            READY   STATUS             RESTARTS      AGE
busybox-crash   0/1     CrashLoopBackOff   3 (45s ago)   1m15s

Step 3: Query the Restart Metric in Prometheus

Open your Prometheus UI (exposed via kubectl port-forward svc/prometheus-kube-prometheus-stack-prometheus 9090:9090 -n monitoring) and enter the following query:

kube_pod_container_status_restarts_total{namespace="default"}

Click Execute and switch to the Graph tab. You will see a step-like graph showing the restarts increasing over time.

Conceptually: Kubernetes uses an exponential back-off delay algorithm for crashing pods. The delay between restarts increases progressively (e.g., 10s, 20s, 40s, 80s, up to a maximum of 5 minutes) to prevent a failing application from overloading the node's container runtime or CPU. This back-off behavior is clearly visible in the Prometheus graph as the horizontal plateaus between steps grow wider.

7. Querying with PromQL: Basic Selectors and Aggregations

PromQL allows you to filter, aggregate, and evaluate time series data in real time.

Filtering with Labels

You can filter metrics by appending curly braces {} containing label matchers:

• Equality Matcher (=): Get restarts only in the default namespace.
  PROMQL

  Copy

  kube_pod_container_status_restarts_total{namespace="default"}
  

• Inequality Matcher (!=): Get restarts in all namespaces except kube-system.
  PROMQL

  Copy

  kube_pod_container_status_restarts_total{namespace!="kube-system"}
  

Aggregation Operators

Raw metrics can be extremely noisy when you have hundreds of pods. PromQL provides aggregation operators to summarize data.

1. sum()

Calculates the total sum over all matching dimensions. For example, to find the total number of container restarts across the entire cluster:

sum(kube_pod_container_status_restarts_total)

To group the sum by a specific dimension, such as namespace, use the by modifier:

sum(kube_pod_container_status_restarts_total) by (namespace)

2. avg()

Calculates the average value across dimensions. This is highly useful for infrastructure metrics like CPU and memory utilization:

avg(node_cpu_seconds_total{mode="idle"}) by (instance)

8. Prometheus vs. Grafana: The Visualization & Access Control Layer

While Prometheus includes a built-in web interface, production environments use Grafana for visualization and dashboard management.

9. Building a Custom Grafana Dashboard

Let's build a custom dashboard in Grafana to track our crashing pods.

Step 1: Access Grafana

Expose the Grafana service locally:

kubectl port-forward svc/prometheus-grafana 3000:80 -n monitoring

Navigate to http://localhost:3000 in your browser. Log in using the default credentials (Username: admin, Password: prom-operator or your custom configured password).

Step 2: Create a New Dashboard

1. Click the + (New) button in the top right and select Dashboard.
2. Click Add a new panel.
3. In the Data source dropdown, select Prometheus.

Step 3: Configure the Query

In the query editor, input our PromQL query:

kube_pod_container_status_restarts_total{namespace="default"}

Select Run query.

Step 4: Customize and Save

1. On the right-hand panel, change the Title to Default Namespace Pod Restarts.
2. Change the graph style if desired (e.g., line, bar, or state timeline).
3. Click Apply in the top right corner.
4. Click the Save icon, name your dashboard Kubernetes App Health, and assign it to a folder.

10. Looking Ahead: Custom Metrics, Metric Types, and Service Monitors

While Node Exporter and Cube State Metrics cover infrastructure and cluster state, they do not capture application-level business logic (e.g., payment transaction failures, active user sessions, or API latency).

Metric Types in Prometheus

When developers instrument custom metrics in their code, they must choose from four primary metric types:

1. Counter: A cumulative metric that only increases or resets to zero on restart (e.g., http_requests_total).
2. Gauge: A metric that can go up and down (e.g., active_db_connections, memory_usage_bytes).
3. Histogram: Samples observations (usually things like request durations or response sizes) and counts them in configurable buckets.
4. Summary: Similar to a histogram, but it calculates configurable quantiles (e.g., p95, p99) over a sliding time window.

The Service Monitor Pattern

In a large cluster with thousands of pods, Prometheus cannot scrape everything without running out of memory and CPU. To solve this, Prometheus Operator introduces a Custom Resource Definition (CRD) called a ServiceMonitor.

A ServiceMonitor declaratively specifies how groups of Kubernetes services should be monitored. Prometheus reads these resources to dynamically update its scrape configuration, ensuring it only targets services that are explicitly configured to expose metrics.

11. Summary and Key Takeaways

• Pull-Based Architecture: Prometheus collects metrics by periodically pulling data from HTTP /metrics endpoints exposed by targets or exporters.
• Exporters as Translators: Exporters bridge the gap between systems that do not natively support Prometheus and the Prometheus TSDB.
• Node Exporter: Deployed as a DaemonSet to collect OS and hardware-level metrics from every node in the cluster.
• Cube State Metrics: Deployed as a Deployment to query the Kubernetes API server and generate metrics about the state of cluster objects (pods, deployments, configmaps).
• TSDB Storage: Metrics are stored as time series data, combining metric names, key-value labels, timestamps, and numeric values.
• PromQL: A powerful query language used to filter and aggregate raw time series data using functions like sum(), avg(), and label matchers.
• Grafana's Role: Grafana acts as a dedicated visualization and access control layer, decoupling dashboard presentation from the underlying Prometheus data source.
• Exponential Back-Off: Kubernetes automatically delays restarts for failing pods, a pattern that is easily visualized in Prometheus using the kube_pod_container_status_restarts_total metric.