Day-4 | Custom Metrics Instrumentation, Scraping with Prom Client, and Real-Time Email Alerts

This chapter explores the practical implementation of custom metrics instrumentation within application code and the configuration of real-time alerting pipelines. You will learn how to write application-specific metrics using a Node.js client library, expose them to Prometheus via Service Discovery, and configure Alertmanager to route critical system alerts directly to an email inbox.

1. The Core Concept of Instrumentation: Why Exporters Aren't Enough

In modern observability, setting up a robust monitoring stack (Prometheus, Grafana, EFK, Jaeger) is only half the battle. Observability is a collective responsibility shared between platform/SRE teams and application developers.

If your applications do not actively emit metrics, write structured logs, or propagate trace contexts, your observability platform remains empty and ineffective.

The Limitations of Standard Exporters

While standard exporters (such as node_exporter or mysqld_exporter) are excellent for capturing system-level metrics (CPU, memory, disk I/O) or database performance, they operate as external plugins. They cannot capture domain-specific business logic or granular application behaviors, such as:

• The number of users who logged in or registered an account over the last 30 days.
• The processing latency of a highly specific internal function or API endpoint.
• The number of items added to a shopping cart but abandoned before checkout.

To capture these high-value business and application-level metrics, you must implement instrumentation—the process of writing code directly inside your application to measure and expose its internal state.

2. Mental Model: Metric Types vs. Programming Data Types

To understand how to record metrics, it helps to use a programming analogy.

Real-Life Analogy

Mental Hook: Think of metric types as programming data types. Just as you cannot represent every piece of data as a single type—using strings for text, lists for ordered collections, and dictionaries for key-value pairs—you cannot represent every system behavior with a single metric type.

You must analyze the nature of the data you want to track and map it to the correct metric type.

3. Deep Dive into Prometheus Metric Types

Prometheus defines four primary metric types to represent different kinds of telemetry data.

Understanding Histogram Buckets

When tracking latency, a single average value is often misleading (the "flaw of averages"). Instead, histograms group observations into cumulative buckets. For example, if you define buckets at $0.5\text{s}$, $1.0\text{s}$, and $5.0\text{s}$:

• A request taking $0.3\text{s}$ increments the $\le 0.5\text{s}$, $\le 1.0\text{s}$, and $\le 5.0\text{s}$ buckets.
• A request taking $0.8\text{s}$ increments the $\le 1.0\text{s}$ and $\le 5.0\text{s}$ buckets.
• This allows you to calculate precise percentiles (e.g., 95th percentile latency) using the histogram_quantile() function in PromQL.

4. Step-by-Step Code Walkthrough: Instrumenting a Node.js Application

To implement custom metrics in a Node.js application, we use the official prom-client library. Below is a realistic implementation showing how to initialize the client, define a Counter and a Histogram, and expose them on a /metrics endpoint.

// index.js
const express = require('express');
const promClient = require('prom-client');

const app = express();
const port = 8080;

// 1. Initialize the Prometheus client and collect default system metrics
const register = new promClient.Registry();
promClient.collectDefaultMetrics({ register });

// 2. Define a Custom Counter Metric
const httpRequestCounter = new promClient.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests received',
  labelNames: ['method', 'route', 'status_code'],
});
register.registerMetric(httpRequestCounter);

// 3. Define a Custom Histogram Metric with explicit latency buckets (in seconds)
const httpRequestDurationHistogram = new promClient.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.1, 0.5, 1.0, 5.0, 10.0] // Buckets representing latency thresholds
});
register.registerMetric(httpRequestDurationHistogram);

// Middleware to track request metrics
app.use((req, res, next) => {
  const end = httpRequestDurationHistogram.startTimer();
  
  res.on('finish', () => {
    const labels = {
      method: req.method,
      route: req.route ? req.route.path : req.url,
      status_code: res.statusCode
    };
    
    // Increment the counter
    httpRequestCounter.inc(labels);
    
    // End the timer and record the duration in the histogram
    end(labels);
  });
  
  next();
});

// 4. Expose the /metrics endpoint for Prometheus scraping
app.get('/metrics', async (req, res) => {
  res.setHeader('Content-Type', register.contentType);
  res.send(await register.metrics());
});

// Sample Application Routes
app.get('/healthy', (req, res) => {
  res.status(200).send({ status: 'UP' });
});

app.get('/crash', (req, res) => {
  // Intentional crash endpoint to test alerting
  process.exit(1);
});

app.listen(port, () => {
  console.log(`Application running on port ${port}`);
});

5. The Three-Step Lifecycle of Custom Metrics

Simply writing the instrumentation code does not automatically make the metrics visible in Prometheus. You must complete a three-step operational lifecycle:

Step 1: Instrument Code
   |
   | Developers write prom-client code
   v
Step 2: Expose Endpoint
   |
   | Application serves metrics on /metrics
   v
Step 3: Service Discovery
   |
   | SREs configure ServiceMonitor to tell Prometheus where to scrape
   v
Prometheus Scrapes Metrics

1. Instrument: Write the code in the application using a client library (e.g., prom-client for Node.js, client_golang for Go).
2. Expose: Deploy the application and expose the /metrics endpoint over HTTP.
3. Service Discovery: Configure Prometheus to discover and scrape the application. In Kubernetes, this is typically handled by deploying a ServiceMonitor custom resource.

6. Deploying the Application and Configuring Service Discovery via ServiceMonitor

Once your application is containerized and running in a Kubernetes cluster, you must tell the Prometheus Operator how to find it. This is achieved using a ServiceMonitor.

The ServiceMonitor Manifest

The following manifest instructs Prometheus to scrape any Service in the dev namespace that matches the label app: service-a on the port named http every 30 seconds.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: service-a-monitor
  namespace: dev
  labels:
    release: prometheus-stack # Must match the label selector of your Prometheus Operator
spec:
  selector:
    matchLabels:
      app: service-a
  namespaceSelector:
    matchNames:
      - dev
  endpoints:
  - port: http
    path: /metrics
    interval: 30s

Applying the Manifests

To deploy the application and configure the service monitor, execute the following commands:

# 1. Create the development namespace
kubectl create namespace dev

# 2. Deploy the application services (Service A and Service B)
kubectl apply -k ./day4/kubernetes-manifests/ -n dev

# 3. Apply the ServiceMonitor to register the target with Prometheus
kubectl apply -f service-monitor.yaml -n dev

Note: After applying the ServiceMonitor, Prometheus will automatically discover the new target within 1-2 minutes. You can verify this by navigating to the Status -> Targets page in the Prometheus UI and searching for service-a-monitor.

7. Configuring Alertmanager for Real-Time Email Notifications

Alertmanager handles alerts sent by client applications such as Prometheus. It takes care of deduplicating, grouping, and routing them to the correct receiver integration (e.g., Email, Slack, PagerDuty).

To route alerts to an email address (such as Gmail), you must configure SMTP authentication.

Step 1: Generate a Gmail App Password

If you are using Gmail, you cannot use your standard password due to security restrictions. You must generate an App Password:

1. Go to your Google Account settings.
2. Navigate to Security and enable 2-Step Verification (if not already enabled).
3. Search for App Passwords.
4. Create a new app password (e.g., Name: alertmanager).
5. Copy the generated 16-character password.

Step 2: Create a Kubernetes Secret for SMTP Credentials

Convert your generated password to Base64 and store it securely in a Kubernetes Secret.

# Convert your password to base64
echo -n "your-16-char-app-password" | base64

Create the email-secrets.yaml file:

apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-email-secrets
  namespace: monitoring
type: Opaque
data:
  smtp-password: <BASE64_ENCODED_PASSWORD>

Apply the secret to your cluster:

kubectl apply -f email-secrets.yaml

Step 3: Configure Alertmanager Routing Rules

Update your Alertmanager configuration to define the SMTP server, authentication credentials, and the destination email address.

global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: 'your-alert-sender@gmail.com'
  smtp_auth_username: 'your-alert-sender@gmail.com'
  smtp_auth_password: 'your-16-char-app-password'
  smtp_require_tls: true

route:
  group_by: ['alertname', 'namespace']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'email-receiver'

receivers:
- name: 'email-receiver'
  email_configs:
  - to: 'your-personal-destination@gmail.com'
    send_resolved: true

Apply the updated configuration to your Prometheus/Alertmanager stack using Helm or kubectl:

helm upgrade prometheus-stack prometheus-community/kube-prometheus-stack \
  -n monitoring \
  -f alertmanager-values.yaml

8. Testing the Alerting Pipeline: Simulating Production Failures

To verify that the entire pipeline—from application failure to Prometheus detection to Alertmanager email dispatch—works seamlessly, we can intentionally trigger a pod crash.

The Crash Workflow

Step-by-Step Execution

1. Trigger the Crash: Access the application's /crash endpoint via your browser or curl:
   BASH

   Copy

   curl http://<external-ip>/crash
   

2. Verify Pod Status: Check the status of the pod in the dev namespace. You should see the restart count increment:
   BASH

   Copy

   kubectl get pods -n dev
   

   Output:
   TEXT

   Copy

   NAME                         READY   STATUS             RESTARTS      AGE
   service-a-67b89df5c4-xxxxx   0/1     CrashLoopBackOff   2 (15s ago)   5m
   

3. Check Prometheus Alerts: Open the Prometheus UI and navigate to the Alerts tab. You will see the alert rule (e.g., KubePodCrashLooping or PodRestartRate) transition from PENDING to FIRING.
4. Verify Email Delivery: Within seconds of the alert entering the FIRING state, check your destination email inbox. You will receive a structured notification from Alertmanager detailing the failing pod, namespace, and error state.

9. Summary and Key Takeaways

• Instrumentation is Essential: Standard infrastructure exporters cannot capture internal application state or business logic. Developers must write instrumentation code to expose custom metrics.
• Metric Types Map to Behaviors: Use Counters for values that only increase (e.g., request counts), Gauges for values that fluctuate (e.g., memory usage), and Histograms for measuring durations and latencies in buckets.
• The 3-Step Lifecycle: Custom metrics require application instrumentation, exposing a /metrics endpoint, and configuring Service Discovery (via ServiceMonitor) so Prometheus can scrape them.
• Service Discovery Automates Scraping: The Prometheus Operator uses ServiceMonitor custom resources to dynamically discover and scrape targets based on Kubernetes label selectors.
• Alertmanager Decouples Alerting: Prometheus detects anomalies and forwards them to Alertmanager, which handles deduplication, grouping, and routing to endpoints like email (SMTP), Slack, or PagerDuty.
• Secure Secret Management: Sensitive credentials (such as SMTP passwords) must be stored in Kubernetes Secrets and injected into the Alertmanager configuration rather than hardcoded.