Spring Boot 4 Microservices Project: Home Energy Tracker - Observability (Part 14)

As microservices architectures scale, manually inspecting logs across multiple distributed services becomes impractical. This chapter introduces Observability—specifically system monitoring—using industry-standard tools: Prometheus for metrics collection and Grafana for visualization.

1. Why Do We Need Observability?

In a monolithic application, diagnosing a performance bottleneck or a service crash requires checking a single log file. In a distributed microservices system (such as our Home Energy Tracker containing a User Service, Device Service, Ingestion Service, Usage Service, Alert Service, Insight Service, and API Gateway), failure can happen anywhere in the network mesh.

• The Status Quo Challenge: Without centralized monitoring, developers must SSH into separate containers or tail dozens of independent log streams to find out which service is down or why error rates are spiking.
• The Solution: An observability pipeline that aggregates application metrics in real-time, displays them on a unified dashboard, and alerts operators instantly when anomalies occur.

2. Core Mental Model & Architecture Analogy

To understand how Prometheus, Grafana, and Micrometer work together, consider a large corporate office building:

• Micrometer (The Odometer/Counter inside each room): Embedded directly inside every Spring Boot microservice, Micrometer measures internal metrics (like JVM heap usage, request counts, and execution times) and translates them into a universal format.
• Prometheus (The Central Data Archivist / Brain): Running independently on port 9090, Prometheus periodically walks down the hallway (pulling/scraping data) and records the numbers reported by each service's odometer into its local time-series database.
• Grafana (The Executive Dashboard / Display Wall): Running on port 3000, Grafana pulls the historical and real-time records from Prometheus and presents them in clean, human-readable charts, graphs, and status panels.

Observability Data Flow Walkthrough

Spring Boot Microservice (Micrometer collects metrics)
   |
   | exposes endpoint: /actuator/prometheus
   v
Prometheus (Scrapes endpoint every 15s & stores in time-series DB)
   |
   | queries data
   v
Grafana UI (Renders charts on Dashboard, Port 3000)

3. Overview of Ports and Components

4. Configuring Docker Compose for Infrastructure

To run Prometheus and Grafana locally alongside our microservices, we define them inside our docker-compose.yml file. Prometheus requires a configuration file (prometheus.yml) to know which endpoints to scrape.

Docker Compose Service Definitions

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin

5. Setting Up Prometheus Scrape Configuration (prometheus.yml)

Prometheus uses a pull model. The prometheus.yml file lists all microservices in our system and instructs Prometheus where to pull metrics from. Because our Spring Boot services run on the host machine while Prometheus runs inside a Docker container, we use host.docker.internal to bridge the network boundary.

global:
scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'user-service'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['host.docker.internal:8080'] # (Adjust respective ports per service)

  - job_name: 'device-service'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['host.docker.internal:8081']

  - job_name: 'ingestion-service'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['host.docker.internal:8082']

6. Configuring Grafana Provisioning

Grafana needs to know two things automatically upon startup:

1. Data Source: Where to fetch the metrics from (Prometheus).
2. Dashboards: Which dashboard JSON templates to render on the user interface.

Grafana Data Source Provisioning (grafana/provisioning/datasources/prometheus.yaml)

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      timeInterval: 15s

Grafana Dashboard Provisioning (grafana/provisioning/dashboards/dashboards.yaml)

apiVersion: 1

providers:
  - name: 'Home Energy Tracker Dashboards'
    orgId: 1
    folder: ''
    type: file
    disableDeletion: false
    updateIntervalSeconds: 10
    options:
      path: /etc/grafana/provisioning/dashboards

Inside /grafana/provisioning/dashboards/, we place our dashboard definition file (home-energy-tracker-overview.json), which configures color palettes, panel titles (e.g., HTTP Request Rate by Status Code, Client Error Rate, JVM Heap Usage), and layout grids.

7. Adding Dependencies to Spring Boot Microservices

To expose metrics, every microservice's pom.xml must include two critical dependencies: Spring Boot Actuator and Micrometer Prometheus Registry.

Maven Dependencies (pom.xml)

<!-- Spring Boot Actuator for application health and status endpoints -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

<!-- Micrometer Prometheus Registry to translate metrics into Prometheus format -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

Note: For the API Gateway, the spring-boot-starter-actuator dependency is typically already present; we only need to append the micrometer-registry-prometheus registry library.

8. Exposing Actuator Endpoints in application.properties

By default, Spring Boot secures and hides sensitive actuator endpoints. We must explicitly enable web exposure for health, information, and Prometheus metrics, while assigning an application tag so metrics can be filtered per service.

Configuration Properties (application.properties)

# Expose health, info, and prometheus actuator endpoints over HTTP
management.endpoints.web.exposure.include=health,info,prometheus

# Tag every metric with its specific application name for isolation in Grafana
management.metrics.tags.application=user-service

9. Launching and Verifying the Observability Pipeline

With code and configurations in place, we spin up our infrastructure containers and microservices.

Step 1: Start Infrastructure via Docker Compose

Open your terminal in the root directory and run:

docker compose -v up -d

Step 2: Start Microservices

Boot up each Spring Boot application (User Service, Device Service, Ingestion Service, Usage Service, Insight Service, Alert Service, and API Gateway).

Step 3: Verify Raw Metrics Endpoint

Navigate in your browser to:

http://localhost:8080/actuator/prometheus

You will see plain text metrics generated by Micrometer in the format expected by Prometheus.

10. Exploring Prometheus Target Health

1. Open your browser and go to http://localhost:9090 (Prometheus UI).
2. Navigate to Status -> Targets.
3. Here, you will see the list of all registered microservices. If a service is running, its status will be marked as UP (Green).
4. Debugging Mindset (Handling Downtime): If you manually stop a service (e.g., the alert-service), Prometheus will reflect this change on the Target Health page, marking it as DOWN after the scrape timeout interval elapses.

11. Visualizing Metrics in Grafana Dashboards

1. Open your browser and go to http://localhost:3000 (Grafana UI).
2. Log in using default credentials:
   • Username: admin
   • Password: admin
3. Navigate to Dashboards and select Home Energy Tracker.
4. Live Validation Walkthrough:
   • Generate traffic by sending GET/POST requests to your services using Postman or an API client.
   • Observe real-time spikes in the HTTP Request Rate by Service and Status panel.
   • Stop a service (e.g., Ingestion Service) and watch the request rate drop to zero on the dashboard.
   • Restart the service and observe JVM heap usage metrics repopulating on the screen.

Summary and Key Takeaways

• Centralized Monitoring Need: Microservices architectures require unified observability tools like Prometheus and Grafana to prevent the operational overhead of checking fragmented logs.
• Prometheus Role: Acts as the brain and time-series database that periodically pulls metrics from target endpoints.
• Grafana Role: Acts as the visualization layer, connecting to Prometheus data sources to render clean, executive dashboards.
• Micrometer's Bridge: Acts as the metrics collection library inside Spring Boot that translates internal application data into a readable format.
• Actuator Configuration: management.endpoints.web.exposure.include=health,info,prometheus unlocks the /actuator/prometheus endpoint.
• Docker Networking Bridge: Using host.docker.internal allows containers running Prometheus to scrape Spring Boot apps running on the host machine.
• Production Resilience: Real-time dashboards provide instant visual feedback on JVM heap usage, HTTP status error rates (such as 4xx/5xx responses), and service health.