Spring Boot 4 - Full Microservices Course - Home Energy Tracker - System Design (Part 1)

Welcome to the foundational system design session of the Home Energy Tracker project. This course covers building an enterprise-grade, event-driven microservices architecture using Spring Boot 4. Throughout this series, we will transition from raw mathematical requirements and capacity planning all the way to a robust distributed system design capable of ingesting millions of high-frequency telemetry events in real time.

1. Prerequisites and Technology Stack

To successfully follow along with this microservices implementation, certain baseline competencies and local development tools are required.

Core Prerequisites

Java Object-Oriented Programming: Solid understanding of OOP principles (inheritance, polymorphism, encapsulation, and exception handling).
Spring Boot Fundamentals: Familiarity with dependency injection, Spring MVC, REST controllers, and application configuration.

Required Tooling

Java Development Kit (JDK): Modern LTS version compatible with Spring Boot 4.
Build Tool: Maven or Gradle.
Containerization: Docker and Docker Compose (for running localized infrastructure like MySQL, Apache Kafka, and time-series databases).
Messaging Infrastructure: Apache Kafka.
Databases: MySQL (relational data) and a Time-Series Database (e.g., TimeScaleDB or InfluxDB).

Note: The source code for this course is hosted on GitHub. Each distinct section of the course is tagged corresponding to its implementation milestone, allowing you to check out any specific commit tag if you experience issues or join the curriculum midway.

2. Electrical Energy Foundations and Calculations

Before designing database tables or writing microservices, we must understand the electrical domain we are modeling. Electrical units can often be confusing; let's break down the core equations.

Key Electrical Metrics

Power ( $W$ or $kW$ ): Indicates how much energy a device consumes per second (the instantaneous rate of energy consumption). For example, a $1,200\text{ W}$ hair dryer consumes $1.2\text{ kilowatts}$ of power per second.
Energy ( $kWh$ ): Kilowatt-hour. This is the cumulative energy used over a specific duration of time (typically 1 hour).
Utility Price ( $/kWh$ ): The monetary rate charged by the electrical utility provider for each kilowatt-hour consumed.

Practical Calculation Example

Let's calculate the energy consumption and running cost for a $1.2\text{ kW}$ hair dryer:

1-Hour Usage:

\text{Energy} = 1.2\text{ kW} \times 1\text{ hour} = 1.2\text{ kWh}

1-Minute Usage:

\text{Energy} = 1.2\text{ kW} \times \left(\frac{1}{60}\text{ hours}\right) = 0.02\text{ kWh}

Running Cost Calculation: Assuming an electricity tariff of $0.30 per kWh:
- Cost for 1 Hour: $1.2\text{ kWh} \times \char36 0.30 = \char36 0.36$
- Cost for 1 Minute: $0.02\text{ kWh} \times \char36 0.30 = \char36 0.006$

These foundational formulas dictate how our Java applications will process sensor payloads and model device usage metrics.

3. System Requirements & Functional Scope

Our Home Energy Tracker microservices platform must satisfy several core functional requirements:

Consumption Tracking: Track daily and hourly energy consumption aggregated per individual device and per room.
Trend Analysis: Provide historical analytics and trends over customized time windows.
Real-Time Anomaly Alerting: Allow users to configure usage thresholds, triggering instant high-consumption alerts when limits are breached.
AI-Powered Recommendations: Integrate an AI service to deliver personalized energy-saving tips based on usage patterns.
Real-Time IoT Ingestion: Accept high-frequency sensor readings in real time from millions of connected household appliances.

4. Capacity Planning & Load Estimation

To design an architecture that doesn't collapse under production loads, we must perform realistic back-of-the-envelope calculations based on our target scale assumptions.

Scale Assumptions

Active Monthly Users: $500,000$ users
Devices per User: Average of $5$ devices
Telemetry Frequency: Each device sends 1 reading every minute
Data Model: Each reading translates to 1 database write operation

Mathematical Derivations

Total Active Devices:

500,000 \text{ users} \times 5 \text{ devices/user} = \mathbf{2,500,000 \text{ devices}}

Readings Per Minute (System-Wide Ingestion Rate):

2,500,000 \text{ devices} \times 1 \text{ reading/minute} = \mathbf{2,500,000 \text{ readings/minute}}

Readings Per Day:

2,500,000 \times 60 \text{ minutes} \times 24 \text{ hours} = \mathbf{3,600,000,000 \text{ rows/day (3.6 billion)}}

Storage Footprint Calculation:
- Assume each reading record contains: id, device_id, kilowatt_used, and timestamp (~ $72\text{ bytes}$ per row).
- Storage Required Per Day:

3,600,000,000 \text{ rows} \times 72 \text{ bytes} \approx \mathbf{259 \text{ GB/day}}

Storage Required Per Month:

259 \text{ GB} \times 30 \text{ days} \approx \mathbf{7.77 \text{ TB/month}}

Applying a conservative 50% data compression rate reduces this to approximately $4\text{ TB/month}$ .
Write Throughput (Write Operations Per Second):

\frac{2,500,000 \text{ readings}}{60 \text{ seconds}} \approx \mathbf{41,666 \text{ inserts/second (~40,000 inserts/sec)}}

Engineering Takeaway: Processing ~40,000 writes per second with payloads arriving every minute requires an asynchronous, decoupled event-driven streaming architecture rather than synchronous REST calls directly hitting a relational database.

5. Architectural Evolution: Naive Approach vs. Event-Driven Microservices

The Naive Monolithic / Direct-Write Trap

If our IoT devices sent HTTP requests directly to an API Gateway, which in turn executed synchronous JDBC inserts into a standard relational database (like MySQL):

Database connection pools would exhaust almost instantly under 40,000 concurrent writes/sec.
Network latency spikes or database locks would cause cascading failures up to the API Gateway and disconnect client IoT devices.
Relational databases are poorly optimized for multi-billion-row time-series append workloads.

The Production-Grade Event-Driven Evolution

To solve this, we decouple our ingestion layer from our storage engine using Apache Kafka as a high-throughput distributed commit log, and leverage a specialized Time-Series Database (TSDB) for metrics persistence.

6. Comprehensive Microservices Architecture Walkthrough

Below is the complete architectural layout of our Home Energy Tracker system, illustrating how traffic flows from IoT sensors and web clients through our decoupled Spring Boot microservices.

Structure & Concept Flow

[IoT Devices / Sensors (2.5M devices)]

| HTTPS / MQTT

[API Gateway (Proxy)]

+----------------------------------+

| |

v v

[Device Service] [Ingestion Service]

(MySQL DB) (Kafka Producer)

[Apache Kafka Topic]

[Usage Service]

+-----------------------+-----------------------+

| |

v v

[Time-Series DB (TSDB)] [Kafka Alert Topic]

(InfluxDB / TimeScaleDB) |

[Client (Web / Mobile App)] <---> [Insight Service] <---> [Alert Service]

| |

+-----------------> [User Service] <--------------------+

(MySQL DB)

7. Deep Dive Into Microservice Components

Let's examine the specific responsibilities of each microservice in our ecosystem:

Microservice / Component	Primary Responsibility	Data Store / Tech Stack
API Gateway	Acts as the single entry point for clients; proxies requests securely without business logic overhead.	Spring Cloud Gateway
Device Service	Manages appliance registration, metadata, room mapping, and device lifecycles.	MySQL Database
User Service	Handles user profiles, authentication preferences, contact details, and notification thresholds.	MySQL Database
Ingestion Service	Lightweight producer service that receives raw telemetry from the API Gateway and pushes messages to Kafka.	Spring Boot + Kafka Producer
Apache Kafka	Absorbs traffic spikes, buffering ~40,000 writes/sec to ensure fault tolerance and asynchronous processing.	Distributed Commit Log
Usage Service	Consumes telemetry events from Kafka, runs scheduled threshold checks, and writes bulk records to storage.	Spring Boot + Time-Series DB
Time-Series DB	Optimized columnar storage for high-frequency time-stamped metrics (billions of rows).	TimeScaleDB / InfluxDB
Insight Service	Interfaces with local AI models to analyze consumption trends and generate energy-saving recommendations.	Spring Boot + AI Integration
Alert Service	Consumes threshold-breach events, queries user contact details, dispatches alerts, and writes audit logs.	Spring Boot + MySQL (Audit Log)

During initial system design, an engineer might be tempted to wire the API Gateway directly as a Kafka producer forwarding messages from IoT devices. However, this introduces architectural coupling:

Gateway Bloat: The API Gateway should strictly handle cross-cutting concerns (routing, authentication, rate limiting, SSL termination). It should not contain data ingestion or event-publishing logic.
Blast Radius Control: If Kafka experiences latency or broker degradation, a direct producer in the API Gateway would saturate gateway thread pools, crashing user web applications and device management endpoints alike.
Dedicated Ingestion Tier: By isolating telemetry intake into a dedicated Ingestion Service, we can scale the ingestion tier independently from routing and management APIs.

9. The Alerting & Scheduling Workflow

The alert pipeline demonstrates how our system handles background processing and asynchronous notifications:

Structure & Concept Flow

[Usage Service (Scheduled Job)]

| Evaluates user thresholds every N minutes

[Threshold Exceeded?]

+---> Yes ---> [Produces Kafka Alert Event]

[Alert Service]

+---> Queries [User Service] for email address

+---> Dispatches Notification Email

[MySQL Audit Table] (Records sent alert for auditing)

Scheduled Evaluation: The Usage Service runs background schedulers that periodically query aggregated time-series data against user-defined limits stored in the User Service.
Event Dispatch: If a threshold violation is detected, the Usage Service publishes an event to a dedicated Kafka alert topic.
Notification & Auditing: The Alert Service consumes the alert event, fetches recipient details from the relational database, triggers the email dispatch, and persists an immutable audit record into MySQL.

Summary and Key Takeaways

Domain Understanding: Accurate energy tracking requires distinguishing between instantaneous power ( $W$ / $kW$ ), cumulative energy ( $kWh$ ), and monetary tariffs ( $/kWh$ ).
Scale Realities: Supporting $2.5$ million devices sending telemetry every minute generates $\sim$ 3.6 billion rows per day, requiring robust capacity planning and high-throughput infrastructure.
Throughput Demands: Processing $\sim$ 40,000 write operations per second makes synchronous relational database inserts impossible without crashing under load.
Event-Driven Decoupling: Apache Kafka acts as a shock absorber between lightweight ingestion services and persistent data stores.
Storage Strategy: Relational databases (MySQL) are ideal for transactional metadata (users, devices, audit logs), while Time-Series Databases (TimeScaleDB/InfluxDB) are essential for high-frequency metrics.
Gateway Separation: The API Gateway must remain lightweight and act strictly as a reverse proxy, delegating heavy telemetry ingestion to dedicated microservices.
Asynchronous Alerting: Background scheduler jobs combined with event topics allow decoupled alert services to evaluate thresholds and notify users without blocking real-time telemetry ingestion.

Spring Boot 4 - Full Microservices course - Home Energy Tracker - System Design (Part 1)