What is Kafka? How to use Confluent Kafka in Spring Boot Application?

This comprehensive guide covers the fundamentals of Apache Kafka, its internal architecture (topics, partitions, offsets, consumers, consumer groups, and commits), running a local multi-broker Kafka cluster, and integrating Confluent Cloud Kafka into a enterprise Spring Boot application for asynchronous decoupled event-driven messaging.

1. Why Do We Need Kafka? (The Problem Statement & Status Quo)

In modern web applications, user actions (such as liking a post, uploading a story, or commenting) generate constant, continuous streams of data.

The Naive Synchronous Approach (Without Kafka)

When a user performs an action, the application directly fires synchronous REST API calls to multiple downstream services simultaneously:

• User clicks Like -> Direct HTTP POST to Analytics Service.
• User clicks Like -> Direct HTTP POST to Notification/Email Service.
• User clicks Like -> Direct HTTP POST to Feed Update Service.

Critical Production Pitfalls of Direct API Communication:

1. Tight Coupling: The primary service is tightly bound to all downstream consumers. If any consuming service is slow or down, the producer thread blocks or fails.
2. Cascading Failures & Data Loss: If the Notification Service goes down during a traffic spike, incoming likes fail to trigger notifications, resulting in permanent data/event loss.
3. Overwhelming Load: Under heavy traffic (e.g., 1 million requests/sec), downstream microservices collapse under the sheer volume of synchronous API requests.

2. Core Mental Model: The Kafka Event Streaming Architecture

Kafka acts as an Open-Source Distributed Event Streaming Platform that sits between Producers (sources generating data) and Consumers (services processing data).

Real-Life Analogy: The Postal Mailroom Workflow

Think of Kafka as a central Postal Sorting Facility and Mailbox Hub. When millions of people drop letters (events) into mailboxes, the post office doesn't hand-deliver them instantly to every recipient one by one. Instead, it collects them temporarily, sorts them into organized bins (partitions), and lets postal carriers (consumers) pick them up at their own pace without overwhelming the final destination.

The Asynchronous Decoupled Flow:

3. Kafka Internal Components & Terminology

To understand Kafka under the hood, we must explore its core building blocks:

4. Topics, Partitions, and Offsets

Topics (The Logical Table)

A Topic is a named category or feed channel (similar to a database table) where similar events are stored. For example, a food topic stores food-related records, while a weekly-sentiments topic stores sentiment analysis records.

Partitions (Scalability and Parallelism)

A single topic is split into multiple physical chunks called Partitions distributed across brokers.

• Actual Message Storage: Data sent by producers eventually lands inside partitions.
• Creation Configuration: When creating a topic, you define how many partitions it should have.

Offsets (The Consumer Bookmark)

• Each message inside a partition is assigned a sequential, monotonically increasing integer ID known as its Offset (0, 1, 2, 3, ...).
• Ordering Guarantee: Ordering is strictly maintained only at the Partition level, not across the entire topic.

5. Message Keys, Partitioners, and Ordering Guarantees

Producers send messages containing a Key and a Value (where the Key is optional).

A. Sending Messages WITHOUT a Key (Round-Robin)

• If no key is provided, the producer's partitioner distributes messages across available partitions in a Round-Robin fashion (Partition 0 -> Partition 1 -> Partition 2).
• Result: Message ordering is not guaranteed across partitions because consumers poll multiple partitions concurrently.

B. Sending Messages WITH a Key (Hashing & Stickiness)

• When a key is provided (e.g., key: "hello"), the partitioner applies a hashing function to the key:

• Result: All messages with the exact same key always route to the same partition. Consequently, strict ordering is maintained for all messages sharing that key.

6. Consumer Groups and Offsets Management

Consumer Groups

A Consumer Group is a set of independent consumer instances working together to consume data from a topic.

• Partition Assignment: The Group Coordinator ensures each partition is consumed by only one consumer within a given consumer group at a time (preventing duplicate processing).
• Sticky Assignment: Coordinators assign partitions in a "sticky" fashion so consumers continue reading from the same partitions they handled previously, ensuring operational stability.

The Built-In __consumer_offsets Topic

Kafka tracks consumer progress using a special internal built-in topic named __consumer_offsets.

• Each consumer group periodically commits its latest read position (Consumer Offset) to this internal topic.
• If a consumer crashes and restarts, it queries __consumer_offsets to resume reading exactly where it left off, preventing data loss.

7. Storage Engine: Segments, Commit Logs, and Retention Policies

How Kafka Stores Data on Disk

Kafka does not store everything in a single massive file. Instead:

1. Each partition is divided into smaller files called Segments.
2. New records are appended to an active log file (.log on disk in directories like /tmp/kafka-logs/).
3. Once a segment reaches its maximum size limit, a new segment file is created.

Retention Policies

To prevent disk exhaustion, Kafka implements two primary retention policies managed by a background Log Cleaner process:

• Size-Based Retention: If partition logs exceed a maximum byte size (e.g., log.segment.bytes), the oldest segment files are purged.
• Time-Based Retention: If records exceed a specified time duration (e.g., default 168 hours / 7 days), older records are automatically deleted.

8. Setting Up a Multi-Broker Local Kafka Cluster

To establish high availability and fault tolerance, you can run multiple local brokers sharing a ZooKeeper instance by duplicating and configuring server.properties.

Step-by-Step Local Multi-Broker Configuration:

1. Start ZooKeeper:
   BASH

   Copy

   bin/zookeeper-server-start.bat config/zookeeper.properties
   

2. Configure Broker 0 (config/server.properties):
   PROPERTIES

   Copy

   broker.id=0
   listeners=PLAINTEXT://localhost:9092
   log.dirs=/tmp/kafka-logs-0
   

3. Configure Broker 1 (config/server-1.properties):
   PROPERTIES

   Copy

   broker.id=1
   listeners=PLAINTEXT://localhost:9094
   log.dirs=/tmp/kafka-logs-1
   

4. Configure Broker 2 (config/server-2.properties):
   PROPERTIES

   Copy

   broker.id=2
   listeners=PLAINTEXT://localhost:9096
   log.dirs=/tmp/kafka-logs-2
   

5. Start all Brokers using their respective property files.

Creating a Topic with Replication Factor 3:

To ensure fault tolerance, replicate data across multiple brokers:

bin/kafka-topics.bat --create --bootstrap-server localhost:9092,localhost:9094,localhost:9096 --replication-factor 3 --partitions 3 --topic gadgets

9. Integrating Confluent Cloud Kafka in Spring Boot

Instead of managing local brokers in production, enterprise applications use fully managed cloud providers like Confluent Cloud Kafka.

Step 1: Add Maven Dependencies (pom.xml)

Include the Spring Kafka starter in your project:

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

Step 2: Configure Application Properties (application.yml)

Configure connection credentials, security protocols, serializers, and deserializers for Confluent Cloud:

spring:
  kafka:
    bootstrap-servers: <YOUR-CONFLUENT-BOOTSTRAP-SERVER>:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
    consumer:
      group-id: weekly-sentiment-group
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring.json.trusted.packages: "*"
        security.protocol: SASL_SSL
        sasl.mechanism: PLAIN
        sasl.jaas.config: 'org.apache.kafka.common.security.plain.PlainLoginModule required username="<YOUR-API-KEY>" password="<YOUR-API-SECRET>";'
        session.timeout.ms: 45000

10. Building the Spring Boot Producer & Consumer

A. Producer Service (Decoupling Email Scheduling)

Instead of sending emails synchronously (which risks failures during downtime), publish sentiment events to Kafka:

@Service
public class UserScheduler {

    private static final String TOPIC = "weekly-sentiments";

    @Autowired
    private KafkaTemplate<String, SentimentData> kafkaTemplate;

    public void publishSentiment(SentimentData sentimentData) {
        // Publish event asynchronously to Confluent Kafka topic
        kafkaTemplate.send(TOPIC, sentimentData.getEmail(), sentimentData);
    }
}

B. Consumer Service (Async Event Processing)

Listen to the Kafka topic asynchronously using @KafkaListener and process the data safely:

@Service
public class SentimentConsumerService {

    @Autowired
    private EmailService emailService;

    @KafkaListener(topics = "weekly-sentiments", groupId = "weekly-sentiment-group")
    public void consumeSentimentEvent(SentimentData sentimentData) {
        // Safely process message and send email asynchronously
        emailService.sendEmail(
            sentimentData.getEmail(),
            "Weekly Sentiment Report",
            "Your sentiment for last week was: " + sentimentData.getSentiment()
        );
    }
}

C. Payload POJO (SentimentData.java)

public class SentimentData {
    private String email;
    private String sentiment;

    // Constructors, Getters, and Setters
    public SentimentData() {}

    public SentimentData(String email, String sentiment) {
        this.email = email;
        this.sentiment = sentiment;
    }

    public String getEmail() { return email; }
    public void setEmail(String email) { this.email = email; }

    public String getSentiment() { return sentiment; }
    public void setSentiment(String sentiment) { this.sentiment = sentiment; }
}

11. Summary and Key Takeaways

• Distributed Streaming: Kafka is an open-source distributed event streaming platform built to handle massive, high-throughput, continuous data feeds across multiple servers.
• Decoupling Systems: Introducing Kafka between producers and consumers eliminates tight coupling and prevents cascading failures when downstream microservices go down.
• Topics & Partitions: Data is organized into logical Topics which are physically split into Partitions across cluster brokers for horizontal scalability.
• Partition Ordering: Message ordering is guaranteed only at the partition level. Sending messages with a Key ensures related events route to the same partition.
• Consumer Groups & Offsets: Consumer groups coordinate parallel processing, while the __consumer_offsets topic tracks reading progress (bookmarks) to ensure zero data loss.
• Fault Tolerance via Replication: Replication factors greater than 1 ensure that if a broker crashes, follower replicas take over seamlessly as leaders.
• Cloud Managed Kafka: Using Confluent Cloud eliminates local infrastructure overhead, allowing secure, production-ready integration with Spring Boot via spring-kafka.