Kubernetes Networking: Services, StatefulSets, Ingress, and ExternalDNS

Kubernetes implements a flat network model where every Pod receives its own IP address and can communicate directly with other Pods without Network Address Translation (NAT). However, because Pods are ephemeral, their IP addresses constantly change, necessitating robust abstractions for internal routing, sticky network identity, external exposure, and automated DNS synchronization.

1. The Core Networking Problem: Ephemeral Pods and Dynamic Topologies

In Kubernetes, when an application pod is scheduled, the Container Network Interface (CNI) assigns it an internal IP address dynamically from the cluster's Pod CIDR block.

If a client application or an upstream microservice hardcodes a Pod's IP address, any Pod crash, restart, rescheduling, or scaling event breaks communication immediately.

Furthermore, Pod IP addresses are routed internally within the software-defined overlay network. External clients on the public internet or corporate LAN cannot route traffic directly to a Pod's private IP.

To solve this, Kubernetes decouples the consumer of a workload from the backend Pods using two primitives:

1. Stable Network Abstractions (Service): Provides a single, static virtual IP (VIP) and DNS record that sits in front of dynamic sets of Pods.
2. Predictable Lifecycle Workloads (StatefulSet): Guarantees persistent storage bindings and deterministic hostnames for databases and stateful distributed systems.

2. Under the Hood: Linux Namespaces, Veth Pairs, and the Pause Container

Before diving into Services, understanding how a Pod connects to the host network is critical. A Pod is not a virtualization boundary; it is a collection of Linux namespaces (net, ipc, uts, pid, mount) and cgroups.

The Multi-Container Pod Mental Model

1. The Pause Container: When Kubernetes creates a Pod, it first initializes an infrastructure container known as the pause container. This lightweight container holds open the network namespace (netns), IPC namespace, and UTS namespace. All application containers defined in the Pod spec join the network namespace of this pause container using Docker/containerd container-sharing flags.
2. Virtual Ethernet (veth) Pairs: The CNI plugin (such as Calico, Cilium, or Flannel) provisions a virtual Ethernet pair. One end of the pair (veth*) resides in the host's root network namespace, while the peer end appears as eth0 inside the Pod’s network namespace.

Inspecting Pod Networking on the Worker Node

To verify this architecture directly on a Kubernetes node:

# List all network namespaces managed on the node
ip netns list

# Inspect processes sharing the namespace (locating the pause container)
lsns -t net

# Identify the host-side veth pair corresponding to the Pod's eth0 interface
ip netns exec <netns-name> ip link show

Packet Flow: Intra-Node vs. Inter-Node Communication

• Intra-Node (Pod A to Pod B on Node 1): Traffic leaves Pod A's eth0, traverses the veth pair into the host root namespace, hits the local bridge or routing table, and is switched/routed directly to Pod B's peer veth interface via Address Resolution Protocol (ARP).
• Inter-Node (Pod A on Node 1 to Pod C on Node 2): Because the destination IP belongs to a different subnet, the host's routing table directs the packet to the default gateway or overlay tunnel (e.g., VXLAN/Geneve) managed by the CNI, encapsulating or routing the packet across the physical network to Node 2.

3. The Kubernetes Service Abstraction

A Kubernetes Service is a native API resource that defines a logical set of Pods and a policy by which to access them. It gives workloads a single, stable IP address and a stable CoreDNS entry.

How Endpoints and EndpointSlices Work

A Service discovers its backend Pods dynamically using Label Selectors:

apiVersion: v1
kind: Service
metadata:
  name: backend-service
spec:
  selector:
    app: backend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

1. The control plane watches the API server for Pods matching app: backend.
2. Kubernetes automatically creates an EndpointSlice (or Endpoints) resource containing the IP addresses and ports of all healthy Pods passing readiness probes.
3. Every EndpointSlice holds up to 100 endpoints, enhancing scalability across massive clusters compared to monolithic Endpoints resources.

Note: If a Service is defined without selectors, Kubernetes will not automatically create EndpointSlices. This pattern is commonly used to manually route cluster traffic to external static workloads or databases.

4. Deep Dive into Kubernetes Service Types

Kubernetes provides multiple Service types depending on where traffic originates and how it should be routed.

1. ClusterIP (Default)

• Behavior: Allocates an internal, non-routable virtual IP from the cluster's Service CIDR.
• Reachability: Accessible only from within the cluster (or via kubectl proxy / kubectl port-forward during debugging).
• Primary Use Case: Inter-service communication between internal architectural tiers (e.g., Frontend to Backend, Backend to Internal Cache).

apiVersion: v1
kind: Service
metadata:
  name: internal-api
spec:
  type: ClusterIP
  selector:
    app: internal-api
  ports:
    - port: 80
      targetPort: 5000

2. NodePort

• Behavior: Builds on top of ClusterIP. Kubernetes allocates a high-range dedicated port across every node in the cluster (default range: 30000–32767).
• Reachability: External traffic hitting <Any-Node-IP>:<NodePort> is forwarded via kube-proxy packet translation to the backend Pods.
• Primary Use Case: Development testing, internal VPN access (e.g., exposing an internal dashboard like Rancher or Kubernetes Dashboard without purchasing public IPs).

apiVersion: v1
kind: Service
metadata:
  name: nodeport-service
spec:
  type: NodePort
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 80
      nodePort: 32033

3. LoadBalancer

• Behavior: Builds on top of NodePort and ClusterIP. Interacts with the Cloud Controller Manager (CCM) to provision an external cloud load balancer (e.g., AWS NLB/ALB, Google Cloud LB, Civo Load Balancer).
• Reachability: The cloud provider assigns a public IP address or DNS hostname, which automatically forwards traffic to the allocated NodePort and ClusterIP.
• Limitation: In bare-metal or environments like KillerCoda without a CCM, a LoadBalancer service remains permanently in <pending>.

apiVersion: v1
kind: Service
metadata:
  name: cloud-loadbalancer
spec:
  type: LoadBalancer
  selector:
    app: production-web
  ports:
    - port: 80
      targetPort: 80

5. Debugging Services: Inspecting iptables Rules and Packet Counters

Kubernetes Services do not run as listening processes on the host. Instead, kube-proxy programs the Linux kernel using Netfilter/iptables (or IPVS) to intercept traffic sent to the Service VIP or NodePort and translate it to the target Pod IP via DNAT (Destination NAT).

Verifying NodePort iptables Entry

When a NodePort service is created on port 32033, kube-proxy injects rules into the KUBE-NODEPORTS chain on every node:

# Inspect rules matching the specific NodePort
iptables -t nat -L KUBE-NODEPORTS -n -v | grep 32033

The output displays packet counters alongside the forwarding target:

    0     0 KUBE-EXT-XXXXX  tcp  --  *  *  0.0.0.0/0  0.0.0.0/0  tcp dpt:32033

When you execute repeated curl requests against the NodePort:

curl http://<Node-IP>:32033

Rerunning the iptables inspection shows incrementing packet and byte counters:

    4   240 KUBE-EXT-XXXXX  tcp  --  *  *  0.0.0.0/0  0.0.0.0/0  tcp dpt:32033

Production Debugging Tip: If traffic to a NodePort or ClusterIP hangs or times out, checking the iptables counters immediately isolates whether the packets are successfully reaching the node's kernel or being blocked upstream by external firewalls and security groups.

6. Deployments vs. StatefulSets: Handling State and Storage

Standard Deployment controllers are designed strictly for stateless workloads (e.g., REST APIs, web UIs). They treat Pods as interchangeable, disposable units.

The Problem with Stateful Workloads in Deployments

If a database (e.g., PostgreSQL, MongoDB) is managed by a Deployment:

1. Unpredictable Identifiers: Pod names are randomly hashed. A replica cannot identify if it is the primary writer or a secondary read replica.
2. Storage Contention: All replicas share the same persistent volume claims unless configured manually, causing read/write locks or database corruption.
3. Chaotic Scaling & Termination: Pods scale up and down in arbitrary order, making distributed quorum consensus and log replication unreliable.

The StatefulSet Controller

A StatefulSet provides guarantees tailored for stateful systems:

• Sticky Network Identity: Pods receive a deterministic, zero-indexed ordinal name: <statefulset-name>-0, <statefulset-name>-1, <statefulset-name>-2.
• Ordered Deployment and Scaling: Pods are provisioned sequentially ($0 \to 1 \to 2$). Pod $N$ must be fully Running and Ready before Pod $N+1$ is initialized.
• Ordered Termination: Scale-down occurs in reverse order ($2 \to 1 \to 0$), preventing distributed state inconsistency.
• Dedicated Persistent Volume Claims: Each Pod replica gets a distinct Persistent Volume via the volumeClaimTemplates field.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: "postgres-headless"
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:15
          ports:
            - containerPort: 5432
              name: postgres
          env:
            - name: POSTGRES_PASSWORD
              value: "password"
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "local-path"
        resources:
          requests:
            storage: 1Gi

Industry Reality: Managing raw distributed databases (clustering, failovers, backups) manually via pure StatefulSets requires significant operational overhead. In production, teams frequently use purpose-built Kubernetes Operators such as CloudNativePG, Percona Operators, or KubeDB, which automate primary-standby failovers, WAL replication, and disaster recovery.

7. Headless Services (clusterIP: None)

Stateful distributed systems (like clustered databases or Kafka brokers) require direct peer-to-peer communication without an intermediary virtual IP load-balancing requests randomly between master and replicas.

To solve this, Kubernetes supports Headless Services by explicitly declaring:

spec:
  clusterIP: None

apiVersion: v1
kind: Service
metadata:
  name: postgres-headless
spec:
  clusterIP: None
  selector:
    app: postgres
  ports:
    - port: 5432
      name: postgres

DNS Resolution of Headless Services

When clusterIP: None is set:

1. No virtual ClusterIP is allocated.
2. An A record lookup for postgres-headless.default.svc.cluster.local returns the raw IP addresses of all individual backend Pods directly.
3. Every individual Pod associated with the StatefulSet receives its own deterministic DNS A record:

<pod-name>.<service-name>.<namespace>.svc.cluster.local

For instance:

This allows applications to route read-write traffic specifically to postgres-0 (the primary) and read-only queries to postgres-1 or postgres-2.

8. Cross-Namespace and External Routing with ExternalName Services

The ExternalName service is a specialized service type that does not use selectors, endpoints, or proxying. Instead, it acts as an internal alias returning a DNS CNAME record.

Key Use Cases

1. Routing to External Infrastructure: Connecting cluster applications to managed cloud databases (e.g., AWS RDS) outside the cluster using an internal stable alias.
2. Cross-Namespace Service Discovery: Abstracting multi-tenant or enterprise architectures where microservices in one namespace need to consume services in another without hardcoding target namespaces into the application source code.

apiVersion: v1
kind: Service
metadata:
  name: external-db-service
  namespace: app-namespace
spec:
  type: ExternalName
  externalName: my-database-service.db-namespace.svc.cluster.local

If the database moves to an external cloud or a different namespace later, only the externalName definition changes; application configs remain completely untouched.

9. Advanced Edge Routing: Ingress and Ingress Controllers

While a Service operates at Layer 4 (TCP/UDP transport layer), modern microservice architectures require Layer 7 (HTTP/HTTPS) routing, SSL termination, and path-based request distribution.

Provisioning a separate cloud LoadBalancer Service for every microservice is financially prohibitive and architecturally unmanageable.