Kubernetes Bootcamp - Part 10: Kubernetes Volumes

By default, container filesystems are ephemeral, meaning any data generated or modified inside a container is permanently lost when the container stops or crashes. Kubernetes Volumes solve this fundamental problem by providing persistent data abstractions that decouple storage from the pod and container lifecycles.

1. The Ephemeral Container Dilemma

Containers are designed as lightweight, isolated Linux processes. Because of this design, the writable container layer is bound directly to the container process lifetime.

The Problem in Stateful Applications

• Stateless Workloads: Microservices, API gateways, and web frontends do not rely on local state. If a container dies, a new instance spins up without issue.
• Stateful Workloads: Databases (MySQL, PostgreSQL, Redis), search indices, and machine learning pipelines require data to persist across container crashes, restarts, and pod rescheduled events.

Note: Relying on default container storage for stateful applications leads to catastrophic data loss during back-off restarts or node failures.

2. Fundamentals of Kubernetes Volumes

At its core, a Kubernetes Volume is a directory on disk or in remote storage accessible to the containers within a pod.

Key Characteristics

1. Directory Abstraction: A volume is exposed to the container as a filesystem directory mounted at a user-specified path (mountPath).
2. Pod-Scoped Lifecycle: Unlike container storage, Kubernetes volumes persist across container restarts within the same pod.
3. Shared File System: Multiple containers running inside the same pod can mount the same volume simultaneously to share files.

Real-Life Analogy

Think of a Kubernetes volume as a shared external hard drive plugged into a server rack (the Pod). Even if the server CPU or operating system (the Container) restarts, the external drive remains plugged in and its files stay completely intact.

3. Storage Architecture Evolution: In-Tree vs. CSI Drivers

Historically, Kubernetes shipped with "in-tree" storage plugins, meaning code for providers like AWS EBS, GCP PD, and Azure Disk resided inside the core Kubernetes codebase.

Modern Container Storage Interface (CSI)

As of Kubernetes 1.30+, in-tree storage plugins are deprecated and removed to keep the core Kubernetes binary lightweight and modular.

• Out-of-Tree Architecture: Storage vendors develop and maintain their own drivers independently using the Container Storage Interface (CSI) standard.
• Popular Storage Drivers:
  • Cloud-Native / Open Source: Longhorn, Rook-Ceph, Local Path Provisioner, Portworx.
  • Cloud Managed: AWS EBS CSI Driver, GCP Persistent Disk CSI Driver, Azure Disk CSI Driver.

4. Ephemeral Volume Type: emptyDir

An emptyDir volume is created when a Pod is assigned to a Node. It initially starts as an empty directory.

Key Characteristics

• Lifecycle: Bound strictly to the Pod lifecycle (not the container lifecycle). If a container crashes, data in emptyDir survives. If the Pod is removed from the node, the emptyDir is erased permanently.
• Storage Options:
  • Disk: Saved on the host node's SSD/HDD (default).
  • Memory (RAM): Saved as a tmpfs ramdisk (medium: Memory). Fast read/write performance suitable for temporary caching.

Primary Use Cases

1. Scratch Space: Disk-based space for large computations or temporary sorting files.
2. Inter-Container Communication: Multi-container pods where one container downloads or generates data (e.g., an init container or sidecar) and another reads it.
3. In-Memory Caching: Extremely fast read/write cache using RAM.

Manifest Example: emptyDir Pod

apiVersion: v1
kind: Pod
metadata:
  name: emptydir-demo-pod
spec:
  containers:
  - name: writer-container
    image: alpine
    command: ["/bin/sh", "-c"]
    args:
    - echo "Hello from KubeSimplify!" > /data/hello.txt; sleep 3600
    volumeMounts:
    - name: cache-volume
      mountPath: /data
  volumes:
  - name: cache-volume
    emptyDir: {}

Memory Medium Variant:

YAML

Copy

  volumes:
  - name: cache-volume
    emptyDir:
      medium: Memory
      sizeLimit: 1Gi

5. Host-Bound Storage: hostPath

A hostPath volume mounts a pre-existing file or directory from the host node's filesystem directly into your pod.

Common hostPath Types

Manifest Example: hostPath Pod

apiVersion: v1
kind: Pod
metadata:
  name: hostpath-demo-pod
spec:
  containers:
  - name: app-container
    image: nginx
    volumeMounts:
    - name: node-log-volume
      mountPath: /app/logs
  volumes:
  - name: node-log-volume
    hostPath:
      path: /tmp/host-logs
      type: DirectoryOrCreate

Security Warning: hostPath creates security risks because a compromised container can gain elevated read/write access to the host node's underlying filesystem. Avoid using hostPath for general application workloads in production.

6. Decoupled Storage Architecture: PV, PVC, and StorageClass

To keep infrastructure separated from application development, Kubernetes introduces two distinct resources: PersistentVolume (PV) and PersistentVolumeClaim (PVC).

Roles and Responsibilities

1. PersistentVolume (PV):
   • Represent physical storage in the cluster (provisioned by an admin or CSI driver).
   • Has its own lifecycle independent of any pod using it.
2. PersistentVolumeClaim (PVC):
   • A request for storage by a user/developer.
   • Specifies requirements: size (e.g., 10Gi), access modes (ReadWriteOnce), and storage class constraints.
3. StorageClass:
   • Profiles the "class" of storage available (e.g., fast-ssd, slow-hdd, nfs-storage).
   • Enables Dynamic Provisioning.

7. Provisioning Strategies: Static vs. Dynamic Provisioning

Storage can be provisioned in Kubernetes through two primary models:

Strategy Comparison

Manifest Example: StatefulSet with Dynamic volumeClaimTemplates

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql-set
spec:
  serviceName: "mysql"
  replicas: 2
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:8.0
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "passthrough"
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
  volumeClaimTemplates:
  - metadata:
      name: mysql-persistent-storage
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "local-path"
      resources:
        requests:
          storage: 10Gi

8. Deep Dive: Access Modes, Reclaim Policies, and Binding Modes

1. Access Modes

• ReadWriteOnce (RWO): The volume can be mounted as read-write by a single node. Multiple pods on the same node can read/write to it.
• ReadOnlyMany (ROX): The volume can be mounted as read-only by many nodes.
• ReadWriteMany (RWX): The volume can be mounted as read-write by many nodes simultaneously (e.g., NFS, GlusterFS).
• ReadWriteOncePod (RWOP): Restricts access to a single pod across the entire cluster (introduced in newer K8s versions).

2. Reclaim Policies (What happens when a PVC is deleted?)

• Retain: Manual reclamation. When the PVC is deleted, the PV remains intact, and the data is preserved. No other PVC can bind to it until clean-up occurs.
• Delete: Automatic deletion. Deleting the PVC deletes both the PersistentVolume object and the actual underlying physical storage (e.g., AWS EBS volume).
• Recycle (Deprecated): Performs a basic scrub (rm -rf /volume/*) to make the volume available for a new claim.

3. Volume Binding Modes

• Immediate: Storage volume and PV are provisioned and bound as soon as the PersistentVolumeClaim is created.
• WaitForFirstConsumer: Delays the binding and provisioning of a PV until a Pod using the PVC is created. This ensures the volume is provisioned in a topology (Availability Zone / Host Node) where the Pod is actually scheduled.

9. Network & Remote Storage Walkthrough: NFS Setup

Network File System (NFS) provides an ideal environment for ReadWriteMany (RWX) workloads, allowing multiple pods across different worker nodes to access the same directory.

Step 1: Setting Up the NFS Server on Host/VM

# Install NFS Kernel Server
sudo apt-get update
sudo apt-get install -y nfs-kernel-server

# Create exported directory and set permissions
sudo mkdir -p /var/nfs/general
sudo chown nobody:nogroup /var/nfs/general
sudo chmod 777 /var/nfs/general

# Configure exports file
# Add rule: /var/nfs/general *(rw,sync,no_subtree_check,no_root_squash)
echo "/var/nfs/general *(rw,sync,no_subtree_check,no_root_squash)" | sudo tee -a /etc/exports

# Export shares and restart service
sudo exportfs -a
sudo systemctl restart nfs-kernel-server

Step 2: Define Static NFS PersistentVolume (nfs-pv.yaml)

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  nfs:
    path: /var/nfs/general
    server: 172.31.2.2  # IP address of the NFS Server

Step 3: Define PersistentVolumeClaim (nfs-pvc.yaml)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

Step 4: Mount PVC into Pod Workload (nfs-pod.yaml)

apiVersion: v1
kind: Pod
metadata:
  name: nfs-app-pod
spec:
  containers:
  - name: app
    image: alpine
    command: ["/bin/sh", "-c", "echo 'Written to NFS Shared Volume' > /mnt/nfs/hello.txt; sleep 3600"]
    volumeMounts:
    - name: nfs-storage
      mountPath: /mnt/nfs
  volumes:
  - name: nfs-storage
    persistentVolumeClaim:
      claimName: nfs-pvc

10. Enterprise & Distributed Cloud-Native Storage Options

Production environments require robust, scalable, self-healing, and replicated storage solutions.

Highlighted Enterprise Solutions

1. Longhorn:
   • Lightweight, cloud-native, distributed block storage built specifically for Kubernetes.
   • Provides incremental block-level backups to external secondary storage (S3/NFS) and snapshot management.
   • Features a built-in UI dashboard for monitoring volume health and replica attachments.
2. AWS EBS CSI Driver:
   • Connects AWS Elastic Block Store directly to Kubernetes clusters (EKS or self-managed).
   • Supports dynamic provisioning using custom StorageClasses (e.g., gp2, gp3).
3. AWS S3 CSI Driver (mountpoint-s3):
   • Mounts S3 object storage buckets directly into pods as standard file systems.
   • Commonly used for massive data lakes, machine learning datasets, and LLM model storage (e.g., serving open-source models like Llama 3 via vLLM).

11. Complete Workflow: Pod-to-Storage Mounting Pipeline

The diagram below illustrates how a Pod consumes storage dynamically through Kubernetes abstractions.

Summary and Key Takeaways

• Container Ephemerality: Default container storage is temporary; when a container crashes or dies, its writable layer is wiped out.
• Volume Purpose: Kubernetes Volumes decouple storage lifecycle from individual container runtimes, ensuring data survives container restarts.
• CSI Standardization: As of K8s 1.30+, legacy in-tree storage providers are deprecated in favor of external Container Storage Interface (CSI) drivers.
• emptyDir Scenarios: Best used for scratch space, temporary caches, and multi-container sharing within the same pod. It resides on disk or memory (tmpfs) and dies when the pod is deleted.
• hostPath Scenarios: Mounts direct host node filesystems into containers. Useful for node monitoring and system utilities, but introduces security risks.
• PV vs. PVC Abstraction: PersistentVolumes (PVs) represent physical/remote storage resources, while PersistentVolumeClaims (PVCs) act as developer storage requests.
• Dynamic Provisioning: StorageClasses automate storage allocation by triggering CSI drivers to provision physical volumes dynamically when a PVC is applied.
• Access Modes: RWO (ReadWriteOnce) limits access to a single node, while RWX (ReadWriteMany) enables multi-node simultaneous write access (essential for NFS/GlusterFS).
• WaitForFirstConsumer: A critical volume binding mode that delays volume provisioning until a pod is scheduled, ensuring storage is created in the correct availability zone or node topology.