Kubernetes Deployments, ReplicaSets, Health Probes, and Rollout Strategies

This chapter establishes the core architectural foundations of application scaling, continuous updates, and self-healing within Kubernetes. It covers the structural relationship between Pods, ReplicaSets, and Deployments, dissects update strategies (RollingUpdate vs. Recreate), details health-checking mechanisms (Startup, Liveness, and Readiness probes), and explores deployment patterns including Blue-Green and Canary releases.

1. High Availability Architecture: Moving Beyond Standalone Pods

In a production environment, running a standalone Pod introduces a single point of failure (SPOF). When thousands of concurrent user requests hit a single Pod, the underlying container will inevitably exhaust its assigned compute resources (CPU/Memory) and terminate abruptly with an Out-of-Memory (OOMKilled) or unhandled failure state.

To guarantee high availability, fault tolerance, and zero-downtime scalability, workloads are distributed horizontally across multiple identical Pod instances (replicas). Inbound traffic is routed across these replicas via Ingress controllers, Cloud Load Balancers, or the Kubernetes Gateway API.

To manage this fleet of Pods dynamically without manual intervention, Kubernetes uses layered controller abstractions:

2. Core Mental Model: The Hierarchical Object Relationship

Real-Life Analogy

Think of a school organization:

• The Pod is the individual student doing the actual work.
• The ReplicaSet is the Class Monitor, whose sole responsibility is headcount—ensuring exactly $N$ students are sitting in the room at any given moment.
• The Deployment is the Head Teacher / Principal, who manages curriculum changes, upgrades textbooks across semesters, orchestrates transitions, and oversees the class monitor.

Deployment Controller
   |
   | Manages revisions & declarative state
   v
ReplicaSet Controller (Hash: e.g., rs-7998b8c57d)
   |
   | Enforces replica count via Label Selectors
   v
Managed Pods (Hash: rs-7998b8c57d-xxxx)

3. ReplicaSets: Declarative Specification & Selector Mechanics

A ReplicaSet's sole duty is to maintain a stable set of replica Pods running at any given time. It is driven by the ReplicaSet Controller running inside the kube-controller-manager.

3.1 ReplicaSet Manifest (nginx-rs.yaml)

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: nginx-rs
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

3.2 Label Selector Decoupling and "Non-Homogeneous" Pod Adoption

ReplicaSets discover and manage Pods strictly through Label Selectors (spec.selector.matchLabels), not by Pod identity, name, or container image. This loose coupling leads to specific operational behaviors:

ReplicaSet (spec.replicas: 3, selector: app=nginx)
       |
       | Discovers existing Pods matching "app=nginx"
       v
Active Pool: [Pod A (app=nginx), Pod B (app=nginx), Pod C (app=nginx)]

Behavior 1: Unintentional Pod Adoption or Immediate Eviction

1. If $3$ replicas are running under nginx-rs, and an independent standalone Pod is deployed with the identical label (app: nginx), the ReplicaSet Controller detects $4$ running Pods against a desired count of $3$. It will immediately terminate one Pod to restore the desired state.
2. Conversely, if an independent Pod with app: nginx is already running before the ReplicaSet is applied, the ReplicaSet adopts that Pod into its pool and only spins up $2$ additional Pods.

Behavior 2: Quarantining / Debugging Faulty Pods in Production

When a specific Pod in a ReplicaSet misbehaves in production, you can isolate it without deleting it:

• Change or remove the label from the running Pod (e.g., update app: nginx to app: debug-target).
• The ReplicaSet Controller notices the active count dropped from $3$ to $2$, and instantly provisions a new healthy Pod to maintain capacity.
• The isolated Pod remains running independently, allowing engineering teams to attach debuggers, inspect memory dumps, and examine logs in place.

4. Deletion Propagation Policies & Scale-Down Precedence

When deleting a ReplicaSet through the Kubernetes API (/apis/apps/v1/namespaces/{ns}/replicasets/{name}), you can control how child Pods are handled using the propagationPolicy field.

4.1 ReplicaSet Scale-Down Precedence

When a ReplicaSet scales down (e.g., from 5 replicas to 3), the ReplicaSet Controller selects Pods for termination in a deterministic sequence:

1. Pending Pods First: Unscheduled or unready Pods are terminated before running Pods.
2. Deletion Cost Annotation: Pods with the lowest value in the annotation controller.kubernetes.io/pod-deletion-cost (default: 0) are terminated first.
3. Node Distribution: Pods residing on nodes hosting the highest number of replicas of this ReplicaSet are terminated to preserve cluster balance.
4. Creation Timestamp: The most recently created Pods are terminated before older, established Pods.

5. Kubernetes Deployments: Declarative Lifecycle Management

Deployments provide declarative updates for Pods and ReplicaSets. You describe a desired state in a Deployment, and the Deployment Controller changes the actual state to the desired state at a controlled rate.

5.1 Generating Scaffolding Imperatively

To quickly generate a production-ready manifest without memorizing boilerplate fields, use client-side dry runs:

kubectl create deployment boot-camp \
  --image=nginx:1.14.2 \
  --replicas=4 \
  --dry-run=client \
  -o yaml > deployment.yaml

5.2 Declarative Deployment Manifest (deployment.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: boot-camp
  labels:
    app: boot-camp
spec:
  replicas: 4
  selector:
    matchLabels:
      app: boot-camp
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
  template:
    metadata:
      labels:
        app: boot-camp
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "250m"
            memory: "256Mi"

6. Deployment Update Strategies: RollingUpdate vs. Recreate

Kubernetes provides two primary built-in deployment update strategies configured via spec.strategy.type.

6.1 RollingUpdate Strategy (Default)

The RollingUpdate strategy replaces old Pods with new ones gradually, ensuring continuous application availability with zero downtime.

Initial State: [Pod v1] [Pod v1] [Pod v1] [Pod v1]  (RS-v1: 4 replicas)
      |
      | Step 1: MaxSurge creates 1 new Pod
      v
Surge:         [Pod v1] [Pod v1] [Pod v1] [Pod v1] [Pod v2 (Starting)]
      |
      | Step 2: Pod v2 becomes Ready; MaxUnavailable terminates 1 old Pod
      v
Transition:    [Pod v1] [Pod v1] [Pod v1] [Pod v2 (Ready)]
      |
      | Step 3: Continues until all replicas run v2
      v
Final State:   [Pod v2] [Pod v2] [Pod v2] [Pod v2]  (RS-v2: 4 replicas, RS-v1: 0 replicas)

Tuning Parameters: maxSurge and maxUnavailable

• maxSurge: The maximum number of Pods that can be created above the desired replica count during an update. Can be an absolute number (e.g., 1) or a percentage (e.g., 25%).
• maxUnavailable: The maximum number of Pods that can be unavailable during the update process. Can be an absolute number or a percentage.

6.2 Recreate Strategy

The Recreate strategy terminates all existing Pods simultaneously before any new Pods are provisioned.

Initial State:  [Pod v1] [Pod v1] [Pod v1]
       |
       | All v1 Pods terminated at once
       v
Downtime Window: (0 Pods Active / Serving Traffic)
       |
       | All v2 Pods scheduled and started
       v
Final State:    [Pod v2] [Pod v2] [Pod v2]

7. Rollout Management, History Tracking, and Rollback Workflows

Every time a Deployment's Pod template (spec.template) is modified (e.g., updating container images, environment variables, or resource limits), a rollout is triggered. The Deployment creates a new ReplicaSet and manages the transition.

7.1 Execution Commands for Deployment Management

# Trigger an image update across all containers named 'nginx'
kubectl set image deployment/boot-camp nginx=nginx:1.16.0

# Track rollout progress in real time
kubectl rollout status deployment/boot-camp

# Inspect revision history
kubectl rollout history deployment/boot-camp

# View details of a specific revision
kubectl rollout history deployment/boot-camp --revision=2

# Roll back to the immediate previous revision
kubectl rollout undo deployment/boot-camp

# Roll back to a specific revision target
kubectl rollout undo deployment/boot-camp --to-revision=1

7.2 Production Debugging: Mitigating Faulty Rollouts

When an invalid image tag (e.g., nginx:invalid-tag) or breaking application code is deployed, the update stalls:

Note on Annotations vs. --record: The --record flag in kubectl has been deprecated. To maintain clear audit trails in kubectl rollout history, apply change-cause metadata using annotations directly:

BASH

Copy

kubectl annotate deployment/boot-camp kubernetes.io/change-cause="Upgraded nginx to 1.16.0 for security patch"

8. Container Health Checks: Startup, Liveness, and Readiness Probes

By default, Kubernetes considers a container healthy as long as its primary process (PID 1) is running. However, applications can crash internally, enter deadlocks, or take minutes to initialize while PID 1 remains active. Health probes solve this by monitoring actual application state.

8.1 The Three Probe Archetypes

1. Startup Probe (startupProbe):

   • Purpose: Verifies whether legacy or slow-starting applications have fully initialized.
   • Behavior: Disables Liveness and Readiness checks until the Startup probe succeeds.
   • Failure Action: Kills and restarts the container if it fails past failureThreshold.

2. Readiness Probe (readinessProbe):

   • Purpose: Determines if the container is ready to accept inbound network traffic.
   • Behavior: Checks application readiness (e.g., database connections established, cache warmed).
   • Failure Action: Removes the Pod IP from the associated Kubernetes Service endpoints. The container is not killed or restarted.

3. Liveness Probe (livenessProbe):

   • Purpose: Detects internal deadlocks, corrupted state, or frozen runtimes.
   • Behavior: Periodically verifies that the application remains functional.
   • Failure Action: Kills the container and initiates a restart according to the Pod's restartPolicy.

9. Probe Handlers and Tuning Parameters

9.1 The Four Probe Detection Mechanisms

Probes can check container health using four distinct mechanisms:

9.2 Complete Multi-Probe Manifest (probes-demo.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: probe-bootcamp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: probe-app
  template:
    metadata:
      labels:
        app: probe-app
    spec:
      containers:
      - name: web-app
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        startupProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 10
          periodSeconds: 5
          failureThreshold: 30
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 15
          periodSeconds: 10
          timeoutSeconds: 2
          failureThreshold: 3

9.3 Probe Configuration Parameters

10. Progressive Delivery Patterns: Blue-Green and Canary Deployments

Beyond native Rolling Updates, production systems often use advanced release strategies to minimize risk when releasing new code.

10.1 Blue-Green Deployments

In a Blue-Green deployment, two identical environments exist side by side:

• Blue: The current live production version ($v1$).
• Green: The new target version ($v2$).

• Workflow:
  1. Version $v1$ (Blue) is active behind the Service.
  2. Version $v2$ (Green) is deployed with its own Deployment manifest.
  3. Green undergoes smoke tests and automated verification with zero user traffic.
  4. The Service selector is updated (version: v2), instantly switching 100% of user traffic to Green.
  5. If unexpected issues arise, changing the selector back to version: v1 provides immediate rollback.

10.2 Canary Deployments

Canary releases route a small percentage of user traffic to the new version ($v2$) while serving the majority from the stable version ($v1$). As confidence increases, traffic to $v2$ is increased incrementally.

Native Kubernetes Workaround (Using Label Selectors & Replica Weights)

A basic Canary setup can be built natively using two Deployments behind a single Service:

• Deployment-Stable: 9 replicas with label app: api, track: stable (v1).
• Deployment-Canary: 1 replica with label app: api, track: canary (v2).
• Service: Selects app: api.

Production Note: Native selector-based traffic splitting is coarse and hard to scale (e.g., achieving a $1\%$ split requires 100 total Pods). In modern production environments, use the Kubernetes Gateway API, an Ingress Controller (e.g., NGINX with canary annotations), or a Service Mesh (e.g., Istio, Linkerd) to split traffic precisely by percentage, HTTP headers, or cookies.

# Conceptual Gateway API HTTPRoute Weighted Traffic Split
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-canary-route
spec:
  rules:
  - forwardTo:
    - serviceName: api-stable-svc
      port: 80
      weight: 90
    - serviceName: api-canary-svc
      port: 80
      weight: 10

11. Summary and Key Takeaways

• Hierarchical Abstraction: Deployments manage ReplicaSets, and ReplicaSets manage Pods. Never deploy unmanaged standalone Pods in production.
• Label-Driven Associations: ReplicaSets and Services track Pods solely through label matching (spec.selector.matchLabels), allowing pods to be quarantined for debugging by modifying their labels.
• Cascading Deletion Controls: The propagationPolicy (Foreground, Background, Orphan) governs whether child Pods terminate with or outlive their parent ReplicaSet.
• Deterministic Scale-Down: When scaling down, Kubernetes prioritizes terminating unassigned/pending Pods, low-deletion-cost Pods (pod-deletion-cost), Pods on dense nodes, and finally the newest Pods.
• Zero-Downtime Rollouts: RollingUpdate uses maxSurge and maxUnavailable to cycle Pods gradually. Recreate incurs downtime by killing all existing Pods before starting new ones.
• Rollback Safety: Revisions are tracked automatically. If a new deployment fails, kubectl rollout undo reverts the cluster to the previous healthy ReplicaSet.
• Health Probe Trio:
  • startupProbe: Protects slow-starting containers from early termination.
  • readinessProbe: Controls traffic routing; failed probes remove the Pod from Service endpoints without restarting it.
  • livenessProbe: Detects unrecoverable deadlocks and restarts failing containers.
• Traffic Routing Evolution: While native label tricks allow rudimentary Blue-Green and Canary releases, modern architectures rely on the Kubernetes Gateway API, advanced Ingress controllers, or Service Meshes for reliable, weighted traffic splitting.