Kubernetes Bootcamp: Chapter 11 - End-to-End Production Project Deployment with HTTPS

A comprehensive guide to transitioning from basic Kubernetes concepts to deploying a production-ready, highly available microservices application. This walkthrough covers cloud cluster provisioning, automated database management using Kubernetes Operators, ingress routing, automated SSL/TLS certificate issuance via Cert-Manager, horizontal autoscaling, and real-time load testing.

1. Production Architecture & System Overview

Deploying an application to production requires more than wrapping a container in a Pod. A resilient cloud infrastructure demands decoupling state, managing external network traffic safely, securing communications with valid TLS certificates, and handling dynamic traffic spikes.

The Production Traffic Flow

2. Microservice Application Mechanics & Containerization

The sample application is a Flask-based web application backed by a PostgreSQL database. It features basic CRUD operations (viewing, adding, and removing goals) and exposes a health-check endpoint (/health) essential for production liveness and readiness probes.

Flask Application Logic (app.py)

import os
import psycopg2
from flask import Flask, render_template, request, redirect, jsonify

app = Flask(__name__)

def get_db_connection():
    conn = psycopg2.connect(
        host=os.environ.get("DB_HOST"),
        database=os.environ.get("DB_NAME"),
        user=os.environ.get("DB_USER"),
        password=os.environ.get("DB_PASSWORD"),
        port=os.environ.get("DB_PORT", 5432)
    )
    return conn

@app.route("/")
def index():
    conn = get_db_connection()
    cur = conn.cursor()
    cur.execute("SELECT id, goal_name FROM goals;")
    goals = cur.fetchall()
    cur.close()
    conn.close()
    return render_template("index.html", goals=goals)

@app.route("/add-goal", methods=["POST"])
def add_goal():
    goal = request.form.get("goal")
    if goal:
        conn = get_db_connection()
        cur = conn.cursor()
        cur.execute("INSERT INTO goals (goal_name) VALUES (%s);", (goal,))
        conn.commit()
        cur.close()
        conn.close()
    return redirect("/")

@app.route("/remove-goal", methods=["POST"])
def remove_goal():
    goal_id = request.form.get("goal_id")
    if goal_id:
        conn = get_db_connection()
        cur = conn.cursor()
        cur.execute("DELETE FROM goals WHERE id = %s;", (goal_id,))
        conn.commit()
        cur.close()
        conn.close()
    return redirect("/")

@app.route("/health")
def health():
    return jsonify(status="UP"), 200

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

Containerization Strategy (Dockerfile)

To run efficiently in Kubernetes, we package the application using Python slim images and Gunicorn as the WSGI production HTTP server.

FROM python:3.9-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8080

CMD ["gunicorn", "--bind", "0.0.0.0:8080", "app:app"]

Production Tip: When building multi-architecture Docker images (e.g., building on Apple Silicon M1/M2/M3 chips for linux/amd64 cloud nodes), specify the target platform explicitly: docker build --platform linux/amd64 -t registry/my-app:v1 .

3. Database Strategy: Naive Approaches vs. Operator Pattern

Running databases on Kubernetes has evolved significantly over time:

Why Use Operators for Databases?

While standard StatefulSets provide stable network identifiers and ordered deployment, they do not handle database-level state management (such as primary-replica replication, failover switching, and continuous WAL archiving).

The Operator Pattern extends Kubernetes by introducing Custom Resource Definitions (CRDs) along with custom controllers that encode human operational domain knowledge directly into automated software routines.

For PostgreSQL, CloudNativePG (CNPG) is a cloud-native operator that manages PostgreSQL clusters natively without relying on heavy external wrapper dependencies.

4. Deploying PostgreSQL using Custom Resource Definitions (CRDs)

Step 1: Install CloudNativePG Operator

kubectl apply --server-side -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/main/releases/cnpg-1.22.0.yaml

Step 2: Define Database Secret Credentials

apiVersion: v1
kind: Secret
metadata:
  name: my-postgres-credentials
type: Opaque
stringData:
  username: goals_user
  password: NewSecurePassword123

Step 3: Define the CloudNativePG PostgreSQL Cluster (postgres-cluster.yaml)

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: my-postgres-cluster
spec:
  instances: 3
  storage:
    size: 1Gi
  bootstrap:
    initdb:
      database: goals_db
      owner: goals_user
      secret:
        name: my-postgres-credentials

When applied, the CloudNativePG operator provisions 3 instances, handles master/replica election automatically, and creates distinct Kubernetes services for Read-Write (my-postgres-cluster-rw) and Read-Only (my-postgres-cluster-ro) operations.

Step 4: Schema Initialization

Forward the local port to access the primary PostgreSQL container and execute the database setup:

kubectl port-forward svc/my-postgres-cluster-rw 5432:5432

Execute initialization query:

CREATE TABLE IF NOT EXISTS goals (
    id SERIAL PRIMARY KEY,
    goal_name VARCHAR(255) NOT NULL
);

5. Kubernetes Application Deployment & Environment Integration

The production deployment manifest binds the application container to database configurations via Kubernetes Secrets, sets up resource limits, and configures liveness and readiness probes.

Production Deployment Manifest (deployment.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-flask-app
  labels:
    app: my-flask-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-flask-app
  template:
    metadata:
      labels:
        app: my-flask-app
    spec:
      containers:
      - name: flask-container
        image: kubesimplify/demo-app:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        env:
        - name: DB_HOST
          value: "my-postgres-cluster-rw.default.svc.cluster.local"
        - name: DB_NAME
          value: "goals_db"
        - name: DB_PORT
          value: "5432"
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: my-postgres-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: my-postgres-credentials
              key: password
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "200m"
            memory: "256Mi"
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10

ClusterIP Service Manifest (service.yaml)

apiVersion: v1
kind: Service
metadata:
  name: my-app-service
spec:
  type: ClusterIP
  selector:
    app: my-flask-app
  ports:
  - port: 80
    targetPort: 8080

6. Exposing Applications: NGINX Ingress Controller & DNS Mapping

To expose internal ClusterIP services to the public internet securely and efficiently, we use an Ingress Controller.

# Deploy NGINX Ingress Controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.2/deploy/static/provider/cloud/deploy.yaml

Executing this command provisions a Cloud Load Balancer with an External Public IP Address (e.g., 51.8.29.232).

DNS Infrastructure Configuration

In your Domain Name Registrar (e.g., Cloudflare, Route53, Namecheap), add an A Record:

Fallback for Local Testing: If you do not own a custom domain name, use dynamic DNS services like nip.io or xip.io (e.g., demo.51.8.29.232.nip.io).

7. Automated SSL/TLS Management: Cert-Manager & Let's Encrypt

Cert-Manager automates the issuance and renewal of TLS certificates using the ACME protocol (Let's Encrypt).

Step 1: Install Cert-Manager

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.1/cert-manager.yaml

Step 2: Define ClusterIssuer (cluster-issuer.yaml)

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@kubesimplify.com
    privateKeySecretRef:
      name: letsencrypt-prod-private-key
    solvers:
    - http01:
        ingress:
          class: nginx

Step 3: Define TLS-Enabled Ingress Manifest (ingress.yaml)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app-ingress
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - demo.kubesimplify.com
    secretName: app-tls-cert
  rules:
  - host: demo.kubesimplify.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-app-service
            port:
              number: 80

8. Dynamic Auto-Scaling: Horizontal Pod Autoscaler (HPA)

To prevent cluster failure during unexpected traffic surges, the Horizontal Pod Autoscaler (HPA) scales the number of running Pod replicas based on observed metrics like CPU and memory utilization.

Prerequisites: Metrics Server

HPA requires the Metrics Server to collect resource metrics from kubelets.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

HPA Configuration Manifest (hpa.yaml)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-flask-app
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 20

Formula for Scaling Calculation:

9. Real-World Verification: Load Testing with Grafana k6

To verify automated scaling under heavy traffic load, we use Grafana k6, an open-source load-testing tool.

JavaScript Load Test Script (loadtest.js)

import http from 'k6/http';
import { sleep } from 'k6';

export const options = {
  vus: 100, // 100 Concurrent Virtual Users
  duration: '60s',
};

export default function () {
  http.get('https://demo.kubesimplify.com/');
  sleep(0.1);
}

Running the Load Test Execution

# Execute k6 script
k6 run loadtest.js

# Observe real-time pod creation in a separate terminal
kubectl get pods -l app=my-flask-app -w

Scaling Event Log

NAME                            READY   STATUS    RESTARTS   AGE
my-flask-app-759d5786c5-x9z4p   1/1     Running   0          5m
my-flask-app-759d5786c5-8f2ba   0/1     ContainerCreating 0  2s
my-flask-app-759d5786c5-l9q1m   0/1     ContainerCreating 0  2s
my-flask-app-759d5786c5-8f2ba   1/1     Running   0          12s
my-flask-app-759d5786c5-l9q1m   1/1     Running   0          14s

10. Comparative Analysis of Production Components

11. Troubleshooting & Debugging Common Production Pitfalls

When setting up complex Kubernetes environments, issues can arise at several points in the stack:

1. Pending StatefulSet or Database Volumes

• Symptom: CloudNativePG pods remain in Pending state.
• Root Cause: Lack of dynamic storage provisioner (StorageClass) in the cloud cluster.
• Diagnosis Command:
  BASH

  Copy

  kubectl describe pvc
  kubectl get storageclass
  

2. Cert-Manager Certificate Generation Stalled (FALSE State)

• Symptom: kubectl get certificate shows READY: False.
• Root Cause: Let's Encrypt HTTP-01 challenge failed because DNS records were not properly propagated or Port 80 was blocked by firewall rules.
• Diagnosis Command:
  BASH

  Copy

  kubectl describe certificate app-tls-cert
  kubectl get challenges
  kubectl describe clusterissuer letsencrypt-prod
  

3. HPA Unable to Target Utilization (<unknown> / 20%)

• Symptom: kubectl get hpa displays target metric utilization as <unknown>.
• Root Cause: Missing CPU/Memory resources.requests definitions inside the target Deployment manifest, or the Metrics Server is not running.
• Fix: Ensure resources.requests.cpu is declared in the container spec.

Summary and Key Takeaways

• Platform-Agnostic Design: Developing cloud-native applications on Kubernetes ensures seamless portability across AWS (EKS), Azure (AKS), Google Cloud (GKE), or bare-metal clusters.
• Database Operators Over StatefulSets: Relying on operators like CloudNativePG abstracts database operational complexities, automating high availability, backup strategies, and failover management.
• Production Probes Are Essential: Always configure livenessProbe and readinessProbe to allow Kubernetes to automatically restart unhealthy containers and avoid routing traffic to non-ready pods.
• Centralized Secrets Integration: Secure database connection strings using Kubernetes Secrets and inject them into container environments dynamically rather than hardcoding credentials.
• Automated SSL/TLS Lifecycles: Pairing NGINX Ingress Controller with Cert-Manager and Let's Encrypt completely automates HTTPS certificate issuance and renewals.
• Metrics-Driven Autoscaling: Define CPU resource requests inside application deployment specs to allow the Metrics Server and HPA to automatically scale application pods under heavy load.
• Validation Through Load Testing: Always run load tests (using tools like k6) against your infrastructure to verify autoscaling targets, resource limits, and system resiliency under load.