Day-5: Kubernetes Security Tutorial - RBAC, Network Policies, Kyverno, Secrets, and External Secrets

1. Why Do We Need Kubernetes Security? (Problem Statement & Status Quo)

When deploying containerized workloads across a shared Kubernetes cluster, treating the cluster as a single flat environment introduces severe operational and security risks. Without proper isolation, guardrails, and access controls, multi-tenant environments quickly become unstable and vulnerable to lateral movement, accidental data loss, and privilege escalation.

To prevent this chaos, enterprise-grade Kubernetes security relies on six foundational pillars:

1. Namespaces: Logical cluster partitioning and resource quotas.
2. RBAC (Role-Based Access Control): Restricting service accounts and user permissions.
3. Network Policies: Firewalling inter-pod traffic at the network/IP layer.
4. Kyverno: Dynamic policy enforcement using validation admission control.
5. Kubernetes Secrets: Native, base64-encoded credential storage.
6. External Secrets Operator (ESO): Secure secret synchronization with enterprise vaults (e.g., HashiCorp Vault) for GitOps workflows.

2. Core Mental Model & Real-Life Analogy

Real-Life Analogy: Imagine an entire apartment building managed by a single landlord. Without interior walls, doors, or locks:

• Anyone can walk into any apartment and accidentally (or maliciously) throw away someone else's furniture (Lack of Namespaces / RBAC).
• One loud tenant can hog 90% of the building's water and electricity (Lack of Resource Quotas).
• Anyone can wander from the lobby straight into the master vault (Lack of Network Policies).
• Tenants can bring in unverified, dangerous equipment with no oversight (Lack of Kyverno Policy Enforcement).

Kubernetes security introduces walls (Namespaces), keycards and permission logs (RBAC), internal security guards and doors (Network Policies), building inspectors (Kyverno), and secure safety deposit boxes connected to an off-site bank vault (External Secrets Operator).

3. Namespaces and Resource Quotas

By default, all workloads in a Kubernetes cluster share the same flat namespace (default). Namespaces provide logical isolation to divide a single cluster among multiple teams or environments.

The Multi-Tenant Namespace Workflow

Key Namespace Commands

# 1. Create a Kubernetes cluster using Kind (Kubernetes in Docker)
kind create cluster --name k8s-demo

# 2. Create namespaces for logical isolation
kubectl create ns payments-ns
kubectl create ns score-ns

# 3. View all namespaces
kubectl get ns

# 4. Deploy workloads into specific namespaces
kubectl create deployment nginx-payments --image=nginx -n payments-ns
kubectl create deployment nginx-scores --image=nginx -n score-ns

# 5. Verify pods running inside specific namespaces
kubectl get pods -n payments-ns

Resource Quotas

Logical isolation alone does not prevent resource hogging. Administrators bind a ResourceQuota object to a namespace to hard-cap CPU and memory consumption regardless of how many pods a team deploys.

4. Role-Based Access Control (RBAC) in Kubernetes

Pods and service accounts should never operate with wildcard (*) cluster permissions. RBAC ensures that workloads and users operate under the principle of least privilege.

The Three Core RBAC Components

1. Service Account (SA): The identity used by pods or processes inside the cluster (analogous to a user account on an operating system). If unassigned, Kubernetes automatically assigns the default service account.
2. Role / ClusterRole: A collection of permissions (verbs) on Kubernetes resources (apiGroups, resources).
   • Role: Namespaced scope.
   • ClusterRole: Cluster-wide scope across all namespaces.
3. RoleBinding / ClusterRoleBinding: Maps a Service Account to a Role or ClusterRole.

RBAC Implementation Walkthrough

# 1. Create a dedicated service account
kubectl create service account payments-user -n payments-ns

# 2. Create a Role manifest (pod-reader.yaml)
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: payments-ns
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
---
# 3. Create a RoleBinding manifest (pod-reader-binding.yaml)
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods-binding
  namespace: payments-ns
subjects:
- kind: ServiceAccount
  name: payments-user
  namespace: payments-ns
roleRef:
  kind: Role
  name: pod-reader
apiGroup: rbac.authorization.k8s.io/v1

# Apply RBAC manifests
kubectl apply -f pod-reader.yaml
kubectl apply -f pod-reader-binding.yaml

# 4. Verify permissions using 'kubectl auth can-i'
kubectl auth can-i list pods --as=system:serviceaccount:payments-ns:payments-user -n payments-ns
# Output: yes

kubectl auth can-i create pods --as=system:serviceaccount:payments-ns:payments-user -n payments-ns
# Output: no

5. Network Policies

By default, Kubernetes cluster networks are flat: any pod can communicate with any other pod across namespaces. Network Policies act as software-defined firewalls for pod traffic.

Ingress & Egress Traffic Control

• Ingress: Restricts incoming traffic to a pod based on pod selectors, namespace selectors, or IP blocks.
• Egress: Restricts outgoing traffic leaving a pod.

Setting Up Network Policies (Calico CNI)

Standard local kind clusters do not support network policies out-of-the-box. A Container Network Interface (CNI) like Calico must be installed (managed clusters like AWS EKS with VPC CNI support network policies natively).

# Deploy sample pods with labels
kubectl run frontend --image=busybox -n payments-ns -- labels="role=frontend"
kubectl run attacker --image=busybox -n payments-ns # No labels
kubectl run backend --image=busybox -n payments-ns -- labels="app=my-app"
kubectl expose pod backend --port=80 --name=backend-svc -n payments-ns

Network Policy Manifest Example

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-only
  namespace: payments-ns
spec:
  podSelector:
    matchLabels:
      app: my-app
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          role: frontend

Behavior: When applied, any curl or ping request from the attacker pod to the backend-svc will time out and fail, while requests from the frontend pod will succeed.

6. Advanced Policy Enforcement with Kyverno

Organizations often enforce compliance rules across their clusters (e.g., banning the :latest image tag to prevent unpredictable production breaks). Kyverno is a Kubernetes-native policy engine that acts as a dynamic admission controller.

Admission Control Workflow

Kyverno Installation & Cluster Policy

# 1. Install Kyverno CRDs and Controller
kubectl apply -f https://github.com/kyverno/kyverno/releases/download/v1.11.4/kyverno.yaml

# 2. Wait for Kyverno pods to initialize
kubectl get pods -n kyverno -w

# 3. Create a ClusterPolicy to ban the ':latest' tag (disallow-latest-tag.yaml)
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
spec:
  validationFailureAction: Enforce
  rules:
  - name: validate-image-tag
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "Using 'latest' image tag is strictly prohibited in this cluster."
      pattern:
        spec:
          containers:
          - image: "!*:latest"

7. Kubernetes Secrets and Their Limitations

Native Kubernetes Secrets store sensitive configuration data (passwords, tokens, keys) as base64-encoded strings.

Managing Native Secrets

# 1. Create a native secret
kubectl create secret generic db-secret \
  --from-literal=username=admin \
  --from-literal=password=strongpassword123 \
  -n payments-ns

# 2. Consume secret inside a Pod via Environment Variables

apiVersion: v1
kind: Pod
metadata:
  name: secret-demo
  namespace: payments-ns
spec:
  containers:
  - name: app
  - image: nginx
    env:
    - name: DB_USER
      valueFrom:
        secretKeyRef:
          name: db-secret
          key: username
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: db-secret
          key: password

The Security Flaw of Native Secrets

Warning: Kubernetes secrets are not encrypted at rest by default—they are only base64 encoded. Anyone with read access to the namespace or RBAC permissions to read secrets can decode them instantly using standard command-line utilities:

BASH

Copy

echo "c3Ryb25ncGFzc3dvcmQxMjM=" | base64 --decode
# Output: strongpassword123

Consequently, storing raw Kubernetes secret manifests in Git repositories is an anti-pattern.

8. External Secrets Operator (ESO) & Vault Integration

To safely manage secrets in GitOps pipelines without exposing plaintext or base64 secrets in version control, teams deploy the External Secrets Operator (ESO) alongside an external vault provider (such as HashiCorp Vault).

External Secrets Architecture

3-Step ESO Implementation Workflow

# Step 1: Install External Secrets Operator using Helm
helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace

# Step 2: Install and configure HashiCorp Vault (or connect to managed Vault)
# (Deploy Vault deployment manifest in vault-ns)

# Step 3: Configure Vault as a SecretStore in Kubernetes

# SecretStore Manifest (vault-secret-store.yaml)
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: payments-ns
spec:
  provider:
    vault:
      server: "http://vault.vault-ns.svc.cluster.local:8200"
      path: "secret"
      version: "v2"
      auth:
        tokenSecretRef:
          name: vault-token
          key: token
---
# ExternalSecret Manifest stored safely in Git (external-secret.yaml)
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-secret-external
  namespace: payments-ns
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-secret # Creates/updates native Kubernetes secret automatically
    creationPolicy: Owner
  data:
  - secretKey: username
    remoteRef:
      key: secret/payments/db
      property: username
  - secretKey: password
    remoteRef:
      key: secret/payments/db
      property: password

9. Summary & Key Takeaways

• Namespaces & Resource Quotas: Logical isolation divides physical clusters between teams while quotas prevent resource monopolization (CPU/RAM caps).
• Service Accounts & RBAC: Every pod should run with a dedicated, non-default service account restricted by scoped Roles and RoleBindings following the principle of least privilege.
• Network Policies: Act as internal firewalls that restrict ingress and egress traffic between pods and namespaces, isolating sensitive database tiers from compromised frontends.
• Kyverno Admission Control: Enforces cluster compliance dynamically (e.g., blocking deployment manifests using :latest tags or missing health probes) before workloads reach etcd.
• Base64 Encoding Is Not Encryption: Kubernetes native secrets are only encoded, making them insecure for direct storage in Git version control systems.
• External Secrets Operator (ESO): Solves GitOps secret management by keeping secret references in Git while letting ESO synchronize live credentials securely from external vaults like HashiCorp Vault.