Kubernetes Bootcamp: Architecture, Kubeconfig, RBAC, GVK/GVR, and REST API Integration

Kubernetes is a production-grade container orchestration system designed to automate the deployment, scaling, and management of containerized applications across cluster nodes. This chapter covers the internal mechanics of Kubernetes, exploring control plane and worker node architectures, request execution lifecycles, configuration contexts (kubeconfig), fine-grained Role-Based Access Control (RBAC), API schema structures (GVK vs. GVR), and direct interaction with the kube-apiserver REST API.

1. Virtual Machines vs. Containers: Recapping Core Concepts

Traditional infrastructure relied heavily on Virtual Machines (VMs) running on top of hypervisors. While VMs provide hardware-level isolation, they suffer from significant resource overhead, slow startup times (measured in minutes), and rigid environment configurations between development and production.

Containers decouple applications from the underlying host infrastructure. By sharing the host system's kernel and packaging only the application binary alongside its required runtime dependencies, containers spin up in seconds, consume minimal memory, and provide predictable execution across varying environments.

2. Demystifying Container Isolation: Myths, Distroless Images, and OS Limits

A common misconception is that every container contains a full-blown operating system complete with basic utilities like bash, ls, or package managers (apt, yum).

The Scratch & Distroless Image Reality

Containers are simply isolated host processes. When building container images using a Dockerfile, specifying FROM scratch or utilizing Distroless base images strips away all OS utilities, leaving strictly the application binary and minimal runtime dependencies.

# Example: Minimal multi-stage build using a scratch base
FROM golang:1.21 AS builder
WORKDIR /app
COPY main.go .
RUN CGO_ENABLED=0 GOOS=linux go build -o server main.go

# Scratch contains ZERO OS binaries (no /bin/sh, no ls, no curl)
FROM scratch
COPY --from=builder /app/server /server
ENTRYPOINT ["/server"]

Note: Attempting to execute kubectl exec -it <pod_name> -- /bin/bash on a distroless or scratch-based container will fail because no shell binary exists inside the container image.

Linux vs. Windows & VM Containers

Container isolation mechanisms vary based on the underlying host kernel and technology stack:

1. Linux Containers: Rely strictly on Linux Namespaces (for visibility isolation: PID, NET, MNT, IPC, UTS, USER) and Control Groups / cgroups (for resource constraint allocation: CPU, Memory, I/O).
2. Windows Containers: Utilize Windows-specific process isolation or Hyper-V isolation boundaries.
3. VM Containers (Kata Containers / Firecracker): Wrap container workloads inside lightweight microVMs to provide hardware-enforced isolation boundaries for multi-tenant security requirements.

3. Kubernetes Architecture: Control Plane & Worker Node Deep-Dive

A Kubernetes cluster consists of two primary layers: the Control Plane (which makes global decisions about the cluster) and Worker Nodes (which maintain running workloads).

Control Plane Components

• kube-apiserver: The centralized management engine exposing the Kubernetes REST API. All administrative tasks, worker node communications, and internal controller interactions pass through the API server.
• etcd: A consistent, highly available distributed key-value store. It serves as the single source of truth, persisting the complete state specification and runtime status of the entire cluster.
• kube-scheduler: Assigns newly created Pods without assigned nodes to optimal worker nodes based on resource specifications (CPU/Memory requests), node affinity rules, taints, and tolerations.
• kube-controller-manager: Runs core controller loops (e.g., ReplicaSet Controller, Deployment Controller, Node Controller) that continuously monitor cluster state via kube-apiserver and drive the actual state toward the desired target state.
• cloud-controller-manager (CCM): Integrates cluster components with underlying cloud provider APIs (AWS EKS, Azure AKS, GCP GKE, Civo) to provision external resources like cloud load balancers, managed storage volumes, and routing tables.

Worker Node Components

• kubelet: The primary agent running on each worker node. It monitors pod specifications provided by kube-apiserver and ensures that the declared containers are running and healthy.
• Container Runtime Interface (CRI): The low-level software executing containers (e.g., containerd, CRI-O).
• Container Network Interface (CNI): Networking plugins (e.g., Calico, Cilium, Flannel) responsible for assigning pod IP addresses and managing overlay network routing across nodes.
• Container Storage Interface (CSI): Storage drivers (e.g., Longhorn, OpenEBS, cloud volume drivers) enabling persistent volume provisioning and attachment.
• kube-proxy: Maintains network routing rules (iptables or IPVS entries) on each node to route network traffic smoothly across internal pod endpoints and external Services.

4. Workload Lifecycle: Step-by-Step Request Execution Flow

Understanding how Kubernetes processes requests requires tracing the life of an imperative command, such as kubectl run nginx --image=nginx.

Real-Life Analogy: Office Building Security Checkpoint

To understand the three API validation phases, consider entering a high-security corporate facility:

1. Authentication (AuthN): The security guard inspects your corporate ID card to verify who you are.
2. Authorization (AuthZ): The access control system checks whether your badge grants permission to enter the restricted server room (Access Control Lists / RBAC permissions).
3. Admission Control: A building policy restricts entry to specific hours (e.g., maintenance window policies or webhook rules enforcing security contexts on incoming pods).

5. Kubeconfig Deep-Dive: Structure, Precedence, and Multi-Cluster Management

The kubeconfig file orchestrates access parameter definitions for connecting to one or multiple Kubernetes clusters.

File Location Precedence

When executing commands via kubectl, the client resolves configuration settings using the following lookup order:

1. Flag Override: --kubeconfig=/path/to/custom/config
2. Environment Variable: $KUBECONFIG (supports colon-separated paths for merging multiple files: export KUBECONFIG=~/.kube/config:~/.kube/dev-config)
3. Default File Location: ~/.kube/config

Kubeconfig Architecture

A standard kubeconfig structure consists of three decoupled entities bound together by Contexts:

apiVersion: v1
kind: Config
preferences: {}

# 1. Cluster Endpoints & CA Certificates
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FUR...
    server: https://192.168.1.100:6443
  name: production-cluster

# 2. User Credentials & Client Keys
users:
- name: developer-user
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FUR...
    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRV...

# 3. Contexts (Mapping Clusters + Users + Default Namespaces)
contexts:
- context:
    cluster: production-cluster
    namespace: default
    user: developer-user
  name: dev-context

current-context: dev-context

6. End-to-End Hands-on: Creating a Custom User with Restricted RBAC Roles

To grant an engineer restricted permissions within a cluster, you must issue custom X.509 certificates and bind the authenticated subject to specific Kubernetes RBAC roles.

Step 1: Generate Client Private Key and Certificate Signing Request (CSR)

# 1. Generate private key for user 'sayam'
openssl genrsa -out sayam.key 2048

# 2. Generate a Certificate Signing Request (CSR) specifying user subject
openssl req -new -key sayam.key -out sayam.csr -subj "/CN=sayam/O=dev-team"

# 3. Base64 encode the CSR file contents (removing newlines)
BASE64_CSR=$(cat sayam.csr | tr -d '\n' | base64 | tr -d '\n')

Step 2: Submit and Approve the Kubernetes CSR Object

Create a manifest file named csr.yaml:

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: sayam-csr
spec:
  request: <INSERT_BASE64_CSR_STRING_HERE>
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: 86400 # 24 Hours
  usages:
  - client auth

Submit and approve the certificate request using kubectl:

# Apply CSR to cluster
kubectl apply -f csr.yaml

# Approve the certificate request as cluster admin
kubectl certificate approve sayam-csr

# Extract the signed public client certificate
kubectl get csr sayam-csr -o jsonpath='{.status.certificate}' | base64 --decode > sayam.crt

Step 3: Define Restricted RBAC Roles and Bindings

Create a Role restricting operations to reading Pods, and a RoleBinding attaching user sayam to that role.

# role-and-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""] # Core group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods-sayam
  namespace: default
subjects:
- kind: User
  name: sayam
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Apply the RBAC policy:

kubectl apply -f role-and-binding.yaml

Step 4: Configure Kubeconfig Credentials and Test Permissions

# 1. Register user credentials inside kubeconfig
kubectl config set-credentials sayam \
  --client-certificate=sayam.crt \
  --client-key=sayam.key \
  --embed-certs=true

# 2. Create a dedicated context for user 'sayam'
kubectl config set-context sayam-context \
  --cluster=kubernetes \
  --namespace=default \
  --user=sayam

# 3. Switch active execution context
kubectl config use-context sayam-context

Authorization Verification:

# Attempting an authorized action (READ Pods) -> SUCCESS
kubectl get pods
# OUTPUT: No resources found in default namespace. (Status Code 200 OK)

# Attempting an unauthorized action (CREATE Deployment) -> FORBIDDEN
kubectl create deployment test-app --image=nginx
# OUTPUT: Error from server (Forbidden): deployments.apps is forbidden: 
# User "sayam" cannot create resource "deployments" in API group "apps" in the namespace "default"

7. Understanding API Architecture: GVK (Group-Version-Kind) vs. GVR (Group-Version-Resource)

Every resource definition processed by Kubernetes mapping schemas relies on API categorization constructs: GVK and GVR.

Real-Life Analogy: Grocery Store Classification

Consider how inventory is organized inside a large departmental supermarket:

• Group: Department (e.g., Groceries, Bakery, Dairy).
• Version: Packaging Revision (e.g., 2023-Revision, v1).
• Kind: Object Template Class (e.g., Biscuit, Milk Carton).
• Resource: Physical Aisles/Endpoints where items are collected (e.g., biscuits, milks).

Decoupling GVK and GVR

• GVK (Group, Version, Kind):
  • Used in Declarative YAML/JSON Manifests and Go client code.
  • Defines the schema endpoint specification details.
  • Example: apiVersion: apps/v1, kind: Deployment.
• GVR (Group, Version, Resource):
  • Used by the HTTP REST Engine paths inside kube-apiserver.
  • Constructs URI endpoints. Uses pluralized lowercase strings.
  • Example: /apis/apps/v1/namespaces/default/deployments.

# Manifest Level Serialization Uses GVK
apiVersion: apps/v1  # Group: "apps", Version: "v1"
kind: Deployment     # Kind: "Deployment"
metadata:
  name: web-app
spec:
  replicas: 2

When this YAML is applied, kubectl serializes the manifest to JSON and converts the GVK (apps/v1, Deployment) into an HTTP REST GVR target (/apis/apps/v1/namespaces/default/deployments).

8. Direct API Server Interaction: Bypassing kubectl via REST Calls

kubectl is simply a CLI wrapper that makes HTTP REST calls to the kube-apiserver. You can interact directly with the Kubernetes API server using standard tools like curl.

Method A: Establishing an Authenticated Local Proxy

Running kubectl proxy opens a local HTTP server on port 8001 that automatically injects user authentication credentials from your kubeconfig:

# Start proxy in background
kubectl proxy --port=8001 &

# Query cluster core API groups via REST
curl http://localhost:8001/api/v1/namespaces/default/pods

Method B: Directly Authenticating via HTTP Header Tokens

To query the API server directly without kubectl proxy, extract an authorized bearer token and construct explicit HTTP requests.

# 1. Obtain Cluster API Server URL
APISERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')

# 2. Extract Bearer Token (ServiceAccount Secret Token)
TOKEN=$(kubectl create token default)

# 3. Query all Deployments in 'default' namespace via raw REST API
curl -k -X GET "$APISERVER/apis/apps/v1/namespaces/default/deployments" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json"

Creating Workloads Directly Using Raw HTTP POST Requests

Save the JSON payload as deployment.json:

{
  "apiVersion": "apps/v1",
  "kind": "Deployment",
  "metadata": {
    "name": "api-demo-nginx",
    "namespace": "default"
  },
  "spec": {
    "replicas": 1,
    "selector": {
      "matchLabels": {
        "app": "api-demo"
      }
    },
    "template": {
      "metadata": {
        "labels": {
          "app": "api-demo"
        }
      },
      "spec": {
        "containers": [
          {
            "name": "nginx",
            "image": "nginx:latest",
            "ports": [
              {
                "containerPort": 80
              }
            ]
          }
        ]
      }
    }
  }
}

Post the JSON definition payload directly to the REST endpoint:

curl -k -X POST "$APISERVER/apis/apps/v1/namespaces/default/deployments" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d @deployment.json

Verify that the Deployment was created via standard CLI commands:

kubectl get deployments
# OUTPUT: api-demo-nginx   1/1     1            1           10s

9. Architectural Synthesis & Comparison Matrix

Control Plane vs. Worker Node Responsibilities

GVK vs. GVR Mapping Specifications

Summary and Key Takeaways

• Containers Are Not Full OSs: Container images built from minimal bases (e.g., scratch or Distroless) lack full operating systems, package managers, and shell binaries (/bin/sh), running purely as isolated host processes.
• Isolation Varies by Platform: Linux relies on namespaces (for visibility) and cgroups (for resource boundaries), whereas Windows containers and microVM technologies (e.g., Kata Containers) utilize platform-specific isolation boundaries.
• Control Plane Components: The kube-apiserver serves as the central API gateway, etcd acts as the persistent key-value store, kube-scheduler selects worker nodes, and kube-controller-manager drives state reconciliation.
• Worker Node Components: The kubelet acts as the node agent interacting with CRI, CNI, and CSI drivers, while kube-proxy configures local network packet routing (iptables/IPVS).
• API Security Pipeline: Every incoming API request passes through three sequential checks: Authentication (AuthN), Authorization (AuthZ via RBAC), and Admission Control.
• Kubeconfig Structural Hierarchy: Resolves configurations via --kubeconfig flags, $KUBECONFIG environment variables, or ~/.kube/config. Maps cluster endpoints, user credentials, and active execution contexts.
• X.509 PKI User Authentication: Custom users are onboarded by generating OpenSSL key pairs, submitting a CertificateSigningRequest (CSR) to Kubernetes, approving it, and binding the user subject to fine-grained RBAC Roles via RoleBindings.
• GVK vs. GVR Mechanics: Group-Version-Kind (GVK) categorizes declarative YAML specs (apps/v1, Deployment), whereas Group-Version-Resource (GVR) constructs lower-case, pluralized HTTP REST paths (/apis/apps/v1/namespaces/default/deployments).
• Direct REST API Access: kubectl is an HTTP client; clusters can be fully inspected, managed, and mutated using raw REST API endpoints with curl using bearer tokens or kubectl proxy.