Azure Zero to Hero: Day-20 | Azure Key Vault Integration with AKS

Welcome to Day 20 of the Azure Zero to Hero series. This comprehensive guide covers integrating Azure Key Vault with an Azure Kubernetes Service (AKS) cluster using the Kubernetes Secret Store CSI Driver and Azure Key Vault Provider. This setup enables pods within an AKS cluster to securely consume and reference secrets and keys directly from Azure Key Vault without storing them as native, plaintext-equivalent Kubernetes Secrets.

1. Why Do We Need Centralized Secret Management?

The Status Quo and Its Limitations

In modern cloud-native architectures, sensitive information—such as database connection strings, passwords, API tokens, and TLS certificates—is frequently scattered across multiple independent systems:

• Kubernetes Secrets stored natively inside etcd.
• CI/CD Environment Variables in GitHub Actions or GitLab CI.
• Configuration files and deployment manifests.

Organizational Compliance and Rotation Challenges

Enterprises enforce strict security and compliance policies requiring sensitive credentials to be rotated regularly (e.g., every 90, 180, or 270 days).

• When secrets are decentralized across dozens of tools and clusters, tracking rotation schedules becomes an administrative nightmare.
• Failing to update a single downstream consumer results in application outages.

The Centralized Solution: Azure Key Vault

Azure Key Vault provides a single, audited, highly secure source of truth for all secrets, keys, and certificates.

• Centralized Auditing & RBAC: Access is strictly governed using Azure Role-Based Access Control (RBAC) and Managed Identities.
• Elimination of Duplication: Multiple Azure resources (Virtual Machines, App Services, and AKS Pods) can reference the exact same secret in Key Vault, simplifying rotation to a single update in the vault.

2. Core Mental Model: The External Secret Store Analogy

Real-Life Analogy: Think of an Azure Key Vault as a high-security downtown bank vault containing important property deeds (passwords and certificates). Your AKS cluster is a suburban residential neighborhood. Instead of printing copies of property deeds and handing them out to every resident (storing raw strings in Kubernetes Secrets), residents carry a specialized secure badge (Managed Identity). When a resident needs to verify a document, they present their badge to a specialized courier service (the Secret store CSI Driver) which travels to the bank vault, retrieves the document on-demand, and brings it directly into the resident's home workspace as a temporary mounted file.

3. Naive vs. Better Approach: Kubernetes Secrets vs. CSI Driver

4. Step-by-Step Architecture & Component Breakdown

To bridge an external Azure Key Vault with an internal AKS Pod, four primary components must work in harmony:

Azure Key Vault (External Secret Store)
   ^
   | accessed via Managed Identity & oidc
   |
AKS Pod (Inside Cluster)
   ^
   | mounts volume via
   |
Secret Store CSI Driver + Azure Key Vault Provider

1. Secret Store CSI Driver (secrets-store.csi.k8s.io): A Kubernetes-native Container Storage Interface (CSI) plugin that allows pods to mount external secret stores as volumes.
2. Azure Key Vault Provider: A plugin extension for the CSI driver that translates requests specifically for Azure Key Vault REST APIs.
3. Azure Managed Identity & OIDC Issuer: Establishes a secure, passwordless trust relationship between the Kubernetes service account and Azure Active Directory.
4. SecretProviderClass: A custom Kubernetes Custom Resource Definition (CRD) that tells the CSI driver which specific keys/secrets to fetch from Key Vault.

5. Setting Up the AKS Cluster with CSI Driver Add-ons

Before provisioning the Key Vault, create the AKS cluster with the required add-ons enabled: the Azure Key Vault Secrets Provider and Open ID Connect (OIDC) Issuer (essential for workload identity federation).

Step 5.1: Create the Resource Group

az group create --name keyvault-demo --location eastus

Step 5.2: Create the AKS Cluster

az aks create \
  --name keyvault-demo-cluster \
  --resource-group keyvault-demo \
  --node-count 1 \
  --enable-addons azure-keyvault-secrets-provider \
  --enable-oidc-issuer \
  --generate-ssh-keys

Step 5.3: Fetch Kubernetes Credentials

Configure your local kubectl context to point to the newly created cluster:

az aks get-credentials --resource-group keyvault-demo --name keyvault-demo-cluster

Verify the connection:

kubectl config current-context
# Expected output: keyvault-demo-cluster

Verify that the Secret Store CSI Driver and Azure Provider daemon sets are running successfully on your cluster nodes:

kubectl get pods -n kube-system -l app=secrets-store-csi-driver
kubectl get pods -n kube-system -l app=secrets-store-provider-azure

6. Creating and Configuring Azure Key Vault

Step 6.1: Provision the Key Vault Instance

Create a Key Vault with RBAC authorization enabled:

az keyvault create \
  --name aks-demo-vault \
  --resource-group keyvault-demo \
  --location eastus \
  --enable-rbac-authorization true

Step 6.2: Assign Key Vault Administrator Role

Ensure your user identity has permission to manage objects inside the vault:

# Fetch your signed-in user principal name or object ID
USER_ID=$(az ad signed-in-user show --query id -o tsv)

# Assign Key Vault Administrator role
az role assignment create \
  --role "Key Vault Administrator" \
  --assignee $USER_ID \
  --scope "/subscriptions/<YOUR_SUBSCRIPTION_ID>/resourceGroups/keyvault-demo/providers/Microsoft.KeyVault/vaults/aks-demo-vault"

Step 6.3: Add a Secret and Key to Key Vault

Using either the Azure Portal or Azure CLI, add a test secret:

az keyvault secret set --vault-name aks-demo-vault --name "secret-one" --value "abhishek-virala"
az keyvault key create --vault-name aks-demo-vault --name "key-one" --ops get

7. Configuring Managed Identity and Workload Federation

To allow AKS pods to authenticate against Azure Key Vault without embedding credentials, configure a User-Assigned Managed Identity and federate it with the Kubernetes Service Account.

Step 7.1: Export Environment Variables

export SUBSCRIPTION_ID="<your-subscription-id>"
export RESOURCE_GROUP="keyvault-demo"
export KEYVAULT_NAME="aks-demo-vault"
export CLUSTER_NAME="keyvault-demo-cluster"
export IDENTITY_NAME="aks-keyvault-secret-provider-id"

Step 7.2: Create User-Assigned Managed Identity

az identity create --name $IDENTITY_NAME --resource-group $RESOURCE_GROUP

Extract the Client ID and Principal ID of the managed identity:

export IDENTITY_CLIENT_ID=$(az identity show --name $IDENTITY_NAME --resource-group $RESOURCE_GROUP --query clientId -o tsv)
export IDENTITY_PRINCIPAL_ID=$(az identity show --name $IDENTITY_NAME --resource-group $RESOURCE_GROUP --query principalId -o tsv)

Step 7.3: Grant Managed Identity Access to Key Vault

Assign the Key Vault Secrets User role to the managed identity so it can read secrets:

az role assignment create \
  --role "Key Vault Secrets User" \
  --assignee $IDENTITY_PRINCIPAL_ID \
  --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.KeyVault/vaults/$KEYVAULT_NAME"

Step 7.4: Retrieve OIDC Issuer URL and Create Service Account

Fetch the OIDC issuer URL from your AKS cluster:

export AKS_OIDC_ISSUER="$(az aks show --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --query oidcIssuerProfile.issuerUrl -o tsv)"

Create a Kubernetes Service Account (workload-identity-sa) in your desired namespace:

kubectl create namespace demo-ns
kubectl create serviceaccount workload-identity-sa --namespace demo-ns

Establish federated identity credentials linking the Azure Managed Identity to the Kubernetes Service Account:

az identity federated-credential create \
  --name "aks-federated-credential" \
  --identity-name $IDENTITY_NAME \
  --resource-group $RESOURCE_GROUP \
  --issuer "$AKS_OIDC_ISSUER" \
  --subject "system:serviceaccount:demo-ns:workload-identity-sa"

8. Defining the SecretProviderClass

The SecretProviderClass CRD instructs the CSI driver on how to connect to Azure Key Vault and which specific secrets or keys to download.

Create a file named secret-provider-class.yaml:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-keyvault-provider
  namespace: demo-ns
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "false"
    userAssignedIdentityID: "${IDENTITY_CLIENT_ID}" # Replace with actual client ID string if not templated
    keyvaultName: "aks-demo-vault"
    cloudName: "AzurePublicCloud"
    objects: |
      array:
        - objectName: "secret-one"
          objectType: "secret"
          objectVersion: ""
        - objectName: "key-one"
          objectType: "key"
          objectVersion: ""
    tenantId: "<your-tenant-id>"

Apply the manifest to the cluster:

kubectl apply -f secret-provider-class.yaml

9. Deploying the Consumer Pod and Verifying Mounts

Deploy a sample BusyBox pod configured with the service account and volume mount pointing to our SecretProviderClass.

Create a file named pod-consume-secret.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: busybox-secret-pod
  namespace: demo-ns
spec:
  serviceAccountName: workload-identity-sa
  containers:
    - name: busybox
      image: registry.k8s.io/e2e-test-images/busybox:1.36-1
      command:
        - sleep
        - "3600"
      volumeMounts:
        - name: secret-volume
          mountPath: "/mnt/secret-store"
          readOnly: true
  volumes:
    - name: secret-volume
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-keyvault-provider"

Apply the pod manifest:

kubectl apply -f pod-consume-secret.yaml

Step 9.1: Verify Pod Execution and Mounted Files

Check that the pod started successfully and entered the Running state:

kubectl get pods -n demo-ns

Inspect the files mounted inside the container volume:

kubectl exec -it busybox-secret-pod -n demo-ns -- ls -la /mnt/secret-store

Expected Output: You will see files named secret-one and key-one mapped directly into the directory as files.

Read the contents of the mounted secret:

kubectl exec -it busybox-secret-pod -n demo-ns -- cat /mnt/secret-store/secret-one

Expected Output: abhishek-virala

10. Real-World Practical Pitfalls & Debugging Mindset

1. Vault Not Found (VaultNotFound) or RBAC Errors:

   • Symptom: Pod gets stuck in ContainerCreating or logs show permission denied when talking to Key Vault.
   • Fix: Ensure the Managed Identity has been assigned Key Vault Secrets User and that the userAssignedIdentityID matches the exact Client ID string in your SecretProviderClass.

2. OIDC Federation Issuer Mismatch:

   • Symptom: Authentication fails during token exchange.
   • Fix: Verify that --enable-oidc-issuer was passed during AKS cluster creation and that the --subject string in your federated credential matches your exact namespace and service account name (system:serviceaccount:<namespace>:<serviceaccount>).

3. Typos in Object Names:

   • Symptom: Secret mounting fails because the objectName specified in SecretProviderClass does not match the exact name of the secret in Azure Key Vault. Always cross-check vault object names.

11. Summary and Key Takeaways

• Centralized Security: Azure Key Vault eliminates the sprawl of sensitive credentials across CI/CD tools, configuration files, and application repositories.
• Strict Compliance: Enables seamless centralized rotation of passwords, API keys, and certificates without modifying application or pod configurations.
• Container Storage Interface (CSI): The Secret Store CSI Driver bridges external cloud secret storage directly into standard Kubernetes pod volumes.
• Passwordless Authentication: Workload Identity and OIDC Federation bind Kubernetes Service Accounts to Azure Managed Identities securely, removing stored credentials.
• File-Based Consumption: Secrets are mounted securely into ephemeral pod directories as files, preventing exposure via environment variable inspection (kubectl describe pod).
• Hands-On Mastery: Practical implementation requires aligning Resource Providers, Key Vault RBAC, Managed Identities, OIDC Federation, and SecretProviderClass CRDs.