Ultimate Azure CI/CD Pipeline using Azure Pipelines & Argo CD

1. Why Do We Need This? (Problem Statement & Status Quo)

In the previous session (Day-14), continuous integration (CI) was successfully established, automating the build of Docker container images and pushing them to the Azure Container Registry (ACR). However, building and pushing images to a registry does not deliver code to end users.

Manual deployments—such as running arbitrary shell scripts, executing kubectl apply from local developer terminals, or relying on ad-hoc Ansible playbooks—introduce severe production vulnerabilities:

• Configuration Drift: Manual cluster changes cause the actual cluster state to diverge from intended repository configurations.
• Lack of Auditability: Without a single source of truth, tracking who deployed what, when, and why becomes nearly impossible.
• Operational Overhead: Manual rollouts require direct cluster access, increasing the risk of human error during deployments.

To achieve robust, enterprise-grade continuous delivery (CD), this architecture adopts the GitOps model, utilizing Argo CD to continuously reconcile the desired state in a Git repository with the live state of an Azure Kubernetes Service (AKS) cluster.

2. Core Mental Model & Architecture Analogy

Real-Life Analogy: The Automated Mailroom and Smart Dispatcher

Imagine a high-volume publishing company:

• The Developers (Writers): Write articles and commit changes to the main repository (The Central Manuscript).
• Azure Pipelines (The Printing Press / CI): Automatically picks up manuscript updates, formats them into finished books (Docker images), and boxes them up in the warehouse (Azure Container Registry).
• The Update Script (The Shipping Clerk): Automatically stamps the inventory logbook with the exact crate number of the newest shipment.
• Argo CD (The Smart Dispatcher / GitOps): Constantly watches the logbook. The moment the logbook changes, the dispatcher immediately carries the new crate out to the storefront shelves (AKS Cluster) without anyone needing to ask. If a customer sneaks behind the counter and rearranges the shelves manually, the dispatcher instantly notices the drift, pulls out the logbook, and restores the correct arrangement within seconds.

End-to-End CI/CD Data Flow

3. The Multi-Microservice Application Architecture

The demonstration project is a multi-microservice opinion-polling application developed by the Docker samples team. It showcases microservice interaction across disparate language runtimes and data stores:

4. Azure Kubernetes Service (AKS) Cluster Setup

To host the production workloads and Argo CD, an AKS cluster is provisioned via the Azure CLI and Azure Portal.

Creating the AKS Cluster via Azure Portal

1. Resource Group: Use an existing resource group (Azure-CICD-Cluster).
2. Preset Configuration: Select Dev/Test (recommended for proofs-of-concept and demos; standard/economy/enterprise are reserved for heavy production workloads).
3. Region: US West 2 (selected to avoid resource quota limitations in East US).
4. Node Pool (VMSS - Virtual Machine Scale Sets):
   • Scale Method: Autoscale enabled.
   • Node Count Limits: Minimum 1, Maximum 2 nodes.
   • Security Warning: While setting max nodes to higher numbers (e.g., 10 or 20) helps withstand Denial of Service (DoS) attacks, it can result in catastrophic billing charges if exploited by malicious traffic.
5. Node Ports: Max pods per node set to the default 30.

Connecting to AKS via Azure CLI

Once the cluster is provisioned, configure local kubectl credentials:

az aks get-credentials --name Azure-DevOps --resource-group Azure-CICD-Cluster --overwrite-existing

Verify cluster connectivity:

kubectl get nodes -o wide

5. Installing and Configuring Argo CD on AKS

Step 1: Deploy Argo CD

Create the dedicated argocd namespace and apply the official installation manifests:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Verify that all Argo CD system pods are running:

kubectl get pods -n argocd

Step 2: Retrieve the Initial Admin Password

Argo CD auto-generates an initial admin password stored as a Kubernetes secret. Decode it:

kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}" | base64 --decode
echo ""

(Username is admin, and the decoded output is the password).

Step 3: Expose Argo CD via NodePort

To access the Argo CD Web UI externally without an Ingress controller for this demo, edit the argocd-server service type from ClusterIP to NodePort:

kubectl edit svc argocd-server -n argocd

Change type: ClusterIP to type: NodePort. Find the assigned NodePort (e.g., 30461) by running:

kubectl get svc -n argocd

Step 4: Configure Azure Network Security Group (NSG)

Because the AKS nodes run behind Virtual Machine Scale Sets (VMSS), inbound traffic to the NodePort must be explicitly allowed:

1. Navigate to the Azure Portal -> VMSS -> Node Pool -> Instances.
2. Select the active instance -> Networking.
3. Add an Inbound Port Rule:
   • Destination Port Range: 30461 (Argo CD) or 31000 (Voting App NodePort).
   • Protocol: Any.
   • Action: Allow.

Access the UI at https://<Node-External-IP>:<NodePort>.

6. Connecting Argo CD to Azure Repositories

To allow Argo CD to monitor your Azure DevOps Git repository, authenticate using a Personal Access Token (PAT) with at least Read permissions on Repositories.

1. In Argo CD UI, navigate to Settings -> Repositories -> Connect Repo.
2. Select HTTPS and Default project.
3. Provide the Repository URL copied from Azure Repos.
4. Replace your organization username in the URL string with your generated PAT token:
   TEXT

   Copy

   https://<PAT_TOKEN>@dev.azure.com/<Org>/<Project>/_git/<Repo>
   

5. Click Connect (Status must reflect Successful).

7. Writing the Update Shell Script (CI-to-Git Bridge)

Because Argo CD monitors Git (not ACR directly), an automated bridge is required. After Azure Pipelines pushes a new container image to ACR, a shell script updates the Kubernetes manifest in the Git repository with the new image tag.

Script: updateK8sManifest.sh

#!/bin/bash
set -x

# Arguments:
# $1 = Microservice name prefix (e.g., vote)
# $2 = Repository Name in ACR (e.g., oting-app)
# $3 = Build Tag / Image Version (e.g., 65)

REPO_URL="https://$PAT_TOKEN@dev.azure.com/myorg/myproject/_git/myrepo"
TEMP_DIR=$(mktemp -d)

# Clone repository into temporary directory
git clone "$REPO_URL" "$TEMP_DIR"
cd "$TEMP_DIR/k8s-specifications"

# Update image tag inside deployment YAML using sed
# Target format: image: <registry>/<repo>:<tag>
sed -i "s|image: .*/$2:.*\vert image: abishekazurecicd.azurecr.io/$2:$3\vert g" "$1-deployment.yaml"

# Commit and push changes back to Azure Repos
git config user.email "pipeline@azure.com"
git config user.name "Azure Pipeline Agent"
git add "$1-deployment.yaml"
git commit -m "CI Pipeline updated image tag to $3 [skip ci]"
git push origin HEAD:main

8. Azure Pipelines CI/CD Configuration File

The complete pipeline integrates Build, Push, and Update stages.

azure-pipelines.yml

trigger:
  - main

stages:
- stage: Build
  displayName: 'Build Docker Image'
  jobs:
  - job: BuildJob
    pool: Default # Self-hosted Linux Agent
    steps:
    - task: Docker@2
      inputs:
        command: 'build'
        Dockerfile: 'vote/Dockerfile'
        tags: '$(Build.BuildId)'

- stage: Push
  displayName: 'Push to Azure Container Registry'
  jobs:
  - job: PushJob
    pool: Default
    steps:
    - task: Docker@2
      inputs:
        containerRegistry: 'my-acr-service-connection'
        command: 'push'
        repository: 'oting-app'
        tags: '$(Build.BuildId)'

- stage: Update
  displayName: 'Update Git Manifests'
  jobs:
  - job: UpdateJob
    pool: Default
    steps:
    - script: |
        bash scripts/updateK8sManifest.sh vote oting-app $(Build.BuildId)
      displayName: 'Execute Manifest Update Script'

9. Handling Private Registry Authentication (Image Pull Secrets)

When AKS attempts to pull private container images from ACR, it will fail with an ImagePullBackOff error unless a Kubernetes Secret is provided to authenticate with the registry.

Creating the Image Pull Secret in AKS

Run the following command against your cluster, making sure the namespace matches where your pods are deployed (e.g., default):

kubectl create secret docker-registry acr-secret \
  --namespace=default \
  --docker-server=abishekazurecicd.azurecr.io \
  --docker-username=<ACR_ADMIN_USERNAME> \
  --docker-password=<ACR_ADMIN_PASSWORD>

Attaching the Secret to Kubernetes Deployments

Inside your microservice deployment manifest (vote-deployment.yaml), add the imagePullSecrets block directly beneath the template spec:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vote
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vote
  template:
    metadata:
      labels:
        app: vote
    spec:
      imagePullSecrets:
        - name: acr-secret
      containers:
        - name: vote
          image: abishekazurecicd.azurecr.io/oting-app:65
          ports:
            - containerPort: 80

10. Argo CD Application Configuration & Reconciliation

With the repository connected and manifests updated, configure Argo CD to manage the application lifecycle:

1. In Argo CD UI, click New App:
   • Application Name: azure-cd-project
   • Project: default
   • Sync Policy: Automatic (Enables self-healing and continuous synchronization).
2. Repository URL: Select your connected Azure Repo URL.
3. Revision: HEAD
4. Path: k8s-specifications
5. Destination:
   • Cluster: https://kubernetes.default.svc (In-cluster)
   • Namespace: default
6. Click Create.

Tuning Argo CD Reconciliation Timeout

By default, Argo CD checks Git for changes every 180 seconds (3 minutes). To accelerate this reconciliation window to 10 seconds for rapid testing, edit the Argo CD ConfigMap:

kubectl edit cm argocd-cm -n argocd

Add the reconciliation timeout parameter under data:

data:
  timeout.reconciliation: 10s

11. End-to-End Flow Walkthrough

12. Summary and Key Takeaways

• GitOps Paradigm: Using Git as the single source of truth guarantees full auditability, eliminates configuration drift, and ensures automated self-healing.
• Continuous Reconciliation: Argo CD continuously monitors Git repositories and automatically applies updates to the target Kubernetes cluster without manual intervention.
• Separation of Concerns: Continuous Integration (Azure Pipelines) builds and publishes artifacts, while Continuous Delivery (Argo CD) handles deployment and synchronization.
• Image Pull Secrets: Private container registries (like Azure Container Registry) require explicit Kubernetes docker-registry secrets linked via imagePullSecrets in deployment manifests to prevent ImagePullBackOff errors.
• Scripted Manifest Updates: In the absence of general-availability tools like Argo Image Updater, lightweight shell scripts utilizing git clone, sed, and git push provide a reliable bridge between CI pipelines and GitOps repositories.
• Automated Rollouts in Action: Modifying source code automatically triggers builds, updates tags, reconciles cluster states, and renders new application features live in minutes.