Day 18 | GitOps With ArgoCD & GitLab | 30 Days Of GitLab DevOps

Welcome to comprehensive study notes for Day 18 of the 30 Days of GitLab DevOps Course. This chapter explores the principles of GitOps, contrasting traditional deployment pipelines with continuous reconciliation models using Argo CD and GitLab. You will learn how to set up an AWS EKS cluster, deploy and configure Argo CD, manage Kubernetes manifest files, establish secure SSH connections between tools, configure webhooks for real-time synchronization, and enforce self-healing against configuration drifts.

1. Why Do We Need GitOps? (The Problem of Configuration Drift)

In modern containerized deployments, maintaining consistency between what is defined in version control and what is actually running in a production Kubernetes cluster is vital.

• Single Source of Truth: Git repositories (such as GitLab) hold the ultimate authority on what infrastructure and applications should exist. If a YAML manifest defines replicas: 2, the cluster should reflect precisely that.
• Configuration Drift: This occurs when the actual state of the cluster diverges from the desired state defined in Git.
• The Manual Override Problem: Engineers often log into production clusters directly to troubleshoot or patch issues (e.g., running kubectl edit deployment to scale pods from 2 to 10). These manual tweaks bypass version control, leaving zero audit trails in Git and causing out-of-sync states.

2. Core Mental Model: The Watcher and Reconciler

Real-Life Analogy: The Thermostat

Think of Argo CD as a digital household thermostat:

1. Desired State: You set the thermostat to 72°F (analogous to defining replicas: 2 in your GitLab YAML files).
2. Observation: The thermostat continuously checks the room temperature (analogous to Argo CD observing the Kubernetes cluster and the Git repository).
3. Action: If a window is left open and the room temperature drops to 65°F, the heater automatically turns on to force the temperature back to 72°F.

Similarly, Argo CD constantly compares Git and Kubernetes, automatically reverting unauthorized manual changes and ensuring the system converges back to the desired state.

3. Traditional Deployments vs. GitOps

Before GitOps, application delivery relied heavily on push-based CI/CD pipelines. The table below compares both approaches across key operational vectors:

4. The Three Pillars of GitOps Architecture

A standard GitOps deployment pipeline using GitLab and Argo CD relies on three foundational components working in concert:

GitLab Repository (G1)
   |
   | (Watched via SSH / Webhooks)
   v
Argo CD Controller (Running as an in-cluster Pod)
   |
   | (Continuous Reconciliation & Sync)
   v
Kubernetes Cluster (EKS)

1. The Git Repository (GitLab): Acts as the single source of truth, storing all Kubernetes YAML manifest files (Deployments, Services, ConfigMaps).
2. Argo CD: Deployed directly inside the Kubernetes cluster as a normal application workload. It connects to the Git repository, watches for changes, and manages deployments.
3. The Target Cluster: The runtime environment (e.g., AWS EKS) where workloads are executed and reconciled.

5. Step-by-Step Infrastructure Setup (AWS EKS & Connector VM)

To build a practical GitOps workflow, we first provision an AWS EC2 "Connector VM" to orchestrate our Kubernetes cluster creation.

Step A: Launch and Configure the Connector VM

• AMI: Ubuntu 24.04 LTS
• Instance Type: t3.medium
• Storage: 20 GB gp3
• Connect via SSH/MobaXterm and update system packages:
  BASH

  Copy

  sudo apt update && sudo apt upgrade -y
  

Step B: Install Essential CLI Tools

Execute the following commands on the Connector VM to install the AWS CLI, eksctl, and kubectl:

# 1. Install AWS CLI v2
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws --version

# 2. Install eksctl
ARCH=amd64
PLATFORM=$(uname_s | tr '[:upper:]' '[:lower:]')
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$PLATFORM_$ARCH.tar.gz" | tar xz -v -C /tmp
sudo mv /tmp/eksctl /usr/local/bin
eksctl version

# 3. Install kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv kubectl /usr/local/bin/
kubectl version --client

Step C: Configure AWS Credentials

Generate a programmatic Access Key and Secret Access Key in AWS IAM, then configure your environment:

aws configure
# Provide AWS Access Key ID
# Provide AWS Secret Access Key
# Default region name: ap-south-1
# Default output format: json

Step D: Create the EKS Cluster

Run eksctl to provision a managed Kubernetes cluster with node scaling limits:

eksctl create cluster --name day18-cluster --region ap-south-1 --node-type t3.medium --nodes 1 --nodes-min 1 --nodes-max 3

Note: Cluster creation typically takes 15–20 minutes. Once complete, configure kubectl context:

aws eks update-kubeconfig --region ap-south-1 --name day18-cluster
kubectl get nodes

6. Installing and Configuring Argo CD on Kubernetes

With our EKS cluster running, we install Argo CD into its own dedicated namespace.

# 1. Create Argo CD Namespace
kubectl create namespace argocd

# 2. Apply official Argo CD manifest installation
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# 3. Wait for all pods to reach Running status
kubectl get pods -n argocd

Exposing the Argo CD Server via LoadBalancer

By default, Argo CD runs behind a ClusterIP service. To access the UI externally, patch the service type to LoadBalancer:

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'
kubectl get svc argocd-server -n argocd

Retrieving the Admin Password

Argo CD generates an initial autogenerated password stored as a Kubernetes secret. Decode it with:

kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}" | base64 -d
echo ""

Username: admin
Password: (Decoded string from the command above)

7. Creating GitLab Repositories and Manifests

Next, create a public GitLab repository named day-18-gitops and add your Kubernetes deployment and service manifests.

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: default
spec:
  type: LoadBalancer
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

8. Connecting Argo CD to GitLab Using SSH Keys

To allow Argo CD to authenticate securely with your private or public GitLab repository without exposing passwords, use SSH key pairs.

1. Generate SSH Key Pair on the Connector VM:
   BASH

   Copy

   ssh-keygen -t rsa -b 4096 -C "argocd-connector"
   

2. Add Public Key to GitLab:
   • Copy the contents of ~/.ssh/id_rsa.pub.
   • In GitLab, navigate to Preferences -> SSH Keys and add the public key.
3. Register Repository in Argo CD:
   • In the Argo CD UI, go to Settings -> Repositories -> Connect Repo.
   • Select SSH connection method.
   • Provide the SSH repository URL (e.g., git@gitlab.com:<username>/day-18-gitops.git).
   • Paste the private key contents (~/.ssh/id_rsa) into the SSH Private Key field and click Connect.

9. Creating and Managing Argo CD Applications

With the repository successfully linked, deploy your application through the Argo CD dashboard:

1. Click New App in the Argo CD UI.
2. Configure application settings:
   • Application Name: nginx-app
   • Project Name: default
   • Sync Policy: Automatic (Enable Prune Resources and Self Heal).
   • Repository URL: Select your connected GitLab repository URL.
   • Revision: HEAD
   • Path: . (Root directory)
   • Cluster URL: https://kubernetes.default.svc (Default in-cluster URL)
   • Namespace: default (Check Auto-Create Namespace if required).
3. Click Create. Argo CD immediately syncs the repository, provisions the Nginx deployment, and spins up the pods.

10. Understanding Sync Mechanisms and Webhooks

Argo CD supports three primary synchronization and event-triggering strategies:

Configuring a GitLab Webhook for Instant Sync

1. In your Argo CD UI, locate your application server API endpoint (http://<argocd-loadbalancer-ip>/api/webhook).
2. In your GitLab repository, navigate to Settings -> Webhooks -> Add new webhook.
3. URL: Paste the Argo CD webhook URL.
4. Trigger: Select Push events. Uncheck SSL verification if using self-signed certificates.
5. Click Add webhook. Now, every git push triggers an instantaneous deployment update in Kubernetes.

11. Testing Self-Healing and Configuration Drift

To witness Argo CD's auto-healing capabilities in action, simulate a manual configuration drift directly inside the Kubernetes cluster.

1. Check Current Pod Count:

   BASH

   Copy

   kubectl get pods
   

   Observation: Exactly 1 pod is running (as defined in deployment.yaml).

2. Manually Force Drift via kubectl: Manually scale the deployment replicas directly inside the cluster:

   BASH

   Copy

   kubectl scale deployment nginx-deployment --replicas=3
   

   Observation: Running kubectl get pods temporarily shows 3 running pods.

3. Argo CD Auto-Heal Reaction: Within seconds, Argo CD detects that the cluster state (replicas: 3) does not match the desired state in the GitLab repository (replicas: 1). It automatically prunes the extra pods and forces the cluster back to 1 replica.

Summary and Key Takeaways

• GitOps Paradigm: Git serves as the single source of truth for all infrastructure and application states, removing manual interventions.
• Elimination of Configuration Drift: Continuous monitoring by tools like Argo CD ensures that unauthorized cluster alterations are instantly identified and reverted.
• Pull-Based Security: Unlike traditional push-based CI/CD pipelines that require hardcoded AWS/K8s credentials, Argo CD runs securely inside the cluster.
• Argo CD Core Architecture: Combines repository observation, state comparison, and automated reconciliation loops into a unified control plane.
• Automatic Sync & Webhooks: Leveraging GitLab webhooks ensures sub-second deployment propagation upon every code commit.
• Self-Healing Mechanics: Manual changes made via kubectl edit or kubectl scale are automatically overwritten by Argo CD to enforce desired repository states.
• Resource Cleanup: Always clean up cloud resources post-lab to prevent incurring AWS charges:
  BASH

  Copy

  eksctl delete cluster --name day18-cluster --region ap-south-1