Argo CD: Basics to Production — What Is GitOps? | The Definitive Guide

GitOps is not merely a deployment tool or a CI/CD pattern; it is an operating model that redefines how applications and infrastructure are managed by establishing Git as the absolute single source of truth. By separating application code from environment configurations and leveraging software agents for continuous reconciliation, GitOps eliminates environment drift, enhances security, and simplifies disaster recovery.

1. The Genesis of GitOps: Why Do We Need It?

Historically, software development and operations teams treated application code and infrastructure configurations with entirely different levels of discipline.

Application Code Discipline

Application code has long adhered to strict version control practices:

• Structured Branching: Developers isolate work in feature branches and merge changes into protected environment branches (e.g., dev, stage, prod) or a single main branch (using trunk-based development).
• Rigorous Gatekeeping: Direct pushes to production branches are blocked. Changes require Pull Requests (PRs), peer reviews, and automated testing before integration.

The Infrastructure Gap

In contrast, infrastructure and environment configurations were historically managed in a highly fragmented manner:

• Manual and Ad-Hoc Executions: Even when teams adopted Infrastructure as Code (IaC) tools like Terraform, Ansible, Chef, or Puppet, the actual execution of these templates was often manual.
• Scattered Templates: Configuration files, Kubernetes manifests, Helm charts, and Kustomize templates were frequently stored on individual DevOps engineers' laptops, jump hosts, or scattered across disparate file shares.
• Lack of Version Control: Because these files did not live in a centralized, version-controlled repository governed by the same PR-and-review discipline as application code, there was no reliable history of who changed what, when, or why.

The core mission of GitOps is to bridge this gap by bringing infrastructure and configuration files under the exact same Git-based workflow and discipline used for application code.

2. Core Challenges of Traditional Deployment Models

Without a unified, Git-centric operating model, operations teams face several critical challenges in production environments:

A. No Single Source of Truth

When configurations are applied manually from local workstations or jump hosts, there is no single authoritative location that defines the expected state of the infrastructure. If a cluster needs to be recreated from scratch, finding the exact, up-to-date manifests becomes a forensic exercise.

B. Desired State vs. Current State Mismatch (Drift)

Every running system has two distinct states:

1. Desired State: The configuration defined in your templates (e.g., a Kubernetes deployment manifest specifying replicas: 5).
2. Current State: What is actually running in the live cluster.

In traditional models, if an engineer manually alters the live cluster (e.g., running kubectl edit deployment to scale replicas down to 3 to troubleshoot an issue), the current state diverges from the desired state. This divergence is known as drift. Traditional systems have no automated way to detect this drift, let alone correct it.

C. Weak Audit Trails and Fragile Rollbacks

If a manual change breaks an application, identifying the root cause is incredibly difficult. There is no commit history linking the live state to a specific author or pull request. Furthermore, rolling back a change requires manual intervention, which is highly error-prone—especially when multiple engineers are working with different versions of local configuration files.

D. Environment Drift

Over time, manual hotfixes and ad-hoc adjustments cause environments (Development, Staging, and Production) to drift significantly. An application that deploys successfully in Staging may fail in Production because of undocumented configuration differences.

E. Security Risks (The "Push" Model Vulnerability)

In traditional CI/CD pipelines (e.g., Jenkins, GitHub Actions, or GitLab CI pushing directly to a cluster), the CI server must be provisioned with administrative credentials (such as a kubeconfig file or cloud provider access keys) to execute deployment commands.

If the CI/CD server is compromised, or if a malicious pipeline script is executed, the entire production cluster is exposed.

3. Defining GitOps: The CNCF & Open GitOps Standard

According to Open GitOps (a CNCF sandbox project), GitOps is defined as:

Definition: GitOps is an operating model where the desired state of applications and infrastructure is declared in Git, and software agents continuously reconcile the running system to match that state.

The Open GitOps standard defines four core principles that any true GitOps implementation must satisfy:

4. The Core Mental Model: Push-Based vs. Pull-Based Deployments

To understand GitOps, you must contrast the traditional Push-Based deployment model with the GitOps-native Pull-Based model.

5. Architectural Separation: Application vs. Configuration Repositories

A foundational best practice in production GitOps is the strict separation of concerns between your Application Source Code and your Application Configuration. Mixing these into a single repository leads to infinite pipeline loops, security boundary violations, and poor access control.

A. The Application Repository

• Purpose: Build, test, and package the application code into a versioned, deployable artifact (e.g., a container image). No deployment actions occur here.
• Primary Contents:
  • Application source code (Go, Java, Python, Node.js, etc.).
  • Unit and integration tests.
  • Dockerfile (defining how to package the application).
  • CI pipeline definitions (e.g., Jenkinsfile, .github/workflows/ci.yml).
• Ownership: Primarily owned by the Application/Development team. DevOps engineers maintain secondary ownership to assist with CI pipeline templates.

B. The Application Configuration Repository

• Purpose: Define and control how approved artifacts run across various environments (Dev, Stage, Prod).
• Primary Contents:
  • Kubernetes manifests (Deployments, Services, Ingresses, ConfigMaps).
  • Helm charts or Kustomize bases and overlays.
  • Terraform or CloudFormation templates.
  • Network policies and security contexts.
• Ownership: Solely owned and managed by the DevOps/Platform Engineering team. Developers may raise PRs to modify environment variables or resource limits, but DevOps acts as the primary gatekeeper.

6. Step-by-Step GitOps Workflow: From Code Commit to Continuous Reconciliation

The following workflow illustrates how a code change moves from a developer's workstation to a production Kubernetes cluster using a decoupled, pull-based GitOps pipeline.

The End-to-End GitOps Pipeline Flow

Step-by-Step Execution Breakdown

Step 1: Code Modification & Isolation

A developer modifies the application source code. Adhering to git discipline, they push these changes to a feature branch in the Application Repository and open a Pull Request (PR) targeting the main branch.

Step 2: Continuous Integration (CI) & Verification

The PR triggers the CI server (e.g., Jenkins). The CI pipeline executes:

• Code compilation and unit testing.
• Security scanning (SAST, DAST, Software Bill of Materials/SBOM).
• Container image building via the Dockerfile.

Step 3: Artifact Publishing

Once all tests pass and the PR is merged into the main branch, the CI pipeline builds the final production-ready container image. It pushes this image to a secure registry (e.g., Amazon ECR or Docker Hub).

Production Best Practice: The image is tagged not just with a mutable tag (like latest or v1.0.0), but with its unique, immutable Image Digest (e.g., sha256:e3b0c442...).

Step 4: The Automated Handoff (Config Update)

The CI pipeline automatically clones the Application Configuration Repository, updates the target deployment manifest with the new image digest, commits the change to a feature branch, and opens a PR in the Config Repository.

Step 5: Configuration Validation & Ephemeral Testing

The configuration PR triggers a validation pipeline. This pipeline may spin up an ephemeral environment (e.g., a temporary Kubernetes namespace) to deploy the manifests and run smoke tests, checking liveness, readiness, and startup probes. Once validated, the PR is merged into the main branch of the Config Repository.

Step 6: Continuous Reconciliation (The GitOps Loop)

The GitOps Controller (e.g., Argo CD) running inside the Kubernetes cluster is continuously polling the Config Repository.

• It detects that the desired state in Git has changed (new image digest).
• It pulls the updated manifests.
• It applies the changes to the cluster, updating the running pods to the new version without requiring any external push credentials.

7. The Role of GitOps Controllers: Why Kubernetes is the Perfect Fit

A common question is: Why is GitOps highly mature on Kubernetes, but difficult to implement for traditional virtual machines or cloud resources managed by Terraform/Ansible?

The Reconciliation Loop

Kubernetes was designed from the ground up around the concept of the reconciliation loop. Internally, Kubernetes controllers (like the Deployment Controller or ReplicaSet Controller) constantly compare the actual state of the cluster with the desired state stored in etcd.

GitOps controllers like Argo CD and Flux CD simply extend this native Kubernetes reconciliation loop one step further: they treat the external Git Repository as the source of truth for the desired state, rather than just etcd.

The Challenge with Non-Kubernetes Tools

• Terraform: While Terraform has a state file, it does not run as a continuous, self-healing agent inside your infrastructure. To detect drift, you must manually run terraform plan or terraform apply. There is no native "Terraform Controller" running on bare-metal VMs that automatically reverts manual security group changes in real-time.
• Ansible/Chef/Puppet: These tools are typically executed on an ad-hoc, push-based schedule or run as local agents, but they lack the native, declarative, API-driven reconciliation architecture that makes Kubernetes the ideal runtime for GitOps.

Note on Crossplane: Crossplane allows you to manage cloud infrastructure (like RDS databases or S3 buckets) using Kubernetes Custom Resource Definitions (CRDs). While this brings infrastructure into the Kubernetes control plane, it is only "true GitOps" when those CRDs are managed and reconciled from a Git repository by a GitOps controller like Argo CD.

8. Real-World Production Practices: Image Digests vs. Tags

In production GitOps pipelines, relying on standard image tags (like v1.1.0 or latest) introduces significant stability and security risks.

The Danger of Mutable Tags

Image tags are mutable pointers. A developer can build a new container image, tag it with an existing tag (e.g., overwriting v1.1.0), and push it to the registry.

• The Drift Problem: If your Kubernetes nodes pull this image, some nodes might use a locally cached version of the old v1.1.0 image, while newly scaled nodes pull the new v1.1.0 image. This results in an inconsistent, untraceable runtime state.
• The GitOps Blindspot: Because the manifest in Git still points to image: app:v1.1.0, the GitOps controller sees no change in the desired state and will not trigger a reconciliation, leaving the cluster in a partially upgraded state.

The Security of Immutable Digests

An Image Digest is a unique, cryptographic SHA-256 hash of the container image's content and configuration (e.g., app@sha256:85738f...).

• Guaranteed Immutability: If even a single bit of code changes in the container, the digest changes entirely.
• Explicit Git Tracking: When your CI pipeline writes the image digest directly into the configuration repository, Git records a clear, auditable change. Argo CD detects this change instantly and rolls out the exact, cryptographically verified image across all cluster nodes.

# Naive / Dangerous Configuration (Using Tags)
spec:
  containers:
  - name: web-app
    image: myregistry.azurecr.io/web-app:v1.2.0 # Mutable and prone to silent overwrites

# Production-Grade Configuration (Using Digests)
spec:
  containers:
  - name: web-app
    image: myregistry.azurecr.io/web-app@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 # Cryptographically secure & immutable

9. Summary and Key Takeaways

• GitOps is an Operating Model: It is a methodology and culture—not a tool you can buy. Tools like Argo CD and Flux CD exist to implement and enforce this model.
• Git is the Single Source of Truth: Both application code and infrastructure/configuration manifests must be version-controlled, audited, and managed via Git pull requests.
• Continuous Reconciliation is Key: GitOps relies on an in-cluster software agent that continuously compares the desired state (Git) with the current state (Cluster) and automatically corrects any drift.
• Decoupled Repositories are Mandatory: Production systems must separate the Application Repository (owned by developers, focused on artifact creation) from the Application Configuration Repository (owned by DevOps, focused on environment state).
• Pull-Based Security is Superior: By pulling configurations from inside the cluster, you eliminate the need to expose administrative cluster credentials to external CI/CD servers like Jenkins or GitHub Actions.
• Disaster Recovery is Simplified: Because the entire cluster state is declared in Git, recovering from a total cluster failure is as simple as spinning up a new cluster and pointing your GitOps controller to the configuration repository.
• Always Use Image Digests in Production: Avoid mutable tags in your deployment manifests. Use immutable cryptographic SHA-256 digests to guarantee consistency, security, and accurate drift detection.