Argo CD End-to-End Project Demo: Private Git, Secrets, Pruning & Self-Healing

Production GitOps environments mandate strict zero-trust network boundaries: application source code, Kubernetes manifests, and container image registries are entirely private by default. This guide details the complete end-to-end lifecycle of deploying a private containerized Python service via Argo CD into Kubernetes, managing credentials securely, and enforcing declarative synchronization through automated sync, pruning, and self-healing.

1. Production Architecture: The Two-Repository Model

In enterprise environments, maintaining application source code and infrastructure configurations inside the same repository breaks separation of concerns and creates security bottlenecks. A robust GitOps workflow decouples these responsibilities into two distinct, private repositories.

Separation of Concerns: App Repo vs. Config Repo

1. Application Code Repository (app-1):

   • Contains business logic (app.py), language dependency locks (requirements.txt), and container specifications (Dockerfile).
   • Managed primarily by software developers.
   • Triggers continuous integration (CI) workflows to run unit tests, security scans, and build immutable container artifacts.

2. Application Configuration Repository (app-1-config):

   • Contains pure declarative manifests (deployment.yaml, service.yaml) and Argo CD Application custom resources (app-crd.yaml).
   • Managed by platform engineers, DevOps teams, or automated deployment bots.
   • Forms the single source of truth for the desired state in the Kubernetes cluster. Argo CD monitors only this repository.

2. End-to-End GitOps Workflow: Artifact Creation vs. Reconciliation

The GitOps delivery pipeline consists of two distinct lifecycle phases: pre-GitOps artifact creation and post-commit GitOps reconciliation.

Identity and Access: PATs, Deploy Keys, and Machine Users

Authentication mechanisms must strictly follow the Principle of Least Privilege:

3. Local Workspace & Application Artifact Preparation

The demo application utilizes a lightweight Python Flask application structured to run inside an unprivileged container environment listening on port 5000.

Directory Layout

Application Source (app-1-code/app.py)

from flask import Flask

app = Flask(__name__)

@app.route("/")
def home():
    return "GitOps Managed Flask Application Deployed to Kubernetes via Argo CD!\n"

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

Dependency Definition (app-1-code/requirements.txt)

flask==3.0.0

Multi-Stage Production Dockerfile (app-1-code/Dockerfile)

FROM python:3.11-slim

WORKDIR /app

# Cache dependency installations
COPY requirements.txt /app/
RUN pip install --no-cache-dir -r requirements.txt

# Copy application artifacts
COPY app.py /app/

EXPOSE 5000

CMD ["python", "app.py"]

Note: The --no-cache-dir flag prevents pip from caching wheels locally, keeping the container image lean and minimizing attack surface.

4. Building and Publishing to a Private Registry

Private container registries block unauthenticated access by default. The local daemon builds the image, tags it with semver-compliant metadata, and pushes it to Docker Hub.

# Navigate to the code directory
cd app-1-code

# Build and tag the image
docker build -t cloudwithvarun/app-1:v1.0.0 .

# Authenticate local Docker daemon (if not using Docker Desktop)
docker login -u cloudwithvarun

# Push to private Docker Hub repository
docker push cloudwithvarun/app-1:v1.0.0

5. Kubernetes Ingestion: Namespace & Secret Architecture

When Kubernetes schedules a pod whose specification references a private registry image, the container runtime (e.g., containerd) fails with ImagePullBackOff or ErrImagePull unless supplied with valid credentials.

Secret Generation (docker-registry)

Create the target namespace and provision a Kubernetes secret of type kubernetes.io/dockerconfigjson:

# Create dedicated namespace
kubectl create namespace app-1-ns

# Generate the image pull secret
kubectl create secret docker-registry docker-hub-secret \
  --docker-server=https://index.docker.io/v1/ \
  --docker-username=cloudwithvarun \
  --docker-password=<DOCKER_HUB_PERSONAL_ACCESS_TOKEN> \
  --docker-email=varun@example.com \
  --namespace=app-1-ns

Security Realities & Anti-Patterns

1. Imperative Generation Risk: Running kubectl create secret directly writes plain base64-encoded credentials to cluster etcd.
2. Access Risks: Anyone with kubectl get secret -o yaml can decode credentials instantly:
   BASH

   Copy

   echo "<BASE64_PAYLOAD>" | base64 --decode
   

3. The GitOps Secret Dilemma: Never store plain secrets in git. For true declarative production setups, utilize:
   • Cloud-Native IAM: AWS IRSA (IAM Roles for Service Accounts) pulling directly from ECR without static keys.
   • External Secrets Operator (ESO): Synchronizes secrets securely from HashiCorp Vault or AWS Secrets Manager into native Kubernetes secrets at runtime.

6. Manifest Definitions: Workloads and Application CRD

The declarative state is maintained inside the app-1-config repository.

Deployment Manifest (app-1-config/app-1-deploy.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-1
  labels:
    app: app-1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-1
  template:
    metadata:
      labels:
        app: app-1
    spec:
      imagePullSecrets:
        - name: docker-hub-secret
      containers:
        - name: web
          image: cloudwithvarun/app-1:v1.0.0
          ports:
            - containerPort: 5000

Service Manifest (app-1-config/app-1-svc.yaml)

apiVersion: v1
kind: Service
metadata:
  name: app-1-svc
spec:
  type: ClusterIP
  selector:
    app: app-1
  ports:
    - protocol: TCP
      port: 80
      targetPort: 5000

Argo CD Application CRD (app-1-config/app-1-app-crd.yaml)

The Application custom resource bridges the desired state in Git with the target Kubernetes cluster.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-1
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/cloudwithvarun/app-1-config.git
    targetRevision: HEAD
    path: .
  destination:
    server: https://kubernetes.default.svc
    namespace: app-1-ns

Note on Namespace Inheritance: Although the deployment and service manifests do not explicitly hardcode metadata.namespace, Argo CD automatically injects the destination namespace (app-1-ns) defined within the Application spec during the manifest rendering phase.

7. Connecting Argo CD to Private Git via CLI

To reconcile private repositories, Argo CD requires credentials. These can be added declaratively or imperatively via the Argo CD CLI.

CLI Connection and Credential Provisioning

# Log in to the Argo CD API Server
argocd login localhost:8080 --username admin --password <ADMIN_PASSWORD> --insecure

# Register private Git repository credentials
argocd repo add https://github.com/cloudwithvarun/app-1-config.git \
  --username argo-bot \
  --password <GITHUB_PERSONAL_ACCESS_TOKEN>

8. Initial Synchronization & Manual State Verification

With credentials in place and the code pushed to both repositories, apply the Application CRD:

kubectl apply -f app-1-app-crd.yaml

Because automated sync is disabled by default, the application registers in an OutOfSync state. Argo CD compares the cluster's live state with Git and detects that the manifests have not yet been applied.

# Inspect resources within the application namespace
kubectl get pods,svc -n app-1-ns

Triggering a manual sync via the UI or CLI reconciles the state:

argocd app sync app-1

Local Application Verification via Port Forwarding

kubectl port-forward svc/app-1-svc -n app-1-ns 8081:80

Accessing http://localhost:8081 outputs: GitOps Managed Flask Application Deployed to Kubernetes via Argo CD!

9. Advanced Sync Governance: Automated Synchronization

In mature GitOps architectures, platform teams eliminate manual interventions. Any merge into the default branch automatically triggers synchronization.

Updating the Application CRD

Enable automated synchronization by adding the syncPolicy block:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-1
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/cloudwithvarun/app-1-config.git
    targetRevision: HEAD
    path: .
  destination:
    server: https://kubernetes.default.svc
    namespace: app-1-ns
  syncPolicy:
    automated: {}

Verification: Scaling via Git

1. Edit app-1-deploy.yaml to update spec.replicas from 1 to 2.
2. Commit and push the change to app-1-config.
3. Force a quick sync detection (or wait up to 180 seconds for the reconciliation loop):
   BASH

   Copy

   argocd app get app-1 --refresh
   

4. Argo CD immediately identifies the diff and deploys a second pod.

10. Declarative Resource Cleanup: Pruning

By default, if an engineer deletes a Kubernetes manifest from Git, Argo CD preserves the live resource in the cluster to prevent accidental data loss. This default behavior leaves behind orphaned resources during refactoring.

Pruning enforces complete parity: if a manifest is removed from Git, the corresponding live resource is automatically deleted from Kubernetes.

Enabling Prune Governance

spec:
  syncPolicy:
    automated:
      prune: true

Validation Workflow

1. Remove app-1-svc.yaml from the app-1-config repository:
   BASH

   Copy

   git rm app-1-svc.yaml
   git commit -m "refactor: remove clusterIP service for gateway migration"
   git push origin main
   

2. Argo CD detects that app-1-svc exists in the cluster but is absent from Git.
3. Because prune: true is configured, the controller deletes the live Service resource:
   BASH

   Copy

   kubectl get svc -n app-1-ns
   # Output: No resources found in app-1-ns namespace.
   

11. Configuration Drift Correction: Self-Healing

Even with Git configured as the authoritative source, engineers may make out-of-band changes directly in the cluster using kubectl edit, kubectl scale, or manual patches. This creates configuration drift.

Self-Healing forces the cluster back to the Git-defined desired state whenever unapproved modifications occur on the live system.

Enabling Self-Healing Governance

spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Step-by-Step Drift Simulation

1. Update app-1-deploy.yaml in Git to specify replicas: 3. Commit and push changes.
2. Argo CD brings the live cluster up to 3 running pods.
3. An operator manually introduces drift via kubectl:
   BASH

   Copy

   kubectl scale deployment app-1 --replicas=1 -n app-1-ns
   

4. Pods begin terminating.
5. The Argo CD Application Controller detects the drift against its cached desired state and immediately overrides the manual change, restoring the deployment back to 3 replicas within seconds.

12. Complete Production Application Specification

Below is the fully-reconciled, production-ready Application manifest combining private Git access, automated deployment, cleanup pruning, and continuous self-healing:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-1
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://github.com/cloudwithvarun/app-1-config.git
    targetRevision: HEAD
    path: .
  destination:
    server: https://kubernetes.default.svc
    namespace: app-1-ns
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Sync Governance Feature Matrix

Summary and Key Takeaways

• Strict Repository Decoupling: Isolate application source code from Kubernetes manifests using two distinct repositories. Argo CD requires access only to the configuration repository.
• Principle of Least Privilege: Configure human developers with read/write personal access tokens or SSH keys, but restrict Argo CD to read-only fine-grained tokens or repository-specific Deploy Keys.
• Private Image Pull Mechanism: Pulling images from private registries requires a kubernetes.io/dockerconfigjson secret assigned to the pod specification via spec.imagePullSecrets.
• Imperative Secret Limitations: Imperative secrets created via kubectl create secret are vulnerable to cluster privilege escalation. Production architectures should use the External Secrets Operator or native cloud IAM roles (such as AWS IRSA).
• Default Polling vs. Cache Invalidation: The Argo CD Application Controller polls Git repositories every 180 seconds (timeout.reconciliation). Use webhooks or the manual Refresh / Hard Refresh controls in the UI/CLI to trigger immediate checks.
• Automated Synchronization: Adding syncPolicy.automated: {} allows Argo CD to reconcile Git-pushed changes without manual intervention.
• Pruning Prevents Resource Leaks: Enabling prune: true ensures that deleting a file in Git drops the matching resource in Kubernetes, preventing abandoned workloads when services are refactored.
• Self-Healing Enforces Immutability: Enabling selfHeal: true actively prevents configuration drift by overriding out-of-band manual changes (kubectl edit/scale) and returning the cluster to the Git-defined desired state.