Argo CD Projects Tutorial | GitOps Governance, RBAC & Production-Ready Demo

This chapter explores the architecture and implementation of Argo CD Projects (AppProject). We will examine why the default project is highly discouraged for production environments and how custom projects establish robust GitOps governance, security guardrails, and clear ownership boundaries. Through a hands-on, two-tier application deployment demo, we will configure source repository restrictions, destination cluster/namespace constraints, resource whitelisting, and application-level Role-Based Access Control (RBAC).

1. The Production Scaling Problem: Why the Default Project Fails

When first learning Argo CD, it is common to deploy all applications under the default project. While convenient for local development and quick experiments, this approach introduces severe security and operational risks in a production environment.

The Core Vulnerabilities of the Default Project

• Unrestricted Source Repositories: By default, any engineer with access to create an Argo CD application can point it to any Git repository (including personal forks or unauthorized public repositories). This introduces a massive supply-chain security risk.
• Uncontrolled Deployment Destinations: In multi-cluster environments managed by a single Argo CD instance, a simple typo in an application's destination field can result in workloads being deployed to the wrong cluster or namespace, potentially overwriting production services.
• No Resource-Level Safety: Kubernetes supports hundreds of resource types (both native and Custom Resource Definitions). Without restrictions, an application manifest could contain cluster-scoped resources (e.g., ClusterRole, PersistentVolume) or highly privileged resources that compromise the cluster.
• Missing Application-Level RBAC: While Kubernetes RBAC controls who can create or modify Argo CD Custom Resources (like Application or AppProject), it cannot govern Argo CD-specific operations. For example, Kubernetes RBAC cannot easily restrict a user from clicking the Sync, Prune, or Rollback buttons inside the Argo CD UI or CLI.
• Single-Tenant Trust Model: Under the default project, all applications share identical permissions. There is no isolation, no blast-radius control, and no multi-tenant segregation.

2. The Core Mental Model: Argo CD Projects (AppProject)

An Argo CD Project is defined by the AppProject Custom Resource Definition (CRD). It acts as a logical security boundary and governance wrapper around a group of applications.

Key Concept: Argo CD projects enforce hard GitOps boundaries before the synchronization process even begins. If an application manifest violates any rule defined in its assigned project, Argo CD will block the sync and report a policy violation.

Unlike Kubernetes namespaces—which isolate running workloads at the container engine and API level—Argo CD Projects isolate the GitOps control plane. They govern what can be deployed, from where it can be sourced, to where it can be sent, and who can perform lifecycle operations on those applications.

3. Repository Layout Architectures: Naive vs. Enterprise

When structuring code and configuration repositories for multi-tier applications (e.g., an application with a frontend and a backend tier), organizations typically choose one of three repository designs.

The Role of the Fifth Repository: argocd-platform-config

In the enterprise-grade 5-Repository Design, the fifth repository is dedicated entirely to the platform team. It contains:

• AppProject manifests.
• Global GitOps guardrails.
• Argo CD RBAC policies.
• Application onboarding configurations.

This ensures application developers cannot modify the security boundaries (the AppProject definitions) governing their own applications.

4. Step-by-Step Infrastructure Setup: Namespaces & Repositories

To demonstrate these concepts, we will deploy a two-tier application (composed of a frontend tier and a backend tier) using the recommended multi-repository structure.

Step 1: Create the Target Kubernetes Namespace

We will manually create the target namespace where our application tiers will run.

kubectl create ns app1-ns

Note: In advanced GitOps setups, namespaces are often managed declaratively using sync waves or hooks. For this foundational governance demo, we create the namespace manually to isolate the project's resource-whitelisting behavior.

Step 2: Initialize the Config Repositories

Create three public repositories on GitHub (or your preferred Git provider):

1. app1-frontend-config: Holds the Kubernetes manifests for the frontend tier.
2. app1-backend-config: Holds the Kubernetes manifests for the backend tier.
3. argocd-platform-config: Holds the AppProject governance manifests.

5. Deep Dive: Anatomy of a Production-Grade AppProject Manifest

Below is the declarative manifest for our custom project, app1-project. This file is stored in our platform configuration repository (argocd-platform-config).

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: app1-project
  namespace: argocd
spec:
  description: "Project for app1 frontend and backend workloads"
  
  # 1. Source Repository Guardrails
  sourceRepos:
  - "https://github.com/cloudwithvarjosh/app1-frontend-config.git"
  - "https://github.com/cloudwithvarjosh/app1-backend-config.git"
  
  # 2. Destination Guardrails
  destinations:
  - server: "https://kubernetes.default.svc"
    namespace: "app1-ns"
    
  # 3. Cluster-Scoped Resource Whitelist
  clusterResourceWhitelist:
  - group: ""
    kind: Namespace

  # 4. Namespace-Scoped Resource Whitelist
  namespaceResourceWhitelist:
  - group: "apps"
    kind: Deployment
  - group: ""
    kind: Service
  - group: ""
    kind: ConfigMap
  - group: ""
    kind: Secret

  # 5. Application-Level RBAC Roles
  roles:
  - name: dev-read-only
    description: "Read-only access to app1 applications"
    groups:
    - "app1-devs"
    policies:
    - "p, proj:app1-project:dev-read-only, applications, get, app1-project/*, allow"
    
  - name: platform-admin
    description: "Admin access to app1 applications"
    groups:
    - "platform-team"
    policies:
    - "p, proj:app1-project:platform-admin, applications, *, app1-project/*, allow"

Dissecting the AppProject Fields

sourceRepos

An explicit whitelist of Git repositories. Any Argo CD application referencing app1-project is strictly prohibited from pulling manifests from any repository not listed here.

destinations

Specifies the target clusters and namespaces. The wildcard pattern (e.g., app1-*) can be used, but here we restrict deployments strictly to the local cluster (https://kubernetes.default.svc) and the app1-ns namespace.

clusterResourceWhitelist vs. namespaceResourceWhitelist

These fields control which Kubernetes resource kinds Argo CD is allowed to apply.

• Cluster-Scoped: We only allow Namespace creation. If a developer attempts to deploy a ClusterRole or PersistentVolume, Argo CD will block it.
• Namespace-Scoped: We whitelist only four essential resources: Deployment, Service, ConfigMap, and Secret.

Understanding API Groups: In Kubernetes, resources are organized by API groups.

• The core group (e.g., Service, Secret, ConfigMap, Namespace) is represented by an empty string "".
• The workloads group (e.g., Deployment) is represented by "apps".
• RBAC resources belong to "rbac.authorization.k8s.io".

roles and policies

Defines in-project RBAC. This integrates with external Single Sign-On (SSO) providers (like Okta, Keycloak, or Azure AD) by mapping identity provider groups (e.g., app1-devs) to Argo CD project roles.

• The policy syntax follows the Casbin format:

• platform-admin is granted wildcard * permissions (create, update, sync, delete, override) on all applications within app1-project.
• dev-read-only is restricted to get (view) operations.

Apply this project manifest to your cluster:

kubectl apply -f app1-proj.yaml

6. Defining the Application Manifests (Frontend & Backend)

Now we define the Argo CD Application manifests. These manifests reference our newly created app1-project and point to their respective configuration repositories.

Backend Application Manifest (app1-backend-app.yaml)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app1-backend-app
  namespace: argocd
spec:
  project: app1-project
  source:
    repoURL: "https://github.com/cloudwithvarjosh/app1-backend-config.git"
    targetRevision: HEAD
    path: backend
  destination:
    server: "https://kubernetes.default.svc"
    namespace: app1-ns
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Frontend Application Manifest (app1-frontend-app.yaml)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app1-frontend-app
  namespace: argocd
spec:
  project: app1-project
  source:
    repoURL: "https://github.com/cloudwithvarjosh/app1-frontend-config.git"
    targetRevision: HEAD
    path: frontend
  destination:
    server: "https://kubernetes.default.svc"
    namespace: app1-ns
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Apply both application manifests to register them with Argo CD:

kubectl apply -f app1-backend-app.yaml
kubectl apply -f app1-frontend-app.yaml

7. The Two-Tier Application Code & Containerization

Our application consists of a simple Python Flask backend and a Python Flask frontend that communicates with the backend.

Backend Application Code (app.py)

The backend runs on port 5000 and returns a simple JSON payload.

from flask import Flask, jsonify

app = Flask(__name__)

@app.route('/api/v1/data')
def get_data():
    return jsonify({
        "status": "success",
        "message": "Hello from backend API"
    })

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

Backend Dockerfile

FROM python:3.9-slim
WORKDIR /app
RUN pip install flask
COPY app.py /app/app.py
EXPOSE 5000
CMD ["python", "app.py"]

Frontend Application Code (app.py)

The frontend runs on port 4000. It reads an environment variable BACKEND_URL, queries the backend, and displays the response.

import os
import requests
from flask import Flask, render_template_string

app = Flask(__name__)

BACKEND_URL = os.getenv("BACKEND_URL", "http://localhost:5000/api/v1/data")

@app.route('/')
def index():
    try:
        response = requests.get(BACKEND_URL)
        data = response.json()
        backend_msg = data.get("message", "No message found")
    except Exception as e:
        backend_msg = f"Failed to connect to backend: {str(e)}"
        
    html = f"""
    <html>
        <head><title>Two-Tier App</title></head>
        <body>
            <h1>Welcome to the Frontend Tier</h1>
            <p>Backend Status: <strong>{backend_msg}</strong></p>
        </body>
    </html>
    """
    return render_template_string(html)

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=4000)

Frontend Dockerfile

FROM python:3.9-slim
WORKDIR /app
RUN pip install flask requests
COPY app.py /app/app.py
EXPOSE 4000
CMD ["python", "app.py"]

Build and Push Container Images

Execute these commands in your local terminal to build and push the images to Docker Hub (replace cloudwithvarjosh with your Docker Hub username):

# Build and Push Backend
docker build -t cloudwithvarjosh/app1-backend:v1.0.0 ./application1-backend
docker push cloudwithvarjosh/app1-backend:v1.0.0

# Build and Push Frontend
docker build -t cloudwithvarjosh/app1-frontend:v1.0.0 ./application1-frontend
docker push cloudwithvarjosh/app1-frontend:v1.0.0

8. Deploying and Verifying the GitOps Pipeline

With our container images pushed, we populate our configuration repositories with the Kubernetes deployment and service manifests.

Backend Manifests (app1-backend-config repo under /backend)

deploy.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend-deployment
  namespace: app1-ns
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
      - name: backend
        image: cloudwithvarjosh/app1-backend:v1.0.0
        ports:
        - containerPort: 5000

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: backend-svc
  namespace: app1-ns
spec:
  selector:
    app: backend
  ports:
  - protocol: TCP
    port: 5000
    targetPort: 5000

Frontend Manifests (app1-frontend-config repo under /frontend)

deploy.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend-deployment
  namespace: app1-ns
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend
        image: cloudwithvarjosh/app1-frontend:v1.0.0
        ports:
        - containerPort: 4000
        env:
        - name: BACKEND_URL
          value: "http://backend-svc:5000/api/v1/data"

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: frontend-svc
  namespace: app1-ns
spec:
  selector:
    app: frontend
  ports:
  - protocol: TCP
    port: 80
    targetPort: 4000

Once these files are committed and pushed to their respective repositories, Argo CD's automated synchronization engine will detect the changes (within its standard 3-minute polling interval, or immediately upon manual refresh) and apply them.

# Verify running pods and services
kubectl get all -n app1-ns

9. Accessing the Application (Port Forwarding Walkthrough)

To verify that the frontend can successfully communicate with the backend over the Kubernetes service network, we will port-forward our local machine to the frontend service.

Run the port-forwarding command:

kubectl port-forward -n app1-ns svc/frontend-svc 8081:80

Open your browser and navigate to http://localhost:8081. You should see:

Welcome to the Frontend Tier

Backend Status: Hello from backend API

10. Chaos Engineering: Testing Resource Whitelisting Guardrails

To prove that our AppProject guardrails are active and enforced, we will perform a chaos test. We will attempt to deploy a cluster-scoped PersistentVolume (PV) and a namespace-scoped PersistentVolumeClaim (PVC). Neither of these resource kinds is whitelisted in our app1-project manifest.

Step 1: Commit Unauthorized Manifests to Git

We commit a new file, pv-pvc.yaml, to the /backend directory of our app1-backend-config repository:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: unauthorized-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/tmp/data"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: unauthorized-pvc
  namespace: app1-ns
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Step 2: Observe Argo CD Policy Enforcement

When Argo CD attempts to synchronize the repository, the controller compares the incoming manifests against the AppProject rules. The sync fails immediately, displaying the following policy violation errors in the UI and CLI:

This confirms that even though the Git repository contains these manifests, Argo CD blocks them from ever reaching the Kubernetes API server, protecting the cluster from unauthorized resource creation.

Step 3: Cleanup

Delete the pv-pvc.yaml file from the Git repository and trigger a refresh in Argo CD to return the application to a healthy, synchronized state.

11. Implementing and Testing Local User RBAC

While production environments should always integrate Argo CD with an Identity Provider (IdP) via OIDC or SAML, we can simulate and test our RBAC policies locally by creating a local user.

Step 1: Create a Local User in Argo CD

We edit the global Argo CD ConfigMap (argocd-cm) to register a new local user named cwvj with login capabilities.

kubectl edit cm argocd-cm -n argocd

Add the following configuration under the data block:

data:
  accounts.cwvj: login

Step 2: Set a Password for the Local User

Log in to your Argo CD instance via the CLI as the administrator, then update the password for the new user:

# Log in as admin
argocd login localhost:8080 --username admin --password <admin-password>

# Update password for cwvj
argocd account update-password \
  --account cwvj \
  --current-password <admin-password> \
  --new-password "CloudwithVarjo@123"

Step 3: Verify RBAC Restrictions

Log out of the Argo CD UI and log back in using the credentials for the cwvj user.

Upon logging in, the dashboard will appear completely empty. The user cwvj cannot see app1-backend-app or app1-frontend-app.

This occurs because our app1-project RBAC policies only grant read access (get) to users belonging to the app1-devs group and admin access to the platform-team group. Since our local user cwvj is not associated with either group, Argo CD's default-deny security model blocks all visibility.

12. Summary and Key Takeaways

• Default Project Risk: The default Argo CD project is fully permissive and lacks the guardrails necessary for production environments, creating significant security and operational risks.
• GitOps Control Plane Isolation: Custom AppProject resources establish logical security boundaries, validating manifests before they are applied to the Kubernetes API.
• Enterprise Repository Layout: The 5-repository design isolates application code, environment configurations, and platform-level security policies into separate repositories to minimize the blast radius of changes.
• Source and Destination Constraints: Projects restrict applications to verified Git source repositories and specific target clusters and namespaces, preventing accidental cross-environment deployments.
• Resource Whitelisting: By explicitly defining allowed cluster-scoped and namespace-scoped resource kinds, platform teams can prevent developers from deploying unauthorized or privileged resources.
• Granular Application RBAC: Argo CD projects support Casbin-formatted policy rules, allowing organizations to map SSO groups to specific application-level actions (such as sync, refresh, or rollback).
• Default-Deny Security Model: Any resource type, source repository, or user action not explicitly whitelisted in the AppProject configuration is denied by default.