Argo CD ApplicationSets: Generators, Templates, and Production Design Patterns

This chapter focuses on scaling GitOps workflows across multiple Kubernetes clusters and environments. We will explore the operational challenges of multicluster management, understand why Argo CD ApplicationSets exist, and dive deep into five production-grade design patterns using different generators: List, Cluster, Git Directory, Git File, and Matrix. By the end of this guide, you will have a clear architectural framework for designing highly scalable, production-ready GitOps pipelines.

1. Understanding Multicluster Kubernetes Topologies

To understand why we need ApplicationSets, we must first define what "multicluster" means in a Kubernetes ecosystem and analyze the operational pressure it places on GitOps pipelines.

Multicluster setups generally fall into two categories:

A. Independent Clusters

In this topology, clusters are logically and operationally isolated. There are no shared workloads or applications between them.

B. Operationally Connected Clusters

In this topology, a single logical project spans multiple clusters. The same applications are deployed across different clusters for high availability, environment separation, or compliance.

Why Teams Adopt Operationally Connected Multicluster Topologies

1. Failure Domain Isolation (High Availability): Deploying applications across multiple geographic regions (e.g., Mumbai ap-south-1 and N. Virginia us-east-1). If an entire cloud region suffers an outage, traffic is seamlessly routed to the surviving region.
2. Environment Separation: Isolating development, staging, and production workloads onto physically distinct clusters to prevent "noisy neighbor" issues and secure production data.
3. Geographic Proximity & Latency: Serving users from the cluster closest to their physical location to minimize network latency.
4. Compliance & Data Residency: Adhering to local laws (e.g., GDPR or local data localization mandates) requiring citizen data to remain within specific national boundaries.

2. The Scaling Problem: Why We Need ApplicationSets

In a standard Argo CD setup, every application deployment requires an instance of the Application Custom Resource Definition (CRD).

The Naive Approach (Manual Application Management)

If you have 50 microservices deployed across 3 environments (Dev, Stage, Prod) on 2 regional clusters, you must write, track, and maintain:

Managing 300 individual Argo CD Application manifests leads to several critical pain points:

• Configuration Drift at Scale: Keeping common configurations (such as sync policies, destination namespaces, and repository URLs) consistent across hundreds of files becomes nearly impossible.
• Manual Cluster Bootstrapping: When a new cluster is provisioned, operators must manually write and apply new Application manifests for every service destined for that cluster.
• High Operational Overhead: The sheer volume of boilerplate YAML increases the risk of human error, leading to misconfigured environments and deployment failures.

3. Core Mental Model: ApplicationSets vs. Deployments

To understand how Argo CD ApplicationSets solve this scaling problem, we can use a familiar Kubernetes analogy:

Real-Life Analogy: Pods vs. Deployments

Note: ApplicationSets do not deploy your application workloads (such as Pods, Services, or Ingresses) directly. The ApplicationSet controller only generates and manages Argo CD Application resources. Those generated Application resources are then responsible for pulling manifests from Git and deploying workloads to target clusters.

4. Architectural Anatomy: Generators and Templates

An ApplicationSet resource is split into two primary functional blocks: Generators and Templates.

1. Generators

Generators define how many applications to create and for whom (which target clusters, environments, or directories). They scan your infrastructure or Git repositories to produce a list of parameters (key-value pairs).

2. Templates

Templates define what the generated Argo CD Applications should look like. The template uses standard Go templating syntax (e.g., {{parameter}}) to dynamically inject the key-value pairs produced by the generator.

Go Templating Best Practices

When writing production ApplicationSets, always enable strict Go templating options to catch configuration errors early:

spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]

• goTemplate: true: Enables the more powerful Go templating engine instead of the legacy fast-template engine.
• missingkey=error: Instructs the controller to fail immediately with an error if a template references a variable that was not produced by the generator, preventing the silent generation of broken manifests.

5. Step-by-Step Infrastructure Setup & Cluster Registration

To demonstrate these patterns in a realistic environment, we will configure a two-cluster topology across two AWS regions using Amazon EKS.

Step 1: Provision the EKS Clusters

We use eksctl to provision two clusters. The Mumbai cluster acts as our Argo CD control plane, while the N. Virginia cluster acts as a managed target.

Mumbai Cluster Config (01-eks-mumbai.yaml)

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: eks-mumbai
  region: ap-south-1
vpc:
  cidr: 10.10.0.0/16
managedNodeGroups:
  - name: workers-mumbai
    instanceType: t3.small
    desiredCapacity: 2

N. Virginia Cluster Config (02-eks-virginia.yaml)

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: eks-virginia
  region: us-east-1
vpc:
  cidr: 10.20.0.0/16
managedNodeGroups:
  - name: workers-virginia
    instanceType: t3.small
    desiredCapacity: 1

Apply these configurations to provision the infrastructure:

eksctl create cluster -f 01-eks-mumbai.yaml
eksctl create cluster -f 02-eks-virginia.yaml

Step 2: Install Argo CD on the Control Plane (Mumbai)

Switch your Kubernetes context to the Mumbai cluster and install Argo CD using the official Helm chart. We will use version 9.4.0 (an $n-1$ stable release for production environments):

# Switch context to Mumbai
kubectl config use-context <mumbai-cluster-context-name>

# Add and update Helm repository
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

# Create namespace and install
kubectl create namespace argocd
helm install my-argo-cd argo/argo-cd --version 9.4.0 -n argocd

Step 3: Register the N. Virginia Cluster with Argo CD

By default, Argo CD only has visibility into the cluster it is running on (referred to as the in-cluster destination). To deploy workloads to the N. Virginia cluster, we must register it.

# Log into Argo CD via CLI
argo cd login localhost:8080 --username admin --password <password>

# Get your kubeconfig contexts
kubectl config get-contexts

# Register the N. Virginia cluster
argo cd cluster add <eks-virginia-context-name>

Under the Hood: When you run argo cd cluster add, Argo CD reads your local kubeconfig credentials, connects to the target cluster, and provisions an argocd-manager ServiceAccount with administrative ClusterRole privileges. It then packages these access credentials into a Kubernetes Secret inside the argocd namespace on the Mumbai control plane cluster.

6. Deep Dive: The List Generator (Demo 1)

The List Generator is the most straightforward generator. It passes an explicit, hardcoded list of key-value elements directly to the template.

Production Use Case

Ideal for small, static environments where target clusters and regions rarely change, and you want to keep all configurations in a single, easy-to-read file.

Complete ApplicationSet Manifest (list-generator.yaml)

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: app-one-multi-region
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - list:
        elements:
          - region: mumbai
            clusterName: in-cluster
            namespace: app-1-mumbai-ns
          - region: virginia
            clusterName: eks-virginia
            namespace: app-1-virginia-ns
  template:
    metadata:
      name: 'app-one-{{.region}}'
    spec:
      project: default
      source:
        repoURL: 'https://github.com/cloudwithvarun/app-1-config.git'
        targetRevision: main
        path: '{{.region}}/manifests'
      destination:
        name: '{{.clusterName}}'
        namespace: '{{.namespace}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Applying and Verifying the Resources

kubectl apply -f list-generator.yaml -n argocd

# Verify generated applications
kubectl get applications -n argocd

Output:

NAME               SYNC STATUS   HEALTH STATUS
app-one-mumbai     Synced        Healthy
app-one-virginia   Synced        Healthy

7. Deep Dive: The Cluster Generator (Demo 2)

The Cluster Generator dynamically discovers target clusters registered with Argo CD. It queries the Kubernetes secrets in the argocd namespace that carry the label argocd.argoproj.io/secret-type: cluster.

Production Use Case

Perfect for dynamic environments where clusters are frequently added or removed (e.g., ephemeral testing clusters or auto-scaling regional clusters).

Step 1: Label the Cluster Secrets

To target specific clusters, we apply metadata labels to their corresponding Argo CD secrets.

# Label the N. Virginia cluster secret
argo cd cluster add <eks-virginia-context-name> \
  --label app=app-one \
  --label region=virginia \
  --upsert

# Label the Mumbai (in-cluster) secret
# Note: Since the in-cluster secret is created automatically, we label it via kubectl
kubectl label secret -n argocd -l argocd.argoproj.io/secret-type=cluster app=app-one region=mumbai --overwrite

Step 2: Write the Cluster Generator Manifest (cluster-generator.yaml)

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: app-one-cluster-dynamic
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - clusters:
        selector:
          matchLabels:
            app: app-one
  template:
    metadata:
      name: 'app-one-{{.metadata.labels.region}}'
    spec:
      project: default
      source:
        repoURL: 'https://github.com/cloudwithvarun/app-1-config.git'
        targetRevision: main
        path: '{{.metadata.labels.region}}/manifests'
      destination:
        name: '{{.name}}'
        namespace: 'app-1-{{.metadata.labels.region}}-ns'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

8. Deep Dive: The Git Directory Generator (Demo 3)

The Git Directory Generator scans a specified Git repository path and dynamically creates applications based on the subdirectories it discovers.

Production Use Case

Ideal for monorepos where each directory represents a distinct environment, microservice, or tenant. Adding a new environment is as simple as creating a new directory in Git.

Parameter Resolution

The Git Directory Generator exposes two key parameters:

• {{.path.path}}: The full relative path to the discovered directory (e.g., mumbai/dev).
• {{.path.basename}}: The name of the leaf directory (e.g., dev).

Complete ApplicationSet Manifest (git-directory-generator.yaml)

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: app-one-git-directories
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: 'https://github.com/cloudwithvarun/app-1-config.git'
        revision: main
        directories:
          - path: 'mumbai/*'
  template:
    metadata:
      name: 'app-one-{{.path.basename}}'
    spec:
      project: default
      source:
        repoURL: 'https://github.com/cloudwithvarun/app-1-config.git'
        targetRevision: main
        path: '{{.path.path}}/manifests'
      destination:
        name: in-cluster
        namespace: 'app-1-{{.path.basename}}-ns'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

9. Deep Dive: The Git File Generator (Demo 4)

The Git File Generator reads structured configuration files (JSON or YAML) directly from a Git repository. It parses the file contents and generates applications based on the defined data structure.

Production Use Case

This pattern decouples the platform architecture from application configuration. The platform team maintains the ApplicationSet manifest, while application teams simply modify a simple clusters.yaml file to provision or update environments.

Step 1: Create the Configuration File (clusters.yaml)

Commit this file to the root of your Git repository:

clusters:
  - region: mumbai
    clusterName: in-cluster
    namespace: app-1-mumbai-ns
  - region: virginia
    clusterName: eks-virginia
    namespace: app-1-virginia-ns

Step 2: Write the Git File Generator Manifest (git-file-generator.yaml)

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: app-one-git-files
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: 'https://github.com/cloudwithvarun/app-1-config.git'
        revision: main
        files:
          - path: 'clusters.yaml'
  template:
    metadata:
      name: 'app-one-{{.region}}'
    spec:
      project: default
      source:
        repoURL: 'https://github.com/cloudwithvarun/app-1-config.git'
        targetRevision: main
        path: '{{.region}}/manifests'
      destination:
        name: '{{.clusterName}}'
        namespace: '{{.namespace}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

10. Deep Dive: The Matrix Generator (Demo 5)

The Matrix Generator is a powerful combinator. It does not generate parameters on its own; instead, it combines two or more generators to produce a Cartesian product of applications.

Production Use Case

Perfect for enterprise-scale deployments where you need to deploy a set of environments (discovered via Git directories) across multiple regional clusters (discovered via cluster labels or defined in a list).

Complete ApplicationSet Manifest (matrix-generator.yaml)

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: app-one-matrix
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
          - git:
              repoURL: 'https://github.com/cloudwithvarun/app-1-config.git'
              revision: main
              directories:
                - path: '*'
          - list:
              elements:
                - region: mumbai
                  clusterName: in-cluster
                - region: virginia
                  clusterName: eks-virginia
  template:
    metadata:
      name: 'app-one-{{.path.basename}}-{{.region}}'
    spec:
      project: default
      source:
        repoURL: 'https://github.com/cloudwithvarun/app-1-config.git'
        targetRevision: main
        path: '{{.path.path}}/manifests'
      destination:
        name: '{{.clusterName}}'
        namespace: 'app-1-{{.path.basename}}-{{.region}}-ns'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Verifying the Matrix Output

Applying this manifest generates six distinct applications:

kubectl apply -f matrix-generator.yaml -n argocd
kubectl get applications -n argocd

Output:

NAME                     SYNC STATUS   HEALTH STATUS
app-one-dev-mumbai       Synced        Healthy
app-one-dev-virginia     Synced        Healthy
app-one-stage-mumbai     Synced        Healthy
app-one-stage-virginia   Synced        Healthy
app-one-prod-mumbai      Synced        Healthy
app-one-prod-virginia    Synced        Healthy

11. Production Design Patterns & Choosing the Right Generator

Selecting the correct generator depends on your team's operational model, repository structure, and infrastructure scale.

Generator Comparison Matrix

12. Production Debugging Workflows

When working with ApplicationSets, issues can occur at the controller level or the generated application level. Use this structured workflow to diagnose and resolve deployment failures:

13. Summary and Key Takeaways

• Multicluster GitOps requires robust automation to prevent configuration drift, manual bootstrapping bottlenecks, and human error at scale.
• ApplicationSets act as a controller-of-controllers, templating and managing the lifecycle of individual Argo CD Application resources.
• Generators define how many applications to create and for whom, while Templates define what those applications look like.
• The List Generator is best for small, static environments with minimal cluster changes.
• The Cluster Generator dynamically targets clusters using Kubernetes metadata labels on cluster secrets.
• The Git Directory Generator automates monorepo environments by mapping directory structures directly to application instances.
• The Git File Generator decouples platform configuration from application configuration by reading structured YAML/JSON files from Git.
• The Matrix Generator combines multiple generators to easily manage complex, multi-region, multi-environment deployments.
• Strict Go Templating (missingkey=error) is a production best practice that prevents silent failures during application generation.