Argo CD App of Apps Pattern: Multi-Component GitOps Orchestration & Sync Options

This chapter explores the App of Apps pattern in Argo CD, a design pattern used to manage multi-component software products as a single declarative unit. You will learn why this pattern is critical for microservices and multi-tier architectures, how parent and child applications interact, and how to configure advanced sync options to optimize resource reconciliation. The chapter concludes with an end-to-end hands-on deployment of a two-tier application (front-end and back-end) featuring automated self-healing.

1. The Architectural Problem: Managing Multi-Component Products in GitOps

In production environments, modern software products are rarely deployed as monoliths. Instead, they are designed as multi-tier systems or microservices architectures. This architectural reality introduces significant operational challenges when using GitOps.

Clarifying Terminology

To avoid confusion, we must establish a clear distinction between two terms:

• Product (or Software): The business application that end-users or external tools access (e.g., an e-commerce platform, a payment gateway).
• Argo CD Application: The custom resource definition (CRD) within Kubernetes that connects a Git source repository to a target cluster namespace.

The Challenges of Scaling Without App of Apps

1. One Product Maps to Multiple Argo CD Applications
   A single product might consist of a front-end, a back-end, a database, and helper tiers (such as caching or authentication services). In a microservices architecture, a product could easily scale to dozens of individual services. Adhering to GitOps best practices means creating a separate Argo CD application for each component.
2. No Single Control Plane for Product Deployment
   Without a unifying layer, deploying a product requires manually applying the Argo CD application manifests for every individual component. If a product has 5 microservices deployed across 3 environments (Dev, Staging, Prod), you must manage $5 \times 3 = 15$ separate Argo CD applications. This approach is highly manual, error-prone, and difficult to scale.
3. Scattered Product Health Metrics
   To determine if a product is fully functional, an administrator must check the health status of every individual Argo CD application. There is no single "pane of glass" or root resource that aggregates the health status of all components.
4. Cumbersome Multi-Cluster Bootstrapping
   Replicating the entire product stack on a new Kubernetes cluster requires redeploying every individual component application manifest manually, increasing bootstrap complexity.

2. The App of Apps Mental Model: Parent and Child Applications

The App of Apps pattern addresses these challenges by using Argo CD's native capabilities. It is a pattern, not a separate feature or a new Custom Resource Definition (CRD). It leverages the standard Argo CD Application CRD to manage other Application CRDs.

The Parent-Child Hierarchy

• Parent Application (Root): A single Argo CD application that points to a Git repository containing the manifests of other Argo CD applications.
• Child Applications (Branches): Argo CD applications defined as manifests within the parent's source directory. Each child application points to the actual Kubernetes manifests (Deployments, Services, Ingresses) of a specific product component.

Role Separation and Blast Radius Reduction

By keeping child application manifests in their respective component repositories and referencing them in a parent application, organizations can enforce clean role separation:

• Junior Administrators / Developers: Manage the configuration repositories and manifests for individual tiers (e.g., front-end or back-end).
• Senior Administrators / Platform Engineers: Manage the root GitOps repository containing the parent application, controlling which components are allowed into the cluster.

3. Deep Dive: Sync Order and Execution Flow

When deploying a product via the App of Apps pattern, Argo CD executes the reconciliation in a structured, multi-step sequence.

Step 1: Parent Application Sync

The administrator applies the parent application manifest to the cluster. Argo CD pulls the parent's Git repository and reads the directory containing the child application manifests.

Step 2: Child Application Creation

Argo CD applies the child application manifests. Because these are custom resources of kind Application, they are created inside the target namespace where Argo CD is running (typically argocd).

Step 3: Child Application Sync

Once the child applications exist in the cluster, they begin their own reconciliation loops. Each child application pulls its designated source repository and deploys the actual Kubernetes workloads (Deployments, Services, ConfigMaps) to the target application namespace.

4. Sync Options: Optimizing Kubernetes Resource Reconciliation

To control how Argo CD applies and updates resources, you can define syncOptions under the syncPolicy of an application manifest. The table below outlines the key sync options used to manage complex deployments:

5. Directory Structure and Component Breakdown

To implement this pattern, we organize our product configuration into a modular directory structure. This structure separates application code, component configurations, and the root GitOps configuration.

6. Declarative Manifest Analysis: Parent vs. Child Applications

The Parent Application (product-1-root.yaml)

The parent application uses Argo CD's multi-source capability to track multiple repositories simultaneously. This allows a single root application to orchestrate components stored in different repositories.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: product-1-root
  namespace: argocd
spec:
  project: default
  sources:
    - repoURL: 'https://github.com/cloudwithvaros/product-1-backend-config.git'
      targetRevision: HEAD
      path: .
    - repoURL: 'https://github.com/cloudwithvaros/product-1-frontend-config.git'
      targetRevision: HEAD
      path: .
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Note: The destination namespace for the parent application is argocd. This is because the child resources it creates are Argo CD Application CRDs, which must reside in the namespace where the Argo CD controller is running.

The Backend Child Application (product-1-backend-app-crd.yaml)

This manifest defines the child application for the backend service. It is applied by the parent and points to the repository containing the actual backend Kubernetes manifests.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: product-1-backend
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'https://github.com/cloudwithvaros/product-1-backend-config.git'
    targetRevision: HEAD
    path: .
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: product-1
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true
      - ApplyOutOfSyncOnly=true

The Frontend Child Application (product-1-frontend-app-crd.yaml)

Similarly, the frontend child application points to the frontend configuration repository and deploys resources to the same target namespace (product-1).

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: product-1-frontend
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'https://github.com/cloudwithvaros/product-1-frontend-config.git'
    targetRevision: HEAD
    path: .
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: product-1
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true
      - ApplyOutOfSyncOnly=true

7. Hands-On Step-by-Step Evolution: Building and Deploying the Product

Step 1: Build and Push Container Images

First, build and push the custom Docker images for both the front-end and back-end components. Replace cloudwithvaros with your Docker Hub username.

# Build and push Backend Image
cd product-1/product-1-backend
docker build --tag cloudwithvaros/product-1b:1.0.0 .
docker push cloudwithvaros/product-1b:1.0.0

# Build and push Frontend Image
cd ../product-1-frontend
docker build --tag cloudwithvaros/product-1f:1.0.0 .
docker push cloudwithvaros/product-1f:1.0.0

Step 2: Create GitHub Repositories

Create three public repositories on GitHub to store your configurations:

1. product-1-backend-config
2. product-1-frontend-config
3. product-1-gitops

Upload the respective configuration files to each repository:

• Backend Config Repo: Upload deploy.yaml, service.yaml, and product-1-backend-app-crd.yaml.
• Frontend Config Repo: Upload deploy.yaml, service.yaml, and product-1-frontend-app-crd.yaml.
• GitOps Repo: Upload product-1-root.yaml.

Step 3: Apply the Parent Application Manually

Apply the parent application manifest to your Kubernetes cluster. This is the only manual step required to bootstrap the entire product.

cd ../product-1-gitops
kubectl apply -f product-1-root.yaml

Once applied, Argo CD will automatically detect the parent application, pull the child manifests from the referenced repositories, and create the child applications.

Step 4: Verify the Deployment

Verify that the parent and child applications have been successfully created and synced:

# List all Argo CD applications
kubectl get apps -n argocd

Expected output:

NAME                 SYNC STATUS   HEALTH STATUS
product-1-root       Synced        Healthy
product-1-backend    Synced        Healthy
product-1-frontend   Synced        Healthy

Verify that the Kubernetes resources have been deployed to the product-1 namespace (which was automatically created by the CreateNamespace=true sync option):

kubectl get all -n product-1

Step 5: Access the Application

Port-forward the frontend service to access the running application locally:

kubectl port-forward -n product-1 svc/frontend 8081:80

Open your browser and navigate to http://localhost:8081 to verify that the two-tier application is fully functional.

8. Chaos Engineering & Self-Healing Verification

To demonstrate the resilience of the App of Apps pattern combined with Argo CD's self-healing capabilities, we can simulate a manual deletion of a child application.

The Chaos Test

An operator accidentally or maliciously deletes the frontend child application directly from the Kubernetes cluster:

kubectl delete app -n argocd product-1-frontend

The Self-Healing Workflow

Because the parent application (product-1-root) is configured with selfHeal: true, Argo CD continuously compares the live cluster state with the Git repository.

When it detects that the product-1-frontend application CRD has been deleted, it immediately recreates the child application. The recreated child application then reconciles its own state and restores any missing frontend workloads in the product-1 namespace.

Summary and Key Takeaways

• Pattern Over Feature: The App of Apps pattern is a design methodology that uses standard Argo CD Application CRDs to manage other applications, requiring no custom plugins or extensions.
• Unified Control Plane: It provides a single root application (product-1-root) that acts as a single pane of glass for monitoring the health and sync status of an entire multi-component product.
• Multi-Source Support: Argo CD's multi-source capability allows a single parent application to orchestrate components across multiple independent Git repositories.
• Declarative Namespace Management: Using the CreateNamespace=true sync option removes the need to pre-create target namespaces manually or via external scripts.
• Optimized Reconciliation: The ApplyOutOfSyncOnly=true option reduces API server load by ensuring only modified manifests are sent during sync operations.
• Safe Deletions: The PruneLast=true option ensures that resource deletions occur only after new resources are healthy, minimizing potential downtime during updates.
• Automated Self-Healing: Enabling selfHeal: true on the parent application guarantees that if a child application is deleted or modified manually, Argo CD will automatically recreate and restore it to match the Git source of truth.