Argo CD Sync Waves: Architecture, Resource Ordering, and Production Workflows

Argo CD sync waves provide an explicit, metadata-driven mechanism to dictate the exact sequence in which Kubernetes manifests are applied during deployment phases. By overriding default client-side or kind-based resource ordering, sync waves solve critical race conditions and ensure that runtime dependencies—such as network policies, quotas, and underlying services—are fully established before workloads are created.

1. Kubernetes Resource Creation Execution Models

Every interaction with a Kubernetes cluster—whether initiated by human operators, CI/CD pipelines, or deployment agents—must traverse the Kubernetes API Server via REST endpoints. Regardless of the client tool used, objects defined in manifest files are eventually stored independently inside etcd.

Human / Client Tools / Argo CD
        |
        |  REST API Requests (HTTPS)
        v
  kube-apiserver
        |
        |  Persists objects independently
        v
      etcd

To understand why custom resource ordering is necessary, we must compare the two primary execution models used to interact with the API Server:

Note: Argo CD does not execute kubectl commands under the hood. It leverages official Kubernetes client libraries (such as client-go) to communicate directly with the kube-apiserver REST interface.

2. Default Client-Side Ordering (kubectl) and Its Pitfalls

When utilizing the kubectl client, manifest application follows a rigid, non-logical evaluation order based strictly on how files and documents are structured.

Case A: Multi-Document Order within a Single File

If multiple Kubernetes resources are defined within a single YAML file using the --- separator, kubectl submits them to the API server strictly top-to-bottom.

Naive Manifest Sequence (test1.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-one-deployment
  namespace: app-1-ns
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-one
  template:
    metadata:
      labels:
        app: app-one
    spec:
      containers:
      - name: nginx
        image: nginx:latest
---
apiVersion: v1
kind: Namespace
metadata:
  name: app-1-ns

Execution Result

Executing kubectl apply -f test1.yaml fails on the deployment stage:

$ kubectl apply -f test1.yaml
Error from server (NotFound): namespaces "app-1-ns" not found
namespace/app-1-ns created

kubectl attempted to apply the Deployment first. Because the namespace app-1-ns did not exist in etcd, the API server rejected the request. It then proceeded to the next block and created app-1-ns.

Corrected Manifest Sequence

To make kubectl apply this file cleanly, the Namespace object must be manually moved to the top of the file:

apiVersion: v1
kind: Namespace
metadata:
  name: app-1-ns
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-one-deployment
  namespace: app-1-ns
# ... rest of spec

Case B: Directory-Based File Execution Order

When executing kubectl apply -f <directory>/, kubectl applies files in lexicographical (alphabetical) order based on their filenames.

Running kubectl apply -f manifests/ causes test1.yaml to execute before test2.yaml, resulting in an API error because test1.yaml requires resources declared in test2.yaml.

3. Argo CD Control-Plane Driven Ordering & Deterministic Execution

Unlike kubectl, Argo CD analyzes all target manifests in advance before issuing REST calls to the API server. It calculates a deterministic execution plan that prioritizes API correctness—ensuring that dependent objects exist at the API schema level before referencing workloads are submitted.

Argo CD guarantees that explicit references inside workload specs (such as serviceAccountName, configMapRef, or secretRef) are created before the workloads themselves. However, default Argo CD ordering aims purely for API correctness, not runtime readiness or business logic sequence.

4. The Precedence Hierarchy in Argo CD Sync Operations

When a sync operation is initiated, Argo CD evaluates every resource against a strict four-tier precedence hierarchy.

Precedence Hierarchy Summary

Default Resource Kind Ordering Order

When no sync waves are defined, Argo CD sorts resources within the same phase using built-in kind priorities:

1. Namespaces (e.g., Namespace)
2. Custom Resource Definitions (e.g., CustomResourceDefinition)
3. Cluster-Scoped Resources (e.g., ClusterRole, ClusterRoleBinding)
4. Namespaced Resources (e.g., ResourceQuota, ConfigMap, Secret, ServiceAccount, Service, Deployment)

Note: While Argo CD applies Custom Resource Definitions (CRDs) before Custom Resources (CRs), all standard namespaced resources (ConfigMaps, Services, Deployments) reside within the same priority tier. Argo CD relies on explicit syntax dependencies to separate them, or treats them as peers.

5. Explicit vs. Implicit Dependencies in Kubernetes

Understanding why default ordering is insufficient in enterprise environments requires distinguishing between explicit and implicit dependencies.

Comparison Matrix

Real-World Pitfalls of Implicit Dependencies

1. The NetworkPolicy Race Condition

If a Deployment and a restrictive NetworkPolicy are deployed in the same default sync wave, the pods may start up and accept unencrypted or non-isolated traffic for a brief window before the API server processes and enforces the NetworkPolicy.

2. The ResourceQuota Scheduling Failure

If a Deployment is processed before a ResourceQuota or LimitRange object, pods may be scheduled without enforced resource boundaries, violating cluster governance policies or consuming unexpected node resources.

3. Backend vs. Frontend Deployment Sequence

In microservice architectures, a frontend application may crash-loop or trigger alerts if deployed simultaneously with its backend API. The backend must reach a healthy state before the frontend manifests are submitted.

6. Fundamentals of Argo CD Sync Waves

Sync waves allow administrators to convert implicit dependencies into explicit deployment steps by annotating Kubernetes manifests with integer wave numbers.

Annotation Syntax

To assign a resource to a specific wave, add the argocd.argoproj.io/sync-wave annotation under the object's metadata.annotations field:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"

Core Rules of Sync Waves

1. Integer Scale: Sync waves accept both positive and negative integers (e.g., $-99, -5, 0, 1, 10, 99$).
2. Ascending Execution ($W_i < W_j$): Lower numerical values are processed first.
3. Phase-Bound Scope: Sync waves operate within a single lifecycle phase (e.g., waves inside PreSync complete before waves inside Sync). They cannot bypass phase boundaries.
4. Wave Completion Criteria: Argo CD applies all resources in Wave $N$, waits for them to reach a Healthy state (or Synced for non-status objects), and only then proceeds to Wave $N+1$.
5. Peer Processing: Resources assigned to the same wave (e.g., Wave $0$) are applied concurrently as peers.

7. Step-by-Step Architecture Walkthrough: Wave-Based Resource Deployment

To achieve reliable deployments, resources should be categorized into ordered sync waves based on their architectural layer:

Wave Allocation Strategy

Production Best Practice: Leave numerical gaps between sync waves (e.g., use $-5, 0, 1, 5$ instead of $1, 2, 3, 4$). This creates room to inject intermediate resources (such as database migrations or vault sidecars) in future iterations without re-annotating existing manifests.

8. Hands-On Implementation: Manifest Configuration & Annotation Syntax

The following production manifests demonstrate a complete multi-tier application configured with Argo CD sync waves.

Project Directory Structure

Manifest Specifications

Wave $-5$: Core Foundation Layer

config/01-namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: app-1-ns
  annotations:
    argocd.argoproj.io/sync-wave: "-5"

config/02-resource-quota.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: app-1-quota
  namespace: app-1-ns
  annotations:
    argocd.argoproj.io/sync-wave: "-5"
spec:
  hard:
    pods: "2"
    requests.cpu: "500m"
    requests.memory: "512Mi"
    limits.cpu: "1000m"
    limits.memory: "1Gi"

config/03-service-account.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-1-sa
  namespace: app-1-ns
  annotations:
    argocd.argoproj.io/sync-wave: "-5"

config/04-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: app-1-secret
  namespace: app-1-ns
  annotations:
    argocd.argoproj.io/sync-wave: "-5"
type: Opaque
stringData:
  API_KEY: "cloud-with-varun-is-awesome"

Wave $0$: Configuration Layer

config/05-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-1-cm
  namespace: app-1-ns
  annotations:
    argocd.argoproj.io/sync-wave: "0"
data:
  index.html: |
    <!DOCTYPE html>
    <html>
    <body>
      <h1>Welcome to Cloud with Varun!</h1>
      <p>Your happy place for Cloud & DevOps learning. Build, Break, Learn, Repeat!</p>
    </body>
    </html>

Wave $1$: Networking Layer

config/06-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: app-1-service
  namespace: app-1-ns
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  type: ClusterIP
  selector:
    app: app-1
  ports:
  - port: 80
    targetPort: 80

Wave $5$: Application Workload Layer

config/07-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-1-deployment
  namespace: app-1-ns
  annotations:
    argocd.argoproj.io/sync-wave: "5"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-1
  template:
    metadata:
      labels:
        app: app-1
    spec:
      serviceAccountName: app-1-sa
      containers:
      - name: web
        image: nginx:alpine
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "250m"
            memory: "256Mi"
        volumeMounts:
        - name: html-volume
          mountPath: /usr/share/nginx/html
        env:
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: app-1-secret
              key: API_KEY
      volumes:
      - name: html-volume
        configMap:
          name: app-1-cm

Argo CD Application Definition (Argo/app1-app-cd.yaml)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-1-config
  namespace: argocd
spec:
  project: default
  source:
    repoURL: 'https://github.com/your-username/app1-config.git'
    targetRevision: HEAD
    path: config
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: app-1-ns
  syncPolicy:
    automated: null # Manual trigger configured for demo observation

9. Live Application Lifecycle & Verification Workflow

Step 1: Apply Application Custom Resource

Submit the Argo CD application manifest to the cluster to register the application tracking controller:

kubectl apply -f app1-config/Argo/app1-app-cd.yaml

Step 2: Trigger Synchronisation and Observe Wave Sequencing

When initiating a sync via the UI or CLI, Argo CD sorts the manifests into wave steps and applies them sequentially.

Step 3: Verify Deployed Resources

Check the deployed objects in the target namespace:

kubectl get all,resourcequota,configmap,secret -n app-1-ns

NAME                                   READY   STATUS    RESTARTS   AGE
pod/app-1-deployment-7d4554b77-x82l9   1/1     Running   0          45s

NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/app-1-service   ClusterIP   10.96.142.18   <none>        80/TCP    50s

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/app-1-deployment   1/1     1            1           45s

NAME                        AGE
resourcequota/app-1-quota   55s

NAME                   DATA   AGE
configmap/app-1-cm     1      50s

NAME                  TYPE     DATA   AGE
secret/app-1-secret   Opaque   1      55s

Step 4: Verify Traffic Forwarding and Content

Forward local port 8081 to the cluster service port 80:

kubectl port-forward svc/app-1-service 8081:80 -n app-1-ns

Accessing http://localhost:8081 via curl or browser returns the HTML served from the ConfigMap mounted inside the Deployment pod:

$ curl http://localhost:8081
<!DOCTYPE html>
<html>
<body>
  <h1>Welcome to Cloud with Varun!</h1>
  <p>Your happy place for Cloud & DevOps learning. Build, Break, Learn, Repeat!</p>
</body>
</html>

10. Summary and Key Takeaways

• kubectl Client Limitations: kubectl applies manifests in linear file order or alphabetical filename sequence. It lacks dependency intelligence, causing errors when workload dependencies appear later in the execution stream.
• Argo CD Execution Intelligence: Argo CD evaluates all manifests up front using client libraries to issue REST calls directly to the kube-apiserver, automatically resolving explicit dependencies.
• Precedence Hierarchy: Sync operations obey a four-tier evaluation hierarchy: Lifecycle Phase $\rightarrow$ Sync Wave $\rightarrow$ Resource Kind $\rightarrow$ Resource Name.
• Explicit vs. Implicit Dependencies: Argo CD handles explicit dependencies (e.g., serviceAccountName, mounted ConfigMap) by default. Implicit dependencies (e.g., ResourceQuota, NetworkPolicy, service startup ordering) require explicit wave annotations.
• Sync Wave Syntax: Sync waves are controlled using the argocd.argoproj.io/sync-wave: "<integer>" annotation in manifest metadata.
• Ascending Execution Order: Waves execute in ascending integer order (e.g., $-5$ before $0$, $0$ before $1$). Lower values have higher priority.
• Health Progression Gating: Argo CD applies Wave $N$ and waits for all included resources to reach a Healthy state before proceeding to Wave $N+1$.
• Production Spacing Strategy: Leave numerical gaps between wave annotations (e.g., $-5, 0, 1, 5$) to allow room for future architecture layers without requiring global re-annotation.