Day 19: Multi-Project and Parent-Child Pipelines in GitLab CI/CD

In modern enterprise architectures, continuous integration and continuous delivery (CI/CD) pipelines frequently scale beyond the boundaries of a single configuration file. As organizations adopt microservices, monorepos, and cross-functional team structures (Dev, Security, Cloud/Terraform, and Operations), managing continuous integration requires sophisticated orchestration techniques. This chapter explores Parent-Child Pipelines (intra-repository pipeline splitting) and Multi-Project Pipelines (inter-repository pipeline chaining), demonstrating how to decouple large configurations, improve execution speed, prevent team merge conflicts, and enforce deterministic deployment orders.

1. The Scaling Problem in Enterprise CI/CD

As software systems expand, two common architectural paradigms emerge, each presenting distinct challenges to standard CI/CD configurations:

Scenario A: Microservices Across Multiple Repositories

Consider an architecture divided across several independent git repositories:

• auth-service
• backend-service
• payment-service
• frontend-service

Each service maintains its own repository with independent build, test, and release cycles. However, end-to-end integration or coordinated production rollouts require these microservices to be deployed in a strict sequence:

Relying on developers to manually navigate into each repository, trigger pipelines one by one, and verify health checks is error-prone, uncoordinated, and lacks centralized visibility.

Scenario B: Monorepos with Massive Single .gitlab-ci.yml Files

In a monorepo containing application source code, Terraform infrastructure code, Kubernetes manifests, security configurations, and monitoring definitions, housing all CI/CD logic inside a single .gitlab-ci.yml leads to significant operational friction:

• Massive File Sizes: Single configuration files often balloon to 500, 1,000, or 2,000+ lines of YAML.
• Cross-Team Merge Conflicts: Development, Security, Cloud, and Operations teams constantly edit the same .gitlab-ci.yml file simultaneously, leading to frequent merge conflicts.
• Pipeline Slowness and Complexity: Unnecessary stages are evaluated, creating slow execution times and bloated visual pipeline graphs.

2. Core Mental Model: Parent-Child vs. Multi-Project Pipelines

GitLab provides two native orchestration mechanisms to resolve these architectural bottlenecks:

Real-Life Analogy: The General Contractor and Subcontractors

Think of a Parent Pipeline as a General Contractor overseeing the construction of a building. The General Contractor does not personally lay every brick, wire every electrical outlet, or install the plumbing. Instead, the contractor holds the master schedule (the Parent Pipeline) and delegates specialized tasks to dedicated Subcontractors (the Child Pipelines):

• The Electrical Subcontractor (Security Child Pipeline) works from their own blueprint.
• The Plumbing Subcontractor (Terraform Child Pipeline) operates independently.
• The Framing Subcontractor (Build/Test Child Pipeline) executes its tasks in isolation.

If a subcontractor fails an inspection, the General Contractor's master schedule pauses until the failure is rectified.

3. The Core Syntactic Primitives: trigger and strategy: depend

To orchestrate child or downstream pipelines, GitLab CI/CD utilizes the trigger keyword instead of standard job execution primitives (script, before_script).

The strategy: depend Directive

By default, triggering a downstream pipeline is an asynchronous (fire-and-forget) operation. The parent job triggers the child and immediately marks itself as passed.

When strategy: depend is added:

1. The parent job enters a running state and waits for the downstream child/project pipeline to finish.
2. The parent job mirrors the downstream pipeline's final status:
   • If the child pipeline succeeds, the parent job succeeds.
   • If the child pipeline fails, the parent job fails.

4. Lab 1: Basic Parent-Child Pipeline (Single Repo, Single Child)

This scenario demonstrates how a primary parent pipeline in .gitlab-ci.yml triggers a localized downstream child pipeline defined in an auxiliary YAML file within the same repository.

Step 1: Define the Child Pipeline Configuration

Create a dedicated child YAML file (e.g., child-pipeline.yaml) containing localized stages and jobs:

# File: child-pipeline.yaml
stages:
  - test
  - deploy

child_test_job:
  stage: test
  script:
    - echo "Executing child pipeline unit/integration tests..."

child_deploy_job:
  stage: deploy
  script:
    - echo "Executing child pipeline deployment steps..."

Note: Committing this file alone does not trigger any pipeline run, as GitLab looks exclusively for .gitlab-ci.yml on push events unless configured otherwise.

Step 2: Define the Parent Pipeline Configuration

Create the master .gitlab-ci.yml file to include and trigger child-pipeline.yaml:

# File: .gitlab-ci.yml
stages:
  - trigger

trigger_child:
  stage: trigger
  trigger:
    include: child-pipeline.yaml
    strategy: depend

Execution Flow:

5. Lab 2: Multi-Child Pipelines (Modular Monorepos)

When handling front-end and back-end modules within a single monorepo, each team maintains their own isolated pipeline file. The parent pipeline triggers both child pipelines concurrently or sequentially.

Child Configuration 1: backend.gitlab-ci.yml

# File: backend.gitlab-ci.yml
stages:
  - build

backend_build_job:
  stage: build
  script:
    - echo "Compiling backend binaries and running unit tests..."

Child Configuration 2: frontend.gitlab-ci.yml

# File: frontend.gitlab-ci.yml
stages:
  - build

frontend_build_job:
  stage: build
  script:
    - echo "Bundling frontend assets and executing UI tests..."

Parent Orchestration: .gitlab-ci.yml

# File: .gitlab-ci.yml
stages:
  - trigger

trigger_backend:
  stage: trigger
  trigger:
    include: backend.gitlab-ci.yml
    strategy: depend

trigger_frontend:
  stage: trigger
  trigger:
    include: frontend.gitlab-ci.yml
    strategy: depend

Monorepo Execution Flow:

6. Lab 3: Multi-Project Pipelines (Cross-Repository Orchestration)

In distributed microservice ecosystems, independent projects live in separate repositories under a shared namespace or group. Multi-project pipelines allow Project A to trigger Project B across repository boundaries.

Project Setup

• Upstream Project (Project A): group-name/day-19-project-a
• Downstream Project (Project B): group-name/day-19-project-b

Step 1: Define the Downstream Target Pipeline (Project B)

Inside the repository for day-19-project-b, create a standard .gitlab-ci.yml:

# Project B: .gitlab-ci.yml
stages:
  - deploy

deploy_service_b:
  stage: deploy
  script:
    - echo "Deploying Project B microservice to target environment..."

Step 2: Configure Upstream Cross-Project Trigger (Project A)

Inside the repository for day-19-project-a, configure .gitlab-ci.yml to target Project B using the project directive:

# Project A: .gitlab-ci.yml
stages:
  - trigger

trigger_project_b:
  stage: trigger
  trigger:
    project: group-name/day-19-project-b
    branch: main
    strategy: depend

Cross-Project Pipeline Execution Flow:

7. Pipeline Strategy Comparison

8. Real-World Practical Pitfalls, Edge Cases, and Debugging

1. The Missing strategy: depend Anti-Pattern

• Pitfall: When omitting strategy: depend, the parent pipeline immediately reports success as soon as the trigger job fires, even if the child/downstream pipeline subsequently fails during deployment.
• Resolution: Always include strategy: depend when the parent pipeline must act as an authoritative quality or release gate.

2. Permissions and Namespace Visibility in Multi-Project Pipelines

• Pitfall: Upstream triggers fail with 404 Not Found or 403 Forbidden errors when attempting to trigger cross-project pipelines.
• Resolution: The user triggering the upstream pipeline must have at least Developer access to the downstream project. For automated cross-group pipelines, ensure CI/CD Job Tokens (or Pipeline Trigger Tokens) are properly authorized under Settings > CI/CD > Job Token Permissions.

3. File Path Inaccuracies in Parent-Child Includes

• Pitfall: Incorrect local path references inside the include directive cause syntax errors before pipeline instantiation.
• Resolution: Paths defined under trigger: include: are evaluated relative to the repository root. Ensure exact relative paths (e.g., ci/security.gitlab-ci.yml or child-pipeline.yaml) are used.

4. Runner Concurrency and Deadlocks

• Pitfall: If GitLab Runners have a concurrency limit of 1, and a parent pipeline occupies that slot while waiting (strategy: depend) for a child pipeline, the child job may stay in pending indefinitely, causing a deadlock.
• Resolution: Configure runner concurrency (concurrent >= 2) in /etc/gitlab-runner/config.toml to permit parent waiting jobs and child execution jobs to run simultaneously.

## Summary and Key Takeaways

• Architectural Decoupling: Parent-child and multi-project pipelines prevent bloated, unmaintainable .gitlab-ci.yml files by splitting monolithic CI/CD logic into modular, domain-specific configurations.
• Parent-Child Boundaries: Used within a single repository (monorepos) using trigger: include: <local-file.yaml>.
• Multi-Project Boundaries: Used across distinct repositories (microservices) using trigger: project: <group/repo> and optional branch: targeting.
• Deterministic Synchronization: The strategy: depend directive forces the upstream/parent trigger job to remain active until downstream execution completes, propagating failure or success statuses accurately.
• Team Autonomy: Domain teams (Dev, Security, Cloud/Terraform, QA) can independently update their own CI/CD modules without introducing merge conflicts or risking instability in the central root pipeline.
• Visual Graph Clarity: GitLab UI renders downstream and child pipelines as clean, expandable hierarchical nodes rather than unreadable monolithic linear graphs.