GitLab CI/CD: Creating & Using Common Pipeline Templates

As projects scale from simple learning experiments to massive enterprise-level repositories, managing continuous integration and delivery (CI/CD) pipelines becomes complex. Hardcoding every single job, stage, and script across dozens of branches and multiple repositories leads to severe duplication, maintenance overhead, and operational headaches. This chapter explores how to solve these scaling issues by introducing Pipeline Templates—a modular, DRY (Don't Repeat Yourself) approach to GitLab CI/CD pipeline management.

1. Why Do We Need Pipeline Templates? (The Status Quo Problem)

In basic GitLab CI/CD setups, a pipeline is defined entirely within a single .gitlab-ci.yml file sitting at the root of a branch. This file contains everything: rules, stages, job definitions, scripts, artifact storage declarations, and dependency management (needs).

While this works well for simple tutorials, real-world enterprise projects face severe organizational bottlenecks:

• Massive File Size: A realistic enterprise pipeline often contains 7 to 8 stages, with 2 to 3 jobs per stage. The configuration file quickly grows into hundreds of lines of dense YAML.
• Branch Proliferation: A single repository might have 50, 60, or even over 100 active feature branches, hotfix branches, and release branches. If every branch contains its own duplicated pipeline definition, changing a global setting becomes a nightmare.
• Multi-Project Sprawl: Enterprise environments rarely consist of just one project. Organizations often maintain 50 to 100+ separate microservices or applications.
• The Maintenance Nightmarish Loop: If a security policy changes, a deployment tool updates, or a build command needs a modification, engineers would theoretically have to update dozens of branches across multiple repositories manually.

2. Core Mental Model: The Software Function Analogy

Real-Life Analogy: Think about software programming. If you need to calculate taxes across an application, you don't copy and paste the tax calculation logic into every single file. Instead, you write a function once, store it in a shared library, and call (or import) that function wherever you need it.

Pipeline templates apply this exact programming principle (DRY - Don't Repeat Yourself) to CI/CD pipelines. You write your job definitions once in a centralized template repository, and individual projects simply pull and assemble those templates when a pipeline runs.

3. Naive vs. Modular Approach: Code & Structure Evolution

To understand how templates clean up your workflow, let's contrast the naive monolithic approach with the modular template architecture.

The Naive Monolithic Approach (Single File)

In a naive setup, every single job definition lives crammed inside one massive file:

# .gitlab-ci.yml (Monolithic & Hard to Maintain)
stages:
  - build
  - test
  - release
  - deploy

maven_build:
  stage: build
  script:
    - mvn clean package
  artifacts:
    paths:
      - target/*.jar

unit_test:
  stage: test
  script:
    - mvn test

# ... dozens more lines for release and deploy jobs ...

The Modular Template Approach (Centralized Repository)

Instead of stuffing everything into one file, we break the job definitions down into logical component files housed inside a dedicated central project (e.g., pipeline-templates).

Inside maven-build.yml, a single job is defined cleanly:

# pipeline-templates/build/maven-build.yml
maven_build:
  stage: build
  script:
    - echo "Running Maven Build..."
    - mvn clean package
  artifacts:
    paths:
      - target/*.jar

4. Step-by-Step Implementation: Creating the Central Template Project

To set up and consume common pipeline templates in GitLab, follow this systematic workflow:

1. Establish a Central Project: Create a dedicated project (e.g., common-pipeline-jobs) inside your organization's group or namespace (e.g., under devops/).
2. Modularize Job Definitions: Break down your pipeline into separate YAML files categorized by stage (e.g., build/, test/, release/, deploy/).
3. Pull Templates Using the include Keyword: In your individual consumer projects, clear out local job definitions and pull remote templates using GitLab's native include: keyword with project references.

5. Consuming Templates in Individual Projects

In your actual application repository (on a branch like feat/1.0), you discard the monolithic YAML and use the include keyword to pull jobs from your central template repository.

Consuming a Single Template First

stages:
  - build

include:
  - project: 'devops/common-pipeline-jobs'
    ref: 'feat/1.0'
    file: '/build/maven-build.yml'

Note: You must explicitly define the stages block in your local .gitlab-ci.yml file, and the stage names must precisely match the stage declarations inside the imported template jobs (e.g., stage: build).

Scaling to Full Multi-Stage Pipelines

To assemble a complete pipeline pulling build, test, release, and deploy jobs from the central repository, your consumer .gitlab-ci.yml shrinks down to a clean, declarative manifest:

stages:
  - build
  - test
  - release
  - deploy

include:
  - project: 'devops/common-pipeline-jobs'
    ref: 'feat/1.0'
    file: 
      - '/build/maven-build.yml'
      - '/test/unit-test.yml'
      - '/release/github-release.yml'
      - '/deploy/k8s-deploy.yml'

Complete Execution Flow Walkthrough

6. Real-World Maintenance & Management Benefits

Moving to a centralized template model transforms operational overhead in enterprise environments:

Summary and Key Takeaways

• The Scaling Challenge: As projects grow to dozens of branches and multiple repositories, maintaining monolithic .gitlab-ci.yml files creates severe duplication and maintenance bottlenecks.
• The DRY Principle: Pipeline templates apply software programming best functions to CI/CD, allowing you to write job definitions once and reuse them everywhere.
• Centralized Storage: Store reusable job definitions in a dedicated central repository (e.g., devops/common-pipeline-jobs) organized logically by folders and stages.
• The include Keyword: Individual projects pull remote template files dynamically using the include directive by specifying the target project, branch reference (ref), and file path (file).
• Stage Alignment Principle: Local .gitlab-ci.yml files must declare stages matching the exact stage names defined inside the imported remote job templates.
• Single Point of Control: Making a global update (such as appending a script command or security scan) requires changing only the central template file, instantly benefiting all consuming projects.
• Pipeline as Code Maturity: Moving to modular templates represents an advanced maturity level in GitLab CI/CD, paving the way for advanced project-specific custom overrides and extensibility.