Day-8 | Introduction To GitLab CI/CD | 30 Days Of GitLab DevOps

This chapter introduces the core concepts of Continuous Integration and Continuous Delivery/Deployment (CI/CD) within the GitLab ecosystem. You will learn the theoretical foundations of automation pipelines, understand how GitLab orchestrates jobs using runners and executors, and build a hands-on, multi-stage pipeline from scratch. By the end of this guide, you will understand how to manage job execution order, configure custom container environments, and optimize pipeline execution using dependencies.

1. Understanding Continuous Integration (CI)

In modern software development, multiple developers push code changes to a shared repository daily, often multiple times a day. Without automation, integrating these changes is highly error-prone and leads to integration delays (often referred to as "integration hell").

Continuous Integration (CI) is the practice of automatically integrating code changes from multiple contributors into a single, shared repository and validating those changes through automated steps.

Core Phases of Continuous Integration

1. Code Push: A developer commits and pushes code to a branch in the GitLab repository.
2. Automated Trigger: GitLab detects the push and automatically triggers the configured CI pipeline.
3. Build Phase: The application code is compiled, and dependencies are resolved. For containerized applications, this phase often includes building Docker images.
4. Test & Validation Phase: Automated test suites (unit tests, integration tests, linters, and security scans) run against the build.
5. Feedback Loop: If any test fails, the pipeline fails, and the developer is notified immediately. This allows teams to catch issues early before they reach production.

2. Continuous Delivery vs. Continuous Deployment (CD)

While CI focuses on preparing and validating code, Continuous Delivery and Continuous Deployment (both abbreviated as CD) handle the release of that code to target environments (such as staging or production servers).

Continuous Delivery Workflow

In a Continuous Delivery model, the code is built, tested, and validated automatically. However, the actual deployment to production requires human intervention (e.g., a release manager clicking a "Deploy" button).

Continuous Deployment Workflow

In a Continuous Deployment model, there are no manual gates. Every change that passes all stages of the automated pipeline is deployed directly to production.

3. The Core Benefits of CI/CD in DevOps

CI/CD is considered the backbone of DevOps. It bridges the gap between development and operations teams by automating manual, repetitive tasks.

• Elimination of Manual Work: Instead of manually running test suites, building Docker images, and executing SSH commands to deploy code, everything is defined in a structured script (the pipeline) and executed automatically.
• Faster Time-to-Market: Automated pipelines allow teams to deliver new features, bug fixes, and updates to users within minutes of the code passing validation.
• Early Bug Detection: By running comprehensive test suites on every single commit, bugs are identified and resolved during the early stages of development, reducing debugging costs and production outages.
• Consistency and Reliability: Automated pipelines execute the exact same steps in the exact same order every time, eliminating human errors during deployment.

4. GitLab CI/CD Architecture: Pipelines, Stages, and Jobs

To write effective pipelines, you must understand the hierarchical relationship between Pipelines, Stages, and Jobs in GitLab.

1. Jobs

Jobs are the smallest unit of work in a GitLab pipeline. A job defines what to do by executing a series of shell commands or scripts. Examples of jobs include running a linter, compiling Java code, or running a vulnerability scan.

2. Stages

Stages define when to run jobs. They act as logical groupings for jobs.

• Jobs belonging to the same stage run in parallel by default (provided there are enough runner resources).
• Stages execute sequentially. Stage 2 (test) will only begin after all jobs in Stage 1 (build) have completed successfully.

3. Pipelines

A pipeline is the top-level component that encompasses all stages and jobs. It represents the entire workflow configured for your project's CI/CD process.

5. GitLab Runners and Executors Explained

GitLab Server does not execute your pipeline scripts. Instead, it delegates job execution to a separate component called the GitLab Runner.

How Runners Work

1. Polling: GitLab Runners continuously poll the GitLab Server to check if there are pending jobs in the queue.
2. Job Assignment: When a pipeline is triggered, the GitLab Server assigns the jobs to an available, compatible runner.
3. Execution: The runner executes the scripts defined in the job and streams the execution logs back to the GitLab UI in real-time.

Public vs. Private Runners

• Public Runners (Shared Runners): Provided out-of-the-box by GitLab. They are free to use (within monthly limits) and require zero configuration.
• Private Runners (Specific Runners): Virtual machines or physical servers that you provision, configure, and register with your GitLab instance. These are used when you need custom dependencies, specialized hardware, or strict security isolation.

What is an Executor?

An Executor determines the environment in which the runner executes the job scripts. When setting up a runner, you must choose an executor type. Common executors include:

• Shell Executor: Runs scripts directly on the runner machine's host operating system.
• Docker Executor: Spins up a Docker container using a specified image and runs the job scripts inside that container.
• Docker+Machine Executor: An auto-scaling executor that dynamically provisions temporary virtual machines, installs Docker, runs the job inside a container, and tears down the VM afterward.

6. The Configuration File: .gitlab-ci.yml

The entire configuration of a GitLab CI/CD pipeline is defined in a single YAML file.

Key Characteristics of the Configuration File

• Default Name: .gitlab-ci.yml
• Location: Must be placed at the root directory of your repository.
• Syntax: Written in standard YAML, which relies on strict indentation (spaces, not tabs).

Note on Customizing File Paths: If you want to use a custom name or path for your pipeline configuration file (e.g., project-ci.yml), you can change it in your GitLab project settings: Settings -> CI/CD -> General pipelines -> CI/CD configuration file.

7. Step-by-Step Hands-on: Creating Your First Pipeline

Let's walk through creating a basic multi-stage pipeline using a custom configuration file name.

Step 1: Create a New Project

1. Log in to GitLab and click New project -> Create blank project.
2. Name the project d8-cicd-basics.
3. Set the visibility level to Public and click Create project.

Step 2: Configure a Custom Pipeline File Name

1. In your project's left sidebar, navigate to Settings -> CI/CD.
2. Expand the General pipelines section.
3. Locate the CI/CD configuration file field.
4. Change the value from the default to project-ci.yml.
5. Click Save changes.

Step 3: Create the Pipeline Configuration File

1. Go back to the project repository homepage.
2. Click the + icon and select New file.
3. Name the file project-ci.yml.
4. Paste the following basic pipeline configuration:

stages:
  - build
  - test
  - deploy

compile job:
  stage: build
  script:
    - echo "Compiling the application source code..."
    - echo "Compilation complete."

validate job:
  stage: build
  script:
    - echo "Validating code syntax and structure..."
    - echo "Validation complete."

unit test job:
  stage: test
  script:
    - echo "Running unit tests..."
    - sleep 10
    - echo "Unit tests passed."

lint test job:
  stage: test
  script:
    - echo "Running code linter..."
    - echo "Linting complete."

deploy job:
  stage: deploy
  script:
    - echo "Deploying application to production server..."
    - echo "Deployment successful."

Step 4: Commit and Monitor the Pipeline

1. Write a commit message (e.g., feat: add initial multi-stage pipeline).
2. Click Commit changes.
3. Navigate to Build -> Pipelines in the left sidebar.
4. Click on the running pipeline to view the visual execution graph.

Observe that compile job and validate job run in parallel because they both belong to the build stage. Similarly, unit test job and lint test job run in parallel during the test stage.

8. Under the Hood: How GitLab Executes Jobs (Docker + Machine)

When using GitLab's public shared runners, you will notice specific logs at the beginning of every job execution. Let's analyze what happens behind the scenes.

Analyzing the Job Logs

When you open the logs for compile job, you will see output similar to this:

Running with gitlab-runner 15.11.0 (91a27b2a)
  on blue-5.shared.runners-manager.gitlab.com/ip-10-0-12-34 tZ3yXyZ
Preparing the "docker+machine" executor
Using Docker Executor with image ruby:3.1 ...
Pulling docker image ruby:3.1 ...
Using docker image sha256:abc123xyz... for ruby:3.1 with digest ruby:3.1 ...

The Execution Lifecycle (Docker + Machine)

When a job is picked up by a GitLab public runner using the docker+machine executor, the following steps occur:

1. VM Provisioning: The runner manager dynamically provisions a temporary, clean virtual machine (VM) in the cloud.
2. Docker Initialization: Docker is started on this temporary VM.
3. Container Creation: The runner pulls a default Docker image—which is ruby:3.1 by default if no image is specified in the YAML file.
4. Script Execution: The runner starts a container from that image and executes your job's script commands inside it.
5. Teardown (Ephemeral Environment): Once the job completes (whether it succeeds or fails), the temporary VM is completely destroyed. This ensures that every job runs in a clean, isolated, and secure environment.

9. Customizing Execution Environments with Global Images

Using the default ruby:3.1 image is rarely ideal for non-Ruby applications. If your application requires specific runtimes (like Node.js, Python, Java, or Ubuntu utilities), you must define a custom container image.

You can define a container image at the global level (affecting all jobs) or at the job level (affecting only that specific job).

Configuring a Global Ubuntu Image

Let's modify our pipeline to run all jobs inside an Ubuntu container instead of the default Ruby container.

Update your project-ci.yml file to include the image keyword at the top level:

image: ubuntu:24.04

stages:
  - build
  - test
  - deploy

compile job:
  stage: build
  script:
    - cat /etc/os-release
    - echo "Compiling the application source code..."

validate job:
  stage: build
  script:
    - echo "Validating code syntax..."

unit test job:
  stage: test
  script:
    - echo "Running unit tests..."

lint test job:
  stage: test
  script:
    - echo "Running linter..."

deploy job:
  stage: deploy
  script:
    - echo "Deploying application..."

Verifying the Environment Change

Commit these changes and open the logs for the new compile job. You will observe the following changes:

Preparing the "docker+machine" executor
Using Docker Executor with image ubuntu:24.04 ...
Pulling docker image ubuntu:24.04 ...
...
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"

The job now executes inside an isolated Ubuntu 24.04 container environment.

10. Managing Job Dependencies and Parallelism with the needs Keyword

By default, GitLab executes jobs within a stage in parallel. However, there are scenarios where you want a job to start as soon as its specific prerequisite job completes, without waiting for the entire preceding stage to finish.

The needs keyword allows you to establish direct dependencies between jobs, bypassing the strict sequential execution of stages.

Real-World Scenario

Suppose you have two jobs in the build stage: compile job and validate job.

• Naive approach: They run in parallel.
• Better approach: You want validate job to run only after compile job completes successfully, while keeping both in the build stage.

Implementing the needs Dependency

Update your project-ci.yml file to configure this dependency:

image: ubuntu:24.04

stages:
  - build
  - test
  - deploy

compile job:
  stage: build
  script:
    - echo "Compiling application..."
    - sleep 5

validate job:
  stage: build
  needs: ["compile job"]
  script:
    - echo "Validating compiled code..."

unit test job:
  stage: test
  script:
    - echo "Running unit tests..."

lint test job:
  stage: test
  script:
    - echo "Running linter..."

deploy job:
  stage: deploy
  script:
    - echo "Deploying application..."

Observing the Execution Behavior

1. Commit the changes and navigate to the running pipeline.
2. In the build stage, you will see that only compile job starts running initially.
3. The validate job remains in a pending state.
4. As soon as compile job completes successfully, validate job is triggered immediately.
5. Meanwhile, the test stage jobs (unit test job and lint test job) still run in parallel once the build stage completes.

Summary and Key Takeaways

• Continuous Integration (CI) automates the process of merging, building, and testing code changes to identify bugs as early as possible.
• Continuous Delivery (CD) ensures code is always in a deployable state but requires manual approval to release to production.
• Continuous Deployment (CD) automates the entire release process, deploying code to production without manual intervention as soon as it passes the pipeline.
• GitLab Pipelines are composed of Stages (which execute sequentially) and Jobs (which execute in parallel within their stage by default).
• GitLab Runners are the execution agents that poll GitLab for jobs, while Executors (like Docker or Shell) define the environment where the scripts run.
• The docker+machine Executor provides clean, isolated, and ephemeral environments by spinning up a temporary VM and running jobs inside a container.
• The Default Image used by GitLab's shared runners is ruby:3.1 unless overridden by the image keyword.
• The needs Keyword allows you to define explicit dependencies between jobs, enabling sequential execution within a single stage or across stages to optimize pipeline speed.