Day 13: Pipeline Debugging in GitLab CI/CD

Pipeline debugging is the systematic process of identifying why a CI/CD job failed and resolving the underlying issue using execution logs, configuration files, and environmental clues. In a complex DevOps pipeline, jobs run in isolated, ephemeral environments, making robust log analysis and defensive scripting essential skills for maintaining smooth delivery workflows.

1. Navigating the GitLab UI for Debugging

When a pipeline fails, the GitLab User Interface (UI) serves as the primary diagnostic window. To locate and analyze the failure, follow this sequential path:

By isolating the specific job log, you can view the exact standard output (stdout) and standard error (stderr) streams generated during execution.

2. Anatomy of a GitLab CI/CD Job Log

Every GitLab CI/CD job log follows a standardized, sequential execution structure. Understanding this structure allows you to quickly isolate whether a failure occurred due to infrastructure issues, code checkout problems, caching, or your custom scripts.

The Four Core Execution Phases

1. Prepare Environment: The GitLab Runner initializes, pulls the specified container image (if using the Docker executor), and configures system resources.
2. Get Source: The runner clones the project repository and checks out the exact commit SHA that triggered the pipeline.
3. Restore Cache: If caching is configured, the runner downloads the archived cache from the remote storage and extracts it into the working directory.
4. Run Script: The runner executes the commands declared under the script block of the active job.

3. The Golden Rule of Log Analysis

The Golden Rule of Debugging: Always scroll to the first error that appears in the log from top to bottom.

Do not jump straight to the very last line of a failed job log. Often, the final lines of a log show generic exit codes (e.g., ERROR: Job failed: exit code 1) which are merely symptoms of an earlier failure. The root cause—such as a syntax error, missing dependency, or logic issue—is typically printed higher up at the moment the first command failed.

4. Common Failure Patterns & Troubleshooting

The table below synthesizes the most common failure patterns encountered in GitLab CI/CD pipelines, their root causes, and how to resolve them.

5. The "Retry Job" Mechanism

A Retry is the execution of a failed or completed job again without making any changes to the source code or the .gitlab-ci.yml configuration.

When to Use Retry

• Temporary Network Glitches: Intermittent timeouts when pulling external dependencies or Docker images.
• Runner Glitches: Transient infrastructure issues on the host machine hosting the GitLab Runner.
• Cache Corruption: Temporary issues downloading or extracting remote cache archives.

Warning: Retrying a job does not fix bugs in your source code or pipeline configuration. If a job fails due to a syntax error or a missing dependency, retrying it will yield the exact same failure.

6. Proactive Debugging Techniques

When standard logs do not provide enough context, you can inject diagnostic commands into your .gitlab-ci.yml script block to expose the state of the execution environment.

• Adding Debug Prints: Use environmental discovery commands to verify paths and file structures:
  YAML

  Copy

  script:
    - pwd          # Prints the current working directory
    - ls -la       # Lists all files, including hidden ones, with permissions
  

• Printing Variables: Verify that GitLab CI/CD predefined or custom variables are populated correctly:
  YAML

  Copy

  script:
    - echo "Current branch is $CI_COMMIT_BRANCH"
  

• Enabling Shell Execution Tracing (set -x): Adding set -x (or set -o xtrace) at the beginning of a script forces the shell to print each command before executing it. This makes it easy to see exactly how variables are expanding during execution.

7. Practical Lab 1: Resolving Missing Artifacts

The Problem

In a multi-stage pipeline, jobs run in isolated, clean environments. If a job in the build stage generates a file, that file is lost when the container exits. A subsequent job in the test stage will fail with a No such file or directory error if it tries to access that file.

Naive (Failing) Configuration

stages:
  - build
  - test

build_job:
  stage: build
  image: ubuntu:latest
  script:
    - mkdir build
    - echo "app content" > build/app.txt

test_job:
  stage: test
  image: ubuntu:latest
  script:
    - cat build/app.txt

• Result: test_job fails with cat: build/app.txt: No such file or directory because the build/ directory was not persisted.

Decoupled & Fixed Configuration (Using Artifacts)

To pass directories or files between stages, you must declare them as artifacts. GitLab will upload these files to its coordinator and automatically download them into subsequent stages.

stages:
  - build
  - test

build_job:
  stage: build
  image: ubuntu:latest
  script:
    - mkdir build
    - echo "app content" > build/app.txt
  artifacts:
    paths:
      - build/ # Persists the build directory for subsequent stages

test_job:
  stage: test
  image: ubuntu:latest
  script:
    - cat build/app.txt

• Result: Success. The runner downloads the build/ artifact before executing the test_job script.

8. Practical Lab 2: Resolving Cache-Induced Logic Errors

The Problem

Caching is designed to speed up pipelines by reusing downloaded dependencies (like node_modules). However, if your script contains non-idempotent commands (commands that fail if run more than once, like mkdir), restoring a cache can break your pipeline on subsequent runs.

Naive (Failing) Configuration

stages:
  - build

build_job:
  stage: build
  image: ubuntu:latest
  cache:
    key: node-modules-cache
    paths:
      - node_modules/
  script:
    - mkdir node_modules # Fails on the second run because the directory is restored from cache

Defensive Scripting Fix

To prevent this logic error, write defensive shell scripts that check for the existence of a directory before attempting to create it.

stages:
  - build

build_job:
  stage: build
  image: ubuntu:latest
  cache:
    key: node-modules-cache
    paths:
      - node_modules/
  script:
    - |
      if [ -d "node_modules" ]; then
        echo "node_modules directory already exists (restored from cache)."
      else
        echo "node_modules not found. Creating directory..."
        mkdir node_modules
      fi

• Result: Success on all subsequent runs. The script gracefully handles both cold starts (no cache) and warm starts (cache restored).

9. Practical Lab 3: Resolving "Command Not Found" with Custom Images

The Problem

By default, generic runner images (like ubuntu:latest) are minimal operating system environments. They do not come pre-installed with runtime engines or package managers like Node.js, npm, Python, or Pip. Running these tools directly results in a command not found error.

Naive (Failing) Configuration

stages:
  - test

test_job:
  stage: test
  image: ubuntu:latest
  script:
    - echo '{"dependencies": {}}' > package.json
    - npm install # Fails: "npm: command not found"

Fixed Configuration (Using Specialized Images)

Instead of manually installing Node.js and NPM during runtime (which slows down execution), configure the job to run inside a pre-configured Docker image that contains the required tools.

stages:
  - test

test_job:
  stage: test
  image: node:18 # Uses an official Node.js image containing npm
  script:
    - echo '{"dependencies": {}}' > package.json
    - npm install

• Result: Success. The runner pulls the node:18 image, which has npm pre-installed, allowing the package installation to run immediately.

Summary and Key Takeaways

• Pipeline Debugging Definition: The process of identifying and fixing job failures using logs, configuration files, and environmental clues.
• Log Navigation: Access logs by navigating to CI/CD > Pipelines, selecting the active pipeline, and clicking on the specific failed job.
• Standard Log Structure: Logs execute in four distinct phases: Prepare Environment, Get Source, Restore Cache, and Run Script.
• The Golden Rule: Always scroll to the first error in the log from top to bottom to find the root cause, rather than looking only at the final exit code.
• Artifacts vs. Cache: Use artifacts to pass files between different stages in a pipeline; use cache to speed up build times by reusing dependencies across different runs of the same job.
• Defensive Scripting: Write shell scripts that check for file or directory existence (e.g., if [ -d "dir" ]) to prevent logic errors when caches are restored.
• Image Selection: Resolve command not found errors by matching your job's image directive to a specialized Docker image that contains your required runtimes (e.g., using node:18 instead of ubuntu:latest for Node.js projects).
• The Limits of Retrying: Use the Retry button only for transient runner or network issues; code or configuration bugs require a new commit to resolve.