Day 11: Artifacts & Caches in GitLab CI/CD

This chapter explores how to optimize GitLab CI/CD pipelines using Artifacts and Caches. You will learn how to resolve the challenges of isolated runner environments, prevent redundant work, and configure pipelines that are fast, efficient, and production-ready.

1. The Problem of Isolated Runner Environments

In GitLab CI/CD, every job runs in a completely isolated, fresh environment. When using shared or public runners, GitLab provisions a new runner instance (often a temporary Docker container) for each individual job. Once the job completes, this container is destroyed along with all its local data.

The Runner Lifecycle Workflow

Key Problems Caused by Isolation

1. No Native Data Sharing: Files generated in one job (such as compiled binaries, test reports, or build packages) are not natively visible to subsequent jobs.
2. Repeated Work: If Job 1 installs project dependencies and Job 2 requires those same dependencies, Job 2 must download and install them all over again.
3. Pipeline Slowness: Downloading packages and rebuilding binaries repeatedly consumes massive amounts of network bandwidth and CPU time, making pipelines slow and resource-heavy.

2. Mental Model: Artifacts vs. Caching

To solve the problems of isolated runner environments, GitLab provides two distinct mechanisms: Artifacts and Caches. While they may seem similar at first glance, they serve entirely different purposes.

Real-Life Analogy

• The Postal Package (Artifacts)

  Imagine you are working on an assembly line. Step 1 (Build) creates a physical product and packages it into a box. This box is sent to a central warehouse (GitLab's temporary storage). Step 2 (Test) and Step 3 (Deploy) request this specific box from the warehouse to perform their tasks. The package is a formal, guaranteed output of that specific run.

• The Local Kitchen Pantry (Caching)

  Imagine you are cooking a meal. Instead of driving to the grocery store (the internet) to buy salt, flour, and sugar every single time you cook, you store these reusable ingredients in your kitchen pantry (the cache). If you run out of an ingredient, you fetch it from the store once and put the remainder back in the pantry for future meals. The pantry speeds up your cooking process, but if the pantry is suddenly cleared, you can still complete the meal by going back to the store.

3. Deep Dive into GitLab Artifacts

Artifacts are files or directories generated by a job that are uploaded to GitLab's central storage. They are guaranteed to be passed to subsequent jobs in the same pipeline.

Key Properties of Artifacts

• Pipeline-Scoped: Artifacts are created within a specific pipeline run and are passed down to subsequent stages of that same pipeline.
• Guaranteed Availability: Subsequent jobs in the pipeline will automatically download these artifacts before executing their scripts.
• UI Accessibility: Artifacts can be browsed and downloaded directly from the GitLab web interface.
• Expiration: Artifacts can be configured to expire after a set duration (e.g., 90 days by default, or 1 hour for temporary test outputs) to save storage space.

4. Deep Dive into GitLab Caching

Cache is used to store reusable files (such as external dependencies, package managers' download directories, or compilation caches) to speed up subsequent jobs and future pipeline runs.

Key Properties of Caching

• Performance-Oriented: The primary goal of caching is to avoid doing the same work twice (e.g., downloading the same NPM packages or Maven dependencies).
• Cross-Pipeline Availability: Unlike artifacts, caches can persist across different runs of the same pipeline, as well as across different branches.
• Not Guaranteed: Caches are meant for temporary, non-essential files. If a cache is cleared or unavailable, the job must still be able to download or regenerate the files from scratch.
• No UI Download: Caches are stored internally on the runner or distributed object storage and cannot be downloaded directly from the GitLab UI.

5. Tabular Comparison: Artifacts vs. Caches

6. Step-by-Step Evolution: Implementing Artifacts

Let us look at how to implement artifacts in a .gitlab-ci.yml file. We will contrast a naive approach (which fails due to runner isolation) with a decoupled, artifact-enabled configuration.

The Naive (Failing) Approach

In this configuration, the test_job tries to read a file created by the build_job. Because they run on separate, isolated runners, the file is missing, and the test job fails.

stages:
  - build
  - test

build_job:
  stage: build
  image: ubuntu:latest
  script:
    - echo "Building application..."
    - mkdir build
    - echo "My app content" > build/app.txt

test_job:
  stage: test
  image: ubuntu:latest
  script:
    - echo "Testing application..."
    # This command will FAIL because build/app.txt does not exist on this runner
    - cat build/app.txt 

The Correct (Artifact-Enabled) Approach

By defining the artifacts keyword, GitLab uploads the build/ directory to central storage at the end of the build_job and automatically downloads it at the start of the test_job.

stages:
  - build
  - test

build_job:
  stage: build
  image: ubuntu:latest
  script:
    - echo "Building application..."
    - mkdir build
    - echo "My app content" > build/app.txt
  artifacts:
    paths:
      - build/
    expire_in: 1 hour  # Automatically deletes the artifact after 1 hour to save space

test_job:
  stage: test
  image: ubuntu:latest
  script:
    - echo "Testing application..."
    # This command succeeds because GitLab automatically restored the build/ directory
    - cat build/app.txt

7. Step-by-Step Evolution: Implementing Caches

Caching is typically defined globally or per-job. In this example, we configure a global cache for a Node.js project to preserve the node_modules/ directory across jobs and future pipeline runs.

Global Cache Configuration

stages:
  - build
  - test

# Define cache globally so all jobs can access and update it
cache:
  key: my-cache-unique-key
  paths:
    - node_modules/

build_job:
  stage: build
  image: ubuntu:latest
  script:
    - echo "Restoring cache and installing dependencies..."
    # Simulate installing dependencies by creating the folder and a dummy file
    - mkdir -p node_modules
    - echo "Dependency file content" > node_modules/dependency.txt
    - echo "Build completed."

test_job:
  stage: test
  image: ubuntu:latest
  script:
    - echo "Running tests..."
    # The node_modules folder is restored automatically via the cache
    - cat node_modules/dependency.txt

How the Cache Key Works

The key directive determines how the cache is grouped and isolated.

• If you use a static string like my-cache-unique-key, every branch and pipeline run will share and overwrite the exact same cache.
• To prevent different branches from overwriting each other's caches, you can use GitLab CI/CD predefined variables:
  YAML

  Copy

  cache:
    key: "$CI_COMMIT_REF_SLUG" # Creates a unique cache per branch
    paths:
      - node_modules/
  

8. Complete Pipeline Flow Walkthrough

Let us trace the exact execution logs and runner behaviors during a pipeline run that uses caching.

Execution Flow of the First Pipeline Run

Execution Flow of the Second Pipeline Run (Future Trigger)

When the pipeline is triggered a second time, the build_job immediately benefits from the cache:

9. Real-World Pitfalls and Debugging Workflows

When implementing artifacts and caches in production environments, you may encounter several common failure modes.

Pitfall 1: Default Runner Image Conflicts

• The Issue: If you do not explicitly define an image in your .gitlab-ci.yml file, GitLab shared runners default to a pre-configured image (such as a Ruby image). This can cause jobs to fail if they rely on tools like bash, cat, or npm that might not be present or configured correctly in that default environment.
• The Fix: Always declare a explicit, lightweight base image (e.g., ubuntu:latest, alpine:latest, or language-specific images like node:alpine) at the top level or per job.

Pitfall 2: Confusing Cache with Artifacts for Critical Files

• The Issue: A developer uses cache to pass a compiled .jar file from the build stage to the deploy stage. On a busy day, the runner's cache storage fills up, and the cache is cleared. The deploy job runs, cannot find the .jar file, and fails.
• The Fix: Never use cache for files that are strictly required for subsequent stages to function. Use artifacts for mandatory intermediate files, and cache only for optional speed-up files.

Pitfall 3: Cache Overwrite Collisions

• The Issue: Multiple branches (e.g., feature-a and feature-b) run pipelines simultaneously. They use different versions of dependencies but share the same static cache key. They constantly overwrite each other's caches, leading to corrupted dependencies and build failures.
• The Fix: Use dynamic cache keys based on the branch name or lock files:
  YAML

  Copy

  cache:
    key:
      files:
        - package-lock.json # Recreates cache only when dependencies change
    paths:
      - node_modules/
  

Debugging Artifacts via the GitLab UI

If a downstream job fails because it is missing an artifact, you can verify if the artifact was created successfully by inspecting the upstream job:

1. Navigate to Build > Pipelines in your GitLab sidebar.
2. Click on the completed pipeline, then click on the specific build job (e.g., build_job).
3. On the right-hand sidebar, look for the Job artifacts section.
4. Click Keep (to prevent expiration), Download, or Browse to inspect the files inside the uploaded archive.

10. Summary and Key Takeaways

• Isolated Runner Environments: Every GitLab job runs in a clean, temporary container. Files do not persist between jobs or runs unless explicitly saved.
• Artifacts for Job Communication: Use artifacts to pass build outputs (like binaries, compiled code, or test reports) from one stage to subsequent stages in the same pipeline.
• Caches for Speed: Use caching to store reusable dependencies (like node_modules/ or .m2/) to accelerate future pipeline runs.
• Artifact Expiration: Always set an expiration time (e.g., expire_in: 1 hour or expire_in: 7 days) on artifacts to prevent your GitLab instance from running out of storage space.
• Cache is Optional: Design your pipelines with the assumption that the cache might be cleared at any time. The pipeline must still succeed even if it has to download dependencies from scratch.
• Dynamic Cache Keys: Use dynamic keys (such as $CI_COMMIT_REF_SLUG or dependency lock files) to prevent cache collisions across different branches.