GitHub Actions Artifacts & Caching: File Persistence & Build Optimization

GitHub Actions workflows execute inside isolated, ephemeral virtual machines (runners) that are completely destroyed upon job completion. To pass data between jobs, retain build deliverables, and avoid re-downloading identical dependencies on every run, GitHub Actions provides two distinct storage mechanisms: Artifacts and Caching.

1. Defining Software Artifacts vs. Runtime Outputs

In a Software Development Life Cycle (SDLC), any file generated during application compilation, testing, scanning, or packaging is considered an artifact. Broadening beyond simple compiled binaries, artifacts encompass every persistent deliverable produced across the end-to-end pipeline.

Examples of Workflow Artifacts

• Build Artifacts: application.jar, dist/, compiled Docker tarballs.
• Test & Coverage Reports: test-results.html, coverage_report.html, UI test execution recordings.
• Security & Compliance: securityscan.json, Software Bill of Materials (S-bomb.json), vulnerability logs.
• Operational Logs: deployment.log, environment provisioning traces.
• LLM / AI Pipelines: Benchmark evaluation reports, vector index dumps, RAG evaluation JSONs.

Distinguishing Outputs from Artifacts

A common point of confusion is the distinction between workflow outputs and workflow artifacts.

2. Ephemeral Runners & The Artifact Architecture

The Ephemeral Storage Problem

GitHub Actions runners provision a clean file system for every job execution. Once a job finishes, its runner is immediately decommissioned and destroyed.

If Job 1 (Build) creates application.jar, that file exists exclusively on Job 1's local runner. When Job 2 (Deploy) starts, it runs on an entirely new runner without access to Job 1's file system. Re-generating application.jar inside Job 2 wastes compute time, inflates execution costs, and delays pipeline feedback loops.

The Solution: GitHub-Managed Artifact Storage

To enable file sharing across jobs and workflows—as well as human access—GitHub Actions intercepts generated files and uploads them to an intermediate object storage layer before the runner is destroyed.

Conceptually: Under the hood, GitHub manages this intermediate storage using cloud object storage (such as Azure Blob Storage). This infrastructure layer is completely abstracted from the user, handling secure upload, download, authentication, and lifecycle expiration automatically.

3. Artifact Storage Lifecycle & Retention Rules

GitHub Actions artifacts are designed specifically for temporary pipeline persistence, not permanent binary hosting.

Key Lifecycle Principles

1. Machine-to-Machine Sharing: Primary use case. Allows decoupled pipeline jobs (e.g., Build -> Test -> Deploy) to pass heavy files downstream.
2. Human Inspection: Engineers can manually download zipped artifacts directly from the GitHub Actions UI run summary for auditing or debugging.
3. Retention Boundaries:
   • Workflow Default: Artifacts are retained for 90 days by default.
   • Workflow Configurable Limit: Can be adjusted down to 1 day or up to 90 days via retention-days in the YAML workflow.
   • Repository/Organization Upper Limit: Can be extended up to a maximum of 400 days within Repository Settings (Settings -> Actions -> General -> Artifact and log retention).

Production Pitfall: Artifact storage incurs cloud storage costs after free-tier quotas are exceeded. Never treat GitHub Artifacts as a perpetual binary repository. For permanent, versioned enterprise releases, upload deliverables to dedicated artifact registries such as AWS CodeArtifact, JFrog Artifactory, or Sonatype Nexus.

4. Walkthrough: Sharing Artifacts Across Jobs

The following hands-on workflow demonstrates generating smoke test logs and container output in a primary CI job (Flask CI Job), uploading them as a zipped artifact, and downloading them in a decoupled downstream job (Review Job).

Project Source Files

Application Code (app.py)

import os
from flask import Flask

app = Flask(__name__)

@app.route("/")
def home():
    print("home endpoint invoked")
    return "Cloud with Varun's Flask application started"

@app.route("/health")
def health():
    print("health endpoint invoked")
    return {"status": "healthy"}

if __name__ == "__main__":
    port = int(os.environ.get("PORT", 5000))
    app.run(host="0.0.0.0", port=port)

Dependencies (requirements.txt)

Flask==3.1.0
requests==2.32.3
pytest==8.3.4

Complete Workflow Definition (01-artifacts-demo.yaml)

name: 01-artifacts-demo

on:
  workflow_dispatch:

jobs:
  flask-ci-job:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Build Docker image
        uses: docker/build-push-action@v6
        with:
          context: .
          push: false
          tags: cwvj-flask-app:latest

      - name: Run container
        run: |
          docker run -d -p 5000:5000 --name cwvj-flask-container cwvj-flask-app:latest

      - name: Generate smoke test report
        run: |
          sleep 5
          curl http://localhost:5000/ > smoke-test-report.txt
          curl http://localhost:5000/health >> smoke-test-report.txt

      - name: Export container logs
        run: |
          docker logs cwvj-flask-container > container-logs.txt

      - name: Upload artifacts
        uses: actions/upload-artifact@v4
        with:
          name: Flask-CI-Artifacts
          path: |
            smoke-test-report.txt
            container-logs.txt
          retention-days: 7

  review-job:
    runs-on: ubuntu-latest
    needs: flask-ci-job
    steps:
      - name: Download artifacts
        uses: actions/download-artifact@v4
        with:
          name: Flask-CI-Artifacts

      - name: Review downloaded artifacts
        run: |
          echo "--- Smoke Test Report ---"
          cat smoke-test-report.txt
          echo "--- Container Logs ---"
          cat container-logs.txt

5. Contextual Deep-Dive: CI/CD Testing Sequences

When generating test artifacts, understanding where specific test suites belong within the delivery pipeline ensures efficient resource utilization.

Key Rule of Thumb: Fast-running suites (Unit, Integration, Smoke) execute on every commit push. Exhaustive suites (System, Regression, Acceptance) execute later in staging or pre-production pipelines due to execution duration.

6. Fundamentals of Dependency Caching

While artifacts preserve workflow-generated outputs, caching preserves reusable external dependencies and tool caches across pipeline runs.

Core Benefits of Caching

1. Reduces Execution Time: Bypasses network fetch and compilation steps for static packages.
2. Saves Network Bandwidth: Minimizes egress traffic toward public registries (PyPI, npm, Maven Central).
3. Decreases CI/CD Costs: Reduces total runner execution minutes.

Key Conceptual Comparison

7. Package Manager Cache Paths & Key Generation

Build tools maintain internal directories where downloaded binaries and dependencies are saved locally. GitHub Actions caching works by capturing these specific directories and restoring them on subsequent runs.

Standard Build Tool & Package Manager Cache Paths

Cache Key Mechanics: Hit vs. Miss

A cache key uniquely identifies a cache archive. Keys are generated dynamically using contextual functions such as hashFiles().

key: pip-cache-${{ runner.os }}-${{ hashFiles('requirements.txt') }}

8. Walkthrough: Implementing Pip Caching with actions/cache

The following workflow demonstrates explicit dependency caching using actions/cache@v4.

Caching Workflow (01-caching-demo.yaml)

name: 01-caching-demo

on:
  workflow_dispatch:

jobs:
  build-job:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.13'

      - name: Restore or create pip cache
        id: pip-cache
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: pip-cache-${{ runner.os }}-${{ hashFiles('requirements.txt') }}

      - name: Display cache status
        run: |
          echo "Cache Hit Status: '${{ steps.pip-cache.outputs.cache-hit }}'"

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

  test-job:
    runs-on: ubuntu-latest
    needs: build-job
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.13'

      - name: Restore pip cache
        id: pip-cache
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: pip-cache-${{ runner.os }}-${{ hashFiles('requirements.txt') }}

      - name: Display cache status
        run: |
          echo "Cache Hit Status: '${{ steps.pip-cache.outputs.cache-hit }}'"

      - name: Install dependencies
        run: |
          pip install -r requirements.txt

Execution Lifecycle Breakdown

1. First Execution (Initial Run):

   • build-job: Evaluates key pip-cache-Linux-<hash1>. Key does not exist (Cache Miss). outputs.cache-hit returns an empty string ''.
   • Dependencies are fetched from PyPI.
   • Post Restore or create pip cache automatically triggers at job completion, saving ~/.cache/pip under key pip-cache-Linux-<hash1>.
   • test-job: Evaluates key pip-cache-Linux-<hash1>. Matching key found (Cache Hit). outputs.cache-hit returns 'true'. Dependencies are restored instantaneously from cache.

2. Subsequent Runs (Unchanged requirements.txt):

   • Both build-job and test-job result in exact primary key matches (Cache Hit). Zero external PyPI requests are made.

3. Dependency Modification:

   • If a version in requirements.txt changes (e.g., Flask==3.1.0 updated to Flask==3.0.0), hashFiles('requirements.txt') produces <hash2>.
   • Evaluates key pip-cache-Linux-<hash2> (Cache Miss). Dependencies are redownloaded from PyPI and saved under the new key.

9. Advanced Caching Strategies: Fallback Keys (restore-keys)

The Exact Match Problem

When a single dependency version is updated in requirements.txt, the generated hash changes completely. Without fallback handling, an exact match failure forces a complete Cache Miss, downloading all dependencies from scratch—even those that did not change.

The Solution: Fallback Search via restore-keys

restore-keys provides a list of prefix fallback keys that GitHub Actions checks sequentially (top to bottom) if the primary key misses.

- name: Restore pip cache with fallbacks
  uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: pip-cache-${{ runner.os }}-${{ hashFiles('requirements.txt') }}
    restore-keys: |
      pip-cache-${{ runner.os }}-
      pip-cache-

Prefix Matching Order Rules

1. GitHub Actions attempts an exact lookup on key.
2. If missed, it evaluates restore-keys sequentially from top to bottom.
3. If multiple older caches match a prefix string (e.g., pip-cache-Linux-), GitHub Actions selects the most recently created cache.
4. During partial hits, existing dependencies are restored locally, meaning pip or maven only downloads the specific packages that were newly added or changed.

Summary and Key Takeaways

• Artifacts vs. Caching Purpose: Artifacts store build-generated outputs (.jar, test logs) for downstream jobs or human download; caches preserve reusable static dependencies (~/.cache/pip, ~/.m2) to speed up future runs.
• Ephemeral Runner Isolation: GitHub Actions runners are completely destroyed after job completion. Storage abstractions are necessary to persist data across jobs or workflows.
• Outputs are Text, Artifacts are Files: Workflow outputs handle lightweight key-value metadata strings (URLs, versions); artifacts handle complete files and directories.
• Artifact Storage Layer: Artifacts are uploaded out-of-band to managed object storage (Azure Blob abstraction) and default to a 90-day retention period (configurable between 1–90 days per workflow, up to 400 days at repository level).
• Temporary Nature of Actions Artifacts: Artifacts should never replace production artifact registries (such as AWS CodeArtifact or Artifactory) for long-term versioned binary hosting.
• Cache Key Integrity: Cache keys leverage dynamic cryptographic hashes via hashFiles('manifest_file'). A change in the file changes the hash, invalidating the primary key.
• Cache Hit vs. Miss Dynamics: An exact primary key match yields a cache hit (cache-hit = 'true'); an missing key yields a cache miss (cache-hit = '').
• Optimized Re-downloads via restore-keys: Utilizing prefix fallbacks in restore-keys prevents full re-downloads during minor dependency updates by restoring partial dependency trees.