Day 9 | GitLab Runner Setup & Architecture

GitLab itself does not execute the jobs defined in your CI/CD pipelines; instead, it acts as an orchestrator. The actual execution of your pipeline scripts is offloaded to an agent known as a GitLab Runner. This guide covers the core architecture of GitLab Runners, compares runner types and executor environments, and provides a step-by-step practical guide to provisioning and configuring a self-hosted Docker-based runner on an AWS EC2 instance.

1. Why Do We Need Runners? (The Core Execution Engine)

In GitLab CI/CD, there is a strict separation of concerns between the Control Plane (GitLab Server) and the Execution Plane (GitLab Runner).

• GitLab Server: Hosts the repository, parses the .gitlab-ci.yml file, orchestrates pipeline stages, and displays logs and statuses.
• GitLab Runner: A lightweight, highly scalable agent installed on a separate machine (virtual machine, bare-metal server, or container) that polls GitLab for pending jobs, executes them, and streams the execution logs back to the server.

The Polling Mechanism Workflow

The runner operates on a pull-based architecture. It constantly pings the GitLab instance via API to check if there are jobs waiting to be processed.

2. Shared Runners vs. Self-Hosted Runners

When setting up pipelines, you must choose where your jobs will execute. GitLab offers two primary categories of runners:

Shared Runners (SaaS / Public)

Shared runners are managed directly by GitLab. They are automatically available to all projects (including free tier accounts) with zero configuration required.

• Pros: Zero maintenance, completely free up to monthly quota limits, ideal for small projects and rapid prototyping.
• Cons: Shared resources mean slower execution times, queue wait times during peak hours, and limited compute capacity. You also have zero host-level control (e.g., you cannot inspect the underlying workspace or access local files generated during builds).

Self-Hosted Runners (Specific / Private)

Self-hosted runners are machines that you provision, configure, and manage within your own infrastructure (e.g., AWS, GCP, Azure, or on-premises bare metal).

• Pros: Full control over host resources (CPU, RAM, storage), custom software pre-installation, faster execution, and enhanced security (runs within your private network).
• Cons: Requires manual setup, ongoing maintenance, OS patching, and infrastructure costs.

The Cloning Overhead Optimization

In a shared runner environment, every single job in a pipeline typically spins up on a completely different, clean virtual machine. This requires the runner to clone the entire repository from scratch for every single job.

For example, if a pipeline has 10 jobs and each clone operation takes 6 seconds:

On a self-hosted runner, all jobs can execute on the same persistent virtual machine. The repository is cloned once, and subsequent jobs perform fast incremental updates or reuse the existing workspace:

This saves significant pipeline execution time ($54\text{ seconds}$ saved in this scenario).

Comparison Matrix

3. Understanding Executor Types

An Executor is a setting inside the GitLab Runner configuration that defines the environment in which your pipeline scripts will run.

1. Shell Executor

The simplest executor. It runs pipeline commands directly inside the terminal of the host machine's Operating System (e.g., Bash on Ubuntu, PowerShell on Windows).

• Pros: Extremely fast startup; simple to understand.
• Cons: No isolation. If Job A installs a dependency or modifies a system file, it directly affects Job B. Running untrusted code on a shell executor poses severe security risks to the host machine.

2. Docker Executor

The industry standard for most CI/CD pipelines. The runner spins up a clean Docker container using a specified image (e.g., ubuntu:latest, node:20, maven:3.9), runs the job scripts inside that container, and destroys the container immediately after completion.

• Pros: Complete isolation; reproducible builds; clean environment for every job.
• Cons: Requires Docker to be installed and running on the host machine; slight startup overhead to pull images.

3. Kubernetes Executor

Designed for cloud-native, large-scale systems. The runner connects to a Kubernetes cluster and dynamically provisions a new Pod for each pipeline job, terminating the Pod upon job completion.

• Pros: Highly scalable; self-healing; optimal resource utilization across a cluster.
• Cons: Complex initial setup and cluster management overhead.

Executor Comparison

4. Scope of Runners

Runners can be registered at different levels of your GitLab hierarchy to control access and resource sharing:

• Project-Level Runners: Dedicated to a single, specific project. Useful for projects with unique build requirements or security constraints.
• Group-Level Runners: Shared across all projects within a specific GitLab Group or Subgroup. This reduces management overhead for teams managing multiple microservices.
• Instance-Level Runners (Shared): Available to every project across the entire GitLab instance (managed by system administrators).

5. Step-by-Step Infrastructure Provisioning (AWS EC2)

To set up a self-hosted runner, we must first provision a virtual machine. In this guide, we use an AWS EC2 instance running Ubuntu.

Step 1: Security Group Configuration

Ensure your EC2 Security Group allows outbound traffic to internet resources (so the runner can poll gitlab.com and download packages) and inbound SSH traffic for configuration.

• Inbound Rules:
  • Port 22 (SSH) allowed from your local IP address.
• Outbound Rules:
  • All traffic allowed (0.0.0.0/0).

Step 2: Launch the EC2 Instance

1. Navigate to the AWS EC2 Dashboard and click Launch Instance.
2. Name: gitlab-runner-vm
3. AMI: Ubuntu Server 24.04 LTS (or 22.04 LTS)
4. Instance Type: t2.medium (2 vCPUs, 4 GiB RAM recommended for stable Docker builds)
5. Key Pair: Create a new key pair. Select ED25519 as the key pair type.

   Why ED25519? ED25519 is a modern public-key signature system that offers higher security and faster performance with significantly smaller key sizes compared to traditional RSA keys.

6. Storage: Increase the root volume size to at least 20 GB to accommodate Docker images and build caches.
7. Click Launch Instance.

Step 3: Establish SSH Connection

Open your terminal or an SSH client (such as MobaXterm) to connect to the instance.

MobaXterm Optimization Tip: Go to Settings -> Configuration -> SSH and ensure SSH Keepalive is checked. This prevents your terminal session from dropping during long periods of inactivity or build execution.

# Set correct permissions on your downloaded private key
chmod 400 your-key.pem

# Connect to the EC2 instance
ssh -i "your-key.pem" ubuntu@<YOUR_EC2_PUBLIC_IP>

Once connected, update the local package index:

sudo apt update && sudo apt upgrade -y

6. Installing and Registering the GitLab Runner

With the host machine ready, we can now install the GitLab Runner agent and register it with our GitLab project.

Step 1: Create a Project Runner in GitLab UI

1. Navigate to your GitLab project.
2. Go to Settings -> CI/CD -> Expand Runners.
3. Click New project runner.
4. Tags: Enter self-hosted.

   Note: Tags are critical. They tell GitLab which jobs should be routed to this specific runner.

5. Click Create runner.
6. GitLab will display a registration command containing a unique Runner Token. Keep this page open.

Step 2: Install the GitLab Runner Binary on the VM

Execute the following commands on your Ubuntu EC2 instance to download and install the official GitLab Runner package:

# Download the GitLab Runner repository setup script
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash

# Install the gitlab-runner package
sudo apt-get install gitlab-runner -y

Step 3: Register the Runner with GitLab

Run the registration command to link your local runner agent with your GitLab project:

sudo gitlab-runner register

During the interactive prompt, provide the following details:

1. Enter the GitLab instance URL: Press Enter to accept the default (https://gitlab.com/) or enter your self-hosted GitLab URL.
2. Enter the registration token: Paste the token generated from the GitLab UI.
3. Enter a name for the runner: e.g., test-runner-12
4. Enter an executor: Type docker
5. Enter the default Docker image: Type ubuntu:latest

Upon successful registration, you will see:

Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

7. Installing Docker & Configuring Permissions

Since we selected the Docker Executor, the GitLab Runner agent must have permission to communicate with the Docker daemon on the host machine to spin up containers.

Step 1: Install Docker on the Host VM

Execute the official Docker installation commands:

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl -y
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update

# Install Docker packages
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y

Step 2: Configure User Group Permissions (Crucial Step)

By default, the Docker daemon binds to a Unix socket owned by the root user and members of the docker group. Because the GitLab Runner agent runs under its own system user account (gitlab-runner), it will fail to access Docker, resulting in permission denied errors during pipeline execution.

To resolve this, add both the gitlab-runner user and the default ubuntu user to the docker group:

# Add users to the docker group
sudo usermod -aG docker gitlab-runner
sudo usermod -aG docker ubuntu

# Apply group changes to the current terminal session
newgrp docker

# Restart the GitLab Runner service to apply the new group permissions
sudo systemctl restart gitlab-runner

Verify that the GitLab Runner service is active and running:

sudo systemctl status gitlab-runner

8. Configuring the CI/CD Pipeline (.gitlab-ci.yml)

To force your pipeline jobs to run on your newly configured self-hosted runner, you must specify the corresponding tag in your .gitlab-ci.yml file.

Example Configuration

Below is a multi-job pipeline configuration. Each job uses the tags parameter to target the self-hosted runner:

stages:
  - build
  - test

variables:
  DOCKER_IMAGE: "ubuntu:latest"

build_job:
  stage: build
  image: $DOCKER_IMAGE
  tags:
    - self-hosted
  script:
    - echo "Starting the compilation and build process..."
    - echo "Running inside an isolated Docker container managed by our self-hosted runner!"
    - hostname
    - cat /etc/os-release

test_job:
  stage: test
  image: $DOCKER_IMAGE
  tags:
    - self-hosted
  script:
    - echo "Executing unit and integration tests..."
    - echo "Test suite completed successfully."

9. Real-World Troubleshooting & Common Pitfalls

When setting up self-hosted runners for the first time, you may encounter common configuration bottlenecks. Use this reference guide to resolve them.

1. Jobs Stuck in "Pending" State

• Symptom: The pipeline is triggered, but jobs remain in a "Pending" state indefinitely with a message: "This job is stuck because the project doesn't have any active runners that can run this job."
• Causes & Solutions:
  1. Runner Service is Stopped: The runner agent is not running on the host VM.
     BASH

     Copy

     # Check status
     sudo gitlab-runner status
     # Start service if stopped
     sudo gitlab-runner start
     

  2. Tag Mismatch: The tags defined in .gitlab-ci.yml do not match the tags assigned to the runner in the GitLab UI. Ensure your job has tags: [self-hosted] and your runner has the self-hosted tag.
  3. Untagged Jobs Disabled: By default, runners only pick up jobs that match their tags. If your job has no tags, the runner will ignore it. To allow your runner to pick up untagged jobs, go to GitLab UI -> Runners -> Edit your runner -> Check Run untagged jobs -> Save.

2. Docker Permission Denied Errors

• Symptom: The job starts but immediately fails with an error similar to: dial unix /var/run/docker.sock: connect: permission denied or Is the docker daemon running?
• Cause: The gitlab-runner system user does not have permission to access the Docker socket.
• Solution:
  BASH

  Copy

  # Add user to group
  sudo usermod -aG docker gitlab-runner
  # Restart the runner service to reload group memberships
  sudo systemctl restart gitlab-runner
  

3. Runner Shows "Offline" in GitLab UI

• Symptom: The runner appears with a red dot and "Offline" status in the project settings.
• Cause: Network connectivity issues or the runner process crashed.
• Solution: Check host system logs to diagnose the crash:
  BASH

  Copy

  sudo journalctl -u gitlab-runner -n 50 --no-pager
  

10. Summary and Key Takeaways

• The Runner is the Engine: GitLab coordinates pipelines, but the GitLab Runner is the actual agent that executes the scripts.
• Shared Runners are Ephemeral: Shared runners are convenient and free but offer limited performance, lack customization, and suffer from cloning overhead across multiple jobs.
• Self-Hosted Runners Offer Control: Setting up your own runner on an EC2 instance provides full environment customization, faster builds, and persistent workspace caching.
• Executors Dictate the Environment: The Docker Executor is the industry standard, providing clean, isolated, and reproducible containerized environments for every job.
• Tags Route the Work: GitLab uses tags (e.g., self-hosted) to match pipeline jobs to the appropriate registered runners.
• Permissions are Critical: When using the Docker executor, the gitlab-runner system user must be added to the host's docker group to successfully spin up containers.