GitLab CI/CD: Deploy Self-Hosted Runner (Docker Executor) on AWS

Deploying a self-hosted GitLab CI/CD Runner with a Docker Executor on an AWS EC2 instance is a crucial skill for production pipelines. While shell-based runners execute jobs directly on the host machine's operating system, Docker-based runners spin up isolated containers for every single job, ensuring a clean, reproducible, and sandboxed environment.

1. Why Do We Need a Docker Executor Runner?

The Problem with Shell-Based Runners

• Environmental Pollution: Every job executed on a shell runner leaves artifacts, installed dependencies, or modified configuration files on the host machine, potentially breaking subsequent builds.
• Dependency Conflicts: If Job A requires Python 3.8 and Job B requires Python 3.11, installing both globally on a single EC2 host causes conflicts.
• Lack of Isolation: Shell jobs have direct access to the host machine, which can pose security risks if an untrusted script or dependency executes malicious commands.

The Solution: Docker Executor

• Ephemeral Containers: Every CI/CD job runs inside its own fresh, isolated Docker container based on an image you specify.
• Consistent Environments: Guaranteed repeatability. A build running on your local machine will execute identically on the AWS EC2 runner because the environment is defined entirely by the Docker image.
• Tooling Flexibility: Different jobs can use completely different toolsets (e.g., Node.js, Ruby, Python, Java) without needing those tools pre-installed on the host EC2 instance.

2. Core Mental Model: How GitLab CI/CD Interacts with Docker on AWS

When a GitLab project triggers a pipeline, GitLab communicates with the self-hosted runner installed on your EC2 instance. If configured as a Docker Executor, the runner instructs the local Docker daemon on the EC2 instance to spin up a container, execute the job steps inside it, capture the logs, and destroy the container once finished.

GitLab Server (Cloud)
   |
   | polls for job / receives trigger
   v
EC2 Instance (Self-Hosted Runner)
   |
   | instructs Docker daemon
   v
Docker Daemon spawns isolated container (e.g., Ruby / Node.js)
   |
   | executes pipeline script steps
   v
Container destroys, logs returned to GitLab

3. High-Level Step-by-Step Architecture

To successfully deploy and register a Docker-based GitLab Runner on AWS, follow these four distinct phases:

4. Step 1: Launch and Connect to AWS EC2 Instance

First, provision a virtual machine in AWS EC2 where your runner will live.

1. Launch Instance: Go to the AWS EC2 Console and click Launch Instance.
2. Name & Tags: Name your instance (e.g., my-gitlab-runner).
3. AMI (OS Image): Select Ubuntu (Latest LTS version, Free Tier eligible, 64-bit x86 architecture).
4. Instance Type: Choose t2.micro or t3.micro (Free Tier eligible for demonstration purposes).
5. Key Pair: Select or create an SSH key pair (.pem or .ppk) so you can securely log into the instance.
6. Network Settings:
   • Choose your VPC and a Public Subnet (to allow outbound internet access for downloading packages and communicating with GitLab.com).
   • Configure Security Groups: Ensure Port 22 (SSH) is open (Source: Anywhere or your specific IP).
7. Connect via SSH: Once running, connect to your instance from your terminal:
   BASH

   Copy

   ssh -i "your-key-pair.pem" ubuntu@<your-ec2-public-ip>
   

5. Step 2: Install GitLab Runner on Ubuntu EC2

With your EC2 instance running and accessible via SSH, update the system packages and install the official GitLab Runner package.

Update System Packages

sudo apt-get update -y
sudo apt-get upgrade -y

Install GitLab Runner Package

Run the following official installation commands to add the GitLab repository and install the runner daemon:

# Download and add GitLab official repository
curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64

# Give it execution permissions
sudo chmod +x /usr/local/bin/gitlab-runner

# Create a dedicated gitlab-runner user
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash

# Install and run as service
sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
sudo gitlab-runner start

Note: Alternatively, you can use the official package repository manager (apt) for easier updates:

BASH

Copy

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
sudo apt-get install gitlab-runner -y

6. Step 3: Install and Configure Docker Engine on EC2

Because this runner uses the Docker Executor, Docker must be installed directly on the host EC2 instance so that containers can be spawned locally.

Run Installation Script / Commands

Execute the standard Docker installation sequence on Ubuntu:

# Remove any conflicting older packages
sudo apt-get remove docker docker-engine docker.io containerd runc -y

# Install prerequisite packages
sudo apt-get update -y
sudo apt-get install apt-transport-https ca-certificates curl gnupg lsb-release -y

# Add Docker’s official GPG key
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

# Set up the stable Docker repository
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker Engine and Containerd
sudo apt-get update -y
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin -y

Add GitLab Runner User to Docker Group

To ensure the gitlab-runner system user has permission to communicate with the Docker socket without requiring sudo privileges every time, add it to the docker group:

sudo usermod -aG docker gitlab-runner
sudo systemctl restart docker

Verify Docker Installation

Check if the Docker daemon is active and running properly:

sudo systemctl status docker
docker --version

Expected Output: Service status should show active (running) and return the installed Docker version.

7. Step 4: Register the Runner with Your GitLab Project

Registration connects your newly created AWS runner instance to your specific GitLab repository or group.

Navigate to GitLab Project Settings

1. Go to your repository on GitLab.
2. Navigate to Settings -> CI/CD -> Runners -> Click Expand.
3. Click New project runner.
4. Configure the runner settings:
   • Tags: Add a descriptive tag (e.g., docker-runner or docker-build).
   • Run untagged jobs: Check this box if you want this runner to automatically pick up jobs that do not explicitly specify a tag (making it act as a default runner for untagged tasks).
5. Click Create runner, which will reveal the registration command containing your unique authentication token.

Execute Registration Command on EC2

Run the registration command provided by GitLab in your EC2 terminal:

sudo gitlab-runner register

You will be prompted interactively for configuration details:

1. GitLab instance URL: Enter https://gitlab.com/ (or your self-hosted GitLab URL).
2. Registration token: Paste the token provided in the GitLab UI.
3. Description: Enter a friendly name (e.g., my-docker-runner).
4. Executor: Type docker (crucial step: do not use shell here).
5. Default Docker image: Enter a fallback image if jobs don't specify one (e.g., ruby:3.0 or node:18).

Verify Registration

Run a dry-run check to ensure the runner is responsive:

gitlab-runner run

Expected Output: The runner connects successfully to the GitLab coordinator and listens for jobs.

8. Step 5: Writing and Testing the GitLab CI/CD Pipeline (.gitlab-ci.yml)

To test both shell-based and Docker-based runners, define a multi-stage .gitlab-ci.yml configuration file in the root of your repository.

Sample .gitlab-ci.yml

stages:
  - compile
  - test

# Job 1: Runs on Shell-based Runner
compile_job:
  stage: compile
  tags:
    - shell-runner
  script:
    echo "Compiling code on EC2 Shell Executor..."
    echo "Current Node/Ruby version on host:"
    node -v || ruby -v

# Job 2: Runs on Docker-based Runner (Default Image)
test_job_1:
  stage: test
  tags:
    - docker-runner
  script:
    echo "Running tests inside default Docker container..."
    ruby -v

# Job 3: Runs on Docker-based Runner with a Custom Image
test_job_2:
  stage: test
  tags:
    - docker-runner
  image: node:18-alpine
  script:
    echo "Running tests inside custom Node 18 Alpine container..."
    node -v

9. Real-World Pitfalls & Debugging Common Runner Issues

When operating self-hosted runners in production on AWS, you may encounter specific operational edge cases:

1. Job Stuck in Pending State:
   • Cause: No active runner matches the tags specified in .gitlab-ci.yml, or the runner is offline.
   • Fix: Check sudo gitlab-runner status on the EC2 instance and verify that the tags in your YAML file match the runner tags registered in GitLab.
2. Permission Denied / Docker Socket Errors:
   • Cause: The gitlab-runner user does not have read/write access to /var/run/docker.sock.
   • Fix: Re-run sudo usermod -aG docker gitlab-runner and restart the GitLab Runner service (sudo systemctl restart gitlab-runner).
3. Shared vs. Custom Runner Selection Conflict:
   • Cause: If you have multiple runners registered with Run untagged jobs enabled, GitLab assigns jobs non-deterministically.
   • Fix: Always use explicit tags: in your .gitlab-ci.yml jobs to guarantee execution on the intended runner (Shell vs. Docker).

10. Summary and Key Takeaways

• Docker Executor Isolation: Using a Docker executor ensures that every CI/CD pipeline job runs inside a clean, sandboxed, and reproducible container environment, preventing host machine pollution.
• AWS EC2 Setup: Setting up a self-hosted runner requires provisioning an EC2 instance, installing the GitLab Runner binary, and installing the Docker Engine daemon.
• Privilege Management: The gitlab-runner system user must be added to the local docker group to securely spawn child containers.
• Interactive Registration: During gitlab-runner register, specifying docker as the executor and defining a default image establishes the core operational baseline.
• Tag-Based Routing: Proper tag management in .gitlab-ci.yml allows fine-grained control over whether a job executes on a lightweight shell runner or an isolated Docker container runner.