GitLab CI/CD: Deploy Self-Hosted Runner (Shell Executor) on AWS

This comprehensive guide covers the transition from GitLab's default shared runners to a fully custom, self-hosted GitLab Runner using a Shell executor on an AWS EC2 instance. You will learn the security and architectural justifications for self-hosted runners, how to provision and configure an EC2 instance, install the GitLab Runner daemon, register the runner with your GitLab project, and route specific CI/CD jobs using pipeline tags.

1. Why Do We Need Self-Hosted Runners?

In GitLab CI/CD, pipeline jobs execute scripts (such as compilation, testing, and deployment) on external machines. Up to this point, jobs may have executed seamlessly without explicit runner configurations because GitLab provides Shared Runners out-of-the-box.

The Problem with Shared Runners in Production

• Security Risks: Shared runners are multi-tenant machines managed by GitLab. Your code is cloned onto infrastructure shared with other users and organizations, creating potential security vulnerabilities, intellectual property exposure, or code leakage.
• Network Isolation: Shared runners live in public infrastructure and cannot directly communicate with internal databases, private corporate applications, or secured staging environments behind corporate firewalls.
• Enterprise Standard: Real-world enterprise organizations and startups never rely on public shared runners for production pipelines. They deploy Self-Hosted (Self-Managed) Runners within their own cloud accounts or physical datacenters.

2. Core Mental Model & Architectural Workflow

A self-hosted runner acts as a dedicated worker agent sitting on a remote machine (such as an AWS EC2 instance) that constantly polls the GitLab server for pending jobs matching its registered capabilities.

GitLab Project (CI/CD Pipeline Triggered)
   |
   | 1. Dispatches job matching tag: "build-shell-runner"
   v
GitLab Server Queue
   |
   | 2. AWS EC2 Runner Agent polls and accepts job
   v
AWS EC2 Instance (Self-Hosted Runner)
   |
   | 3. Executes Shell commands locally (git clone, mvn compile, etc.)
   v
Pipeline Status Reported Back to GitLab Dashboard

3. Comparison of GitLab Runner Types and Executors

4. Step 1: Provisioning an AWS EC2 Instance

To host our custom runner, we provision an Amazon EC2 instance running Ubuntu. For simplicity in this tutorial, we place it in a public subnet, though enterprise architectures typically isolate runners inside private subnets using NAT Gateways.

EC2 Launch Configuration Steps:

1. Name: my-shell-runner
2. AMI (Operating System): Ubuntu (Latest LTS, 64-bit x86 architecture)
3. Instance Type: t2.micro (Eligible for AWS Free Tier)
4. Key Pair: Select an existing SSH key pair (or create a new one) to allow secure remote shell access.
5. Network Settings:
   • Subnet: Public Subnet (to allow direct outbound internet connectivity for package downloads and GitLab communication).
   • Auto-assign Public IP: Enabled.
6. Security Group:
   • Allow SSH (Port 22) from your IP (or 0.0.0.0/0 for demo simplicity).
7. Storage: Standard root EBS volume (default or slightly increased).

Connecting to the EC2 Instance via SSH:

# Set appropriate permissions for your private key
chmod 400 my-key-pair.pem

# Connect to the EC2 instance using its Public IPv4 address
ssh -i "my-key-pair.pem" ubuntu@<YOUR_EC2_PUBLIC_IP>

5. Step 2: Installing GitLab Runner on Ubuntu EC2

Once connected to your EC2 instance via terminal, update the system packages and install the official GitLab Runner package using the repository setup scripts provided by GitLab.

Execution Workflow on EC2 Terminal:

# 1. Update existing package lists
sudo apt-get update -y

# 2. Download and run the official GitLab Runner repository installation script
curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64

# Alternative official Debian/Ubuntu repository installation method:
curl -s https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash

# 3. Install the gitlab-runner package
sudo apt-get install gitlab-runner -y

6. Step 3: Registering the Runner with a GitLab Project

To link your newly created EC2 runner to a specific GitLab project, generate a registration token from the GitLab web interface and execute the interactive registration command on the EC2 terminal.

Obtaining Credentials from GitLab UI:

1. Navigate to your GitLab Project.
2. Go to Settings -> CI/CD -> Runners.
3. Expand the Runners section to view Project runners.
4. Click New project runner.
5. Assign tags (e.g., build-shell-runner) and click Create runner. Copy the generated runner authentication token.

Interactive Registration on EC2 Terminal:

Run the following command on your EC2 terminal:

sudo gitlab-runner register

Follow the interactive prompts in your terminal:

• GitLab instance URL: Enter https://gitlab.com/ (or your enterprise GitLab URL).
• Registration token: Paste the token copied from the GitLab project settings.
• Description for the runner: Enter my-shell-runners.
• Executor: Choose shell (this executes pipeline scripts directly on the host operating system terminal).

Note: If registration succeeds, the terminal will confirm successful registration, and the GitLab UI will instantly reflect the new active runner under your project settings.

Verifying the Registration:

Run the verification command on the EC2 instance to ensure active communication with GitLab:

sudo gitlab-runner verify

7. Step 4: Configuring and Routing Pipeline Jobs via Tags

By default, jobs without tags run on GitLab's shared runners. To route specific jobs exclusively to our newly created self-hosted shell runner, we must define tags in the .gitlab-ci.yml pipeline configuration file.

Naive vs. Tag-Routed Pipeline Configuration:

# --- Naive Pipeline (Runs on Default Shared Runners) ---
compile_job:
  stage: compile
  script:
    - echo "Compiling code..."
    - mvn clean compile

# --- Production-Ready Pipeline (Routed to Self-Hosted Shell Runner) ---
compile_job_self_hosted:
  stage: compile
  script:
    - echo "Compiling code on dedicated AWS EC2 runner..."
    - mvn clean compile
  tags:
    - build-shell-runner

8. Real-World Debugging & Execution Inspection

When observing pipeline execution in the GitLab UI:

1. Navigate to Build -> Pipelines -> Click on your running/completed pipeline.
2. Open the individual job logs (e.g., compile_job_self_hosted).
3. Observe the right-side panel:
   • Runner Details: Displays the custom runner description (my-shell-runners).
   • Executor Type: Confirms Using Shell executor....
   • Execution Environment: Proves the script executed directly on the AWS EC2 Linux OS terminal rather than a shared container.

9. Summary and Key Takeaways

• Self-Hosted Runners eliminate security risks associated with shared multi-tenant runners by executing code on dedicated infrastructure owned by your organization.
• Network Integration allows self-hosted runners (such as AWS EC2 instances inside a VPC) to securely connect with internal databases, private repositories, and corporate staging environments.
• Provisioning Flow involves launching an AWS EC2 instance, installing the gitlab-runner daemon package via official repository scripts, and registering the agent.
• The Shell Executor runs CI/CD script commands directly within the host machine's native terminal environment (Linux shell).
• Pipeline Tagging (tags: [build-shell-runner]) acts as a routing switch, directing specific pipeline jobs to execute on your custom self-hosted runner instead of default public runners.