Day 22: HashiCorp Vault in GitLab CI/CD

Master production-grade secret management by deploying an external HashiCorp Vault server on AWS EC2, configuring secure programmatic authentication, and integrating dynamic credential retrieval inside GitLab CI/CD pipelines.

1. Why Do We Need External Secret Management?

In any DevOps or software engineering project, applications rely heavily on sensitive data to function. Hardcoding these credentials directly into source code repositories or exposing them in plain text within CI/CD pipeline execution logs creates severe security vulnerabilities.

Common Sensitive Data Types

• Cloud Provider Keys: AWS Access Keys (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY).
• Database Credentials: Production database usernames, master passwords, and connection strings.
• API Tokens: Third-party service tokens, webhook secrets, and OAuth client secrets.
• Cryptography & Access: SSH private keys, TLS/SSL certificates, and Kubernetes kubeconfig files.

The Dangers of Hardcoding Secrets

2. Core Mental Model: GitLab Native Variables vs. External Vault

When deciding how to manage sensitive data, engineering teams typically choose between two primary approaches:

3. Deep Dive: GitLab CI/CD Native Variables & Their Limitations

GitLab offers robust native features to secure variables within project settings (Settings -> CI/CD -> Variables), but they come with specific configuration toggles and inherent boundaries.

Powerful Native Security Features

1. Masked Variables: When enabled, GitLab automatically replaces the sensitive string value with stars (*****) in all pipeline execution logs, preventing accidental leakage.
2. Protected Variables: Restricts the variable's availability so it can only be injected into pipelines running on protected branches (e.g., main or release branches).

Limitations of Native GitLab Variables

• Centralized Platform Lock-in: Your sensitive material remains bound strictly within GitLab's database.
• Scaling Constraints: Managing hundreds of microservices across multiple cloud environments with static GitLab variables becomes unmaintainable.

4. HashiCorp Vault Architecture & Features

To overcome native limitations, production environments leverage HashiCorp Vault, a centralized secrets management system built specifically to control access to sensitive data securely.

Core Vault Capabilities

• Encrypted Storage: Secrets are encrypted at rest and in transit, ensuring unauthorized parties cannot read raw database files.
• Dynamic Credentials: Vault can generate on-demand, temporary credentials (e.g., creating a temporary AWS IAM user or database login) that automatically expire after a set TTL (Time-To-Live).
• Automatic Secret Rotation: Integrates with backend systems to automatically rotate credentials without requiring application downtime.
• Granular Access Control: Uses token-based and policy-based authorization to strictly define who or what can read specific paths.

5. Infrastructure Setup: Deploying Vault on AWS EC2

Because GitLab CI/CD pipelines run on cloud-hosted runners, hosting HashiCorp Vault strictly on local loopback (localhost) prevents external pipeline access. Vault must be deployed on an instance with a public IP address or accessible network routing.

Step 1: Launching the EC2 Instance

• Name: Vault-Server
• AMI: Ubuntu 24.04 LTS
• Instance Type: t3.medium (Recommended minimum for memory stability)
• Security Group Configuration:
  • Open SSH (Port 22)
  • Open Vault HTTP API (Port 8200) — Crucial for client/pipeline communication

Step 2: Installing Docker on Ubuntu

Connect to your EC2 instance via SSH and execute the following commands to install Docker and grant execution permissions to the ubuntu user:

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Enable non-root Docker execution for user 'ubuntu'
sudo usermod -aG docker ubuntu
newgrp docker

Step 3: Running Vault Container in Dev Mode

Run the HashiCorp Vault container in development mode (dev), passing the appropriate environment variables and memory lock capabilities:

docker run -d \
  --name=vault \
  -p 8200:8200 \
  --cap-add=IPC_LOCK \
  -e 'VAULT_DEV_ROOT_TOKEN_ID=root' \
  -e 'VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200' \
  hashicorp/vault:latest

Note: --cap-add=IPC_LOCK prevents Linux from swapping Vault's memory (containing unencrypted secrets) to disk, keeping sensitive data strictly in RAM.

Verify the container status:

docker ps
docker logs vault

6. Configuring and Testing Vault CLI

To interact with your Vault server from the command line, export your Vault connection environment variables and install the Vault CLI binary.

Exporting Environment Variables

Replace <PUBLIC_IP> with your EC2 instance's public IP address:

export VAULT_ADDR='http://<PUBLIC_IP>:8200'
export VAULT_TOKEN='root'

Verifying Server Health via Curl

curl -s http://<PUBLIC_IP>:8200/v1/sys/health | jq .

Expected output snippet: {"initialized": true, "sealed": false, ...}

Installing Vault CLI on EC2

sudo apt-get install -y wget gpg
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt-get update && sudo apt-get install -y vault

# Verify installation version
vault version

7. Writing and Reading Secrets in Vault

HashiCorp Vault organizes secrets using Key-Value (KV) version secrets engines.

Writing Secrets

Store a set of test credentials under the secret path secret/data/my-app:

vault kv put secret/my-app username="admin_user" password="SuperSecretPassword123"

Reading Secrets

Retrieve the stored credentials back from Vault to verify success:

vault kv get secret/my-app

8. Integrating HashiCorp Vault with GitLab CI/CD

To allow your GitLab pipeline to fetch secrets from your external Vault server, configure GitLab project variables first.

Step 1: Configure GitLab Project Variables

Navigate to your GitLab repository:
Settings -> CI/CD -> Variables -> Click Add variable:

1. Key: VAULT_ADDR | Value: http://<EC2_PUBLIC_IP>:8200 (Visible)
2. Key: VAULT_TOKEN | Value: root (Visible)

9. Step-by-Step Evolution of the CI/CD Pipeline

We will build two pipeline iterations: a basic verification pipeline, followed by a robust production-style pipeline that securely parses JSON secrets into environment artifacts.

Approach 1: Basic Secret Retrieval Pipeline (.gitlab-ci.yml)

This simple pipeline uses an official image containing the Vault CLI to verify direct connectivity and credential printing.

stages:
  - secrets

fetch_secrets_job:
  stage: secrets
  image: hashicorp/vault:latest
  script:
    - echo "Connecting to Vault at $VAULT_ADDR"
    - vault login -method=token $VAULT_TOKEN
    - vault kv get secret/my-app

Approach 2: Production-Grade Pipeline with Artifact Exporting

In real-world enterprise environments, jobs require raw tools (curl, jq, unzip), robust API parsing, and structured environment variable handoffs between pipeline stages (secrets -> build -> deploy).

Create or update your .gitlab-ci.yml:

stages:
  - secrets
  - build
  - deploy

variables:
  VAULT_SECRET_PATH: "secret/data/my-app"

fetch_secrets:
  stage: secrets
  image: alpine:latest
  before_script:
    - apk add --no-cache curl jq unzip
    - |
      echo "Installing Vault CLI..."
      wget https://releases.hashicorp.com/vault/1.15.2/vault_1.15.2_linux_amd64.zip
      unzip vault_1.15.2_linux_amd64.zip
      mv vault /usr/local/bin/
      vault version
  script:
    - echo "Fetching secrets from Vault server..."
    - >
      curl -s --header "X-Vault-Token: $VAULT_TOKEN" \
      "$VAULT_ADDR/v1/$VAULT_SECRET_PATH" > secrets.json
    
    # Parse JSON using jq to extract specific keys
    - USERNAME=$(jq -r '.data.data.username' secrets.json)
    - PASSWORD=$(jq -r '.data.data.password' secrets.json)
    
    # Output into a secure environment file for downstream stages
    - echo "DB_USER=$USERNAME" >> secrets.env
    - echo "DB_PASS=$PASSWORD" >> secrets.env
    - cat secrets.env
  artifacts:
    reports:
      dotenv: secrets.env

build_application:
  stage: build
  image: alpine:latest
  dependencies:
    - fetch_secrets
  script:
    - echo "Building application using fetched credentials..."
    - echo "Using Username -> $DB_USER"
    # Password remains masked/secure in environment context
    - echo "Build completed successfully!"

deploy_application:
  stage: deploy
  image: alpine:latest
  dependencies:
    - fetch_secrets
  script:
    - echo "Deploying application to production target..."
    - echo "Deployment verified for user $DB_USER"

10. Complete Flow Walkthrough

11. Summary and Key Takeaways

• Security Risks: Hardcoding passwords, AWS keys, or API tokens directly inside Git repositories or pipeline logs compromises security and exposes systems to breaches.
• Alternative Approaches: Native GitLab CI/CD variables offer built-in masking and branch protection, but external managers like HashiCorp Vault are required for large-scale enterprise deployments.
• Vault Advantages: HashiCorp Vault provides centralized secret encryption, automatic secret rotation, and dynamic temporary credentials with strict Time-To-Live (TTL) limits.
• Infrastructure Requirements: When integrating Vault with cloud GitLab runners, the Vault server must be hosted on an externally accessible machine (such as an AWS EC2 instance) with open API ports (e.g., 8200).
• Memory Protection: Running Vault with Docker capability flags like --cap-add=IPC_LOCK keeps secrets safely stored in RAM, preventing sensitive data from swapping to disk storage.
• CLI Interaction: The Vault CLI allows developers and automation pipelines to write (vault kv put) and read (vault kv get) secrets securely from designated paths.
• Pipeline Integration: Production pipelines utilize lightweight Linux runner images (alpine), install parser utilities (curl, jq), query Vault via REST APIs, and export parsed values safely using GitLab dotenv artifacts.