Day 10 | Variables & Secrets | 30 Days of GitLab DevOps

Welcome to Day 10 of the 30 Days of GitLab DevOps course. This comprehensive guide covers Variables and Secrets in GitLab CI/CD—one of the most crucial sections for building secure, production-ready, and flexible pipelines.

1. Why Do We Need CI/CD Variables?

In GitLab pipelines, individual jobs run inside temporary, isolated environments. A common anti-pattern among beginners is hardcoding sensitive values, configuration settings, or credentials directly into pipeline configuration files (.gitlab-ci.yml).

The Dangers of Hardcoding Credentials

When values such as database passwords, API keys, or Docker registry credentials are hardcoded directly into scripts:

• Credential Leaks in Logs: Even if masked or hidden in configuration files, execution logs often echo commands that expose plaintext secrets to anyone with view access to the pipeline.
• Environment Rigidity: Hardcoded configurations tie a pipeline to a single target environment (e.g., hardcoding a development database URL prevents the same pipeline from deploying to staging or production).
• Maintenance Nightmares: Changing a password requires modifying code across multiple pipeline definitions rather than updating a single centralized secret store.

# BAD PRACTICE: Hardcoding secrets directly in pipeline scripts
stages:
  - deploy

deploy_job:
  stage: deploy
  script:
    - docker login -u admin -p "SuperSecretPassword123!" registry.example.com
    - ./deploy.sh

2. Core Mental Model & Architecture

To eliminate hardcoded values, variables allow you to store data outside of your pipeline code and inject them securely at runtime.

Real-Life Analogy: Environment Variables as Secure Key-Value Lockboxes

Think of a GitLab CI/CD variable as a secure, labeled lockbox stored in a vault outside your house (the pipeline script).

• When your application (the pipeline job) runs, it does not keep the house keys in its pocket where anyone can steal them.
• Instead, it reaches into the vault using a secure lookup handle (e.g., $DB_PASSWORD), and the system dynamically unlocks and injects the value precisely when execution demands it, keeping the actual secret concealed from logs and source control.

3. What is a CI Variable?

A CI Variable is a key-value pair injected into your pipeline environment.

• Key: The identifier used to reference the variable (e.g., DB_USER).
• Value: The underlying data assigned to that key (e.g., admin).

Using Variables in Pipeline Scripts

To reference a variable inside a .gitlab-ci.yml script, prefix the key name with a dollar sign ($). GitLab automatically resolves and replaces it with the underlying value during execution.

stages:
  - test

test_job:
  stage: test
  script:
    - echo "Connecting to database with user: $DB_USER"

4. Types of GitLab Variables

GitLab categorizes variables into two primary groups based on their origin and management:

1. Predefined (Built-in) Variables: Automatically populated by GitLab when a pipeline runs.
2. Custom (User-Defined) Variables: Created explicitly by developers or administrators for specific operational needs.

Predefined Variables Reference

GitLab provides dozens of built-in variables out of the box. Common examples include:

5. Custom Variables & Scope Levels

Custom variables are configured via the GitLab UI by navigating to:

Project -> Settings -> CI/CD -> Variables -> Add Variable

Depending on organizational structure and access controls, variables can be defined at three distinct scopes:

• Project Level: Usable across all pipelines belonging to that specific project repository.
• Group Level: Inherited and shared across multiple projects nested under a common GitLab group or subgroup.
• Pipeline Level: Injected manually or via API parameters during pipeline triggering, existing only for the duration of that specific run.

6. Securing Pipelines: Masked & Protected Variables

When dealing with sensitive data (passwords, tokens, API keys), standard variables are insufficient. GitLab provides two critical security modifiers:

A. Masked Variables

• Purpose: Prevents sensitive variable values from appearing in job logs. If a job accidentally or intentionally prints a masked variable (e.g., via echo $API_PASSWORD), GitLab replaces the value with asterisks ([MASKED]).
• Strict Constraints:
  • Must be at least 8 characters long.
  • Must not contain spaces.
  • Must consist of valid characters supported by GitLab's regex masking engine.

B. Protected Variables

• Purpose: Restricts the availability of a variable exclusively to protected branches (such as main or production).
• Security Impact: Prevents malicious or compromised feature branches (e.g., a developer creating an untrusted branch xyz and running echo $DB_PASSWORD) from reading production secrets.

7. Variable Precedence Hierarchy

If a variable with the exact same name is defined across multiple scopes, GitLab resolves conflicts using a strict precedence hierarchy. Variables higher on the list override those lower down:

1. Pipeline-level variables (Highest Priority)
2. CI/CD variables set in project settings
3. CI/CD variables set in group settings
4. Predefined / Built-in variables (Lowest Priority)

8. Practical Step-by-Step Laboratory Implementation

Follow this hands-on workflow to implement and test variables, masking, protection, and predefined attributes in a live GitLab repository.

Lab 1: Creating Basic CI Variables

1. Navigate to your GitLab project -> Settings -> CI/CD -> Variables -> Add variable.
2. Create the first variable:
   • Key: APP_NAME
   • Value: my-app
   • Flags: Leave defaults (Visible, Unprotected).
3. Create a second variable:
   • Key: ENVIRONMENT
   • Value: dev
4. Update your .gitlab-ci.yml pipeline configuration via Build -> Pipeline Editor:

stages:
  - test

variables:
  GLOBAL_VAR: "default-value"

test_variables_job:
  stage: test
  script:
    - echo "Application Name is $APP_NAME"
    - echo "Target Environment is $ENVIRONMENT"

5. Commit changes to main and verify pipeline logs show the injected values successfully.

Lab 2: Enforcing Variable Masking

1. Return to Settings -> CI/CD -> Variables.
2. Edit APP_NAME and ENVIRONMENT, ensuring their values meet the 8-character minimum requirement (e.g., update ENVIRONMENT value to dev-env-2024).
3. Check the Mask variable checkbox for both entries and save.
4. Trigger a new pipeline run and inspect the execution logs for test_variables_job.
   • Expected Output: The printed values will appear as [MASKED] rather than plaintext.

Lab 3: Implementing Manual Jobs

To control when specific validation or deployment steps run, use the when: manual keyword.

stages:
  - build
  - test

build_job:
  stage: build
  script:
    - echo "Building application artifact..."

unit_test_job:
  stage: test
  when: manual
  script:
    - echo "Running manual QA test suite..."
    - echo "App: $APP_NAME running in $ENVIRONMENT"

• Behavior: When you commit changes, the build_job runs automatically. The unit_test_job pauses, presenting a Play button in the GitLab pipeline UI, waiting for explicit manual approval before executing.

Lab 4: Testing Protected Variables & Branch Security

1. Create a new secure variable in project settings:
   • Key: DB_PASSWORD
   • Value: ProdSecurePass99!
   • Flags: Check Protect variable.
2. Update .gitlab-ci.yml to print $DB_PASSWORD.
3. Commit and run the pipeline on the main branch (which is protected by default). Notice that the password prints successfully.
4. Create a new branch named develop from the repository page.
5. Switch to the develop branch in the Pipeline Editor, commit a minor comment change, and trigger a pipeline.
6. Check the job logs on the develop branch.
   • Expected Output: $DB_PASSWORD evaluates to an empty string because protected variables are inaccessible on non-protected branches.

Lab 5: Utilizing Predefined Variables & Real Secret Usage (cURL Integration)

In production workflows, secrets are rarely echoed to logs; instead, they are passed as secure parameters into automation utilities like curl or SDK clients.

stages:
  - test

real_world_secret_test:
  stage: test
  script:
    - echo "Running pipeline ID: $CI_PIPELINE_ID on branch: $CI_COMMIT_REF_NAME"
    - echo "Project: $CI_PROJECT_NAME"
    - echo "Connecting to secure API..."
    - curl -u "$API_USER:$API_PASS" https://httpbin.org/basic-auth/user/passwd

Summary and Key Takeaways

• Avoid Hardcoding: Never store passwords, tokens, or configuration keys directly inside .gitlab-ci.yml files to prevent security breaches and environment rigidity.
• Centralized Management: Store variables securely at the Project, Group, or Pipeline scope via Settings -> CI/CD -> Variables.
• Masking Protection: Use Masked variables (minimum 8 characters, no spaces) to automatically redact sensitive strings from public pipeline logs with [MASKED].
• Branch Security: Use Protected variables to ensure sensitive production credentials are never leaked or accessed by feature or development branches.
• Predefined Context: Leverage built-in variables (CI_COMMIT_REF_NAME, CI_PIPELINE_ID, CI_PROJECT_NAME) to inject dynamic runtime context into build scripts.
• Variable Precedence: Pipeline-level variables take highest precedence, overriding project-level, group-level, and predefined variables in that order.
• Manual Control: Combine variables with when: manual pipelines to gate production deployments behind human verification.