Day 17: DevOps Environments & GitLab CI/CD Pipeline Integration

Comprehensive study notes covering the theoretical foundations, structural architectures, GitLab configurations, and practical implementations of software environments (Dev, Staging, Production, and Dynamic Review Apps) within modern DevOps pipelines.

1. Why Do Environments Exist? (The Problem Statement)

In modern software development, pushing code changes directly from a developer's workstation into a live production environment without intermediate validation is a recipe for disaster.

Real-Life Analogy: The Food Delivery App Disaster

Imagine you are a developer at a major food delivery service like Zomato. You write code for a new user authentication feature, test it locally, and push it directly to production (developer -> production). Initially, the service appears stable. However, twenty minutes later, authentication sessions fail entirely, the core backend service crashes, and thousands of active users cannot place orders.

The Question: Could this bug have been caught before reaching actual users? Absolutely. This exact failure mode is why isolated environments exist as systematic checkpoints.

The Status Quo vs. The Safer Approach

• Naïve Approach (Developer -> Production): Zero checkpoints, immediate feedback loops for users (via crashes), high business risk, financial loss, and severe downtime.
• Mature DevOps Approach: Introduces sequential quality gates. Code flows through progressive validation tiers where each stage systematically increases confidence before reaching real users:
  Structure & Concept Flow

  Copy DiagramCopy

  Developer Workspace

  |

  v

  Dev Environment (Developer functional checks)

  |

  v

  Staging Environment (QA simulation & automated test suites)

  |

  v

  Production Environment (Live users & revenue generation)

   

2. Defining a DevOps Environment

A common misconception among beginners is that an "environment" is synonymous with a physical or virtual server.

Definition: An environment is a dedicated, isolated execution space where an application runs with its own distinct configuration, backing data (databases), network rules, and specific operational purpose.

To visualize this, think of three identical houses built from the same architectural blueprint:

1. House 1 (Dev Environment): Used for active construction, framing adjustments, and rough wiring tests by the builders.
2. House 2 (Staging Environment): Used for thorough code inspection, safety audits, and municipal code validation by the quality inspection team.
3. House 3 (Production Environment): The finished home where real residents live, operate, and interact daily.

3. Deep-Dive: The Three Static Environments

Static environments are permanent infrastructure tiers that always exist regardless of active feature branches.

A. Dev Environment

• Primary Purpose: A sandbox playground for developers to verify if newly written code executes properly.
• Characteristics: Highly unstable, frequent deployments, minimal manual gatekeeping.
• Key Verification: Database connections, basic API responses, and baseline syntax/compilation checks. Developers are not hunting for deep architectural bugs here; they are simply answering: "Does this code run?"

B. Staging Environment

• Primary Purpose: Quality Assurance (QA) validation and pre-release simulation.
• Characteristics: Engineered to be an exact replica of the production environment (sharing identical infrastructure configurations, database schemas, network policies, and OS versions).
• Key Verification:
  • Functional Testing: Verifying features meet business requirements.
  • Regression Testing: Ensuring new changes did not break legacy features.
  • UAT (User Acceptance Testing): Stakeholder review.
  • Performance & Load Checks: Simulating traffic spikes.
• If an application passes all criteria in Staging, high confidence is established that it will behave identically in Production.

C. Production Environment

• Primary Purpose: Serving live end-users, processing real payments, and storing core business data.
• Characteristics: The most protected and critical tier. Any mistake results in immediate financial and reputational loss.
• Key Constraints:
  • Strict role-based access control (RBAC)—not everyone can access it.
  • Changes must undergo formal approvals.
  • Deployments are strictly controlled and often require manual approval gates rather than pure automation.

4. The Critical Rule of Database Isolation

One of the most catastrophic failures in early engineering setups is sharing a single database across multiple environments.

Why Isolation is Mandatory

• Data Pollution: Test seed data in Dev corrupts financial metrics in Production.
• Accidental Wiping: Destructive queries (DROP TABLE, DELETE) run during developer testing will instantly wipe out live user records if environments share a database.
• Rule: Every environment (dev, staging, prod, and dynamic test environments) must maintain an entirely isolated database instance.

5. GitLab Environments & Static Configurations

GitLab tracks operational deployments by injecting metadata into pipelines via the environment keyword. This allows GitLab to answer three core auditing questions: Where was it deployed? When was it deployed? Who deployed it?

Basic Pipeline YAML with Static Environments

stages:
  - deploy

deploy-dev:
  stage: deploy
  script:
    - echo "Deploying application to Development..."
    - kubectl apply -f deployment.yaml
  environment:
    name: dev
    url: http://dev.myapp.local

deploy-prod:
  stage: deploy
  script:
    - echo "Deploying application to Production..."
    - kubectl apply -f deployment.yaml
  environment:
    name: production
    url: http://myapp.com
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      when: manual  # Production requires manual gatekeeping

Note on URLs: The url parameter inside a GitLab environment block does not magically provision an ingress or build a server. It is metadata that tells the GitLab UI where the application can be accessed post-deployment so engineers can click a direct link to test it.

6. Dynamic Environments & Review Apps

The Multi-Developer Branching Bottleneck

Imagine a team of 20 developers working simultaneously on features (feature/login, feature/payment, feature/cart). If all developers try to deploy their distinct, unmerged feature branches to a single shared staging environment, they will continuously overwrite each other's work, leading to endless code conflicts and blocked testing.

GitLab Review Apps

Review Apps are GitLab's native implementation of dynamic environments. They provide automated, temporary, and isolated environments for every single feature branch.

Workflow of a Review App

Implementing Review Apps & Auto-Stop in GitLab CI

review-app:
  stage: deploy
  script:
    - echo "Deploying review app for branch $CI_COMMIT_REF_SLUG"
    - kubectl apply -f deployment.yaml
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    url: http://$CI_COMMIT_REF_SLUG.myapp.local
    on_stop: stop-review-app
    auto_stop_in: 1 day

stop-review-app:
  stage: deploy
  script:
    - echo "Cleaning up review environment..."
    - kubectl delete -f deployment.yaml || true
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    action: stop
  rules:
    - when: manual

7. Environment-Scoped CI/CD Variables

Different environments require different configuration parameters (e.g., a Dev database connection string must never point to the Production database). GitLab solves this using Environment Scopes.

How Scoping Works

You can define a variable with the exact same key name (DATABASE_URL) multiple times in GitLab's CI/CD settings, binding each instance to a specific environment scope:

When a pipeline executes a job targeted at the production environment, GitLab automatically injects the DATABASE_URL scoped strictly to production, preventing configuration mix-ups.

8. Real-World Practical Project: AWS EKS & GitLab Deployment

To demonstrate these concepts in a production-grade infrastructure, we build an AWS Elastic Kubernetes Service (EKS) cluster using a dedicated connector virtual machine and integrate it with a multi-environment GitLab pipeline.

Step-by-Step Infrastructure Setup Workflow

1. IAM Configuration: Create an administrative IAM user (eksuser) in AWS and generate programmatic access keys (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY).
2. Connector VM Provisioning:
   • Launch an AWS EC2 instance (Ubuntu 24.04, T3.Medium instance type to avoid resource constraints).
   • Install required CLI utilities: AWS CLI, kubectl, and eksctl.
3. Cluster Provisioning:
   • Configure credentials on the connector VM via aws configure.
   • Run eksctl to provision the managed Kubernetes control plane and worker nodes:
     BASH

     Copy

     eksctl create cluster --name devops-cluster --region ap-south-1 --node-type t3.medium --nodes 2
     

   • Verify cluster connectivity:
     BASH

     Copy

     kubectl get nodes
     

4. GitLab CI/CD Variable Injection: Configure the required secure variables inside GitLab (Settings -> CI/CD -> Variables):
   • AWS_ACCESS_KEY_ID
   • AWS_SECRET_ACCESS_KEY
   • AWS_REGION (ap-south-1)

9. Real-World Troubleshooting & Production Debugging

The NodePort Exhaustion Bug

The Problem

During multi-environment deployments (dev, staging, prod) using Kubernetes NodePort services on the same cluster, pipelines began failing with errors stating that ports (e.g., default ports around 30000) were already allocated.

Root Cause Analysis

Kubernetes NodePorts bind globally across all cluster nodes. When deploying the exact same static manifest (deployment.yaml) across multiple namespaces (dev, staging, prod) simultaneously without explicitly dynamic port routing, Kubernetes attempts to bind the same fixed port twice, causing allocation conflicts.

The Resolution

1. Widen Security Group Ingress Range: Update the AWS EC2 Security Group inbound rules to support the entire default Kubernetes NodePort allocation range ($30000$ to $32767$):
   CODE

   Copy

   Port Range: 30000 - 32767
   Source: Anywhere (0.0.0.0/0)
   

2. Decouple Fixed Ports in Kubernetes Manifest: Remove explicit nodePort declarations from the YAML manifest, allowing Kubernetes to dynamically assign available ports from the valid range:
   YAML

   Copy

   apiVersion: v1
   kind: Service
   metadata:
     name: nginx-service
   spec:
     type: NodePort
     ports:
       - port: 80
         targetPort: 80
         # nodePort: 30008  <-- REMOVED to let Kubernetes auto-assign ports
     selector:
       app: nginx
   

10. Complete Production Pipeline Manifest

Below is the comprehensive, end-to-end GitLab CI pipeline combining static environments, namespaces, review apps, auto-stopping, and manual production gates.

stages:
  - deploy

variables:
  AWS_DEFAULT_REGION: "ap-south-1"

before_script:
  - aws configure set aws_access_key_id "$AWS_ACCESS_KEY_ID"
  - aws configure set aws_secret_access_key "$AWS_SECRET_ACCESS_KEY"
  - aws configure set region "$AWS_DEFAULT_REGION"
  - aws eks update-kubeconfig --name devops-cluster --region "$AWS_DEFAULT_REGION"

deploy-dev:
  stage: deploy
  script:
    - kubectl create namespace dev --dry-run=client -o yaml | kubectl apply -f -
    - kubectl apply -f deployment.yaml -n dev
  environment:
    name: dev
    url: http://<worker-node-ip>:30008
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'

deploy-staging:
  stage: deploy
  script:
    - kubectl create namespace staging --dry-run=client -o yaml | kubectl apply -f -
    - kubectl apply -f deployment.yaml -n staging
  environment:
    name: staging
    url: http://<worker-node-ip>:31190
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'

deploy-prod:
  stage: deploy
  script:
    - kubectl create namespace prod --dry-run=client -o yaml | kubectl apply -f -
    - kubectl apply -f deployment.yaml -n prod
  environment:
    name: production
    url: http://<worker-node-ip>:31414
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      when: manual

review-app:
  stage: deploy
  script:
    - kubectl create namespace review-$CI_COMMIT_REF_SLUG --dry-run=client -o yaml | kubectl apply -f -
    - kubectl apply -f deployment.yaml -n review-$CI_COMMIT_REF_SLUG
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    url: http://<worker-node-ip>:$REVIEW_PORT
    on_stop: stop-review-app
    auto_stop_in: 1 day
  rules:
    - if: '$CI_COMMIT_BRANCH != "main"'

stop-review-app:
  stage: deploy
  script:
    - kubectl delete namespace review-$CI_COMMIT_REF_SLUG --ignore-not-found=true
  environment:
    name: review/$CI_COMMIT_REF_SLUG
    action: stop
  rules:
    - if: '$CI_COMMIT_BRANCH != "main"'
      when: manual

Summary and Key Takeaways

• Environments as Checkpoints: Environments (Dev, Staging, Prod) act as progressive quality gates that systematically catch bugs and failures before code reaches live users.
• Isolation is Mandatory: Never share databases across environments to prevent accidental catastrophic data loss (e.g., executing accidental delete queries in shared spaces).
• Static vs. Dynamic Environments: Static environments (dev, staging, prod) are permanent infrastructure tiers; dynamic environments (Review Apps) are ephemeral, isolated spaces provisioned per feature branch.
• GitLab Environment Metadata: Using the environment: keyword allows GitLab to track deployment history, active URLs, and user accountability directly within the UI.
• Environment-Scoped Variables: Sensitive variables like database URLs and API keys must be scoped to specific environments to prevent staging or dev credentials from bleeding into production pipelines.
• Review Apps Enhance Collaboration: Review Apps give developers, QA engineers, and clients a dedicated URL to preview unmerged feature branches without causing merge conflicts on staging.
• Auto-Stop Efficiency: Configuring auto_stop_in ensures temporary review environments do not linger indefinitely and consume cloud infrastructure compute costs.
• Kubernetes Port Management: When deploying multiple replica applications across namespaces on a shared cluster, avoid hardcoding nodePort values to prevent port allocation collisions.
• Cluster Resource Cleanup: Always clean up and delete cloud Kubernetes clusters (eksctl delete cluster) post-testing to prevent runaway cloud provider billing.