Day 2: What is CI/CD? | CI/CD Explained | How Pipelines Work in DevOps

This chapter explores the foundational mechanics of modern software delivery, detailing how development teams collaborate using Git and transition code through automated pipelines. It contrasts the industry's two primary branching patterns—Git Flow and Trunk-Based Development—and establishes the core architectural principle of building once and promoting by immutable image digest. Finally, it positions Jenkins as the central orchestration engine that automates this entire lifecycle from commit to production release.

1. Git Essentials for CI/CD Collaboration

Git is a Distributed Version Control System (DVCS) designed to track changes in source code over time. Unlike centralized version control systems, Git provides every developer with a complete local copy of the repository's history on their machine.

Note: In the lecture transcript, the term "comet" is used phonetically. Conceptually and technically, this refers to a commit, which is a local snapshot of code changes at a specific juncture in the development phase.

Why Distributed Version Control Matters

• Local Autonomy: Developers can commit, revert, and branch locally without network dependency or fear of breaking the shared codebase.
• Collaboration and Review: A remote repository acts as a highly available, centralized source of truth where developers push their local changes to share them with the team.
• History Traversal: If a local test fails, Git allows developers to instantly traverse back to any previous stable commit, eliminating the need to manually maintain multiple versions of files (e.g., app_v1.js, app_v2_final.js).

Core Git Collaboration Workflow

• git clone: Executed by a new team member to download the entire remote repository, including its history, to their local machine.
• git pull: Fetches and merges the latest changes from the remote repository to keep the local copy up to date.
• git push: Sends local commits to the remote repository so other collaborators can access them.

2. Git Repository Hosting & High Availability

Because local development environments are prone to hardware failures and are not highly available, teams rely on dedicated hosting platforms to manage their remote Git repositories. These platforms offer Service Level Agreements (SLAs) and robust High Availability (HA) configurations.

Note on Legacy Services: Cloud-native Git hosting services like AWS CodeCommit and GCP's Cloud Source Repositories (CSR) are increasingly considered legacy options for new teams. Modern engineering organizations favor feature-rich platforms like GitHub or GitLab to leverage advanced collaboration and security capabilities.

3. Branches, Pull Requests, and Branch Protection

A branch in Git represents an independent, named line of development. It allows developers to isolate their work from the main codebase to build features, fix bugs, or run experiments.

Key Branching Concepts

• Main Branch: Often named main (historically master). It represents the stable source of truth for the application.
• Feature Branches: Short-lived branches created from main to implement specific changes (e.g., a topup-loan branch for a banking application). Multiple developers can collaborate on a single feature branch.
• Pull Request (PR) / Merge Request (MR): Terms used interchangeably (GitHub uses PR; GitLab uses MR). A PR/MR is a formal request to merge changes from a feature branch into a target branch (such as main or dev). It serves as the primary forum for code review, discussion, and automated testing.

Protected Branches and Status Gates

To maintain the integrity of critical branches (like main or dev), teams configure Branch Protection Rules. These rules enforce strict quality gates before code can be merged:

1. No Direct Pushes: Developers cannot push commits directly to protected branches; they must submit a PR/MR.
2. Mandatory Reviews: Requires at least one (often more) peer review and approval before merging.
3. Status Checks Must Pass: Automated Continuous Integration (CI) pipelines must execute successfully (returning a "green" status) on the PR branch.
4. Up-to-Date Requirements: The feature branch must be updated with the latest commits from the target branch before merging to prevent integration conflicts.

4. The Continuous Practices in DevOps (10,000-Foot View)

DevOps relies on a suite of continuous practices that span the entire Software Development Lifecycle (SDLC). While these practices are logically distinct, they run concurrently to form a cohesive delivery pipeline.

5. Branching Strategies: Git Flow vs. Trunk-Based

The choice of branching strategy dictates how code moves from a developer's laptop to production. The two dominant industry patterns are Git Flow and Trunk-Based Development.

6. The Core Principle: Build Once, Promote by Digest

A fundamental anti-pattern in CI/CD is rebuilding application artifacts (such as jar files or Docker images) for each target environment. Rebuilding introduces environmental drift, where the code tested in staging is not identical to the code deployed in production.

The Golden Rule of CI/CD: Build your artifact once during the initial CI phase, and promote that exact same artifact through all subsequent environments (Dev $\rightarrow$ Stage $\rightarrow$ Prod).

Tagging Conventions vs. Immutable Digests

When working with container images, teams apply various tagging strategies to balance human readability with technical precision:

1. Commit Tag: Named after the Git commit hash (e.g., 777yt99). This allows developers and SREs to map a running container directly back to the exact line of code that produced it.
2. Version Tag: Follows Semantic Versioning (e.g., v1.44.3). This is used by QA, product owners, and support teams to track releases and generate change logs.
3. Combined Tag: Combines both elements (e.g., v1.44.3-777yt99) to provide complete context at a single glance.

# Building a single Docker image with multiple tags
docker build \
  -t cloudwithvarun/loans:777yt99 \
  -t cloudwithvarun/loans:v1.44.3 \
  -t cloudwithvarun/loans:v1.44.3-777yt99 .

# Pushing all tags to Docker Hub
docker push -a cloudwithvarun/loans

While tags are highly readable, they are mutable. If tag immutability is not enforced in the container registry, a user can overwrite an existing tag (e.g., pushing a new image to v1.44.3).

To guarantee absolute immutability in production, enterprise pipelines deploy images using their SHA-256 Digest rather than a tag. The digest is a unique cryptographic hash of the image's contents. If even a single byte of the image changes, the digest changes.

The Deployment Manifest (manifest.json)

To track which artifact digest is currently active, the CI pipeline generates a metadata file (often called manifest.json) during the initial build. This file is stored in a shared location (such as an S3 bucket or Git repository) and acts as the single source of truth for downstream deployment jobs.

{
  "timestamp": "2023-10-27T10:00:00Z",
  "repository": "cloudwithvarun/loans",
  "commit_sha": "777yt99",
  "version": "v1.44.3",
  "digest": "sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",
  "tags": [
    "777yt99",
    "v1.44.3",
    "v1.44.3-777yt99"
  ]
}

7. Git Flow Walkthrough: Feature to Dev

In a Git Flow-based architecture, the first transition occurs when a developer moves code from a local feature branch into the shared development (dev) branch.

Why Merge Before Deploy?

In the Feature-to-Dev phase, the deployment to the Dev environment occurs after the PR is merged.

• Fast Feedback: The primary goal is to integrate code from multiple developers into a single shared environment as quickly as possible.
• Avoid Merge Queues: Forcing developers to wait for a full deployment to succeed before merging would create bottlenecks, blocking other team members.
• Fail Fast, Recover Faster: The Dev environment is designed to be highly dynamic and tolerant of transient failures. If a deployment fails post-merge, the developer simply pushes a fix directly to dev.

8. Git Flow Walkthrough: Dev to Stage (Promotion PR)

Promoting code from the development branch to the staging environment (stage) requires a more rigorous workflow. Staging serves as the pre-production environment and must remain highly stable.

Why Deploy Before Merge?

Unlike the Feature-to-Dev flow, the promotion to Staging executes the deployment before the PR is merged.

• Branch Cleanliness: The stage branch must always reflect a verified, working state of the application.
• Zero Tolerance for Broken Deployments: If the deployment or post-deploy smoke tests fail, the PR remains unmerged, keeping the stage branch clean and unaffected by the broken code.

Deployment Execution Example

During the deployment phase, Jenkins reads the immutable digest from the manifest.json file and uses a package manager like Helm to update the Kubernetes cluster:

# Extract the immutable digest from the manifest
IMAGE_DIGEST=$(jq -r '.digest' manifest.json)

# Deploy to the staging cluster using the exact digest and staging configurations
helm upgrade --install loans ./charts/loans \
  --namespace staging \
  --set image.repository="cloudwithvarun/loans" \
  --set image.digest="$IMAGE_DIGEST" \
  --values values-stage.yaml

9. Stage to Prod Promotion & Production Deployment Strategies

Promoting code from Staging to Production (prod) follows the same "deploy-before-merge" pattern but incorporates stricter validation gates, multiple manual approvals, and advanced traffic routing strategies.

Stage Branch (Verified Commit)
   |
   | 1. Open Promotion PR to prod branch
   v
Prod Branch (Highly Protected)
   |
   | 2. Pre-flight Validation (Verify staging-proven digest)
   | 3. Multi-Party Manual Approvals (Tech Lead, QA Lead, Change Manager)
   | 4. Automated Production Rollout (Canary / Blue-Green)
   | 5. Post-Deploy Health Verification (SLOs & Synthetic Monitoring)
   v
Merge PR into prod branch (Only after successful rollout & verification)

Advanced Production Deployment Strategies

1. Feature Flags (Flags Off)

Applications are deployed to production with new features disabled by default via configuration flags.

• Rationale: Decouples code deployment from feature release. The team can verify the core application's stability in production before enabling the new feature.
• Rollout: Once stable, the feature flag is enabled in a staggered manner (e.g., 10% of users $\rightarrow$ 50% $\rightarrow$ 100%).

2. Canary Deployments

Gradually routes a small percentage of production traffic to the new version of the application while monitoring system health.

• Guardrails: The rollout is monitored against Service Level Objectives (SLOs). If error rates or latency spike on the canary version, the pipeline triggers an automated rollback.

3. Blue-Green Deployments

Maintains two identical physical environments: "Blue" (active production) and "Green" (idle/new release).

• Trade-off: Highly reliable with near-zero downtime and instant rollback capabilities, but doubles infrastructure costs because two full environments must be maintained.

10. Trunk-Based CI/CD Pipeline Walkthrough

In Trunk-Based Development, all environments are deployed from a single source of truth: the main branch. Promotions are managed by updating environment-specific configuration files rather than merging code across multiple branches.

In this model, the codebase remains unified. If a bug is introduced and passes the PR gates, it is merged into main. The team must prioritize fixing the issue directly on main to unblock the pipeline, requiring highly robust automated testing suites.

11. The Role of Jenkins as the Orchestration Engine

Jenkins is an open-source automation server that serves as the central orchestrator for CI/CD pipelines. It does not replace specialized tools but coordinates them into a unified workflow.

What Jenkins Is

• An Orchestration Engine: Jenkins executes defined pipelines (written as code in a Jenkinsfile) to pull code, trigger tests, build container images, push to registries, and update target environments.
• Extensible Platform: Supports thousands of plugins to integrate with cloud providers (AWS, Azure, GCP), testing frameworks, and security scanners.
• Gatekeeper: Communicates execution statuses (green/red) back to Git hosts to enforce branch protection rules.

What Jenkins Is Not

• Not a Git Host: It does not store source code; it integrates with platforms like GitHub or GitLab.
• Not a Container Registry: It does not host container images; it pushes them to registries like Docker Hub or GHCR.
• Not a Monitoring Tool: It does not monitor production application health; it hands off deployment control to runtime environments and integrates with monitoring suites like Prometheus.
• Not a GitOps Controller: In GitOps workflows, Jenkins handles the CI and artifact creation phases, then hands off deployment execution to tools like Argo CD or Flux.

12. Summary and Key Takeaways

• Git is a Distributed Version Control System: It provides every developer with a complete local history, enabling offline work and safe code traversal.
• Branch Protection is Critical: Restricting direct pushes to branches like main and requiring passing CI status checks prevents unstable code from reaching production.
• Build Once, Promote by Digest: Rebuilding artifacts for different environments introduces drift. Build a container image once, tag it for traceability, and deploy it using its immutable SHA-256 digest.
• Git Flow Uses Environment Branches: Code is promoted by merging PRs sequentially through dev, stage, and prod branches.
• Trunk-Based Development Uses One Branch: All developers merge directly into main. Promotions to different environments are managed by updating configuration files rather than merging branches.
• Deploy-Before-Merge for Staging and Production: To keep pre-production and production branches clean, run deployments and post-deploy validation tests before allowing a promotion PR to merge.
• Merge-Before-Deploy for Development: To avoid merge queues and provide fast feedback, merge feature PRs into dev first, then trigger the deployment.
• Jenkins is an Orchestrator: It is an automation engine that coordinates Git hosts, registries, testing suites, and deployment targets to execute pipelines as code.