Day 7: Jenkins Multibranch Pipelines | Production-Ready CI/CD with GitHub PRs

Multibranch Pipelines represent a fundamental architectural shift in continuous integration and deployment. Instead of manually configuring isolated jobs for individual branches, a Multibranch Pipeline dynamically scans your source control management (SCM) system—discovering branches, tags, and pull requests—and provisions dedicated child pipelines on demand. This paradigm forms the operational engine of modern trunk-based development, guaranteeing that every line of code is validated in isolation before merging, while keeping the main trunk continuously releasable.

1. The Core Mental Model: From Single Pipelines to Multibranch Pipelines

In standard Jenkins configurations, a traditional pipeline job points to a single repository and a statically configured branch (typically main or master). This model breaks down in collaborative software teams where multiple feature branches, bug fixes, and pull requests exist concurrently.

The SCM-Scanning Paradigm

A Multibranch Pipeline (MBP) operates as an umbrella container job. It does not execute build steps directly at the root level. Instead:

1. It queries the SCM provider's API (such as GitHub, GitLab, or Bitbucket) using configured credentials.
2. It detects references that match specified behavioral criteria (e.g., branches, open pull requests, tags).
3. It checks for the existence of a pipeline definition file (by default, Jenkinsfile) in the root of each detected reference.
4. It dynamically instantiates, executes, and prunes child pipeline jobs corresponding to each valid branch or pull request.

Note: While Jenkins implements this through the Multibranch Pipeline Job type and the SCM Branch Source plugin, the underlying concept is universal across modern CI/CD systems. In GitHub Actions and GitLab CI, similar logic is expressed natively within workflow files via event triggers (on: [push, pull_request]).

2. Trunk-Based CI/CD Architecture

Trunk-based development requires all engineers to merge small, frequent updates into a single core branch (often main or trunk). To maintain a clean, deployable baseline without long-lived, divergent feature branches, the CI/CD pipeline acts as an automated gatekeeper.

The Fast-Feedback Imperative

In trunk-based pipelines, branch and PR validation stages must be strictly optimized:

• Target Execution Duration: Pipelines validating branches should ideally complete in under 5 minutes.
• Blast Radius Isolation: If tests fail on a feature branch, only that feature's child pipeline turns red. The shared main trunk remains untouched, stable, and ready for deployment.
• Merge Hygiene: Developers must keep their feature branches up to date with the trunk. If main advances while a PR is open, the PR pipeline must validate against a fresh synthetic merge of the branch and the latest main commit.

3. The Production Validation Lifecycle

A production-grade CI/CD pipeline enforces sequential verification tiers. Each phase guards the system against distinct categories of failure:

Verification Stages Defined

4. Branch Discovery Strategies in Jenkins

When integrating Jenkins with GitHub through the GitHub Branch Source plugin, the pipeline controller requires explicit rules governing which Git references generate automated jobs.

Branch Discovery Behaviors

Pull Request Discovery Strategies

When discovering pull requests from origin repositories, Jenkins provides three operational evaluation modes:

GitHub Repository State:
Base: main (Commit M1)
Branch: feature/login (Commit F1)

1. The Current Pull Request Revision (PR Head):
   Jenkins tests Commit F1 in isolation.
   Risk: Passes in isolation, but may break when combined with newer commits on main.

2. Merging the PR with Current Target Revision (Synthetic Merge):
   Jenkins executes an internal temporary merge: [M1 + F1].
   Benefit: Proves whether the code will actually work when merged into the latest trunk.

3. Both PR Revision and Merged PR Revision:
   Jenkins generates TWO parallel jobs per PR:
   - PR-XX (Head)
   - PR-XX (Merge)

5. Application and Infrastructure Baseline

To demonstrate the complete lifecycle, we configure a lightweight, containerized Python Flask application managed by a declarative Jenkinsfile.

Directory Structure

Application Manifests

Application Code (app.py)

from flask import Flask

app = Flask(__name__)

@app.route("/")
def home():
    return {
        "message": "Welcome to Cloud with Varun Joshi",
        "tip": "Built with Flask, shipped by Jenkins, running in Docker",
        "status": "Healthy"
    }

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

Dependencies (requirements.txt)

flask==3.0.0

Container Specification (Dockerfile)

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 5000

CMD ["python", "app.py"]

6. Production Declarative Jenkinsfile Architecture

The declarative Jenkinsfile orchestrates the build, containerization, image registration, deployment, and metadata archiving stages:

pipeline {
    agent any

    environment {
        DOCKER_HUB_REPO = 'cloudwithvarun/flask-app'
        CONTAINER_NAME  = 'flask-production-app'
        HOST_PORT       = '5000'
        APP_PORT        = '5000'
        DOCKER_CREDS_ID = 'docker-hub-credentials'
    }

    stages {
        stage('Checkout Source') {
            steps {
                checkout scm
            }
        }

        stage('Build Container Image') {
            steps {
                script {
                    // Tag using build number and static latest
                    sh "docker build -t ${DOCKER_HUB_REPO}:${BUILD_NUMBER} -t ${DOCKER_HUB_REPO}:latest ."
                }
            }
        }

        stage('Registry Authentication & Push') {
            steps {
                withCredentials([usernamePassword(credentialsId: "${DOCKER_CREDS_ID}", 
                                                  usernameVariable: 'DOCKER_USER', 
                                                  passwordVariable: 'DOCKER_PASS')]) {
                    sh '''
                        echo "$DOCKER_PASS" \vert  docker login -u "$DOCKER_USER" --password-stdin
                        docker push ${DOCKER_HUB_REPO}:${BUILD_NUMBER}
                        docker push ${DOCKER_HUB_REPO}:latest
                    '''
                }
            }
        }

        stage('Generate Deployment Metadata') {
            steps {
                sh """
                    mkdir -p build-metadata
                    echo "BUILD_NUMBER=${BUILD_NUMBER}" > build-metadata/deploy-info.txt
                    echo "GIT_COMMIT=${GIT_COMMIT}" >> build-metadata/deploy-info.txt
                    echo "GIT_BRANCH=${GIT_BRANCH}" >> build-metadata/deploy-info.txt
                    echo "IMAGE_TAG=${DOCKER_HUB_REPO}:${BUILD_NUMBER}" >> build-metadata/deploy-info.txt
                    echo "TIMESTAMP=\$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> build-metadata/deploy-info.txt
                """
                // Preserve deployment trace in Jenkins master jobs directory
                archiveArtifacts artifacts: 'build-metadata/deploy-info.txt', fingerprint: true
            }
        }

        stage('Deploy to Host') {
            steps {
                script {
                    // Idempotent deployment: Stop and remove stale container if present
                    sh """
                        if [ \$(docker ps -aq -f name=${CONTAINER_NAME}) ]; then
                            docker stop ${CONTAINER_NAME} || true
                            docker rm ${CONTAINER_NAME} || true
                        fi
                        docker run -d --name ${CONTAINER_NAME} -p ${HOST_PORT}:${APP_PORT} ${DOCKER_HUB_REPO}:${BUILD_NUMBER}
                    """
                }
            }
        }
    }

    post {
        success {
            echo "Pipeline completed successfully. Deployed revision: ${BUILD_NUMBER}."
        }
        failure {
            echo "Critical Pipeline Error: Execution failed on branch/PR ${env.BRANCH_NAME}."
        }
        always {
            echo "Execution cycle finished for job: ${env.JOB_NAME}."
        }
    }
}

7. Step-by-Step Implementation Walkthrough

Phase 1: Configuring the Multibranch Pipeline

1. Navigate to Jenkins Web UI (localhost:8080) -> New Item.
2. Set the item name to MVP-flask and select Multibranch Pipeline.
3. Under Branch Sources:
   • Choose GitHub.
   • Input the Repository URL (e.g., https://github.com/<org>/flask-private-repo).
   • Select valid Personal Access Token (PAT) credentials. Click Validate.
4. Configure Behaviors:
   • Discover branches: Exclude branches that are also filed as PRs.
   • Discover pull requests from origin: The current pull request revision (or Both if concurrency handles it).
5. Set Script Path to Jenkinsfile.
6. Click Save. Jenkins immediately performs its initial repository scan.

Jenkins Scan Console Output:
Started
Checking branch main
Jenkinsfile found. Met criteria.
Scheduled build for branch: main
1 branches were processed.
Checking pull requests...
0 pull requests were processed.
Finished branch indexing. Result: SUCCESS

Phase 2: Feature Branch Creation and Modifications

Clone the private repository locally and construct the feature branch:

# Clone private repository using credentials
git clone https://<PAT_TOKEN>@github.com/<org>/flask-private-repo.git
cd flask-private-repo

# Create and checkout feature branch
git branch feature/ui-font
git switch feature/ui-font

# Modify app.py to introduce new UI update
sed -i 's/"Healthy"/"Healthy: New UI Font Loaded"/g' app.py

# Commit changes locally
git add app.py
git commit -m "feat(ui): update font and health response"

# Push feature branch to upstream remote
git push origin feature/ui-font

Phase 3: SCM Discovery and Verification

Once the remote receives the feature branch, re-scan the repository in Jenkins (via Scan Repository Now or an automated webhook):

1. Jenkins indexes feature/ui-font.
2. A dedicated child pipeline MVP-flask/feature%2Fui-font is automatically created.
3. The branch pipeline builds the image, archives deployment metadata, and runs the container on host port 5000.

Verification via terminal:

$ curl http://localhost:5000
{
  "message": "Welcome to Cloud with Varun Joshi",
  "status": "Healthy: New UI Font Loaded",
  "tip": "Built with Flask, shipped by Jenkins, running in Docker"
}

Phase 4: Raising a Pull Request

With branch checks passing, the developer opens a Pull Request on GitHub:

• Base: main
• Compare: feature/ui-font
• Title: feat(ui): update font and health response

Because Exclude branches that are also filed as PRs is active, Jenkins automatically halts future branch-level jobs for feature/ui-font. From this point forward, every subsequent push to this branch triggers the PR job instead, avoiding duplicate builds and conserving CI agent resources.

Phase 5: Merging, Trunk Rebuilding, and Post-Build Pruning

1. Ensure all required GitHub status checks turn green.
2. In GitHub repository Settings -> General:
   • Check Automatically delete head branches.
3. Under Branches -> Branch protection rules:
   • Target branch: main.
   • Enable Require a pull request before merging.
4. Merge the Pull Request on GitHub.
5. GitHub automatically deletes the remote feature/ui-font branch.
6. Trigger an SCM scan on Jenkins:
   • The main pipeline runs Build #2, deploying the newly merged code.
   • Jenkins marks the child pipeline feature/ui-font and PR-1 as Disabled or Pruned, unlinking the jobs and cleaning up associated workspace storage.

8. Real-World Practical Pitfalls and Debugging

Port Allocation Collisions on Fixed Docker Hosts

Root Cause

Executing automated deployments directly against a single shared host Docker daemon causes race conditions. If a single PR triggers both the Head revision build and the Synthetic Merge build concurrently, both pipelines attempt to bind to 0.0.0.0:5000 simultaneously, causing one build to fail.

Production Resolution

1. Dynamic Ephemeral Port Mapping: Do not hardcode host ports. Let Docker bind dynamically:
   BASH

   Copy

   docker run -d -p 0:5000 --name flask-${BUILD_TAG} ${DOCKER_HUB_REPO}:${BUILD_NUMBER}
   ASSIGNED_PORT=$(docker port flask-${BUILD_TAG} 5000 | cut -d: -f2)
   

2. Namespace Isolation: Deploy into isolated, short-lived Kubernetes namespaces (e.g., preview-pr-${CHANGE_ID}) instead of binding host ports.

Workspace Leaks from Ephemeral Branches

Short-lived feature branches create dedicated directories on disk under $JENKINS_HOME/workspace/. If branches are frequently created and deleted, disk space can quickly fill up.

Production Resolution

Add the Workspace Cleanup Plugin to the Jenkinsfile post-actions to clean up after every execution:

post {
    always {
        cleanWs deleteDirs: true, notFailBuild: true
    }
}

Stale Base Revisions (The "Merge Drift" Bug)

If a feature branch remains open while multiple other PRs merge into main, testing only the Head Revision creates a false sense of security. The feature branch code may be bug-free in isolation, but it might fail when merged into the updated main.

Production Resolution

Configure GitHub Branch Discovery to use The pull request merged with the current target branch revision (synthetic merge). This forces Jenkins to fetch the latest main commit and verify the merged result before allowing the PR to be approved.

9. Summary and Key Takeaways

• Dynamic Job Lifecycle: Multibranch Pipelines eliminate manual job maintenance by scanning SCM repositories and automatically provisioning child pipelines for branches, tags, and pull requests.
• Trunk-Based Enabler: Serves as the operational engine for trunk-based development, ensuring that main remains continuously releasable while feature branches stay small and short-lived.
• Strict Pipeline Detection: Jenkins only activates builds on branches containing a valid Jenkinsfile at their root (or at a configured custom script path).
• Branch Exclusion Rules: Using the strategy Exclude branches that are also filed as PRs prevents duplicate builds, seamlessly shifting CI runs from the branch to the pull request once opened.
• Synthetic PR Validation: Testing the pull request merged with the current target branch revision validates the code against the latest trunk commit, catching merge conflicts and regression bugs before merging.
• Artifact Immutability: Binaries and container images should be built and packaged only once, tagged uniquely with the $BUILD_NUMBER, and promoted across environments without recompilation.
• Automated Resource Housekeeping: Merging pull requests should trigger the immediate deletion of remote head branches in GitHub, which prompts Jenkins to retire and prune the corresponding child jobs and workspaces.