AWS Zero to Hero: AWS End-to-End Continuous Integration (CI) Pipeline

1. Overview & Architecture Introduction

In this chapter, we implement a fully managed, end-to-end Continuous Integration (CI) pipeline on the AWS platform using native managed services. Rather than writing custom build scripts on self-hosted infrastructure (such as managing Jenkins worker nodes), we orchestrate modern cloud-native developer tooling: GitHub as the Version Control System (VCS), AWS CodePipeline as the pipeline orchestrator, and AWS CodeBuild as the managed build environment.

The core objective of a CI pipeline is to automatically fetch code changes, run prerequisite dependency installs, package applications into container images, and securely push artifacts to a container registry without manual intervention.

2. Core Mental Model: The Factory Assembly Line

To understand managed AWS CI services, think of the architecture as an automated manufacturing plant:

• GitHub (The Raw Materials Warehouse): Developers push code changes here. It stores raw material components (source code, Dockerfile, dependencies).
• AWS CodePipeline (The Floor Manager / Orchestrator): It doesn't build anything itself. Instead, it watches the warehouse doors, catches incoming commit notifications, and hands off the work order to the assembly floor.
• AWS CodeBuild (The Managed Assembly Worker): A managed server environment that spins up on demand, executes pre-configured build steps (installing dependencies, compiling code, building Docker images), and cleans itself up when finished.
• AWS Systems Manager Parameter Store (The Secure Vault): A secure locking cabinet where sensitive credentials (passwords, API tokens, registry keys) are stored so they never get hardcoded into public build files.

3. Preparing the Source Repository Structure

Before configuring AWS services, our GitHub repository (aws-devops-zero-to-hero, under folder day-14/sample-python-app) contains the required files for a containerized Python Flask application:

1. app.py: A basic Python Flask "Hello World" application.
2. requirements.txt: Specifies application dependencies (e.g., flask).
3. Dockerfile: Defines the container packaging steps.
4. buildspec.yaml: The instruction manual executed by AWS CodeBuild.

Sample Dockerfile Structure

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 5000
CMD ["python", "app.py"]

4. Setting Up AWS CodeBuild Project Configuration

We configure AWS CodeBuild first to establish our build steps before wrapping it inside an orchestrating pipeline.

Step-by-Step Configuration:

1. Navigate to the AWS CodeBuild console and click Create build project.
2. Project Name: sample-python-flask-service.
3. Source Provider: Select GitHub (supports OAuth and Personal Access Tokens).
4. Repository: Choose either a Public repository URL or connect via your personal GitHub account OAuth.
5. Environment Settings:
   • Environment image: Managed image.
   • Operating System: Ubuntu.
   • Runtime(s): Standard.
   • Image: Choose the latest available standard Ubuntu runtime image (aws/codebuild/standard:7.0 or similar).
   • Compute: Choose standard small instance types.
   • Privileged Mode: CRITICAL CHECK — You must enable the "Privileged" checkbox (Enable this flag if you want to build Docker images or get security updates). Without privileged mode, the build container lacks root access to the Docker daemon (/var/run/docker.sock), causing Cannot connect to the Docker daemon errors.
6. Service Role: Choose an existing IAM role or allow CodeBuild to automatically create a new service role.

5. Managing Secrets Securely with AWS Systems Manager Parameter Store

Hardcoding registry passwords or tokens in configuration files introduces critical security vulnerabilities. We use AWS Systems Manager (SSM) Parameter Store to manage sensitive variables.

Parameter Store Configuration Table

Note on IAM Permissions: The IAM Service Role attached to AWS CodeBuild must have explicit permissions to read parameters from Systems Manager. Attach the managed policy AmazonSSMReadOnlyAccess (or custom inline policies granting ssm:GetParameters) to the CodeBuild IAM role.

6. Writing the Build Specification (buildspec.yaml)

The buildspec.yaml file defines the lifecycle phases executed by CodeBuild: install, pre_build, build, and post_build.

version: 0.2

env:
  parameter-store:
    DOCKER_REGISTRY_URL: "/my-app/docker/registry-url"
    DOCKER_REGISTRY_USERNAME: "/my-app/docker/username"
    DOCKER_REGISTRY_PASSWORD: "/my-app/docker/password"

phases:
  install:
    runtime-versions:
      python: 3.11
  pre_build:
    commands:
      - echo "Installing application dependencies..."
      - cd day-14/sample-python-app
      - pip install -r requirements.txt
  build:
    commands:
      - echo "Logging into Docker Hub..."
      - echo "$DOCKER_REGISTRY_PASSWORD" \vert  docker login -u "$DOCKER_REGISTRY_USERNAME" --password-stdin "$DOCKER_REGISTRY_URL"
      - echo "Building Docker image..."
      - docker build -t $DOCKER_REGISTRY_USERNAME/sample-python-flask-app:latest .
      - echo "Pushing Docker image to Docker Hub..."
      - docker push $DOCKER_REGISTRY_USERNAME/sample-python-flask-app:latest
  post_build:
    commands:
      - echo "Build and push completed successfully on `date`"

7. Real-World Debugging & Production Pitfalls

When implementing CI pipelines from scratch, engineers frequently encounter specific failure modes. Analyzing these errors ensures robust pipeline design:

1. Path Resolution Errors (No such file or directory)

• Symptom: CodeBuild fails during the pre_build phase with ERROR: [Errno 2] No such file or directory: 'requirements.txt'.
• Root Cause: The build container defaults to the root of the repository, but source files are nested inside subfolders (e.g., day-14/sample-python-app/).
• Fix: Explicitly change directories (cd day-14/sample-python-app) before executing dependency installation commands.

2. Docker Daemon Connection Failure

• Symptom: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
• Root Cause: AWS CodeBuild runs inside an isolated container by default. Without elevated privileges, it cannot spawn sibling containers or interact with the host docker daemon.
• Fix: Enable Privileged Mode in the CodeBuild project environment settings.

3. Registry Authentication Failures

• Symptom: unauthorized: authentication required or denied: requested access to the resource is denied.
• Root Cause: Missing docker login step in the buildspec, or the IAM role lacks permissions to fetch credentials from SSM Parameter Store.
• Fix: Inject parameters using env.parameter-store and pipe the password securely into docker login --password-stdin.

8. Integrating with AWS CodePipeline (Orchestration)

While CodeBuild executes builds manually, AWS CodePipeline automates invocation upon every code commit.

Pipeline Creation Workflow:

1. Navigate to AWS CodePipeline -> Create pipeline.
2. Pipeline Settings: Enter pipeline name (sample-python-pipeline) and select service role.
3. Add Source Stage:
   • Source Provider: GitHub (Version 2).
   • Click Connect to GitHub, authorize OAuth access, select your repository (aws-devops-zero-to-hero), and branch (main).
4. Add Build Stage:
   • Build provider: AWS CodeBuild.
   • Project name: Select sample-python-flask-service created earlier.
   • Build type: Single build.
5. Add Deploy Stage: Skip for this CI-only implementation (Continuous Delivery will be covered separately).
6. Review and Create: Finalize pipeline creation.

9. End-to-End Verification Flow

To test the automated pipeline:

1. Make a minor modification (e.g., adding a space or updating a comment) directly in your GitHub repository app.py.
2. Commit the change to the main branch.
3. Observe the AWS CodePipeline console:
   • The Source stage automatically turns green as it detects the GitHub webhook event.
   • The Build stage transitions to In Progress, automatically invoking AWS CodeBuild.
4. Open the CodeBuild execution logs to monitor real-time compilation, dependency installation, Docker build output, and registry push confirmation.
5. Verify your Docker Hub repository (your-username/sample-python-flask-app) to confirm that the latest image tag reflects a timestamp of Just now.

10. Summary and Key Takeaways

• Managed CI Architecture: AWS CodePipeline acts as the pipeline orchestrator, while AWS CodeBuild functions as the managed build executor, eliminating the need to provision and maintain self-hosted build worker nodes (like Jenkins agents).
• GitHub Integration: Connecting GitHub via OAuth enables webhook-driven automated triggers upon every code commit to the repository.
• Privileged Mode Requirement: Building and pushing Docker containers inside AWS CodeBuild requires enabling Privileged Mode in project environment settings to interact with the Docker daemon.
• Secure Credential Management: Sensitive registry credentials, passwords, and tokens must never be hardcoded into configuration files; instead, store them securely in AWS Systems Manager (SSM) Parameter Store as SecureString parameters.
• IAM Role Least Privilege: AWS services communicate using IAM service roles. CodeBuild requires explicit IAM permissions (AmazonSSMReadOnlyAccess) to retrieve secrets from Parameter Store at runtime.
• Directory Context Awareness: Build scripts must account for repository folder structures by navigating (cd) into the appropriate application subdirectory before running install or build commands.
• Iterative Debugging: Real-world CI pipelines often fail due to missing syntax, incorrect paths, or permission gaps; analyzing CodeBuild build logs is essential for diagnosing and resolving runtime errors.