Day-7: Ultimate End-

to-End DevSecOps Project Implementation

This guide provides a production-grade, end-to-end implementation of DevSecOps practices across a modern three-tier application stack. It covers local validation, Infrastructure as Code (IaC) with Terraform on AWS, containerization and orchestration using Docker and Kubernetes (EKS), automated CI/CD pipelines via GitHub Actions, and continuous GitOps delivery through Argo CD. Security controls—including static analysis, container image vulnerability scanning, IaC posture evaluation, admission policies, and secret management—are integrated into every phase of the delivery lifecycle.

1. Problem Statement and Architectural Overview

Enterprise delivery models require transforming raw source code written by development teams into hardened, production-ready workloads running on cloud infrastructure. Rather than treating security as an isolated perimeter check at the conclusion of development, DevSecOps shifts verification, isolation, and policy enforcement directly into each phase of the deployment lifecycle.

The 3-Tier Architecture Mental Model

The target system is a microservices-based blogging application named Journey, comprising three interdependent layers:

Presentation Layer (Frontend): Written in ReactJS, served behind an NGINX reverse proxy on port 80. NGINX handles static asset routing, SSL offloading, and user-facing boundary interactions.
Application Layer (Backend): Written in Node.js utilizing Express, exposed internally on port 5000. Acts as the business logic coordinator and data abstraction gateway.
Persistence Layer (Database): PostgreSQL relational database running on standard port 5432. Stores posts, relational links, and user comments.

Structure & Concept Flow

Incoming User Traffic (Port 80)

NGINX Web Server / Reverse Proxy

ReactJS Single Page App (Frontend)

| HTTP REST Calls (Port 5000)

Node.js / Express Engine (Backend)

| SQL Queries / TCP (Port 5432)

PostgreSQL Database Storage Engine

Naive vs. Production Delivery Models

Naive Model: Developers provision a monolithic Virtual Machine (e.g., an AWS EC2 instance), manually install packages via SSH, hardcode secrets within source trees, execute scripts without verification, and leave standard database ports exposed directly to external networks.
Production DevSecOps Model:
- Infrastructure is codified, versioned, and immutable via Terraform.
- Workloads run within isolated containers built with multi-stage boundaries and non-root execution contexts.
- Network policies govern ingress and egress between pods inside private Kubernetes subnets.
- CI pipelines automatically validate code quality, software composition (SCA), container vulnerability profiles, and IaC compliance.
- Continuous Delivery pipelines employ declarative GitOps controllers (Argo CD) to eliminate configuration drift.

2. Phase 1: Local Verification on AWS EC2

DevOps workflows begin by verifying the raw application logic in an environment matching the target deployment OS before introducing container abstraction layers.

Structure & Concept Flow

Git Clone Repository (Main Branch)

Inspect deployment/setup.sh

Execute Package Management Setup (Node.js, PostgreSQL, NGINX)

Bind EC2 Security Group (Expose Port 80, Restrict Port 22)

Verify Browser Rendering at Public IPv4

Initial Node Provisioning

Deploy an Ubuntu LTS instance on AWS (recommended: t3.small to allocate sufficient CPU and memory for compiling the frontend and running PostgreSQL concurrently; storage minimum: 16 GB EBS GP3).
SSH into the instance with strict key permissions:

BASH

chmod 400 sample-keypair.pem
ssh -i sample-keypair.pem ubuntu@<EC2_PUBLIC_IP>

Update core repositories and pull source code:

BASH

sudo apt update && sudo apt upgrade -y
git clone https://github.com/iam-veeramalla/journey.git
cd journey/deployment

Local Provisioning Script Breakdown

Review setup.sh to identify system dependencies, database initialization steps, and reverse proxy routes:

BASH

#!/usr/bin/env bash
set -e

# Update and install base runtime packages
sudo apt update
sudo apt install -y curl gnupg build-essential postgresql postgresql-contrib nginx

# Install Node.js LTS distribution
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt install -y nodejs

# Configure PostgreSQL user, database, and schema
sudo -u postgres psql -c "CREATE USER journey_user WITH PASSWORD 'journey_password';"
sudo -u postgres psql -c "CREATE DATABASE journey_db OWNER journey_user;"

# Install and build frontend static assets
cd ../frontend
npm install
npm run build
sudo cp -r dist/* /var/www/html/

# Start Backend API service
cd ../backend
npm install
nohup node src/index.js > backend.log 2>&1 &

# Restart and bind NGINX Reverse Proxy
sudo systemctl restart nginx
echo "Application successfully initialized on Port 80."

Grant execution permissions and run:

BASH

chmod 700 setup.sh
./setup.sh

Navigate to http://<EC2_PUBLIC_IP> to confirm database write operations (creating a sample post, querying data).

3. Phase 2: Infrastructure as Code (IaC) with Terraform & EKS Auto Mode

To achieve enterprise repeatability, the foundational infrastructure—Virtual Private Cloud (VPC), routing tables, and managed Kubernetes control planes—is provisioned using declarative HashiCorp Configuration Language (HCL).

Structure & Concept Flow

Terraform CLI (Local Machine / Admin Node)

| Authenticates via AWS CLI (Access Key / Secret Key)

AWS API Gateway

+---> Creates VPC (10.0.0.0/16, Public & Private Subnets, NAT Gateway)

+---> Provisions EKS Cluster (v1.32, Auto Mode Enabled)

Karpenter Dynamic Node Controller (Serverless Data Plane)

1. Terraform Architecture Configurations

`provider.tf`

Configures the cloud provider, API version requirements, and state locking backend:

HCL

terraform {
  required_version = ">= 1.5.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = var.aws_region

  default_tags {
    tags = {
      Environment = var.environment
      Project     = "Journey-DevSecOps"
      ManagedBy   = "Terraform"
    }
  }
}

`variables.tf`

Establishes typed configurations, descriptions, and defaults:

HCL

variable "aws_region" {
  description = "Target AWS deployment region"
  type        = string
  default     = "us-east-1"
}

variable "environment" {
  description = "Deployment lifecycle stage"
  type        = string
  default     = "dev"
}

variable "cluster_name" {
  description = "Kubernetes EKS cluster identifier"
  type        = string
  default     = "journey-eks"
}

variable "cluster_version" {
  description = "Target Kubernetes API version"
  type        = string
  default     = "1.32" # Intentionally set to demonstrate lifecycle/vulnerability management
}

variable "vpc_cidr" {
  description = "Base network CIDR block for VPC"
  type        = string
  default     = "10.0.0.0/16"
}

`main.tf`

Invokes upstream vetted modules to create the isolated network and the dynamic compute platform:

HCL

data "aws_availability_zones" "available" {}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.environment}-journey-vpc"
  cidr = var.vpc_cidr

  azs             = slice(data.aws_availability_zones.available.names, 0, 2)
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true # Optimized for lab cost-reduction
  enable_dns_hostnames = true

  public_subnet_tags = {
    "kubernetes.io/role/elb" = 1
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = 1
  }
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = var.cluster_name
  cluster_version = var.cluster_version

  cluster_endpoint_public_access = true

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  # Enable EKS Auto Mode for dynamic, Karpenter-managed compute provisioning
  cluster_compute_config = {
    enabled    = true
    node_pools = ["general-purpose"]
  }
}

`outputs.tf`

Surfaces connection endpoints required for downstream tooling:

HCL

output "cluster_name" {
  value = module.eks.cluster_name
}

output "cluster_endpoint" {
  value = module.eks.cluster_endpoint
}

output "vpc_id" {
  value = module.vpc.vpc_id
}

2. Execution and State Lifecycle

BASH

# Initialize plugins and backend configuration
terraform init

# Validate structural integrity and compile execution graph
terraform plan -out=tfplan.binary

# Apply resource changes to AWS
terraform apply tfplan.binary

Note: EKS cluster provisioning involves creating VPC subnets, NAT gateways, security groups, IAM roles, and control-plane endpoints. This process typically requires 12 to 18 minutes.

3. Verification & EKS Auto Mode Mechanics

Connect to the cluster using the AWS CLI:

BASH

aws eks update-kubeconfig --region us-east-1 --name journey-eks
kubectl get nodes

Mechanism: In EKS Auto Mode, running kubectl get nodes immediately after provisioning returns empty results. Unlike standard static node groups with fixed compute nodes running continuously, EKS Auto Mode utilizes an integrated Karpenter dynamic controller. Compute nodes are spun up on-demand only when a non-tolerated pod enters a Pending state.

4. Phase 3: Multi-Stage Secure Containerization (Docker)

To minimize attack surfaces, image layers must not contain development tooling, compilers, shell histories, or package manager caches.

Structure & Concept Flow

STAGE 1: Builder Stage (Node:20-alpine)

|-- Mount /app

|-- Copy package*.json

|-- Execute: npm ci

|-- Compile source files: npm run build

Output Artifact: Static Production Files (/app/dist)

| COPY --from=builder /app/dist /usr/share/nginx/html

STAGE 2: Production Stage (NGINX:1.27-alpine / Distroless)

|-- Strip package managers and unneeded shells

|-- Drop privileges: USER nginx (Non-Root)

|-- Configure custom nginx.conf

Final Minimal Runtime Image

1. Frontend Dockerfile (`frontend/Dockerfile`)

DOCKERFILE

# Stage 1: Build Application Artifacts
FROM node:20-alpine AS builder
WORKDIR /app

# Copy dependency manifests separately to leverage Docker layer caching
COPY package.json package-lock.json ./
RUN npm ci && npm cache clean --force

# Copy application source and compile
COPY . .
RUN npm run build

# Stage 2: Minimal Production Runtime
FROM nginx:1.27-alpine AS production

# Remove default boilerplate configuration
RUN rm -rf /etc/nginx/conf.d/* /usr/share/nginx/html/*

# Inject hardened reverse-proxy configuration
COPY nginx.conf /etc/nginx/conf.d/default.conf

# Copy build output from Stage 1
COPY --from=builder /app/dist /usr/share/nginx/html

# Establish non-root permissions across writable paths
RUN chown -R nginx:nginx /usr/share/nginx/html /var/cache/nginx /var/log/nginx && \
    chmod -R 755 /usr/share/nginx/html

USER nginx
EXPOSE 8080

CMD ["nginx", "-g", "daemon off;"]

2. Backend Dockerfile (`backend/Dockerfile`)

DOCKERFILE

# Stage 1: Dependency Assembly
FROM node:20-alpine AS builder
WORKDIR /app

COPY package.json package-lock.json ./
RUN npm ci --only=production && npm cache clean --force

# Stage 2: Hardened Runtime Stage
FROM node:20-alpine AS production
WORKDIR /app

# Add security-compliant, unprivileged service account
RUN addgroup -S appgroup && adduser -S appuser -G appgroup

# Pull runtime dependencies from Stage 1
COPY --from=builder /app/node_modules ./node_modules
COPY src/ ./src
COPY package.json ./

# Secure filesystem ownership
RUN chown -R appuser:appgroup /app

USER appuser
EXPOSE 5000

CMD ["node", "src/index.js"]

5. Phase 4: Local Orchestration with Docker Compose

docker-compose.yaml links multiple containers into a unified local stack, configuring service discovery, internal networks, isolated volumes, and dependency startup sequences.

Structure & Concept Flow

docker compose up -d

Create Internal Bridge Network (journey-network)

+---> 1. Spin up journey-db (Postgres:16-alpine)

| |

| +--- Healthcheck: pg_isready -U journey_user

+---> 2. Spin up journey-backend (NodeJS) [depends_on: db == healthy]

| |

| +--- Connects via DB URI over internal DNS

+---> 3. Spin up journey-frontend (NGINX) [depends_on: backend]

+--- Exposes Port 80 to Host Localhost

Manifest Implementation: `docker-compose.yaml`

YAML

version: '3.8'

services:
  db:
    image: postgres:16-alpine
    container_name: journey-db
    restart: unless-stopped
    environment:
      POSTGRES_USER: journey_user
      POSTGRES_PASSWORD: journey_password
      POSTGRES_DB: journey_db
    ports:
      - "5432:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U journey_user -d journey_db"]
      interval: 5s
      timeout: 5s
      retries: 5
    networks:
      - journey-network

  backend:
    build:
      context: ./backend
      dockerfile: Dockerfile
    container_name: journey-backend
    restart: unless-stopped
    environment:
      PORT: 5000
      DB_HOST: db
      DB_PORT: 5432
      DB_USER: journey_user
      DB_PASSWORD: journey_password
      DB_NAME: journey_db
    ports:
      - "5000:5000"
    security_opt:
      - no-new-privileges:true
    depends_on:
      db:
        condition: service_healthy
    networks:
      - journey-network

  frontend:
    build:
      context: ./frontend
      dockerfile: Dockerfile
    container_name: journey-frontend
    restart: unless-stopped
    ports:
      - "80:8080"
    security_opt:
      - no-new-privileges:true
    depends_on:
      - backend
    networks:
      - journey-network

volumes:
  pgdata:
    driver: local

networks:
  journey-network:
    driver: bridge

Execute and confirm runtime operations:

BASH

docker compose up -d --build
docker compose ps
curl -I http://localhost

6. Phase 5: Production Kubernetes Manifests with Shift-Left Security

When deploying multi-tier applications to a Kubernetes cluster, network isolation, persistent volumes, and non-root execution contexts must be defined explicitly.

Structure & Concept Flow

Namespace (journey)

+---> StorageClass (EBS GP3) <--- PVC (10Gi) <--- Deployment (PostgreSQL)

| ^

| | DB Secret

| |

| NetworkPolicy (Allow ONLY Backend) | Port 5432

+--------------------------------------------------------+

+---> Service (ClusterIP: 5000) <------------------- Deployment (Backend)

| ^

| |

| NetworkPolicy (Allow ONLY Frontend) | Port 5000

+--------------------------------------------------------+

+---> Service (ClusterIP: 8080) <------------------- Deployment (Frontend)

| Port Forward / Ingress Controller

Ingress Traffic

Complete Declarative Specification: `k8s/journey.yaml`

YAML

apiVersion: v1
kind: Namespace
metadata:
  name: journey
  labels:
    app.kubernetes.io/part-of: journey
    pod-security.kubernetes.io/enforce: restricted
---
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: journey
type: Opaque
stringData:
  DB_USER: "journey_user"
  DB_PASSWORD: "journey_password"
  DB_NAME: "journey_db"
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-gp3-sc
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain
parameters:
  type: gp3
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: db-pvc
  namespace: journey
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ebs-gp3-sc
  resources:
    requests:
      storage: 10Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: journey-db
  namespace: journey
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: journey-db
  template:
    metadata:
      labels:
        app: journey-db
    spec:
      containers:
        - name: postgres
          image: postgres:16-alpine
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: DB_USER
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: DB_PASSWORD
            - name: POSTGRES_DB
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: DB_NAME
          volumeMounts:
            - name: pgdata
              mountPath: /var/lib/postgresql/data
      volumes:
        - name: pgdata
          persistentVolumeClaim:
            claimName: db-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: db-service
  namespace: journey
spec:
  selector:
    app: journey-db
  ports:
    - protocol: TCP
      port: 5432
      targetPort: 5432
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-network-policy
  namespace: journey
spec:
  podSelector:
    matchLabels:
      app: journey-db
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: journey-backend
      ports:
        - protocol: TCP
          port: 5432
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: journey-backend
  namespace: journey
spec:
  replicas: 2
  selector:
    matchLabels:
      app: journey-backend
  template:
    metadata:
      labels:
        app: journey-backend
    spec:
      containers:
        - name: backend
          image: ghcr.io/iam-veeramalla/journey-backend:latest
          ports:
            - containerPort: 5000
          env:
            - name: DB_HOST
              value: "db-service"
            - name: DB_PORT
              value: "5432"
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: DB_USER
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: DB_PASSWORD
            - name: DB_NAME
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: DB_NAME
---
apiVersion: v1
kind: Service
metadata:
  name: backend-service
  namespace: journey
spec:
  selector:
    app: journey-backend
  ports:
    - protocol: TCP
      port: 5000
      targetPort: 5000
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-network-policy
  namespace: journey
spec:
  podSelector:
    matchLabels:
      app: journey-backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: journey-frontend
      ports:
        - protocol: TCP
          port: 5000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: journey-frontend
  namespace: journey
spec:
  replicas: 2
  selector:
    matchLabels:
      app: journey-frontend
  template:
    metadata:
      labels:
        app: journey-frontend
    spec:
      containers:
        - name: frontend
          image: ghcr.io/iam-veeramalla/journey-frontend:latest
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: frontend-service
  namespace: journey
spec:
  type: ClusterIP
  selector:
    app: journey-frontend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

Deploying and testing connectivity:

BASH

kubectl apply -f k8s/journey.yaml
kubectl get pods -n journey

# Establish local tunnel directly to cluster frontend service
kubectl port-forward svc/frontend-service 8172:80 -n journey

7. Phase 6: Automated DevSecOps Pipeline via GitHub Actions

The CI/CD pipeline acts as an automated security gatekeeper. Every commit or pull request triggers linting, static code analysis, software composition analysis (SCA), vulnerability scanning, and infrastructure checks.

Structure & Concept Flow

Git Push / Pull Request

+---> Job 1: Linting Matrix (ESLint across Frontend & Backend)

+---> Job 2: Software Composition Analysis (npm audit / SCA)

+---> Job 3: Dockerfile Linting (Hadolint check for unpinned versions)

+---> Job 4: Infrastructure as Code Scan (Checkov on Terraform & K8s)

+---> Job 5: Container Build & Push (GHCR with Commit SHA tags)

+---> Job 6: Image Vulnerability Scanning (Trivy scanner for CVEs)

+---> Job 7: Dynamic Manifest Mutation (sed updates K8s image tag)

+---> Job 8: Git Commit & Push back to DevOps Branch

Complete Pipeline Implementation: `.github/workflows/ci.yaml`

YAML

name: Journey DevSecOps Workflow

on:
  push:
    branches: ["*"]
  pull_request:
    branches: ["*"]

env:
  REGISTRY: ghcr.io
  BACKEND_IMAGE_NAME: ${{ github.repository }}/journey-backend
  FRONTEND_IMAGE_NAME: ${{ github.repository }}/journey-frontend

jobs:
  lint:
    name: Source Code Linting
    runs-on: ubuntu-latest
    strategy:
      matrix:
        component: [frontend, backend]
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Install Dependencies
        run: |
          cd ${{ matrix.component }}
          npm ci

      - name: Run Linters
        run: |
          cd ${{ matrix.component }}
          npm run lint --if-present

  sca:
    name: Software Composition Analysis (SCA)
    runs-on: ubuntu-latest
    needs: lint
    strategy:
      matrix:
        component: [frontend, backend]
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Audit Production Dependencies
        run: |
          cd ${{ matrix.component }}
          npm audit --audit-level=high

  docker-lint:
    name: Dockerfile Best Practice Lint
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Run Hadolint
        uses: hadolint/hadolint-action@v3.1.0
        with:
          dockerfile: "**/Dockerfile"
          failure-threshold: error

  iac-scan:
    name: Infrastructure Security Scan (Checkov)
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Run Checkov Static Analysis
        uses: bridgecrewio/checkov-action@master
        with:
          framework: terraform,kubernetes
          output_format: cli
          soft_fail: false

  build-and-push:
    name: Container Build & Push
    runs-on: ubuntu-latest
    needs: [sca, docker-lint, iac-scan]
    permissions:
      contents: read
      packages: write
    outputs:
      commit_tag: ${{ steps.set_tag.outputs.tag }}
    strategy:
      matrix:
        include:
          - component: frontend
            context: ./frontend
            image: ghcr.io/${{ github.repository }}/journey-frontend
          - component: backend
            context: ./backend
            image: ghcr.io/${{ github.repository }}/journey-backend
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Set Image Tag
        id: set_tag
        run: echo "tag=${GITHUB_SHA::7}" >> $GITHUB_OUTPUT

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and Push Docker Image
        uses: docker/build-push-action@v5
        with:
          context: ${{ matrix.context }}
          push: true
          tags: |
            ${{ matrix.image }}:${{ steps.set_tag.outputs.tag }}
            ${{ matrix.image }}:latest

  image-scan:
    name: Container Image Scan (Trivy)
    runs-on: ubuntu-latest
    needs: build-and-push
    strategy:
      matrix:
        image:
          - ghcr.io/${{ github.repository }}/journey-frontend
          - ghcr.io/${{ github.repository }}/journey-backend
    steps:
      - name: Run Trivy Vulnerability Scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: "${{ matrix.image }}:${{ needs.build-and-push.outputs.commit_tag }}"
          format: "table"
          exit-code: "1"
          ignore-unfixed: true
          vuln-type: "os,library"
          severity: "CRITICAL,HIGH"

  update-manifest:
    name: Mutate K8s Manifest & GitOps Commit
    runs-on: ubuntu-latest
    needs: [build-and-push, image-scan]
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v4
        with:
          ref: devops
          token: ${{ secrets.GITHUB_TOKEN }}

      - name: Update Image Tags with In-Place Stream Editor (sed)
        run: |
          NEW_TAG="${{ needs.build-and-push.outputs.commit_tag }}"
          sed -i -E "s|(image:\s*ghcr.io/${{ github.repository }}/journey-frontend:).*\vert \1${NEW_TAG}|" k8s/journey.yaml
          sed -i -E "s|(image:\s*ghcr.io/${{ github.repository }}/journey-backend:).*\vert \1${NEW_TAG}|" k8s/journey.yaml

      - name: Commit and Push Updated Manifest
        run: |
          git config --global user.name "github-actions[bot]"
          git config --global user.email "github-actions[bot]@users.noreply.github.com"
          git commit -am "chore(gitops): update image tags to commit ${GITHUB_SHA::7}"
          git push origin devops

8. Phase 7: GitOps Continuous Delivery via Argo CD

Argo CD handles Continuous Delivery by implementing a pull-based model that continually synchronizes the state of the live cluster with the configuration stored in Git.

Structure & Concept Flow

Developer / Pipeline Push

GitHub Repository (k8s/journey.yaml updated with new SHA tag)

| Continuous Polling Loop (Every 180s or Webhook Trigger)

Argo CD Application Controller (Running inside EKS)

| Compares Live vs. Desired State (Out of Sync Detected)

Triggers Kubernetes API Rolling Update

+---> Terminates Old Pods

+---> Pulls New Digest from GHCR

+---> Validates Pod Health Probes

Live Cluster Synchronized with Git Repository (Synced & Healthy)

1. Bootstrapping Argo CD on the EKS Cluster

BASH

# Create dedicated namespace
kubectl create namespace argo-cd

# Deploy upstream manifests
kubectl apply -n argo-cd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Wait for control-plane readiness
kubectl wait --for=condition=available deployment/argocd-server -n argo-cd --timeout=300s

# Extract initial generated admin secret
kubectl get secret argocd-initial-admin-secret -n argo-cd \
  -o jsonpath="{.data.password}" | base64 -d; echo

# Expose Argo CD Web Interface locally
kubectl port-forward svc/argocd-server 7180:80 -n argo-cd

2. Declarative GitOps Application Definition

Apply an Application manifest inside the argo-cd namespace to establish the sync loop:

YAML

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: journey-production-app
  namespace: argo-cd
spec:
  project: default
  source:
    repoURL: 'https://github.com/iam-veeramalla/journey.git'
    targetRevision: devops
    path: k8s
  destination:
    server: 'https://kubernetes.default.svc'
    namespace: journey
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

9. Comprehensive Tooling and Security Matrix

The table below summarizes the DevSecOps tooling used across each phase of the project:

Phase	Technology	Tool Category	Security and Operational Rationale
IaC	Terraform	Infrastructure as Code	Enforces immutable infrastructure configurations, eliminates manual configuration drift, and maintains an auditable execution graph.
Compute	AWS EKS (Auto Mode)	Managed Kubernetes Engine	AWS handles dynamic control and data-plane provisioning via Karpenter, reducing operational complexity and cost.
Packaging	Docker (Multi-Stage)	Immutable Workload Artifact	Separates compilers and dev dependencies from production runtime images, dropping root privileges to isolate containers.
Orchestration	Docker Compose	Local Integration Tool	Emulates multi-service dependency trees, network bridging, and storage mounts on local developer workstations.
Network Security	K8s NetworkPolicy	In-Cluster Micro-Segmentation	Implements zero-trust isolation by allowing only the Frontend to access the Backend, and only the Backend to access PostgreSQL.
Code Hygiene	ESLint	Static Linting	Catches syntax errors, code styling issues, and potential code smells before dependencies are resolved.
Dependencies	npm audit	Software Composition Analysis	Scans application packages for known Common Vulnerabilities and Exposures (CVEs) during the pipeline build phase.
IaC Security	Checkov	Static Code Analysis for IaC	Prevents security misconfigurations (such as public S3 buckets or overly broad IAM roles) in Terraform and Kubernetes manifests.
Image Security	Trivy	Container Vulnerability Scanner	Blocks production deployments containing container base image vulnerabilities matching High or Critical severity scores.
Delivery	Argo CD	Declarative GitOps Engine	Synchronizes cluster workloads with Git as the single source of truth, automating rollouts and self-healing configuration drift.

10. Real-World Debugging and Failure Analysis

During pipeline construction, several intentional failure modes were analyzed to establish operational playbooks for troubleshooting production issues.

1. Docker Build Failure: Unpinned Package Manager Additions

Symptom: Hadolint or pipeline build fails during backend Docker compilation:

TEXT

[Hadolint DL3018] Pin versions in apk add. Instead of 'apk add dumb-init', use 'apk add dumb-init=1.2.5-r3'

Root Cause: Omitting exact package version tags introduces non-deterministic builds. Upstream package revisions can silently alter container behavior or introduce new vulnerabilities.
Resolution: Pin the package version explicitly:
DOCKERFILE
```
RUN apk add --no-cache dumb-init=1.2.5-r3
```

2. Checkov IaC Failure: Overly Permissive Terraform Configurations

Symptom: The iac-scan job returns exit code 1:

TEXT

Check: CKV_AWS_20: "Ensure that S3 bucket has an ACL defined that grants public read access"
FAILED for resource: module.vpc.aws_s3_bucket.state

Root Cause: A Terraform module exposed an unencrypted endpoint or omitted an explicit public access block.
Resolution: Ensure S3 buckets define explicit aws_s3_bucket_public_access_block attachments and enforce AES256 server-side encryption.

3. Trivy Container Scan Failure: Deprecated Base Images

Symptom: The image-scan stage terminates with critical vulnerability flags:

TEXT

Total: 47 (HIGH: 34, CRITICAL: 13)
CVE-2023-5363 OpenSSL vulnerability inside node:20-alpine

Root Cause: Using an unpatched base image tag.
Resolution: Shift from older images to updated patch distributions (e.g., update from node:20.0-alpine to node:20.18-alpine or an active distroless container variant).

4. GitOps Drift and Sync Halts

Symptom: Argo CD displays an OutOfSync status indefinitely:

TEXT

ComparisonError: PersistentVolumeClaim "db-pvc" is invalid: spec: Forbidden: spec is immutable after creation

Root Cause: Modifying a static volume claim size within k8s/journey.yaml after it has already been provisioned on EBS.
Resolution: Decouple stateful database operations from automated GitOps mutation cycles, or provision independent storage expansion sequences using AWS CSI volume expansion drivers.

11. End-to-End DevSecOps Execution Workflow

This walkthrough illustrates the complete path of a code change, from the initial commit to its deployment in the production Kubernetes cluster:

Structure & Concept Flow

Step 1: Developer Commits Code Change

Step 2: GitHub Actions Triggers CI Matrix

+---> Static Analysis (ESLint) validates application source code

+---> SCA (npm audit) evaluates package dependencies against known CVEs

+---> Linter (Hadolint) scans Dockerfiles for security anti-patterns

+---> Checkov assesses Terraform and Kubernetes manifests for misconfigurations

Step 3: Secure Container Artifact Generation

+---> Multi-stage Docker builds isolate the runtime environment

+---> The resulting image drops root privileges to an unprivileged system user

+---> Docker image is pushed to GitHub Container Registry tagged with the short Git SHA

Step 4: Image Vulnerability Gate (Trivy)

+---> Pulls image directly from GHCR

+---> Blocks deployment if CRITICAL or HIGH vulnerabilities are detected

Step 5: Automated Manifest Mutation

+---> In-place stream editing updates the deployment image tag in k8s/journey.yaml

+---> Pipeline commits the updated manifest directly back to the devops branch

Step 6: Declarative GitOps Reconciliation (Argo CD)

+---> Argo CD detects configuration drift between the Git repository and live cluster

+---> Initiates rolling deployment to replace pods with the new container image

+---> Enforces network policies and mounts EBS storage via StorageClass

Production Cluster Successfully Updated (Zero-Downtime, Cryptographically Traced)

Summary and Key Takeaways

Three-Tier Architecture Isolation: Production applications divide responsibilities across frontend presentation layers (React/NGINX), stateless backend logic (Node.js Express), and stateful database engines (PostgreSQL).
Shift-Left Security Principle: Incorporate security validation directly into developer workflows—using static code analysis, software composition analysis (SCA), and linting—rather than relying solely on perimeter defenses.
Infrastructure as Code (IaC): Utilizing modular, versioned Terraform code ensures infrastructure deployments to AWS VPC and EKS environments remain reproducible and auditable.
Dynamic Resource Provisioning: Using EKS Auto Mode with Karpenter provisions compute worker nodes on-demand based on actual pod requirements, lowering idle operational costs.
Multi-Stage Container Construction: Multi-stage Docker builds isolate compilation tools to the build phase, keeping development packages out of minimal, production runtime containers that run as non-root users.
Defense-in-Depth Network Policies: Kubernetes NetworkPolicies implement zero-trust network boundaries by ensuring the database pod accepts connections solely from authorized backend components.
Automated Security Pipelines: GitHub Actions automates sequential validation stages—including linting, npm audits, Hadolint checks, Checkov scans, Docker builds, and Trivy CVE scans.
Continuous GitOps Delivery: Argo CD decouples delivery pipelines from developer workstations by actively tracking Git as the single source of truth, automating zero-downtime rolling updates while preventing configuration drift.

Day-7 | Ultimate End to End DevSecOps Project Implementation | 3 Hours Tutorial

Day-7: Ultimate End-

1. Problem Statement and Architectural Overview

The 3-Tier Architecture Mental Model

Naive vs. Production Delivery Models

2. Phase 1: Local Verification on AWS EC2

Initial Node Provisioning

Local Provisioning Script Breakdown

3. Phase 2: Infrastructure as Code (IaC) with Terraform & EKS Auto Mode

1. Terraform Architecture Configurations

`provider.tf`

`variables.tf`

`main.tf`

`outputs.tf`

2. Execution and State Lifecycle

3. Verification & EKS Auto Mode Mechanics

4. Phase 3: Multi-Stage Secure Containerization (Docker)

1. Frontend Dockerfile (`frontend/Dockerfile`)

2. Backend Dockerfile (`backend/Dockerfile`)

5. Phase 4: Local Orchestration with Docker Compose

Manifest Implementation: `docker-compose.yaml`

6. Phase 5: Production Kubernetes Manifests with Shift-Left Security

Complete Declarative Specification: `k8s/journey.yaml`

7. Phase 6: Automated DevSecOps Pipeline via GitHub Actions

Complete Pipeline Implementation: `.github/workflows/ci.yaml`

8. Phase 7: GitOps Continuous Delivery via Argo CD

1. Bootstrapping Argo CD on the EKS Cluster

2. Declarative GitOps Application Definition

9. Comprehensive Tooling and Security Matrix

10. Real-World Debugging and Failure Analysis

1. Docker Build Failure: Unpinned Package Manager Additions

2. Checkov IaC Failure: Overly Permissive Terraform Configurations

3. Trivy Container Scan Failure: Deprecated Base Images

4. GitOps Drift and Sync Halts

11. End-to-End DevSecOps Execution Workflow

Summary and Key Takeaways

You've Reached the Final Chapter!

Day-7 | Ultimate End to End DevSecOps Project Implementation | 3 Hours Tutorial

Day-7: Ultimate End-

1. Problem Statement and Architectural Overview

The 3-Tier Architecture Mental Model

Naive vs. Production Delivery Models

2. Phase 1: Local Verification on AWS EC2

Initial Node Provisioning

Local Provisioning Script Breakdown

3. Phase 2: Infrastructure as Code (IaC) with Terraform & EKS Auto Mode

1. Terraform Architecture Configurations

provider.tf

variables.tf

main.tf

outputs.tf

2. Execution and State Lifecycle

3. Verification & EKS Auto Mode Mechanics

4. Phase 3: Multi-Stage Secure Containerization (Docker)

1. Frontend Dockerfile (frontend/Dockerfile)

2. Backend Dockerfile (backend/Dockerfile)

5. Phase 4: Local Orchestration with Docker Compose

Manifest Implementation: docker-compose.yaml

6. Phase 5: Production Kubernetes Manifests with Shift-Left Security

Complete Declarative Specification: k8s/journey.yaml

7. Phase 6: Automated DevSecOps Pipeline via GitHub Actions

Complete Pipeline Implementation: .github/workflows/ci.yaml

8. Phase 7: GitOps Continuous Delivery via Argo CD

1. Bootstrapping Argo CD on the EKS Cluster

2. Declarative GitOps Application Definition

9. Comprehensive Tooling and Security Matrix

10. Real-World Debugging and Failure Analysis

1. Docker Build Failure: Unpinned Package Manager Additions

2. Checkov IaC Failure: Overly Permissive Terraform Configurations

3. Trivy Container Scan Failure: Deprecated Base Images

4. GitOps Drift and Sync Halts

11. End-to-End DevSecOps Execution Workflow

Summary and Key Takeaways

You've Reached the Final Chapter!

`provider.tf`

`variables.tf`

`main.tf`

`outputs.tf`

1. Frontend Dockerfile (`frontend/Dockerfile`)

2. Backend Dockerfile (`backend/Dockerfile`)

Manifest Implementation: `docker-compose.yaml`

Complete Declarative Specification: `k8s/journey.yaml`

Complete Pipeline Implementation: `.github/workflows/ci.yaml`