DevSecOps Pipeline: Multi-AZ Amazon EKS, Jenkins, Trivy, SonarQube, & Amazon ECR
A production-grade DevSecOps pipeline integrates security analysis and automated policy gates directly into the continuous integration and continuous deployment (CI/CD) lifecycle. By shifting security left, vulnerabilities in third-party libraries, application source code, and base container images are detected and mitigated before software reaches a production cluster. This guide walks through the architectural principles and operational commands required to build an automated, zero-trust deployment engine spanning Jenkins, SonarQube, Trivy, Amazon ECR, and a multi-Availability Zone (AZ) Amazon EKS cluster.
1. Pipeline Architecture and Security Mental Models
Traditional DevOps pipelines treat security as an afterthought, scanning applications in staging environments or post-deployment. DevSecOps mandates that every commit undergoes layered inspection across different vectors:
The Three Security Layers (SCA vs. SAST vs. Container Scanning)
2. Infrastructure Topology and Security Groups
The pipeline infrastructure runs across three dedicated EC2 instances running Ubuntu 22.04 LTS alongside an Amazon EKS managed cluster in the ap-south-1 (Mumbai) region.
Note: The Jenkins Agent requires an EBS allocation of at least 30 GB. Maven downloads project dependencies to its local cache (
~/.m2/repository), Trivy pulls complete vulnerability databases locally, and Docker caches container layers. Allocating 15 GB or less will trigger out-of-disk failures during image build phases.
3. Jenkins Controller and Distributed Agent Setup
Production CI/CD pipelines decouple scheduling from execution. The Jenkins Controller executes no build steps directly; it delegates pipeline tasks to dedicated build agents.
Jenkins Controller (SSH Client)
|
| 1. Generates ED25519 Key Pair (~/.ssh/jenkins_agent_key)
| 2. Connects over Port 22 using SSH Username + Private Key
v
Jenkins Agent (SSH Server)
|
| 3. Authenticates using authorized_keys (/home/jenkins/.ssh/authorized_keys)
| 4. Launches remoting agent process in workspace (/home/jenkins)
Step 1: Base Controller Initialization
Log in to the Jenkins Controller instance and run:
# Update host identity and system timezone
sudo hostnamectl set-hostname jenkins-controller
exec bash
sudo timedatectl set-timezone Asia/Kolkata
# Install JDK 21 (Required for Jenkins Core and Plugins)
sudo apt update && sudo apt install -y openjdk-21-jdk
# Add Jenkins Official Repositories and Install Core
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee \
/etc/apt/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/etc/apt/keyrings/jenkins-keyring.asc] \
https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
/etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt update && sudo apt install -y jenkins
# Adjust JVM system timezone to prevent UI timestamp skew
sudo systemctl edit jenkins
In the systemd override editor, append the JVM timezone argument:
[Service]
Environment="JAVA_OPTS=-Duser.timezone=Asia/Kolkata"
Save, reload, and verify the service:
sudo systemctl daemon-reload
sudo systemctl enable --now jenkins
sudo systemctl status jenkins
Retrieve the initial unlock key:
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
Step 2: Provisioning the Jenkins Agent Node
Log in to the Jenkins Agent instance and establish a clean user identity:
sudo hostnamectl set-hostname jenkins-agent
exec bash
sudo timedatectl set-timezone Asia/Kolkata
# Install Java Development Kit 21 (Required for Maven build lifecycle execution)
sudo apt update && sudo apt install -y openjdk-21-jdk
# Create isolated service user for Jenkins
sudo useradd -m -s /bin/bash jenkins
Step 3: Configuring ED25519 Mutual Authentication
Generate a cryptographically secure key pair under the jenkins user on the Jenkins Controller:
# Switch to the jenkins user on the Controller
sudo su - -s /bin/bash jenkins
mkdir -p ~/.ssh && cd ~/.ssh
# Generate ED25519 key without a passphrase
ssh-keygen -t ed25519 -f jenkins_agent_key -C "jenkins-controller-to-agent"
cat jenkins_agent_key.pub
On the Jenkins Agent, install the public key into the agent user's authorization file:
sudo su - -s /bin/bash jenkins
mkdir -p ~/.ssh && chmod 700 ~/.ssh
nano ~/.ssh/authorized_keys
# Paste the contents of jenkins_agent_key.pub here
chmod 600 ~/.ssh/authorized_keys
On the Jenkins Controller, prime the SSH known_hosts store using Trust On First Use (TOFU) to prevent interactive prompt failures during automated Jenkins SSH launches:
# From the controller (as user jenkins), initiate SSH using the Agent's PRIVATE IP
ssh -i ~/.ssh/jenkins_agent_key -o StrictHostKeyChecking=accept-new jenkins@<AGENT_PRIVATE_IP> hostname
Step 4: Registering the Agent in Jenkins UI
- Navigate to Manage Jenkins Nodes New Node.
- Node Name:
jenkins-agent - Type: Permanent Agent
- Number of Executors:
1(or matching dedicated vCPU limits) - Remote Root Directory:
/home/jenkins - Labels:
docker-maven-trivy - Launch Method: Launch agents via SSH
- Host:
<AGENT_PRIVATE_IP> - Credentials: Add SSH Username with Private Key:
- ID:
jenkins-agent-key - Username:
jenkins - Private Key: Paste contents of
~/.ssh/jenkins_agent_keyfrom the Controller.
- ID:
- Host Key Verification Strategy: Known hosts file Verification Strategy.
- Host:
- Click Save and select Launch Agent. Ensure the agent reports
Agent successfully connected and online.
4. SonarQube Server with Dedicated PostgreSQL
SonarQube running on an embedded H2 database is not production-ready; H2 does not support concurrent access, scales poorly, and cannot be cleanly backed up. A reliable deployment requires an external, relational database like PostgreSQL.
Jenkins Agent (Build Runtime)
|
| HTTP POST /api/ce/submit (Analysis Report + JaCoCo Metrics)
v
SonarQube Instance (:9000)
|
| TCP 5432 (State Storage, Quality Gate Rules, Project Indices)
v
PostgreSQL Server (sonar / sonar)
Step 1: PostgreSQL Installation and Initialization
Log into the dedicated SonarQube instance:
sudo hostnamectl set-hostname sonarqube-server
exec bash
sudo timedatectl set-timezone Asia/Kolkata
# Install JRE 21 and PostgreSQL
sudo apt update && sudo apt install -y openjdk-21-jre postgresql postgresql-contrib
# Start and enable PostgreSQL
sudo systemctl enable --now postgresql
# Create Database and Dedicated User
sudo -u postgres psql
Inside the PostgreSQL terminal:
CREATE ROLE sonar WITH LOGIN ENCRYPTED PASSWORD 'sonar';
CREATE DATABASE sonarqube OWNER sonar;
GRANT ALL PRIVILEGES ON DATABASE sonarqube TO sonar;
\q
Step 2: OS Kernel Tuning for ElasticSearch
SonarQube embeds ElasticSearch, which requires specific operating system kernel limits for memory mapping and file descriptors:
sudo bash -c 'cat <<EOF >> /etc/sysctl.d/99-sonarqube.conf
vm.max_map_count=524288
fs.file-max=131072
EOF'
sudo sysctl --system
sudo bash -c 'cat <<EOF >> /etc/security/limits.d/99-sonarqube.conf
sonar soft nofile 131072
sonar hard nofile 131072
sonar soft nproc 8192
sonar hard nproc 8192
sonar soft memlock unlimited
sonar hard memlock unlimited
EOF'
Step 3: SonarQube Binaries and Systemd Configuration
# Create the service user
sudo useradd -b /opt/sonarqube -d /opt/sonarqube -m -s /bin/bash sonar
# Download and extract SonarQube Community Edition
cd /tmp
wget https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-25.1.0.102122.zip
sudo apt install -y unzip
sudo unzip sonarqube-*.zip -d /opt/
sudo mv /opt/sonarqube-* /opt/sonarqube
sudo chown -R sonar:sonar /opt/sonarqube
Edit /opt/sonarqube/conf/sonar.properties to point to the local PostgreSQL database:
sonar.jdbc.username=sonar
sonar.jdbc.password=sonar
sonar.jdbc.url=jdbc:postgresql://localhost:5432/sonarqube
sonar.web.host=0.0.0.0
sonar.web.port=9000
Real-World Debugging Insight: A common misconfiguration is setting the JDBC port to
5431instead of5432. If the SonarQube service stops immediately after starting, check/opt/sonarqube/logs/sonar.logand/opt/sonarqube/logs/web.log. AConnection to localhost:5431 refusedentry confirms an incorrect port assignment insonar.properties.
Create the systemd unit file at /etc/systemd/system/sonarqube.service:
[Unit]
Description=SonarQube service
After=syslog.target network.target postgresql.service
[Service]
Type=forking
ExecStart=/opt/sonarqube/bin/linux-x86-64/sonar.sh start
ExecStop=/opt/sonarqube/bin/linux-x86-64/sonar.sh stop
User=sonar
Group=sonar
Restart=always
LimitNOFILE=131072
LimitNPROC=8192
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now sonarqube
Access the UI at http://<SONARQUBE_PUBLIC_IP>:9000, authenticate using default credentials (admin/admin), and reset the master password.
5. Security Integration: SonarQube Quality Gates and Trivy
Integrating security gates allows pipelines to detect vulnerabilities early and halt builds before deployment.
SonarQube Quality Gate Configuration
- In SonarQube, go to Administration Security Users Tokens.
- Generate an analysis token named
jenkins-sonar-tokenfor a dedicated service account. - In Jenkins, go to Manage Jenkins Credentials (global) Add Credentials:
- Kind: Secret text
- Secret:
<COPIED_SONAR_TOKEN> - ID:
sonarqube-token
- In SonarQube, go to Quality Gates Create:
- Name:
cloud-quality-gate - Add Condition: Overall Code Coverage is less than
80%.
- Name:
- Associate this quality gate with the project
cloud-with-vjosh-devsecops-demo.
Agent Tooling: Trivy Installation
Install Trivy directly on the Jenkins Agent node so the Jenkins executor can invoke it from a shell step:
sudo apt-get install wget apt-transport-https gnupg lsb-release -y
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo gpg --dearmor -o /usr/share/keyrings/trivy.gpg
echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" | sudo tee -a /etc/apt/sources.list.d/trivy.list
sudo apt update && sudo apt install -y trivy
The Anatomy of Quality Gate Failures
The pipeline enforces quality gates using non-zero exit codes:
# Filesystem and SCA Scan
trivy fs --exit-code 1 --severity HIGH,CRITICAL .
# Container Base Image Scan
trivy image --exit-code 1 --severity HIGH,CRITICAL ${IMAGE_REPO}:${BUILD_NUMBER}
If Trivy discovers a vulnerability matching the targeted severity levels, it returns exit code 1, immediately terminating the Jenkins stage and halting the build.
For SonarQube, passing the build flag -Dsonar.qualitygate.wait=true causes the Maven execution to poll the SonarQube server after report processing. If the project metrics violate quality gate conditions (such as low code coverage or new security hotspots), the goal throws an exception and halts the pipeline.
6. Container Packaging and Amazon ECR Integration
Container builds run on the Jenkins Agent and push directly to Amazon ECR. This requires the Docker daemon, the AWS CLI, and an attached AWS IAM role.
Agent Setup: Docker & AWS CLI
Run the following commands on the Jenkins Agent:
# 1. Install AWS CLI v2
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip -q awscliv2.zip
sudo ./aws/install
rm -rf aws awscliv2.zip
# 2. Install Docker CE
sudo install -m 0755 -d /etc/apt/keyrings
sudo
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update && sudo apt install -y docker-ce docker-ce-cli containerd.io
# 3. Grant Docker Socket Access to Jenkins Service Account
sudo usermod -aG docker jenkins
Real-World Debugging Insight: Modifying supplementary groups (
usermod -aG docker jenkins) does not automatically update active background processes. If Jenkins fails with aGot permission denied while trying to connect to the Docker daemon socketerror, navigate to Manage Jenkins Nodes jenkins-agent, disconnect the node, and relaunch it via SSH to reload the group permissions.
AWS IAM Role and Instance Profile Association
Never embed long-lived static AWS access keys on build machines. Instead, assign an IAM Role directly to the EC2 Jenkins Agent using an Instance Profile.
Create an inline policy named JenkinsEKSDescribePolicy on the IAM Role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"eks:DescribeCluster"
],
"Resource": "*"
}
]
}
Attach JenkinsAgentRole as an Instance Profile to the Jenkins Agent EC2 instance in the AWS Management Console.
ECR Repository Provisioning with Tag Immutability
aws ecr create-repository \
--repository-name cloud-with-vjosh-devsecops-demo \
--image-tag-mutability IMMUTABLE \
--image-scanning-configuration scanOnPush=true \
--region ap-south-1
Why Immutability Matters: In production systems, enabling Tag Immutability prevents bad actors or broken build jobs from overwriting existing container tags (such as
v1.2.0). This ensures that every deployed image tag references an unalterable artifact and prevents cache poisoning across your Kubernetes worker nodes.
7. Multi-AZ Amazon EKS Cluster Provisioning
To ensure high availability, the Kubernetes control plane requires redundant worker nodes distributed across multiple Availability Zones.
Declarative Cluster Configuration (eks-config.yaml)
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: cloud-with-vjosh-devsecops-eks
region: ap-south-1
version: "1.32"
tags:
Owner: CloudWithVJosh
Environment: Production
Project: DevSecOps
availabilityZones: ["ap-south-1a", "ap-south-1b"]
iam:
withOIDC: true
managedNodeGroups:
- name: cwj-ng-public
instanceType: t3.medium
minSize: 2
maxSize: 3
desiredCapacity: 2
volumeSize: 20
ssh:
allow: false
labels:
environment: production
role: worker
tags:
nodegroup-role: worker
Initialize the cluster using eksctl:
eksctl create cluster -f eks-config.yaml
Once provisioned, verify multi-AZ distribution across the nodes:
kubectl get nodes -o wide --show-labels
8. AWS Authentication and Kubernetes RBAC
For the Jenkins Agent to interact with the EKS API server, it must be mapped within the cluster's authentication layer and granted permissions through Role-Based Access Control (RBAC).
Jenkins Agent (Executing kubectl)
|
| 1. AWS IAM Identity Verification (STS Caller Identity)
v
EKS Control Plane (AWS IAM Authenticator)
|
| 2. Maps IAM Role ARN to Kube Identity via aws-auth ConfigMap
v
Kubernetes RBAC Engine
|
| 3. Matches mapped user to ClusterRoleBinding
v
Executes API Operation (e.g., pods/create, deployments/update)
Step 1: Mapping the IAM Role to the aws-auth ConfigMap
Edit the authentication configuration map in the kube-system namespace:
kubectl edit configmap aws-auth -n kube-system
Append the Jenkins Agent IAM Role under the mapRoles key:
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::ACCOUNT_ID:role/eksctl-managed-node-role
username: system:node:{{EC2PrivateDNSName}}
- groups:
- system:authenticated
rolearn: arn:aws:iam::ACCOUNT_ID:role/JenkinsAgentRole
username: jenkins-agent-role
Step 2: Defining Kubernetes RBAC Roles and Bindings
Create a dedicated ClusterRole (cluster-role-deployer.yaml) enforcing the Principle of Least Privilege:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: jenkins-deployer-role
rules:
- apiGroups: ["", "apps"]
resources: ["deployments", "replicasets", "pods", "services", "namespaces"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: jenkins-deployer-binding
subjects:
- kind: User
name: jenkins-agent-role
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: jenkins-deployer-role
apiGroup: rbac.authorization.k8s.io
Apply the RBAC definitions to your cluster:
kubectl apply -f cluster-role-deployer.yaml
9. Kubernetes Topology-Aware Manifests and Rollout Guards
To prevent downtime, applications running in multi-AZ clusters must be evenly distributed across failure domains using topologySpreadConstraints.
Deployment and Service Manifest (deploy-svc.yaml)
apiVersion: apps/v1
kind: Deployment
metadata:
name: cloud-with-vjosh-devsecops-demo
namespace: default
labels:
app: cloud-with-vjosh-devsecops-demo
spec:
replicas: 2
selector:
matchLabels:
app: cloud-with-vjosh-devsecops-demo
template:
metadata:
labels:
app: cloud-with-vjosh-devsecops-demo
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: "topology.kubernetes.io/zone"
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: cloud-with-vjosh-devsecops-demo
containers:
- name: application
image: ACCOUNT_ID.dkr.ecr.ap-south-1.amazonaws.com/cloud-with-vjosh-devsecops-demo:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
name: cloud-with-vjosh-devsecops-service
namespace: default
spec:
type: NodePort
selector:
app: cloud-with-vjosh-devsecops-demo
ports:
- port: 8080
targetPort: 8080
nodePort: 31000
Pod Distribution Logic: maxSkew Explained
The constraint metric follows the formula:
Where:
- is the number of matching pods in Availability Zone A.
- is the number of matching pods in Availability Zone B.
- Setting ensures that the scheduler allows at most an imbalance of one pod between zones.
- Setting
whenUnsatisfiable: DoNotSchedulehalts pod placement if scheduling causes skew to exceed the limit, preventing pods from clumping into a single zone during node recovery.
Rollout Checks and Automatic Rollbacks
Deployment stages should never assume that kubectl apply guarantees a working application. The pipeline checks the deployment status and automatically rolls back if pods fail to stabilize:
# Check rollout status within a 60-second window; trigger a rollback on timeout
kubectl rollout status deployment/cloud-with-vjosh-devsecops-demo -n default --timeout=60s || {
echo "Rollout failed! Rolling back to previous release..."
kubectl rollout undo deployment/cloud-with-vjosh-devsecops-demo -n default
exit 1
}
10. Complete Declarative Pipeline Implementation
This end-to-end declarative Jenkinsfile runs on the distributed agent. It checks out source code, performs SCA, SAST, and container security scans, pushes artifacts to Amazon ECR, and orchestrates zero-downtime rollouts to Amazon EKS.
pipeline {
agent {
node {
label 'docker-maven-trivy'
}
}
tools {
maven 'Maven 3'
}
environment {
AWS_REGION = 'ap-south-1'
AWS_ACCOUNT_ID = '123456789012'
ECR_REGISTRY = "${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com"
IMAGE_REPO = "${ECR_REGISTRY}/cloud-with-vjosh-devsecops-demo"
EKS_CLUSTER_NAME = 'cloud-with-vjosh-devsecops-eks'
SONAR_HOST_URL = 'http://172.31.2.15:9000' // Use Private IP inside VPC
}
stages {
stage('SCM Checkout') {
steps {
checkout scm
}
}
stage('Trivy FileSystem & SCA Scan') {
steps {
echo "Running Software Composition Analysis (SCA) on dependencies and repo..."
sh 'trivy fs --exit-code 1 --severity HIGH,CRITICAL .'
}
}
stage('Build, Unit Test & SAST') {
steps {
withCredentials([string(credentialsId: 'sonarqube-token', variable: 'SONAR_TOKEN')]) {
echo "Executing Maven Compilation, JaCoCo Coverage & SonarQube Scanner..."
sh """
mvn clean verify sonar:sonar \
-Dsonar.projectKey=cloud-with-vjosh-devsecops-demo \
-Dsonar.host.url=${SONAR_HOST_URL} \
-Dsonar.token=${SONAR_TOKEN} \
-Dsonar.qualitygate.wait=true
"""
}
}
}
stage('Amazon ECR Authentication') {
steps {
echo "Authenticating Docker with Amazon ECR Registry..."
sh """
aws ecr get-login-password --region ${AWS_REGION} | \
docker login --username AWS --password-stdin ${ECR_REGISTRY}
"""
}
}
stage('Build Container Image') {
steps {
echo "Building application container with architecture constraints..."
sh """
export DOCKER_BUILDKIT=0
docker build --platform linux/amd64 \
-t ${IMAGE_REPO}:${BUILD_NUMBER} \
-t ${IMAGE_REPO}:latest .
"""
}
}
stage('Trivy Base Image Security Scan') {
steps {
echo "Scanning generated container image layers for OS and library CVEs..."
sh "trivy image --exit-code 1 --severity HIGH,CRITICAL ${IMAGE_REPO}:${BUILD_NUMBER}"
}
}
stage('Push Container Image to ECR') {
steps {
echo "Pushing immutable tags to Amazon ECR..."
sh """
docker push ${IMAGE_REPO}:${BUILD_NUMBER}
docker push ${IMAGE_REPO}:latest
"""
}
}
stage('Inject Dynamic Image Tag') {
steps {
echo "Patching Kubernetes deployment manifest with build identifier ${BUILD_NUMBER}..."
sh """
sed -i "s|image:.*|image: ${IMAGE_REPO}:${BUILD_NUMBER}|g" deploy-svc.yaml
"""
}
}
stage('Deploy & Verify Rollout on Amazon EKS') {
steps {
echo "Applying manifests and monitoring cluster rollout state..."
sh """
# Generate temporary cluster credentials
aws eks update-kubeconfig --region ${AWS_REGION} --name ${EKS_CLUSTER_NAME}
# Deploy Manifests
kubectl apply -f deploy-svc.yaml
# Monitor zero-downtime progressive deployment
kubectl rollout status deployment/cloud-with-vjosh-devsecops-demo --timeout=60s || {
echo "Health check failed within 60s window! Executing immediate cluster rollback..."
kubectl rollout undo deployment/cloud-with-vjosh-devsecops-demo
exit 1
}
"""
}
}
}
post {
always {
echo "Pipeline run completed for build: ${BUILD_NUMBER}."
cleanWs notFailBuild: true
}
success {
echo "DevSecOps Pipeline executed successfully. Deployment is live."
}
failure {
echo "Critical Alert: DevSecOps Pipeline failed! Check the security or rollout logs above."
}
}
}
11. Production Hardening and Maturity Roadmap
- Adopt Ephemeral Jenkins Agents on EKS: Running long-lived build agent VMs introduces configuration drift and requires manual patch maintenance. Replace static EC2 agents with the Jenkins Kubernetes Plugin, which spins up dynamic, single-use build pods that terminate immediately after a job finishes.
- Enforce Cryptographic Image Signing with Cosign: Rather than relying solely on tag names, use Sigstore's Cosign to sign container images during the build phase. Configure Kubernetes admission controllers to block images that lack a valid cryptographic signature.
- Adopt Ingress Controllers Over NodePorts: NodePorts expose application ports directly on the public IP addresses of your worker nodes. In production environments, place nodes in private subnets and route external traffic through the AWS Load Balancer Controller, using managed Target Groups and automated TLS termination.
- Implement Admission Controllers (Kyverno / OPA Gatekeeper): Move beyond pipeline-only checks by deploying admission controllers inside your cluster. These controllers can reject pods that run as root, require CPU and memory requests, and block images pulled from registries other than your approved Amazon ECR account.
- Transition to GitOps Using ArgoCD: Decouple the build phase from deployment by having Jenkins push built artifacts and update an image tag in a separate Git repository. A GitOps operator like ArgoCD can then reconcile cluster state with the repository, removing cluster deployment privileges from your CI system entirely.
Summary and Key Takeaways
- Shift-Left Security Implementation: Layering SCA (Trivy fs), SAST (SonarQube/JaCoCo), and container scanning (Trivy image) catches security vulnerabilities and code regressions early in the delivery cycle.
- Zero Static IAM Secrets: Using an EC2 Instance Profile with an IAM Role eliminates the need to hardcode static AWS credentials on build nodes, improving overall operational security.
- Database Decoupling for SonarQube: Production SonarQube deployments should always use an external, production-grade database like PostgreSQL, paired with tuned Linux kernel parameters (
vm.max_map_count), rather than relying on the embedded H2 database. - Immutable Container Registries: Enabling Tag Immutability in Amazon ECR prevents existing release tags from being overwritten, ensuring traceable and repeatable deployments across your infrastructure.
- Kubernetes Architecture Parity: Setting
platform: linux/amd64during container builds ensures binary compatibility with x86-based Kubernetes worker nodes. - Multi-AZ High Availability: Using
topologySpreadConstraintswithmaxSkew: 1spreads pods evenly across different Availability Zones, preventing single-zone outages from taking down your applications. - Automated Rollout Verification: Pairing
kubectl rollout statuschecks with automatedrollout undofallbacks prevents failed builds from lingering and causing production downtime.