AWS Zero to Hero: Day 7 - Production-Grade AWS Architecture & Implementation

Deploying applications securely in a production environment requires isolating workloads from the public internet while remaining accessible via load balancers. This chapter walks through the complete end-to-end implementation of a highly available, secure, two-tier AWS architecture utilizing VPCs, public/private subnets, Auto Scaling Groups (ASGs), Application Load Balancers (ALBs), NAT Gateways, and Bastion Hosts.

1. Why Do We Need Production-Grade Architecture?

The Problem with Single-Zone / Flat Deployments

In early proof-of-concept stages, engineers often deploy applications into a single public subnet inside a single Availability Zone (AZ) with direct public IP addresses attached to EC2 instances. This creates critical operational risks:

• Single Point of Failure (SPOF): If an AWS data center or AZ experiences an outage, the entire application goes offline.
• Security Vulnerabilities: Exposing raw EC2 instances directly to the public internet increases the surface area for brute-force attacks, vulnerability exploits, and unauthorized access.
• Lack of Outbound Isolation: If internal servers need to fetch updates or call external third-party APIs directly using public IPs, they risk inbound exposure if compromised.

The Production Solution

To mitigate these risks, production workloads are isolated inside a VPC (Virtual Private Cloud) split across multiple Availability Zones, separating public-facing entry points from private backend compute instances.

2. Core Mental Model & Real-Life Analogy

Real-Life Analogy: The Secure Corporate Headquarters

Think of this AWS production architecture like a high-security corporate headquarters:

• Internet Gateway (IGW): The main highway entrance gate to the corporate campus.
• Application Load Balancer (ALB): The front desk receptionist who greets visitors (incoming HTTP requests) and directs them to available internal employees.
• Public Subnet: The visitor lobby where outsiders are permitted to arrive.
• Private Subnet: The secure inner office floors containing sensitive company servers and databases. Outsiders cannot walk directly onto these floors.
• Bastion Host (Jump Box): A security checkpoint guard station. System administrators must first log into this guard station before stepping through to service inner-office machines.
• NAT Gateway: The outgoing mail room clerk. When internal servers need to order supplies from the outside world (fetch external API data), they hand their requests to the mail clerk, who masks the internal room numbers and sends the mail out with the headquarters' official public return address.

3. High-Level Architecture Flow

The traffic and operational workflow across the provisioned AWS components follows this clean downward path:

And for outgoing requests from private instances:

Private EC2 Instance
   |
   | requests external API data
   v
NAT Gateway [Public Subnet]
   |
   | masks private IP with Elastic IP (EIP)
   v
Internet (External API / Third-Party Services)

4. Architecture Components Breakdown

5. Step-by-Step Implementation: VPC & Networking

To avoid manually configuring route tables, subnets, and internet gateways individually, AWS provides a streamlined "VPC and more" configuration wizard.

Step 5.1: Provisioning the VPC via AWS Console

1. Navigate to the VPC Console in the AWS Management Console.
2. Click Create VPC and select VPC and more.
3. Configure the parameters:
   • Name tag: aws-prod-example
   • IPv4 CIDR block: Default (10.0.0.0/16)
   • Number of Availability Zones (AZs): 2 (e.g., us-east-1a and us-east-1b)
   • Number of public subnets: 2
   • Number of private subnets: 2
   • NAT gateways: 1 per AZ (or 1 total depending on cost/availability requirements)
   • VPC endpoints: None (disable S3 endpoint generation for this project)
4. Click Create VPC.

Note: If you encounter an error stating Maximum number of addresses has been reached when allocating Elastic IPs for the NAT Gateway, navigate to the EC2 console, locate unused Elastic IPs from past projects, and release them before retrying.

6. Step-by-Step Implementation: Auto Scaling Group & Launch Template

Backend applications should never be deployed as standalone EC2 instances in production; instead, they are managed via Launch Templates and Auto Scaling Groups.

Step 6.1: Creating the Launch Template

1. Search for EC2 in the console, scroll down to Auto Scaling, and select Launch Templates.
2. Click Create launch template:
   • Launch template name: aws-prod-example
   • AMI: Ubuntu (latest LTS version)
   • Instance type: t2.micro (free-tier eligible)
   • Key pair: Select your SSH key pair.
   • Network settings: Create a new Security Group named aws-prod-example-sg attached to your custom VPC.
3. Configure Inbound Security Group Rules:
   • SSH (Port 22) from Anywhere (0.0.0.0/0) — Note: In ultra-secure environments, restrict this strictly to the Bastion host IP.
   • Custom TCP (Port 8000) from Anywhere (0.0.0.0/0) — Used for our python test application.
4. Click Create launch template.

Step 6.2: Creating the Auto Scaling Group (ASG)

1. In the EC2 console, click Auto Scaling Groups -> Create auto scaling group.
2. Select the aws-prod-example launch template.
3. Configure network placement:
   • Select your custom VPC aws-prod-example.
   • Select both private subnets across AZ-1A and AZ-1B.
4. Skip load balancer attachment for now (we will attach the ALB via Target Groups later).
5. Configure group capacity:
   • Desired capacity: 2
   • Minimum capacity: 2
   • Maximum capacity: 4
6. Complete the wizard and click Create auto scaling group. Verify in the EC2 instances tab that two instances have spun up—one in us-east-1a and one in us-east-1b.

7. Step-by-Step Implementation: Bastion Host & Secure SSH Access

Because the application EC2 instances reside in private subnets, they possess no public IP addresses and cannot be reached directly via SSH from your local machine. We use a Bastion Host (Jump Box) deployed in the public subnet to bridge this gap.

Step 7.1: Launching the Bastion Host

1. Launch a new standalone EC2 instance named bastion-host.
2. Choose Ubuntu, t2.micro, and your standard key pair.
3. Under Network settings:
   • Select your custom VPC.
   • Place it in a public subnet.
   • Auto-assign public IP: Enable.
4. Attach a security group allowing inbound SSH (Port 22) from your IP address.

Step 7.2: Securely Copying Private Keys to the Bastion Host

To allow the Bastion host to SSH into private instances, you must securely copy your .pem private key file from your local machine to the Bastion host using the Secure Copy Protocol (SCP).

Run this command from your local terminal:

scp -i /path/to/aws-login.pem /path/to/aws-login.pem ubuntu@<BASTION_PUBLIC_IP>:/home/ubuntu/

Step 7.3: Connecting to Private Instances via Bastion

1. SSH into the Bastion host:
   BASH

   Copy

   ssh -i aws-login.pem ubuntu@<BASTION_PUBLIC_IP>
   

2. Once inside the Bastion host, verify the key was transferred (ls).
3. Set proper permissions on the key file:
   BASH

   Copy

   chmod 400 aws-login.pem
   

4. SSH from the Bastion host into a private instance using its private IP address:
   BASH

   Copy

   ssh -i aws-login.pem ubuntu@<PRIVATE_INSTANCE_IP>
   

8. Deploying the Test Application & Load Balancer Integration

Step 8.1: Setting up a Simple Python HTTP Server

Once logged into one of the private EC2 instances, create a basic HTML test page:

cat <<EOF > index.html
<!DOCTYPE html>
<html>
<head><title>AWS Production Demo</title></head>
<body>
  <h1>My first AWS project to demonstrate apps in private subnet</h1>
</body>
</html>
EOF

Start a lightweight Python HTTP server on port 8000:

python3 -m http.server 8000

Demo Note: To clearly observe load balancing behavior later, leave the second private instance without this application running, or deploy a distinct message (e.g., "Second Instance") to observe round-robin routing and health check responses.

Step 8.2: Creating the Application Load Balancer (ALB) & Target Group

1. Navigate to the EC2 Console -> Target Groups -> Create target group:

   • Target type: Instances
   • Target group name: aws-prod-example-tg
   • Protocol: HTTP, Port: 8000
   • VPC: Select your custom VPC.
   • Health check protocol: HTTP

2. Select both private EC2 instances, click Include as pending below, and click Create target group.

3. Navigate to Load Balancers -> Create Load Balancer -> Application Load Balancer:

   • Name: aws-prod-example-alb
   • Scheme: Internet-facing
   • VPC: Select custom VPC.
   • Mappings: Select both Availability Zones and check their respective public subnets.
   • Security groups: Select a security group allowing inbound HTTP (Port 80) from 0.0.0.0/0.
   • Listeners and routing: Listen on Port 80 and forward traffic to aws-prod-example-tg.

4. Click Create load balancer.

9. Real-World Troubleshooting & Production Edge Cases

Issue 1: Load Balancer Listener Port Unreachable (502 / Timeout Errors)

• Symptom: Attempting to access the ALB DNS name results in connection timeouts or target group health check failures.
• Root Cause: The security group attached to the ALB or target instances does not permit inbound traffic on the expected listener port (Port 80 for ALB, Port 8000 for target instances).
• Resolution: Edit the ALB Security Group to explicitly allow inbound HTTP traffic (Port 80) from Anywhere (0.0.0.0/0). Ensure target instance security groups allow incoming traffic from the ALB security group on Port 8000.

Issue 2: Health Check Failures in Target Groups

• Symptom: Target instances show as Unhealthy in the Target Group console.
• Root Cause: The web server process is not running on the instance, or the health check path (/) returns a non-200 HTTP status code.
• Resolution: Verify the HTTP server is actively running (python3 -m http.server 8000) and that an index.html file exists in the root directory.

10. Summary and Key Takeaways

• Production Isolation: Production architectures isolate compute workloads in private subnets across multiple Availability Zones to guarantee high availability and fault tolerance.
• VPC and More Wizard: Streamlines the creation of VPCs, public/private subnets, route tables, Internet Gateways, and NAT Gateways in a single automated step.
• NAT Gateway Role: Enables secure outbound internet access for private instances while masking their internal private IP addresses behind static Elastic IPs.
• Bastion Hosts: Act as secure jump servers (SSH gateways) that allow administrators to manage private subnet resources without exposing them directly to the public internet.
• Auto Scaling Groups (ASG): Automatically manage EC2 instance lifecycles, ensuring application redundancy and dynamic scaling based on demand.
• Application Load Balancers (ALB): Operate at Layer 7 to distribute incoming client traffic across healthy backend targets residing across multiple Availability Zones.
• Target Groups & Health Checks: Continuously monitor application health, routing traffic exclusively to responsive, healthy server instances.
• Security Group Hygiene: Strict adherence to principle-of-least-privilege firewall rules ensures only authorized ports (SSH, HTTP, application ports) remain open between network tiers.