GCP Zero to Hero (Day 8): Production-Grade Cloud & DevOps Architecture on GCP

This comprehensive guide covers the design, provisioning, and deployment of a production-ready, highly secure, and autoscaling enterprise infrastructure on Google Cloud Platform (GCP). Based on a real-world enterprise scenario, this architecture encapsulates private networking, outward egress securing, granular firewall management, automated instance template configuration via systemctl services, stateless Managed Instance Groups (MIG), Regional Application Load Balancing, and DNS mapping.

1. The Production Scenario & Architectural Objective

In an enterprise environment, cloud engineers typically do not receive step-by-step instructions. Instead, they receive high-level requirements—often initiated via a ticket (e.g., Jira)—from development teams asking to securely host a newly built application.

The Real-World Jira Requirement

• Application: Python Flask REST application listening on port 8080.
• Endpoints:
  • /: Primary application response ("Hello from flask on GCP").
  • /health: Health status probe returning HTTP 200 OK.
• Security & Reliability Constraints:
  1. Virtual machines (VMs) hosting the code must not possess public IPv4 addresses (no direct external exposure).
  2. VMs must securely download dependencies (Python packages, system utilities) from the internet without exposing internal IP addresses.
  3. Traffic from end-users must pass through a managed, highly available entry point (Load Balancer).
  4. Infrastructure must automatically adjust instance capacity based on user traffic spikes ($N$ users scaling up to $10^6$ users).

2. End-to-End Request & Security Flow

To satisfy these strict corporate isolation guarantees, traffic flows through a layered enterprise architecture:

3. Custom VPC Design & Subnet Strategy

GCP offers two primary network creation modes: Auto mode and Custom mode.

• Auto Mode: Automatically provisions one subnet per GCP region using pre-defined CIDR blocks. This results in $40+$ default subnets across global regions, creating excessive routing table pollution and unnecessary security exposures for enterprise setups.
• Custom Mode: Empowers network architects to strictly define local subnets, explicitly control IP ranges, and eliminate unnecessary regional subnets.

Subnet Allocation Strategy

To separate distinct architectural responsibilities, we define three discrete subnets within a custom Virtual Private Cloud (prod-vpc):

1. Application Subnet (app-subnet): Hosts the private computing workloads (Compute Engine VMs managed by MIG).
2. NAT Subnet (nat-subnet): Hosts the Cloud NAT gateway service for outbound internet routing.
3. Proxy-Only Subnet (proxy-subnet): Utilized by GCP Regional Envoy-based Application Load Balancers to process and forward HTTP/HTTPS requests to internal backends.

Subnet CIDR Math

For a standard subnet prefix of /24, the number of total available IP addresses is computed as:

In GCP networking, 4 IP addresses are reserved per subnet (network address, gateway address, and two reserved addresses), yielding $256 - 4 = 252$ usable addresses for workloads.

4. Securing Egress Traffic with Cloud NAT

Applications deployed in enterprise app subnets must remain strictly private—meaning they receive no public (external) IPv4 address. However, during initial boot or maintenance cycles, these instances must fetch third-party packages (e.g., pip install flask, apt update).

The Middleman Attack Prevention Model

If a VM is granted a direct public IP, malicious actors can perform port scanning, attempt brute-force SSH attacks, or inspect network traffic.

By routing outbound traffic through a Cloud NAT Gateway:

1. Virtual machines send outbound traffic using their private IP (10.10.0.X) to the Cloud NAT Router.
2. Cloud NAT performs Source Network Address Translation (SNAT), replacing the private source IP with the Cloud NAT external gateway IP.
3. Third-party repositories reply to Cloud NAT, which maps the connection back to the internal VM.

Key Takeaway: External attackers only see the external IP address of the Cloud NAT instance. Because Cloud NAT does not accept unsolicited inbound requests, direct attacks on internal VM workloads are blocked.

5. Granular Network Security via GCP Firewalls

Unlike AWS (which splits security rules into Security Groups and Network Access Control Lists) or Azure (Network Security Groups), GCP uses a unified, global Firewall Rule abstraction attached directly to the VPC network.

To support our Application Load Balancer and internal health checks, we must configure two explicit Ingress firewall rules.

Rule 1: Allow Health Check

Google's managed load balancers continuously test target VMs to confirm application availability. GCP health check probes originate from specific, deterministic Google IP ranges:

• 130.211.0.0/22
• 35.191.0.0/16

Firewall Name  : allow-health-check-8080
Network        : prod-vpc
Direction      : Ingress
Action         : Allow
Source IP Range: 130.211.0.0/22, 35.191.0.0/16
Target         : All instances in the network
Protocol/Port  : TCP: 8080

Rule 2: Allow Proxy Traffic

To permit incoming application requests routed through the Application Load Balancer:

Firewall Name  : allow-proxy-only
Network        : prod-vpc
Direction      : Ingress
Action         : Allow
Source IP Range: 0.0.0.0/0 (or strictly bounded to 10.10.30.0/24)
Target         : All instances in the network
Protocol/Port  : TCP: 8080

6. Application Code & Instance Template Setup

Python Flask Code Base

The development team provided a lightweight Python Flask service containing two routes:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def index():
    return "Hello from flask on GCP", 200

@app.route('/health')
def health():
    return "OK", 200

if __name__ == '__main__':
    # Listen globally on port 8080 inside the container/VM
    app.run(host='0.0.0.0', port=8080)

Automation via Instance Template

An Instance Template is a immutable resource that defines machine properties (vCPU, RAM, boot disk image, network tags, startup scripts) so that scaling groups can launch identical VM instances dynamically in seconds.

Cost Optimization Settings

• Machine Type: e2-micro (2 vCPUs, 1 GB RAM)
• Provisioning Model: Spot Instance (preemptible capacity resulting in up to 60–80% cost reduction, dropping instance costs down to approximately ~$4.14/month).
• External IP: Set strictly to None.

Embedded Startup Script & Systemd Daemon Setup

To make the deployment completely self-healing, the instance startup script writes the source code to disk, provisions an isolated Python virtual environment (venv), installs dependencies, and configures a systemd daemon.

#!/bin/bash
# 1. Update OS package repos & install base tooling via Cloud NAT egress
apt-get update
apt-get install -y python3-pip python3-venv

# 2. Prepare application directory
mkdir -p /app
cat << 'EOF' > /app/main.py
from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello():
    return "Hello from flask on GCP"

@app.route('/health')
def health():
    return "OK", 200

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)
EOF

# 3. Create isolated virtual environment and install Flask
python3 -m venv /app/venv
/app/venv/bin/pip install flask

# 4. Configure systemd unit file for daemon management
cat << 'EOF' > /etc/systemd/system/flask.service
[Unit]
Description=Flask Python GCP Application
After=network.target

[Service]
User=root
WorkingDirectory=/app
ExecStart=/app/venv/bin/python /app/main.py
Restart=always

[Install]
WantedBy=multi-user.target
EOF

# 5. Enable and launch system service
systemctl daemon-reload
systemctl enable flask.service
systemctl start flask.service

7. Auto-Healing & Scaling via Managed Instance Groups (MIG)

A Managed Instance Group (MIG) uses an Instance Template to operate a homogeneous pool of virtual machines.

MIG Workload Classifications

1. Stateful MIG: Used for workloads maintaining persistent local data (e.g., databases like MySQL, PostgreSQL, Redis). IP addresses and attached storage disks are preserved across VM restarts.
2. Stateless MIG: Ideal for web servers and stateless REST APIs (e.g., Flask, Node.js, Go microservices). Instances can be destroyed, recreated, or auto-scaled horizontally without data loss.

Scaling & Healing Configuration

• Group Type: Stateless
• Template: flask-template
• Minimum Replica Bounds: $N_{\text{min}} = 2$
• Maximum Replica Bounds: $N_{\text{max}} = 4$ (or higher based on peak user load)
• Auto-healing Policy: Linked directly to the /health endpoint on TCP port 8080. If a VM fails $3$ consecutive health probes, the MIG automatically terminates and recreates that specific instance.

8. Regional Application Load Balancing (Layer 7)

To serve external end-users, we provision a Regional External Application Load Balancer.

Load Balancer Sub-Components

1. Frontend Configuration:
   • Protocol: HTTP
   • Port: 80
   • IP Address: Ephemeral or Static Public IPv4 Address.
2. Proxy-Only Subnet Allocation:
   • Uses 10.10.30.0/24 to process HTTP connections and pass headers down to backend VMs.
3. Backend Service (mig-backend):
   • Target Backend: MIG-flask-app
   • Target Port: 8080
   • Attached Health Check:
     • Protocol: HTTP
     • Port: 8080
     • Request Path: /health
     • Interval: Every $5$ seconds.

9. Global Domain Mapping via Cloud DNS

External IP addresses (34.X.Y.Z) are difficult for users to memorize. Enterprise setups map friendly hostnames (e.g., walmart.com or abishekviramala.shop) to the Application Load Balancer via GCP Cloud DNS.

Steps for Mapping Domains

1. Create Public Zone in Cloud DNS: Specify your registered DNS name (e.g., example.com).
2. Copy Name Server (NS) Records: Cloud DNS provides 4 authoritative name server addresses:
   • ns-cloud-a1.googledomains.com
   • ns-cloud-a2.googledomains.com
   • ns-cloud-a3.googledomains.com
   • ns-cloud-a4.googledomains.com
3. Update Domain Registrar: Log into your domain registrar (e.g., GoDaddy, Namecheap) and replace default name servers with Google's NS records.
4. Create 'A' Record: Point example.com directly to the Load Balancer's public IPv4 address.

Note on Propagation Delay: Full global DNS propagation across all Tier-1 Internet Service Providers can take up to 24 to 48 hours.

10. Verification, Debugging & Troubleshooting

Once all resources are provisioned, follow this verification workflow to validate deployment integrity:

1. Verification of Private VMs via SSH

Log in to an individual VM instance through GCP Cloud Console (using Identity-Aware Proxy / Internal SSH):

# Verify process status managed by systemd
systemctl status flask.service

# Local application health validation
curl -v http://localhost:8080/
# Expected Output: Hello from flask on GCP (HTTP status 200)

curl -v http://localhost:8080/health
# Expected Output: OK (HTTP status 200)

2. Verification of Load Balancer Ingress

Fetch the public IP address of the newly provisioned Application Load Balancer:

curl http://<LOAD_BALANCER_EXTERNAL_IP>/
# Output: Hello from flask on GCP

3. Failure Domain Testing

Simulate an instance crash or terminate a VM manually in the GCP console:

• Result: The MIG detects the missing instance or health probe failure, automatically launches a new e2-micro instance using flask-template, runs the startup script via Cloud NAT egress, attaches it to the Backend Service, and resumes balancing load—maintaining continuous uptime.

11. Complete Architectural Resource Matrix

The following matrix synthesizes all infrastructure assets configured throughout this enterprise deployment:

Summary and Key Takeaways

• Custom VPC Selection: Always build custom VPC networks in production to explicitly manage IP ranges and prevent automatic generation of unnecessary regional subnets.
• Egress Isolation via Cloud NAT: Secure private workloads by granting no external public IP addresses to individual VMs. Route outbound requests through Cloud NAT to prevent external source address exposure and guard against middleman attacks.
• Granular Firewalls: Restrict ingress access strictly to necessary application ports (e.g., port 8080), and scope Google Health Checks directly to Google's official probe IP blocks (130.211.0.0/22, 35.191.0.0/16).
• Automated Bootstrapping: Embed application installation and startup scripts directly into Instance Templates, managing application runtimes with Linux system daemons (systemd) to ensure reliable operational readiness.
• Stateless Workload Management: Utilize Managed Instance Groups (MIGs) with health checks to achieve continuous application auto-healing and auto-scaling without human intervention.
• Proxy-Only Subnet Obligation: Regional Envoy-based Application Load Balancers in GCP require a dedicated proxy-only subnet (e.g., /24) to negotiate downstream connections to application backends.
• Domain Integration: Map frontend Load Balancer IP addresses to user-facing custom domain names using Cloud DNS A Records and custom Registrar Name Servers.
• Clean Resource Lifecycle Management: When testing demo deployments, disable project billing or delete host projects promptly upon completion to avoid unexpected cloud charges.