Cloud Load Generation and Network Bottlenecks

Handling massive request throughput at the scale of one million requests per second ($1\text{M RPS}$) introduces infrastructural and networking challenges that go far beyond standard application-level optimizations. When pushing a cloud-hosted infrastructure to its theoretical limits, resource saturation shifts unexpectedly from CPU compute to physical network interface bandwidth, exposing steep architectural trade-offs and astronomical financial costs.

1. High-Throughput Test Architecture Setup

To test high-scale concurrency without the variability of public internet transit speeds, the infrastructure is isolated entirely within a cloud provider's high-speed private Virtual Private Cloud (VPC).

Isolated Private Network Topology

• Network Isolation: Operating on a private VPC eliminates public gateway throttling, routing overhead, and internet speed bottlenecks.
• Database Connection: PostgreSQL runs as an independent tier attached directly to the application host via private networking.
• Scale Goal: Evaluate system behavior and identify primary hardware/network bottlenecks when pushing throughput past $1\text{M RPS}$.

2. Server Cluster Initialization and Baseline Verification

To fully utilize a massive multi-core server instance, the application must run concurrently across all physical and virtual cores. Node.js operates on a single-threaded event loop by default, necessitating a process manager such as PM2 configured in cluster mode.

# Start 128 Node.js processes across all available vCPUs
pm2 start ecosystem.config.js

Baseline Endpoint Verification

Before starting automated load generation, a manual health check verifies baseline functionality:

• Route Tested: GET /simple
• Response Payload: {"message": "hi"}
• Baseline Health: All 128 worker processes register $0\%$ CPU utilization at idle.

3. Benchmarking the Minimal /simple Endpoint

To measure maximum raw compute capacity, load generation begins against the lightweight /simple JSON route using Autocannon.

# Monitor CPU metrics with 1-second intervals
mpstat 1
# Or using a custom alias:
cpu-usage

Autocannon Stress Test Parameters

autocannon -m GET \
  -c 1000 \
  -d 20 \
  -p 100 \
  -w 120 \
  http://<power-server-private-ip>:3000/simple

• -c 1000: 1,000 concurrent TCP connections
• -d 20: 20-second test duration
• -p 100: Pipelining factor of 100 requests per connection
• -w 120: 120 worker threads distributed on the load tester

Test Results and Resource Utilization

• Throughput Achieved: $\approx 6,000,000\text{ RPS}$ ($6\text{M RPS}$).
• Server CPU State: Idle metric dropped to $0\%$ (100% full CPU saturation).
• Load Generator CPU State: Remained with $\approx 50\%$ idle capacity, proving the load generator was not the bottleneck.
• Financial Cost Context: Sustaining this hardware footprint costs approximately $\char36 5,000/\text{month}$.

4. Workload Complexity and Network Saturation

Transitioning from a zero-logic endpoint to a complex PATCH endpoint immediately shifts the bottleneck from CPU processing power to network bandwidth.

# Executing heavy PATCH load test
autocannon -m PATCH \
  -c 500 \
  -d 20 \
  -p 50 \
  -w 120 \
  http://<power-server-private-ip>:3000/patch-endpoint

Observed Anomalies

• Throughput Collapse: Throughput dropped from $6,000,000\text{ RPS}$ to approximately $100,000\text{ RPS}$.
• Low CPU Utilization:
  • Tester Host: $\approx 80\%$ idle
  • Application Server Host: $\approx 50\%$ idle
• Core Question: Why did throughput plummet by $98.3\%$ while the CPU remained half idle?

5. Identifying the Physical Network Bottleneck

Analyzing the data transfer metrics from the 20-second test output uncovers the real bottleneck:

Converting gigabytes per second to gigabits per second ($1\text{ Byte} = 8\text{ bits}$):

Note: Even though the server had substantial idle CPU capacity, the instance's 50 Gbps physical Network Interface Controller (NIC) was completely saturated. The application was bandwidth-bound, not compute-bound.

6. Scaling Network Capacity: Tiers and Infrastructure Costs

Overcoming a network interface bottleneck requires vertically scaling the instance type to access wider network pipes.

Diminishing Returns of Vertical Scaling

• Upgrading to a $100\text{ Gbps}$ tier doubles theoretical throughput from $100\text{k RPS}$ to $\approx 200\text{k RPS}$, but still falls far short of the $1\text{M RPS}$ objective for payload-heavy requests.
• Reaching $1\text{M RPS}$ for data-intensive routes on single-node or tightly coupled clusters requires upwards of $300\text{ to }3,000\text{ Gbps}$, resulting in exorbitant infrastructure bills.

7. The Macro Economics of $1\text{M RPS}$

Sustaining a consistent $1,000,000\text{ requests per second}$ continuously across standard pay-per-use public APIs or cloud primitives incurs massive financial costs.

• Micro-Costs Compound Exponentially: Even a modest rate of $\char36 0.90\text{ per } 1\text{M requests}$ results in millions of dollars in monthly operational costs at full scale.
• Payload Size Direct Impact: Payload sizes directly dictate network costs. An endpoint returning a few kilobytes per response will saturate gigabit connections orders of magnitude faster than a minimal JSON status flag.

Summary and Key Takeaways

• Private VPC Isolation: Stress testing extreme throughput requires private network topologies to eliminate public internet gateway throttling and unpredictable latency.
• Compute vs. Data Payload Dynamics: A minimal JSON endpoint (/simple) achieved $6\text{M RPS}$ at $100\%$ CPU usage, whereas a data-rich PATCH route choked at $100\text{k RPS}$ with $50\%$ idle CPU due to network saturation.
• Network Interface Saturation: Generating $5.95\text{ GB/s}$ of throughput exhausts a $50\text{ Gbps}$ network card ($47.6\text{ Gbps}$ consumed), halting performance regardless of spare CPU or RAM capacity.
• Process Clustering via PM2: Single-threaded runtimes like Node.js require full multi-process orchestration (e.g., 128 instances) across all physical vCPUs to achieve maximum compute saturation.
• Cost Realities of Extreme Throughput: Scaling physical hardware from $50\text{ Gbps}$ to $3,000\text{ Gbps}$ can push dedicated hosting bills from $\char36 5,000/\text{month}$ to over $\char36 30,000/\text{month}$.
• Scale Math: Sustaining $1\text{M RPS}$ translates to roughly $2.59\text{ trillion requests per month}$, turning even tiny fractions of a cent per request into massive operational expenses.