Surpassing 1M RPS: Native C++ Benchmarking

This chapter explores the engineering challenges, infrastructure design, and benchmarking strategies required to push past the 1 million Requests Per Second (RPS) barrier. When operating at this scale, standard benchmarking setups break down. We will examine how to identify client-side bottlenecks, coordinate a distributed load-testing cluster, and manage the system limits of high-performance network environments.

1. The 30% Idle CPU Bottleneck

During single-instance high-performance testing, a common engineering roadblock emerges: the target server's CPU utilization plateaus (e.g., hovering around 70% utilization, leaving 30% idle), yet the system cannot process more requests.

The Diagnostic Dilemma

When a high-performance server (such as a native C++ web server) fails to reach 100% CPU utilization under load, the bottleneck typically lies in one of three areas:

1. Cloud Provider Network Limits: Cloud providers like AWS impose strict limits on Packets Per Second (PPS) and bandwidth (e.g., 300 Gbps limits) at the virtual network interface (vNIC) level.
2. Operating System Limits: The Linux kernel on either the server or the tester machine may restrict performance due to default TCP stack configurations, open file descriptor limits (ulimit -n), or local port range exhaustion.
3. Client-Side Generator Exhaustion: A single benchmarking instance (the "tester") runs out of resources. It cannot open connections, manage timers, or parse responses fast enough to saturate a highly optimized server.

2. The Client-Side Exhaustion Problem

To generate 1 million requests per second, the benchmarking client must work just as hard as the server. When using a single "beast" instance as a load generator, the client inevitably hits a physical wall.

Real-Life Analogy

Think of a high-speed sorting warehouse (the Server) capable of processing 10,000 packages per minute. If you only hire a single delivery driver (the Client) to bring packages to the warehouse, the warehouse will sit idle. The driver cannot load, drive, and unload packages fast enough to keep up with the warehouse's capacity. To test the warehouse's true limits, you must deploy a coordinated fleet of delivery trucks arriving simultaneously.

Why Single-Tester Benchmarking Fails

• Ephemeral Port Exhaustion: A single IP address only has 65,535 available outbound ports. When connections are opened and closed rapidly, ports enter the TIME_WAIT state, preventing immediate reuse and halting new connection requests.
• Single-Threaded Event Loop Saturation: Many modern benchmarking tools (such as Autocannon or wrk) rely on single-threaded event loops. Even on a multi-core machine, the tool's event loop can saturate a single CPU core, capping the maximum request generation rate.

3. Distributed Load Testing Architecture

To bypass client-side bottlenecks and drive the target server's CPU idle capacity down to 0%, we must transition from a single load generator to a Distributed Load Testing Cluster.

Infrastructure Specifications

• Target Server: A high-compute, network-optimized instance (e.g., AWS EC2 c8gn series).
• Distributed Tester Nodes: 60 x c8gn.2xlarge instances.
  • vCPUs per Node: 8 Cores
  • RAM per Node: 16 GB
  • Placement Group: Cluster placement group (ensures low-latency, high-throughput physical proximity within the AWS data center).
• Orchestration Layer: A centralized control script that triggers load generation across all 60 nodes simultaneously and aggregates the performance metrics.

4. Overcoming Cloud Provider Quotas

Deploying massive distributed infrastructure requires navigating cloud provider resource limits. By default, cloud accounts have protective limits to prevent runaway costs and resource abuse.

5. Orchestrating the 60-Node Cluster

Manually configuring and running benchmarks on 60 individual servers is impractical. We use an automated Bash script to orchestrate the load generators. The script connects to each tester node via SSH, triggers the load generator (autocannon), and pipes the results back to a central controller.

Distributed Execution Script (run-benchmark.sh)

#!/bin/bash

# List of IP addresses of the 60 tester instances
TESTER_IPS=(
  "192.168.1.101"
  "192.168.1.102"
  # ... truncated for readability ...
  "192.168.1.160"
)

TARGET_URL="http://192.168.1.50:8080/hello"
CONNECTIONS=400
PIPELINING=5
DURATION="1h" # 1-hour stress test

echo "Launching distributed benchmark across ${#TESTER_IPS[@]} nodes..."

for ip in "${TESTER_IPS[@]}"; do
  # Run autocannon in the background on each remote node
  ssh -o StrictHostKeyChecking=no ubuntu@"$ip" \
    "autocannon -c $CONNECTIONS -p $PIPELINING -d $DURATION $TARGET_URL" > "results_$ip.log" 2>&1 &
done

# Wait for all background SSH processes to complete
wait
echo "All benchmark runs completed. Aggregating logs..."

Note: Autocannon outputs its progress and metrics to stderr by default. When redirecting output to files, ensure you capture both standard output and standard error using the 2>&1 redirection operator.

6. Mathematical Model of the Load Test

To understand the sheer volume of concurrent traffic hitting the server, we can calculate the total active requests in flight at any given millisecond.

The Formula

Where:

• $N_{\text{servers}}$ = Number of distributed tester nodes ($60$)
• $C_{\text{connections}}$ = Connections opened per tester ($400$)
• $P_{\text{pipelining}}$ = Number of pipelined requests per connection ($5$)

Calculation

This means that at any single point in time, the C++ server's network buffer and event loop are actively processing 120,000 concurrent requests, translating to over 1,000,000 requests completed per second.

7. Real-World Pitfalls: The 1-Hour Log Failure

During long-running, high-throughput benchmarks, unexpected failure modes can occur. A key example is the 1-Hour Log Failure, where short tests (e.g., 20 seconds to 30 minutes) return clean metrics, but a continuous 1-hour run results in corrupted, missing, or truncated log outputs.

Root Cause Analysis

1. SSH Session Timeouts: Long-running SSH connections without keep-alive packets can be silently dropped by intermediate firewalls or routers, terminating the log collection stream before the test finishes.
2. Buffer Overflow: Autocannon generates continuous progress updates. Over a 1-hour period, piping this high-frequency output over SSH can overwhelm terminal buffers or local disk storage, leading to truncated files.
3. Memory Exhaustion on Testers: Accumulating raw JSON metrics in the memory of the tester nodes over an hour can cause the benchmarking process to be terminated by the Linux Out-Of-Memory (OOM) killer.

Mitigation Strategies

• Write Logs Locally: Configure the load testing tool to write results to a local file on the tester node first, then pull the completed files using scp or rsync after the test finishes.
• Enable SSH Keep-Alives: Configure your SSH client to send periodic keep-alive packets to prevent connection drops:
  BASH

  Copy

  ssh -o ServerAliveInterval=60 ubuntu@$IP
  

• Reduce Log Verbosity: Run the benchmarking tool in quiet or silent mode to minimize the volume of log data written to disk.

Summary and Key Takeaways

• Client-Side Bottlenecks: A single load generator cannot saturate a highly optimized native C++ server. The client will hit OS, port, or CPU limits first, leaving the server partially idle.
• Distributed Testing: To achieve true stress testing at scale, you must distribute the load across a cluster of coordinated, smaller instances (e.g., 60 x c8gn.2xlarge nodes).
• Placement Groups: Always launch benchmarking instances within a cluster placement group to minimize network latency and maximize throughput between nodes.
• AWS Resource Quotas: High-scale testing requires proactive planning. You must request vCPU limit increases (e.g., raising limits from 32 to 800+ cores) from your cloud provider ahead of time.
• Pipelining Power: Combining distributed nodes with connection pipelining allows you to maintain over 120,000 concurrent requests in flight, simulating millions of active users.
• Log Management: Long-running benchmarks (such as 1-hour stress tests) are prone to SSH timeouts and log corruption. Always write benchmark outputs to local files on the tester nodes and retrieve them after the run completes.