Distributed Load Fleet: Orchestrating 60 Load Generators

This chapter explores the engineering challenges and architectural solutions involved in orchestrating a distributed fleet of 60 load-generation servers. It covers transitioning from manual terminal-based execution to automated, synchronized scripting, leveraging AWS Systems Manager (SSM) for concurrent execution, and navigating the physical capacity limits of AWS Network Load Balancers (NLBs) when pushing millions of requests per second.

1. The Problem: Why Ad-Hoc Load Generation Fails at Scale

When attempting to generate 1 million requests per second (RPS), running load-generation tools (like Autocannon) manually via individual terminals or uncoordinated scripts fails due to two primary issues:

• Desynchronization (The Staggering Problem): If the 60 load-testing instances do not start sending traffic at the exact same millisecond, the target server experiences a staggered, rolling load rather than a concentrated peak. This prevents the system from reaching its true saturation point.
• Terminal Overhead and Crash Vulnerability: Running Autocannon directly inside a standard interactive terminal session can cause local buffer overflows, SSH disconnects, or incomplete log captures.

Real-Life Analogy

Imagine trying to break down a heavy security door with 60 people. If each person runs up and hits the door one by one (staggered), the door easily absorbs the impact. To break the door, all 60 people must link arms and charge at the exact same second.

2. The Solution: Programmatic Autocannon & Exact Time Logging

To solve the synchronization problem, the manual terminal execution of Autocannon was replaced with a programmatic Node.js script (patch.js). This script wraps the Autocannon engine, dynamically accepts parameters from the command line, and appends precise execution metadata to a local log file.

The Programmatic Wrapper (patch.js)

const autocannon = require('autocannon');
const fs = require('fs');

// Retrieve arguments from CLI: node patch.js <connections> <duration> <host>
const args = process.argv.slice(2);
const connections = parseInt(args[0], 10) || 10;
const duration = parseInt(args[1], 10) || 20;
const host = args[2] || 'http://localhost:8080';

const config = {
  url: host,
  connections: connections,
  duration: duration,
  headers: {
    'm-connection': 'keep-alive'
  }
};

const startTime = new Date().toISOString();
console.log(`[${startTime}] Launching load test against ${host}...`);

autocannon(config, (err, result) => {
  if (err) {
    console.error('Error running Autocannon:', err);
    process.exit(1);
  }

  const endTime = new Date().toISOString();
  
  // Format output results
  const output = `
==========================================
TEST RUN METADATA
==========================================
Start Time: ${startTime}
End Time:   ${endTime}
Target Host: ${host}
Connections: ${connections}
Duration:    ${duration} seconds

==========================================
AUTOCANNON RESULTS
==========================================
Requests/Sec: ${result.requests.average}
Transfer/Sec: ${result.throughput.average} bytes
Latency (Avg): ${result.latency.average} ms
Errors:        ${result.errors}
timeouts:      ${result.timeouts}
==========================================
\n`;

  // Write results to a local file
  try {
    fs.appendFileSync('/var/log/results.txt', output);
    console.log(`[${endTime}] Test completed successfully. Results saved to /var/log/results.txt`);
  } catch (writeErr) {
    console.error('Failed to write results to file. Ensure you have sudo privileges.', writeErr);
  }
});

3. Orchestrating 60 Servers Simultaneously via AWS SSM

To control 60 EC2 instances simultaneously without SSHing into each one individually, we use AWS Systems Manager (SSM) Run Command.

Critical AWS SSM Parameters

• AWS-RunShellScript: The document name used to execute shell commands on remote managed instances.
• max-concurrency 100%: By default, AWS SSM throttles command delivery (e.g., sending to 50 instances first, waiting, then sending to the next). Setting this to 100% forces SSM to dispatch the command to all 60 instances at the exact same instant.
• S3 Output Delivery: Configures the instances to write their stdout/stderr logs directly to a centralized Amazon S3 bucket.
• Amazon CloudWatch Logs: Streams real-time console output from all 60 instances to a single CloudWatch log group.
• IAM (Identity and Access Management): The security framework that grants the EC2 instances permission to write to S3 and CloudWatch without hardcoding access keys.

The Orchestration Command

# Execute the load test across all 60 instances simultaneously
aws ssm send-command \
    --document-name "AWS-RunShellScript" \
    --targets "Key=tag:Role,Values=LoadGenerator" \
    --parameters 'commands=["cd /home/ubuntu/1M-RPS-tester","sudo node patch.js 10000 1800 http://target-server-ip:8080"]' \
    --max-concurrency "100%" \
    --output-s3-bucket-name "my-load-test-results-bucket" \
    --cloud-watch-output-config '{"CloudWatchLogGroupName":"aws/ssm/benchmarks","CloudWatchOutputEnabled":true}' \
    --query "Command.CommandId" \
    --output text

4. Real-Time Monitoring and Emergency Cancellation

When running large-scale distributed tests, things can go wrong quickly (e.g., misconfigured target IPs, incorrect durations). You must be able to monitor the fleet in real time and cancel operations instantly.

Real-Time Log Tailing

Using the AWS CLI, you can aggregate and tail the stdout of all 60 servers directly in your local terminal:

aws logs tail aws/ssm/benchmarks --follow

Emergency Cancellation Workflow

If a test is misconfigured (for example, setting a duration of 3 minutes instead of 30 minutes), retrieve the CommandId returned by the execution command and cancel it immediately:

# Cancel the active SSM command across the entire fleet
aws ssm cancel-command --command-id "12345678-1234-1234-1234-1234567890ab"

5. The Load Balancer Bottleneck: Theory vs. Production Reality

To scale beyond a single target server, a common architectural pattern is to place a Network Load Balancer (NLB) in front of multiple target servers.

While NLBs operate at the transport layer (TCP Layer 4) and are advertised to handle millions of requests per second, they exhibit severe performance degradation out of the box when subjected to sudden, massive traffic spikes.

The AWS Support Discovery: Load Balancer Capacity Units (LCUs)

During testing, introducing an NLB dropped throughput from 1 million RPS to a mere 5 Gbps (a massive degradation).

AWS Support confirmed that the NLB hit its Consumed Load Balancer Capacity Unit (LCU) limit. NLBs scale elastically, but they do so gradually. When hit with an instantaneous burst of 1 million RPS, the NLB cannot scale fast enough, resulting in dropped packets and severe throttling.

Solution: Pre-Warming / Reserving Capacity

To handle enterprise-grade traffic spikes, you must contact AWS Support in advance to pre-warm the load balancer or reserve capacity by specifying:

1. Target Gbps (e.g., 300 to 600 Gbps)
2. Expected Connection Count (e.g., 100,000+ concurrent connections)
3. Availability Zones (AZs) (distributing the capacity across specific physical datacenters)

6. Comparison: Direct Target vs. Un-Warmed Network Load Balancer

Summary and Key Takeaways

• Programmatic Wrappers are Essential: Do not run raw CLI load tools over SSH for large-scale tests. Wrap them in scripts that handle errors, format outputs, and log precise start/end timestamps.
• Enforce 100% Concurrency: When using AWS SSM to orchestrate load fleets, always set --max-concurrency "100%" to bypass default rolling deployments and ensure all generators strike the target simultaneously.
• Centralize Your Logs: Use IAM roles to allow EC2 instances to stream logs directly to CloudWatch and S3, enabling real-time fleet monitoring via aws logs tail.
• Load Balancers Have Limits: Network Load Balancers (NLBs) cannot handle instantaneous, massive traffic spikes out of the box. They require manual pre-warming and LCU reservation via AWS Support.
• Scale Costs Money: Running a 60-server load fleet moving hundreds of terabytes of data over several hours generates significant bandwidth and infrastructure costs. Plan and coordinate your test windows carefully.