Final Results Verification, Log Analysis, and Cost Breakdown

This chapter covers the final phase of our massive load-testing experiment: verifying the results of our 1-million-requests-per-second load test, aggregating distributed logs using AWS Systems Manager (SSM) and Bash scripting, analyzing performance metrics, and breaking down the real-world infrastructure costs of running terabyte-scale experiments in the cloud.

1. The Challenge of Distributed Log Verification

When running high-throughput load tests across dozens of distributed client machines, verifying the final results is not as simple as checking a single console output.

The Setup

• Target Server: A high-performance C++ server built with the Drogon framework.
• Load Generators: 60 distributed EC2 instances ("testers") configured to generate massive traffic.
• Test Duration: 30 minutes of continuous load.
• Target Load: 100,000 concurrent connections per client machine, pushing millions of requests per second.

Each of the 60 tester machines saves its individual execution metrics locally to a file named results.txt. To analyze the experiment as a cohesive whole, we must aggregate these isolated files from across the AWS fleet without manually logging into 60 separate terminals.

2. Aggregating Logs via AWS SSM and S3

To collect the distributed results.txt files efficiently, we leverage AWS Systems Manager (SSM) and Amazon S3. This approach avoids the need to manage SSH keys or write complex multi-threaded SSH scripts.

Distributed Testers (60 EC2 Instances)
   |
   | 1. Run 'cat results.txt' via AWS SSM
   v
AWS Systems Manager (SSM)
   |
   | 2. Stream standard output (stdout)
   v
Amazon S3 Bucket (Organized by Command ID)
   |
   | 3. Sync to local admin machine
   v
Local Administrator Machine

Step 1: Execute Remote Commands with AWS SSM

We dispatch a command to all instances tagged with the name tester to read their local results.txt file and stream the output directly to an S3 bucket.

# Execute remote command across all tester instances and save output to S3
aws ssm send-command \
    --document-name "AWS-RunShellScript" \
    --targets "Key=tag:Name,Values=tester" \
    --parameters 'commands=["cat results.txt"]' \
    --output-s3-bucket-name "my-load-test-results-bucket" \
    --query "Command.CommandId" \
    --output text

Step 2: Sync S3 Results to the Local Machine

Once the SSM command completes, the output of each instance is stored in S3 under a folder named after the unique SSM Command ID (e.g., starting with 97...). We sync this remote S3 directory to our local workspace:

# Sync the S3 command output folder to a local directory named 'results'
aws s3 sync s3://my-load-test-results-bucket/97-command-id-here/ ./results

This downloads a clean, nested directory structure containing the standard output (stdout) of each of the 60 instances.

3. Harnessing Bash Scripting for Data Synthesis

With 60 individual output files downloaded locally, we use native Unix command-line utilities to concatenate, clean, and verify our dataset.

Concatenating Distributed Outputs

We use the find command to locate all stdout files within our nested results directory and concatenate them into a single consolidated file:

# Find all stdout files and concatenate them into a single master results file
find ./results -type f -name "stdout" -exec cat {} + > 30min-60instances-results.txt

Verifying Dataset Integrity

To ensure that we successfully collected results from all 60 load-testing instances, we search for the keyword read (which appears once per successful test summary line) and count the lines using wc -l:

# Count the number of successful test summaries in our master file
grep "read" 30min-60instances-results.txt | wc -l

Result: The output returns exactly 60, confirming that no tester logs were lost or dropped during the aggregation process.

4. Performance Metrics & Log Analysis

Analyzing the consolidated log file reveals the sheer scale of traffic handled by our C++ Drogon server.

Error and Timeout Analysis

At this extreme scale, network timeouts are the primary point of failure. We query our master log file to count the total number of connection timeouts:

# Search for timeouts or connection errors in the results
grep -i "timeout" 30min-60instances-results.txt | wc -l

• Total Timeouts: 40
• Context: Out of billions of requests transmitted over 100,000 concurrent connections, only 40 timed out. This represents an exceptionally high success rate, demonstrating the stability of the Drogon C++ network stack under extreme load.

Aggregating Total Requests and Data Volume

To calculate the total requests sent and the total volume of data transferred across all 60 instances, we write an awk script. This script parses the metrics from each log line, sums them up, and formats the output:

# Parse and sum total requests and data transferred
awk '
/read/ {
    # Assuming $1 is the request count in thousands (K) and $5 is the data in GB
    req += $1
    data += $5
}
END {
    # Convert requests to Billions and data to Terabytes (TB)
    printf "Total Requests: %.2f Billion\n", req / 1000000
    printf "Total Data Read: %.2f TB\n", data / 1024
}' 30min-60instances-results.txt

Final Test Metrics (30-Minute Run)

5. Real-World Infrastructure Cost Breakdown

Running high-throughput, terabyte-scale load tests in public clouds like AWS incurs significant costs. Moving terabytes of data per minute requires substantial compute, database, and network resources.

The total cost for the month of testing and research was approximately $2,000.

Cost Distribution

• Database Infrastructure ($800): Provisioning high-IOPS databases to handle persistent state, write-heavy workloads, and concurrent connections during the research and testing phases.
• EC2 Compute Fleet ($1,200): The cost of running the 60 distributed load-testing instances alongside the high-performance target server.

Lessons Learned for Cost Optimization

1. Clean Up Unused Resources: A portion of the cost was driven by minor configuration oversights (e.g., leaving high-performance instances or provisioned IOPS volumes running longer than necessary).
2. Use Spot Instances for Testers: For load generators, leveraging AWS Spot Instances instead of On-Demand instances can reduce compute costs by up to 90%.
3. Minimize Data Transfer Costs: Data transfer out of AWS or across Availability Zones (AZs) can quickly become expensive at terabyte scale. Keeping load generators and target servers within the same AZ minimizes cross-AZ data transfer fees.

6. The Limits of AI at Extreme Scale

During the preparation and execution of this 1-million-RPS test, several highly specialized infrastructure and networking bottlenecks emerged.

• The Support Experience: When troubleshooting these extreme scale issues, AWS Business Support was engaged. They ran specialized AI diagnostics to resolve the bottlenecks.
• AI Limitations: The AI diagnostic tool provided approximately 10 architectural and configuration suggestions, none of which resolved the issue.
• Why AI Struggled: Large Language Models (LLMs) and diagnostic AIs are trained on public datasets, forums, and documentation. Because only a tiny fraction of global technology companies operate at a scale of 1 million requests per second, there is very little training data available for these edge cases.
• The Human Element: The issue was ultimately resolved through deep, manual systems engineering and direct collaboration with senior AWS support engineers. At this level of engineering, deep domain expertise and structured systems thinking remain irreplaceable.

Summary and Key Takeaways

• Distributed Log Collection: AWS SSM allows you to run commands across large fleets of instances simultaneously, streaming outputs directly to S3 for centralized access.
• Power of Command-Line Utilities: Standard Unix tools like find, grep, awk, and wc easily process and analyze gigabytes of log data without requiring custom analysis scripts.
• Astounding Performance: The C++ Drogon server successfully processed 2 billion requests and transferred over 60 terabytes of data in just 30 minutes, with only 40 timeouts.
• High-Throughput Costs: Running massive load tests is expensive; our testing and research phase cost **$2,000**, split between EC2 compute ($1,200) and database resources ($800).
• AI Limits at Scale: AI models struggle with ultra-high-throughput networking and infrastructure issues due to a lack of training data at the 1-million-RPS scale.
• Real-World Engineering: Building systems for real production traffic requires a deep understanding of systems programming, network protocols, and cost management.