CPU Core Saturation and Multithreading Principles

To build systems capable of handling high-throughput workloads (such as 1 million requests per second), you must understand how software interacts with the underlying hardware. At this scale, performance tuning is impossible without continuous, precise resource monitoring.

This chapter explores how CPU cores execute instructions, how operating systems calculate core and CPU utilization, and how single-threaded versus multithreaded execution models impact hardware saturation.

1. Core Utilization: The Fundamental Formula

At the hardware level, a Central Processing Unit (CPU) consists of one or more physical cores. Each core is a discrete processing unit designed to execute instructions (such as arithmetic operations, logical comparisons, or memory reads/writes).

At any given instant, a CPU core is in one of two states:

• Active: Executing instructions.
• Idle: Doing absolutely nothing (sitting idle, waiting for interrupts or new instruction cycles).

To measure how heavily a single core is being worked over a specific time window, operating systems calculate Core Utilization using the following formula:

Real-Life Analogy

Think of a CPU core as a kitchen chef. If you monitor the chef over a $60$-minute dinner service ($\text{Total Time}$), and find that they spent a combined total of $30$ minutes leaning against the counter waiting for orders ($\text{Total Idle Time}$), their utilization for that hour was:

2. CPU Utilization: The Two Reporting Methods

Because modern computers feature multi-core processors, operating systems must aggregate individual core metrics to report overall system CPU utilization. There are two primary methods used by different operating systems and monitoring tools to display this metric.

Mathematical Comparison

Consider a system with 4 CPU cores where all 4 cores are running at $100\%$ capacity:

• Method 1 (Cumulative): $100\% + 100\% + 100\% + 100\% = \mathbf{400\%}$ CPU Utilization.
• Method 2 (Normalized): $\frac{100\% + 100\% + 100\% + 100\%}{4} = \mathbf{100\%}$ CPU Utilization.

3. Single-Threaded Execution and Core Limits

A thread is the smallest sequence of programmed instructions that can be managed independently by an operating system scheduler.

The Core Constraint

Rule of Thread Scheduling: A single thread can only execute on one CPU core at any given instant of time.

If you write a program that runs on a single thread and contains an infinite loop (e.g., while(true)), that thread will continuously request instruction cycles from the OS scheduler. The OS will assign it to a core, instantly saturating that single core to $100\%$ utilization.

However, because the program is single-threaded, it cannot spill over to other cores. The remaining cores on your system will stay idle unless other processes utilize them.

4. Code Demonstration: Single-Threaded Saturation

To observe this behavior, we can write a simple JavaScript script (singlethread.js) executed via Node.js. Node.js runs on a single-threaded event loop for executing user code.

Naive Single-Threaded Code (singlethread.js)

// This infinite loop will run continuously, pinning a single thread to 100% of a core.
while (true) {
    // The CPU core is forced to execute this jump/evaluation instruction repeatedly
}

Execution and Monitoring Workflow

1. Run the script in your terminal:
   BASH

   Copy

   node singlethread.js
   

2. Open your operating system's resource monitor:
   • macOS: Activity Monitor
   • Windows: Task Manager
   • Linux: System Monitor or top
3. Search for the node process and inspect its CPU usage:
   • On macOS (using Method 1), you will see the process consuming exactly 100% CPU.
   • On a Windows machine with 4 cores (using Method 2), you will see the process consuming 25% CPU.

5. Multithreaded Execution and Core Saturation

To perform multiple tasks simultaneously and utilize the full computing power of a multi-core machine, you must write multithreaded code.

If you spawn as many active threads as your machine has physical cores, and each thread runs an intensive computation (like a while(true) loop), you will saturate every core on the processor.

6. Code Demonstration: Multithreaded Saturation

The following Node.js script (multithread.js) uses the native worker_threads module to spawn multiple parallel execution threads.

To run this successfully, determine your CPU's core count and set the THREAD_COUNT variable accordingly.

Multithreaded Code (multithread.js)

const { Worker, isMainThread, parentPort, workerData } = require('worker_threads');

// Define the number of threads based on your CPU cores (e.g., 12 cores)
const THREAD_COUNT = 12; 

if (isMainThread) {
  console.log(`Master thread running. Spawning ${THREAD_COUNT} worker threads...`);
  
  // Spawn workers
  for (let i = 0; i < THREAD_COUNT; i++) {
    new Worker(__filename, {
      workerData: i
    });
  }
} else {
  // This code runs inside each individual worker thread
  console.log(`Worker thread ${workerData} started.`);
  
  // Infinite loop to saturate the assigned CPU core
  while (true) {
    // Continuous execution
  }
}

Execution and Monitoring Workflow

1. Run the multithreaded script:
   BASH

   Copy

   node multithread.js
   

2. Inspect your system's resource monitor:
   • Process CPU Usage: On a 12-core macOS system, the node process will show a CPU usage of 1200% (or slightly lower, such as 900% to 1100%, if other processes like video recording or system tasks are competing for CPU cycles).
   • Total System Idle CPU: The overall system Idle metric will drop to 0%. Every core is fully saturated, leaving no breathing room for the processor.

7. Real-World Pitfalls of Multithreading

While multithreading allows you to maximize hardware utilization, writing multithreaded applications introduces complex software engineering challenges:

• Race Conditions: Occur when multiple threads attempt to read and write to a shared memory location simultaneously, leading to unpredictable data corruption.
• Synchronization Overhead: To prevent race conditions, developers must use synchronization primitives like mutexes, semaphores, or locks. However, excessive locking can cause threads to block each other, degrading performance and potentially causing deadlocks (where threads wait on each other indefinitely).
• Context Switching: If you spawn significantly more active threads than the number of physical CPU cores, the operating system must constantly swap threads in and out of the cores. This process, known as context switching, consumes CPU cycles and degrades overall application throughput.

Summary and Key Takeaways

• Resource Monitoring is Mandatory: When designing systems for extreme scale (e.g., 1 million requests per second), continuous resource monitoring is the only way to identify bottlenecks.
• Core Utilization Formula: Calculated by subtracting idle time from total time, dividing by total time, and multiplying by 100.
• Method 1 CPU Reporting: Sums up all core percentages. A fully saturated 12-core system will report $1200\%$ CPU usage.
• Method 2 CPU Reporting: Normalizes the sum across all cores, capping the maximum reported CPU usage at $100\%$.
• Single-Thread Constraint: A single thread can only run on one core at a time. A single-threaded application can never utilize more than $100\%$ of a single core's capacity.
• Hardware Saturation: To achieve maximum hardware throughput, an application must run multiple threads in parallel to distribute the workload across all available physical CPU cores.
• Concurrency Trade-offs: Multithreading introduces complex engineering challenges, including race conditions, synchronization overhead, and context switching, which must be carefully managed.