Concurrency & Parallelism: IO Bound vs CPU Bound

This video provides a comprehensive mental model for how backend systems handle multiple simultaneous requests, focusing on the differences between concurrency and parallelism, and explaining the underlying mechanisms such as threads, event loops, and virtual threads (goroutines). It emphasizes the importance of understanding I/O-bound vs CPU-bound workloads to design efficient backend architectures.

1. The Need for Concurrency in Backend Systems

Backend servers must handle multiple user requests concurrently; otherwise, processing one request at a time leads to significant delays or dropped requests under load.

• Synchronous, single-request processing causes the CPU to remain idle during I/O operations like database queries, leading to resource waste.
• Modern CPUs can execute billions of instructions per second, and waiting for I/O means underutilizing this power.
• Typical API calls involve multiple I/O operations (database queries, external APIs, caching), which can amount to hundreds of milliseconds waiting.
• During this waiting, CPU usage is minimal (e.g., 10 ms CPU work vs 250 ms waiting), meaning 95% of server resources can be idle without concurrency.

Key insight: Over 70% of backend operations are I/O-bound, waiting on network or file system responses, making concurrency critical for efficient resource use.

2. Concurrency vs Parallelism: Defining the Terms

• Concurrency is about structuring a program to deal with multiple things at once by rapidly switching between tasks, even on a single CPU core.
• Parallelism means actually executing multiple instructions simultaneously, requiring multiple CPU cores.

Comparison Table

• Concurrency is about managing tasks so that CPU is not idle while waiting for I/O.
• Parallelism requires hardware that supports simultaneous execution and is ideal for CPU-bound tasks.

3. Typical Backend Request Timeline (Single-Core CPU Example)

• Requests arrive nearly simultaneously (Request A, Request B).
• Request A uses CPU for initial processing (validation, routing).
• When Request A performs an I/O wait (e.g., database query), the CPU switches to Request B.
• Request B uses CPU for some processing, then hits I/O wait.
• CPU switches back to Request A once the database response arrives.
• This cycle continues, interleaving processing so the CPU is never idle.

Conclusion: Even with a single core, concurrency efficiently utilizes CPU by switching between tasks during I/O waits. Each request appears to progress simultaneously from the client’s perspective.

4. I/O Bound vs CPU Bound Workloads

• Backend services mostly deal with I/O-bound workloads, so concurrency models are more important.
• CPU-bound tasks benefit from parallelism to utilize multiple cores efficiently.
• Understanding what workload type dominates helps select the right concurrency model.

5. Two Fundamental Concurrency Mechanisms

5.1. Threads

• A thread is an OS-managed, independent unit of CPU execution.
• Each thread has its own stack (memory for function calls, local variables) and an instruction pointer.
• Threads are scheduled by the Operating System scheduler using preemptive scheduling:
  • Each thread gets a CPU time slice.
  • After the slice, context switches save thread state and restore another.
• Threads can share memory if they belong to the same process, enabling fast communication via pointers.
• However, threads are expensive:
  • Each thread's stack can consume megabytes of memory.
  • Creating threads involves system calls with milliseconds overhead.
  • Frequent context switches add latency and CPU overhead.

5.2. Event Loops

• Event loop model typically uses a single thread which never blocks.
• When hitting an I/O operation, the task registers a callback and yields control.
• The event loop iterates, checking completion of I/O operations, running callbacks on completion.
• No blocking or extensive context switching: more memory and CPU efficient.
• Operating Systems support event loops via mechanisms like epoll (Linux) or kqueue (MacOS).
• Language features like async/await are syntactic sugar that hide callback complexity.
• Trade-off: event loop cannot block on CPU-bound tasks; heavy CPU work blocks the entire loop.

6. Practical Implementations

6.1. Threading Model (Example with Python)

• For each request, a thread executes CPU tasks (parsing, validation).
• Database query blocks the thread.
• OS scheduler pauses this thread, switches to another thread.
• Once database responds, the thread resumes and processes the response.
• Drawback: thread-per-request not scalable due to high memory/latency overhead.

6.2. Event Loop Model (Example with JavaScript)

• Single thread handles multiple requests.
• On database queries or I/O, await relinquishes control.
• Event loop continues processing other requests.
• When I/O completes, the registered callback resumes execution.
• Avoids thread overhead and context switch delays.
• Prevent blocking with CPU-heavy tasks to keep responsiveness.

6.3. Virtual Threads / Goroutines (Go Programming Language)

• Go implements lightweight "virtual threads" called goroutines managed by the Go runtime scheduler.
• The runtime multiplexes many goroutines onto a limited number of OS threads.
• Goroutines are extremely lightweight compared to OS threads (much smaller stack memory).
• When one goroutine blocks on I/O, the runtime scheduler switches to another goroutine.
• Enables spawning thousands or millions of concurrent goroutines without memory exhaustion.
• Combines concurrency with efficient scheduling—ideal for backend services.

7. Async/Await Internals: State Machine Model

• Async functions internally modeled as state machines.
• Every await transitions the function to a new state, pausing execution until the awaited operation completes.
• This explains why await can only be used inside async functions.
• Prevents blocking the event loop because execution is resumed only when I/O completes.
• Enhances readability over deeply nested callbacks.

8. Concurrency Pitfalls: Race Conditions and Shared Memory

• Multiple threads accessing shared variables can cause race conditions.
• Example: two threads incrementing a shared counter may overwrite updates, leading to lost counts.
• Similarly, even single-threaded async code can suffer from concurrency issues if shared states are modified unsafely (e.g., balance withdrawal example).
• Known solutions:
  • Locks/Mutexes: allow exclusive access to critical sections.
  • Channels (Go-specific): message passing avoids shared state by serializing access.
• Proper synchronization is essential to avoid data corruption in concurrent code.

9. Summary of Recommendations

• Most backend applications are I/O-bound and benefit significantly from concurrency.
• Parallelism is essential for CPU-heavy tasks requiring raw processing power.
• Understanding the difference guides architectural and technology choices (Node.js event loop, Go goroutines, Java threads).
• Concurrency avoids wasting CPU cycles during waiting times.
• Parallelism uses hardware capabilities for speed.

In essence, mastering concurrency vs parallelism and their implementations helps backend engineers design scalable, efficient systems that utilize server resources optimally while handling thousands of simultaneous requests without blocking or crashes.