Real-Time Backends

This video explores the evolution and architecture of real-time communication between browsers and servers, focusing on task management applications where live updates are required. It covers the limitations of traditional HTTP communication, introduces various techniques for enabling server-to-client updates, and analyzes their performance, scalability, and trade-offs.

1. From Request-Response to Real-Time Updates

Traditional web backends follow a strict request-response model: the client initiates every communication, and the server replies. This model struggles with real-time scenarios such as task boards where multiple users collaborate simultaneously and changes must be reflected immediately across clients.

• Current backend architecture cannot send unsolicited updates to browsers.
• Polling is the naive solution: the client repeatedly asks the server at fixed intervals if there is new data.
• Polling causes unnecessary overhead because most requests return no new data.
• Polling interval choices affect latency and server load:
  • Longer intervals increase update delay (average half the interval).
  • Shorter intervals reduce latency but multiply request volume and resource consumption.

Key limitations of polling:

• Wasteful resource usage independent of actual event frequency.
• Higher server load with scale (number of users, not number of changes).
• Negative user experience due to update delays.
• Inefficient for mobile due to frequent radio wake-ups, draining battery and data.

2. Improving Polling: Long Polling and Server-Sent Events

Long Polling

Long polling improves over simple polling by holding the client’s HTTP request open on the server until an event occurs or a timeout happens.

• The server delays responding until new data is available or timeout occurs.
• When a response is sent, the client immediately opens a new long poll request.
• Advantages:
  • Reduces unnecessary responses.
  • Decreases latency compared to fixed interval polling.
• Drawbacks:
  • There is a gap between response closing and new request initiation.
  • Average latency can still be one network transit time; worst can be worse.
  • Connection overhead repeats for every event.

Server-Sent Events (SSE)

SSE takes long polling further by keeping a single HTTP connection open indefinitely and streaming updates from server to client as text/event-stream data.

• The client issues a single GET request with accept: text/event-stream.
• The server responds with status 200 and keeps the connection open forever.
• The server pushes new events in the stream without closing the connection.
• Client-side features:
  • Automatic reconnection on drop.
  • Last event ID tracking for resuming missed events.
• SSE is simple and widely supported by browsers.
• Example usage by companies like Uber and LinkedIn.

Limitations of SSE:

• Unidirectional: only server-to-client streaming.
• Client-to-server must use separate HTTP requests.
• May suffice for many real-time features but not full-duplex needs.

3. WebSockets: Full-Duplex, Bidirectional Communication

WebSockets solve SSE’s main limitation by enabling clients and servers to send messages to each other over a single persistent connection.

• Connection starts as a normal HTTP GET request with "Upgrade" headers.
• Server responds with 101 Switching Protocols and a special key to confirm upgrade.
• After handshake, protocol switches from HTTP to WebSocket:
  • No headers, methods, or statuses in data frames.
  • Data transmitted as binary/text frames.
  • Both sides can send messages independently when needed.

WebSocket Frame Structure and Masking

• Frames include fields like FIN (final frame), opcode (text, binary, ping, pong, close), mask bit, and payload length.
• Small payloads have minimal overhead (~2 bytes per frame header).
• Client-to-server frames are masked for protection against cache poisoning attacks.
• Masking uses XOR with a random 4-byte key per frame to prevent malicious proxies from caching WebSocket traffic as HTTP responses.

Heartbeat Mechanism (Ping/Pong)

• To detect dead connections (due to network changes or client exits), either party can send a ping frame.
• The other side must immediately respond with pong.
• If no pong within a timeout (e.g., 30 seconds), connection considered dead and closed.
• Keeps TCP connections alive and servers aware of client state.

4. Scalability and Resource Limits of WebSockets

File Descriptor Limits

• Each WebSocket connection corresponds to a file descriptor (FD) on the server OS.
• Default soft limits vary by language/runtime:
  • Go runtime raises FD limit automatically (~148,000).
  • Python, Node.js, and Java may have lower limits (e.g., 1024).
• This limits simultaneous open connections per machine.

Port Exhaustion on Clients

• Ports used by client sockets are limited (typically 28,000+ range).
• Max concurrent connections from a single client IP limited by port availability.
• Solution: use multiple client IP addresses to increase connection capacity.

Memory Usage Per Connection

• In Go, an idle WebSocket consumes ~9.75 KB of heap memory.
• 1 million idle connections require ~10 GB heap memory.
• Limiting factor for scalability is server RAM availability.

5. Multi-Instance Deployment and Connection State

• Real-world deployments run several server instances behind a load balancer.
• WebSocket connections are stateful and tied to a specific instance in memory.
• Load balancer forwards different client connections to different instances.
• Issue: Updates to one client on instance A are not visible to clients connected to instance B.

Attempts and Solutions

• Sticky sessions (connecting client repeatedly to same instance) only partly solve issue; no coordination between instances.
• Correct architecture requires inter-instance communication.

Publish-Subscribe (Pub/Sub) Architecture

• Event-producing instance publishes messages to a common topic.
• Messaging systems (Redis pub/sub, NATS, Kafka) fan out events to all subscribing instances.
• Each instance filters messages relevant to connections it holds and forwards events.
• Challenges:
  • Redis pub/sub offers at most once delivery — no guaranteed retries.
  • Kafka and Redis Streams provide at least once semantics with persistence and acknowledgments.
• Design choice impacts implementable consistency and reliability.

6. Handling Failures and Reconnection

• Clients disconnect due to network changes, tab closes, or server restarts.
• Need strategy for:
  • Catching up on missed events by resending from last received event ID.
  • Load balancing reconnect bursts ("storm problem").
• Example: Slack caches initial client states at edge to avoid origin overload during reconnect storms.

7. Performance Implications and Fan-out Problem

• When many clients watch the same resource, each event causes many writes.
• Example: 30,000 concurrent subscribers require 30,000 writes per event.
• Solutions:
  • Distribute event delivery across several nodes/instances owning subsets of connections.
  • Use caching layers to optimize reconnects and reduce backend load.

Summary Table: Comparison of Real-Time Communication Techniques

Key Insights and Takeaways

• Traditional HTTP request-response model is insufficient for true real-time updates.
• Polling and long polling work but have significant latency and scale cost trade-offs.
• Server-Sent Events offer efficient one-way streaming with built-in browser support but lack client-to-server messaging on same connection.
• WebSockets provide full duplex real-time communication with minimal overhead after handshake, suited for interactive features.
• WebSocket scalability depends heavily on OS file descriptor limits, client ports, and memory.
• Multi-instance deployments need inter-instance message propagation via pub/sub systems.
• Pub/Sub solutions have different guarantees which affect event delivery reliability.
• Reconnection and state resynchronization are critical for production-quality systems.
• Systems with large fan-out loads must distribute work efficiently to avoid bottlenecks.
• Real-time collaboration and presence detection features build on top of these foundational architectures and introduce additional complexity (e.g., collaborative editing algorithms).

This comprehensive overview sets the foundation for building scalable, efficient real-time applications by understanding the trade-offs of different communication protocols and architectural patterns.