Chapter 6: REPLICATION & CONSISTENCY - System Design Course: Complete Production Architecture | PlayBook Library

SLIDE 48: Replication: How It Works

Database Replication Fundamentals

Leader-Follower Replication:

CODE

#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db{font-family:ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI Variable Display","Segoe UI",Helvetica,"Apple Color Emoji","Noto Sans Arabic","Noto Sans Hebrew",Arial,sans-serif,"Segoe UI Emoji","Segoe UI Symbol";font-size:16px;fill:#000000;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .error-icon{fill:#552222;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .error-text{fill:#552222;stroke:#552222;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .edge-thickness-normal{stroke-width:1px;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .edge-thickness-thick{stroke-width:3.5px;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .edge-pattern-solid{stroke-dasharray:0;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .marker{fill:#666;stroke:#666;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .marker.cross{stroke:#666;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db svg{font-family:ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI Variable Display","Segoe UI",Helvetica,"Apple Color Emoji","Noto Sans Arabic","Noto Sans Hebrew",Arial,sans-serif,"Segoe UI Emoji","Segoe UI Symbol";font-size:16px;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db p{margin:0;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .label{font-family:ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI Variable Display","Segoe UI",Helvetica,"Apple Color Emoji","Noto Sans Arabic","Noto Sans Hebrew",Arial,sans-serif,"Segoe UI Emoji","Segoe UI Symbol";color:#000000;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .cluster-label text{fill:#333;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .cluster-label span{color:#333;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .cluster-label span p{background-color:transparent;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .label text,#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db span{fill:#000000;color:#000000;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .node rect,#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .node circle,#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .node ellipse,#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .node polygon,#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .node path{fill:#eee;stroke:#999;stroke-width:1px;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .rough-node .label text,#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .node .label text,#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .image-shape .label,#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .icon-shape .label{text-anchor:middle;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .rough-node .label,#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .node .label,#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .image-shape .label,#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .icon-shape .label{text-align:center;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .node.clickable{cursor:pointer;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .root .anchor path{fill:#666!important;stroke-width:0;stroke:#666;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .arrowheadPath{fill:#333333;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .edgePath .path{stroke:#666;stroke-width:1px;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .flowchart-link{stroke:#666;fill:none;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .edgeLabel{background-color:white;text-align:center;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .edgeLabel p{background-color:white;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .edgeLabel rect{opacity:0.5;background-color:white;fill:white;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .labelBkg{background-color:rgba(255, 255, 255, 0.5);}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .cluster rect{fill:hsl(0, 0%, 98.9215686275%);stroke:#707070;stroke-width:1px;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .cluster text{fill:#333;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .cluster span{color:#333;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI Variable Display","Segoe UI",Helvetica,"Apple Color Emoji","Noto Sans Arabic","Noto Sans Hebrew",Arial,sans-serif,"Segoe UI Emoji","Segoe UI Symbol";font-size:12px;background:hsl(-160, 0%, 93.3333333333%);border:1px solid #707070;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#000000;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db rect.text{fill:none;stroke-width:0;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .icon-shape,#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .image-shape{background-color:white;text-align:center;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .icon-shape p,#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .image-shape p{background-color:white;padding:2px;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .icon-shape .label rect,#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .image-shape .label rect{opacity:0.5;background-color:white;fill:white;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db .node .neo-node{stroke:#999;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db [data-look="neo"].node rect,#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db [data-look="neo"].cluster rect,#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db [data-look="neo"].node polygon{stroke:url(#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db [data-look="neo"].swimlane.cluster rect{filter:none;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db [data-look="neo"].node path{stroke:url(#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db-gradient);stroke-width:1px;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db [data-look="neo"].node .outer-path{filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db [data-look="neo"].node .neo-line path{stroke:#999;filter:none;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db [data-look="neo"].node circle{stroke:url(#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db [data-look="neo"].node circle .state-start{fill:#000000;}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db [data-look="neo"].icon-shape .icon{fill:url(#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db [data-look="neo"].icon-shape .icon-neo path{stroke:url(#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-ff5e63b5-f73f-476c-a777-ec1440d426db :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}ApplicationLeaderFollower 1Follower 2Follower 3WritesReplicationReplicationReplicationReadsReadsReads

How It Works:

Step	Description
1	Application sends write to leader
2	Leader writes to local storage
3	Leader writes to replication log (WAL/binlog)
4	Followers read replication log
5	Followers apply changes to local storage
6	Followers acknowledge to leader (sync) or not (async)

SLIDE 49: Replication: How It Works (continued)

Synchronous vs Asynchronous Replication:

Type	Behavior	Pros	Cons
Synchronous	Leader waits for follower ACK	No data loss	Higher latency
Asynchronous	Leader does not wait	Lower latency	Potential data loss
Semi-synchronous	Leader waits for 1 of N followers	Balanced	Complexity

Replication Lag:

CODE

Replication lag = Time for change to appear on follower

Typical values:
- Same datacenter: 1-10ms
- Cross-region: 50-200ms
- Under heavy load: 1-60 seconds
- During maintenance: Minutes to hours

Why Lag Matters:

CODE

1. User creates post (writes to leader)
2. User refreshes page (reads from follower)
3. Follower is 500ms behind
4. User does not see their post
5. User thinks system is broken

SLIDE 50: Replication: Read-Your-Writes Consistency

Solving the "Where Is My Post?" Problem

The Problem:

CODE

#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e{font-family:ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI Variable Display","Segoe UI",Helvetica,"Apple Color Emoji","Noto Sans Arabic","Noto Sans Hebrew",Arial,sans-serif,"Segoe UI Emoji","Segoe UI Symbol";font-size:16px;fill:#000000;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .error-icon{fill:#552222;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .error-text{fill:#552222;stroke:#552222;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .edge-thickness-normal{stroke-width:1px;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .edge-thickness-thick{stroke-width:3.5px;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .edge-pattern-solid{stroke-dasharray:0;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .marker{fill:#666;stroke:#666;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .marker.cross{stroke:#666;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e svg{font-family:ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI Variable Display","Segoe UI",Helvetica,"Apple Color Emoji","Noto Sans Arabic","Noto Sans Hebrew",Arial,sans-serif,"Segoe UI Emoji","Segoe UI Symbol";font-size:16px;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e p{margin:0;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .label{font-family:ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI Variable Display","Segoe UI",Helvetica,"Apple Color Emoji","Noto Sans Arabic","Noto Sans Hebrew",Arial,sans-serif,"Segoe UI Emoji","Segoe UI Symbol";color:#000000;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .cluster-label text{fill:#333;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .cluster-label span{color:#333;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .cluster-label span p{background-color:transparent;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .label text,#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e span{fill:#000000;color:#000000;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .node rect,#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .node circle,#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .node ellipse,#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .node polygon,#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .node path{fill:#eee;stroke:#999;stroke-width:1px;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .rough-node .label text,#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .node .label text,#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .image-shape .label,#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .icon-shape .label{text-anchor:middle;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .rough-node .label,#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .node .label,#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .image-shape .label,#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .icon-shape .label{text-align:center;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .node.clickable{cursor:pointer;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .root .anchor path{fill:#666!important;stroke-width:0;stroke:#666;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .arrowheadPath{fill:#333333;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .edgePath .path{stroke:#666;stroke-width:1px;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .flowchart-link{stroke:#666;fill:none;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .edgeLabel{background-color:white;text-align:center;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .edgeLabel p{background-color:white;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .edgeLabel rect{opacity:0.5;background-color:white;fill:white;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .labelBkg{background-color:rgba(255, 255, 255, 0.5);}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .cluster rect{fill:hsl(0, 0%, 98.9215686275%);stroke:#707070;stroke-width:1px;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .cluster text{fill:#333;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .cluster span{color:#333;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI Variable Display","Segoe UI",Helvetica,"Apple Color Emoji","Noto Sans Arabic","Noto Sans Hebrew",Arial,sans-serif,"Segoe UI Emoji","Segoe UI Symbol";font-size:12px;background:hsl(-160, 0%, 93.3333333333%);border:1px solid #707070;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#000000;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e rect.text{fill:none;stroke-width:0;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .icon-shape,#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .image-shape{background-color:white;text-align:center;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .icon-shape p,#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .image-shape p{background-color:white;padding:2px;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .icon-shape .label rect,#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .image-shape .label rect{opacity:0.5;background-color:white;fill:white;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e .node .neo-node{stroke:#999;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e [data-look="neo"].node rect,#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e [data-look="neo"].cluster rect,#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e [data-look="neo"].node polygon{stroke:url(#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e [data-look="neo"].swimlane.cluster rect{filter:none;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e [data-look="neo"].node path{stroke:url(#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e-gradient);stroke-width:1px;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e [data-look="neo"].node .outer-path{filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e [data-look="neo"].node .neo-line path{stroke:#999;filter:none;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e [data-look="neo"].node circle{stroke:url(#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e [data-look="neo"].node circle .state-start{fill:#000000;}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e [data-look="neo"].icon-shape .icon{fill:url(#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e [data-look="neo"].icon-shape .icon-neo path{stroke:url(#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-39f576bd-6cb1-4245-a7da-b9c2948a5e6e :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}User WritesLeaderFollowerLagReadsStale Data

Solution 1: Read From Leader After Write

CODE

def create_post(user_id, content):
    post_id = leader_db.insert("INSERT INTO posts ...")

    # Mark user as "recently wrote"
    cache.set(f"recent_write:{user_id}", True, ttl=5)
    return post_id

def get_posts(user_id):
    if cache.get(f"recent_write:{user_id}"):
        # Read from leader for consistency
        return leader_db.query("SELECT * FROM posts WHERE user_id = ?", user_id)
    else:
        # Read from follower for performance
        return follower_db.query("SELECT * FROM posts WHERE user_id = ?", user_id)

SLIDE 51: Replication: Read-Your-Writes Consistency (continued)

Solution 2: Include Timestamp in Response

CODE

def create_post(user_id, content):
    post_id = leader_db.insert("INSERT INTO posts ...")
    write_timestamp = time.now()

    return {
        "post_id": post_id,
        "write_timestamp": write_timestamp
    }

def get_posts(user_id, min_timestamp=None):
    if min_timestamp:
        # Wait for follower to catch up
        wait_for_replication(min_timestamp, timeout=2)

    return follower_db.query("SELECT * FROM posts WHERE user_id = ?", user_id)

Solution 3: Sticky Sessions to Same Replica

CODE

All reads for user_123 go to follower-2
Follower-2 has consistent view for that user

Trade-offs:

Solution	Consistency	Load Distribution	Complexity
Read from leader	Strong	Poor (leader overloaded)	Low
Timestamp-based	Strong	Good	Medium
Sticky sessions	Eventual	Uneven	Low

SLIDE 52: Replication: Failover

What Happens When Leader Dies

Failover Process:

Structure & Concept Flow

graph LR

A[Leader Fails] --> B[Detect Failure]

B --> C[Elect New Leader]

C --> D[Reconfigure Followers]

D --> E[Update Application]

Failure Detection:

Method	Detection Time	False Positives
Heartbeat timeout	10-30 seconds	Medium
Consensus (Raft/Paxos)	1-5 seconds	Low
Manual intervention	Minutes to hours	None

Leader Election:

CODE

Criteria for new leader:
1. Most up-to-date data (smallest replication lag)
2. Healthy and responsive
3. In majority partition (network split scenario)

SLIDE 53: Replication: Failover (continued)

Data Loss During Failover:

CODE

Scenario:
- Leader has transactions A, B, C
- Follower has transactions A, B (lag)
- Leader crashes
- Follower becomes new leader
- Transaction C is lost

Mitigation Strategies:

Strategy	Data Loss	Availability Impact
Synchronous replication	None	Higher latency, lower availability
Semi-synchronous	Minimal	Moderate impact
Asynchronous	Possible	Best availability

Application Considerations:

CODE

def write_with_retry(data):
    try:
        return db.insert(data)
    except ConnectionError:
        # Leader may have changed
        time.sleep(1)
        refresh_db_connection()
        return db.insert(data)  # May create duplicate if original succeeded

Key Insight:
Failover is messy. Applications must handle duplicates (idempotency) and retries.

SLIDE 54: Consistency: Strong vs Eventual

Understanding Consistency Models

Strong Consistency:

CODE

After write completes, all subsequent reads return that write.

Structure & Concept Flow

graph LR

A[Write X=5] --> B[Commit]

B --> C[Read X]

C --> D[Returns 5]

Eventual Consistency:

CODE

After write completes, reads may return old value temporarily.
Eventually, all reads return the new value.

Structure & Concept Flow

graph LR

A[Write X=5] --> B[Commit]

B --> C[Read X]

C --> D[Returns 3 or 5]

D --> E[Eventually Returns 5]

SLIDE 55: Consistency: Strong vs Eventual (continued)

When to Use Each:

Use Strong Consistency	Use Eventual Consistency
Bank account balance	Social media feed
Inventory count (purchasing)	Like counts
User authentication	View counts
Payment processing	Recommendation systems
Booking/reservation systems	Analytics dashboards

The CAP Theorem Trade-off:

CODE

In presence of network partition, choose either:
- Consistency: Refuse writes until partition heals
- Availability: Accept writes, reconcile later

Most systems choose availability with eventual consistency.

Production Reality:

CODE

Strong consistency = Higher latency + Lower availability
Eventual consistency = Lower latency + Higher availability

Choose based on business requirements, not technical preference.

SLIDE 56: Consistency: Dealing with Stale Reads

Strategies for Acceptable Staleness

Strategy 1: Time-Bounded Staleness

CODE

# Read is acceptable if data is less than 5 seconds old
def get_user(user_id):
    cached = cache.get(f"user:{user_id}")
    if cached and cached.timestamp > time.now() - 5:
        return cached.data
    else:
        fresh = db.query("SELECT * FROM users WHERE id = ?", user_id)
        cache.set(f"user:{user_id}", {"data": fresh, "timestamp": time.now()})
        return fresh

Strategy 2: Version-Based Reads

CODE

# Client tracks version, rejects older versions
def get_user(user_id, min_version=None):
    user = db.query("SELECT * FROM users WHERE id = ?", user_id)
    if min_version and user.version < min_version:
        # Read from leader to get latest
        user = leader_db.query("SELECT * FROM users WHERE id = ?", user_id)
    return user

SLIDE 57: Consistency: Dealing with Stale Reads (continued)

Strategy 3: Causal Consistency

CODE

# Track dependencies between operations
def update_user_and_get(user_id, new_name):
    # Write returns a token
    token = db.execute("UPDATE users SET name = ? WHERE id = ?", new_name, user_id)

    # Read waits for token to be visible
    user = db.query("SELECT * FROM users WHERE id = ? AFTER ?", user_id, token)
    return user

Strategy 4: Read Repair

CODE

# If stale read detected, trigger refresh
def get_user(user_id):
    user = cache.get(f"user:{user_id}")
    if user and is_stale(user):
        # Async refresh
        queue.enqueue("refresh_user_cache", user_id)
    return user  # Return stale data, refresh in background

Key Insight:
Most users tolerate 1-5 seconds of staleness. Design for this explicitly rather than fighting for strong consistency everywhere.

SLIDE 58: Consistency: Compensating Transactions

When Things Go Wrong

The Problem: Distributed Transactions

Structure & Concept Flow

graph LR

A[Order Service] --> B[Inventory Service]

A --> C[Payment Service]

A --> D[Shipping Service]

If payment succeeds but inventory fails, how do you rollback?

Solution: Saga Pattern

Structure & Concept Flow

graph LR

A[Create Order] --> B[Reserve Inventory]

B --> C[Charge Payment]

C --> D[Schedule Shipping]

D --> E[Complete]

B -->|Fail| F[Cancel Order]

C -->|Fail| G[Release Inventory]

D -->|Fail| H[Refund Payment]

Each step has a compensating action that undoes it.

SLIDE 59: Consistency: Compensating Transactions (continued)

Saga Implementation:

CODE

class OrderSaga:
    def execute(self, order):
        try:
            # Step 1: Reserve inventory
            inventory_reservation = inventory_service.reserve(order.items)

            # Step 2: Charge payment
            payment = payment_service.charge(order.total)

            # Step 3: Schedule shipping
            shipping = shipping_service.schedule(order.address)

            return Success(order)

        except PaymentError:
            # Compensate step 1
            inventory_service.release(inventory_reservation)
            return Failure("Payment failed")

        except ShippingError:
            # Compensate steps 1 and 2
            payment_service.refund(payment)
            inventory_service.release(inventory_reservation)
            return Failure("Shipping failed")

Key Properties:

Property	Requirement
Idempotent steps	Same request twice = same result
Compensatable steps	Every action can be undone
Ordered execution	Steps must run in sequence
Failure handling	Each failure triggers compensation

SLIDE 60: Hot Partitions: The Problem

When One Shard Gets All the Traffic

Scenario: Sharding by User ID

CODE

shard_id = user_id % 1000

Normal Distribution:

Shard	Users	Requests/sec
0	2M	1000
1	2M	1000
…	…	…
999	2M	1000

Hot Partition Scenario:

CODE

Celebrity user (100M followers) is on Shard 345
All followers viewing celebrity feed hit Shard 345

Shard	Users	Requests/sec
0	2M	1000
345	2M	500,000
…	…	…

SLIDE 61: Hot Partitions: The Problem (continued)

Common Causes of Hot Partitions:

Cause	Example
Celebrity users	Kim Kardashian posts, millions view
Viral content	One post shared millions of times
Time-based keys	All writes to “today” partition
Sequential IDs	Auto-increment creates hot tail
Event-driven spikes	Black Friday, sports events

Symptoms:

CODE

- One database shard at 100% CPU, others at 10%
- Latency spikes for subset of users
- Connection pool exhaustion on one shard
- Replication lag on one shard only

Key Insight:
Uniform hashing does not prevent hot partitions. Traffic is not uniform by nature.

SLIDE 62: Hot Partitions: Solutions

Strategies for Handling Hot Keys

Solution 1: Caching Hot Data

CODE

def get_celebrity_posts(user_id):
    cache_key = f"posts:{user_id}"

    # Check if user is "hot"
    if is_celebrity(user_id):
        # Use dedicated cache with higher TTL
        posts = celebrity_cache.get(cache_key)
        if posts:
            return posts

    posts = db.query("SELECT * FROM posts WHERE user_id = ?", user_id)

    if is_celebrity(user_id):
        celebrity_cache.set(cache_key, posts, ttl=60)

    return posts

Solution 2: Read Replicas for Hot Shards

Structure & Concept Flow

graph LR

A[Shard 345 Leader] --> B[Replica 1]

A --> C[Replica 2]

A --> D[Replica 3]

A --> E[Replica 4]

A --> F[Replica 5]

Add more read replicas specifically for hot shards.

SLIDE 63: Hot Partitions: Solutions (continued)

Solution 3: Key Salting

CODE

# Instead of: cache_key = f"user:{user_id}"
# Use: cache_key = f"user:{user_id}:{salt}"

def get_user_with_salt(user_id):
    # Distribute across multiple cache keys
    salt = random.randint(0, 9)
    cache_key = f"user:{user_id}:{salt}"

    result = cache.get(cache_key)
    if not result:
        result = db.query("SELECT * FROM users WHERE id = ?", user_id)
        cache.set(cache_key, result)

    return result

For reads: Random salt distributes load across cache nodes.
For writes: Must update all 10 salted keys (fan-out).

Solution 4: Separate Hot and Cold Data

Structure & Concept Flow

graph LR

A[Request] --> B{Is Hot User?}

B -->|Yes| C[Hot Data Store]

B -->|No| D[Cold Data Store]

Route celebrity traffic to dedicated infrastructure.

SLIDE 64: Hot Partitions: Time-Based Partitioning Pitfalls

Why Time-Based Keys Create Hot Spots

Problematic Pattern:

CODE

CREATE TABLE events (
    event_date DATE,
    event_id UUID,
    data JSONB,
    PRIMARY KEY (event_date, event_id)
);

Problem:

CODE

All writes go to "today" partition
Yesterday partition: 0 writes
Today partition: 100,000 writes/second
Tomorrow partition: 0 writes

Better Pattern: Time + Random Prefix

CODE

CREATE TABLE events (
    partition_key INT,  -- random 0-99
    event_date DATE,
    event_id UUID,
    data JSONB,
    PRIMARY KEY (partition_key, event_date, event_id)
);

-- Insert with random partition
INSERT INTO events (partition_key, event_date, event_id, data)
VALUES (random() * 100, CURRENT_DATE, uuid_generate_v4(), '{}');

Writes distributed across 100 partitions for each date.

SLIDE 65: Hot Partitions: Time-Based Partitioning Pitfalls (continued)

Querying with Partition Scatter-Gather:

CODE

def get_events_for_date(date):
    results = []

    # Query all partitions in parallel
    futures = []
    for partition in range(100):
        futures.append(
            executor.submit(query_partition, partition, date)
        )

    for future in futures:
        results.extend(future.result())

    return results

Trade-offs:

Approach	Write Distribution	Read Complexity	Use Case
Date only	Hot (today)	Simple	Low write volume
Date + random prefix	Even	Scatter-gather	High write volume
Date + hour	Moderate	Simple	Medium write volume

Key Insight:
Write-heavy time series data needs synthetic partitioning. Accept read complexity for write scalability.