SLIDE 1a: What Senior and Staff Interviews Actually Test
Beyond Architecture Diagrams
At the senior and staff level, interviewers expect you to reason about what happens after the system is deployed:
- How do you know if your system is healthy right now?
- How do you deploy changes to millions of users without breaking things?
- How do you debug a problem that spans 15 microservices?
- How do you design for cost when infrastructure bills hit millions per year?
- How do you respond when something breaks at 3am?
These questions test whether you have operated real systems at scale — not just designed them on a whiteboard.
This course gives you that knowledge — structured, practical, and interview-ready.
SLIDE 1b: Course Overview
What We Will Cover
Total Runtime: Approximately 4 hours 55 minutes
Each section builds on the previous one. We finish with exactly how to frame these concepts in senior and staff interviews.
Let’s get started with observability and debugging.
SLIDE 2: Section: Observability
Understanding What Your System is Doing
CODE
#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0{font-family:ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI Variable Display","Segoe UI",Helvetica,"Apple Color Emoji","Noto Sans Arabic","Noto Sans Hebrew",Arial,sans-serif,"Segoe UI Emoji","Segoe UI Symbol";font-size:16px;fill:#000000;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .error-icon{fill:#552222;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .error-text{fill:#552222;stroke:#552222;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .edge-thickness-normal{stroke-width:1px;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .marker{fill:#666;stroke:#666;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .marker.cross{stroke:#666;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 svg{font-family:ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI Variable Display","Segoe UI",Helvetica,"Apple Color Emoji","Noto Sans Arabic","Noto Sans Hebrew",Arial,sans-serif,"Segoe UI Emoji","Segoe UI Symbol";font-size:16px;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 p{margin:0;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .label{font-family:ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI Variable Display","Segoe UI",Helvetica,"Apple Color Emoji","Noto Sans Arabic","Noto Sans Hebrew",Arial,sans-serif,"Segoe UI Emoji","Segoe UI Symbol";color:#000000;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .cluster-label text{fill:#333;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .cluster-label span{color:#333;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .cluster-label span p{background-color:transparent;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .label text,#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 span{fill:#000000;color:#000000;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .node rect,#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .node circle,#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .node ellipse,#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .node polygon,#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .node path{fill:#eee;stroke:#999;stroke-width:1px;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .rough-node .label text,#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .node .label text,#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .image-shape .label,#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .icon-shape .label{text-anchor:middle;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .rough-node .label,#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .node .label,#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .image-shape .label,#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .icon-shape .label{text-align:center;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .node.clickable{cursor:pointer;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .root .anchor path{fill:#666!important;stroke-width:0;stroke:#666;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .arrowheadPath{fill:#333333;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .edgePath .path{stroke:#666;stroke-width:1px;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .flowchart-link{stroke:#666;fill:none;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .edgeLabel{background-color:white;text-align:center;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .edgeLabel p{background-color:white;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .edgeLabel rect{opacity:0.5;background-color:white;fill:white;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .labelBkg{background-color:rgba(255, 255, 255, 0.5);}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .cluster rect{fill:hsl(0, 0%, 98.9215686275%);stroke:#707070;stroke-width:1px;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .cluster text{fill:#333;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .cluster span{color:#333;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI Variable Display","Segoe UI",Helvetica,"Apple Color Emoji","Noto Sans Arabic","Noto Sans Hebrew",Arial,sans-serif,"Segoe UI Emoji","Segoe UI Symbol";font-size:12px;background:hsl(-160, 0%, 93.3333333333%);border:1px solid #707070;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#000000;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 rect.text{fill:none;stroke-width:0;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .icon-shape,#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .image-shape{background-color:white;text-align:center;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .icon-shape p,#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .image-shape p{background-color:white;padding:2px;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .icon-shape .label rect,#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .image-shape .label rect{opacity:0.5;background-color:white;fill:white;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 .node .neo-node{stroke:#999;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 [data-look="neo"].node rect,#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 [data-look="neo"].cluster rect,#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 [data-look="neo"].node polygon{stroke:url(#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 [data-look="neo"].swimlane.cluster rect{filter:none;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 [data-look="neo"].node path{stroke:url(#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0-gradient);stroke-width:1px;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 [data-look="neo"].node .outer-path{filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 [data-look="neo"].node .neo-line path{stroke:#999;filter:none;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 [data-look="neo"].node circle{stroke:url(#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 [data-look="neo"].node circle .state-start{fill:#000000;}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 [data-look="neo"].icon-shape .icon{fill:url(#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 [data-look="neo"].icon-shape .icon-neo path{stroke:url(#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-44da1eb6-62cc-4c08-bec1-d1d42c3bf5c0 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}LogsMetricsTracesAlertingDashboards
The Three Pillars of Observability:
Key Insight:
You cannot fix what you cannot see. Observability is not optional in production.
SLIDE 3: Logs: Structured Logging
Why Structure Matters
Unstructured Log (Bad):
CODE
2025-01-15 10:23:45 ERROR - Payment failed for user 12345 order 67890 amount $99.99 reason: card declined
Parsing this requires regex. Every log format is different.
Structured Log (Good):
CODE
{
"timestamp": "2025-01-15T10:23:45.123Z",
"level": "ERROR",
"message": "Payment failed",
"user_id": 12345,
"order_id": 67890,
"amount": 99.99,
"currency": "USD",
"reason": "card_declined",
"trace_id": "abc123",
"service": "payment-service",
"host": "payment-prod-1"
}
Queryable. Filterable. Aggregatable.
SLIDE 4: Logs: Structured Logging (continued)
Structured Logging Implementation:
CODE
import structlog
import logging
# Configure structured logging
structlog.configure(
processors=[
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.JSONRenderer()
],
wrapper_class=structlog.BoundLogger,
context_class=dict,
logger_factory=structlog.PrintLoggerFactory(),
)
logger = structlog.get_logger()
def process_payment(user_id, order_id, amount):
log = logger.bind(user_id=user_id, order_id=order_id, amount=amount)
log.info("payment_started")
try:
result = charge_card(amount)
log.info("payment_succeeded", transaction_id=result.id)
return result
except CardDeclinedError as e:
log.error("payment_failed", reason="card_declined", error=str(e))
raise
SLIDE 5: Logs: What to Log
Logging Best Practices
Always Log:
Never Log:
SLIDE 6: Logs: What to Log (continued)
Sanitizing Sensitive Data:
CODE
SENSITIVE_FIELDS = {"password", "credit_card", "ssn", "api_key", "token"}
def sanitize(data):
if isinstance(data, dict):
return {
k: "[REDACTED]" if k.lower() in SENSITIVE_FIELDS else sanitize(v)
for k, v in data.items()
}
return data
def log_request(request):
sanitized = sanitize({
"path": request.path,
"method": request.method,
"body": request.body,
"headers": dict(request.headers),
})
logger.info("request_received", **sanitized)
Log Levels:
SLIDE 7: Logs: Correlation IDs
Tracing Requests Across Services
The Problem:
CODE
User complains: "My order failed"
You have 100 services, 1 billion log lines
Which logs are relevant?
Solution: Correlation ID (Trace ID)
Structure & Concept Flow
All logs for one request share the same trace_id.
CODE
Query: trace_id = "abc123"
Result: All logs from all services for that request
SLIDE 8: Logs: Correlation IDs (continued)
Implementation:
CODE
import uuid
from contextvars import ContextVar
trace_id_var: ContextVar[str] = ContextVar("trace_id", default="")
def trace_id_middleware(request, call_next):
# Get trace_id from header or generate new one
trace_id = request.headers.get("X-Trace-ID") or str(uuid.uuid4())
# Store in context for this request
trace_id_var.set(trace_id)
# Add to response headers
response = call_next(request)
response.headers["X-Trace-ID"] = trace_id
return response
def get_logger():
return logger.bind(trace_id=trace_id_var.get())
# Usage
def process_order(order):
log = get_logger()
log.info("processing_order", order_id=order.id)
# When calling another service, pass trace_id
response = requests.post(
payment_url,
headers={"X-Trace-ID": trace_id_var.get()}
)
SLIDE 9: Metrics: Types of Metrics
What to Measure
Counter:
CODE
Total count of events. Only goes up.
CODE
# Examples
requests_total = Counter("requests_total", "Total requests")
errors_total = Counter("errors_total", "Total errors")
orders_created = Counter("orders_created", "Orders created")
def handle_request():
requests_total.inc()
Gauge:
CODE
Current value. Can go up or down.
CODE
# Examples
active_connections = Gauge("active_connections", "Current connections")
queue_depth = Gauge("queue_depth", "Messages in queue")
memory_usage = Gauge("memory_usage_bytes", "Memory usage")
def on_connect():
active_connections.inc()
def on_disconnect():
active_connections.dec()
SLIDE 10: Metrics: Types of Metrics (continued)
Histogram:
CODE
Distribution of values. Tracks buckets.
CODE
# Examples
request_duration = Histogram(
"request_duration_seconds",
"Request latency",
buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
)
response_size = Histogram(
"response_size_bytes",
"Response size",
buckets=[100, 1000, 10000, 100000, 1000000]
)
def handle_request():
start = time.time()
response = process()
request_duration.observe(time.time() - start)
response_size.observe(len(response.body))
Summary:
CODE
Similar to histogram but calculates percentiles client-side.
Use histogram unless you need specific quantiles.
SLIDE 11: Metrics: The RED Method
Request-focused Metrics
RED Metrics:
Implementation:
CODE
from prometheus_client import Counter, Histogram
# Rate
requests_total = Counter(
"http_requests_total",
"Total HTTP requests",
["method", "endpoint", "status"]
)
# Errors (subset of requests_total where status >= 500)
# Duration
request_duration = Histogram(
"http_request_duration_seconds",
"Request duration",
["method", "endpoint"]
)
def handle_request(request):
start = time.time()
response = process(request)
requests_total.labels(
method=request.method,
endpoint=request.path,
status=response.status
).inc()
request_duration.labels(
method=request.method,
endpoint=request.path
).observe(time.time() - start)
SLIDE 12: Metrics: The USE Method
Resource-focused Metrics
USE Metrics:
Apply USE to Each Resource:
SLIDE 13: Metrics: Application Metrics
Business and Application-Level Metrics
Business Metrics:
CODE
# Orders
orders_created = Counter("orders_created_total", "Total orders")
order_value = Histogram("order_value_dollars", "Order value distribution")
orders_by_status = Counter("orders_by_status", "Orders by status", ["status"])
# Users
signups = Counter("user_signups_total", "User signups")
active_users = Gauge("active_users", "Currently active users")
# Revenue
revenue = Counter("revenue_cents_total", "Total revenue in cents")
Dependency Metrics:
CODE
# External service calls
external_calls = Counter(
"external_calls_total",
"External service calls",
["service", "endpoint", "status"]
)
external_duration = Histogram(
"external_call_duration_seconds",
"External call latency",
["service", "endpoint"]
)
# Circuit breaker state
circuit_state = Gauge(
"circuit_breaker_state",
"Circuit breaker state (0=closed, 1=open, 0.5=half-open)",
["service"]
)
SLIDE 14: Metrics: Percentiles and SLIs
Understanding Latency Distribution
Why Percentiles Matter:
CODE
Average latency: 100ms
Sounds good, right?
But:
- P50 (median): 50ms
- P90: 100ms
- P99: 2000ms
- P99.9: 10000ms
1 in 1000 requests takes 10 seconds!
At 1M requests/day, that is 1000 terrible experiences.
Service Level Indicators (SLIs):
SLIDE 15: Metrics: Percentiles and SLIs (continued)
Calculating SLIs:
CODE
# Availability SLI
availability = (
sum(requests where status < 500) /
sum(all requests)
) * 100
# Latency SLI
latency_sli = (
sum(requests where duration < 500ms) /
sum(all requests)
) * 100
# PromQL examples
# Availability
sum(rate(requests_total{status!~"5.."}[5m])) /
sum(rate(requests_total[5m]))
# P99 latency
histogram_quantile(0.99, rate(request_duration_seconds_bucket[5m]))
Error Budget:
CODE
SLO: 99.9% availability
Allowed downtime per month: 0.1% = 43 minutes
If you have used 30 minutes of downtime:
- Error budget remaining: 13 minutes
- Slow down risky deployments
- Focus on reliability
SLIDE 16: Distributed Tracing: Concepts
Following Requests Across Services
Why Tracing:
CODE
User reports: "Checkout is slow"
Without tracing:
- Which service is slow?
- Is it the database? Network? CPU?
- 10 services involved, where to look?
With tracing:
- See exact path of request
- See time spent in each service
- Identify the bottleneck
Trace Structure:
CODE
#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3{font-family:ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI Variable Display","Segoe UI",Helvetica,"Apple Color Emoji","Noto Sans Arabic","Noto Sans Hebrew",Arial,sans-serif,"Segoe UI Emoji","Segoe UI Symbol";font-size:16px;fill:#000000;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .error-icon{fill:#552222;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .error-text{fill:#552222;stroke:#552222;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .edge-thickness-normal{stroke-width:1px;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .marker{fill:#666;stroke:#666;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .marker.cross{stroke:#666;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 svg{font-family:ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI Variable Display","Segoe UI",Helvetica,"Apple Color Emoji","Noto Sans Arabic","Noto Sans Hebrew",Arial,sans-serif,"Segoe UI Emoji","Segoe UI Symbol";font-size:16px;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 p{margin:0;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .label{font-family:ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI Variable Display","Segoe UI",Helvetica,"Apple Color Emoji","Noto Sans Arabic","Noto Sans Hebrew",Arial,sans-serif,"Segoe UI Emoji","Segoe UI Symbol";color:#000000;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .cluster-label text{fill:#333;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .cluster-label span{color:#333;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .cluster-label span p{background-color:transparent;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .label text,#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 span{fill:#000000;color:#000000;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .node rect,#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .node circle,#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .node ellipse,#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .node polygon,#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .node path{fill:#eee;stroke:#999;stroke-width:1px;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .rough-node .label text,#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .node .label text,#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .image-shape .label,#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .icon-shape .label{text-anchor:middle;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .rough-node .label,#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .node .label,#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .image-shape .label,#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .icon-shape .label{text-align:center;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .node.clickable{cursor:pointer;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .root .anchor path{fill:#666!important;stroke-width:0;stroke:#666;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .arrowheadPath{fill:#333333;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .edgePath .path{stroke:#666;stroke-width:1px;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .flowchart-link{stroke:#666;fill:none;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .edgeLabel{background-color:white;text-align:center;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .edgeLabel p{background-color:white;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .edgeLabel rect{opacity:0.5;background-color:white;fill:white;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .labelBkg{background-color:rgba(255, 255, 255, 0.5);}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .cluster rect{fill:hsl(0, 0%, 98.9215686275%);stroke:#707070;stroke-width:1px;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .cluster text{fill:#333;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .cluster span{color:#333;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI Variable Display","Segoe UI",Helvetica,"Apple Color Emoji","Noto Sans Arabic","Noto Sans Hebrew",Arial,sans-serif,"Segoe UI Emoji","Segoe UI Symbol";font-size:12px;background:hsl(-160, 0%, 93.3333333333%);border:1px solid #707070;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#000000;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 rect.text{fill:none;stroke-width:0;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .icon-shape,#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .image-shape{background-color:white;text-align:center;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .icon-shape p,#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .image-shape p{background-color:white;padding:2px;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .icon-shape .label rect,#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .image-shape .label rect{opacity:0.5;background-color:white;fill:white;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 .node .neo-node{stroke:#999;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 [data-look="neo"].node rect,#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 [data-look="neo"].cluster rect,#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 [data-look="neo"].node polygon{stroke:url(#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 [data-look="neo"].swimlane.cluster rect{filter:none;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 [data-look="neo"].node path{stroke:url(#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3-gradient);stroke-width:1px;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 [data-look="neo"].node .outer-path{filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 [data-look="neo"].node .neo-line path{stroke:#999;filter:none;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 [data-look="neo"].node circle{stroke:url(#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 [data-look="neo"].node circle .state-start{fill:#000000;}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 [data-look="neo"].icon-shape .icon{fill:url(#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 [data-look="neo"].icon-shape .icon-neo path{stroke:url(#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-5e37fdf0-7938-4dd1-a443-e98f036607d3 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}Trace: abc123Span: API Gateway 5msSpan: Order Service 100msSpan: DB Query 80msSpan: Payment Service 50msSpan: External API 45ms
SLIDE 17: Distributed Tracing: Concepts (continued)
Trace Components:
Span Example:
CODE
{
"trace_id": "abc123",
"span_id": "span456",
"parent_span_id": "span123",
"operation_name": "db.query",
"service": "order-service",
"start_time": "2025-01-15T10:23:45.000Z",
"duration_ms": 45,
"tags": {
"db.type": "postgresql",
"db.statement": "SELECT * FROM orders WHERE id = ?",
"db.rows_affected": 1
},
"status": "OK"
}
SLIDE 18: Distributed Tracing: Implementation
Adding Tracing to Code
CODE
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
tracer = trace.get_tracer("order-service")
def process_order(order_id):
with tracer.start_as_current_span("process_order") as span:
span.set_attribute("order_id", order_id)
# Child span for database query
with tracer.start_as_current_span("db.get_order") as db_span:
db_span.set_attribute("db.system", "postgresql")
order = db.get_order(order_id)
db_span.set_attribute("db.rows_returned", 1 if order else 0)
# Child span for payment
with tracer.start_as_current_span("payment.charge") as payment_span:
payment_span.set_attribute("payment.amount", order.total)
try:
result = payment_service.charge(order)
payment_span.set_attribute("payment.transaction_id", result.id)
except PaymentError as e:
payment_span.set_status(Status(StatusCode.ERROR, str(e)))
raise
return order
SLIDE 19: Distributed Tracing: Implementation (continued)
Propagating Context Across Services:
CODE
from opentelemetry.propagate import inject, extract
# Outgoing request: inject trace context into headers
def call_payment_service(order):
headers = {}
inject(headers) # Adds trace context headers
response = requests.post(
f"{PAYMENT_URL}/charge",
json=order.to_dict(),
headers=headers
)
return response
# Incoming request: extract trace context from headers
def payment_handler(request):
context = extract(request.headers)
with tracer.start_as_current_span("handle_payment", context=context):
# Process payment
pass
Context Propagation Headers:
SLIDE 20: Distributed Tracing: Sampling
Tracing Everything is Expensive
The Cost:
CODE
1 million requests/day
Average 10 spans per request
= 10 million spans/day
= 300 million spans/month
Storage and processing cost: Significant
Sampling Strategies:
SLIDE 21: Distributed Tracing: Sampling (continued)
Head-Based Sampling:
CODE
import random
SAMPLE_RATE = 0.01 # 1% of requests
def should_sample():
return random.random() < SAMPLE_RATE
# At start of request
if should_sample():
span = tracer.start_span("request", sampled=True)
else:
span = tracer.start_span("request", sampled=False)
Tail-Based Sampling (Collector-Side):
CODE
# OpenTelemetry Collector configuration
processors:
tail_sampling:
policies:
-name: errors
type: status_code
status_code:{status_codes:[ERROR]}
-name: slow_requests
type: latency
latency:{threshold_ms:1000}
-name: sample_rest
type: probabilistic
probabilistic:{sampling_percentage:1}
Keep 100% of errors, 100% of slow requests, 1% of everything else.
SLIDE 22: Alerting: Principles
When to Page Someone
Good Alert:
CODE
Actionable: Someone can do something about it
Urgent: Needs attention now, not tomorrow
Real: Indicates actual user impact
Documented: Runbook exists
Bad Alert:
CODE
"CPU is at 80%"
- Is this a problem? Maybe not.
- What should I do? Unclear.
- Are users affected? Unknown.
Better Alert:
CODE
"Error rate exceeded 1% for 5 minutes"
- Users are seeing errors
- Action: Check logs, rollback if recent deploy
- Runbook: /docs/runbooks/high-error-rate
SLIDE 23: Alerting: Alert Design
Symptom-Based vs Cause-Based Alerts
Cause-Based (Avoid):
Symptom-Based (Prefer):
Rule: Alert on symptoms, investigate causes.
SLIDE 24: Alerting: Alert Design (continued)
Alert Severity Levels:
Alert Configuration:
CODE
groups:
-name: api-alerts
rules:
-alert: HighErrorRate
expr:|
sum(rate(http_requests_total{status=~"5.."}[5m])) /
sum(rate(http_requests_total[5m])) > 0.01
for: 5m
labels:
severity: critical
annotations:
summary:"High error rate detected"
description:"Error rate is {{ $value | humanizePercentage }}"
runbook:"https://wiki/runbooks/high-error-rate"
SLIDE 25: Alerting: Reducing Alert Fatigue
Too Many Alerts = Ignored Alerts
Signs of Alert Fatigue:
CODE
- Oncall ignores alerts
- Alerts auto-acknowledged
- "Alert storms" common
- Real issues missed
Solutions:
SLIDE 26: Alerting: Reducing Alert Fatigue (continued)
Alert Hygiene Process:
CODE
Monthly alert review:
1. Which alerts fired?
2. Which were actionable?
3. Which led to real fixes?
4. Which should be deleted?
If alert fired but no action taken:
- Delete it or raise threshold
If same alert fires repeatedly:
- Fix underlying issue or auto-remediate
Alert Aggregation:
CODE
# Instead of alerting for each pod:
-alert: PodHighCPU
expr: container_cpu_usage > 0.9
# Fires 100 times for 100 pods
# Alert once for the service:
-alert: ServiceHighCPU
expr: avg(container_cpu_usage{service="api"}) > 0.8
# Fires once, indicates service-level issue
SLIDE 27: Dashboards: Design Principles
Making Data Visible
Dashboard Hierarchy:
CODE
#mermaid-644e441e-184d-4a2f-a933-0b178f42584a{font-family:ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI Variable Display","Segoe UI",Helvetica,"Apple Color Emoji","Noto Sans Arabic","Noto Sans Hebrew",Arial,sans-serif,"Segoe UI Emoji","Segoe UI Symbol";font-size:16px;fill:#000000;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .error-icon{fill:#552222;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .error-text{fill:#552222;stroke:#552222;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .edge-thickness-normal{stroke-width:1px;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .edge-thickness-thick{stroke-width:3.5px;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .edge-pattern-solid{stroke-dasharray:0;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .marker{fill:#666;stroke:#666;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .marker.cross{stroke:#666;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a svg{font-family:ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI Variable Display","Segoe UI",Helvetica,"Apple Color Emoji","Noto Sans Arabic","Noto Sans Hebrew",Arial,sans-serif,"Segoe UI Emoji","Segoe UI Symbol";font-size:16px;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a p{margin:0;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .label{font-family:ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI Variable Display","Segoe UI",Helvetica,"Apple Color Emoji","Noto Sans Arabic","Noto Sans Hebrew",Arial,sans-serif,"Segoe UI Emoji","Segoe UI Symbol";color:#000000;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .cluster-label text{fill:#333;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .cluster-label span{color:#333;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .cluster-label span p{background-color:transparent;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .label text,#mermaid-644e441e-184d-4a2f-a933-0b178f42584a span{fill:#000000;color:#000000;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .node rect,#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .node circle,#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .node ellipse,#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .node polygon,#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .node path{fill:#eee;stroke:#999;stroke-width:1px;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .rough-node .label text,#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .node .label text,#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .image-shape .label,#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .icon-shape .label{text-anchor:middle;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .rough-node .label,#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .node .label,#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .image-shape .label,#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .icon-shape .label{text-align:center;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .node.clickable{cursor:pointer;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .root .anchor path{fill:#666!important;stroke-width:0;stroke:#666;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .arrowheadPath{fill:#333333;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .edgePath .path{stroke:#666;stroke-width:1px;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .flowchart-link{stroke:#666;fill:none;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .edgeLabel{background-color:white;text-align:center;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .edgeLabel p{background-color:white;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .edgeLabel rect{opacity:0.5;background-color:white;fill:white;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .labelBkg{background-color:rgba(255, 255, 255, 0.5);}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .cluster rect{fill:hsl(0, 0%, 98.9215686275%);stroke:#707070;stroke-width:1px;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .cluster text{fill:#333;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .cluster span{color:#333;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:ui-sans-serif,-apple-system,BlinkMacSystemFont,"Segoe UI Variable Display","Segoe UI",Helvetica,"Apple Color Emoji","Noto Sans Arabic","Noto Sans Hebrew",Arial,sans-serif,"Segoe UI Emoji","Segoe UI Symbol";font-size:12px;background:hsl(-160, 0%, 93.3333333333%);border:1px solid #707070;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#000000;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a rect.text{fill:none;stroke-width:0;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .icon-shape,#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .image-shape{background-color:white;text-align:center;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .icon-shape p,#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .image-shape p{background-color:white;padding:2px;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .icon-shape .label rect,#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .image-shape .label rect{opacity:0.5;background-color:white;fill:white;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a .node .neo-node{stroke:#999;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a [data-look="neo"].node rect,#mermaid-644e441e-184d-4a2f-a933-0b178f42584a [data-look="neo"].cluster rect,#mermaid-644e441e-184d-4a2f-a933-0b178f42584a [data-look="neo"].node polygon{stroke:url(#mermaid-644e441e-184d-4a2f-a933-0b178f42584a-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a [data-look="neo"].swimlane.cluster rect{filter:none;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a [data-look="neo"].node path{stroke:url(#mermaid-644e441e-184d-4a2f-a933-0b178f42584a-gradient);stroke-width:1px;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a [data-look="neo"].node .outer-path{filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a [data-look="neo"].node .neo-line path{stroke:#999;filter:none;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a [data-look="neo"].node circle{stroke:url(#mermaid-644e441e-184d-4a2f-a933-0b178f42584a-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a [data-look="neo"].node circle .state-start{fill:#000000;}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a [data-look="neo"].icon-shape .icon{fill:url(#mermaid-644e441e-184d-4a2f-a933-0b178f42584a-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a [data-look="neo"].icon-shape .icon-neo path{stroke:url(#mermaid-644e441e-184d-4a2f-a933-0b178f42584a-gradient);filter:drop-shadow( 1px 2px 2px rgba(185,185,185,1));}#mermaid-644e441e-184d-4a2f-a933-0b178f42584a :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;}Service OverviewComponent DetailsIndividual Instance
Level 1: Service Overview
CODE
At a glance:
- Is the service healthy? (Green/Red)
- Error rate trend
- Latency P50/P99
- Request rate
Level 2: Component Details
CODE
Drill down:
- Database metrics
- Cache hit rate
- Dependency health
- Resource utilization
Level 3: Instance Details
CODE
Deep dive:
- Individual pod metrics
- Specific queries
- Debug information
SLIDE 28: Dashboards: The Four Golden Signals
What Every Service Dashboard Needs
The Four Golden Signals (Google SRE):
Dashboard Layout:
Structure & Concept Flow
SLIDE 29: Dashboards: The Four Golden Signals (continued)
Dashboard Implementation:
CODE
Panel 1: Latency
- P50 line (green)
- P95 line (yellow)
- P99 line (red)
- SLO threshold line (dashed)
Panel 2: Traffic
- Requests per second
- Compare to same time yesterday (dashed)
Panel 3: Errors
- Error rate percentage
- Breakdown by error type (stacked)
Panel 4: Saturation
- CPU utilization gauge
- Memory utilization gauge
- Connection pool usage
- Queue depth
Contextual Information:
CODE
- Recent deployments (vertical lines on charts)
- Active incidents (status banner)
- SLO budget remaining (percentage)
- Link to runbooks
SLIDE 30: Observability: Correlation
Connecting Logs, Metrics, and Traces
The Investigation Flow:
CODE
1. Alert fires: "Error rate > 1%"
2. Check dashboard: Error spike at 10:23
3. Check metrics: Errors from /checkout endpoint
4. Filter traces: Traces for /checkout with errors
5. Find trace: trace_id = abc123
6. View logs: All logs with trace_id = abc123
7. Root cause: Database connection timeout
Enabling Correlation:
CODE
# Include trace_id in logs
logger.info(
"checkout_failed",
trace_id=trace_id_var.get(),
user_id=user_id,
error="db_timeout"
)
# Include trace_id in metrics (via exemplars)
request_duration.observe(
duration,
{"trace_id": trace_id_var.get()}
)
SLIDE 31: Observability: Correlation (continued)
Exemplars: Linking Metrics to Traces
CODE
from prometheus_client import Histogram
request_duration = Histogram(
"request_duration_seconds",
"Request duration",
["endpoint"]
)
def handle_request(request):
start = time.time()
response = process(request)
duration = time.time() - start
# Record metric with exemplar (trace_id)
request_duration.labels(endpoint=request.path).observe(
duration,
exemplar={"trace_id": get_trace_id()}
)
In Grafana:
CODE
1. See spike in P99 latency
2. Click on the spike
3. See exemplar trace_id
4. Click to open trace in Jaeger/Tempo
5. See exactly what happened in that slow request
SLIDE 32: Observability: Anti-Patterns
Common Mistakes to Avoid
Anti-Pattern 1: Log Everything
CODE
# Bad: Logs every detail
logger.debug(f"Entering function process_order")
logger.debug(f"Order data:{order}")
logger.debug(f"Calling database")
logger.debug(f"Database returned:{result}")
logger.debug(f"Exiting function process_order")
# Good: Log meaningful events
logger.info("order_processed", order_id=order.id, duration_ms=duration)
Anti-Pattern 2: Metrics Without Labels
CODE
# Bad: Cannot filter by endpoint
requests = Counter("requests_total")
# Good: Can filter and aggregate
requests = Counter("requests_total", "Requests", ["method", "endpoint", "status"])
SLIDE 33: Observability: Anti-Patterns (continued)
Anti-Pattern 3: High Cardinality Labels
CODE
# Bad: user_id has millions of values
requests = Counter("requests_total", labels=["user_id"]) # Explodes metric storage
# Good: Use bounded labels
requests = Counter("requests_total", labels=["user_type"]) # free, premium, enterprise
Anti-Pattern 4: No Correlation IDs
CODE
# Bad: Cannot connect logs across services
logger.info("Payment processed")
# Good: Can trace request across services
logger.info("payment_processed", trace_id=trace_id, order_id=order_id)
Anti-Pattern 5: Alert on Causes, Not Symptoms
CODE
# Bad: Alert when CPU is high (may not affect users)
alert: CPUHigh
expr: cpu_usage > 0.8
# Good: Alert when users are affected
alert: HighErrorRate
expr: error_rate > 0.01
SLIDE 34: Observability: Summary
Key Takeaways
Logging:
- Use structured logging (JSON)
- Include trace_id in every log
- Log events, not debug statements
- Sanitize sensitive data
Metrics:
- Use RED for request metrics (Rate, Errors, Duration)
- Use USE for resource metrics (Utilization, Saturation, Errors)
- Define SLIs and track error budgets
- Avoid high cardinality labels
SLIDE 35: Observability: Summary (continued)
Tracing:
- Trace requests across all services
- Propagate context via headers
- Use sampling to control costs
- Link traces to logs and metrics
Alerting:
- Alert on symptoms, not causes
- Every alert must be actionable
- Include runbook links
- Review and prune alerts regularly
Dashboards:
- Show the four golden signals
- Provide drill-down capability
- Include deployment markers
- Link to traces and logs
Coming Up Next: Section - Deployment Strategies