Chapter 11: HEALTH CHECKS & CHAOS ENGINEERING - System Design Course: Complete Production Architecture | PlayBook Library

SLIDE 162: Health Checks: Liveness vs Readiness

Two Types of Health Checks

Liveness Check:

CODE

Question: Is the process running and not deadlocked?
Action if fails: Restart the container/process

CODE

@app.get("/health/live")
def liveness():
    # Simple check: can we respond?
    return {"status": "alive"}

Readiness Check:

CODE

Question: Is the service ready to handle traffic?
Action if fails: Remove from load balancer, do not restart

CODE

@app.get("/health/ready")
def readiness():
    checks = {
        "database": check_database_connection(),
        "cache": check_cache_connection(),
        "dependencies": check_critical_dependencies(),
    }

    if all(checks.values()):
        return {"status": "ready", "checks": checks}
    else:
        return Response(
            status=503,
            body={"status": "not ready", "checks": checks}
        )

SLIDE 163: Health Checks: Liveness vs Readiness (continued)

When Each Check Fails:

Check	Fails When	Platform Action
Liveness	Deadlock, infinite loop, memory corruption	Kill and restart pod
Readiness	Database down, warming up, overloaded	Remove from load balancer

Common Mistakes:

CODE

# Bad: Liveness check that depends on database
@app.get("/health/live")
def liveness():
    db.query("SELECT 1")  # If DB is slow, pod gets killed!
    return {"status": "alive"}

# Good: Liveness check is simple
@app.get("/health/live")
def liveness():
    return {"status": "alive"}

# Database check belongs in readiness
@app.get("/health/ready")
def readiness():
    try:
        db.query("SELECT 1")
        return {"status": "ready"}
    except:
        return Response(status=503, body={"status": "not ready"})

SLIDE 164: Health Checks: Deep Health Checks

Checking Dependencies

CODE

def check_database_connection():
    try:
        db.query("SELECT 1")
        return True
    except Exception:
        return False

def check_cache_connection():
    try:
        cache.ping()
        return True
    except Exception:
        return False

def check_downstream_service(service_name, url):
    try:
        response = requests.get(f"{url}/health", timeout=2)
        return response.status_code == 200
    except Exception:
        return False

@app.get("/health/deep")
def deep_health():
    checks = {
        "database": check_database_connection(),
        "cache": check_cache_connection(),
        "user_service": check_downstream_service("user", USER_SERVICE_URL),
        "payment_service": check_downstream_service("payment", PAYMENT_SERVICE_URL),
    }

    all_healthy = all(checks.values())
    status_code = 200 if all_healthy else 503

    return Response(
        status=status_code,
        body={"status": "healthy" if all_healthy else "degraded", "checks": checks}
    )

SLIDE 165: Health Checks: Startup Probes

Handling Slow-Starting Applications

The Problem:

CODE

Application takes 60 seconds to start (loading ML model, warming cache)
Liveness check: 10 second timeout
After 10 seconds, platform kills pod
Pod never finishes starting

Solution: Startup Probe

CODE

# Kubernetes configuration
startupProbe:
httpGet:
path: /health/startup
port:8080
failureThreshold:30
periodSeconds:10
  # Total: 30 × 10 = 300 seconds to start

livenessProbe:
httpGet:
path: /health/live
port:8080
initialDelaySeconds:0 # Starts after startup probe succeeds
periodSeconds:10

Startup Endpoint:

CODE

startup_complete = False

@app.get("/health/startup")
def startup():
    if startup_complete:
        return {"status": "started"}
    else:
        return Response(status=503, body={"status": "starting"})

SLIDE 166: Chaos Engineering: Principles

Testing Failure Handling in Production

What is Chaos Engineering:

CODE

Deliberately inject failures to verify system handles them correctly.

Chaos Engineering Principles:

Principle	Description
Build hypothesis	“System will continue serving 95% of requests if cache fails”
Vary real-world events	Simulate actual failure modes
Run in production	Staging cannot replicate production complexity
Minimize blast radius	Start small, contain failures
Automate experiments	Repeatable, scheduled tests

SLIDE 167: Chaos Engineering: Types of Experiments

Common Chaos Experiments

Infrastructure Chaos:

Experiment	What It Tests
Kill instance	Auto-scaling, load balancing
Fill disk	Disk pressure handling
Network latency	Timeout configuration
Network partition	Split-brain handling
CPU stress	Throttling, queue buildup

Application Chaos:

Experiment	What It Tests
Dependency failure	Circuit breakers, fallbacks
Slow responses	Timeout handling
Error injection	Error handling paths
Resource exhaustion	Connection pool limits

SLIDE 168: Chaos Engineering: Implementation

Simple Chaos Injection

CODE

import random

CHAOS_CONFIG = {
    "enabled": True,
    "failure_rate": 0.01,  # 1% of requests
    "latency_rate": 0.05,  # 5% of requests
    "latency_ms": 2000,
}

def chaos_middleware(request, call_next):
    if not CHAOS_CONFIG["enabled"]:
        return call_next(request)

    # Inject random failure
    if random.random() < CHAOS_CONFIG["failure_rate"]:
        log.info("Chaos: injecting failure")
        return Response(status=500, body="Chaos failure")

    # Inject latency
    if random.random() < CHAOS_CONFIG["latency_rate"]:
        delay = CHAOS_CONFIG["latency_ms"] / 1000
        log.info(f"Chaos: injecting{delay}s latency")
        time.sleep(delay)

    return call_next(request)

SLIDE 169: Chaos Engineering: Implementation (continued)

Targeted Chaos:

CODE

class ChaosMonkey:
    def __init__(self):
        self.rules = {}

    def add_rule(self, service, failure_rate=0, latency_ms=0):
        self.rules[service] = {
            "failure_rate": failure_rate,
            "latency_ms": latency_ms
        }

    def should_fail(self, service):
        rule = self.rules.get(service, {})
        return random.random() < rule.get("failure_rate", 0)

    def get_latency(self, service):
        rule = self.rules.get(service, {})
        return rule.get("latency_ms", 0)

chaos = ChaosMonkey()
chaos.add_rule("payment_service", failure_rate=0.1)  # 10% failures
chaos.add_rule("user_service", latency_ms=500)  # 500ms added latency

def call_service(service_name, func):
    if chaos.should_fail(service_name):
        raise ChaosInjectedError(f"Chaos:{service_name} failed")

    latency = chaos.get_latency(service_name)
    if latency > 0:
        time.sleep(latency / 1000)

    return func()

SLIDE 170: Chaos Engineering: Safe Practices

Running Chaos Safely

Blast Radius Control:

CODE

class SafeChaos:
    def __init__(self):
        self.enabled = False
        self.affected_percent = 1  # Only 1% of requests
        self.excluded_paths = ["/checkout", "/payment"]

    def should_inject(self, request):
        if not self.enabled:
            return False

        # Never inject on critical paths
        if any(request.path.startswith(p) for p in self.excluded_paths):
            return False

        # Only affect small percentage
        return random.random() < (self.affected_percent / 100)

Chaos Safety Rules:

Rule	Why
Start with 0.1% of traffic	Validate monitoring first
Exclude critical paths	Do not break payments
Have kill switch	Stop immediately if needed
Run during business hours	Engineers available to respond
Monitor closely	Detect unexpected cascades

SLIDE 171: Reliability: Anti-Patterns

Common Mistakes to Avoid

Anti-Pattern 1: No Timeouts

CODE

# Bad: No timeout
response = requests.get(url)  # Can hang forever

# Good: Always timeout
response = requests.get(url, timeout=(3, 10))

Anti-Pattern 2: Retry Without Backoff

CODE

# Bad: Immediate retry
for _ in range(3):
    try:
        return call_service()
    except:
        pass  # Retry immediately

# Good: Exponential backoff
for attempt in range(3):
    try:
        return call_service()
    except:
        time.sleep(2 ** attempt)

SLIDE 172: Reliability: Anti-Patterns (continued)

Anti-Pattern 3: Retry Non-Idempotent Operations

CODE

# Bad: Retrying payment without idempotency key
for _ in range(3):
    charge_card(amount)  # May charge 3 times!

# Good: Use idempotency key
for _ in range(3):
    charge_card(amount, idempotency_key=order_id)

Anti-Pattern 4: Circuit Breaker on Wrong Level

CODE

# Bad: One circuit for all database operations
db_circuit.call(lambda: db.query(any_query))

# Good: Separate circuits for different operations
user_queries_circuit.call(lambda: db.query(user_query))
order_queries_circuit.call(lambda: db.query(order_query))

Anti-Pattern 5: Health Check That Depends on External Services

CODE

# Bad: Liveness check calls external service
@app.get("/health/live")
def liveness():
    external_service.ping()  # If slow, pod killed!

# Good: Liveness is self-contained
@app.get("/health/live")
def liveness():
    return {"status": "alive"}

SLIDE 173: Reliability: Summary

Key Takeaways

Timeouts:

Every external call must have a timeout
Use cascading timeouts: inner < outer
Separate connection and read timeouts

Retries:

Use exponential backoff with jitter
Implement retry budgets to limit amplification
Only retry idempotent operations (or use idempotency keys)

Circuit Breakers:

Prevent cascade failures by failing fast
Three states: Closed, Open, Half-Open
Configure per-operation, not globally
Always have a fallback strategy

SLIDE 174: Reliability: Summary (continued)

Bulkheads:

Isolate failures by separating resource pools
Size based on expected load and latency
Monitor utilization to detect exhaustion

Graceful Degradation:

Prioritize features: critical, important, optional
Drop optional features under load
Partial functionality beats total failure

Load Shedding:

Reject requests to protect system capacity
Shed low-priority requests first
Return 503 with Retry-After header

Health Checks:

Liveness: is process running?
Readiness: can it handle traffic?
Use startup probes for slow-starting apps

SLIDE 175: Section 6: Observability

Understanding What Your System is Doing

Structure & Concept Flow

graph LR

A[Logs] --> B[Metrics]

B --> C[Traces]

C --> D[Alerting]

D --> E[Dashboards]

The Three Pillars of Observability:

Pillar	What It Captures	Use Case
Logs	Discrete events	Debugging, audit trail
Metrics	Aggregated measurements	Trends, alerting
Traces	Request journey	Performance, dependencies

Key Insight:
You cannot fix what you cannot see. Observability is not optional in production.

SLIDE 176: Logs: Structured Logging

Why Structure Matters

Unstructured Log (Bad):

CODE

2025-01-15 10:23:45 ERROR - Payment failed for user 12345 order 67890 amount $99.99 reason: card declined

Parsing this requires regex. Every log format is different.

Structured Log (Good):

CODE

{
  "timestamp": "2025-01-15T10:23:45.123Z",
  "level": "ERROR",
  "message": "Payment failed",
  "user_id": 12345,
  "order_id": 67890,
  "amount": 99.99,
  "currency": "USD",
  "reason": "card_declined",
  "trace_id": "abc123",
  "service": "payment-service",
  "host": "payment-prod-1"
}

Queryable. Filterable. Aggregatable.

SLIDE 177: Logs: Structured Logging (continued)

Structured Logging Implementation:

CODE

import structlog
import logging

# Configure structured logging
structlog.configure(
    processors=[
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.JSONRenderer()
    ],
    wrapper_class=structlog.BoundLogger,
    context_class=dict,
    logger_factory=structlog.PrintLoggerFactory(),
)

logger = structlog.get_logger()

def process_payment(user_id, order_id, amount):
    log = logger.bind(user_id=user_id, order_id=order_id, amount=amount)

    log.info("payment_started")

    try:
        result = charge_card(amount)
        log.info("payment_succeeded", transaction_id=result.id)
        return result
    except CardDeclinedError as e:
        log.error("payment_failed", reason="card_declined", error=str(e))
        raise

SLIDE 178: Logs: What to Log

Logging Best Practices

Always Log:

Event	Fields to Include
Request received	method, path, user_id, trace_id
Request completed	status_code, duration_ms, trace_id
Error occurred	error_type, error_message, stack_trace
External call made	service, endpoint, duration_ms
Business event	event_type, entity_id, relevant_data

Never Log:

Sensitive Data	Why
Passwords	Security risk
Credit card numbers	PCI compliance
SSN/government IDs	Privacy regulations
API keys/secrets	Security risk
Full request bodies	May contain PII

SLIDE 179: Logs: What to Log (continued)

Sanitizing Sensitive Data:

CODE

SENSITIVE_FIELDS = {"password", "credit_card", "ssn", "api_key", "token"}

def sanitize(data):
    if isinstance(data, dict):
        return {
            k: "[REDACTED]" if k.lower() in SENSITIVE_FIELDS else sanitize(v)
            for k, v in data.items()
        }
    return data

def log_request(request):
    sanitized = sanitize({
        "path": request.path,
        "method": request.method,
        "body": request.body,
        "headers": dict(request.headers),
    })
    logger.info("request_received", **sanitized)

Log Levels:

Level	When to Use
DEBUG	Detailed debugging info (not in production)
INFO	Normal operations, business events
WARNING	Unexpected but handled situations
ERROR	Failures requiring attention
CRITICAL	System is unusable

SLIDE 180: Logs: Correlation IDs

Tracing Requests Across Services

The Problem:

CODE

User complains: "My order failed"
You have 100 services, 1 billion log lines
Which logs are relevant?

Solution: Correlation ID (Trace ID)

Structure & Concept Flow

graph LR

A[API Gateway] -->|trace_id=abc123| B[Order Service]

B -->|trace_id=abc123| C[Payment Service]

B -->|trace_id=abc123| D[Inventory Service]

C -->|trace_id=abc123| E[Bank API]

All logs for one request share the same trace_id.

CODE

Query: trace_id = "abc123"
Result: All logs from all services for that request

SLIDE 181: Logs: Correlation IDs (continued)

Implementation:

CODE

import uuid
from contextvars import ContextVar

trace_id_var: ContextVar[str] = ContextVar("trace_id", default="")

def trace_id_middleware(request, call_next):
    # Get trace_id from header or generate new one
    trace_id = request.headers.get("X-Trace-ID") or str(uuid.uuid4())

    # Store in context for this request
    trace_id_var.set(trace_id)

    # Add to response headers
    response = call_next(request)
    response.headers["X-Trace-ID"] = trace_id

    return response

def get_logger():
    return logger.bind(trace_id=trace_id_var.get())

# Usage
def process_order(order):
    log = get_logger()
    log.info("processing_order", order_id=order.id)

    # When calling another service, pass trace_id
    response = requests.post(
        payment_url,
        headers={"X-Trace-ID": trace_id_var.get()}
    )

SLIDE 182: Metrics: Types of Metrics

What to Measure

Counter:

CODE

Total count of events. Only goes up.

CODE

# Examples
requests_total = Counter("requests_total", "Total requests")
errors_total = Counter("errors_total", "Total errors")
orders_created = Counter("orders_created", "Orders created")

def handle_request():
    requests_total.inc()

Gauge:

CODE

Current value. Can go up or down.

CODE

# Examples
active_connections = Gauge("active_connections", "Current connections")
queue_depth = Gauge("queue_depth", "Messages in queue")
memory_usage = Gauge("memory_usage_bytes", "Memory usage")

def on_connect():
    active_connections.inc()

def on_disconnect():
    active_connections.dec()

SLIDE 183: Metrics: Types of Metrics (continued)

Histogram:

CODE

Distribution of values. Tracks buckets.

CODE

# Examples
request_duration = Histogram(
    "request_duration_seconds",
    "Request latency",
    buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
)

response_size = Histogram(
    "response_size_bytes",
    "Response size",
    buckets=[100, 1000, 10000, 100000, 1000000]
)

def handle_request():
    start = time.time()
    response = process()

    request_duration.observe(time.time() - start)
    response_size.observe(len(response.body))

Summary:

CODE

Similar to histogram but calculates percentiles client-side.
Use histogram unless you need specific quantiles.

SLIDE 184: Metrics: The RED Method

Request-focused Metrics

RED Metrics:

Metric	What It Measures	Example
Rate	Requests per second	1000 req/s
Errors	Failed requests per second	5 errors/s
Duration	Latency distribution	P50=50ms, P99=500ms

Implementation:

CODE

from prometheus_client import Counter, Histogram

# Rate
requests_total = Counter(
    "http_requests_total",
    "Total HTTP requests",
    ["method", "endpoint", "status"]
)

# Errors (subset of requests_total where status >= 500)

# Duration
request_duration = Histogram(
    "http_request_duration_seconds",
    "Request duration",
    ["method", "endpoint"]
)

def handle_request(request):
    start = time.time()
    response = process(request)

    requests_total.labels(
        method=request.method,
        endpoint=request.path,
        status=response.status
    ).inc()

    request_duration.labels(
        method=request.method,
        endpoint=request.path
    ).observe(time.time() - start)

SLIDE 185: Metrics: The USE Method

Resource-focused Metrics

USE Metrics:

Metric	What It Measures	Example
Utilization	% of resource used	CPU 75%
Saturation	Work waiting	Queue depth 100
Errors	Error count	5 disk errors

Apply USE to Each Resource:

Resource	Utilization	Saturation	Errors
CPU	% CPU used	Load average	N/A
Memory	% RAM used	Swap usage	OOM kills
Disk	% capacity	I/O queue depth	Disk errors
Network	Bandwidth used	Packet drops	Connection errors
Thread pool	Active/total	Queue depth	Rejected tasks
Connection pool	Used/max	Wait time	Connection errors

SLIDE 186: Metrics: Application Metrics

Business and Application-Level Metrics

Business Metrics:

CODE

# Orders
orders_created = Counter("orders_created_total", "Total orders")
order_value = Histogram("order_value_dollars", "Order value distribution")
orders_by_status = Counter("orders_by_status", "Orders by status", ["status"])

# Users
signups = Counter("user_signups_total", "User signups")
active_users = Gauge("active_users", "Currently active users")

# Revenue
revenue = Counter("revenue_cents_total", "Total revenue in cents")

Dependency Metrics:

CODE

# External service calls
external_calls = Counter(
    "external_calls_total",
    "External service calls",
    ["service", "endpoint", "status"]
)

external_duration = Histogram(
    "external_call_duration_seconds",
    "External call latency",
    ["service", "endpoint"]
)

# Circuit breaker state
circuit_state = Gauge(
    "circuit_breaker_state",
    "Circuit breaker state (0=closed, 1=open, 0.5=half-open)",
    ["service"]
)

SLIDE 187: Metrics: Percentiles and SLIs

Understanding Latency Distribution

Why Percentiles Matter:

CODE

Average latency: 100ms
Sounds good, right?

But:
- P50 (median): 50ms
- P90: 100ms
- P99: 2000ms
- P99.9: 10000ms

1 in 1000 requests takes 10 seconds!
At 1M requests/day, that is 1000 terrible experiences.

Service Level Indicators (SLIs):

SLI	Definition	Target (SLO)
Availability	% of successful requests	99.9%
Latency	P99 response time	< 500ms
Error rate	% of 5xx responses	< 0.1%
Throughput	Requests per second	> 10,000

SLIDE 188: Metrics: Percentiles and SLIs (continued)

Calculating SLIs:

CODE

# Availability SLI
availability = (
    sum(requests where status < 500) /
    sum(all requests)
) * 100

# Latency SLI
latency_sli = (
    sum(requests where duration < 500ms) /
    sum(all requests)
) * 100

# PromQL examples
# Availability
sum(rate(requests_total{status!~"5.."}[5m])) /
sum(rate(requests_total[5m]))

# P99 latency
histogram_quantile(0.99, rate(request_duration_seconds_bucket[5m]))

Error Budget:

CODE

SLO: 99.9% availability
Allowed downtime per month: 0.1% = 43 minutes

If you have used 30 minutes of downtime:
- Error budget remaining: 13 minutes
- Slow down risky deployments
- Focus on reliability

SLIDE 189: Distributed Tracing: Concepts

Following Requests Across Services

Why Tracing:

CODE

User reports: "Checkout is slow"

Without tracing:
- Which service is slow?
- Is it the database? Network? CPU?
- 10 services involved, where to look?

With tracing:
- See exact path of request
- See time spent in each service
- Identify the bottleneck

Trace Structure:

Structure & Concept Flow

graph LR

A[Trace: abc123] --> B[Span: API Gateway 5ms]

B --> C[Span: Order Service 100ms]

C --> D[Span: DB Query 80ms]

C --> E[Span: Payment Service 50ms]

E --> F[Span: External API 45ms]

SLIDE 190: Distributed Tracing: Concepts (continued)

Trace Components:

Component	Description
Trace	Entire request journey (one trace_id)
Span	Single operation within trace
Parent span	The calling operation
Child span	Operations called by parent
Tags	Key-value metadata
Logs	Events within a span

Span Example:

CODE

{
  "trace_id": "abc123",
  "span_id": "span456",
  "parent_span_id": "span123",
  "operation_name": "db.query",
  "service": "order-service",
  "start_time": "2025-01-15T10:23:45.000Z",
  "duration_ms": 45,
  "tags": {
    "db.type": "postgresql",
    "db.statement": "SELECT * FROM orders WHERE id = ?",
    "db.rows_affected": 1
  },
  "status": "OK"
}

SLIDE 191: Distributed Tracing: Implementation

Adding Tracing to Code

CODE

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("order-service")

def process_order(order_id):
    with tracer.start_as_current_span("process_order") as span:
        span.set_attribute("order_id", order_id)

        # Child span for database query
        with tracer.start_as_current_span("db.get_order") as db_span:
            db_span.set_attribute("db.system", "postgresql")
            order = db.get_order(order_id)
            db_span.set_attribute("db.rows_returned", 1 if order else 0)

        # Child span for payment
        with tracer.start_as_current_span("payment.charge") as payment_span:
            payment_span.set_attribute("payment.amount", order.total)
            try:
                result = payment_service.charge(order)
                payment_span.set_attribute("payment.transaction_id", result.id)
            except PaymentError as e:
                payment_span.set_status(Status(StatusCode.ERROR, str(e)))
                raise

        return order

SLIDE 192: Distributed Tracing: Implementation (continued)

Propagating Context Across Services:

CODE

from opentelemetry.propagate import inject, extract

# Outgoing request: inject trace context into headers
def call_payment_service(order):
    headers = {}
    inject(headers)  # Adds trace context headers

    response = requests.post(
        f"{PAYMENT_URL}/charge",
        json=order.to_dict(),
        headers=headers
    )
    return response

# Incoming request: extract trace context from headers
def payment_handler(request):
    context = extract(request.headers)

    with tracer.start_as_current_span("handle_payment", context=context):
        # Process payment
        pass

Context Propagation Headers:

Header	Purpose
traceparent	W3C trace context (trace_id, span_id, flags)
tracestate	Vendor-specific trace data
X-B3-TraceId	Zipkin/B3 format

SLIDE 193: Distributed Tracing: Sampling

Tracing Everything is Expensive

The Cost:

CODE

1 million requests/day
Average 10 spans per request
= 10 million spans/day
= 300 million spans/month

Storage and processing cost: Significant

Sampling Strategies:

Strategy	How It Works	Use Case
Head-based	Decide at start of trace	Simple, predictable
Tail-based	Decide at end based on outcome	Keep interesting traces
Rate limiting	Fixed traces per second	Control costs
Priority-based	Sample more for errors/slow	Focus on problems

SLIDE 194: Distributed Tracing: Sampling (continued)

Head-Based Sampling:

CODE

import random

SAMPLE_RATE = 0.01  # 1% of requests

def should_sample():
    return random.random() < SAMPLE_RATE

# At start of request
if should_sample():
    span = tracer.start_span("request", sampled=True)
else:
    span = tracer.start_span("request", sampled=False)

Tail-Based Sampling (Collector-Side):

CODE

# OpenTelemetry Collector configuration
processors:
tail_sampling:
policies:
-name: errors
type: status_code
status_code:{status_codes:[ERROR]}
-name: slow_requests
type: latency
latency:{threshold_ms:1000}
-name: sample_rest
type: probabilistic
probabilistic:{sampling_percentage:1}

Keep 100% of errors, 100% of slow requests, 1% of everything else.

SLIDE 195: Alerting: Principles

When to Page Someone

Good Alert:

CODE

Actionable: Someone can do something about it
Urgent: Needs attention now, not tomorrow
Real: Indicates actual user impact
Documented: Runbook exists

Bad Alert:

CODE

"CPU is at 80%"
- Is this a problem? Maybe not.
- What should I do? Unclear.
- Are users affected? Unknown.

Better Alert:

CODE

"Error rate exceeded 1% for 5 minutes"
- Users are seeing errors
- Action: Check logs, rollback if recent deploy
- Runbook: /docs/runbooks/high-error-rate

SLIDE 196: Alerting: Alert Design

Symptom-Based vs Cause-Based Alerts

Cause-Based (Avoid):

Alert	Problem
CPU > 80%	May not affect users
Disk > 90%	How urgent?
Memory > 80%	Normal for some apps
Pod restarted	May be expected

Symptom-Based (Prefer):

Alert	User Impact
Error rate > 1%	Users seeing errors
P99 latency > 2s	Users experiencing slowness
Success rate < 99%	Orders failing
Queue depth > 10000	Processing delayed

Rule: Alert on symptoms, investigate causes.

SLIDE 197: Alerting: Alert Design (continued)

Alert Severity Levels:

Severity	Response Time	Example
P1 Critical	Immediate (page)	Site down, data loss
P2 High	< 1 hour	Partial outage, major feature broken
P3 Medium	< 4 hours	Degraded performance, minor feature
P4 Low	Next business day	Warning threshold, cleanup needed

Alert Configuration:

CODE

groups:
-name: api-alerts
rules:
-alert: HighErrorRate
        expr:|
          sum(rate(http_requests_total{status=~"5.."}[5m])) /
          sum(rate(http_requests_total[5m])) > 0.01
for: 5m
labels:
severity: critical
annotations:
summary:"High error rate detected"
description:"Error rate is {{ $value | humanizePercentage }}"
runbook:"https://wiki/runbooks/high-error-rate"

SLIDE 198: Alerting: Reducing Alert Fatigue

Too Many Alerts = Ignored Alerts

Signs of Alert Fatigue:

CODE

- Oncall ignores alerts
- Alerts auto-acknowledged
- "Alert storms" common
- Real issues missed

Solutions:

Problem	Solution
Too many alerts	Raise thresholds, reduce scope
Flapping alerts	Add hysteresis (for: 5m)
Duplicate alerts	Group related alerts
Non-actionable	Delete or convert to dashboard
No runbook	Add runbook or delete alert

SLIDE 199: Alerting: Reducing Alert Fatigue (continued)

Alert Hygiene Process:

CODE

Monthly alert review:
1. Which alerts fired?
2. Which were actionable?
3. Which led to real fixes?
4. Which should be deleted?

If alert fired but no action taken:
- Delete it or raise threshold

If same alert fires repeatedly:
- Fix underlying issue or auto-remediate

Alert Aggregation:

CODE

# Instead of alerting for each pod:
-alert: PodHighCPU
expr: container_cpu_usage > 0.9
  # Fires 100 times for 100 pods

# Alert once for the service:
-alert: ServiceHighCPU
expr: avg(container_cpu_usage{service="api"}) > 0.8
  # Fires once, indicates service-level issue

SLIDE 200: Dashboards: Design Principles

Making Data Visible

Dashboard Hierarchy:

Structure & Concept Flow

graph LR

A[Service Overview] --> B[Component Details]

B --> C[Individual Instance]

Level 1: Service Overview

CODE

At a glance:
- Is the service healthy? (Green/Red)
- Error rate trend
- Latency P50/P99
- Request rate

Level 2: Component Details

CODE

Drill down:
- Database metrics
- Cache hit rate
- Dependency health
- Resource utilization

Level 3: Instance Details

CODE

Deep dive:
- Individual pod metrics
- Specific queries
- Debug information

HEALTH CHECKS & CHAOS ENGINEERING

SLIDE 162: Health Checks: Liveness vs Readiness

SLIDE 163: Health Checks: Liveness vs Readiness (continued)

SLIDE 164: Health Checks: Deep Health Checks

SLIDE 165: Health Checks: Startup Probes

SLIDE 166: Chaos Engineering: Principles

SLIDE 167: Chaos Engineering: Types of Experiments

SLIDE 168: Chaos Engineering: Implementation

SLIDE 169: Chaos Engineering: Implementation (continued)

SLIDE 170: Chaos Engineering: Safe Practices

SLIDE 171: Reliability: Anti-Patterns

SLIDE 172: Reliability: Anti-Patterns (continued)

SLIDE 173: Reliability: Summary

SLIDE 174: Reliability: Summary (continued)

SLIDE 175: Section 6: Observability

SLIDE 176: Logs: Structured Logging

SLIDE 177: Logs: Structured Logging (continued)

SLIDE 178: Logs: What to Log

SLIDE 179: Logs: What to Log (continued)

SLIDE 180: Logs: Correlation IDs

SLIDE 181: Logs: Correlation IDs (continued)

SLIDE 182: Metrics: Types of Metrics

SLIDE 183: Metrics: Types of Metrics (continued)

SLIDE 184: Metrics: The RED Method

SLIDE 185: Metrics: The USE Method

SLIDE 186: Metrics: Application Metrics

SLIDE 187: Metrics: Percentiles and SLIs

SLIDE 188: Metrics: Percentiles and SLIs (continued)

SLIDE 189: Distributed Tracing: Concepts

SLIDE 190: Distributed Tracing: Concepts (continued)

SLIDE 191: Distributed Tracing: Implementation

SLIDE 192: Distributed Tracing: Implementation (continued)

SLIDE 193: Distributed Tracing: Sampling

SLIDE 194: Distributed Tracing: Sampling (continued)

SLIDE 195: Alerting: Principles

SLIDE 196: Alerting: Alert Design

SLIDE 197: Alerting: Alert Design (continued)

SLIDE 198: Alerting: Reducing Alert Fatigue

SLIDE 199: Alerting: Reducing Alert Fatigue (continued)

SLIDE 200: Dashboards: Design Principles

You've Reached the Final Chapter!