SLIDE 132: Section: Incident Management
When Things Go Wrong
Structure & Concept Flow
Incident Reality:
Key Insight:
The goal is not zero incidents. The goal is fast detection, fast mitigation, and learning from every incident.
SLIDE 133: Incident Detection: Alerting
Finding Problems Before Users Do
Detection Sources:
Alert Severity Mapping:
CODE
SEVERITY_DEFINITIONS = {
"SEV1": {
"criteria": "Complete service outage, data loss risk",
"response_time": "Immediate",
"notification": "Page oncall + management",
"example": "API returning 500 for all requests"
},
"SEV2": {
"criteria": "Major feature broken, significant user impact",
"response_time": "15 minutes",
"notification": "Page oncall",
"example": "Checkout flow failing for 20% of users"
},
"SEV3": {
"criteria": "Minor feature broken, limited impact",
"response_time": "4 hours",
"notification": "Slack alert",
"example": "Email notifications delayed"
}
}
SLIDE 134: Incident Response: First Minutes
The Golden Hour
First 5 Minutes:
CODE
1. ACKNOWLEDGE
- Oncall acknowledges alert
- Join incident channel
- Declare incident severity
2. ASSESS
- What is broken?
- Who is affected?
- What changed recently?
3. COMMUNICATE
- Update status page
- Notify stakeholders
- Set expectations
Incident Commander Role:
SLIDE 135: Incident Response: First Minutes (continued)
Incident Channel Setup:
Structure & Concept Flow
SLIDE 136: Incident Mitigation: Common Actions
Stopping the Bleeding
Mitigation Playbook:
Rollback Decision:
CODE
def should_rollback():
"""
Rollback if:
1. Recent deploy (< 1 hour)
2. Clear correlation with issue
3. Rollback is safe (no DB migrations)
"""
recent_deploy = get_last_deploy()
if recent_deploy.time > now() - timedelta(hours=1):
if metrics.error_rate_increased_after(recent_deploy.time):
if recent_deploy.has_safe_rollback():
return True
return False
SLIDE 137: Incident Mitigation: Common Actions (continued)
Mitigation vs Resolution:
CODE
MITIGATION (Do first):
- Stop the immediate bleeding
- Restore service even if degraded
- Example: Rollback bad deploy
RESOLUTION (Do after stable):
- Fix the root cause
- Prevent recurrence
- Example: Fix the bug, add tests, redeploy
Degraded Operation Modes:
CODE
class DegradedMode:
@staticmethod
def enable_read_only():
"""Disable writes to protect data integrity."""
feature_flags.disable("write_operations")
cache.extend_ttl(factor=10)
@staticmethod
def enable_essential_only():
"""Disable non-essential features to reduce load."""
feature_flags.disable("recommendations")
feature_flags.disable("analytics")
feature_flags.disable("notifications")
@staticmethod
def enable_static_fallback():
"""Serve static content when dynamic fails."""
feature_flags.enable("static_fallback_mode")
SLIDE 138: Incident Communication: Status Updates
Keeping Everyone Informed
Status Page Updates:
CODE
INVESTIGATING (14:35)
We are investigating reports of checkout failures.
Some users may experience errors when completing purchases.
IDENTIFIED (14:45)
We have identified the issue as a problem with our
payment processing service. We are working on a fix.
MONITORING (15:10)
A fix has been deployed. We are monitoring the situation.
Checkout should now be working normally.
RESOLVED (15:45)
This incident has been resolved. Checkout is fully operational.
We apologize for the inconvenience.
Communication Cadence:
SLIDE 139: Incident Communication: Status Updates (continued)
Internal vs External Communication:
Customer Communication Template:
CODE
Subject: [Resolved] Checkout Service Disruption
What happened:
Between 14:32 and 15:45 UTC on January 15, some users
experienced errors when attempting to complete purchases.
Impact:
Approximately 3,000 checkout attempts failed during this period.
Root cause:
A configuration change caused our payment processing to fail
for certain payment methods.
Resolution:
We rolled back the configuration change and verified all
payment methods are working correctly.
Prevention:
We are implementing additional validation for configuration
changes to prevent similar issues.
SLIDE 140: Post-Mortem: Blameless Analysis
Learning from Incidents
Post-Mortem Goals:
Post-Mortem Template:
CODE
INCIDENT POST-MORTEM: 2025-01-15 API Outage
Summary:
API returned 500 errors for 73 minutes due to misconfigured
rate limiter deployed at 14:30 UTC.
Impact:
- Duration: 73 minutes
- Users affected: ~15,000
- Failed requests: ~45,000
- Revenue impact: ~$12,000
Timeline:
[Detailed timeline of events]
Root cause:
[Technical explanation]
Contributing factors:
[Systemic issues that allowed this to happen]
Action items:
[Specific, assigned tasks with due dates]
SLIDE 141: Post-Mortem: Blameless Analysis (continued)
Root Cause Analysis: 5 Whys
Structure & Concept Flow
SLIDE 142: Post-Mortem: Action Items
Turning Learnings into Improvements
Good Action Items:
Bad Action Items:
CODE
❌ "Be more careful"
❌ "Add more monitoring"
❌ "Improve testing"
❌ "Review processes"
Action Item Categories:
CODE
DETECTION (find problems faster):
- Add alert for rate limiter config changes
- Add synthetic checkout monitoring
MITIGATION (reduce impact):
- Implement config rollback automation
- Add rate limiter circuit breaker
PREVENTION (stop it happening):
- Require review for config changes
- Add config validation in CI
SLIDE 143: Post-Mortem: Action Items (continued)
Tracking Action Items:
CODE
class ActionItemTracker:
def __init__(self):
self.items = []
def add_item(self, incident_id, item):
self.items.append({
"incident_id": incident_id,
"description": item.description,
"owner": item.owner,
"due_date": item.due_date,
"status": "open",
"priority": item.priority
})
def weekly_report(self):
overdue = [i for i in self.items
if i["due_date"] < today() and i["status"] == "open"]
if overdue:
alert_engineering_leadership(
f"{len(overdue)} overdue post-mortem action items"
)
return {
"total_open": len([i for i in self.items if i["status"] == "open"]),
"overdue": len(overdue),
"completed_this_week": len([i for i in self.items
if i["completed_date"] > week_ago()])
}
SLIDE 144: Incident Management: On-Call
Sustainable Operations
On-Call Principles:
On-Call Handoff:
CODE
ON-CALL HANDOFF: 2025-01-15
OUTGOING: @alice
INCOMING: @bob
Active issues:
- None
Recent incidents:
- 2025-01-13: Redis failover, resolved
Action items assigned to @charlie
Things to watch:
- Deploying new payment integration Tuesday
- Higher traffic expected for sale on Thursday
Runbooks updated:
- Updated database failover runbook
- Added new Kafka troubleshooting guide
Questions for incoming:
- None
SLIDE 145: Incident Management: On-Call (continued)
Reducing On-Call Burden:
CODE
class OnCallHealthMetrics:
def weekly_report(self):
return {
"pages_received": count_pages(this_week),
"pages_per_day": count_pages(this_week) / 7,
"night_pages": count_pages(this_week, hours=(22, 6)),
"false_positives": count_false_positives(this_week),
"mean_time_to_ack": mean(page_ack_times),
"mean_time_to_resolve": mean(incident_durations),
"toil_hours": estimate_toil_hours(),
}
def suggest_improvements(self):
suggestions = []
if self.night_pages > 5:
suggestions.append("Reduce night pages: review alerting thresholds")
if self.false_positives > 0.3 * self.pages_received:
suggestions.append("High false positive rate: tune alerts")
if self.toil_hours > 10:
suggestions.append("High toil: automate repetitive tasks")
return suggestions
Target Metrics:
SLIDE 146: Incident Management: Summary
Key Takeaways
Detection:
- Multiple detection sources (synthetic, RUM, logs)
- Clear severity definitions
- Alert on symptoms, not causes
Response:
- Acknowledge fast, communicate faster
- Incident commander coordinates
- Mitigate first, investigate second
Post-Mortem:
- Blameless culture
- 5 Whys for root cause
- Specific, assigned action items
- Track completion
On-Call:
- Sustainable practices
- Clear handoffs
- Reduce toil continuously