Site Reliability Engineering (SRE)

Site Reliability Engineering (SRE)

The SRE Philosophy

Google invented SRE when they realized that traditional ops (manually managing infrastructure separately from developers) didn't scale. SRE applies software engineering to operations: build automation, observability, and runbooks so repeated problems are solved permanently.

SLOs + error budgets: SLOs make reliability measurable. If SLO is 99.9%, error budget = 0.1% of the month (~43 minutes). When budget is full: deploy freely, run experiments. When budget is depleted: reliability work takes precedence over features.

Toil elimination: SRE teams track toil fraction. If > 50% of time is manual repeated work (handling the same alert the same way weekly), the team is in a reactive spiral. Goal: identify toil, automate it, redeploy time to proactive reliability work.

The Four Golden Signals

Monitor these to cover the main ways a service fails users:

1. Latency: time to serve a request (P50, P95, P99)
2. Traffic: demand on the system (req/s, active users)
3. Errors: rate of failed requests (5xx, exceptions, timeouts)
4. Saturation: utilization of constrained resources (CPU, memory, connection pool, disk)

Alerting principles:

• Alert on symptoms (user-visible impact), not causes (CPU > 80% may not affect users)
• Every alert must be actionable — if no action exists, remove the alert
• < 2 pages per on-call shift; more degrades response quality
• Alert on SLO burn rate (budget burning too fast), not on individual errors

Incident Response

Severity: P0 (service down), P1 (significant degradation), P2 (partial impact), P3 (minor).

Roles: Incident Commander (coordinates, delegates, communicates), Technical Lead (diagnoses, mitigates), Communications Lead (status page, stakeholder updates).

Mitigation vs resolution: mitigation stops user impact fast (roll back, disable feature, reroute). Resolution fixes the root cause. During an active incident: prioritize mitigation first — get users back, then fix root cause.

Runbooks

Runbooks: step-by-step guides for specific alerts. Linked directly from the alert definition. Contents: what the alert means → diagnostic commands → decision tree (if X high, do Y) → escalation path.

Example "High P99 Latency" runbook:

1. Check error rate (if high → separate error incident)
2. Check recent deploys → roll back candidate
3. Check DB query time (SELECT * FROM pg_stat_activity)
4. Check CPU → flame graph, consider scaling
5. Check external dependencies
6. Escalate to senior engineer if unclear

Blameless Postmortems

Why blameless: blame → engineers hide problems → missed learning → more incidents. Blameless → honesty → systemic fixes → fewer incidents.

Structure:

1. Summary: impact, duration, resolution
2. Timeline: minute-by-minute from monitoring, deploy logs, Slack, git
3. Root cause: 5-whys. "Alert fired" is not a root cause.
4. Contributing factors: missing docs, inadequate monitoring, thin on-call rotation
5. Action items: specific, assigned, time-bound. Bad: "improve monitoring." Good: "Add P99 latency alert at 200ms for payment checkout endpoint with runbook, assigned to @eng, by end of sprint."

Capacity Planning

Measure current utilization → forecast growth (historical trend + planned launches) → determine headroom (target ~70% max utilization for spikes) → plan capacity additions with lead time.

Load testing: validate capacity assumptions. A service handling 1000 req/s should be tested at 2000 req/s to understand failure modes.