Monitoring

SLO-Driven Observability

Turn metrics, logs, traces, and alerts into reliability decisions.

SLO-Driven Observability

Observability is useful when it helps teams make reliability decisions. Service-level objectives (SLOs) connect telemetry to user experience, alerting, incident response, and product tradeoffs.

Use this guide when setting up Prometheus/Grafana, improving alert quality, or formalizing SRE ownership with Assistance.

Start with user journeys#

Pick a small number of critical user journeys before adding more dashboards:

JourneyPossible SLI
User signs inSuccessful login rate and p95 latency.
Customer completes checkoutSuccessful checkout rate and payment provider error rate.
API client reads dataAvailability, p95/p99 latency, and error ratio.
Background job processes eventsQueue age, success ratio, and processing latency.
Search returns resultsQuery success rate, latency, and index freshness.

SLO design checklist#

  • Define the user journey and owner.
  • Choose an SLI that can be measured from the user's perspective where possible.
  • Set an SLO target that leaves room for maintenance and real-world failure.
  • Define the measurement window and data source.
  • Decide which burn-rate alerts page humans and which create tickets.
  • Add dashboards for current health, error budget, and recent incidents.
  • Review SLOs after incidents and major product changes.

Alert quality rules#

Good alerts are actionable, owned, and tied to user impact. Remove or downgrade alerts that only say a host is noisy, a metric moved without consequence, or a condition self-heals before anyone can act.

Alert typeRoute
Fast error-budget burn on a critical journeyPage the on-call responder.
Slow burn or capacity trendCreate a ticket or backlog item.
Backup failure or certificate expiry riskPage only when the remediation window is short; otherwise ticket.
Dashboard-only diagnostic metricNo alert unless it predicts user-impacting failure.

Reference architecture#

1
Applications and infrastructure
2
  -> Metrics, logs, traces, events
3
  -> Prometheus/OpenTelemetry/log pipeline
4
  -> SLI recording rules and dashboards
5
  -> Alertmanager/on-call/ticket routing
6
  -> Incident review and reliability backlog
7
  -> Product and platform SLO review cadence

Reliability review cadence#

SLOs need a review loop, not just dashboards.

CadenceReview focus
Daily or on-call handoffActive incidents, fast burn alerts, noisy alerts, and unusual user-impacting trends.
WeeklyError-budget burn, top regressions, alert changes, capacity risks, and reliability backlog priority.
MonthlySLO target fit, dashboard usefulness, incident themes, toil reduction, and product tradeoff decisions.
QuarterlyOwnership changes, customer commitments, DR evidence, dependency risk, and investment roadmap.

Record decisions that change the SLO, alert route, or reliability backlog. Otherwise the observability system drifts into a collection of graphs nobody trusts.

Assistance engagement fit#

References#