Monitoring & Observability — Seeing Production

What you'll learn

• Why you can't fix what you can't see
• Metrics, logs, and traces (the 3 pillars)
• Alerts — being told before users complain
• You'll build an alerting threshold checker

The dashboard in the cockpit ✈️

A pilot doesn't fly by feel — they watch instruments: altitude, speed, fuel. Running production software without monitoring is flying blind. Once your automated pipeline deploys code to servers you can't see, you need instruments telling you: is it healthy? is it fast? is it about to crash?

The three pillars of observability:

• Metrics — numbers over time. CPU usage, requests per second, error rate, response time. These are your gauges: "error rate is 0.2%, response time 120ms." Tools: Prometheus collects them, Grafana graphs them into dashboards.
• Logs — timestamped text records of events. "User 5 logged in", "Payment failed: timeout." When something breaks, logs tell you what happened. (You practiced log analysis in the security track.)
• Traces — follow a single request across all the services it touches. In a system with 10 microservices, a trace shows where the 3-second delay actually happened.

Metrics say something is wrong, logs and traces say what and where.

Alerts — the whole point. Collecting data is useless if no one looks until users complain. An alert watches a metric and notifies the team (Slack, email, phone) when it crosses a threshold: "error rate > 5% for 2 minutes → wake up the on-call engineer." Good monitoring means you find out before your users do.

The art is setting thresholds well: too sensitive and you get "alert fatigue" (so many false alarms people ignore them); too loose and you miss real problems. This ties back to the SOC triage from the security track — deciding what's a real signal versus noise.

Why it matters for DevOps: automation deploys fast, but speed without visibility is dangerous. Monitoring closes the loop — deploy, observe, learn, improve. It's how teams run reliable systems at scale and sleep at night.

SLIs, SLOs, and error budgets 🎯

Beyond raw metrics, mature DevOps teams speak a precise language for "how reliable is reliable enough?" — three connected terms, easy to confuse but worth telling apart:

• SLI (Service Level Indicator) — the actual measured number. "99.94% of requests succeeded last month."
• SLO (Service Level Objective) — the internal target for that number. "We aim for 99.9% success rate." This is a goal your team sets for itself.
• SLA (Service Level Agreement) — an external, often contractual, promise to customers, usually with penalties if missed. "99.5% uptime or you get a refund."

Notice the SLO (99.9%) is usually stricter than the SLA (99.5%) — you give yourself a safety margin before you'd ever breach a customer-facing promise.

The error budget is what makes an SLO actionable instead of just aspirational: if your SLO is 99.9% uptime, your error budget is the remaining 0.1% — a concrete, spendable allowance for things going wrong (deploys that cause brief blips, planned maintenance, honest bugs). While budget remains, the team can ship features and take reasonable risks. Once the error budget is exhausted for the period, the team's priority automatically shifts: freeze risky releases, focus entirely on stability, until the budget resets next period.

This turns "be more reliable" from a vague slogan into a measurable, self-enforcing rule that balances shipping speed against reliability — the exact tension DevOps exists to manage.

Real-world case study: AWS's 2017 S3 outage 🔍

In February 2017, a large chunk of the internet went down for about four hours — not because of a hack, but because an Amazon engineer, debugging a slow billing system, ran an authorized maintenance command intended to remove a small number of servers from one Amazon S3 subsystem. A typo-like input error caused it to remove far more capacity than intended, taking two critical internal subsystems offline entirely. Because so many popular websites and apps depended on S3 for storage, the outage cascaded into visible failures across a huge portion of the web.

The incident became famous in the industry for a wry detail: Amazon's own service health dashboard, which was supposed to show the outage status to customers, itself depended on the very S3 systems that had just gone down — so the status page showing "everything is fine" couldn't update. It's a textbook lesson in observability: your monitoring and alerting systems must be resilient independently of the systems they monitor, or you lose visibility at the exact moment you need it most.

Amazon's public post-incident report afterward detailed new safeguards — including slower, smaller removal operations and cooldown mechanisms — mirroring exactly the "guardrails around destructive operations" theme from this DevOps track.

Summary

• You can't fix what you can't see — monitoring gives production visibility
• Three pillars: metrics (numbers), logs (events), traces (request journeys)
• Alerts notify you before users complain; tune thresholds to avoid fatigue
• SLIs measure reality, SLOs set internal targets, SLAs are external promises — and the error budget turns "be reliable" into an actionable, self-enforcing rule
• You built an alert threshold checker and an error-budget calculator
• AWS's 2017 S3 outage shows why monitoring systems must stay independent of what they monitor

Next: DevSecOps — baking security into every stage of the automated pipeline.