Engineering teams often hold onto a stubborn belief: the more CPU, memory, and disk sensors you strap onto your servers, the safer production will be from sudden outages. If monitoring watches every breath the OS takes, surely no failure can slip past the on-call engineer, right? In reality, your phone buzzes all night long—and you still find out from morning support tickets that customers couldn't log in or pay for their orders.
What grabbed me in Brent Chapman's article—written by someone who built incident management at Google and Slack—is his razor-sharp diagnosis: we wake people up based on how the machines "feel" instead of checking whether the product actually works for a human. Why does monitoring catch dozens of infrastructure blips while users are still the first to notice a real outage?
The Trap of Easy Sensors and the Cost of Nighttime Alert Fatigue
It all starts with how effortless hardware metrics are to collect. When you spin up monitoring, system stats come free out of the box: query the OS and you instantly get CPU, memory, and network graphs. Measuring how fast a page loads through a customer's eyes, on the other hand, takes real frontend work—you have to account for browser quirks, flaky mobile networks, caching, and the gap between the server sending a response and the UI actually painting on screen.
The temptation is obvious, and that's exactly where teams walk into a trap. Server sensors only describe the state of the hardware. Hook them up to an overnight pager, and a steady stream of false alarms quickly dulls everyone's edge. Chapman lays out a classic scenario: an on-call engineer gets woken up 10 times in one night, and 9 of those pages are brief CPU spikes that resolve on their own or warnings about a disk that won't fill up for a week. That's how alert fatigue sets in. When the tenth page arrives just before dawn—successful logins collapsing to 50%—the exhausted engineer reflexively writes it off as more noise.
To break that cycle, Google introduced a weekly ops review. The team goes through every single alert that woke someone up over the past 7 days and asks whether it demanded immediate action. If a page turned out to be noise, they adjust the threshold, downgrade it to a scheduled ticket, or delete it outright. That gives you a simple yardstick for a healthy on-call rotation: nighttime pages should be rare enough that you can review the entire week's list in part of a single meeting.
Work Metrics vs. Resource Metrics: Shifting from Hardware Health to User Experience
So which signals actually deserve a 3 a.m. wake-up call? In the classic Monitoring 101 guide that Chapman builds on, there's a crisp line between work metrics and resource metrics. Work metrics measure useful output for people: request success rates, response latency, and scheduled background jobs completing on time. Resource metrics only reflect the internal state of components.
Compare two states of a payments service in production. In the first, CPU utilization sits at 95%, yet every payment clears steadily in 80 ms with zero errors. That's not an outage; it's peak efficiency—customers are happy, and you're getting full value out of the cloud compute you're paying for. In the second, CPU drops to a calm 5% because a database connection pool got wedged, and half your buyers are hit with 500 Internal Server Error. Every CPU graph looks peaceful while the business bleeds money in real time.
Trying to rescue CPU alerts with lower thresholds or time-of-day schedules turns into an endless game of whack-a-mole. You're far better off leaning on what the Google SRE book calls symptom-based alerting and the four golden signals—latency, traffic, errors, and saturation—paired with domain-specific product metrics. Contrast a blind alert like "CPU on app-worker-42 exceeded 90%," which forces an engineer to spend 10–15 minutes just figuring out if users are even affected, against concrete signals like these:
- login success rate dropped below 90%, pointing directly to an auth failure;
- homepage load time exceeded 10 seconds for 5 consecutive minutes;
- user accounts are getting locked out en masse due to repeated payment failures;
- transaction failure rate crossed 2%;
- search relevance (the share of useful results returned) fell below threshold.
Any one of those tells the on-call engineer from the very first second what broke for users. A team watching three honest customer-facing metrics is better protected than a department staring at a hundred hardware dashboards. A similar trap shows up in cluster autoscaling too: we looked at why reacting to core utilization lags behind traffic spikes in our piece on MTSS metrics and cold starts.
Urgency Decides Everything: Why Resource Sensors Belong in Tickets and Dashboards
You're probably thinking: wait, can we really ignore disks and memory until a server crashes? Of course you shouldn't ignore them—you just need to separate alerting from paging. The deciding factor is time to failure.
If a disk trend says it will fill up in a week, customers aren't hurting yet, and clearing logs at 3 a.m. makes zero sense. Waking an engineer just to file a morning to-do item is counterproductive: if the runway is over 24 hours, automation should open a ticket in the daytime queue—the same goes for memory growth trends or expiring certificates. A pager wake-up call is only justified when the disk will run out in under 1 hour and a crash is imminent.
Emergency room doctors have a saying: 'Treat the patient, not the monitor.' A flatline on the screen could mean cardiac arrest—or just a sensor lead that slipped off. In infrastructure, a CPU spike is often an artifact of container rescheduling, and rising memory usage might just be the file cache doing its job. Resource metrics belong on dashboards to help diagnose a known problem, not as triggers to hunt for outages.
If you want to see how CPU schedulers can distort utilization numbers on their own, check out our deep dive into CPU limits and throttling mechanics.
An External Radar for Blind Spots: How Slack's Status Page Caught Hidden Outages
Even the sharpest work metrics leave blind spots—novel failure modes your team hasn't written checks for yet. At Slack, Brent Chapman and his colleagues found a remarkably simple independent indicator for those gaps: traffic to their public status.slack.com page.
When people's messages stop sending, they open the status page to see if the problem is on Slack's end. A sudden spike in pageviews means hundreds of users just hit an outage at the same moment. For Slack's engineers, that external signal acted as a reliable radar: even if every internal dashboard was green, it was time to hunt for a hidden failure right away. Pair that kind of external signal with fast escalation from customer support to on-call, and during every postmortem of a user-reported incident, ask which work metric your automation was missing.
In the end, healthy monitoring boils down to one rule: only direct user pain or an imminent failure within the next hour has the right to wake an engineer at night—everything about hardware health belongs in daytime tickets and diagnostic dashboards.



