Alert fatigue and how to avoid it

The fastest way to ruin an on-call rotation is too many alerts. Every noisy false positive trains the on-call engineer to ignore alerts. By the time a real incident fires, nobody reads the page. Alert fatigue is the silent killer of production reliability.

What makes good alerts work. Alert on symptoms users feel (slow requests, errors) not on causes (CPU usage). Make every alert actionable: if nobody can do anything about it, do not page anybody. And tune thresholds based on real incidents: if an alert fires repeatedly with no action needed, turn it off.