Teams that get spammed by downtime alerts make a predictable mistake: they loosen the monitor until it barely pages, then discover outages from Twitter. Teams that tighten until every blip pages make the other mistake: nobody believes the pager. You want a third path.
Where false downs come from
- One probe, one network path, one bad second.
- Timeouts set tighter than your app's p99.
- Deploys without maintenance windows.
- Health checks that fail when a non-critical dependency hiccups.
- Alerting on the first failure with no confirmation.
If your tool pages on a single failed check from a single city, you have volunteered for false positives.
Add evidence, not vibes
Across time: require two or three consecutive failures before opening an incident. Cheap. Adds seconds to minutes of delay. Good filter for one-off blips.
Across space: require more than one region to see the same failure. Filters path weather. This is the core of multi-region consensus.
Both when you can. Consecutive failures from one region still share a fate. Regions that agree in the same round catch real outages without waiting for a long retry ladder from one place.
What not to do
- Mute the channel permanently.
- Raise intervals to fifteen minutes to "reduce noise" (you reduced detection instead).
- Delete monitors you still need because they were loud — fix their rules.
- Add AI that guesses which alerts matter while you still page on single probes. More on that argument here.
A practical starting point
- Critical URLs: one-minute checks.
- Incident opens only with confirmation (regions and/or consecutive fails).
- Log false pages for two weeks (first fortnight habit).
- Demote or fix anything that cried wolf.
You will still get real pages. They should feel rare enough that people answer them. That feeling is the whole point of monitoring.