Back to blog
7 min read
AlertingOpinion

AI won't fix alert fatigue. A quorum will.

The 2026 pitch is that AI ends alert fatigue. Most fatigue isn't a thresholding problem a model must learn away. It's one flaky probe paging you.

NK

Nabin Khair

Founder

The hottest story in monitoring this year is that AI is going to end alert fatigue. Vendors are quoting 90 to 95 percent reductions in alert noise, and the pitch writes itself: enterprise teams field 500 to 1,200 alerts a day with only a fraction of them actionable, so let a model learn what "normal" looks like and surface anomalies instead of firing on fixed thresholds. I believe those numbers describe a real problem. I do not believe AI is the cheapest or most trustworthy way to fix the largest part of it.

Here is the uncomfortable thing the framing skips. A lot of alert fatigue is not a thresholding problem at all. It is a single-probe problem. One flaky vantage point fails to reach your site for thirty seconds, pages you, and your site was never down. No model needs to learn that away. You just need to stop trusting one opinion as if it were the whole truth.

Two very different sources of noise

When people say "alert fatigue," they are usually pointing at two different things that have gotten blurred together.

The first is genuinely a tuning problem. You set a CPU alert at 80 percent, traffic patterns shift, and now it fires every afternoon during a load spike that resolves itself. The threshold was static and the world moved. This is the noise AI observability is actually good at. A model that learns the daily and weekly shape of a metric, and only flags departures from it, can quiet a whole category of "it's high again but it's always high then" pages. That work is real and I am not here to wave it away.

The second source is different in kind. It is the alert that was simply wrong, because the thing that fired it could not tell its own bad moment apart from yours. A single checking location fails to reach your endpoint, and it has no way to know whether your service died or whether the network between that one machine and your origin had a rough thirty seconds. It pages anyway. That is not a threshold that needs learning. That is a decision made on one data point that should never have been made on one data point.

No amount of "learning normal" fixes the second category cleanly, because there was nothing abnormal to learn. The probe did exactly what a healthy probe does when the path in front of it breaks. The defect is in trusting a single probe to call an outage, and you fix that with more evidence, not a smarter guess from the same thin evidence.

A quorum is the boring fix that works

The fix that kills the loudest source of false pages is embarrassingly simple, and that is the whole argument. Check each target from several regions, count each result as a vote, and open an incident only when at least two checked regions agree the target failed in the same round. One region seeing a failure is a rumor. Two or more regions seeing it together, in the same window, is a fact. Tallwatch pages on the fact and stays quiet on the rumor.

There is one trap in naive voting, and it is the interesting one. Sometimes a whole region degrades and starts failing checks across thousands of unrelated sites at once. Count those votes and one cloud region having a bad hour pages half the internet for outages that were never theirs. So a region that is busy failing across many unrelated targets has its vote set aside until it recovers, and the call is left to the regions that are demonstrably healthy. That is the entire mechanism. It runs on every plan, including free, because deciding from many places is where monitoring should start.

Now compare the two approaches on the things that actually matter when you are deciding whether to bet your pager on something.

There is no training window. A quorum is right on its first check. An anomaly model is worth what its baseline is worth, and the baseline takes weeks of clean data to earn, during which it is learning from your noise too.

There is nothing to mistune. A model has a sensitivity dial, and that dial trades false pages against missed outages. Set it tight and you are back to being woken for nothing. Set it loose and it sleeps through the real thing. A quorum has no such dial. Two regions agree or they do not.

And there is nothing to explain after the fact. When a quorum pages you, the reason fits in a sentence: three regions checked, two failed in the same round, here are the response codes. When an anomaly model pages you, the honest answer to "why did this fire" is sometimes "the score crossed a line the model drew from history you cannot easily inspect." That gap matters at 3 a.m. and it matters more in the postmortem.

Where the AI tools genuinely earn their place

I want to be fair, because the contrarian take is only worth reading if it concedes where the other side wins. AI observability is doing real work, and it is not the work a quorum does.

Correlation across telemetry is genuinely hard and genuinely valuable. When forty alerts fire in ninety seconds and they all trace back to one bad deploy, a model that collapses that storm into a single likely cause is saving someone an ugly hour. Surfacing the metric that moved first, pointing at the change that lines up with the break, ranking which of a dozen symptoms is the actual disease, this is where the modern tools shine, and it is a different question from the one I care about here.

That different question is the narrow one: should this wake a human. For that question alone, a quorum is cheaper to run, faster to trust, and explainable to anyone who asks. You do not need a model to tell you a site is down when you can have several independent regions agree on it. The AI is for what comes after you already know something broke, not for the gate that decides whether something broke at all.

This is a deliberate scope choice

Tallwatch is not an AI product, and that is not an admission, it is the position. The decision to page is too important to hand to a black box, so we made it a simple, explainable quorum and stopped there. We do not do AI anomaly detection, we do not do AI root-cause analysis, and we do not do auto-remediation. If you want a model correlating your telemetry and proposing fixes, buy one of the tools that does that well, and buy it on top of a signal you can trust. That clean, trustworthy signal is the thing those tools quietly depend on, and it is the thing I think the "AI ends alert fatigue" story keeps skipping over.

The pager is only useful if you believe it. Belief is built one true alert at a time and demolished one false one at a time, and the fastest way to stop demolishing it is the least glamorous one on offer: make the regions agree before you wake anyone.

Start free.

Keep reading