Tallwatch
Back to blog
7 min read
AlertingGuide

Should you delay downtime alerts? Blips, flapping, and the two-minute rule

Instant alerts sound safer, but most short outages fix themselves before anyone can act. How to choose an alert delay, handle flapping monitors, and stop down/up email pairs.

NK

Nabin Khair

Founder

Should you delay downtime alerts? Blips, flapping, and the two-minute rule

Every uptime tool asks the same question somewhere in its settings: how soon should we tell you? The instinct is to say "immediately." Faster feels safer. If the site is down, you want to know.

But look at the alert history of almost any production site and a pattern shows up. A large share of incidents last a minute or two and end on their own. A deploy rolls pods. A load balancer drains a node. A database failover takes forty seconds. By the time someone reads the alert, opens a laptop, and logs in, the site is already back. The only thing the alert achieved was a down email, a recovered email, and a little less trust in the next one.

This post is about the time dimension of alerting: when to wait, how long, and what to do with monitors that will not sit still.

Detection and notification are different decisions

It helps to separate two questions that tools often blur together:

  1. Is the site really down? That is detection. It is answered with evidence: consecutive failures, checks from several regions, and assertions on the response.
  2. Is it worth interrupting a person? That is notification. It depends on how long the problem lasts and what it costs.

A good detection setup tells you an outage is real. It does not tell you the outage will last long enough to matter. A 45-second failover is a real outage. It is also one nobody can fix, because it is over before they could start.

So the alert delay is not a way to paper over flaky detection. If your monitor pages on network noise, fix that first. The delay is for real, confirmed, short outages that do not need a human.

Why short outages are not worth a page

An alert is a request for someone's attention. It is worth sending when that person can do something useful with it. For a short blip, they usually cannot:

  • It recovers before they act. Acknowledging, opening a laptop, and finding the right dashboard takes a few minutes even for a fast responder.
  • It trains people to ignore alerts. Every down/up pair that resolves itself teaches the team that alerts can wait. Then the real one waits too. We wrote about that effect in why we stopped paging on warnings.
  • It costs sleep. One pointless 3am page can wreck a night, and on-call fatigue is what makes people quit rotations.

None of this means short outages do not matter. They still count toward your uptime, they still show up in your history, and a pattern of them is worth investigating during working hours. They just do not need to wake anyone.

How to choose a delay

There is no universal number, but a few rules of thumb hold up:

  • Zero delay for things where every second has a direct cost and someone can act quickly: payment processing, a login service that blocks every user, anything with a tight SLA.
  • One to three minutes for most websites and public APIs. This removes deploy and failover blips without meaningfully slowing down response to real outages.
  • Five to ten minutes for internal tools, staging, admin panels, and anything that can wait until morning anyway.
  • Zero delay for heartbeat and cron monitors, since a missed heartbeat is already a sign that time has passed. More on those in how to monitor cron jobs and background workers.

A useful way to decide: look at your last month of incidents and how long each lasted. If most real outages that needed a person ran longer than five minutes, and most blips were shorter than two, a two-minute delay separates them cleanly.

The recovery email problem

Here is a small thing that causes a lot of noise. Suppose you set a two-minute delay. The site goes down, comes back after 60 seconds, and the down alert is never sent. Good. But many setups still send the "recovered" message, because recovery is treated as its own event.

The result is a stream of "Your site is back up" emails about outages nobody was told about. They are confusing at best, and at worst people start wondering what else they are missing.

The rule is simple: if nobody was told it went down, nobody needs to be told it came back. A recovery alert should only follow a down alert that was actually delivered.

Flapping: when a monitor will not settle

Sometimes a monitor goes down, recovers, goes down again, and recovers again, over and over. That is flapping, and it needs different handling from a single outage.

Flapping is usually a symptom of something real:

  • A server at the edge of its capacity, failing some requests and passing others.
  • A timeout set too close to normal response times, so ordinary variation tips it over. If your slowest normal responses take eight seconds, a ten-second timeout will flap. See website slow vs website down.
  • A health check that depends on something flaky, like a third-party API that has nothing to do with whether your app is up. The health check endpoint guide covers what to leave out.
  • One unhealthy instance behind a load balancer, so some requests hit a bad node.

The worst way to handle flapping is to open and close a new incident every time. That produces a pile of incidents, a pile of alerts, and no clear picture. Better behavior:

  1. Reopen the same incident if the site goes down again shortly after recovering, instead of starting a new one. One problem, one incident, one timeline.
  2. Wait longer before declaring recovery while a monitor is flapping, so a brief good minute does not close the incident.
  3. Tell people it is flapping, once, rather than paging on every transition.

Then fix the cause in daylight. Flapping is a useful signal; it just should not be delivered as twenty separate pages.

Keep it off the status page too

The same logic applies to your public status page. A 30-second blip that shows up as a red bar makes customers think you had an outage they never noticed. Showing only outages that lasted long enough to affect people keeps the page honest without making it alarming. We wrote about the balance in your status page is only as honest as your monitoring.

How Tallwatch handles it

In Tallwatch, an incident opens only when several regions agree a site is down, so the alert delay is applied to outages that are already confirmed. Each monitor has a Notify setting, from instantly up to an hour. New monitors default to two minutes (heartbeat monitors default to instantly). If the site recovers during the wait, nobody gets an alert, and because the down alert was never sent, the recovery alert is skipped too. If a site goes down again soon after recovering, Tallwatch reopens the same incident and waits longer before calling it resolved. Short outages are also kept off your public status page by default, while your dashboard still records everything.

A starting checklist

  • Fix noisy detection first: require agreement from more than one place.
  • Set a one to three minute delay on most monitors, zero on the few that truly need it.
  • Make sure recovery alerts only follow down alerts that were delivered.
  • Treat flapping as one incident with one notification, then fix the cause.
  • Review short outages weekly, during working hours.

The goal is not fewer alerts for its own sake. It is alerts that arrive when a person can actually help, so that every page gets answered.

Related

Keep reading

False alerts and status pages.

What changed after we stopped paging on warnings

What changed after we stopped paging on warnings

Warning-level pages feel responsible until nobody sleeps. Moving warnings to daytime channels restored trust in the alerts that still wake people.

NKNabin Khair
How to cut false downtime alerts without waiting forever to get paged

How to cut false downtime alerts without waiting forever to get paged

You should not have to choose between a quiet pager and a slow one. Confirmation — across time or across regions — is how you keep both.

NKNabin Khair
Webhook alerts that silently fail: how to know your pager is broken

Webhook alerts that silently fail: how to know your pager is broken

The scariest failure mode is not a loud false page. It is an outage with zero notifications because the webhook path died quietly.

NKNabin Khair