On-call dies in two ways: nobody gets paged when they should, or everybody gets paged when they should not. The second is more common, and it is almost always a sorting problem.
The one-sentence rule
Page if a human has to do something now to stop customer harm, data loss, or an active security issue.
Notify if someone should know soon, but sleep is more important than speed.
Ticket if it can wait for the next working day without making the problem worse.
If you cannot say which bucket an alert belongs in, it is not ready to ship.
Examples that usually deserve a page
- Login or checkout is down for everyone (confirmed, not a single-probe blip).
- Data is being corrupted or not writing.
- A security boundary just failed open.
- A dependency you cannot route around is dead and customers are blocked.
Examples that usually do not
- CPU is high but the product still works.
- A single region looks slow and others are fine — investigate, do not necessarily wake the company.
- Staging is red.
- A disk warning at 70% with days of headroom.
- "Anomaly detected" with no action attached.
Those can go to Slack, email, or a daytime queue. They should not climb an escalation policy.
Channels are not severities
Slack is a great place for awareness. It is a bad exclusive path for "wake a sleeping human." Push, phone bridges (via something like PagerDuty if you need voice), and devices that break through Focus modes exist because chat apps are designed to be polite.
Tallwatch ships email, Slack, Discord, Teams, Telegram, webhooks, and PagerDuty — and is honest that native SMS and voice are not in the box yet. Whatever tools you use, match the urgency to the channel, not the other way around.
A filter that keeps you honest
Before you enable a page, write the first three steps the on-call person will take. If you cannot, you do not have an actionable alert — you have a notification wearing a pager costume.
Then look at last month's pages. Anything that was acked and closed with "no action" should be demoted. Your future selves will sleep better.