Teams configure Slack, escalation, and status pages, then wait for fate to QA them. Fate is a bad QA engineer. Schedule a drill.
What to test
- Detection: a monitor goes red when it should (staging URL you control, or a temporary route).
- Notification: the primary actually receives it on the channel you think is loud enough.
- Ack: button works; escalation stops (what ack means).
- Escalation: primary ignores on purpose once; backup gets the page within the timeout.
- Status: someone posts investigating / resolved on the page or documents why not.
- Resolve / auto-resolve: recovery clears the incident cleanly.
How to fail safely
Prefer staging or a dedicated canary URL. If you must touch production, use a maintenance window and a tiny scope. Never random-kill a payment path on Friday without a plan.
When to retest
- New on-call hire's first week
- Channel migrations (Slack workspace moves)
- Escalation policy edits
- After a real incident where something in the path failed
Capture the awkwardness
Phones on Focus mode. Wrong Slack workspace. PagerDuty key rotated. Secondary schedule empty. These are gifts: you found them without customers present.
Put a "last alert drill" date on the team page. If it is months old, you are guessing.
Monitoring that has never paged in anger is unverified code. Drill until boredom; boredom means the path works.