You wired incidents to an internal webhook that posts to a homemade router, then into Slack. It worked in February. In August the router moved hosts, TLS expired, or a token rotated. Production died on a Sunday. Nobody's phone moved. The monitor was red in a UI nobody opened.
Why webhooks fail quietly
- Receiver returns 500 and the sender gives up
- Auth token expired
- URL typo after a "quick refactor"
- Receiver up, but the downstream chat integration is not
- Success response that drops the message on the floor
Uptime of your product ≠ uptime of your notification pipe.
Defenses that work
Redundancy: SEV1 goes to email or Slack and webhook, or webhook into PagerDuty that has its own delivery guarantees. One sink is a single point of failure.
Delivery visibility: your alerting tool should retry with backoff and surface failed deliveries. If it cannot, add an external check on the receiver's health URL.
Periodic drills: test alerting includes forcing a page through the webhook path, not only through email.
Heartbeats into the sink: a scheduled "testing notifier" message nobody acts on. If it disappears for a day, investigate.
Signed webhooks with programmable payloads (plus chat and email) cover the automation cases. Use webhooks to automate, not as the only wire into a sleeping human. If you build a custom router, give it a /health, monitor that too, and keep a boring email backup for critical policies.
The outage you do not hear about is the one that teaches customers you are unreliable. Audit the pager path with the same seriousness as the app path.