January 10, 2025 • 22 min read
Most incident tooling is sold on dashboards. The dashboards are the least useful part.
The job of an incident stack is to get one person from "a graph went red" to "I've fixed it". That's it. I've stitched these together at The Good Guys and now at 4DMedical, and most of what I learned was what to rip out. Half the tools above I've bought and later cancelled because nobody opened them during an actual incident. So this is less a recommendation than a list of what I'd keep at each size.
Catch it from your own monitoring before the support queue does
Wake the right person. Not the whole team.
One channel, a runbook, and a number to call when you're stuck
A blameless post-mortem, and one thing you actually change
$200-500
Sentry posting to a Slack channel, and one person who reads it. Don't buy an on-call tool yet. You don't have enough alerts to need a rotation.
$2,000-8,000
Buy OpsGenie the first time an alert fires at 2am into a channel nobody's watching. The APM and session replay are for the bugs you can't reproduce from the ticket. DataDog will be the line item that surprises you on the invoice.
$15,000-50,000+
At this size one outage sets off forty alerts across five systems. Most of this spend is grouping them back into one incident so the on-call engineer isn't reading forty pages to find the one that matters.
Start with less than you think you need. Add a tool the week an actual incident makes you wish you'd had it, not before. The stack that works is the one a tired engineer can use at 3am without reading a wiki, and you don't find out which one that is until something breaks.