On-call alert triage that scores how serious each alert is

The decision

A score question from 1 to 5, each level described in one sentence: 1 is "informational, no action needed", 3 is "degraded, fix during working hours", 5 is "customers are affected now, wake someone up". Add a noul question, "this alert repeats an incident that is already open", to fold repeats into one page.

How to build it

1. Point your alerting tool's webhook, from Alertmanager, Grafana or Datadog, at a small service.
2. Build the state from the alert's title and message, the service, the environment and how many times it fired in the last hour.
3. Try your levels from the terminal first: with an API key, systemone decide mapika/decider --request alert.json answers through the System One inference API.
4. Page for 4 and 5, post 2 and 3 to the team's channel, and only log 1. When two levels are close in probability, take the higher one.
5. After two weeks, compare the scores with what the person on call actually did with each alert.

Make it better

Use last quarter's alerts as labels: the ones that became incidents and the ones that were silenced. Fine-tune on them, and keep a short list of alerts that always page, whatever the score says.