Triage an Alert That Never Notified

Triage an Alert That Never Notified

Locate the handoff where an alert stopped on its way to a notification.

1. Confirm the alerting rule is loaded and check its last evaluation. A rule missing here was never loaded from disk, and a stale lastEvaluation means the rule group is not being evaluated on schedule.
2. Ask Prometheus which alerts its rule engine currently considers pending or firing. If the alert is absent here the expression is simply not matching and the problem is upstream of any notification path.
3. Confirm Prometheus has discovered at least one Alertmanager to dispatch to. An empty activeAlertmanagers list means firing alerts have nowhere to go, no matter how correct the rule is.
4. Ask Alertmanager whether the alert actually arrived, deliberately including silenced and inhibited alerts so a suppressed alert still shows up here rather than looking like it never landed.
5. List the silences whose matchers cover this alert. A silence in the active state explains a firing alert that arrives at Alertmanager and produces no notification.

What you'll need to have ready

• alertName: The alertname label of the alert under investigation (e.g. HighRequestLatency).
• alertNameMatcher: The same alert expressed as an Alertmanager label matcher (e.g. alertname="HighRequestLatency").

For your developer

Arazzo workflow prometheus:alert-notification-triage, 5 operations: getRules → $sourceDescriptions.alertsApi.getAlerts → getAlertmanagers → $sourceDescriptions.alertsApi.getAlerts → listSilences. Specs on APIs.io.