Test an Alertmanager Routing Path

Test an Alertmanager Routing Path

Post a synthetic alert and confirm which receiver it routes to.

1. Confirm the Alertmanager is up and its cluster has settled. Posting alerts at a cluster still in the settling state can produce duplicate or missing notifications that have nothing to do with the routing tree.
2. List the receivers defined in the loaded configuration. If the expected receiver is not in this list the routing tree cannot possibly select it, and the test would fail for a reason the alert itself never reveals.
3. Post the synthetic alert exactly as Prometheus would. The labels decide routing; the annotations are carried through to the notification template so the recipient can see this is a test.
4. Read the alert back to confirm Alertmanager accepted and is now tracking it. The receivers field names every receiver it will notify.
5. Read the alert groups to see the receiver the routing tree actually chose and the group labels it was grouped under. This is the answer the test exists to produce.

What you'll need to have ready

• alertName: The alertname for the synthetic alert (e.g. RoutingSmokeTest).
• alertNameMatcher: The same alert as a label matcher (e.g. alertname="RoutingSmokeTest").
• severity: The severity label to exercise a specific branch of the routing tree (e.g. critical).
• expectedReceiver: The receiver name this alert is expected to route to, used to check the routing tree did what was intended.
• startsAt: Alert start time as an RFC3339 timestamp.
• endsAt: Alert end time as an RFC3339 timestamp. Keep this close so the synthetic alert resolves itself shortly after the test.
• generatorURL: Optional URL linking back to whatever generated this alert, for the benefit of whoever receives the test page.

For your developer

Arazzo workflow prometheus:test-alert-routing, 5 operations: getStatus → listReceivers → postAlerts → getAlerts → getAlertGroups. Specs on APIs.io.