Provision Alerting End to End
A 5-step recipe with Grafana. We can help you get access.
Recipe · 5 steps
Provision Alerting End to End
with Grafana · Observability Management
Stand up a contact point, a routing policy, and an alert rule together.
- Create the contact point first. The notification policy in the next step routes to a receiver by name, and Grafana rejects a tree that references a receiver that does not exist yet.
- Read the contact point back by name to confirm it is registered before the policy tree is rewritten to depend on it.
- Replace the notification policy tree with one that routes alerts carrying the given label to the new receiver, falling through to the default receiver for everything else. This is a whole-tree PUT, so the payload must describe the routing y
- Create the alert rule last, now that a receiver exists and the tree routes to it. The rule's labels must match the route's object_matchers or the notification falls through to the default receiver.
- Read the provisioned rule back by UID to confirm the stored definition, including the labels the policy tree depends on.
What you'll need to have ready
- contactPointName: Name of the contact point (e.g. "platform-oncall-slack").
- contactPointType: Integration type for the receiver (e.g. slack, email, pagerduty, webhook, opsgenie).
- contactPointSettings: Type-specific settings for the integration (e.g. {"url": "..."} for slack, {"addresses": "..."} for email).
- disableResolveMessage: Suppress the follow-up notification when the alert resolves.
- defaultReceiver: Name of the receiver at the root of the policy tree, which catches anything the child route below does not match.
- routeLabel: Label name the child route matches on (e.g. "team").
- routeValue: Label value the child route matches on (e.g. "platform").
- groupBy: Labels to group notifications by before sending.
- groupWait: How long to wait to buffer the first notification for a group.
- groupInterval: How long to wait before sending a notification about new alerts in a group.
- repeatInterval: How long to wait before re-sending a notification that has already fired.
- ruleTitle: Title of the alert rule.
For your developer
Arazzo workflow grafana:provision-alerting, 5 operations: routePostContactpoints → routeGetContactpoints → routePutPolicyTree → createAlertRule → routeGetAlertRule. Specs on APIs.io.