Configure custom alerts

You can use Grafana Alerting to view legacy Synthetic Monitoring alert rules, create custom alert rules, and set up alert notifications.

Create custom alert rules

You can create PromQL queries for check metrics to create custom Grafana alerts.

Example query: check if latency exceeds 1 second.

sum by (instance, job) (rate(probe_all_duration_seconds_sum{}[10m]))
/
sum by (instance, job) (rate(probe_all_duration_seconds_count{}[10m])) > 1

Example query: check if the probe error rate is higher than 10%.

1 - (
sum by (instance, job) (rate(probe_all_success_sum{}[10m]))
/
sum by (instance, job) (rate(probe_all_success_count{}[10m]))
) > 0.1

Example query: check if latency exceeds 1 second, only for checks with the custom label environment="prod".

sum by (instance, job, environment) (rate(probe_all_duration_seconds_sum{environment="prod"}[10m]))
/
sum by (instance, job, environment) (rate(probe_all_duration_seconds_count{environment="prod"}[10m])) > 1

Include the custom label in the by () clause so that the resulting alert instances carry it, which makes the label available for alert routing.