Loki alerting
The Loki data source supports Grafana Alerting. You can alert on your logs so Grafana notifies you when a LogQL metric query crosses a threshold, for example when the rate of error logs spikes.
Loki works with two kinds of alert rules:
- Grafana-managed alert rules: Grafana evaluates the rule against a LogQL metric query. This is the most common approach and works with any Loki data source.
- Data source-managed alert rules: The Loki ruler stores and evaluates the rule. Use these when you want alert rules to live alongside your Loki deployment. Enable Manage alert rules in Alerting UI on the data source to create and edit them from Grafana.
Before you begin
Before you create alert rules with Loki data, ensure you have:
- A configured Loki data source.
- Permissions to create alert rules in Grafana.
- Familiarity with Grafana Alerting concepts.
- For data source-managed rules, a Loki deployment with the ruler enabled, and the Manage alert rules in Alerting UI setting turned on in the data source configuration. To route the resulting alerts, add a separate Alertmanager data source.
Supported queries for alerting
Alert rules require queries that return numeric data that Grafana can evaluate against a threshold. Wrap a Loki log query in a LogQL metric function, such as rate or count_over_time, to produce numeric time series.
Note
Use a metric query for alerting, not a plain log query. A log query returns log lines, which Grafana can’t evaluate against a threshold.
The following queries are common starting points for alerting. Replace the label matchers with values from your own logs.
Rate of error lines per second:
sum(rate({app="my-app"} |= `error` [5m]))Error rate broken down by status code, to alert per endpoint or service:
sum by (status) (rate({app="nginx"} | logfmt | status=~`5..` [5m]))Count of slow requests, for a latency alert:
sum(count_over_time({app="my-app"} | logfmt | duration > 1s [5m]))Detect a service that stopped logging by alerting when the count drops to zero:
sum(count_over_time({app="my-app"} [10m]))Create a Grafana-managed alert rule
To create a Grafana-managed alert rule using Loki data:
- Go to Alerting > Alert rules.
- Click New alert rule.
- Enter a name for your alert rule.
- In the Define query and alert condition section:
- Select your Loki data source.
- Enter a LogQL metric query.
- Add a Reduce expression if your query returns multiple series.
- Add a Threshold expression to define the alert condition.
- Configure the Set evaluation behavior section:
- Select or create a folder and evaluation group.
- Set the evaluation interval and pending period.
- Add labels and annotations to provide context for notifications.
- Click Save rule.
For detailed instructions, refer to Create a Grafana-managed alert rule.
Example: alert on error log rate
This example fires when the rate of log lines that contain error exceeds a threshold:
Create a new alert rule.
Configure the query, replacing the label matcher with your own:
sum(rate({app="my-app"} |= `error` [5m]))Add expressions:
- Reduce: Last, to get the most recent value.
- Threshold: Is above
10.
Set evaluation to run every 1 minute with a 5-minute pending period.
Save the rule.
Manage data source-managed alert rules
When you turn on Manage alert rules in Alerting UI for the Loki data source, you can create and edit alert and recording rules that the Loki ruler stores and evaluates. These rules appear in the Grafana Alerting UI alongside your Grafana-managed rules, grouped under the Loki data source.
The Loki data source configuration doesn’t include Alertmanager routing. To route the resulting alerts, add a separate Alertmanager data source. For more information, refer to Configure the Loki data source and Data source-managed alert rules.
Best practices
Follow these recommendations to create reliable alerts with Loki data.
Test queries before alerting
Verify your query returns numeric data before you create an alert:
- Go to Explore.
- Select your Loki data source.
- Run the LogQL metric query you plan to use for alerting.
- Confirm the result is numeric and suitable for threshold evaluation.
Reduce a time series to a single value
A Grafana-managed rule evaluates a single number against the threshold. A range query returns a time series with many points, which the rule can’t evaluate directly. Either set the query Type to instant, or add a Reduce expression, such as Last or Mean, between the query and the Threshold expression. Without one of these, the rule can evaluate incorrectly.