Documentation Grafana Enterprise Logs Manage Rate limits

Enforce rate limits and push request validation

Loki will reject requests if they exceed a usage threshold (rate limit error) or if they are invalid (validation error).

All occurrences of these errors can be observed using the loki_discarded_samples_total and loki_discarded_bytes_total metrics. The sections below describe the various possible reasons specified in the reason label of these metrics.

It is recommended that Loki operators set up alerts or dashboards with these metrics to detect when rate limits or validation errors occur.

Terminology

• sample: a log line with structured metadata
• stream: samples with a unique combination of labels
• active stream: streams that are present in the ingesters - these have recently received log lines within the chunk_idle_period period (default: 30m)

Rate-Limit Errors

Rate-limits are enforced when Loki cannot handle more requests from a tenant.

Rate-limit errors reject the whole request before Loki stores any of it. The tables below list Sample discarded: No for these errors because Loki never accepts the samples in the first place, but the request still increments loki_discarded_samples_total and loki_discarded_bytes_total. Because these errors are retryable, a client can resend the same data after the rate limit condition clears.

rate_limited

This rate limit is enforced when a tenant has exceeded their configured log ingestion rate limit.

One solution if you’re seeing samples dropped due to rate_limited is simply to increase the rate limits on your Loki cluster. These limits can be modified globally in the limits_config block, or on a per-tenant basis in the runtime overrides file. The config options to use are ingestion_rate_mb and ingestion_burst_size_mb.

Note that you’ll want to make sure your Loki cluster has sufficient resources provisioned to be able to accommodate these higher limits. Otherwise your cluster may experience performance degradation as it tries to handle this higher volume of log lines to ingest.

Another option to address samples being dropped due to rate_limits is simply to decrease the rate of log lines being sent to your Loki cluster. Consider collecting logs from fewer targets or setting up drop stages in Alloy to filter out certain log lines. You can also use Alloy’s rate limiting to control the volume of logs sent to your Loki cluster.

per_stream_rate_limit

This limit is enforced when a single stream reaches its rate limit.

Each stream has a rate limit applied to it to prevent individual streams from overwhelming the set of ingesters it is distributed to (the size of that set is equal to the replication_factor value).

This value can be modified globally in the limits_config block, or on a per-tenant basis in the runtime overrides file. The config options to adjust are per_stream_rate_limit and per_stream_rate_limit_burst. The default values are 3MB for per_stream_rate_limit and 15MB for per_stream_rate_limit_burst.

Another option you could consider to decrease the rate of samples dropped due to per_stream_rate_limit is to split the stream that is getting rate limited into several smaller streams. A third option is to use the Alloy stage.limit block to limit the rate of samples sent to the stream hitting the per_stream_rate_limit.

We typically recommend setting per_stream_rate_limit no higher than 5MB, and per_stream_rate_limit_burst no higher than 20MB.

stream_limit

This limit is enforced when a tenant reaches their maximum number of active streams.

Active streams are held in memory buffers in the ingesters, and if this value becomes sufficiently large then it will cause the ingesters to run out of memory.

This value can be modified globally in the limits_config block, or on a per-tenant basis in the runtime overrides file. To increase the allowable active streams, adjust max_global_streams_per_user. Alternatively, the number of active streams can be reduced by removing extraneous labels or removing excessive unique label values.

Validation Errors

Validation errors occur when a request violates a validation rule defined by Loki.

line_too_long

This error occurs when a log line exceeds the maximum allowable length in bytes. The HTTP response will include the stream to which the offending log line belongs as well as its size in bytes.

This value can be modified globally in the limits_config block, or on a per-tenant basis in the runtime overrides file. To increase the maximum line size, adjust max_line_size. We recommend that you do not increase this value above 256kb for performance reasons. Alternatively, Loki can be configured to ingest truncated versions of log lines over the length limit by using the max_line_size_truncate option. When truncation is enabled, you can use max_line_size_truncate_identifier to append a marker string to each truncated log line, so that truncated lines are easy to identify later.

invalid_labels

This error occurs when one or more labels in the submitted streams fail validation.

Loki uses the same validation rules as Prometheus for validating labels.

Label names may contain ASCII letters, numbers, as well as underscores. They must match the regex [a-zA-Z_][a-zA-Z0-9_]*. Label names beginning with __ are reserved for internal use.