Prometheus logo

Prometheus

DocsDownloadCommunitySupport & TrainingBlog

Ctrl + K

GitHub Logo

Querying basics

Prometheus provides a functional query language called PromQL (Prometheus Query Language) that lets the user select and aggregate time series data in real time.

When you send a query request to Prometheus, it can be an instant query, evaluated at one point in time, or a range query at equally-spaced steps between a start and an end time. PromQL works exactly the same in each case; the range query is just like an instant query run multiple times at different timestamps.

In the Prometheus UI, the "Table" tab is for instant queries and the "Graph" tab is for range queries.

Other programs can fetch the result of a PromQL expression via the HTTP API.

Examples

This document is a Prometheus basic language reference. For learning, it may be easier to start with a couple of examples.

Samples

The value of a sample at a given timestamp returned by PromQL may be a float or a native histogram. A float sample is a simple floating point number, whereas a native histograms sample contains a full histogram including count, sum, and buckets.

Note that the term “histogram sample” in the PromQL documentation always refers to a native histogram. The term "classic histogram" refers to a set of time series containing float samples with the _bucket, _count, and _sum suffixes that together describe a histogram. From the perspective of PromQL, these contain just float samples, there are no “classic histogram samples”.

Both float samples and histogram samples can have a counter or a gauge “flavor”. Float samples with a counter or gauge flavor are generally simply called “counters” or “gauges”, respectively, while their histogram counterparts are called “counter histograms” or “gauge histograms”. Float samples do not store their flavor, leaving it to the user to take their flavor into account when writing PromQL queries. (By convention, time series containing float counters have a name ending on _total to help with the distinction.)

Since histogram samples “know” their counter or gauge flavor, this allows reliable warnings about mismatched operations. For example, applying the rate function to gauge floats will most likely produce a nonsensical result, but the query will be processed without complains. However, if applied to gauge histograms, the result of the query will be annotated with a warning.

Expression language data types

In Prometheus's expression language, an expression or sub-expression can evaluate to one of four types:

  • Instant vector - a set of time series containing a single sample for each time series, all sharing the same timestamp
  • Range vector - a set of time series containing a range of data points over time for each time series
  • Scalar - a simple numeric floating point value
  • String - a simple string value; currently unused

Depending on the use case (e.g. when graphing vs. displaying the output of an expression), only some of these types are legal as the result of a user-specified expression. For instant queries, any of the above data types are allowed as the root of the expression. Range queries only support scalar-typed and instant-vector-typed expressions.

Both vectors and time series may contain a mix of float samples and histogram samples.

Reconciliation of histogram bucket layouts

Native histograms can have different bucket layouts, but they are generally convertible to compatible versions to apply binary and aggregation operations to them. Functions acting on range vectors that are applicable to native histograms also perform such reconciliation. In binary operations this reconciliation is performed pairwise, in aggregation operations and functions all histogram samples are reconciled to one compatible bucket layout.

Not all bucket layouts can be reconciled, if incompatible histograms are encountered in an operation, the corresponding output vector element is removed from the result, flagged with a warn-level annotation. More details can be found in the native histogram specification.

Literals

The following section describes literal values of various kinds. Note that there is no “histogram literal”.

String literals

String literals are designated by single quotes, double quotes or backticks.

PromQL follows the same escaping rules as Go . For string literals in single or double quotes, a backslash begins an escape sequence, which may be followed by a, b, f, n, r, t, v or \. Specific characters can be provided using octal (\nnn) or hexadecimal (\xnn, \unnnn and \Unnnnnnnn) notations.

Conversely, escape characters are not parsed in string literals designated by backticks. It is important to note that, unlike Go, Prometheus does not discard newlines inside backticks.

Example:

"this is a string"
'these are unescaped: \n \\ \t'
`these are not unescaped: \n ' " \t`

Float literals and time durations

Scalar float values can be written as literal integer or floating-point numbers in the format (whitespace only included for better readability):

[-+]?(
      [0-9]*\.?[0-9]+([eE][-+]?[0-9]+)?
    | 0[xX][0-9a-fA-F]+
    | [nN][aA][nN]
    | [iI][nN][fF]
)

Examples:

23
-2.43
3.4e-9
0x8f
-Inf
NaN

Additionally, underscores (_) can be used in between decimal or hexadecimal digits to improve readability.

Examples:

1_000_000
.123_456_789
0x_53_AB_F3_82

Float literals are also used to specify durations in seconds. For convenience, decimal integer numbers may be combined with the following time units:

  • ms – milliseconds
  • s – seconds – 1s equals 1000ms
  • m – minutes – 1m equals 60s (ignoring leap seconds)
  • h – hours – 1h equals 60m
  • d – days – 1d equals 24h (ignoring so-called daylight saving time)
  • w – weeks – 1w equals 7d
  • y – years – 1y equals 365d (ignoring leap days)

Suffixing a decimal integer number with one of the units above is a different representation of the equivalent number of seconds as a bare float literal.

Examples:

1s # Equivalent to 1.
2m # Equivalent to 120.
1ms # Equivalent to 0.001.
-2h # Equivalent to -7200.

The following examples do not work:

0xABm # No suffixing of hexadecimal numbers.
1.5h # Time units cannot be combined with a floating point.
+Infd # No suffixing of ±Inf or NaN.

Multiple units can be combined by concatenation of suffixed integers. Units must be ordered from the longest to the shortest. A given unit must only appear once per float literal.

Examples:

1h30m # Equivalent to 5400s and thus 5400.
12h34m56s # Equivalent to 45296s and thus 45296.
54s321ms # Equivalent to 54.321.

Duration expressions

Arithmetic expressions can be used wherever a time duration is expected, that is in range vector selectors and in offset durations.

In range vectors:

rate(http_requests_total[5m * 2])     # 10 minute range
rate(http_requests_total[(5+2) * 1m]) # 7 minute range

In offset durations:

http_requests_total offset (1h / 2)       # 30 minute offset
http_requests_total offset ((2 ^ 3) * 1m) # 8 minute offset

When using offset with a duration expression, you must wrap the expression in parentheses. Without parentheses, only the first duration value is used in the offset calculation.

The following operators are supported, following the usual precedence rules:

  • + – addition
  • - – subtraction
  • * – multiplication
  • / – division
  • % – modulo
  • ^ – exponentiation

The following functions can be used inside duration expressions:

  • step() resolves to the step width of a range query, and to 0s for an instant query.
  • range() resolves to the full range of a range query (end time − start time), and to 0s for an instant query. This is particularly useful in combination with @ end() to look back over the entire query range, e.g. max_over_time(metric[range()] @ end()).
  • min_of(<duration>, <duration>) returns the smaller of the two durations, which is useful for capping a duration at a maximum value.
  • max_of(<duration>, <duration>) returns the larger of the two durations, which is useful for enforcing a minimum value.

For example, max_of(step(), 5s) ensures the duration is never shorter than 5s, while min_of(range(), 1h) caps the duration at 1h.

Note: Duration expressions are not supported in the @ modifier.

Examples of equivalent durations:

  • 5m * 2 is equivalent to 10m or 600s.
  • 10m - 1m is equivalent to 9m or 540s.
  • (5+2) * 1m is equivalent to 7m or 420s.
  • 1h / 2 is equivalent to 30m or 1800s.
  • 4h % 3h is equivalent to 1h or 3600s.
  • (2 ^ 3) * 1m is equivalent to 8m or 480s.
  • step() + 1 is equivalent to the query step width increased by 1s.
  • max_of(step(), 5s) is equivalent to the larger of the query step width and 5s.
  • min_of(2 * step() + 5s, 5m) is equivalent to the smaller of twice the query step increased by 5s and 5m.

Time series selectors

These are the basic building-blocks that instruct PromQL what data to fetch.

Instant vector selectors

Instant vector selectors allow the selection of a set of time series and a single sample value for each at a given timestamp (point in time). In the simplest form, only a metric name is specified, which results in an instant vector containing elements for all time series that have this metric name.

The value returned will be that of the most recent sample at or before the query's evaluation timestamp (in the case of an instant query) or the current step within the query (in the case of a range query). The @ modifier allows overriding the timestamp relative to which the selection takes place. Time series are only returned if their most recent sample is less than the lookback period ago.

This example selects all time series that have the http_requests_total metric name, returning the most recent sample for each:

http_requests_total

It is possible to filter these time series further by appending a comma-separated list of label matchers in curly braces ({}).

This example selects only those time series with the http_requests_total metric name that also have the job label set to prometheus and their group label set to canary:

http_requests_total{job="prometheus",group="canary"}

It is also possible to negatively match a label value, or to match label values against regular expressions. The following label matching operators exist:

  • =: Select labels that are exactly equal to the provided string.
  • !=: Select labels that are not equal to the provided string.
  • =~: Select labels that regex-match the provided string.
  • !~: Select labels that do not regex-match the provided string.

Regex matches are fully anchored. A match of env=~"foo" is treated as env=~"^foo$".

For example, this selects all http_requests_total time series for staging, testing, and development environments and HTTP methods other than GET.

http_requests_total{environment=~"staging|testing|development",method!="GET"}

Label matchers that match empty label values also select all time series that do not have the specific label set at all. It is possible to have multiple matchers for the same label name.

For example, given the dataset:

http_requests_total
http_requests_total{replica="rep-a"}
http_requests_total{replica="rep-b"}
http_requests_total{environment="development"}

The query http_requests_total{environment=""} would match and return:

http_requests_total
http_requests_total{replica="rep-a"}
http_requests_total{replica="rep-b"}

and would exclude:

http_requests_total{environment="development"}

Multiple matchers can be used for the same label name; they all must pass for a result to be returned.

The query:

http_requests_total{replica!="rep-a",replica=~"rep.*"}

Would then match:

http_requests_total{replica="rep-b"}

Vector selectors must either specify a name or at least one label matcher that does not match the empty string. The following expression is illegal: