PromQL in Cloud Monitoring

This document describes using Prometheus Query Language (PromQL) in Cloud Monitoring. PromQL provides an alternative to the Metrics Explorer menu-driven and Monitoring Query Language (MQL) interfaces for creating charts and dashboards.

You can use PromQL to query and chart Cloud Monitoring data from the following sources:

You can also use tools like Grafana to chart metric data ingested into Cloud Monitoring. Available metrics include metrics from Managed Service for Prometheus and Cloud Monitoring metrics documented in the lists of metrics. For information about setting up Grafana and other tools based on the Prometheus API, see the Managed Service for Prometheus documentation about Grafana.

You can also import your Grafana dashboards into Cloud Monitoring.

Naming conventions

The Prometheus metric model and naming conventions differ from those used by Cloud Monitoring. To use PromQL to query Monitoring metric data, you must convert Cloud Monitoring names into PromQL-compatible equivalents. For information about the naming conventions, see Mapping Cloud Monitoring metrics to PromQL.

Accessing PromQL in Cloud Monitoring

You can use PromQL from the Code tab on the following pages in the Google Cloud console:

  • Metrics Explorer
  • Add Chart when creating custom dashboards

For information about accessing and using the editor, see Using the PromQL editor.

Prometheus rules and alerts

You can use PromQL to create recording and alerting rules on any metric in Cloud Monitoring by using Prometheus-style alerting in Cloud Monitoring. For more information, see Managed rule evaluation and alerting or Self-deployed rule evaluation and alerting.

Learning PromQL

To learn the basics of using PromQL, we recommend consulting open source documentation. The following resources can help you get started:

PromQL differences

PromQL for Cloud Monitoring might function slightly differently than upstream PromQL.

PromQL queries in Cloud Monitoring are partially evaluated at the Monarch backend by using an internal query language, and there are some known differences in query results. Other than the differences listed in this section, the PromQL in Cloud Monitoring is at parity with the PromQL available in Prometheus version 2.44.

PromQL functions added after Prometheus version 2.44 might not be supported.

Matching on metric names

Only exact matching on metric names is supported. You must include an exact match on the metric name in your query.

We recommend the following workarounds for common scenarios that use a regular expression matcher on the __name__ label:

  • Prometheus adapter configurations often use the =~ operator to match on multiple metric names. To fix this usage, expand the config to use a separate policy for each metric and name each metric explicitly. This also prevents you from accidentally autoscaling on unexpected metrics.
  • Regular expressions are often used to graph multiple non-dimensional metrics on the same chart. For example, if you have a metric like cpu_servicename_usage, you might use a wildcard to graph all your services together. Using non-dimensional metrics like this is an explicitly bad practice in Cloud Monitoring, and this practice leads to extremely poor query performance. To fix this usage, move all dimensionality into metric labels instead of embedding dimensions in the metric name.
  • Querying over multiple metrics is often used to see what metrics are available to query. We recommend you instead use the /labels/__name__/values call to discover metrics. You can also discover metrics using the Cloud Monitoring UI.
  • Matching multiple metrics is useful for seeing how many samples were scraped, ingested, and charged on a per-metric basis. Cloud Monitoring provides this information to you on the Metrics Management page. You can also access this information as metric data by using the Samples Ingested metric or the Samples Written by Attribution ID metric.

Staleness

Staleness is not supported in the Monarch backend.

Calculation of irate

When the lookback window for the irate function is less than the step size, we increase the window to the step size. Monarch requires this change to ensure that none of the input data is completely ignored in the output. This difference applies to rate calculations as well.

Calculation of rate and increase

When the lookback window for the rate function is less than the step size, we increase the window to the step size. Monarch requires this change to ensure that none of the input data is completely ignored in the output. This difference applies to irate calculations as well.

There are differences in the interpolation and extrapolation calculations. Monarch uses a different interpolation algorithm than Prometheus, and this difference can lead to slightly different results. For example, Monarch counter samples are stored with a time range rather than the single timestamp that Prometheus uses. Therefore, counter samples in Monarch can be included in a rate calculation even though the Prometheus timestamp would exclude them. This generally results in more accurate rate results, especially when querying over the beginning or end of the underlying time series.

Calculation of histogram_quantile

A PromQL histogram_quantile calculation on a histogram with no samples produces a NaN value. The internal query language's calculation produces no value; the point at the timestamp is dropped instead.

The rate-calculation differences can also affect the input to histogram_quantile queries.

Type-specific functions on differently typed metrics

Although upstream Prometheus is weakly typed, Monarch is strongly typed. This means that running functions specific to a single type on a differently typed metric (for example, running rate() on a GAUGE metric or histogram_quantile() on a COUNTER or untyped metric) doesn't work in Cloud Monitoring, even though these functions work in upstream Prometheus.