Query and view PA metrics

After collecting metrics from workloads deployed in your Google Distributed Cloud (GDC) air-gapped project, you can query and view metrics on dashboards from the project's system monitoring instance or query metrics from the Distributed Cloud HTTP API.

Query and view metrics on dashboards

The system monitoring instance of the platform-obs project includes organization-level metrics to monitor organizational components for data observability, such as CPU utilization, storage consumption, network monitoring, server monitoring, logs, alerts, and metrics.

To view metrics, use dashboards on the Explore page of the system monitoring instance. You can also make queries from the Explore page.

Before you begin

Before querying and viewing metrics on dashboards, you must obtain access to the system monitoring instance. For more information, see Get access to dashboards.

Build queries for your metrics

Monitor and view metrics for data observability purposes on your organization through the system monitoring instance of your project. The integrated user interface (UI) of the instance contains default dashboards from standard components and customized dashboards you create for your specific use case needs. Assuming your environment is up and running, you immediately see a few metrics dashboards in the Available Dashboards page.

Query metrics from the UI to visually retrieve metrics from your project and get an integrated view to monitor your organization. To filter results, search for metrics by labels using query language expressions.

System monitoring instance endpoint

The following URL is the endpoint of the monitoring instance of the platform-obs project:

https://GDC_URL/platform-obs/grafana

Replace GDC_URL with the URL of your organization in GDC.

Query metrics

Complete the following steps to retrieve metrics:

In the GDC console, select a project.
On the navigation menu, click Operations > Monitoring.
Click View all in Grafana.

A new page opens the URL of the monitoring instance of your project.
On the UI of the monitoring instance, click explore Explore from the navigation menu to open the Explore page.
From the drop-down menu at the Explore bar, select prometheus to retrieve metrics.
Enter a query to search for metrics using PromQL (Prometheus Query Language) expressions. You can do this step in either of the following ways:
- Select a metric and a label for your query from the Metric and Label filters drop-down menus. Click add Add to add more labels in your query. Then, click Run query.
- Enter your query directly in the Metrics text field and press Shift+Enter to run the query.
The page displays the metrics matching your query.

The prometheus option is selected on the Explore page of the UI to obtain metrics.

Figure 1. Menu option to query metrics from the UI.

In figure 1, the prometheus option displays the interface that lets you build queries from the monitoring instance to retrieve metrics.

For value examples of labels that you can use to query metrics, see Sample queries and labels.

Sample queries and labels

You can query collected metrics using the metric name and key-value labels. For this reason, a PromQL query has the following syntax:

metric_name{label_one="value", label_two="value"}

Labels let you differentiate the characteristics of a metric. In this way, container authors make their workloads generate metrics and add tags to filter those metrics for their needs. For example, you can have an api_http_requests_total metric to count the number of HTTP requests received. Then, you can add a request_method label on this metric, which takes a POST, GET, or PUT value. Consequently, you create three metric streams for each request type you might receive. In this case, to find the number of HTTP GET requests, you run the following query:

api_http_requests_total{request_method="GET"}

See https://prometheus.io/docs/practices/naming/ for more information about metrics and labels.

The following are some of the default labels that the MonitoringTarget custom resource adds and that you can use to query metrics:

_gdch_service: The short name of the service
cluster: The name of the cluster
container_name: The name of the container within a pod
namespace_name: Your project namespace
pod_name: The pod name prefix

The following table describes the labels that the scraping service adds automatically:

Default labels of the scraping service
Metric label	Description
`job`	The internal name of the scrape job used to collect the metric. Jobs created by the `MonitoringTarget` custom resource have a name with the following pattern: `obs-system/OBS_SHADOW_PROJECT_NAME/MONITORINGTARGET_NAME.MONITORINGTARGET_NAMESPACE/I/J` `I` and `J` are unique numbers determined internally to avoid name conflicts.
`instance`	The `$IP:$PORT` of the scrapped service. If a workload resource has multiple replicas, use this field to differentiate them.

The following code samples show the use of labels and values to query different metrics for data observability:

View all the metric streams of the processed operations in your project:
```
processed_ops_total
```
View the processed operations collected in the organization system cluster:
```
processed_ops_total{cluster="ORG_SYSTEM_CLUSTER_NAME"}
```
View the CPU usage collected on the user cluster:
```
cpu_usage{cluster="USER_CLUSTER_NAME"}
```

Use the metrics relabeling tool to add labels not exposed initially by the scrapped containers and rename produced metrics. You must configure the MonitoringTarget custom resource to add labels on the metrics it collects. Specify those labels in the metricsRelabelings field of the custom resource. For more information, see Label metrics.

Query metrics from the HTTP API

The Observability platform exposes an HTTP API endpoint for querying and reading metrics, alerts, and other time series data from your project.

Query metrics directly from the HTTP API to read and export metrics and other time series data to external tools, set up automated tasks, adapt responses, and build integrations according to your system monitoring use case. For example, insert the output into another command, export details to text file formats, or configure a Linux cron job. You can call the API from the command-line interface (CLI) or a web browser and use query language expressions as endpoint parameters to obtain the result in JSON format.

This section explains how to call the HTTP API endpoint from the CLI using the API specification to query metrics for data observability.

Before you begin

You must obtain authorization to query metrics from the CLI. To get the permissions you need to access the HTTP API endpoint, ask your Organization IAM Admin to grant you the corresponding viewer role in the platform-obs namespace.

For information about setting role bindings from the GDC console, see Grant access to resources.

HTTP API endpoint

The following URL is the HTTP API endpoint for accessing metrics in the platform-obs project:

https://GDC_URL/platform-obs/cortex/prometheus/

Replace GDC_URL with the URL of your organization in Distributed Cloud.

Download and install the gdcloud CLI.
Set the gdcloud core/organization_console_url property:
```
gdcloud config set core/organization_console_url https://GDC_URL
```
Replace GDC_URL with the URL of an organization in GDC.
Sign in with the configured identity provider:
```
gdcloud auth login
```
Use your user and password to authenticate and sign in.

When the login is successful, you can use the authorization header in your cURL request through the gdcloud auth print-identity-token command. For more information, see gdcloud auth.

Call the API endpoint

Work through the following steps to reach the HTTP API endpoint from the CLI and query metrics:

Ensure you meet the prerequisites and the login authentication for the cURL request.
Open the CLI.
Use the curl tool to call the HTTP API endpoint and extend the URL using the standard https://prometheus.io/docs/prometheus/latest/querying/api/ to query metrics. For example:
```
curl https://GDC_URL/PROJECT_NAME/cortex/prometheus/api/v1/query?query=my_metric{cluster="org-1-system"}&time=2015-07-01T20:10:51.781Z \
-H "Authorization: Bearer $(gdcloud auth print-identity-token \
--audiences=https://GDC_URL)"
```
Replace the following:
- GDC_URL: the URL of an organization in GDC
- PROJECT_NAME: the name of your project

You obtain the output following the command. The API response is in JSON format.

Query and view PA metrics

Query and view metrics on dashboards

Before you begin

Build queries for your metrics

System monitoring instance endpoint

Query metrics

Sample queries and labels

Query metrics from the HTTP API

Before you begin

HTTP API endpoint

Authenticate the cURL request

Call the API endpoint