Query and view PA logs

After the log collector pulls data from various organization components, the Logging platform aggregates logs from all log sources, adds indexes, and associates logs with labels according to the configuration for data observability. You can then query and view logs on the logs panel of the system monitoring instance or by using the Log Query API.

Query and view logs on the logs panel

You can query logs through the user interface (UI) of the system monitoring instance of the platform-obs project. The system monitoring instance includes organization-level measures to monitor infrastructure components, such as CPU utilization, storage consumption, network monitoring, server monitoring, logs, alerts, and metrics from the Google Distributed Cloud (GDC) air-gapped components.

You can view logs on the logs panel of the system monitoring instance of the platform-obs project. Then, you can make queries to view specific logs for data observability of your use case needs.

Before you begin

Before querying and viewing logs on the logs panel, you must obtain access to the system monitoring instance. For more information, see Get access to dashboards.

Build queries for your logs

Querying logs from the UI of the system monitoring instance lets you choose the data source to retrieve logs and get an integrated view of your queries. To filter results, search for logs by labels using query language expressions. For example, you can set the cluster and namespace labels in your queries to the values that identify your cluster and project namespace, respectively.

System monitoring instance endpoint

The following URL is the endpoint of the monitoring instance of the platform-obs project:

https://GDC_URL/platform-obs/grafana

Replace GDC_URL with the URL of your organization in GDC.

Query operational logs

Complete the following steps to retrieve operational logs:

In the GDC console, select a project.
On the navigation menu, click Operations > Logging.
Click View all in Grafana Loki.

A new page opens the URL of the monitoring instance of your project.
On the UI of the monitoring instance, click explore Explore from the navigation menu to open the Explore page.
From the drop-down menu at the Explore bar, select Operational Logs as the data source to retrieve operational logs.
Enter a query to search for logs by labels using LogQL (Log Query Language) expressions. You can do this step in either of the following ways:
- Use the interactive query-builder UI. Then, click Run query.
- Enter your query using LogQL expressions directly in the text field and press Shift+Enter to run the query.
The page displays the logs matching your query.

After querying logs, you can export them. Click Export to download logs in plain text or CSV formats.

Query audit logs

Complete the following steps to retrieve audit logs:

In the GDC console, select a project.
On the navigation menu, click Operations > Audit Logging.
Click View all in Grafana Loki.

A new page opens the URL of the monitoring instance of your project.
On the UI of the monitoring instance, click explore Explore from the navigation menu to open the Explore page.
From the drop-down menu at the Explore bar, select Audit Logs as the data source to retrieve audit logs.
Enter a query to search for logs by labels using LogQL (Log Query Language) expressions. You can do this step in either of the following ways:
- Use the interactive query-builder UI. Then, click Run query.
- Enter your query using LogQL expressions directly in the text field and press Shift+Enter to run the query.
The page displays the logs matching your query.

After querying logs, you can export them. Click Export to download logs in plain text or CSV formats.

The Audit Logs option is selected on the Explore page of the UI to obtain audit logs.

Figure 1. Menu option to query audit logs from the UI of the system monitoring instance.

In figure 1, the Audit Logs option displays the interface that lets you build log queries from the UI to retrieve audit logs.

Select a time range for your logs

To query logs in a time range from the UI of the monitoring instance, follow these steps:

Click the Time Picker drop-down menu.
From the menu, perform one of the following actions:
- Select relative time range options, for example, the last 30 minutes.
- Set custom absolute time ranges by choosing specific dates and times from the calendar and clicking Apply time range.
Optionally, click Change time settings to change the Timezone and Fiscal year settings from the time range controls.

Time settings are saved on a per-dashboard basis. For more information about queries over a range of time, see https://grafana.com/docs/loki/latest/reference/api/#query-loki-over-a-range-of-time.

Sample queries and labels

The following are some of the default labels that you can use to query logs:

cluster: The name of the cluster
namespace: Your project namespace
node: The node name prefix
pod: The pod name prefix
container: The container name prefix

The following code samples show the use of labels and values to query different logs:

Select server logs:

{cluster="admin", namespace="kube-system", resources="k8s_container", container="kube-apiserver"}

Select cluster audit logs:

{cluster="admin", resources="k8s_audit"}

Query logs from the Log Query API

The Log Query API is a non-Kubernetes gRPC API that lets you programmatically query audit and operational logs, exposing its own endpoints to you.

The API is only accessible within a specific organization in Distributed Cloud, follows standard API access methods, and supports HTTP and gRPC.

Before you begin

To obtain query access to the API, set up a role binding from the GDC console to get the Log Querier (log-query-api-querier) role. For information about setting up role bindings, see Grant and revoke access.

Log Query API endpoint

The Log Query API exposes the following two endpoints to query audit and operational logs:

Audit log endpoint:
```
audit-log-query-api.ORG_DOMAIN/v1
```

Operational log endpoint:

operational-log-query-api.ORG_DOMAIN/v1

Replace ORG_DOMAIN with the domain name of the organization. You can view this property by using the gdcloud config list command. The domain name must follow the syntax org-name.gdch.google.com. For example, an organization named org-1, in the zone zone1, and in a test environment has a domain like org-1.zone1.google.gdch.test.

Send a query

Send a query to the Log Query API endpoint using HTTP or gRPC.

HTTP

Follow the instructions to directly access the API with an HTTP client to set up querying by using kubectl to manage the authentication or by handling the authentication yourself.

The Log Query API has the following three endpoints:

labels: List all labels for a project.
labels/labels/LABEL/values: List specific label values for a project.
logs: List logs for a specific project.

For more information, see the API documentation.

You can query the Log Query API using HTTP clients like curl, wget, or an HTTP client created and managed by you. The following example uses curl to query the API, and you can use a similar format for wget commands.

Authenticate the cURL request:
1. Download and install the gdcloud CLI.
2. Set the gdcloud core/organization_console_url property:
```
gdcloud config set core/organization_console_url https://GDC_URL
```
  Replace GDC_URL with the URL of an organization in GDC.
3. Sign in with the configured identity provider:
```
gdcloud auth login
```
4. Use your user and password to authenticate and sign in.
5. Export the identity token for the specified account to an environment variable:
```
export TOKEN="$($HOME/gdcloud auth print-identity-token --audiences=https://LOG_QUERY_API_ENDPOINT)"
```
  Replace LOG_QUERY_API_ENDPOINT with the Log Query API endpoint from where you want to query logs and the domain you want to connect to. Therefore, the value for the audiences flag can be, for example, https://operational-log-query-api.org-1.zone1.google.gdch.test.
  
  When the login is successful, you can use the authorization header in your cURL request through the gdcloud auth print-identity-token command. For more information, see gdcloud auth print-identity-token.
If you want to list all labels for a project, send the following query:
```
curl -X -H "Authorization: Bearer $(TOKEN)" \
  https://LOG_QUERY_API_ENDPOINT/projects/PROJECT_NAME/labels \
  -H "Content-Type: application/json" -v
```
Replace the following:
- LOG_QUERY_API_ENDPOINT: the Log Query API endpoint from where you want to query logs.
- PROJECT_NAME: the name of your project.
If you want to list specific label values for a project, send the following query:
```
curl -X -H "Authorization: Bearer $(TOKEN)" \
  https://LOG_QUERY_API_ENDPOINT/projects/PROJECT_NAME/labels/labels/LABEL/values \
  -H "Content-Type: application/json" -v
```
Replace the following:
- LOG_QUERY_API_ENDPOINT: the Log Query API endpoint from where you want to query logs.
- PROJECT_NAME: the name of your project.
- LABEL: the specific label for which you want to query its value.
If you want to query logs for a specific project, construct a Log Filter and include it in the body of the request:
```
curl -X -H "Authorization: Bearer $(TOKEN)" \
  https://LOG_QUERY_API_ENDPOINT/projects/PROJECT_NAME/logs \
  -H "Content-Type: application/json" -d \
  '{"logs_filter": {"labels_equal": {"LABEL": "LABEL_VALUE"}}}' -v
```
Replace the following:
- LOG_QUERY_API_ENDPOINT: the Log Query API endpoint from where you want to query logs.
- PROJECT_NAME: the name of your project.
- LABEL: the specific label for which you want to query logs.
- LABEL_VALUE: the label value for which you want to query logs.
Refer to the API documentation to see all options for constructing a Log Filter.

gRPC

gRPC is widely supported across programming languages and provides a more efficient communication method compared to HTTP clients.

To query logs using gRPC, you must meet the following prerequisites:

Create your own client library based on Google-supplied protocol buffers.
Implement authentication in the client.
Implement retries.

For information about the protocol buffers, see the API documentation.

The following example shows how to query logs from a Go program using a non-authenticated gRPC client. The example assumes you created a golang package that includes a Bazel build file to import code dependencies.

Save the following code to a Go program named client.go:

package main
import (
        "context"
        "crypto/tls"
        "flag"
        "fmt"
        "google.golang.org/grpc/credentials"
        "google.golang.org/grpc/metadata"
        pb "<import path to generated log query api protos>/pkg/apis/public/logging/v1/proto"
        "google.golang.org/grpc"
)

var serverAddr = flag.String("server", "localhost:8080", "server address")

func main() {
        flag.Parse()
        tc := credentials.NewTLS(&tls.Config{InsecureSkipVerify: true})
        conn, err := grpc.Dial(*serverAddr, grpc.WithTransportCredentials(tc))

        if err != nil {
                panic(error.Error(fmt.Errorf("create client connection failed: %v", err)))
        }
        defer conn.Close()

        c := pb.NewLogsClient(conn)
        md := metadata.Pairs("clienttest", "test")
        ctx := metadata.NewOutgoingContext(context.Background(), md)

        err = listLabels(ctx, c, "project-foo")
        if err != nil {
                panic(error.Error(err))
        }

        if err := listLabelValues(ctx, c, "project-foo", "resource-bar"); err != nil {
                panic(error.Error(err))
        }

        if err := listLogs(ctx, c, "project-foo", &pb.ListLogsFilter{
                LabelsEqual:    map[string]string{"resource-bar": "resource-bar-value"},
                OrderAscending: true,
        }); err != nil {
                panic(error.Error(err))
        }
}

// List all labels for a project.

func listLabels(ctx context.Context, c pb.LogsClient, project string) error {
        lbr := &pb.ListLabelsRequest{
                Parent:   project,
                PageSize: 1000, // PageSize can be configured to limit the number of responses per page.
        }
        resp, err := c.ListLabels(ctx, lbr)
        if err != nil {
                return fmt.Errorf("list labels: %v", err)
        }
        fmt.Printf("%v", resp)
        return nil
}

// List specific label values for a project.

func listLabelValues(ctx context.Context, c pb.LogsClient, project string, label string) error {
        lbr := &pb.ListLabelValuesRequest{
                Parent:   project,
                Label:    label,
                PageSize: 1000, // PageSize can be configured to limit the number of responses per page.
        }
        resp, err := c.ListLabelValues(ctx, lbr)
        if err != nil {
                return fmt.Errorf("list label values: %v", err)
        }
        fmt.Printf("%v", resp)
        return nil
}

// List logs for a specific project.

func listLogs(ctx context.Context, c pb.LogsClient, project string, lf *pb.ListLogsFilter) error {
        lbr := &pb.ListLogsRequest{
                Parent:     project,
                LogsFilter: lf,
                PageSize:   5, // PageSize can be configured to limit the number of responses per page.
        }
        resp, err := c.ListLogs(ctx, lbr)
        if err != nil {
                return fmt.Errorf("list logs: %v", err)
        }
        fmt.Printf("logs: %v", resp)
        return nil
}

Run the Go program:
```
go run PATH_TO_API/client.go -server=LOG_QUERY_API_ENDPOINT:443
```
Replace the following:
- PATH_TO_API: the path to the API file.
- LOG_QUERY_API_ENDPOINT: the Log Query API endpoint from where you want to query logs.
If the server flag is not specified, the default request gets to localhost.