View application telemetry

Stay organized with collections Save and categorize content based on your preferences.

This document describes how to view the out-of-the-box (OOTB) dashboards for your App Hub applications, services, and workloads. These dashboards can help you understand how your application's resources are performing and to diagnose issues.

The telemetry that these OOTB dashboards display includes log and metric data, and information about open incidents. Application dashboards provide information like the location, criticality, and owners. Service and workload dashboards include information about the App Hub and infrastructure types.

Application-specific labels

App Hub-specific labels let Google Cloud Observability construct the hierarchy of OOTB dashboards and to filter the displayed telemetry to that for a service or workload. You can also use these labels to filter your log and metric data. These labels are attached when your application runs on Google Cloud resources that are integrated with App Hub. These labels might be attached when Google Cloud Managed Service for Prometheus metrics are ingested.

Log labels

Log entries contain an apphub field that lists the labels for the application and for a service or workload. The following illustrates the format of this data:

apphub: {
  application: {
    id: "my-app"
    location: "my-app-location"
  }
  workload: {
    criticalityType: "MEDIUM"
    environmentType: "STAGING"
    id: "my-workload-id"
  }
}

Metric labels

This section describes how application-specific labels are attached to metric data generated by Google Cloud resources and to Prometheus metrics ingested by using Google Cloud Managed Service for Prometheus. In the following, interpret a.b.{x,y} as a.b.x and a.b.y.

The following App Hub labels are attached to metric data from the Google Cloud resources used by your application:

• metadata.system_labels.apphub_application_{container,id,location}
• metadata.system_labels.apphub_{workload,service}_{criticality_type,environment_type,id}

App Hub application labels are attached to Google Cloud Managed Service for Prometheus metrics when any of the following is true:

• A workload is running on a Google Kubernetes Engine cluster with version 1.30 or higher, and the workload's controller type is one of the following:

  • apps.k8s.io/{Deployment,StatefulSet,DaemonSet}
  • batch.k8s.io/CronJob

• When you use OpenTelemetry on Cloud Run.

• When you use OpenTelemetry on Google Kubernetes Engine and follow OTLP Kubernetes Ingest guidance.

Managed Service for Prometheus uses the App Hub API to determine whether an App Hub application exists. When an application is found, the following application-specific labels are added to your trace data:

• metric.labels.apphub_application_{container,id,location}
• metric.labels.apphub_workload_{criticality_type,environment_type,id}

Alerting policy labels

Service and workload labels aren't automatically attached to alerting policies. However, you can associate an alerting policy with a specific workload or service by attaching all of the following labels with the following keys to the alerting policy:

• apphub_application_location
• apphub_application_id
• apphub_application_container
• apphub_service_id or apphub_workload_id

When an alerting policy includes the previous keys, incidents for the policy are shown on the OOTB dashboards.

For more information, see Associate an alerting policy with a service or workload.

Before you begin

Before you can use Application Monitoring, make sure that you configure your App Hub host project. This is the project that you use to view your application's telemetry data:

• To get the permissions that you need to view log and metric data, and to create alerting policies and customize dashboards, ask your administrator to grant you the following IAM roles:

  • Logs Viewer (roles/logging.viewer) on your App Hub host project and any other projects listed its default log scope. This role grants access to most log entries in the _Default log bucket. For more information, see Logging roles.
  • Logs View Accessor (roles/logging.viewAccessor) on the log views listed in the default log scope of your App Hub host project. To learn more about granting access to a log view, see Control access to a log view.
  • Monitoring Editor (roles/monitoring.editor) on your App Hub host project. For principals who don't need to create or modify alerting policies or dashboards, consider granting the Monitoring Viewer role (roles/monitoring.viewer).
  • App Hub viewer on your App Hub host project. For more information, see App Hub roles and permissions.

  For more information about granting roles, see Manage access to projects, folders, and organizations.

  You might also be able to get the required permissions through custom roles or other predefined roles.

• Ensure that you've configured the observability scope of your App Hub host project. For more information, see Set up application monitoring.

  The observability scope controls which data is available to your App Hub host project. The default settings for these scopes might not let you view all log or metric data.

• Update the alerting policies that monitor your application data to include application-specific labels. For more information, see Associate an alerting policy with a service or workload.

View summary of all applications

The top-level OOTB dashboard provides a concise summary of the state of your registered applications. For each application, this page lists the number of open incidents and information about the location, criticality, and environment.

To open the top-level OOTB dashboard, which lists summary information for your applications registered with App Hub, do the following:

1. In the Google Cloud console, go to the Application monitoring page:

   Go to Application monitoring

   If you use the search bar to find this page, then select the result whose subheading is Monitoring.

2. In the toolbar of the Google Cloud console, select your App Hub host project.

3. Optional: To open an application in App Hub, click more_vert More, and select Manage in AppHub.

4. To open the OOTB dashboard for an application, select the application. The next section provides guidance on how to explore or interpret the telemetry.

Explore application telemetry and incidents

When you select an application listed on the Application monitoring page, the top-level OOTB dashboard for the application is opened. This dashboard helps you understand the health of your application, by summarizing the number of open incidents and listing values for several golden signals. For example, if this page displays widespread failures, that might indicate that the source of the failure is infrastructure and not your application.

To open an application's OOTB dashboard, do the following:

1. In the Google Cloud console, go to the Application monitoring page:

   Go to Application monitoring

   If you use the search bar to find this page, then select the result whose subheading is Monitoring.

2. In the toolbar of the Google Cloud console, select your App Hub host project.

3. On the Application monitoring page, select the name of an application.

   The OOTB dashboard for the application opens with the Overview tab selected. This tab lists each service and workload, along with the number of open incidents and values for the golden signals:

   • Traffic: Incoming request rates on the resource over the selected time period.
   • Server error rate: Average percentage of incoming requests that generate or map to 5xx series HTTP error codes over the selected time period.
   • P95 latency: 95th percentile of latency for a service request aggregated over the selected time period, in milliseconds.
   • Saturation: Measures how "full" your service or workload is. For services and workloads that rely on compute-based resources like managed instance groups (MIG), Cloud Run, and Google Kubernetes Engine deployments, this field shows the CPU utilization.

   In the Overview tab, you can do all of the following:

   • Reset the time range. When you open the dashboard, the time range is set to the most recent hour.
   • View information about the application, like operator, developer, and business owners.
   • Open the application, a service, or a workload in App Hub.
   • Open the dashboard for a service or workload.

4. To view application telemetry and information about open incidents, select the Dashboard tab.

   The Dashboard tab displays log, metric, and incident information. This tab also contains filters, which you can use to restrict what data the tab displays. For information about how to explore this tab, see the following sections of this document:

   • Filter a dashboard.
   • Explore log data.
   • Explore metric data.
   • Explore incidents.

Filter a dashboard

The Dashboard tab displays data for services and workloads in an application. However, you can filter this dashboard to a specific service or to a set of services.

For example, the toolbar on the Dashboard tab contains a filter like apphub_service_id: *. If you want to view data for specific services, then click arrow_drop_down Menu, and then select those services from the list.

Explore log data

If you are viewing the Dashboard tab of an application dashboard, then your log data is shown in the Application Logs section. Otherwise, your data data is shown in the Logs section.

The Application Logs and Logs sections of a dashboard display log entries that match a filter and are stored by the resources listed in the default log scope of your App Hub host project. For applications, the filter is similar to the following:

-- Application logs
(apphub_application_location=global)
(apphub_application_id=my-app)

For services and workloads, the filter contains additional clauses. To view the filter applied to your log data, click Show query.

If you want to write queries, then you can either open a flyout or open the Logs Explorer page. Both options let you write and run queries, and to change scope. However, the flyout retains your current context and lets you save your exploration results to a custom dashboard:

• To open a flyout, click query_stats Explore data.

  When your exploration is complete, click Cancel to close the flyout and discard your changes. If you want to save your changes to a custom dashboard, then click Save to dashboard and complete the dialog. You can't save your changes to the dashboard you were viewing when you opened the flyout.

  The flyout doesn't list the filters shown by the Show query button. Those filters are applied at the dashboard level and can't be removed.

• To open the Logs Explorer page in a new browser tab, click View in Logs Explorer.

  The query is preconfigured to show log entries that match specific application labels and that are in the time range specified by the dashboard. When you close the browser tab, your changes are discarded.

For more information about how to explore your logs, see the following documents:

• Build and save queries by using the Logging query language.
• View logs by using the Logs Explorer.

Explore metric data

The charts and tables on your dashboard display golden signals and other metric data that is relevant for the resources used by your application, service, or workload. The displayed data can be stored in any project listed by the metrics scope of the your App Hub host project.

For any chart or table that displays metric data, you can view details like the metric type and the aggregation options. To view this information, go to the toolbar of a chart or table, and then click infoInformation.

To explore a chart or table in more detail, you can either open a flyout or open the Metrics Explorer page. Both options let you try different chart settings and save the results to a custom dashboard. However, the flyout retains your current context:

• To open a flyout, click query_stats Explore data.

  When your exploration is complete, click Cancel to close the flyout and discard your changes. If you want to save your changes to a custom dashboard, then click Save to dashboard and complete the dialog. You can't save your changes to the dashboard you were viewing when you opened the flyout.

• To view the chart or table in the Metrics Explorer, go to the toolbar, click more_vert More, and then select View in Metrics Explorer.

  You now have a temporary chart that you can modify and explore. For example, you might change the aggregation options or change the display style. To save your chart to a custom dashboard, click Save to dashboard and complete the

Explore incidents

The Alerts section of dashboards lists recent incidents for alerting policies that are associated with a service or workload. These alerting policies must be stored in your App Hub host project.

If you don't see any entries in the Alerts section, then you don't have any alerting policies associated with a service or workload. To create this association, add labels to the policy. For more information, see Associate an alerting policy with a service or workload.

To get more information about your incidents and alerting policies, do the following:

• To find details about the alerting policy that caused the incident, click the policy name. The dashboard that opens displays detailed information about the alerting policy, including a chart that displays the monitored data and the labels.

• To view all alerting policies associated with the application, go to the toolbar and click query_stats Explorer data. A flyout opens that lists all policies. To return to the dashboard, click Cancel.

Explore service or workload telemetry and incidents

An OOTB dashboard is provided for each service and workload. These dashboards display log, metric, and incident information, and they can help you understand the health of a specific service or workload. You can explore the data on these OOTB dashboards similarly to exploring the data shown on an application's Dashboard tab.

To open the OOTB dashboard for a service or workload, do the following:

1. In the Google Cloud console, go to the Application monitoring page:

   Go to Application monitoring

   If you use the search bar to find this page, then select the result whose subheading is Monitoring.

2. In the toolbar of the Google Cloud console, select your App Hub host project.

3. On the Application monitoring page, select the name of an application, and then select a service or workload.

   A dashboard displays the log, metric, and incident information for the service or workload you selected. For information about how to explore this data, see the following sections of this document:

   • Explore log data.
   • Explore metric data.
   • Explore incidents.

Create a custom dashboard

Google Cloud Observability supports the creation of custom dashboards, which let you have full control over the content of the dashboard. Custom dashboards are listed in the Dashboards page of the Google Cloud console, and they can be copied, edited, and deleted. They can also be the destination where you save a dashboard widget that you've configured. For example, if you are exploring a chart on an application dashboard, then you can save that chart to a custom dashboard by going to the toolbar and selecting Save to dashboard.

If you plan to create a custom dashboard for an application, service, or workload, then the following might be helpful:

• The Dashboards overview document provides a summary of widget types and it includes links to task-based guidance.

• You can add dashboard-level filters to a custom dashboard. Using these filters might be more efficient, and more flexible, than adding filters to individual widgets. Filters are based on labels. For a list of labels, see the Application-specific labels section of this document.

• Consider whether an existing template might have most of the content that you want. For more information, including a link to the GitHub repository that stores templates, see Install a dashboard template.

Associate an alerting policy with an App Hub application

You associate an alerting policy with a service or workload by adding application-specific labels to the alerting policy. When you add these labels, they are listed in the alerting policy and also included in any incidents. To learn more about labels, see Annotate incidents with labels.

To associate an alerting policy with a workload or service by using the Google Cloud console, do the following:

1. In the Google Cloud console, go to the notifications Alerting page:

   Go to Alerting

   If you use the search bar to find this page, then select the result whose subheading is Monitoring.

2. In the toolbar of the Google Cloud console, select your App Hub host project.
3. Find the alerting policy, click more_vert View more, select Edit, and then go to the Notifications and name section.
4. In the Application labels section, select your application and then select your workload or service.
5. Click Save policy.

After you complete these steps, labels with the following keys are attached to your alerting policy. These labels identify your application and your service or workload:

• apphub_application_location
• apphub_application_id
• apphub_application_container
• apphub_service_id or apphub_workload_id

You can also add user labels to an alerting policy by using the Google Cloud CLI, Terraform, or the Cloud Monitoring API. However, you must use the label keys shown in the previous example. For more information, see the following:

• Manage alerting policies by API.
• Manage alerting policies with Terraform.

Troubleshoot

For information that might help you understand what data you are viewing, or why a dashboard doesn't display data, see Troubleshoot Application Monitoring.