Instrument an application for Application Monitoring

This document describes how to configure an App Hub application so that the metrics and trace data the application generates contains application-specific labels. It also describes how to have your application export a metric that is used by Application Monitoring to report the traffic level, server error rate, or the latency for HTTP requests workloads that run on Google Kubernetes Engine.

About application-specific labels

Application-specific labels refer to labels or attributes that is injected into log, metric, or trace data. These labels identify the service or workload that generated the data. You can use application-specific labels like any other labels. For example, you can filter data by the ID of an application. Telemetry generated by supported infrastructure automatically includes these labels. With instrumentation, metrics and traces written by your application can include these labels.

In this section, interpret a.b.{x,y} as a.b.x and a.b.y.

Metric data from instrumentation you added to your applications can include the following metric labels:

• metric.labels.apphub_application_{container,id,location}
• metric.labels.apphub_workload_{criticality_type,environment_type,id}

Trace spans generated by instrumentation you added to your applications can include the following resource attributes:

• gcp.apphub.application.{container,id,location}
• gcp.apphub.{workload,service}.{criticality_type,environment_type,id}

About OpenTelemetry HTTP server metrics

There are no system metrics for workloads that run on Google Kubernetes Engine that can report the traffic level, server error rate, or the latency for HTTP requests. However, the values for these golden signals can be derived from the http.server.request.duration, which is a metric automatically collected by the OpenTelemetry HTTP client library.

The googlemanagedprometheus exporter converts the http.server.request.duration OpenTelemetry metric into a Prometheus metric that has the following properties:

• Name: prometheus/http_server_request_duration_seconds/histogram
• Metric: prometheus.googleapis.com/http_server_request_duration_seconds/histogram
• Resource types: prometheus_target
• Unit: s
• Kind: CUMULATIVE
• Value type: DISTRIBUTION

When your Google Cloud project contains the http_server_request_duration_seconds metric, then your OOTB dashboards display the traffic level, server error rate and the latency for HTTP requests.

To learn how to configure your application to export the http_server_request_duration_seconds metric, see the Use OpenTelemetry on Kubernetes section of this document.

Add application labels to metric data

Google Cloud Observability attempts to identify the source of Prometheus metrics by comparing the attributes attached to Prometheus metrics that are sent to your project with data returned from the App Hub API. The remainder of this section lists configurations that let Google Cloud Observability identify the App Hub application.

Use OpenTelemetry on Kubernetes

To have Google Cloud Observability attach application labels to metric data generated by your application's workloads that run in Google Kubernetes Engine, do the following:

1. Instrument the application with OpenTelemetry.

   • For general information, see Instrumentation and observability.
   • For links to Go, Java, Node.js, and Python samples along with information about those samples, see Instrumentation samples overview.

2. Deploy either the Google-built Collector or the OpenTelemetry Collector, and configure the collector as follows:

   • Export metric data with the googlemanagedprometheus exporter.
   • Configure the k8sattributes processor to extract metadata, like the namespace, from the environment.
   • Configure the transform/collision processor to set the project_id, location, cluster, and namespace attributes.
   • Configure the transform/aco-gke processor to populate the top_level_controller_name and top_level_controller_type labels.

   For an example, see otlp-k8s-ingest/config/collector.yaml, which is the configuration file for the Google-built Collector. To learn more about this collector, see Deploy Google-Built OpenTelemetry Collector on GKE.

3. Register your application's workloads with App Hub.

Use Google Cloud Managed Service for Prometheus on GKE

To have Google Cloud Observability attach application labels to metric data generated by your application's workloads that runs on Google Kubernetes Engine clusters, do the following:

1. Use Google Cloud Managed Service for Prometheus with managed collection.

2. Deploy your workloads on a Google Kubernetes Engine cluster whose version is at least 1.32.1-gke.1439000.

3. Register your application's workloads with App Hub.

Google Cloud Managed Service for Prometheus discovers the values of the application labels through service discovery metadata and then adds the top_level_controller_{name,type} labels to the targetLabels.metadata. During metric ingestion, Google Cloud Observability uses the top_level_controller_{name,type} labels and the App Hub API to identify your App Hub application and to add the appropriate labels to your metric data.

Use Cloud Run

To have Google Cloud Observability attach application labels to metric data generated by your Cloud Run workloads, do the following:

1. Instrument your application by using either OpenTelemetry or the Managed Service for Prometheus sidecar for Cloud Run. For information about these approaches, see the following documents:

   • Deploy Google-Built OpenTelemetry Collector on Cloud Run
   • Use the Prometheus sidecar for Cloud Run

2. Register your application's workloads with App Hub.

Verify metric labels

To verify that your application is sending Prometheus metrics, to your project, do the following:

1. Verify that your application is sending Prometheus metrics to your project:

   1. In the Google Cloud console, go to the leaderboard Metrics explorer page:

      Go to Metrics explorer

      If you use the search bar to find this page, then select the result whose subheading is Monitoring.

   2. In the toolbar of the Google Cloud console, select your Google Cloud project. For App Hub configurations, select the App Hub host project or the app-enabled folder's management project.
   3. In the Metric element, expand the Select a metric menu, enter Prometheus Target in the filter bar, and then use the submenus to select a specific resource type and metric:
      1. In the Active resources menu, select Prometheus Target.
      2. To select a metric, use the Active metric categories and Active metrics menus.
      3. Click Apply.
   4. Configure how the data is viewed.
      

      To view labels, expand the Filter menu. Each entry in the Filter corresponds to a label.

      For more information about configuring a chart, see Select metrics when using Metrics Explorer.

   If you aren't seeing any Prometheus metrics, then review your configuration.

If your Prometheus metrics don't contain application labels, then do the following:

1. Verify that you registered your workload or service with App Hub.

2. Examine your logs to determine whether there are any errors.

   For example, if you deploy an OpenTelemetry Collector or the Google-built Collector and are running on Google Kubernetes Engine, then you might do the following:

   1. In the Google Cloud console, go to the Workloads page:

      Go to Workloads

      If you use the search bar to find this page, then select the result whose subheading is Kubernetes Engine.

   2. Select your collector deployment, and then select the Logs tab.

3. If you are using an OpenTelemetry Collector or the Google-built Collector, then review the configuration of your collector. Your collector must do the following:

   • Export metric data with the googlemanagedprometheus exporter.
   • Configure the k8sattributes processor to extract metadata, like the namespace, from the environment.
   • Configure the transform/collision processor to set the project_id, location, cluster, and namespace attributes.
   • Configure the transform/aco-gke processor to populate the top_level_controller_name and top_level_controller_type labels.

   For an example, see Google-built Collector configuration file: otlp-k8s-ingest/config/collector.yaml.

Add application attributes to trace spans

To have Cloud Trace attach application-specific resource attributes to trace data generated by application's services and workloads, do the following:

1. Register your services and workloads with App Hub.
2. Instrument your application with OpenTelemetry and send the trace data collected from your application to your project by using the Google Cloud OTLP endpoint.

3. Configure the OpenTelemetry Collector or the Google-built Collector such that the exported span data includes the OpenTelemetry resource attributes that identify the supported Google Cloud resources that are used by your application. The resource attributes must include the following:

   • cloud.account.id
   • One of cloud.{availability_zone,region}
   • Resource-specific attributes. For example, for a Kubernetes workload, spans must have the k8s.cluster.name, k8s.namespace, and the type of Kubernetes deployment set.

   Use processors to have the collector attach resource-specific attributes to your spans. To learn more, see resourcedetectionprocessor and k8sattributesprocessor.

   Google Cloud Observability uses the preceding resource attributes and the App Hub API to identify your App Hub application. Application-specific attributes are added to trace data when an application is identified.

For more information about these attributes, see Google Cloud App Hub semantic conventions.