Correlate metrics and traces by using exemplars

This document describes exemplars, which are example data points attached to a metric data point. One use of exemplars is to store a link to a trace, which lets you correlate your metric and trace data. This document also describes how you can use OpenTelemetry to generate exemplars.

About exemplars

Exemplars are a way to associate arbitrary data with metric data. You can use them to attach non-metric data to measurements. One use of exemplars is to associate trace data with metric data. In Cloud Monitoring, exemplars can be attached to metric data that has the Distribution value type. Distribution-valued metrics collect many measurements and group them into buckets. Each "point" is a histogram, which is an array that lists bucket counts. The actual measurements aren't retained, only the bucket counts. For more information, see About distribution-valued metrics.

When a metric data point is written, an array of Exemplar objects can be attached to that data. Exemplars, when used, attach useful information to the metric data. For example, when OpenTelemetry write an exemplar, it attaches a link to the current span of a trace.

Histogram data is typically displayed by using a heatmap. When a "point" contains an exemplar, Cloud Monitoring annotates the heatmap with a dot:

  • To view summary information about the metric point, activate the tooltip for the annotation.
  • To access the trace, select the annotation.

For example, the sample application described in Generate traces and metrics with Go uses OpenTelemetry to writes metric data, trace data, and exemplars. The following screenshot shows the value of the prometheus/http_server_duration_milliseconds/histogram metric after several executions of the application:

Example of a heatmap showing multiple exemplars.

The previous screenshot shows multiple exemplars. The tooltip for one exemplar is expanded, and it shows the date, the trace name, percentiles, and a latency value.

When the annotation is selected, the trace linked in the exemplar is displayed:

Example of the trace details for an exemplar.

When the displayed information isn't sufficiently detailed, you can select View in Trace, which opens the Trace Explorer page. On that page, you can view trace attributes and access links to log entries.

How to enable OpenTelemetry exemplars

As the previous example illustrates, you can configure OpenTelemetry to write exemplars that link the current span of a trace to a metric measurement. However, for exemplars to be written, you must do the following:

  1. Configure both the OpenTelemetry metric and trace SDKs.

  2. To include exemplars on your custom instrumentation, ensure that the metric is recorded within a span, and that you pass the context from the span with the metric measurement.

    For example, the computeSubrequests function follows the previous guidance. The function subRequestsHistogram.Record is invoked after tracer.Start and before span.End, which is deferred. Also notice that the context, ctx, is passed from tracer.Start to the Record function:

    func computeSubrequests(r *http.Request, subRequests int) error {
	// Add custom span representing the work done for the subrequests
	ctx, span := tracer.Start(r.Context(), "subrequests")
	defer span.End()

	// Make specified number of http requests to the /single endpoint.
	for i := 0; i < subRequests; i++ {
		if err := callSingle(ctx); err != nil {
			return err
		}
	}
	// record number of sub-requests made
	subRequestsHistogram.Record(ctx, int64(subRequests))
	return nil
}

    For more information about this sample, see Add custom traces and metrics to your app.

What's next