This document describes how to add observability code to your application by using OpenTelemetry. OpenTelemetry provides instrumentation libraries that generate telemetry for popular frameworks. You can augment the library-generated telemetry by adding custom instrumentation that measures your application-specific behavior.
The principles and concepts described in this document can be applied to apps written in all languages supported by OpenTelemetry.
The sample code, which is the same Go app that is described in Getting Started guide for Go, is available in GitHub. To view the full sample, click more_vert More, and then select View on GitHub.
Create custom traces
To generate custom traces from your application, you add instrumentation code that creates OpenTelemetry spans. In OpenTelemetry, spans are the building blocks for traces.
To create a span, do the following:
Modify your app to acquire an OpenTelemetry
Tracer
. In OpenTelemetry, a tracer is a creator of spans. You can acquire a tracer as demonstrated in the following code:The tracer name, which is represented by
scopeName
, identifies the instrumentation scope of the generated traces.Use the
tracer
instance to create spans. In the following code sample, thecomputeSubrequests
function generates a span whenever it is called:In the previous code sample, the span generated from the
computeSubrequests
function represents the work done by the entire function. This is because the first step of the function is to start a new span usingtracer.Start
and thedefer
keyword before thespan.End()
ensures that the span is ended right before the function exits.
Create custom metrics
To generate metrics from your application, you add instrumentation code that records measurements taken during your app's execution.
To create metrics, do the following:
Modify your app to acquire an OpenTelemetry
Meter
. In OpenTelemetry, a meter provides access to metric instruments for recording metrics. You can acquire a meter as demonstrated in the following code:The meter name, which is represented by
scopeName
, identifies the instrumentation scope of the generated metrics.Use the
meter
instance to create instruments which can record metrics. For example, in the following code, we use themeter
to create an OpenTelemetry Histogram:This previous code generates a histogram named
sleepHistogram
.Use the
sleepHistogram
instance to record the sleep time, which is determined when the functionrandomSleep
is invoked:The recorded metrics from these instruments are exported from your application based on your OpenTelemetry exporter configuration.
Using OpenTelemetry Exemplars
Exemplars in OpenTelemetry provide examples of measurements recorded with a metric instrument, and provide a link to the current Span. Cloud Monitoring and Google Cloud Managed Service for Prometheus only support exemplars on histograms.
A histogram takes many measurements and groups them together into buckets. Histogram data is typically displayed by using a heatmap. You can also calculate percentiles such as the 95th percentile, based on the bucket distribution. Histograms discards information about individual measurements to efficiently store large numbers of them.
Exemplars provide "example" measurements which link to spans. By selecting an exemplar and navigating to the trace, you can see more detailed information about the example measurement, and also see where the measurement occurred within your broader application.
Exemplars appear as dots in a histogram heatmap:
OpenTelemetry instrumentation libraries automatically include exemplars when you have
configured both the OpenTelemetry metric and trace SDKs. To include exemplars on your
custom instrumentation, you need to ensure the metric is recorded within a span,
and that you pass the context from the span with the metric measurement. In the
computeSubrequests
function previously shown, these actions are performed by
invoking subRequestsHistogram.Record
between tracer.Start
and span.End
(which is deferred), and by passing the context, ctx
, from tracer.Start
to
the Record
function.
For more information on exemplars, see OpenTelemetry documentation on Exemplars.