Migrate from the Trace exporter to the OTLP endpoint

This document describes how to migrate an application that is instrumented with OpenTelemetry and relies on Google Cloud exporters, to use OpenTelemetry's OTLP exporters. Both configurations send telemetry to your Google Cloud project. The steps in this document are for the in-process export performed by the OpenTelemetry SDK.

This document describes how to export trace data when you use manual instrumentation. This guidance, which is provided for Java, Go and Python, doesn't apply to sending log or metric data to your Google Cloud project.

Why you should migrate

The OpenTelemetry SDKs generate log, metric, and trace data in the OTLP format. When an application exports that data to a Google Cloud project by using a Google Cloud exporter, that exporter performs the following steps:

  1. Transforms the recorded data from OTLP-format into a proprietary format defined by the Cloud Logging API, the Cloud Monitoring API, or the Cloud Trace API.
  2. Sends the transformed data to the appropriate API, which is then stored in your Google Cloud project.

For trace data, we recommend that you migrate your application to use the Telemetry (OTLP) API to export data, because this export doesn't require a data transformation. Data transformation might cause loss of some data. For example, the proprietary format might have lower limits for certain fields, or some OTLP fields might not map to a field in the proprietary format.

Migration guide for manual instrumentation

This section describes how to modify your application so that it sends your trace data to your Google Cloud project by using the Telemetry API. You can't send metric or log data to this endpoint.

Add dependencies

The first step is to add dependencies for OpenTelemetry's OTLP trace exporter in your application. Select the dependency versions appropriate for your application and build system.

Java

For a Java application using the Gradle build system:

// build.gradle
implementation("io.opentelemetry:opentelemetry-exporter-otlp:1.47.0")

Go

For a Golang application ensure the go.mod file has the following dependencies:

// go.mod file
require(
  // OTLP exporter that uses grpc protocol for export
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.34.0
  // Alternatively, for export using http/protobuf protocol, use:
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.34.0
)

Python

For a Python application install the following dependencies or update the requirements.txt file:

# Requirements.txt - use appropriate versions
#
# OTLP exporter that uses grcp protocol for export
opentelemetry-exporter-otlp-proto-grpc==1.30.0
grpcio==1.70.0
# Alternatively, for export using http/protobuf protocol, use:
opentelemetry-exporter-otlp-proto-http==1.30.0

Replace the use of Google Cloud exporters with OTLP exporters

Update your application code such that the OpenTelemetry SDK is configured to use the OpenTelemetry OTLP exporters instead of Google Cloud trace exporter. The required changes are language-specific.

Java

import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;
import io.opentelemetry.sdk.trace.export.SpanExporter;
import io.opentelemetry.sdk.trace.samplers.Sampler;

// Initialize OpenTelemetry SDK with OTLP exporters
public static OpenTelemetry initOpenTelemetry() {
    // Initialize the OTLP gRPC exporter
    SpanExporter otlpGrpcSpanExporter =
        OtlpGrpcSpanExporter.builder()
            .setTimeout(2, TimeUnit.SECONDS)
            .build();

    // Initialize OpenTelemetry tracer provider
    SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
        .setResource(resource)
        .setSampler(Sampler.traceIdRatioBased(0.02))
        .addSpanProcessor(
            BatchSpanProcessor.builder(otlpGrpcSpanExporter)
                .setScheduleDelay(100, TimeUnit.MILLISECONDS)
                .build());

    // Configure OpenTelemetry SDK instacne to use the tracer provider
    // configured with OTLP exporter
    OpenTelemetrySdk openTelemetrySdk =
        OpenTelemetrySdk.builder()
            .setTracerProvider(tracerProvider)
            .build();
}

Go

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    // other dependencies
)

// Initializes OpenTelemetry with OTLP exporters
func init() {
    ctx := context.Background()

    // Initialize the OTLP gRPC exporter
    exporter, err := otlptracegrpc.New(ctx)
    if err != nil {
        panic(err)
    }
    // initialize OpenTelemetry tracer provdier
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithSampler(sdktrace.TraceIDRatioBased(0.02)),
        sdktrace.WithBatcher(exporter)
    )

    // configure OpenTelemetry SDK instance to use the tracer provider
    // configured with OTLP exporter
    otel.SetTracerProvider(tp)
}

Python

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
    OTLPSpanExporter,
)
from opentelemetry.sdk.resources import SERVICE_NAME, Resource

# Initialize OpenTelemetry with OTLP exporters
def init():
    # Initialize the OTLP gRPC or http exporter
    otlp_grpc_exporter = OTLPSpanExporter()

    # Initialize OpenTelemetry TracerProvider
    trace_provider = TracerProvider(resource=resource).add_span_processor(
    BatchSpanProcessor(otlp_grpc_exporter)
    )

    # Configure OpenTelemetry tracing API with the initialized tracer provider
    trace.set_tracer_provider(trace_provider)

Configure authentication

With the previous changes to the OpenTelemetry SDK configuration, your application is configured to export traces using the OpenTelemetry OTLP exporters using either gRPC or HTTP. Next, you need to configure the exports to send those traces to your Google Cloud project.

To configure authentication, do the following:

  1. Configure authentication headers for the export calls.
  2. Configure the required OpenTelemetry resource attributes and OTLP headers.
  3. Configure the exporter endpoint.

This section details each of these steps.

Configure authentication headers for the export calls

To configure the exporter with your Google Cloud Application Default Credentials (ADC), add a language-specific Google Auth Library.

Java

To authenticate to Trace, set up Application Default Credentials. For more information, see Set up authentication for a local development environment. 

// build.gradle
// Google Auth Library
implementation("com.google.auth:google-auth-library-oauth2-http:1.32.1")

Go

To authenticate to Trace, set up Application Default Credentials. For more information, see Set up authentication for a local development environment. 

// go.mod file
require (
    // When using gRPC based OTLP exporter, auth is built-in
    google.golang.org/grpc v1.70.0
    // When using http based OTLP exported, use explicit auth library
    golang.org/x/oauth2 v0.26.0
)

Python

To authenticate to Trace, set up Application Default Credentials. For more information, see Set up authentication for a local development environment. 

# requirements.txt
# Google Auth Library
google-auth==2.38.0

Next, update the application code that constructs the OTLP span exporter so that it adds the authorization tokens retrieved from the library to the headers. This step is language-specific, but the implementation is similar for all languages.

Java

We recommend that you use the Google Cloud Authentication Extension when you are using the OpenTelemetry SDK Autoconfigure module. For a complete example that uses this extension, see OTLP Trace with Google Auth Example.

import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;
import io.opentelemetry.sdk.trace.export.SpanExporter;
import io.opentelemetry.sdk.trace.samplers.Sampler;

import com.google.auth.oauth2.GoogleCredentials;

// Initialize OpenTelemetry SDK with OTLP exporters
public static OpenTelemetry initOpenTelemetry() {
    // Retrieve and store application-default credentials
    GoogleCredentials credentials;
    try {
       credentials = GoogleCredentials.getApplicationDefault();
    } catch (IOException e) {
      // Handle authentication error
      throw new RuntimeException(e);
    }

   // Update gRPC span exporter to add the authorization headers
   // If you are using the Autoconfigure module, we recommend using
   // Google Cloud Authentication Extension.
   // See https://github.com/open-telemetry/opentelemetry-java-contrib/tree/main/gcp-auth-extension
   SpanExporter otlpGrpcSpanExporter =
        OtlpGrpcSpanExporter.builder()
            .setHeaders(() -> {
                try {
                    credentials.refreshIfExpired();
                } catch (IOException e) {
                    // Handle authentication error
                    throw new RuntimeException(e);
                }
                return Map.of("Authorization", "Bearer " + credentials.getAccessToken().getTokenValue());
            })
            .setTimeout(2, TimeUnit.SECONDS)
            .build();

  // Other OpenTelemetry configuration remains unaffected
}

Go

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"

    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/oauth"
)

// Initializes OpenTelemetry with OTLP exporters
func init() {
    ctx := context.Background()

    // Retrieve and store Google application-default credentials
    creds, err := oauth.NewApplicationDefault(ctx)
    if err != nil {
        panic(err)
    }

    // Update the previously created OTLP gRPC span exporter to
    // add authorization headers
    exporter, err := otlptracegrpc.New(
        ctx,
        otlptracegrpc.WithDialOption(grpc.WithPerRPCCredentials(creds))
    )

    // Other OpenTelemetry configuration remains unaffected.
}

Python

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
    OTLPSpanExporter,
)
from opentelemetry.sdk.resources import SERVICE_NAME, Resource

import google.auth
import google.auth.transport.grpc
import google.auth.transport.requests
import grpc
from google.auth.transport.grpc import AuthMetadataPlugin

# Initialize OpenTelemetry with OTLP exporters
def init():
    # Retrieve and store Google application-default credentials
    credentials, project_id = google.auth.default()
    # Request used to refresh credentials upon expiry
    request = google.auth.transport.requests.Request()

    # Supply the request and credentials to AuthMetadataPlugin
    # AuthMeatadataPlugin inserts credentials into each request
    auth_metadata_plugin = AuthMetadataPlugin(
        credentials=credentials, request=request
    )

    # Initialize gRPC channel credentials using the AuthMetadataPlugin
    channel_creds = grpc.composite_channel_credentials(
        grpc.ssl_channel_credentials(),
        grpc.metadata_call_credentials(auth_metadata_plugin),
    )

    # Update the previously created OTLP gRPC span exporter to add authorization
    # credentials
    otlp_grpc_exporter = OTLPSpanExporter(credentials=channel_creds)

    # Other OpenTelementry configuration remains unaffected

Configure the required OpenTelemetry resource attributes and OTLP headers

Set the following OpenTelemetry environment variables:

  • OTEL_RESOURCE_ATTRIBUTES: Configure this environment variable to set the gcp.project_id key in the OpenTelemetry resource attributes. Set the value should be the ID of your Google Cloud project.

  • OTEL_EXPORTER_OTLP_HEADERS: Configure this environment variable to add the x-goog-user-project header. The value should be the ID of your Google Cloud project from which the service quota will be consumed. Typically, the value is set to the same value as for the gcp.project_id key.

Example:

export OTEL_RESOURCE_ATTRIBUTES="gcp.project_id=PROJECT_ID"
export OTEL_EXPORTER_OTLP_HEADERS="x-goog-user-project=QUOTA_PROJECT_ID"

For more information about OpenTelemetry environment variables, see General SDK Configuration.

Configure the exporter endpoint

Set the value of the OTEL_EXPORTER_OTLP_ENDPOINT environment variable to the OTLP endpoint for Google Cloud.

Example:

export OTEL_EXPORTER_OTLP_ENDPOINT=https://telemetry.googleapis.com

For more information about OpenTelemetry environment variables, see General SDK Configuration.

Required permissions

There are Identity and Access Management (IAM) roles and permissions that give you, or a service account, the ability to write trace data to the Telemetry API and have that data stored in your Google Cloud project. For example, if you have been granted only Cloud Trace roles or permissions, then those permissions won't let you write data by using the Telemetry API.

  • To get the permissions that you need to for the sample applications to write log, metric, and trace data, ask your administrator to grant you the following IAM roles on your project:

For more information, see How Application Default Credentials works and Set up Application Default Credentials (ADC) for a local development environment.

Required APIs

The Telemetry API is disabled by default. You must enable this API. The following provides information about the APIs required to send telemetry data to a Google Cloud project, and to view this data:

Google Cloud console

Enable the Telemetry API, Cloud Logging, Cloud Monitoring, and Cloud Trace APIs.

Enable the APIs

Google Cloud CLI

Enable the Telemetry API, Cloud Logging, and Cloud Monitoring, and Cloud Trace APIs: 

gcloud services enable telemetry.googleapis.com logging.googleapis.com monitoring.googleapis.com cloudtrace.googleapis.com