Overview of the Google-Built OpenTelemetry Collector

This set of documents describes the Google-Built OpenTelemetry Collector and explains how to deploy the Collector to collect OpenTelemetry Protocol (OTLP) traces, metrics, and logs from instrumented applications and export that data to Google Cloud Observability and other backends.

The Google-Built OpenTelemetry Collector is an open-source, production-ready build of the upstream OpenTelemetry Collector that is built with upstream OpenTelemetry Collector components. OpenTelemetry, which is part of the Cloud Native Computing Foundation, provides open source APIs, libraries, and SDKs to collect distributed traces, metrics, and logs for application monitoring.

The Google-Built OpenTelemetry Collector lets you send correlated OTLP traces, metrics, and logs to Google Cloud Observability and other backends from applications instrumented by using OpenTelemetry SDKs. The Collector also captures metadata for Google Cloud resources, so you can correlate application performance data with infrastructure telemetry data. Using the Google-built Collector with Google Cloud Observability provides insights to improve the performance of your applications and infrastructure. For more information about the Collector, see Description of the Google-Built OpenTelemetry Collector.

Use the Google-Built OpenTelemetry Collector

You can use the Google-built Collector to collect telemetry data from your applications running on Kubernetes (including Google Kubernetes Engine), Container-Optimized OS, or standalone containers. The documents in this section describe how to configure and deploy the Google-built Collector in the following environments:

If you don't have an application ready to use the Collector, then you can deploy the OpenTelemetry demo with the Google-built Collector. For more information, see Try the OpenTelemetry demo.

For information about using OpenTelemetry instrumentation to generate traces, metrics, and logs from your applications, see the following documents:

Description of the Google-Built OpenTelemetry Collector

The Google-Built OpenTelemetry Collector is built by using upstream OpenTelemetry components and tooling, while being built and retrieved entirely from Google build-test-release infrastructure (Artifact Registry). The Google-built Collector is compatible with an OpenTelemetry Collector build from the upstream repository. It is also hosted as a Docker image for flexible deployment on any container-based system, including Kubernetes and GKE.

The Google-built Collector provides a Google-curated package with the components most users will need for a rich observability experience on Google Cloud. You don't need to select components and manually build your own Collector. By using the Google-built Collector, you can:

  • Collect metadata for Google Cloud resources so you can correlate application performance data with infrastructure telemetry data.
  • Route telemetry data to Google Cloud Observability or the backend of your choice by using exporters, including backends that natively support OpenTelemetry.
  • Simplify onboarding with recommended configurations and best practice self-monitoring, including health checks and batch processing.
  • Use the hosted Docker image for flexible deployment on any container-based system, including Kubernetes and GKE.

Best practices

OpenTelemetry maintains a list of best practices for configuring the OpenTelemetry Collector and for scaling the Collector. This section makes some additional recommendations.

Use the health-check extension

The health-check extension enables an HTTP URL that can be probed to check the status of the OpenTelemetry Collector. Using this extension provides the following benefits:

  • Early problem detection: Health checks facilitate proactive monitoring of the Collector's status, enabling the detection of potential issues before these issues negatively impact telemetry data. This preventative measure helps ensure the reliability of the observability pipeline.
  • Improved troubleshooting: When problems occur, health checks offer valuable insights into the Collector's current state. This information simplifies the diagnosis and resolution process, reducing downtime and streamlining troubleshooting efforts.
  • Enhanced reliability: Continuous monitoring of the Collector's health ensures consistent operation and prevents unexpected failures. This proactive measure enhances the overall reliability of the observability system and minimizes the risk of data loss or gaps in telemetry data.

On Kubernetes and GKE, the health-check extension is compatible with Kubernetes liveness and readiness probes. For information about setting up these probes, see Kubernetes best practices: Setting up health checks with readiness and liveness probes.

On Cloud Run, a single health-check extension can serve as the endpoint for both startup and liveness probes in your Cloud Run service configuration.

Use the batch processor

The batch processor collects traces, metrics, or logs and bundles them into batches for transmission. Using the batch processor provides the following benefits:

  • Minimizes outgoing connections: By grouping data transmissions into batches, the OpenTelemetry Collector significantly reduces the number of outgoing connections. This consolidated approach lowers quota usage and has the potential to lower overall network costs.
  • Improved data compression: Batching enables more efficient data compression, reducing the overall size of the data transmitted.
  • Flexibility in batching strategy: Support for both size-based and time-based batching provides flexibility to optimize for different scenarios. Size-based batching ensures that batches reach a certain size before being sent, while time-based batching sends batches after a specific time interval has elapsed. This flexibility lets you fine-tune the batching strategy to align with the specific characteristics of your data and the particular requirements of your application.

Release notes

The Google-Built OpenTelemetry Collector is versioned in sync with the upstream OpenTelemetry Collector. The current version is v0.121.0; the corresponding Docker image, stored in Artifact Registry, is us-docker.pkg.dev/cloud-ops-agents-artifacts/google-cloud-opentelemetry-collector/otelcol-google:0.121.0. For each new version, the changes that are most relevant to Google Cloud users are included on this page.

  • 0.121.0:
    • Google-Built OpenTelemetry Collector: Initial release
    • OpenTelemetry Collector core repository: Release notes
    • OpenTelemetry Collector Contrib repository: Release notes

Supportability

For all Google-Built OpenTelemetry Collector client-side issues, including feature requests, bug reports, and general questions, open an issue in the appropriate GitHub repository. These repositories are monitored by Google, and issues are triaged and addressed on a best-effort basis.

  • Google-Built OpenTelemetry Collector packaging repository, for issues related to the following:

    • Packaging
    • Deployment on Google Cloud
    • Authentication within Google Cloud
    • Requests for the addition of new OpenTelemetry components

  • OpenTelemetry Collector Contrib repository, for issues related to the following:

    • Google-specific OpenTelemetry components, for example:
      • googlecloudexporter
      • googlemanagedprometheusexporter
      • googleclientauthextension
      • resourcedetectionprocessor
    • Non-Google-specific OpenTelemetry components managed by the upstream community

For issues related to the Google-Built OpenTelemetry Collector's use of Google Cloud Observability services and APIs, like server errors or quotas, contact Cloud Customer Care.

Pricing

There is no charge for deploying and using the Google-Built OpenTelemetry Collector.

When you send telemetry data to Google Cloud, you are billed by ingestion volume. For information about costs associated with the ingestion of traces, logs, and Google Cloud Managed Service for Prometheus metrics, see Google Cloud Observability pricing.

Try the OpenTelemetry demo

This section describes how to deploy and run the OpenTelemetry demo for Google Cloud with the Google-Built OpenTelemetry Collector.

This section is optional. If you are ready to integrate the Google-built Collector into your own deployments, see the following documents:

Before you begin

The OpenTelemetry demo requires a Kubernetes cluster that has Workload Identity Federation configured. For information about setting up Workload Identity Federation for the OpenTelemetry demo, see Workload Identity prerequisites.

Update the demo to use the Google-built Collector

By default, the OpenTelemetry demo uses the upstream OpenTelemetry Collector. To use the Google-Built OpenTelemetry Collector instead, do the following:

  1. Clone the OpenTelemetry demo repository:

    git clone https://github.com/GoogleCloudPlatform/opentelemetry-demo.git

  2. Go to the kubernetes directory.

    cd kubernetes

  3. Edit the file opentelemetry-demo.yaml to replace the line for the Collector image to use. The line looks like the following, although the version might be different:

    image: "otel/opentelemetry-collector-contrib:0.108.0"

    Replace the value of the image: field with us-docker.pkg.dev/cloud-ops-agents-artifacts/google-cloud-opentelemetry-collector/otelcol-google:0.121.0, so that the line looks like the following, and then save the file:

    image: "us-docker.pkg.dev/cloud-ops-agents-artifacts/google-cloud-opentelemetry-collector/otelcol-google:0.121.0"

Deploy the demo

Deploy the demo by applying the updated opentelemetry-demo.yaml file:

kubectl apply --namespace otel-demo -f opentelemetry-demo.yaml

Connect to the demo

After you have applied the updated configuration, you can forward the data to a local port. For example, to connect to the demo at localhost:8080, issue the following command:

kubectl port-forward --n otel-demo svc/opentelemetry-demo-frontendproxy 8080:8080

You can then use your browser to connect to the demo at localhost:8080.

View telemetry

The OpenTelemetry demo sends metrics, traces, and logs to Google Cloud by using the Google-Built OpenTelemetry Collector. For information about the specific telemetry sent by the demo, see Seeing telemetry in the documentation for the demo.

View your metrics

The Google-Built OpenTelemetry Collector collects Prometheus metrics that you can view by using the Metrics Explorer. The metrics collected depend on the instrumentation of the app, although the Google-built Collector also writes some self-metrics.

To view the metrics collected by the Google-Built OpenTelemetry Collector, do the following:

  1. In the Google Cloud console, go to the  Metrics explorer page:

    Go to Metrics explorer

    If you use the search bar to find this page, then select the result whose subheading is Monitoring.

  2. In the toolbar of the Google Cloud console, select your Google Cloud project.
  3. In the Metric element, expand the Select a metric menu, enter Prometheus Target in the filter bar, and then use the submenus to select a specific resource type and metric:
    1. In the Active resources menu, select Prometheus Target.
    2. To select a metric, use the Active metric categories and Active metrics menus. Metrics collected by the Google-Built OpenTelemetry Collector have the prefix prometheus.googleapis.com.
    3. Click Apply.
  4. Configure how the data is viewed.

    When the measurements for a metric are cumulative, Metrics Explorer automatically normalizes the measured data by the alignment period, which results in the chart displaying a rate. For more information, see Kinds, types, and conversions.

    When integer or double values are measured, such as with counter metrics, Metrics Explorer automatically sums all time series. To change this behavior, set the first menu of the Aggregation entry to None.

    For more information about configuring a chart, see Select metrics when using Metrics Explorer.

View your traces

To view your trace data, do the following:

  1. In the Google Cloud console, go to the Trace explorer page:

    Go to Trace explorer

    You can also find this page by using the search bar.

  2. In the toolbar of the Google Cloud console, select your Google Cloud project. For App Hub configurations, select the App Hub host project.
  3. In the scatter plot, select a trace.

  4. In the Gantt chart on the Trace details panel, select a span.

    A panel opens that displays information about the traced request. These details include the method, status code, number of bytes, and the user agent of the caller.

  5. To view the logs associated with this trace, select the Logs & Events tab.

    The tab shows individual logs. To view the details of the log entry, expand the log entry. You can also click View Logs and view the log by using the Logs Explorer.

For more information about using the Cloud Trace explorer, see Find and explore traces.

View your logs

From the Logs Explorer, you can inspect your logs, and you can also view associated traces, when they exist.

  1. In the Google Cloud console, go to the Logs Explorer page:

    Go to Logs Explorer

    If you use the search bar to find this page, then select the result whose subheading is Logging.

  2. Locate a log entry from your instrumented app. To view the details, expand the log entry.

  3. Click Traces on a log entry with a trace message, and then select View trace details.

    A Trace details panel opens and displays the selected trace.

For more information about using the Logs Explorer, see View logs by using the Logs Explorer.