Deploy Google-Built OpenTelemetry Collector on Cloud Run
Stay organized with collections Save and categorize content based on your preferences.

This document shows how to run the Google-Built OpenTelemetry Collector on Cloud Run to collect OTLP logs, metrics, and traces from instrumented applications and then export that data to Google Cloud.

Before you begin

Running the Google-Built OpenTelemetry Collector requires the following resources:

A Google Cloud project with the Cloud Monitoring API, Cloud Trace API, and Cloud Logging API enabled.
- If you don't have a Google Cloud project, then do the following:
  1. In the Google Cloud console, go to New Project:
    
    Create a New Project
  2. In the Project Name field, enter a name for your project and then click Create.
  3. Go to Billing:
    
    Go to Billing
  4. Select the project you just created if it isn't already selected at the top of the page.
  5. You are prompted to choose an existing payment profile or to create a new one.
  The Monitoring API, Trace API, and Logging API are enabled by default for new projects.
- If you already have a Google Cloud project, then ensure that the Monitoring API, Trace API, and Logging API are enabled:
  
  Enable the APIs
A Cloud Run service. If you don't have a Cloud Run service, then follow the instructions at Plan and prepare your service.
An installation of gcloud. For information about installing gcloud, see Install the gcloud CLI.

Configure permissions for the Collector

By default, Cloud Run jobs and services use the Compute Engine default service account, PROJECT_NUMBER-compute@developer.gserviceaccount.com. This service account usually has the Identity and Access Management (IAM) roles necessary to write the metrics and logs described in this document:

Ask your administrator to grant you the following IAM roles on your project:

Cloud Run Admin (roles/run.admin)
Secret Manager Secret Accessor (roles/secretmanager.secretAccessor)

You can also configure a user-managed service account for Cloud Run. A user-managed service account must also have these roles. For more information about service accounts for Cloud Run, see Introduction to service identity.

Deploy the Collector

To install the Google-Built OpenTelemetry Collector as a sidecar for your Cloud Run, first create a secret to store the collector's configuration.

gcloud secrets create SECRET_NAME --data-file=config.yaml --project=PROJECT_ID

Next, add the Google-Built OpenTelemetry Collector as a sidecar to your service.yaml:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  annotations:
    run.googleapis.com/launch-stage: ALPHA
  name: google-otel-cloud-run-sample
spec:
  template:
    metadata:
      annotations:
        # [REQUIRED] set the collector as a parent to the app
        run.googleapis.com/container-dependencies: "{app:[collector]}"
        run.googleapis.com/secrets: 'SECRET_NAME:projects/PROJECT_ID/secrets/SECRET_NAME'
    spec:
      containers:
      - image: my-app
        name: app
        ports:
        - containerPort: 8080
        env:
        - name: "OTEL_EXPORTER_OTLP_ENDPOINT"
          value: "http://localhost:4317"
      - image: "us-docker.pkg.dev/cloud-ops-agents-artifacts/google-cloud-opentelemetry-collector/otelcol-google:0.121.0"
        args:
        - --config=/etc/otelcol-google/config.yaml
        name: collector
        startupProbe:
          httpGet:
            path: /
            port: 13133
          timeoutSeconds: 30
          periodSeconds: 30
        livenessProbe:
          httpGet:
            path: /
            port: 13133
          timeoutSeconds: 30
          periodSeconds: 30
        volumeMounts:
        - mountPath: /etc/otelcol-google/
          name: config
      volumes:
      - name: config
        secret:
          items:
          - key: latest
            path: config.yaml
          secretName: 'SECRET_NAME'

Configure the Collector

We provide an OpenTelemetry Collector configuration for you to use with the Google-built Collector. This configuration is designed to deliver high volumes of OTLP metrics, logs, and traces. This configuration is also designed to prevent common ingestion issues. You can add to the configuration, but we strongly recommend that you don't remove elements.

This section describes the provided configuration, the key components like exporters, processors, receivers, and other available components.

Provided Collector configuration

You can find the Collector configuration in the google-built-opentelemetry-collector directory on the opentelemetry-operations-collector repository:

receivers:
  # Open two OTLP servers:
  # - On port 4317, open an OTLP GRPC server
  # - On port 4318, open an OTLP HTTP server
  #
  # Docs:
  # https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317
      http:
        cors:
          # This effectively allows any origin
          # to make requests to the HTTP server.
          allowed_origins:
          - http://*
          - https://*
        endpoint: localhost:4318

  # Using the prometheus scraper, scrape the Collector's self metrics.
  #
  # Docs:
  # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusreceiver
  # https://opentelemetry.io/docs/collector/internal-telemetry/
  prometheus/self-metrics:
    config:
      scrape_configs:
      - job_name: otel-self-metrics
        scrape_interval: 1m
        static_configs:
        - targets:
          - localhost:8888

processors:
  # The batch processor is in place to regulate both the number of requests
  # being made and the size of those requests.
  #
  # Docs:
  # https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor
  batch:
    send_batch_max_size: 200
    send_batch_size: 200
    timeout: 5s

  # The memorylimiter will check the memory usage of the collector process.
  #
  # Docs:
  # https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/memorylimiterprocessor
  memory_limiter:
    check_interval: 1s
    limit_percentage: 65
    spike_limit_percentage: 20

  # The resourcedetection processor is configured to detect GCP resources.
  # Resource attributes that represent the GCP resource the collector is
  # running on will be attached to all telemetry that goes through this
  # processor.
  #
  # Docs:
  # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor
  # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor#gcp-metadata
  resourcedetection:
    detectors: [gcp]
    timeout: 10s

  # The transform/collision processor ensures that any attributes that may
  # collide with the googlemanagedprometheus exporter's monitored resource
  # construction are moved to a similar name that is not reserved.
  transform/collision:
    metric_statements:
    - context: datapoint
      statements:
      - set(attributes["exported_location"], attributes["location"])
      - delete_key(attributes, "location")
      - set(attributes["exported_cluster"], attributes["cluster"])
      - delete_key(attributes, "cluster")
      - set(attributes["exported_namespace"], attributes["namespace"])
      - delete_key(attributes, "namespace")
      - set(attributes["exported_job"], attributes["job"])
      - delete_key(attributes, "job")
      - set(attributes["exported_instance"], attributes["instance"])
      - delete_key(attributes, "instance")
      - set(attributes["exported_project_id"], attributes["project_id"])
      - delete_key(attributes, "project_id")

exporters:
  # The googlecloud exporter will export telemetry to different
  # Google Cloud services:
  # Logs -> Cloud Logging
  # Metrics -> Cloud Monitoring
  # Traces -> Cloud Trace
  #
  # Docs:
  # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/googlecloudexporter
  googlecloud:
    log:
      default_log_name: opentelemetry-collector

  # The googlemanagedprometheus exporter will send metrics to
  # Google Managed Service for Prometheus.
  #
  # Docs:
  # https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/googlemanagedprometheusexporter
  googlemanagedprometheus:

service:
  pipelines:
    logs:
      receivers:
      - otlp
      processors:
      - resourcedetection
      - memory_limiter
      - batch
      exporters:
      - googlecloud
    metrics/otlp:
      receivers:
      - otlp
      processors:
      - transform/collision
      - resourcedetection
      - memory_limiter
      - batch
      exporters:
      - googlemanagedprometheus
    metrics/self-metrics:
      receivers:
      - prometheus/self-metrics
      processors:
      - resourcedetection
      - memory_limiter
      - batch
      exporters:
      - googlemanagedprometheus
    traces:
      receivers:
      - otlp
      processors:
      - resourcedetection
      - memory_limiter
      - batch
      exporters:
      - googlecloud
  telemetry:
    metrics:
      address: localhost:8888

Exporters

The Collector configuration includes the following exporters:

googlecloud exporter, for logs and traces. This exporter is configured with a default log name.
googlemanagedprometheus exporter, for metrics. This exporter does not require any configuration, but there are configuration options. For information about configuration options for the googlemanagedprometheus exporter, see Get started with the OpenTelemetry Collector in the Google Cloud Managed Service for Prometheus documentation.

Processors

The Collector configuration includes the following processors:

batch: Configured to batch telemetry requests at the Google Cloud maximum number of entries per request, or at the Google Cloud minimum interval of every 5 seconds (whichever comes first).
memory_limiter: Caps the Collector's memory usage to prevent out-of-memory crashes by dropping data points when the limit is exceeded.
resourcedetection: Automatically detects Google Cloud resource labels such as project_id.

Receivers

The Collector configuration includes only the otlp receiver. For information about instrumenting your applications to send OTLP traces and metrics to the Collector's OTLP endpoint, see see Choose an instrumentation approach.

Available components

The Google-Built OpenTelemetry Collector contains the components that most users will need to enable a rich experience within Google Cloud Observability. For a complete list of available components, see Components in the opentelemetry-operations-collector repository.

To request any changes or additions to the available components, open a feature request. in the opentelemetry-operations-collector repository.

Generate telemetry

To generate telemetry, you create a Cloud Run application with a sidecar collector. The document Write OTLP metrics by using an OpenTelemetry sidecar provides a tutorial for using the Google-Built OpenTelemetry Collector as a sidecar. You can use this tutorial to generate telemetry with the Google-built Collector.

View telemetry

The Google-Built OpenTelemetry Collector sends metrics, logs, and traces from your instrumented applications to Google Cloud Observability. The Collector also sends self-observability metrics. The following sections describe how to view this telemetry.

View your metrics

The Google-Built OpenTelemetry Collector collects Prometheus metrics that you can view by using the Metrics Explorer. The metrics collected depend on the instrumentation of the app, although the Google-built Collector also writes some self-metrics.

To view the metrics collected by the Google-Built OpenTelemetry Collector, do the following:

In the Google Cloud console, go to the Metrics explorer page:
Go to Metrics explorer

If you use the search bar to find this page, then select the result whose subheading is Monitoring.
In the toolbar of the Google Cloud console, select your Google Cloud project.
In the Metric element, expand the Select a metric menu, enter Prometheus Target in the filter bar, and then use the submenus to select a specific resource type and metric:
1. In the Active resources menu, select Prometheus Target.
2. To select a metric, use the Active metric categories and Active metrics menus. Metrics collected by the Google-Built OpenTelemetry Collector have the prefix prometheus.googleapis.com.
3. Click Apply.
Configure how the data is viewed.
When the measurements for a metric are cumulative, Metrics Explorer automatically normalizes the measured data by the alignment period, which results in the chart displaying a rate. For more information, see Kinds, types, and conversions.

When integer or double values are measured, such as with counter metrics, Metrics Explorer automatically sums all time series. To change this behavior, set the first menu of the Aggregation entry to None.

For more information about configuring a chart, see Select metrics when using Metrics Explorer.

View your traces

To view your trace data, do the following:

In the Google Cloud console, go to the Trace explorer page:
Go to Trace explorer

You can also find this page by using the search bar.
In the toolbar of the Google Cloud console, select your Google Cloud project. For App Hub configurations, select the App Hub host project.
In the scatter plot, select a trace.
In the Gantt chart on the Trace details panel, select a span.

A panel opens that displays information about the traced request. These details include the method, status code, number of bytes, and the user agent of the caller.
To view the logs associated with this trace, select the Logs & Events tab.

The tab shows individual logs. To view the details of the log entry, expand the log entry. You can also click View Logs and view the log by using the Logs Explorer.

For more information about using the Cloud Trace explorer, see Find and explore traces.

View your logs

From the Logs Explorer, you can inspect your logs, and you can also view associated traces, when they exist.

In the Google Cloud console, go to the Logs Explorer page:
Go to Logs Explorer

If you use the search bar to find this page, then select the result whose subheading is Logging.
Locate a log entry from your instrumented app. To view the details, expand the log entry.
Click Traces on a log entry with a trace message, and then select View trace details.

A Trace details panel opens and displays the selected trace.

For more information about using the Logs Explorer, see View logs by using the Logs Explorer.

Observe and debug the Collector

The Google-Built OpenTelemetry Collector automatically provides self-observability metrics to help you monitor its performance and ensure continued uptime of the OTLP ingestion pipeline.

To monitor the Collector, install the sample dashboard for the Collector. This dashboard offers at-a-glance insights into several metrics from the Collector, including uptime, memory usage, and API calls to Google Cloud Observability.

To install the dashboard, do the following:

In the Google Cloud console, go to the Dashboards page:
Go to Dashboards

If you use the search bar to find this page, then select the result whose subheading is Monitoring.
Click Dashboard Templates.
Search for the OpenTelemetry Collector dashboard.
Optional: To preview the dashboard, select it.
Click Add dashboard to your list and then complete the dialog.

The dialog lets you select the name of the dashboard, and add labels to the dashboard.

For more information about installing dashboards, see Install a dashboard template.