Monitor Cloud Data Fusion system, instance, and pipeline health

This guide explains how to monitor Cloud Data Fusion instances and pipelines using the Cloud Data Fusion Monitoring dashboard in Monitoring.

The Cloud Data Fusion dashboard provides customizable views of key metrics. Viewing these metrics directly in Monitoring lets you customize filtering and aggregation of the metrics.

The dashboard includes the following views:

  • Overview: Monitors Cloud Data Fusion system metrics and logs.
  • Instance: Monitors all Cloud Data Fusion instances in the project.
  • Pipeline: Provides information about pipeline runs and performance.

For detailed information about the metrics in any view, hover over the widget and click More > View in Metrics explorer.

Before you begin

Required roles

To get the permissions that you need to view metrics, ask your administrator to grant you the Monitoring Viewer (roles/monitoring.viewer) IAM role on your project. For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Instance view widgets

This table describes the widgets in the Instance view:

Widget Description
Concurrent pipelines running Number of pipelines running concurrently.
Concurrent pipelines launched Number of pipelines in a provisioning or starting state.
API requests received Cumulative count of API requests received.
API responses count Cumulative count of API responses sent.
Authorization check count Cumulative count of authorization checks.
Authorization check time Latency of authorization checks.
Deployed pipeline count Number of deployed pipelines.
Draft pipeline count Number of draft pipelines.
Namespace count Number of namespaces.

Pipeline view widgets

This table describes the widgets in the Pipeline view:

Widget Description
Successful pipeline runs Cumulative count of successful pipeline runs.
Failed pipeline runs Cumulative count of failed pipeline runs.
Killed pipeline runs Cumulative count of killed pipeline runs.
Rejected pipeline runs Cumulative count of rejected pipeline runs.
Successful pipeline run time Time taken for successful pipeline runs to complete.
Pipeline start latency Time taken for a pipeline run to reach a "Running" state.
Dataproc provisioning latency Time taken to provision the Dataproc cluster.
Dataproc API request count Cumulative count of API requests made to Dataproc.
Successful preview run time Time taken for successful preview runs to complete.
Preview runs Number of preview runs.
Pipeline bytes read Cumulative count of bytes read by a pipeline.
Pipeline bytes written Cumulative count of bytes written by a pipeline.
Pipeline bytes shuffled Cumulative count of bytes shuffled in a pipeline.
Plugin records processed in Cumulative count of records entering a plugin.
Plugin records processed out Cumulative count of records exiting a plugin.

View and filter metrics

View and filter metrics directly in Monitoring:

  1. In the Google Cloud console, go to the Metrics explorer page.

    Go to Metrics explorer

  2. Select the Cloud Data Fusion monitored resource.

  3. Choose a metric.

  4. Select filters and aggregation.

What's next