Stay organized with collections
Save and categorize content based on your preferences.
This page describes how to enable Cloud Logging for your
Cloud Data Fusion Dataproc clusters and access advanced
pipeline logs.
Enable Dataproc Cloud Logging
To view pipeline and cluster issues in Cloud Logging, enable advanced logs
in new or existing Cloud Data Fusion instances. To enable advanced logs in
an existing instance, do the following:
In the Google Cloud console, go to the Cloud Data Fusion
Instances page.
In the Advanced monitoring and logging section, for
Dataproc Cloud Logging, click
editEdit.
In the Cloud Logging window, select the
Enable Cloud Logging checkbox.
Click Save.
View logs
Every Cloud Data Fusion pipeline run is assigned a unique RunID.
After you deploy and run your pipeline, find its RunID. Then, in
Logging, use the RunID to view your pipeline
logs.
Get the pipeline's RunID
Go to your instance:
In the Google Cloud console, go to the Cloud Data Fusion page.
To open the instance in the Cloud Data Fusion Studio,
click Instances, and then click View instance.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-29 UTC."],[[["\u003cp\u003eThis guide explains how to enable Cloud Logging for Cloud Data Fusion Dataproc clusters, allowing you to view pipeline and cluster logs.\u003c/p\u003e\n"],["\u003cp\u003eYou can enable advanced logging on new or existing Cloud Data Fusion instances by editing the "Dataproc Cloud Logging" settings in the instance details.\u003c/p\u003e\n"],["\u003cp\u003eEach pipeline run is assigned a unique RunID, which is used to locate specific pipeline logs within Cloud Logging.\u003c/p\u003e\n"],["\u003cp\u003eTo view logs, you must go to the Cloud Logging Logs Explorer, select the correct Dataproc Cluster resource, and then can filter it down by severity or components.\u003c/p\u003e\n"],["\u003cp\u003eLogs can be downloaded directly from the Cloud Logging interface if needed.\u003c/p\u003e\n"]]],[],null,["# View advanced pipeline logs in Cloud Logging\n\nThis page describes how to enable Cloud Logging for your\nCloud Data Fusion Dataproc clusters and access advanced\npipeline logs.\n\nEnable Dataproc Cloud Logging\n-----------------------------\n\nTo view pipeline and cluster issues in Cloud Logging, enable advanced logs\nin new or existing Cloud Data Fusion instances. To enable advanced logs in\nan existing instance, do the following:\n\n1. In the Google Cloud console, go to the Cloud Data Fusion\n **Instances** page.\n\n [Go to Instances](https://console.cloud.google.com/data-fusion/locations/-/instances)\n2. Click the instance name.\n\n3. In the **Advanced monitoring and logging** section, for\n **Dataproc Cloud Logging** , click\n edit **Edit**.\n\n4. In the **Cloud Logging** window, select the\n **Enable Cloud Logging** checkbox.\n\n5. Click **Save**.\n\nView logs\n---------\n\nEvery Cloud Data Fusion pipeline run is assigned a unique `RunID`.\nAfter you deploy and run your pipeline, find its `RunID`. Then, in\nLogging, use the `RunID` to view your pipeline\nlogs.\n\n### Get the pipeline's RunID\n\n1. Go to your instance:\n 1. In the Google Cloud console, go to the Cloud Data Fusion page.\n\n 2. To open the instance in the Cloud Data Fusion Studio,\n click **Instances** , and then click **View instance**.\n\n [Go to Instances](https://console.cloud.google.com/data-fusion/locations/-/instances)\n2. Click **List**.\n3. Click the pipeline for which you want to get the Run ID.\n4. Click **Summary**.\n5. In the **Run history** section, click **Table**.\n6. To copy the Run ID, right-click the **RunID** , and click **Copy**.\n\n### View the logs in Logs Explorer\n\n1. In the Google Cloud console, go to the\n **Cloud Logging \\\u003e Logs Explorer** page:\n\n [Go Logs Explorer](https://console.cloud.google.com/logs/query)\n2. In the **All resources** drop-down,\n select\n **Cloud Dataproc Cluster \\\u003e **`cdap-`\u003cvar translate=\"no\"\u003ePIPELINE_NAME\u003c/var\u003e`-`\u003cvar translate=\"no\"\u003eYOUR_RUNID\u003c/var\u003e****.\n\n### Optional: Filter the logs\n\nUse the filter menus to filter your logs. You can filter by various log\nseverity levels\nor by components such as `datafusion-pipeline-logs`.\n\n### Optional: Download the logs\n\nClick **Download logs**.\n\nFor more information, see\n[downloading log entries](/logging/docs/view/logs-explorer-interface#download_logs).\n\nWhat's next\n-----------\n\n- Learn how to [view audit logs](/data-fusion/docs/how-to/audit-logging).\n- Learn how to [view pipeline logs from the Cloud Data Fusion Pipeline Studio](/data-fusion/docs/how-to/view-and-download-pipeline-logs)."]]
