Monitor and troubleshoot Dataproc Serverless interactive sessions

This document describes how to monitor and troubleshoot Dataproc Serverless for Spark interactive sessions using the Spark UI.

Spark UI

The Spark UI collects Apache Spark execution details from Dataproc Serverless for Spark interactive sessions. There is no charge for the Spark UI feature, which is enabled by default. Data collected by the Spark UI feature is retained for 90 days.

Limitations

The Spark UI does not support the following Dataproc Serverless workloads:

Required Identity and Access Management permissions and roles

The following permissions are required to use the Spark UI feature with interactive sessions:

  • Data collection permission: dataproc.sessions.sparkApplicationWrite. This permission must be granted to the service account that runs interactive sessions. It is included in the Dataproc Worker role, which is automatically granted to the Compute Engine default service account that Dataproc Serverless uses by default. However, if you specify a custom service account for your session or session runtime template, you must add this dataproc.sessions.sparkApplicationWrite permission to that service account (typically, by granting the service account the Dataproc Worker role).

  • Spark UI access permission: dataproc.sessions.sparkApplicationRead. This permission must be granted to a user to access the Spark UI in the Google Cloud console. It is included in the Dataproc Viewer, Dataproc Editor and Dataproc Administrator roles. To open the Spark UI in the Google Cloud console, you must have one of these roles or have a custom role that includes this permission.

Open the Spark UI

The Spark UI page is available in the Google Cloud console for Spark sessions.

  1. Go to Dataproc Interactive Sessions

  2. Click an Interactive Session ID to open the Interactive Session Details page.
  3. Click View Spark UI in the top menu.

Is the View Spark UI link disabled? If a required permission is not granted, or if you clear the Enable Spark UI checkbox on the interactive session details page or set the spark.dataproc.appContext.enabled property to false (default: true) when you create a session or create a session runtime template, the View Spark UI button will be disabled.