Stay organized with collections
Save and categorize content based on your preferences.
Dataproc provides a gcloud CLI dataproc clusters diagnose
command to help you troubleshoot Dataproc on GKE cluster and job issues. This
command gathers and archives cluster-related configuration files, logs, and outputs
into an archive file. and then uploads the archive to the Cloud Storage
staging bucket you specified when you
created your Dataproc on GKE cluster.
Diagnose archive file
The following tables list metrics and other information included in the
dataproc clusters diagnose command archive file.
System information
Item
Archive location
GKE node metrics
where virtual Dataproc on GKE pods run:
CPU usage
Memory usage
/system/NODE_NAME.json
Network metrics and file system status of running pods:
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-03-21 UTC."],[[["The `gcloud CLI dataproc clusters diagnose` command helps troubleshoot issues with Dataproc on GKE by gathering and archiving cluster-related data."],["The archive file created by the diagnose command contains system metrics like CPU and memory usage for GKE nodes and pods, as well as network and file system status."],["Configuration details, including cluster configmap, Kubernetes deployment, and Role-Based Access Control (RBAC) information, are included in the diagnose archive."],["The archive file also contains various logs, including agent logs, Spark engine logs, and recent Spark driver job logs."],["Job and pod-related information, such as JobAttempt and Kubernetes Pod objects, are captured in the archive file for analysis."]]],[]]
