gcloud dataproc jobs wait 5c1754a5-34f7-4553-b667-8a1199cb9cab \
--project my-project-id --region my-cluster-region
Waiting for job output...
... INFO gcs.GoogleHadoopFileSystemBase: GHFS version: 1.4.2-hadoop2
... 16:47:45 INFO client.RMProxy: Connecting to ResourceManager at my-test-cluster-m/
...
[[["易于理解","easyToUnderstand","thumb-up"],["解决了我的问题","solvedMyProblem","thumb-up"],["其他","otherUp","thumb-up"]],[["很难理解","hardToUnderstand","thumb-down"],["信息或示例代码不正确","incorrectInformationOrSampleCode","thumb-down"],["没有我需要的信息/示例","missingTheInformationSamplesINeed","thumb-down"],["翻译问题","translationIssue","thumb-down"],["其他","otherDown","thumb-down"]],["最后更新时间 (UTC)：2025-08-27。"],[[["\u003cp\u003eDataproc automatically gathers job output, making it easily accessible for review without maintaining a constant cluster connection or navigating complex log files.\u003c/p\u003e\n"],["\u003cp\u003eSpark logs include both driver logs, which contain job output, and executor logs, which contain executable output useful for debugging.\u003c/p\u003e\n"],["\u003cp\u003eThe destination of Dataproc job driver output and Spark driver logs varies based on property settings like \u003ccode\u003edataproc:dataproc.logging.stackdriver.job.driver.enable\u003c/code\u003e and \u003ccode\u003espark:spark.submit.deployMode\u003c/code\u003e.\u003c/p\u003e\n"],["\u003cp\u003eJob output can be accessed via the Google Cloud console, the gcloud CLI, Cloud Storage, or Logging.\u003c/p\u003e\n"],["\u003cp\u003eWhen not using the Dataproc \u003ccode\u003ejobs\u003c/code\u003e API, such as direct submission with \u003ccode\u003espark-submit\u003c/code\u003e, Spark driver logs are streamed to the client in \u003ccode\u003eclient\u003c/code\u003e mode or found in Logging under the cluster resource in \u003ccode\u003ecluster\u003c/code\u003e mode, while Spark executor logs are in Logging under the cluster resource.\u003c/p\u003e\n"]]],[],null,["# Dataproc job output and logs\n\nWhen you [submit a Dataproc job](/dataproc/docs/guides/submit-job),\nDataproc automatically gathers the\njob output, and makes it available to you. This means you can\nquickly review job output without having to maintain a\nconnection to the cluster while your jobs run or look through complicated log\nfiles.\n| **System and cluster logs:** This guide describes how to configure and view job output. See [Dataproc cluster logs in Logging](/dataproc/docs/guides/logging#cluster_logs_in) for information on configuring and viewing Dataproc cluster system and daemon logs.\n\nSpark logs\n----------\n\nThere are two types of Spark logs: Spark driver logs and Spark executor logs.\nSpark driver logs contain job output; Spark executor logs contain job executable\nor launcher output, such as a `spark-submit` \"Submitted application xxx\" message, and\ncan be helpful for debugging job failures.\n\nThe Dataproc job driver, which is distinct from the Spark driver,\nis a launcher for many job types. When launching Spark jobs, it runs as a\nwrapper on the underlying `spark-submit` executable, which launches the Spark\ndriver. The Spark driver runs the job on the Dataproc cluster in Spark\n[`client` or `cluster` mode](https://spark.apache.org/docs/latest/running-on-yarn.html#launching-spark-on-yarn):\n\n- `client` mode: the Spark driver runs the job in the `spark-submit` process,\n and Spark logs are sent to the Dataproc job driver.\n\n- `cluster` mode: the Spark driver runs the job in a YARN container. Spark driver\n logs are not available to the Dataproc job driver.\n\nDataproc and Spark job properties overview\n------------------------------------------\n\n| **Note:** The first two properties in the following table must be set at cluster creation time, and cannot be overridden when a job is submitted.\n\nSpark jobs submitted using the Dataproc `jobs` API\n--------------------------------------------------\n\nThe tables in this section list the effect of different property settings on the\ndestination of Dataproc job driver output when jobs are submitted\nthrough the Dataproc `jobs` API, which includes job submission through the\nGoogle Cloud console, gcloud CLI, and Cloud Client Libraries.\n\nThe listed [Dataproc and Spark properties](/dataproc/docs/concepts/configuring-clusters/cluster-properties)\ncan be set with the `--properties` flag when a cluster is created, and will apply\nto all Spark jobs run on the cluster; Spark properties can also be set with the\n`--properties` flag (without the \"spark:\" prefix) when a job is\nsubmitted to the Dataproc `jobs` API, and will apply only to the job.\n\n### Dataproc job driver output\n\nThe following tables list the effect of different property settings on the\ndestination of Dataproc job driver output.\n\n### Spark driver logs\n\nThe following tables list the effect of different property settings on the\ndestination of Spark driver logs.\n\n### Spark executor logs\n\nThe following tables list the effect of different property settings on the\ndestination of Spark executor logs.\n\nSpark jobs submitted without using the Dataproc `jobs` API\n----------------------------------------------------------\n\nThis section lists the effect of different property settings on the\ndestination of Spark job logs when jobs are submitted\nwithout using the Dataproc `jobs` API, for example when submitting\na job directly on a cluster node using `spark-submit` or when using a Jupyter\nor Zeppelin notebook. These jobs do not have Dataproc job IDs or drivers.\n\n### Spark driver logs\n\nThe following tables list the effect of different property settings on the\ndestination of Spark driver logs for jobs not submitted through the Dataproc `jobs` API.\n\n### Spark executor logs\n\nWhen Spark jobs are not submitted through the Dataproc `jobs` API, executor\nlogs are in Logging `yarn-userlogs` under the cluster resource.\n\nView job output\n---------------\n\nYou can access Dataproc job output in the Google Cloud console,\nthe gcloud CLI, Cloud Storage, or Logging.\n\n\u003cbr /\u003e\n\n### Console\n\nTo view job output, go to your project's Dataproc\n[**Jobs**](https://console.cloud.google.com/project/_/dataproc/jobs)\nsection, then click on the **Job ID** to view job output.\n\nIf the job is running, job output periodically refreshes with\nnew content.\n\n### gcloud command\n\nWhen you submit a job with the\n[gcloud dataproc jobs submit](/sdk/gcloud/reference/dataproc/jobs/submit)\ncommand, job output is displayed on the console. You can \"rejoin\"\noutput at a later time, on a different computer, or in\na new window by passing your job's ID to the\n[gcloud dataproc jobs wait](/sdk/gcloud/reference/dataproc/jobs/wait)\ncommand. The Job ID is a\n[GUID](https://wikipedia.org/wiki/Globally_unique_identifier),\nsuch as `5c1754a5-34f7-4553-b667-8a1199cb9cab`. Here's an example. \n\n```\ngcloud dataproc jobs wait 5c1754a5-34f7-4553-b667-8a1199cb9cab \\\n --project my-project-id --region my-cluster-region\n``` \n\n```\nWaiting for job output...\n... INFO gcs.GoogleHadoopFileSystemBase: GHFS version: 1.4.2-hadoop2\n... 16:47:45 INFO client.RMProxy: Connecting to ResourceManager at my-test-cluster-m/\n...\n```\n\n### Cloud Storage\n\nJob output is stored in Cloud Storage in\neither the [staging bucket](/dataproc/docs/concepts/configuring-clusters/staging-bucket)\nor the bucket you specified when you created your cluster. A link to\njob output in Cloud Storage is provided in the\n[Job.driverOutputResourceUri](/dataproc/docs/reference/rest/v1beta2/projects.regions.jobs)\nfield returned by:\n\n- a [jobs.get](/dataproc/docs/reference/rest/v1beta2/projects.regions.jobs/get) API request.\n- a [gcloud dataproc jobs describe](/sdk/gcloud/reference/dataproc/jobs/describe) \u003cvar translate=\"no\"\u003ejob-id\u003c/var\u003e command. \n\n ```\n $ gcloud dataproc jobs describe spark-pi\n ...\n driverOutputResourceUri: gs://dataproc-nnn/jobs/spark-pi/driveroutput\n ...\n ```\n\n| **Cloud Storage Data Retention:** To prevent data loss, job output and logs saved in Cloud Storage remain in storage after a cluster is removed. You must manually delete job output and logs from Cloud Storage.\n\n### Logging\n\nSee [Dataproc Logs](/dataproc/docs/guides/logging) for information on how to view Dataproc job output in Logging.\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e"]]
