Submit a Spark job by using a template

Submit a job

To submit a sample Apache Spark job that calculates a rough value for pi, fill in and execute the Google APIs Explorer Try this API template.

1. Request parameters:

   1. Insert your projectId.
   2. Specify the region where your cluster is located (confirm or replace "us-central1"). Your cluster's region is listed on the Dataproc Clusters page in the Google Cloud console.

2. Request body:

   1. job.placement.clusterName: The name of the cluster where the job will run (confirm or replace "example-cluster").
   2. job.sparkJob.args: "1000", the number of job tasks.
   3. job.sparkJob.jarFileUris: "file:///usr/lib/spark/examples/jars/spark-examples.jar". This is the local file path on the Dataproc cluster's master node where the jar that contains the Spark Scala job code is installed.
   4. job.sparkJob.mainClass: "org.apache.spark.examples.SparkPi". The is the main method of the job's pi calculation Scala application.

3. Click EXECUTE. The first time you run the API template, you may be asked to choose and sign into your Google account, then authorize the Google APIs Explorer to access your account. If the request is successful, the JSON response shows that job submission request is pending.

4. To view job output, open the Dataproc Jobs page in the Google Cloud console, then click the top (most recent) Job ID. Click "LINE WRAP" to ON to bring lines that exceed the right margin into view.

   ...
   Pi is roughly 3.141804711418047
   ...