Dataproc sets special metadata values for the instances that run in your cluster:
| Metadata key | Value |
|---|---|
dataproc-bucket | Name of the cluster's staging bucket |
dataproc-region | Region of the cluster's endpoint |
dataproc-worker-count | Number of worker nodes in the cluster. The value is 0 for single node clusters. |
dataproc-cluster-name | Name of the cluster |
dataproc-cluster-uuid | UUID of the cluster |
dataproc-role | Instance's role, either Master or Worker |
dataproc-master | Hostname of the first master node. The value is either [CLUSTER_NAME]-m in a standard or single node cluster, or [CLUSTER_NAME]-m-0 in a high-availability cluster, where [CLUSTER_NAME] is the name of your cluster. |
dataproc-master-additional | Comma-separated list of hostnames for the additional master nodes in a high-availability cluster, for example, [CLUSTER_NAME]-m-1,[CLUSTER_NAME]-m-2 in a cluster that has 3 master nodes. |
SPARK_BQ_CONNECTOR_VERSION or SPARK_BQ_CONNECTOR_URL | The version or URL that points to a Spark BigQuery connector version to use in Spark applications, for example, 0.42.1 or gs://spark-lib/bigquery/spark-3.5-bigquery-0.42.1.jar. A default Spark BigQuery connector version is pre-installed in Dataproc 2.1 and later image version clusters. For more information, see Use the Spark BigQuery connector. |
You can use these values to customize the behavior of initialization actions.
You can use the --metadata flag in the
gcloud dataproc clusters create
command to provide your own metadata:
gcloud dataproc clusters create CLUSTER_NAME \ --region=REGION \ --metadata=name1=value1,name2=value2... \ ... other flags ...