Stay organized with collections
Save and categorize content based on your preferences.
Spark Engine 3.5
Component
3.5-dataproc-22
3.5-dataproc-17
Apache Spark
3.5.0
3.5.0
Hadoop Libraries
3.3.6
3.3.6
Cloud Storage Connector
3.0.0
3.0.0
Java
11
11
Python
3.8.5
3.8.5
Conda
4.9.2
4.9.2
R
4.3.0
4.3.0
Spark Engine 3.1(Deprecated)
Component
3.1-dataproc-17
3.1-dataproc-16
3.1-dataproc-15
3.1-dataproc-14
Apache Spark
3.1.3
3.1.3
3.1.3
3.1.3
Hadoop Libraries
3.2.3
3.2.3
3.2.3
3.2.3
Cloud Storage Connector
hadoop3-2.2.13
hadoop3-2.2.13
hadoop3-2.2.11
hadoop3-2.2.11
Java
8
8
8
8
Python
3.8.5
3.8.5
3.8.5
3.8.5
Conda
4.9.2
23.5.0
4.9.2
4.9.2
R
4.3.0
4.3.0
4.2.3
4.2.3
Spark Engine 2.4(Deprecated)
Spark 2.4 has reached EOL for support in DPGKE. The public image continues to
be available with no further support.
Component
2.4-dataproc-17
2.4-dataproc-16
2.4-dataproc-15
2.4-dataproc-14
Apache Spark
2.4.8
2.4.8
2.4.8
2.4.8
Hadoop Libraries
2.10.2
2.10.2
2.10.2
2.10.2
Cloud Storage Connector
hadoop2-2.1.9
hadoop2-2.1.9
hadoop2-2.1.9
hadoop2-2.1.9
Java
8
8
8
8
Python
3.7.4
3.7.4
3.7.4
3.7.4
Conda
4.7.12
22.11.1
22.11.1
22.1.0
R
3.6.3
3.6.3
3.6.3
3.6.3
Spark version formats on Dataproc on GKE
A fully qualified Spark engine release version is expressed as: 3.1-dataproc-[NUMBER] or
3.5-dataproc-[NUMBER], for example, 3.1-dataproc-17 or 3.5-dataproc-17.
Spark version formats can also be expressed in alias form, as shown in the
following examples:
3 - Most recent version of Spark engine with a Spark major version of 3.
3.5 - Most recent version of Spark engine with a Spark major.minor version of 3.5.
dataproc-2.2 - Most recent version of Spark engine that is compatible with
Dataproc on Compute Engine 2.2 images.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-25 UTC."],[[["\u003cp\u003eSpark Engine 3.5 is the latest version, featuring Apache Spark 3.5.0, Hadoop Libraries 3.3.6, and Java 11, among other components.\u003c/p\u003e\n"],["\u003cp\u003eSpark Engine 3.1 is deprecated but includes Apache Spark 3.1.3, Hadoop Libraries 3.2.3, and Java 8 across all subversions.\u003c/p\u003e\n"],["\u003cp\u003eSpark Engine 2.4 is deprecated and has reached the End-of-Life (EOL) for support in DPGKE, although the public image remains available without further support.\u003c/p\u003e\n"],["\u003cp\u003eSpark engine release versions are expressed as \u003ccode\u003e3.1-dataproc-[NUMBER]\u003c/code\u003e or \u003ccode\u003e3.5-dataproc-[NUMBER]\u003c/code\u003e, while aliases like \u003ccode\u003e3\u003c/code\u003e, \u003ccode\u003e3.5\u003c/code\u003e, \u003ccode\u003edataproc-2.2\u003c/code\u003e, and \u003ccode\u003elatest\u003c/code\u003e can be used for convenience.\u003c/p\u003e\n"],["\u003cp\u003eVersion aliases are resolved during cluster creation, so using them does not guarantee the same concrete Spark version across multiple clusters.\u003c/p\u003e\n"]]],[],null,["# Dataproc on GKE release versions\n\nSpark Engine 3.5\n----------------\n\nSpark Engine 3.1(Deprecated)\n----------------------------\n\nSpark Engine 2.4(Deprecated)\n----------------------------\n\nSpark 2.4 has reached EOL for support in DPGKE. The public image continues to\nbe available with no further support.\n\nSpark version formats on Dataproc on GKE\n----------------------------------------\n\nA fully qualified Spark engine release version is expressed as: `3.1-dataproc-[NUMBER]` or\n`3.5-dataproc-[NUMBER]`, for example, `3.1-dataproc-17` or `3.5-dataproc-17`.\n\nSpark version formats can also be expressed in alias form, as shown in the\nfollowing examples:\n\n- `3` - Most recent version of Spark engine with a Spark major version of 3.\n- `3.5` - Most recent version of Spark engine with a Spark major.minor version of 3.5.\n- `dataproc-2.2` - Most recent version of Spark engine that is compatible with Dataproc on Compute Engine `2.2` images.\n- `latest` - Most recent version of Spark engine.\n\n| **Note:** Because version aliases are dereferenced at cluster creation time, multiple clusters created using the same alias are not guaranteed to have the same concrete version. Therefore, although aliases are useful to ensure that the cluster is built with the most up-to-date version, they should not be used if exact consistency is required."]]
