Stay organized with collections
Save and categorize content based on your preferences.
This document provides information about Google Cloud Serverless for Apache Spark autoscaling.
When you submit your Spark workload, Serverless for Apache Spark
can dynamically scale workload resources, such as the number of executors,
to run your workload efficiently. Serverless for Apache Spark autoscaling is the
default behavior, and uses
Spark dynamic resource allocation
to determine whether, how, and when to scale your workload.
Serverless for Apache Spark autoscaling V2
Serverless for Apache Spark autoscaling version 2 (V2) adds features and improvements
to default version 1 (V1) to help you manage Serverless for Apache Spark workloads, improve
workload performance, and save costs:
Asynchronous node downscaling: Autoscaling V2 replaces V1's synchronous
downscaling with asynchronous downscaling. Using asynchronous downscaling,
Serverless for Apache Spark downscales workload resources without waiting for all nodes
to finish shuffle migration. This means that long-tail nodes that scale down slowly
won't block upscaling.
Intelligent scaling down node selection: Autoscaling V2 replaces V1's
random node selection with an intelligent algorithm that identifies the best
nodes to scale down first. This algorithm considers factors such as the
node's shuffle data size and idle time.
Configurable Spark grace decommission and shuffle migration behavior:
Autoscaling V2 lets you use standard Spark properties to configure Spark graceful
decommissioning and shuffle migration. This feature can help
you maintain migration compatibility with your customized Spark properties.
Serverless for Apache Spark autoscaling features
Feature
Serverless for Apache Spark Autoscaling V1
Serverless for Apache Spark Autoscaling V2
Node downscaling
Synchronous
Asynchronous
Node selection for downscaling
Random
Intelligent
Spark graceful decommissioning and shuffle migration
Whether to use dynamic resource allocation, which scales up and down the
number of executors based on the workload.
Setting the value to false disables autoscaling
for the workload. Default: true.
true
spark.dynamicAllocation.initialExecutors
The initial number of executors allocated to the workload. After the
workload starts, autoscaling may change the number of active executors.
Minimum value is 2; maximum value is 2000.
2
spark.dynamicAllocation.minExecutors
The minimum number of executors to scale the workload down to.
Minimum value is 2.
2
spark.dynamicAllocation.maxExecutors
The maximum number of executors to scale the workload up to.
Maximum value is 2000.
1000
spark.dynamicAllocation.executorAllocationRatio
Customizes scaling up of the Spark workload. Accepts a value from
0 to 1. A value of 1.0
provides maximum scale-up capability and helps achieve maximum
parallelism. A value of 0.5 sets scale-up capability and
parallelism at one-half the max value.
0.3
spark.reducer.fetchMigratedShuffle.enabled
When set to true, enables fetching the shuffle output
location from the Spark driver after a fetch fails from an executor that was
decommissioned due to Spark dynamic allocation. This reduces
ExecutorDeadException errors caused by shuffle block migration
from decommissioned executors to live executors, and reduces stage retries
caused by FetchFailedException
errors (see
FetchFailedException caused by
ExecutorDeadException).
This property is available in Serverless for Apache Spark
Spark runtime
versions1.1.12 and later and 2.0.20 and later.
false
Spark dynamic allocation metrics
Spark batch workloads generate the following metrics related to Spark dynamic
resource allocation (for additional information on Spark metrics, see
Monitoring and Instrumentation).
Metric
Description
maximum-needed
The maximum number of executors needed under the current load to satisfy
all running and pending tasks.
running
The number of running executors executing tasks.
Spark dynamic allocation issues and solutions
FetchFailedException caused by ExecutorDeadException
Cause: When Spark dynamic allocation scales down an executor, the shuffle file is migrated to
live executors. However, since the Spark reducer task on an executor fetches shuffle
output from the location set by the Spark driver when the reducer task started, if
a shuffle file is migrated, the reducer can continue to attempt to
fetch shuffle output from a decommissioned executor, causing ExecutorDeadException
and FetchFailedException errors.
Solution: Enable shuffle location refetching by setting
the spark.reducer.fetchMigratedShuffle.enabled to true when you run your
Serverless for Apache Spark batch workload (see
Set Spark batch workload properties).
When this property is enabled, the reducer task refetches the shuffle output
location from the driver after a fetch from a decommissioned executor fails.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-28 UTC."],[[["\u003cp\u003eDataproc Serverless for Spark uses dynamic resource allocation to automatically scale workload resources, such as the number of executors, for efficient processing.\u003c/p\u003e\n"],["\u003cp\u003eAutoscaling V2 is an improved version of the default V1 and offers asynchronous node downscaling, intelligent node selection for downscaling, and configurable Spark graceful decommissioning and shuffle migration behavior.\u003c/p\u003e\n"],["\u003cp\u003eThe \u003ccode\u003espark.dataproc.scaling.version\u003c/code\u003e property controls whether autoscaling V1 or V2 is active, and setting it to \u003ccode\u003e2\u003c/code\u003e enables V2's enhanced features.\u003c/p\u003e\n"],["\u003cp\u003eVarious Spark dynamic allocation properties, such as \u003ccode\u003espark.dynamicAllocation.minExecutors\u003c/code\u003e and \u003ccode\u003espark.dynamicAllocation.maxExecutors\u003c/code\u003e, can be configured to control autoscaling behavior when submitting a batch workload.\u003c/p\u003e\n"],["\u003cp\u003eSetting \u003ccode\u003espark.reducer.fetchMigratedShuffle.enabled\u003c/code\u003e to \u003ccode\u003etrue\u003c/code\u003e mitigates \u003ccode\u003eExecutorDeadException\u003c/code\u003e and \u003ccode\u003eFetchFailedException\u003c/code\u003e errors by allowing reducers to refetch shuffle output locations after an executor is decommissioned.\u003c/p\u003e\n"]]],[],null,["# Serverless for Apache Spark autoscaling\n\nThis document provides information about Google Cloud Serverless for Apache Spark autoscaling.\nWhen you submit your Spark workload, Serverless for Apache Spark\ncan dynamically scale workload resources, such as the number of executors,\nto run your workload efficiently. Serverless for Apache Spark autoscaling is the\ndefault behavior, and uses\n[Spark dynamic resource allocation](https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation)\nto determine whether, how, and when to scale your workload.\n\nServerless for Apache Spark autoscaling V2\n------------------------------------------\n\nServerless for Apache Spark autoscaling version 2 (V2) adds features and improvements\nto default version 1 (V1) to help you manage Serverless for Apache Spark workloads, improve\nworkload performance, and save costs:\n\n- **Asynchronous node downscaling**: Autoscaling V2 replaces V1's synchronous downscaling with asynchronous downscaling. Using asynchronous downscaling, Serverless for Apache Spark downscales workload resources without waiting for all nodes to finish shuffle migration. This means that long-tail nodes that scale down slowly won't block upscaling.\n- **Intelligent scaling down node selection**: Autoscaling V2 replaces V1's random node selection with an intelligent algorithm that identifies the best nodes to scale down first. This algorithm considers factors such as the node's shuffle data size and idle time.\n- **Configurable Spark grace decommission and shuffle migration behavior**: Autoscaling V2 lets you use standard Spark properties to configure Spark graceful decommissioning and shuffle migration. This feature can help you maintain migration compatibility with your customized Spark properties.\n\n| **Note:** Set the `spark.dataproc.scaling.version=2` property to enable Serverless for Apache Spark autoscaling V2 when you [submit a Spark batch workload](/dataproc-serverless/docs/quickstarts/spark-batch#submit_a_spark_batch_workload).\n\n### Serverless for Apache Spark autoscaling features\n\nSpark dynamic allocation properties\n-----------------------------------\n\nThe following table lists\n[Spark Dynamic Allocation](https://spark.apache.org/docs/latest/configuration.html#dynamic-allocation)\nproperties that you can set when you submit a batch workload to control autoscaling (see\n[how to set Spark properties](/dataproc-serverless/docs/concepts/properties)).\n| **Note:** Also see [Spark properties](/dataproc-serverless/docs/concepts/properties), which describes additional properties that you can set when you submit a Serverless for Apache Spark Spark batch workload.\n\nSpark dynamic allocation metrics\n--------------------------------\n\nSpark batch workloads generate the following metrics related to Spark dynamic\nresource allocation (for additional information on Spark metrics, see\n[Monitoring and Instrumentation](https://spark.apache.org/docs/latest/monitoring.html)).\n| **Note:** See [Spark metrics](/dataproc-serverless/docs/concepts/metrics), which describes properties you can set to control the collection of Spark metrics.\n\nSpark dynamic allocation issues and solutions\n---------------------------------------------\n\n- **FetchFailedException caused by ExecutorDeadException**\n\n Cause: When Spark dynamic allocation scales down an executor, the shuffle file is migrated to\n live executors. However, since the Spark reducer task on an executor fetches shuffle\n output from the location set by the Spark driver when the reducer task started, if\n a shuffle file is migrated, the reducer can continue to attempt to\n fetch shuffle output from a decommissioned executor, causing `ExecutorDeadException`\n and `FetchFailedException` errors.\n\n Solution: Enable shuffle location refetching by setting\n the `spark.reducer.fetchMigratedShuffle.enabled` to `true` when you run your\n Serverless for Apache Spark batch workload (see\n [Set Spark batch workload properties](/dataproc-serverless/docs/concepts/properties#set_spark_batch_workload_properties)).\n When this property is enabled, the reducer task refetches the shuffle output\n location from the driver after a fetch from a decommissioned executor fails."]]
