Serverless for Apache Spark has API quota limits that are enforced at the project and region level. The quotas reset every sixty seconds (one-minute).
The following table lists the specific and default per-project Serverless for Apache Spark API quota types, quota limits, and the methods to which they apply.
| Quota Type | Limit | API Methods or Description |
|---|---|---|
| ClusterOperationRequestsPerMinutePerProjectPerRegion | 200 | CancelOperation (cancelling batch operation quota is shared with cancelling cluster operation quota). |
| BatchOperationRequestsPerMinutePerProjectPerRegion | 200 | CreateBatch, DeleteBatch |
| SessionOperationRequestsPerMinutePerProjectPerRegion | 200 | CreateSession, DeleteSession, TerminateSession |
| DefaultRequestsPerMinutePerProjectPerRegion | 7500 | GetBatch, ListBatches, GetSession, ListSessions |
| ActiveOperationsPerProjectPerRegion | 5000 | Limit on the total number of concurrent active operations of all types in a project in a region. |
Other Google Cloud quotas
Serverless for Apache Spark batches utilize other Google Cloud products. These products have project-level quotas, which include quotas that apply to Serverless for Apache Spark use. Some services are required to use Serverless for Apache Spark, such as Compute Engine and Cloud Storage. Other services, such as BigQuery and Bigtable, can optionally be used with Serverless for Apache Spark.
Required services
The following services, which enforce quota limits, are required to create Serverless for Apache Spark batches.
Compute Engine
Serverless for Apache Spark batches consume the following Compute Engine resource quotas:
| Compute Tier | Quota |
|---|---|
| Standard | CPUS |
| Premium | N2_CPUS |
| Disk Tier | Quota |
| Standard | DISKS_TOTAL_GB |
| Premium | LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY with N2 VM Family |
GPU accelerator | Quota |
| L4 | NVIDIA_L4_GPUS |
| A100 40GB | NVIDIA_A100_GPUS |
| A100 80GB | NVIDIA_A100_80GB_GPUS |
The Compute Engine quotas
are split into regional and global limits. These limits apply to batches you
create. For example, to run a Spark batch with 4 driver cores
(spark.driver.cores=4) and two executors with 4 cores each (spark.executor.cores=4),
you will use 12 virtual CPUs (4 * 3).
This batch usage will count against the regional quota limit of 24 virtual CPUs.
Default batch resources
When you create a batch with default settings, the following Compute Engine resources are used:
| Item | Resources used |
|---|---|
| Virtual CPUs | 12 |
| Virtual Machine (VM) Instances | 3 |
| Persistent disk | 1200 GiB |
Cloud Logging
Serverless for Apache Spark saves batch output and logs in Cloud Logging. The Cloud Logging quota applies to your Serverless for Apache Spark batches.
Optional services
The following services, which have quota limits, can optionally be used with Serverless for Apache Spark batches.
BigQuery
When reading or writing data into BigQuery, the BigQuery quota applies.
Bigtable
When reading or writing data into Bigtable, the Bigtable quota applies.