Serverless for Apache Spark offers two tiers, a standard tier and a premium (high performance) tier, to allow customers to align performance requirements and feature access with optimal cost.
Premium tier benefits
The premium tier is differentiated from the standard tier by its added capabilities and features. The premium tier provides advanced query acceleration, and accelerated compute, disk and memory often required by complex or latency-sensitive workloads.
Premium tier features and capabilities
The following table highlights the Serverless for Apache Spark exclusive premium tier features and capabilities.
| Feature | Standard | Premium |
|---|---|---|
| Lightning Engine (Query Acceleration) |  |
(Default: Enabled, except for interactive sessions) |
| Interactive Workloads (Sessions) | |
(Default: Enabled) |
| Shuffle-Intensive Jobs (premium Disk) | |
(Default: Disabled) |
| Memory-Intensive Jobs (Extended Memory) | |
(Default: Disabled) |
| GPU Support | |
(Default: Disabled) |
| Batch workloads | ||
| Autoscaling | ||
| Serverless Spark UI | ||
| Lineage, Security and Compliance (VPC-SC, CMEK) |
Premium tier batch workloads and sessions
The following workloads and sessions require the premium tier:
- Lightning Engine: Exclusively available in the premium tier, Lightning Engine is a critical feature that accelerates Spark performance through optimized query execution. It is enabled by default when you select the premium tier for batch workloads by setting the
dataproc.tierproperty topremium. - Interactive sessions: Interactive sessions are automatically treated as a high-value, premium feature. They are initiated through the Dataproc sessions API, and are charged at the premium DCU rate, providing capabilities such as low latency interactivity and state management. For interactive sessions, you enable Lightning Engine by setting the
spark.dataproc.engineproperty tolightningEngine. - GPU support: GPU support is available only in the premium tier. It is disabled by default. You enable it through configuration flags. GPU support uses specialized infrastructure, and is covered by the premium DCU rate with no additional compute cost.
Predictable premium tier pricing model
The premium tier uses a simplified, all-inclusive pricing model for its core capabilities. The premium DCU rate covers access to high-value features such as Lightning Engine, Extended Memory, and the specialized infrastructure needed for shuffle-intensive jobs. When you enable these features through configuration flags, they are included in the base premium DCU rate and incur no additional compute cost. This means you pay a single, predictable premium rate for access to the complete suite of premium tier advanced capabilities.