Throughput quota

Vertex AI offers two ways to manage throughput for your generative AI models, which lets you balance cost, flexibility, and performance. You can either use a flexible pay-as-you-go model or reserve a dedicated amount of throughput for a fixed price.

Pay-as-you-go

For the default pay-as-you-go model, Vertex AI uses Dynamic Shared Quota, which doesn't have a predefined usage limit. Instead, you get access to a large, shared pool of resources that are dynamically allocated based on real-time availability and demand.

This model allows your workloads to use more resources when they are available. If you receive a resource exhausted (429) error, it means the shared pool is temporarily experiencing high demand from many users at once. You should implement retry mechanisms in your application, as availability can change quickly.

Reserved Capacity

For critical production applications that require consistent performance and predictable costs, you can use Provisioned Throughput. Provisioned Throughput is a fixed-cost subscription that reserves a specific amount of throughput for your models in a chosen location.

What's next