Quotas and limits

This document lists the quotas and limits that apply to Dataform. For more information on quotas, see Virtual Private Cloud quotas.

A quota restricts how much of a shared Google Cloud resource your Google Cloud project can use, including hardware, software, and network components. Therefore, quotas are a part of a system that does the following:

  • Monitors your use or consumption of Google Cloud products and services.
  • Restricts your consumption of those resources, for reasons that include ensuring fairness and reducing spikes in usage.
  • Maintains configurations that automatically enforce prescribed restrictions.
  • Provides a means to request or make changes to the quota.

In most cases, when a quota is exceeded, the system immediately blocks access to the relevant Google resource, and the task that you're trying to perform fails. In most cases, quotas apply to each Google Cloud project and are shared across all applications and IP addresses that use that Google Cloud project.

There are also limits on Dataform resources. These limits are unrelated to the quota system. Limits cannot be changed unless otherwise stated.

Quota replenishment

Daily quotas are replenished at regular intervals throughout the day, reflecting their intent to guide rate limiting behaviors. Intermittent refresh is also done to avoid long disruptions when quota is exhausted. More quota is typically made available within minutes rather than globally replenished once daily.

Request a quota increase

To increase or decrease most quotas, use the Google Cloud console. For more information, see Request a higher quota.

Cap quota usage

To learn how you can limit usage of a particular resource by specifying a smaller quota than the default, see Cap usage.

Required permissions

To view and update your Dataform quotas in the Google Cloud console, you need the same permissions as for any Google Cloud quota. For more information, see Quota permissions.


For information about troubleshooting errors related to Dataform quotas and limits, see Troubleshooting.

Dataform quotas

Dataform has the following API quotas:

Quotas Limit
Total requests per project, per region 6000/minute/project/region
Compilation requests per project, per region 120/minute/project/region
File access requests per project, per region 120/minute/project/region
Package installation requests per project, per region 120/minute/project/region
Workflow invocation requests per project, per region 60/minute/project/region
Workflow actions per execution 5000

Quotas for services used by Dataform

Dataform uses other Google Cloud services. These services have project-level quotas, which apply when you use Dataform.

Dataform uses the following Google Cloud services that have their own quota limits:

Dataform compilation resources limits

Dataform enforces the following usage limits on compilation resources:

Usage limit Value
Maximum number of actions per repository compilation 5000
Maximum number of dependencies per action in the compiled graph 50
Maximum total serialized data size of the compiled graph of actions 20 MB
Maximum size of NPM dependencies 200 MB

Additionally, Dataform imposes limits on CPU and memory use during compilation. If these limits are exceeded when you compile your code, the compilation fails and Dataform throws a compilation error.

To estimate the compilation resources usage of your repository, see Estimate compilation resources usage of your repository.

Your repository might exceed the usage limits for the following reasons:

  • An infinite loop bug in the repository code.
  • A memory leak bug in the repository code.
  • Large repository size, approximately more than 1000 SQL workflow nodes.
  • Large number of concurrent queries.

For more information about repository size in Dataform, see Overview of repository size.