This document lists the quotas that apply to Dataplex.
Google Cloud uses quotas to help ensure fairness and reduce spikes in resource use and availability. A quota restricts how much of a Google Cloud resource your Google Cloud project can use. Quotas apply to a range of resource types, including hardware, software, and network components. For example, quotas can restrict the number of API calls to a service, the number of load balancers used concurrently by your project, or the number of projects that you can create. Quotas protect the community of Google Cloud users by preventing the overloading of services. Quotas also help you to manage your own Google Cloud resources.
The Cloud Quotas system does the following:
- Monitors your consumption of Google Cloud products and services
- Restricts your consumption of those resources
- Provides a way to request changes to the quota value
In most cases, when you attempt to consume more of a resource than its quota allows, the system blocks access to the resource, and the task that you're trying to perform fails.
Quotas generally apply at the Google Cloud project level. Your use of a resource in one project doesn't affect your available quota in another project. Within a Google Cloud project, quotas are shared across all applications and IP addresses.
There are also system limits on Dataplex resources. System limits can't be changed.
Dataplex enforces quotas on the number of API requests during a specific period of time and the number of resources that can be created in a project. Quotas protect you from unforeseen spikes in usage and overloaded services. Learn more about why quotas are enforced.
Your quota needs might increase the more that you use Dataplex. If a quota is too low to meet your needs, or you expect an upcoming increase in usage, you can proactively request a quota increase.
Quotas for requests
The Dataplex API enforces quotas for Dataplex Catalog resource management and search requests at different levels, regardless of their origination. This includes the requests originating from API calls directly, client libraries, Google Cloud CLI, and Google Cloud console.
The following quotas apply to Dataplex requests:
| Quota | Default |
|---|---|
| Read requests for entries and aspects per project per region per minute | 6000 |
| Write requests for entries and aspects per project per region per minute | 1500 |
| Read requests for aspect types, entry types, and entry groups per project per user per minute | 100 |
| Read requests for aspect types, entry types, and entry groups per project per minute | 200 |
| Write requests for aspect types, entry types, and entry groups per project per user per minute | 10 |
| Write requests for aspect types, entry types, and entry groups per project per minute | 20 |
| Search requests per project per user per minute | 900 |
| Search requests per project per minute | 1200 |
| Search requests per organization per minute | 2400 |
| Search requests per organization per day | 345600 |
| Read requests for metadata jobs per project per region per minute | 400 |
| Read requests for metadata jobs per project per user per region per minute | 200 |
| Write requests for metadata jobs per project per region per minute | 20 |
| Write requests for metadata jobs per project per user per region per minute | 10 |
For every Dataplex request that is listed in the following table, quota is enforced at both the per-project and per-project-per-user levels.
| Quota | Examples |
|---|---|
| Read requests for resource management | Get a lake, list zones |
| Write requests for resource management | Create a lake, delete an asset |
| Get requests for metadata | Get an entity, get a partition |
| List requests for metadata | List entities, list partitions |
| Write requests for metadata | Update an entity, delete a partition |
| Read requests for task and job resources | Get a task, list jobs |
| Write requests for task and job resources | Create a task, cancel a job |
| Set IAM policy requests | Set an IAM policy |
Quotas for resources
Resource quota is enforced at per-project level for the following resource types: lake, zone, asset, on-demand task, recurring task. You can check your current usage of Dataplex quota on the Quotas page in the Google Cloud console.
Request a quota increase
In the Google Cloud console, go to the Quotas page.
Select the quota you want to modify.
Click More actions > Edit quota.
Enter the new quota value and a request description.
Click Next.
Enter the contact details.
Click Submit request.
To adjust most quotas, use the Google Cloud console. For more information, see Request a quota adjustment.
Quota dependencies
Your Dataplex usage might involve other Google Cloud products. For example, data discovery requires invoking Cloud Storage and/or BigQuery APIs to interact with the underlying resources. These products have project-level quotas, which include quotas that apply to Dataplex use.
Quota pages of other products that you might use with Dataplex:
This list is not exhaustive.
Limits
The following limits apply to Dataplex Catalog and search:
| Limit | Value |
|---|---|
| Size of a request for entry and aspects (including schema aspect) | 2 MB |
| Length of an entry ID | 4000 characters |
| Length of a fully qualified name in an entry | 4000 characters |
| Length of the entry source display name in an entry | 500 characters |
| Length of the entry source description in an entry | 2000 characters |
| Length of the entry source resource in an entry | 4000 characters |
| Length of the entry source system in an entry | 64 characters |
| Length of the entry source platform in an entry | 64 characters |
| Length of the entry source labels in an entry | 128 characters for each key and value |
| Size of the JSON content in an aspect (except schema aspect) | 120 KB |
| Total size of an entry | 5 MB |
| Maximum number of aspects in an entry | 10,000 |
| Length of the display name of entry type, aspect type, and entry group | 256 |
| Length of the description of entry type, aspect type, and entry group | 1024 |
| Length of the field name in an aspect type template | 63 |
| Maximum number of required aspect types in an entry type | 20 |
| Maximum number of enum values in an aspect type template | 1024 |
| Maximum number of items in Dataplex search results | 500 |