Sets the dedicated crawl rate for a crawlRateScope. If the dedicated crawl rate was not set, this will enable vertex AI's crawl bot to use the new dedicated crawl rate for crawling. If the dedicated crawl rate was set, vertex AI's crawl bot will try to update the rate to the new value. If the new value is too high, the crawl bot may crawl at a lower rate to avoid overloading the user's website.
HTTP request
POST https://discoveryengine.googleapis.com/v1alpha/{location=projects/*/locations/*}:setDedicatedCrawlRate
Required. The scope of the crawl rate that the user wants to config. Currently, only domain and host name are supported. A domain name example: example.com. A host name example: www.example.com. Please do not include / in the domain or host name.
crawlRate
integer
Optional. The crawl QPS set by the user. It is not guaranteed that Vertex crawl bot will crawl at this QPS. If the crawl rate is too high, the real QPS may be lower than the value set by the user to avoid overloading the user's website.
Crawl type is used to distinguish user-triggered and auto-refresh. User-triggered crawl rate is for deterministic use cases like crawling urls or sitemaps specified by users. Auto-refresh crawl rate is for best effort use cases like refreshing urls periodically.
Enums
CRAWL_TYPE_UNSPECIFIED
The crawl type is unspecified.
USER_TRIGGERED
The crawl type is user-triggered.
AUTO_REFRESH
The crawl type is auto-refresh.
Mode
Whether the rate is explicitly set by users, or set by vertex AI.
Enums
MODE_UNSPECIFIED
The mode is unspecified. If the user does not specify the mode, default to AUTOMATIC.
AUTOMATIC
Vertex AI automatically splits crawlRate into user-triggered and auto-refresh. Users don't need to specify SetDedicatedCrawlRateRequest.CrawlType if the mode is AUTOMATIC.
EXPLICIT
Users explicitly set user-triggered or auto-refresh crawl rate.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-06-27 UTC."],[[["\u003cp\u003eThis document describes how to set a dedicated crawl rate for Vertex AI's crawl bot, allowing users to control the rate at which their websites are crawled.\u003c/p\u003e\n"],["\u003cp\u003eThe \u003ccode\u003esetDedicatedCrawlRate\u003c/code\u003e method, a POST request, requires a location parameter and a request body containing the desired \u003ccode\u003ecrawlRateScope\u003c/code\u003e, \u003ccode\u003ecrawlRate\u003c/code\u003e, \u003ccode\u003ecrawlType\u003c/code\u003e, and \u003ccode\u003emode\u003c/code\u003e.\u003c/p\u003e\n"],["\u003cp\u003eUsers can define the crawl rate scope by domain or hostname and choose between \u003ccode\u003eUSER_TRIGGERED\u003c/code\u003e or \u003ccode\u003eAUTO_REFRESH\u003c/code\u003e \u003ccode\u003ecrawlType\u003c/code\u003e, and if \u003ccode\u003emode\u003c/code\u003e is set to \u003ccode\u003eEXPLICIT\u003c/code\u003e, crawl rate types must be defined.\u003c/p\u003e\n"],["\u003cp\u003eThe \u003ccode\u003ecrawlRate\u003c/code\u003e parameter specifies the desired QPS (queries per second), but the actual crawl rate may be lower to avoid overloading the user's website.\u003c/p\u003e\n"],["\u003cp\u003eAuthorization for this operation requires the \u003ccode\u003ehttps://www.googleapis.com/auth/cloud-platform\u003c/code\u003e OAuth scope.\u003c/p\u003e\n"]]],[],null,["# Method: projects.locations.setDedicatedCrawlRate\n\n- [HTTP request](#body.HTTP_TEMPLATE)\n- [Path parameters](#body.PATH_PARAMETERS)\n- [Request body](#body.request_body)\n - [JSON representation](#body.request_body.SCHEMA_REPRESENTATION)\n- [Response body](#body.response_body)\n- [Authorization scopes](#body.aspect)\n- [CrawlType](#CrawlType)\n- [Mode](#Mode)\n\nSets the dedicated crawl rate for a crawlRateScope. If the dedicated crawl rate was not set, this will enable vertex AI's crawl bot to use the new dedicated crawl rate for crawling. If the dedicated crawl rate was set, vertex AI's crawl bot will try to update the rate to the new value. If the new value is too high, the crawl bot may crawl at a lower rate to avoid overloading the user's website.\n\n### HTTP request\n\n`POST https://discoveryengine.googleapis.com/v1alpha/{location=projects/*/locations/*}:setDedicatedCrawlRate`\n\nThe URL uses [gRPC Transcoding](https://google.aip.dev/127) syntax.\n\n### Path parameters\n\n### Request body\n\nThe request body contains data with the following structure:\n\n### Response body\n\nIf successful, the response body contains an instance of [Operation](/generative-ai-app-builder/docs/reference/rest/Shared.Types/ListOperationsResponse#Operation).\n\n### Authorization scopes\n\nRequires the following OAuth scope:\n\n- `https://www.googleapis.com/auth/cloud-platform`\n\nFor more information, see the [Authentication Overview](/docs/authentication#authorization-gcp).\n\nCrawlType\n---------\n\nCrawl type is used to distinguish user-triggered and auto-refresh. User-triggered crawl rate is for deterministic use cases like crawling urls or sitemaps specified by users. Auto-refresh crawl rate is for best effort use cases like refreshing urls periodically.\n\nMode\n----\n\nWhether the rate is explicitly set by users, or set by vertex AI."]]