Stay organized with collections
Save and categorize content based on your preferences.
This page shows you how to run a custom training job on a persistent resource by
using the Google Cloud CLI, Vertex AI SDK for Python, and the REST API.
Normally, when you
create a custom training job, you need to
specify compute resources that the job creates and runs on. After you create a
persistent resource, you can instead configure the custom training job to run on
one or more resource pools of that persistent resource. Running a custom
training job on a persistent resource significantly reduces the job startup time
that's otherwise needed for compute resource creation.
Required roles
To get the permission that
you need to run custom training jobs on a persistent resource,
ask your administrator to grant you the
Vertex AI User (roles/aiplatform.user)
IAM role on your project.
For more information about granting roles, see Manage access to projects, folders, and organizations.
This predefined role contains the
aiplatform.customJobs.create
permission,
which is required to
run custom training jobs on a persistent resource.
Create a training job that runs on a persistent resource
To create a custom training jobs that runs on a persistent resource, make the
following modifications to the standard instructions for
creating a custom training job:
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-28 UTC."],[],[],null,["# Run custom training jobs on a persistent resource\n\nThis page shows you how to run a custom training job on a persistent resource by\nusing the Google Cloud CLI, Vertex AI SDK for Python, and the REST API.\n\nNormally, when you\n[create a custom training job](/vertex-ai/docs/training/create-custom-job), you need to\nspecify compute resources that the job creates and runs on. After you create a\npersistent resource, you can instead configure the custom training job to run on\none or more resource pools of that persistent resource. Running a custom\ntraining job on a persistent resource significantly reduces the job startup time\nthat's otherwise needed for compute resource creation.\n\nRequired roles\n--------------\n\n\nTo get the permission that\nyou need to run custom training jobs on a persistent resource,\n\nask your administrator to grant you the\n\n\n[Vertex AI User](/iam/docs/roles-permissions/aiplatform#aiplatform.user) (`roles/aiplatform.user`)\nIAM role on your project.\n\n\nFor more information about granting roles, see [Manage access to projects, folders, and organizations](/iam/docs/granting-changing-revoking-access).\n\n\nThis predefined role contains the\n` aiplatform.customJobs.create`\npermission,\nwhich is required to\nrun custom training jobs on a persistent resource.\n\n\nYou might also be able to get\nthis permission\nwith [custom roles](/iam/docs/creating-custom-roles) or\nother [predefined roles](/iam/docs/roles-overview#predefined).\n\nCreate a training job that runs on a persistent resource\n--------------------------------------------------------\n\nTo create a custom training jobs that runs on a persistent resource, make the\nfollowing modifications to the standard instructions for\n[creating a custom training job](/vertex-ai/docs/training/create-custom-job): \n\n### gcloud\n\n- Specify the `--persistent-resource-id` flag and set the value to the ID of the persistent resource (\u003cvar translate=\"no\"\u003ePERSISTENT_RESOURCE_ID\u003c/var\u003e) that you want to use.\n- Specify the `--worker-pool-spec` flag such that the values for `machine-type` and `disk-type` matches exactly with a corresponding resource pool from the persistent resource. Specify one `--worker-pool-spec` for single node training and multiple for distributed training.\n- Specify a `replica-count` less than or equal to the `replica-count` or `max-replica-count` of the corresponding resource pool.\n\n### Python\n\nTo learn how to install or update the Vertex AI SDK for Python, see [Install the Vertex AI SDK for Python](/vertex-ai/docs/start/use-vertex-ai-python-sdk).\n\nFor more information, see the\n[Python API reference documentation](/python/docs/reference/aiplatform/latest).\n\n def create_custom_job_on_persistent_resource_sample(\n project: str,\n location: str,\n staging_bucket: str,\n display_name: str,\n container_uri: str,\n persistent_resource_id: str,\n service_account: Optional[str] = None,\n ) -\u003e None:\n aiplatform.init(\n project=project, location=location, staging_bucket=staging_bucket\n )\n\n worker_pool_specs = [{\n \"machine_spec\": {\n \"machine_type\": \"n1-standard-4\",\n \"accelerator_type\": \"NVIDIA_TESLA_K80\",\n \"accelerator_count\": 1,\n },\n \"replica_count\": 1,\n \"container_spec\": {\n \"image_uri\": container_uri,\n \"command\": [],\n \"args\": [],\n },\n }]\n\n custom_job = aiplatform.CustomJob(\n display_name=display_name,\n worker_pool_specs=worker_pool_specs,\n persistent_resource_id=persistent_resource_id,\n )\n\n custom_job.run(service_account=service_account)\n\n### REST\n\n- Specify the `persistent_resource_id` parameter and set the value to the ID of the persistent resource (\u003cvar translate=\"no\"\u003ePERSISTENT_RESOURCE_ID\u003c/var\u003e) that you want to use.\n- Specify the `worker_pool_specs` parameter such that the values of `machine_spec` and `disk_spec` for each resource pool matches exactly with a corresponding resource pool from the persistent resource. Specify one `machine_spec` for single node training and multiple for distributed training.\n- Specify a `replica_count` less than or equal to the `replica_count` or `max_replica_count` of the corresponding resource pool, excluding the replica count of any other jobs running on that resource pool.\n\nWhat's next\n-----------\n\n- [Learn about persistent resource](/vertex-ai/docs/training/persistent-resource-overview).\n- [Create and use a persistent resource](/vertex-ai/docs/training/persistent-resource-create).\n- [Get information about a persistent resource](/vertex-ai/docs/training/persistent-resource-get).\n- [Reboot a persistent resource](/vertex-ai/docs/training/persistent-resource-reboot).\n- [Delete a persistent resource](/vertex-ai/docs/training/persistent-resource-delete)."]]