Stay organized with collections
Save and categorize content based on your preferences.
A custom container is a Docker image that you create to run
your training application. By running your machine learning (ML) training job
in a custom container, you can use ML frameworks, non-ML dependencies,
libraries, and binaries that are not otherwise supported
on Vertex AI.
How training with containers works
Your training application, implemented in the ML framework of your choice,
is the core of the training process.
Create an application that trains your model, using the ML framework
of your choice.
Decide whether to use a custom container. There could be a
prebuilt container that already supports
your dependencies. Otherwise, you need to build a custom container for
your training job. In your custom container, you
pre-install your training application and all its dependencies onto an
image that is used to run your training job.
Store your training and verification data in a source that
Vertex AI can access. To simplify authentication and reduce
latency, store your data in Cloud Storage, Bigtable, or another
Google Cloud storage service in the same Google Cloud project
and region that you are using for Vertex AI. Learn more about
the ways Vertex AI can load your data.
When your application is ready to run, you must build your Docker image and
push it to Artifact Registry or Docker Hub, making sure that
Vertex AI can access your registry.
Vertex AI sets up resources for your job. It allocates one or
more virtual machines (called training instances) based on your job
configuration. You set up a training instance by using the custom container
you specify as part of the WorkerPoolSpec object when
you submit your custom training
job.
Vertex AI runs your Docker image, passing through any
command-line arguments you specify when you create the training job.
When your training job succeeds or encounters an unrecoverable error,
Vertex AI halts all job processes and cleans up the
resources.
Advantages of custom containers
Custom containers let you specify and pre-install all the dependencies
needed for your application.
Faster start-up time. If you use a custom container with your dependencies
pre-installed, you can save the time that your training application
would otherwise take to install dependencies when starting up.
Use the ML framework of your choice. If you can't find an
Vertex AI prebuilt container with the ML framework you
want to use, you can build a custom container with your chosen framework and
use it to run jobs on Vertex AI. For example, you
can use a customer container to train with PyTorch.
Extended support for distributed training. With custom containers, you can
do distributed training using any ML framework.
Use the newest version. You can also use the latest build or minor version
of an ML framework. For example, you can
build a custom container to train with tf-nightly.
Hyperparameter tuning with custom containers
To do hyperparameter tuning on Vertex AI, you
specify goal metrics, along with whether to minimize or maximize each metric.
For example, you might want to maximize your model accuracy, or minimize your
model loss. You also list the hyperparameters you'd like to tune, along with
the range of acceptable values for each hyperparameter. Vertex AI
does multiple trials of your training application, tracking and adjusting the
hyperparameters after each trial. When the hyperparameter tuning job is
complete, Vertex AI reports values for the most effective
configuration of your hyperparameters, and a summary for each trial.
To do hyperparameter tuning with custom containers, you need to make
the following adjustments:
For training with GPUs, your custom container needs to meet a few special
requirements. You must build a different Docker image than what you'd use for
training with CPUs.
Pre-install the CUDA toolkit and cuDNN in your Docker image. The recommended
way to build a custom container with support for GPUs is to use the
nvidia/cuda image as your
base image for your custom container. The nvidia/cuda container image has
matching versions of CUDA toolkit and cuDNN pre-installed, and it helps
you set up the related environment variables correctly.
Install your training application, along with your required ML framework and
other dependencies in your Docker image.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-28 UTC."],[],[],null,["# Custom containers overview\n\nA custom container is a Docker image that you create to run\nyour training application. By running your machine learning (ML) training job\nin a *custom container*, you can use ML frameworks, non-ML dependencies,\nlibraries, and binaries that are not otherwise supported\non Vertex AI.\n\nHow training with containers works\n----------------------------------\n\nYour training application, implemented in the ML framework of your choice,\nis the core of the training process.\n\n1. Create an application that trains your model, using the ML framework\n of your choice.\n\n2. Decide whether to use a custom container. There could be a\n [prebuilt container](/vertex-ai/docs/training/pre-built-containers) that already supports\n your dependencies. Otherwise, you need to [build a custom container for\n your training job](/vertex-ai/docs/training/create-custom-container). In your custom container, you\n pre-install your training application and all its dependencies onto an\n image that is used to run your training job.\n\n3. Store your training and verification data in a source that\n Vertex AI can access. To simplify authentication and reduce\n latency, store your data in Cloud Storage, Bigtable, or another\n Google Cloud storage service in the same Google Cloud project\n and region that you are using for Vertex AI. Learn more about\n [the ways Vertex AI can load your data](/vertex-ai/docs/training/code-requirements#loading-data).\n\n4. When your application is ready to run, you must build your Docker image and\n push it to Artifact Registry or Docker Hub, making sure that\n [Vertex AI can access your registry](/vertex-ai/docs/training/create-custom-container#manage-container-registry-permissions).\n\n5. Submit your custom training job by [creating a custom\n job](/vertex-ai/docs/training/create-custom-job) or [creating a custom training\n pipeline](/vertex-ai/docs/training/create-training-pipeline).\n\n6. Vertex AI sets up resources for your job. It allocates one or\n more virtual machines (called *training instances* ) based on your job\n configuration. You set up a training instance by using the custom container\n you specify as part of the [`WorkerPoolSpec`](/vertex-ai/docs/reference/rest/v1/CustomJobSpec#workerpoolspec) object when\n you [submit your custom training\n job](/vertex-ai/docs/training/create-custom-job).\n\n7. Vertex AI runs your Docker image, passing through any\n command-line arguments you specify when you create the training job.\n\n8. When your training job succeeds or encounters an unrecoverable error,\n Vertex AI halts all job processes and cleans up the\n resources.\n\nAdvantages of custom containers\n-------------------------------\n\nCustom containers let you specify and pre-install all the dependencies\nneeded for your application.\n\n- **Faster start-up time.** If you use a custom container with your dependencies pre-installed, you can save the time that your training application would otherwise take to install dependencies when starting up.\n- **Use the ML framework of your choice.** If you can't find an Vertex AI prebuilt container with the ML framework you want to use, you can build a custom container with your chosen framework and use it to run jobs on Vertex AI. For example, you can use a customer container to train with PyTorch.\n- **Extended support for distributed training.** With custom containers, you can do distributed training using any ML framework.\n- **Use the newest version.** You can also use the latest build or minor version of an ML framework. For example, you can build a custom container to train with `tf-nightly`.\n\nHyperparameter tuning with custom containers\n--------------------------------------------\n\nTo do [hyperparameter tuning](/vertex-ai/docs/training/hyperparameter-tuning-overview) on Vertex AI, you\nspecify goal metrics, along with whether to minimize or maximize each metric.\nFor example, you might want to maximize your model accuracy, or minimize your\nmodel loss. You also list the hyperparameters you'd like to tune, along with\nthe range of acceptable values for each hyperparameter. Vertex AI\ndoes multiple *trials* of your training application, tracking and adjusting the\nhyperparameters after each trial. When the hyperparameter tuning job is\ncomplete, Vertex AI reports values for the most effective\nconfiguration of your hyperparameters, and a summary for each trial.\n\nTo do hyperparameter tuning with custom containers, you need to make\nthe following adjustments:\n\n- In your Dockerfile: install [`cloudml-hypertune`](https://github.com/GoogleCloudPlatform/cloudml-hypertune).\n- In your training code:\n - Use `cloudml-hypertune` to report the results of each trial by calling its helper function, [`report_hyperparameter_tuning_metric`](https://github.com/GoogleCloudPlatform/cloudml-hypertune/blob/master/hypertune/hypertune.py#L49).\n - Add command-line arguments for each hyperparameter, and handle the argument parsing with an argument parser such as [`argparse`](https://docs.python.org/3/library/argparse.html).\n\nSee how to [configure a hyperparameter tuning job that uses custom\ncontainers](/vertex-ai/docs/training/using-hyperparameter-tuning) or learn more about\n[how hyperparameter tuning works on Vertex AI](/vertex-ai/docs/training/hyperparameter-tuning-overview).\n\nGPUs in custom containers\n-------------------------\n\nFor training with GPUs, your custom container needs to meet a few special\nrequirements. You must build a different Docker image than what you'd use for\ntraining with CPUs.\n\n- Pre-install the CUDA toolkit and cuDNN in your Docker image. The recommended way to build a custom container with support for GPUs is to use the [`nvidia/cuda`](https://hub.docker.com/r/nvidia/cuda/) image as your base image for your custom container. The `nvidia/cuda` container image has matching versions of CUDA toolkit and cuDNN pre-installed, and it helps you set up the related environment variables correctly.\n- Install your training application, along with your required ML framework and other dependencies in your Docker image.\n\nSee an [example Dockerfile for training with GPUs](https://github.com/GoogleCloudPlatform/cloudml-samples/blob/master/pytorch/containers/quickstart/mnist/Dockerfile-gpu).\n\nWhat's next\n-----------\n\n- Learn more about how to [create a custom container for your training\n job](/vertex-ai/docs/training/create-custom-container)."]]