Create a MIG with GPU VMs

This document describes how to create a managed instance group (MIG) with virtual machine (VM) instances that have attached GPUs. It describes how to add GPU VMs all at once in a zonal MIG using resize requests and the flex-start provisioning model. To create a MIG resize request that can use the features and services available from Cluster Director, see Create a MIG and a resize request in the AI Hypercomputer documentation.

Using a MIG resize request with the flex-start provisioning model improves obtainability of GPU VMs. In the request, specify the number of GPU VMs and a duration for which you want to run those VMs. Dynamic Workload Scheduler (DWS), the underlying scheduler mechanism, schedules resize requests created across Compute Engine based on requested durations and resource availability. When the resources become available, the MIG automatically creates the VMs.

If your job running on these VMs finishes earlier than the requested duration, then you can delete those VMs. Otherwise, the MIG automatically deletes the VMs at the end of their run duration.

You can also read about other basic scenarios for creating a MIG.

Before you begin

  • To make sure that you have sufficient GPU quota for the resources you're requesting, check your GPU quota.
  • To understand quota consumption, read GPU VMs and preemptible allocation quotas.
  • If you haven't already, then set up authentication. Authentication is the process by which your identity is verified for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine by selecting one of the following options:

    Select the tab for how you plan to use the samples on this page:

    Console

    When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.

    gcloud

    1. After installing the Google Cloud CLI, initialize it by running the following command: 

      gcloud init

      If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

    2. Set a default region and zone.

    REST

    To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.

      After installing the Google Cloud CLI, initialize it by running the following command: 

      gcloud init

      If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

    For more information, see Authenticate for using REST in the Google Cloud authentication documentation.

Limitations

Review the limitations for creating a MIG resize request.

Create a MIG and add GPU VMs all at once

To create a MIG and add GPU VMs all at once in the group, do the following:

  1. Create an instance template, which is required in order to create a MIG. The MIG creates each VM in the group based on the instance template. In the template, specify the configuration for GPU VMs and additional configurations required for using resize requests.

    For more information about instance templates, see About instance templates.

  2. Create a MIG and a resize request to add GPU VMs all at once.

Create an instance template

Create an instance template as described in this section, and then use the template to create a MIG.

Console

  1. Go to the Instance templates page.

    Go to Instance templates

  2. Click Create instance template. The Create an instance template page opens.

  3. In the Name field, enter a name for the instance template.

  4. In the Machine configuration section, do the following:

    1. Click the GPUs tab.

    2. In the GPU type list, select the GPU type.

    3. In the Number of GPUs list, select the number of GPUs.

    4. In the Machine type section, select a machine type.

  5. In the Provisioning model section, do the following:

    1. In the VM provisioning model list, select Flex-start.

    2. To set a run duration for the VMs created through the instance template, in the Enter number of hours field, enter the number of hours. The value must be between one hour (1) and seven days (168).

  6. Optional: To change the default value boot disk type or image, in the Boot disk section, click Change. Then, follow the prompts to change the boot disk.

  7. Click Create.

gcloud

Create an instance template by using the beta instance-templates create command:

gcloud beta compute instance-templates create INSTANCE_TEMPLATE_NAME \
    --image-project=IMAGE_PROJECT \
    --image-family=IMAGE_FAMILY \
    --instance-termination-action=DELETE \
    --instance-template-region=REGION \
    --machine-type=MACHINE_TYPE \
    --maintenance-policy=TERMINATE \
    --max-run-duration=RUN_DURATION \
    --provisioning-model=FLEX_START \
    --reservation-affinity=none

Replace the following:

  • INSTANCE_TEMPLATE_NAME: the name of the instance template.

  • IMAGE_PROJECT: the image project that contains the image—for example, debian-cloud. For more information about the supported image projects, see Public images.

  • IMAGE_FAMILY: an image family. This specifies the most recent, non-deprecated OS image. For example, if you specify debian-12, the latest version in the Debian 12 image family is used. For more information about using image families, see Image families best practices.

  • REGION: the region in which to create the instance template.

  • MACHINE_TYPE: a machine type that supports GPUs. If you specify an N1 machine type, then include the --accelerator flag to specify the number and type of GPUs to attach to your VMs.

  • RUN_DURATION: the duration you want the requested VMs to run. You must format the value as the number of days, hours, minutes, or seconds followed by d, h, m, or s respectively. For example, specify 30m for 30 minutes or 1d2h3m4s for one day, two hours, three minutes, and four seconds. The value must be between 10 minutes and seven days.

REST

Create an instance template by making a POST request to the beta.instanceTemplates.insert method:

POST https://compute.googleapis.com/compute/beta/projects/PROJECT_ID/regions/REGION/instanceTemplates

{
  "name": "INSTANCE_TEMPLATE_NAME",
  "properties": {
    "disks": [
      {
        "boot": true,
        "initializeParams": {
          "sourceImage": "projects/IMAGE_PROJECT/global/images/IMAGE
        }
      }
    ],
    "machineType": "MACHINE_TYPE",
    "networkInterfaces": [
      {
        "network": "global/networks/default"
      }
    ],
    "reservationAffinity": {
      "consumeReservationType": "NO_RESERVATION"
    },
    "scheduling": {
      "instanceTerminationAction": "DELETE",
      "maxRunDuration": {
        "seconds": RUN_DURATION
      },
      "onHostMaintenance": "TERMINATE",
      "provisioningModel": "FLEX_START"
    }
  }
}

Replace the following:

  • PROJECT_ID: the ID of the project in which you want to create the instance template.

  • REGION: the region in which to create the instance template.

  • INSTANCE_TEMPLATE_NAME: the name of the instance template.

  • IMAGE_PROJECT: the image project that contains the image—for example, debian-cloud. For more information about the supported image projects, see Public images.

  • IMAGE: specify one of the following:

    • A specific version of the OS image—for example, debian-12-bookworm-v20240617.

    • An image family, which must be formatted as family/IMAGE_FAMILY. This specifies the most recent, non-deprecated OS image. For example, if you specify family/debian-12, the latest version in the Debian 12 image family is used. For more information about using image families, see Image families best practices.

  • MACHINE_TYPE: a machine type that supports GPUs. If you specify an N1 machine type, then include the guestAccelerators field to specify the number and type of GPUs to attach to your VMs.

  • RUN_DURATION: the duration, in seconds, you want the requested VMs to run before the MIG automatically deletes them. The value must be between 600, which is 600 seconds (10 minutes), and 604800, which is 604,800 seconds (seven days).

Create a MIG and add GPU VMs all at once

Create a MIG as described in this section. To create a resize request in the MIG, you must not configure autoscaling and must turn off repairs.

Console

  1. Go to the Instance groups page.

    Go to Instance groups

  2. Click Create instance group. The Create instance group page opens.

  3. In the Name field, enter a name for the MIG.

  4. Before you select an instance template, you must delete the autoscaling configuration and turn off repairs as follows:

    1. To delete the autoscaling configuration, do the following:
      1. In the Autoscaling section, click the Autoscaling mode list, and then click Delete autoscaling configuration.
      2. In the confirmation dialog, click Delete.
    2. To turn off repairs, in the VM instance lifecycle section, click the Default action on failure list, and then select No action.

  5. Go back to the Instance template field. In the Instance template list, select the instance template that you created in the previous section.

  6. Do one of the following:

    • To create a resize request with the MIG, do the following:
      1. In the Number of instances field, enter the number of VMs that you want to create all at once.
      2. Select the Use resize request to create VMs all at once checkbox.
      3. Optional: To specify a different run duration for the VMs than the one set in the instance template, in the Requested run duration field and Unit lists, specify a duration. The duration must be between one hour and seven days.
    • To create a resize request after you create the MIG, in the Number of instances field, enter 0.

  7. In the Location section, specify whether you want to create a zonal or a regional MIG as follows:

    1. To create a zonal MIG, select Single zone. Or, to create a regional MIG, select Multiple zones.
    2. Select the Region and Zones of the MIG.
    3. If you're creating a regional MIG, then do the following:
      1. In the Target distribution shape field, select Any single zone.
      2. In the dialog that appears, click Disable instance redistribution.

  8. Click Create.

gcloud

  1. Create a zonal MIG using the instance-groups managed create command:

    gcloud compute instance-groups managed create INSTANCE_GROUP_NAME \
   --template=INSTANCE_TEMPLATE_NAME \
   --size=0 \
   --zone=ZONE \
   --default-action-on-vm-failure=do_nothing

  2. In the MIG, create a resize request using the instance-groups managed resize-requests create command. Specify the number of GPU VMs that you want and the duration for which you want to run those VMs.

    gcloud compute instance-groups managed resize-requests create INSTANCE_GROUP_NAME \
   --resize-request=RESIZE_REQUEST_NAME \
   --resize-by=COUNT \
   --zone=ZONE

Replace the following:

  • INSTANCE_GROUP_NAME: the name of the MIG.
  • INSTANCE_TEMPLATE_NAME: the name of the instance template for GPU VMs.
  • ZONE: one of the zones available for Compute Engine.
  • RESIZE_REQUEST_NAME: the name of the resize request.
  • COUNT: the number of VMs to add all at once in the group.

REST

  1. Create a zonal MIG by making a POST request to the instanceGroupManagers.insert method.

    POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instanceGroupManagers

{
 "versions": [
   {
     "instanceTemplate": "global/instanceTemplates/INSTANCE_TEMPLATE_NAME"
   }
 ],
 "name": "INSTANCE_GROUP_NAME",
 "targetSize": 0,
 "instanceLifecyclePolicy": {
   "defaultActionOnFailure": "DO_NOTHING"
 }
}

  2. In the MIG, create a resize request by making a POST request to the instanceGroupManagerResizeRequests.insert method. In the request body, specify the number of GPU VMs that you want to create all at once and the duration that you want to run those VMs.

    POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instanceGroupManagers/INSTANCE_GROUP_NAME/resizeRequests

{
 "name": "RESIZE_REQUEST_NAME",
 "resizeBy": COUNT
}

Replace the following:

  • PROJECT_ID: the ID of the project in which you want to create the MIG.
  • INSTANCE_GROUP_NAME: the name of the MIG.
  • INSTANCE_TEMPLATE_NAME: the name of the instance template for GPU VMs.
  • ZONE: one of the zones available for Compute Engine.
  • RESIZE_REQUEST_NAME: the name of the resize request.
  • COUNT: the number of VMs to add all at once in the group.

The resize request that you create stays in the ACCEPTED state until the MIG creates all the requested GPU VMs. After all GPU VMs are created in the group, the state of the request changes to SUCCEEDED.

What's next