Create an AI-optimized Slurm cluster with an A4 machine type

Launch Cloud Shell

In this tutorial you use Cloud Shell which is a shell environment for managing resources hosted on Google Cloud.

Cloud Shell comes preinstalled with the Google Cloud CLI. gcloud CLI provides the primary command-line interface for Google Cloud. Launch Cloud Shell:

1. Go to the Google Cloud console.

   Google Cloud console

2. From the upper-right corner of the console, click the Activate Cloud Shell button:

A Cloud Shell session starts and displays a command-line prompt. You use this shell to run gcloud and Cluster Toolkit commands.

Set environment variables

In Cloud Shell, set the following environment variables to use for the remainder of the tutorial. These environment variable set placeholders values for the following tasks:

• Configures your project with the relevant values to access your reserved a4-highgpu-8g machine .

• Sets up a Cloud Storage bucket to store Cluster Toolkit modules.

Reservation capacity variables

export A4_RESERVATION_PROJECT_ID=A4_RESERVATION_PROJECT_ID
export A4_RESERVATION_NAME=A4_RESERVATION_NAME
export A4_DEPLOYMENT_NAME=A4_DEPLOYMENT_NAME
export A4_REGION=A4_REGION
export A4_ZONE=A4_ZONE
export A4_DEPLOYMENT_FILE_NAME=A4_DEPLOYMENT_FILE_NAME

Replace the following:

• A4_RESERVATION_PROJECT_ID - the Google Cloud project ID that was granted the A4 machine type reservation block.
• A4_RESERVATION_NAME - the name of your GPU reservation block, found in your project. For example, a4high-exr.
• A4_DEPLOYMENT_NAME - a unique name for your Slurm cluster deployment. For example, my-slurm-cluster-deployment.
• A4_REGION - the region that is running the reserved A4 machine reservation block. For example, us-central1.
• A4_ZONE - the zone that contains the reserved machines. This string must contain both the region and zone. For example, us-central1-a.
• A4_ZONE - a unique name for your Slurm blueprint .YAML. If you run through this tutorial more than once, choose a unique deployment name each time.

Storage capacity variables

Create the environment variables for your Cloud Storage bucket.

Cluster Toolkit uses blueprints to define and deploy clusters of VMs. A blueprint defines one or more Terraform modules to provision Cloud infrastructure. This bucket is used to store these blueprints.

export GOOGLE_CLOUD_BUCKET_NAME=GOOGLE_CLOUD_BUCKET_NAME
export GOOGLE_CLOUD_BUCKET_LOCATION=GOOGLE_CLOUD_BUCKET_LOCATION

Replace the following:

• GOOGLE_CLOUD_BUCKET_NAME - the name that you want to use for your Cloud Storage bucket that meets the bucket naming requirements.
• GOOGLE_CLOUD_BUCKET_LOCATION - any Google Cloud region of your choice, where the bucket will be hosted. For example, us-central1.

Switch to your A4-approved project

Run the following command to ensure that you are in the Google Cloud project that has the approved reservation block for the A4 machine type.

gcloud config set project ${A4_RESERVATION_PROJECT_ID}

Create a Cloud Storage bucket

Create the bucket to store your terraform modules, from Cloud Shell using your environment variables, run the following command:

A best practice when working with Terraform is to store the state remotely in a version enabled file. On Google Cloud, you can create a Cloud Storage bucket that has versioning enabled.

gcloud storage buckets create gs://${GOOGLE_CLOUD_BUCKET_NAME} \
    --project=${A4_RESERVATION_PROJECT_ID} \
    --default-storage-class=STANDARD \
    --location=${GOOGLE_CLOUD_BUCKET_LOCATION} \
    --uniform-bucket-level-access

gcloud storage buckets update gs://${GOOGLE_CLOUD_BUCKET_NAME} --versioning

Set up the Cluster Toolkit

To create a Slurm cluster in a Google Cloud project, you can use Cluster Toolkit to handle deploying and provisioning the cluster. Cluster Toolkit is open-source software offered by Google Cloud to simplify the process of deploying workloads on Google Cloud.

Use the following steps to set up Cluster Toolkit.

Clone the Cluster Toolkit GitHub repository

1. In Cloud Shell, clone the GitHub repository:

   git clone https://github.com/GoogleCloudPlatform/cluster-toolkit.git

2. Go to the main working directory:

   cd cluster-toolkit/

Build the Cluster Toolkit binary

1. In Cloud Shell, build the Cluster Toolkit binary from source by running the following command:

   make

2. To verify the build, run the following command:

   To deploy an A4 high accelerator-optimized machine Slurm Cluster, you must use version v1.47.0 or later of the Cluster Toolkit.

   ./gcluster --version

   After building the binary, you are now ready to deploy clusters to run your jobs or workloads.

Create a deployment file

1. In the Cluster Toolkit directory, create a file named

   nano ${A4_DEPLOYMENT_FILE_NAME}.yaml
   

2. Paste the following content into the YAML file.

   ---
   terraform_backend_defaults:
     type: gcs
     configuration:
       bucket: GOOGLE_CLOUD_BUCKET_NAME
   
   vars:
     deployment_name: A4_DEPLOYMENT_FILE_NAME
     project_id: A4_RESERVATION_PROJECT_ID
     region: A4_REGION
     zone: A4_ZONE
     a4h_reservation_name: A4_RESERVATION_NAME
     a4h_cluster_size: 1
   

3. To save and exit the file, press Ctrl+O > Enter > Ctrl+X.

Provision the Slurm cluster

To provision the Slurm cluster, run the following deployment command. This command provisions the Slurm cluster with the following A4 Cluster Toolkit blueprint.

In Cloud Shell, start the cluster creation.

./gcluster deploy -d ${A4_DEPLOYMENT_FILE_NAME}.yaml examples/machine-learning/a4-highgpu-8g/a4high-slurm-blueprint.yaml --auto-approve

Connect to the cluster

After deploying, connect to the Cloud Console to view your cluster.

1. Go to the Compute Engine > VM instances page in the Google Cloud console.

   Go to VM instances

2. Locate the login node (a4high-login-001 or similar).

3. Click SSH to connect.