Migrate containers that were deployed on VMs during VM creation

The container startup agent in Compute Engine is deprecated. This agent lets you deploy containers on Compute Engine instances when you create VMs.

This document describes how to migrate existing containers that the startup agent created on your VMs or managed instance groups (MIGs) to other Google Cloud services.

Based on your requirements, choose one of the following options to migrate the containers that were deployed on VMs using the deprecated method:

• If you want to continue running containers on individual VMs and MIGs, use startup scripts or cloud-init.
• If you have stateless container applications and small to medium jobs, use Cloud Run.
• If your container is a batch job that has a definite end state and requires additional computing resources, use Batch.
• If you need advanced control and scalability or if you can't meet your requirements with the other options, use GKE on Google Cloud.

For more use cases and alternative solutions, see Compare the container deployment options.

Deprecated options for configuring containers on VMs

When you configure a container on a VM during VM creation, Compute Engine uses the container startup agent to generate the gce-container-declaration metadata key for the instance. This key stores the container information. Because the container startup agent is deprecated, you can no longer store container information in the gce-container-declaration metadata key.

The following options for deploying containers on a VM or MIG use the container startup agent to configure containers on the VM.

Identify instances that use the deprecated container metadata

To determine whether any instances in your project use the deprecated container metadata, run the following Google Cloud CLI command that lists instances with the gce-container-declaration metadata key and value:

gcloud compute instances list --filter="metadata.items.key:gce-container-declaration"

This command provides a list of all VM instances in your configured project that contain the gce-container-declaration metadata key. The metadata key uniquely identifies VMs that are in scope of the deprecation. If you are using multiple projects, run this command across all of the active projects.

If you have a specific instance that you want to validate, run the following Google Cloud CLI command:

  gcloud compute instances describe VM_NAME --format="(metadata.items)"

Replace VM_NAME with the name of the VM instance that you want to validate.

For more information about viewing metadata, see View and query metadata.

Compare the container deployment options

The following table summarizes the use cases for running containers on VMs and recommends alternative container solutions for migrating your workloads:

When you transition from the Compute Engine container startup agent to an alternative solution, consider the following required changes and the potential effort of implementing them:

• VMs running Container-Optimized OS: Take full ownership of VM and container runtime setup, configuration, security, and maintenance, which often involves scripting with startup scripts or cloud-init.
• Cloud Run or Batch: Ensure that your applications are stateless and fit the request-driven or job-based execution model. This approach might involve adapting applications to work with external state management services.
• GKE: Adopt Kubernetes principles, define workloads by using Kubernetes manifest files, and manage cluster resources.

Use startup scripts to deploy containers on VMs

You can run a basic container on a VM using a startup script.

Consider the following points when you use a startup script to configure containers:

• You can use a startup script for basic scenarios. For advanced configuration, consider using cloud-init.
• Because you are creating a new VM with a container configured using the startup script, you must plan for the transition of any workloads deployed on the existing VMs.
• Test and ensure that everything works as expected before you route traffic to your newly created VM with a container.

To create a VM and to deploy a container on a VM or a MIG, do the following:

1. Map current container on VM metadata to startup script command
2. Create a startup script based on the existing metadata configuration
3. Create a VM by using the startup script or Create a MIG by using the startup script.

Map your container metadata to docker run command

You can map the VM metadata or gcloud flags to docker run arguments and include this in your startup script for creating VMs.

Some gcloud flags translate directly to VM metadata. These flags also translate directly to docker run flags. If you have an existing container on a VM, you can read the VM metadata configuration and build a startup script using the equivalent docker run commands.

  # Get your existing VM instance configuration in yaml format
  gcloud compute instances describe VM_NAME --format="(metadata.items)"

The output is similar to the following:

  metadata:
    items:
    - key: gce-container-declaration
      value: |
        spec:
          containers:
          - args:
            - '"hello world!"'
            command:
            - echo
            env:
            - name: ONE
              value: '1'
            image: docker.io/library/busybox
            name: my-instance
            securityContext:
              privileged: true
            stdin: true
            tty: true
          restartPolicy: Always
    - key: google-logging-enabled
      value: 'true'

Use the following table to map existing specification to docker run commands:

Example startup scripts

The following examples show how to include the docker commands in your startup script:

• Example 1: runs a standalone container in a VM based on Container-Optimized OS.

• Example 2: runs a web server container in a VM based on Container-Optimized OS.

#!/bin/bash

# A name for the container
CONTAINER_NAME="my-app-container"

# Stop and remove the container if it exists
docker stop $CONTAINER_NAME || true
docker rm $CONTAINER_NAME || true

# Run docker container from image in docker hub
docker run \
  docker.io/library/busybox \
  echo "hello world!"

#!/bin/bash

# Enable incoming traffic for TCP protocol
sudo iptables -A "INPUT" -p "tcp" -j "ACCEPT"
sudo iptables -A "FORWARD" -p "tcp" -j "ACCEPT"

# A name for the container
CONTAINER_NAME="my-app-container"

# Stop and remove the container if it exists
docker stop $CONTAINER_NAME || true
docker rm $CONTAINER_NAME || true

# Run docker container from image in docker hub
docker run \
  --name=$CONTAINER_NAME \
  --privileged \
  --restart=always \
  --tty \
  --detached \
  --network="host" \
  nginx:1.29.0

Additional configuration options for container deployment

This section describes the additional configuration parameters for deploying containers on your VMs.

For more information about these options, see Configure options to run a container.

Access to Artifact Registry images

If you need access to container images from gcr.io or pkg.dev, use the docker-credential-gcr tool, which is preinstalled in Container-Optimized OS, and configure authentication to Artifact Registry for Docker. Run the following command before you run the container:

  # Set home directory to save docker credentials
  HOME=/home/appuser

  # Configure docker with credentials for gcr.io and pkg.dev
  docker-credential-gcr configure-docker

For more information, see Configure authentication to Artifact Registry for Docker.

Configure logging

We recommend using Cloud Logging by enabling a logging agent on a VM.

Alternatively, if you want to change the logging driver, you can include the --log-driver parameter with your docker run command:

  # Enable Cloud Logging
  --log-driver=gcplogs

For more information, see Using Cloud Logging with Container-Optimized OS

Configure internal firewall

Container-Optimized OS denies incoming traffic by default, so you must add iptables rules to allow that traffic. Note that these commands configure the host operating system's internal firewall. Additionally, you must configure your Virtual Private Cloud firewall to allow that traffic to the new VM

For more information, see Use VPC firewall rules.

  # Enable all incoming traffic for TCP protocol
  iptables -A "INPUT" -p "tcp" -j "ACCEPT"
  iptables -A "FORWARD" -p "tcp" -j "ACCEPT"

  # Enable all incoming traffic for UDP protocol
  iptables -A "INPUT" -p "udp" -j "ACCEPT"
  iptables -A "FORWARD" -p "udp" -j "ACCEPT"

  # Enable all incoming traffic for ICMP protocol
  iptables -A "INPUT" -p "icmp" -j "ACCEPT"
  iptables -A "FORWARD" -p "icmp" -j "ACCEPT"

For more information, see Configuring the host firewall.

Attach volumes to the container

If volumes are attached to the container, the container metadata includes the volumes entry and a volumeMounts array. The name of an entry in volumes corresponds to the name of an entry in volumeMounts, and the other way around. For each volume that you collect, gather the required information either from the volumes or from the volumeMounts entry.

If no volumes are attached to the container, you can skip this section and directly create a VM by using the startup script.

Mount tmpfs file system

  # generic command
  --tmpfs <mount-path>[:opts]

  # for our example
  --tmpfs tmpfs-mount-path

Directory or host-path mounts

  # generic command
  --mount type=bind,source=HOSTPATH,target=MOUNTPATH[,readonly]

  # for our example
  --mount type=bind,source=directory-host-path,target=directory-mount-path,readonly

Mount a persistent disk to the container

Mounting a disk to the container requires additional steps. To mount a disk, first mount it on the VM, and then mount that disk to the container:

1. To mount the disk to the VM, run the following command:

   #!/bin/bash
   
   DISK_DEVICE_NAME="my-persistent-disk" # This name MUST match the 'device-name' in the gcloud --disk flag
   DISK_BY_ID_PATH="/dev/disk/by-id/google-${DISK_DEVICE_NAME}"
   HOST_MOUNT_POINT="/mnt/disks/my-persistent-data" # This is where the disk will be mounted on the VM
   
   # format a disk as an ext4 filesystem, if it doesn't already contain one
   file -sL $DISK_BY_ID_PATH | grep -q filesystem || \
           mkfs.ext4 -m 0 -E lazy_itable_init=0,lazy_journal_init=0,discard $DISK_BY_ID_PATH
   
   # create a directory for mounting point
   sudo mkdir -p "${HOST_MOUNT_POINT}"
   
   # mount a disk to the VM
   sudo mount -o defaults,discard "${DISK_BY_ID_PATH}" "${HOST_MOUNT_POINT}"
   

2. After you mount the disk to the VM, add the --mount flag with the docker run command to mount the disk to the container:

   --mount type=bind,source="${HOST_MOUNT_POINT}",target=/disk,readonly
   

Create a VM by using the startup script

After creating a startup script with your container configuration, use this startup script to create a VM.

For more information about using startup scripts, see Using startup scripts on Linux VMs.

    gcloud compute instances create VM_NAME \
        --zone=ZONE \
        [--image=IMAGE | --image-family=IMAGE_FAMILY] \
        --image-project=IMAGE_PROJECT \
        IMAGE_FLAG \
        --machine-type=MACHINE_TYPE \
        --metadata-from-file=startup-script=STARTUP_SCRIPT_FILE

  # Configuration Variables
  PROJECT_ID="my_project_id"
  REGION="us-central1"
  ZONE="us-central1-c"
  INSTANCE_NAME="instance-startup-script"
  STARTUP_SCRIPT_PATH="./startup.sh"

  # Disk and Policy Configuration
  RESOURCE_POLICY_NAME="disk-resource-policy"
  DISK_DEVICE_NAME="my-persistent-disk"

  # Derived Variables (do not change)
  RESOURCE_POLICY_PATH="projects/${PROJECT_ID}/regions/${REGION}/resourcePolicies/${RESOURCE_POLICY_NAME}"

  # Create the VM instance, attach the boot disk with a resource policy,
  # Attach a secondary read-only disk, and runs a startup script.
  gcloud compute instances create "${INSTANCE_NAME}" \
    --project="${PROJECT_ID}" \
    --zone="${ZONE}" \
    --machine-type="e2-medium" \
    --scopes=devstorage.read_only,logging.write,monitoring.write,servicecontrol \
    --image-family="cos-stable" \
    --image-project="cos-cloud" \
    --metadata-from-file=startup-script="${STARTUP_SCRIPT_PATH}" \
    --create-disk=boot=yes,disk-resource-policy="${RESOURCE_POLICY_PATH}" \
    --disk=name="${DISK_DEVICE_NAME}",mode=ro
 

resource "google_compute_instance" "my_container_vm" {
  name         = "my-container-vm-startup"
  machine_type = "e2-medium"
  zone         = "us-central1-a"

  boot_disk {
    initialize_params {
      image = "cos-cloud/cos-stable"
    }
  }
  attached_disk {
    device_name = "my-persistent-disk"
    mode = "READ_WRITE"
  }
  network_interface {
    network = "default"
  }

  metadata_startup_script = <<EOF
#!/bin/bash
docker pull docker.io/nginx:latest
docker rm -f my-nginx || true
docker run -d -p 80:80 --name my-nginx docker.io/nginx:latest
EOF
}

Create a MIG by using the startup script

After creating an instance template using the startup script, use one of the following methods to create a MIG.

For more information about creating MIGs, see Create a managed instance group.

  # Create the persistent disk
  gcloud compute disks create my-persistent-disk \
    --region=us-central1 \
    --replica-zones=us-central1-a,us-central1-b \
    --size=20GB

  # Create the instance template that uses a startup script
  gcloud compute instance-templates create startup-template \
      --machine-type=e2-medium \
      --image-family=cos-stable \
      --image-project=cos-cloud \
      --disk=name=my-persistent-disk,device-name=my-persistent-disk,mode=rw,boot=no \
      --metadata-from-file=startup-script=startup_script.sh

  # Create the managed instance group
    gcloud compute instance-groups managed create startup-mig \
      --template=startup-template \
      --size=2 \
      --zone=us-central1-a

resource "google_compute_instance_template" "startup_template" {
name_prefix = "startup-template-"
machine_type = "e2-medium"
disk {
  source_image = "cos-cloud/cos-stable"
  auto_delete  = true
  boot         = true
}
disk {
  source_image = ""
  auto_delete  = false
  boot = false
  device_name = "my-persistent-disk"
  disk_size_gb = 20
}
network_interface {
  network = "default"
}
metadata_startup_script = <<EOF
  #!/bin/bash
  docker pull docker.io/nginx:latest
  docker rm -f my-nginx || true
  docker run -d -p 80:80 --name my-nginx docker.io/nginx:latest
  EOF
lifecycle {
  create_before_destroy = true
}
}
resource "google_compute_region_disk" "my_persistent_disk" {
name = "my-persistent-disk"
region = "us-central1"
replica_zones = ["us-central1-a", "us-central1-b"]
size = 20
}

resource "google_compute_instance_group_manager" "startup_mig" {
name               = "startup-mig"
base_instance_name = "startup-vm"
zone               = "us-central1-a"
version {
  instance_template = google_compute_instance_template.startup_template.id
}
target_size = 2
}

Shut down a container VM by using a shutdown script

You can set up a shutdown script to gracefully shut down the container. For example, to stop a Docker container, add a shutdown script similar to the following:

     #!/bin/sh
     docker stop my_container
   

Test and clean up

After successful creation of a VM or a MIG, validate that your application is running on the container and working as expected. To fix any issues, see Troubleshooting.

If the application is running successfully on your new VMs created using the startup script, you can delete the VMs and MIGs that use the deprecated method of deploying containers.

Troubleshooting

For troubleshooting issues that might occur during migration of containers, see the following documents:

• Cloud Logging overview
• Using Cloud Logging with Container-Optimized OS
• Troubleshooting the Docker daemon
• Troubleshoot and diagnose
• Troubleshoot Terraform
• Troubleshooting when running a basic web server
• Building internet connectivity for private VMs

Use cloud-init with Container-Optimized OS

You can use cloud-init, an industry-standard and cross-platform solution, to deploy containers on VMs running Container-Optimized OS. This tool lets you run custom configuration during the VM creation or startup. For more information, see Using cloud-init with the Cloud config format.

Use managed services for container deployment

This section describes the managed services provided by Google Cloud that you can use to deploy containers.

Cloud Run

Cloud Run is a good option for stateless container applications and small to medium jobs.

Key features of Cloud Run include the following:

• You can choose to only allocate CPUs during request processing, or always allocate CPUs.
• You can run a stateless container application or execute a job as one-off, on a schedule, or as part of a workflow.
• You can configure timeouts for each request or task.
• It's highly scalable and secure.
• It has integrated load balancing and autoscaling.

For more information about deploying containers on Cloud Run, see Deploying container images to Cloud Run

Batch

Batch is a fully managed service that lets you schedule, queue, and execute batch processing workloads on Google Cloud resources. It's designed for running batch-style, parallelizable workloads, including those packaged in containers.

For more information about deploying containers on Batch, see the following documents:

• Get started
• Create and run a basic job

Google Kubernetes Engine

If you are running complex applications, microservices, continuous operation and need fine-grained control and scalability, Google Kubernetes Engine (GKE) is the offering that is best suited. For more information about deploying containers on GKE. see the following documents:

• GKE overview
• Quickstart: Deploy an app to a GKE cluster
• Deploying a containerized web application

Get support

If you have any questions about the migration process or if you need assistance, review the FAQ or contact Google Cloud Support.