Building Windows Server multi-arch images

Building multi-arch images manually

Building multi-arch images manually provides you with the flexibility to build an image that includes any Windows Server versions that you need. To build a multi-arch image manually:

1. Create an LTSC 2019 Docker single-arch image. See details about creating Docker images in Deploying a Windows Server application. For example, us-docker.pkg.dev/my-project/docker-repo/foo:1.0-2019.
2. Create an LTSC 2022 Docker single-arch image. For example, us-docker.pkg.dev/my-project/docker-repo/foo:1.0-2022
3. Create a SAC 20H2 Docker single-arch image. For example, us-docker.pkg.dev/my-project/docker-repo/foo:1.0-20h2.
4. Create a Windows Server VM, for example version 20H2. See the Quickstart using a Windows Server VM.
5. Use RDP to connect to the VM.
6. Open a PowerShell window to run the commands in the next steps.

7. Enable the docker manifest experimental feature. A Docker manifest is a list of images to push to a registry:

   PS C:\> $env:DOCKER_CLI_EXPERIMENTAL = 'enabled'
   

8. Create the multi-arch manifest:

   docker manifest create `
     REGISTRY_REGION-docker.pkg.dev/PROJECT_ID/REPOSITORY/foo:1.0 `
     REGISTRY_REGION-docker.pkg.dev/PROJECT_ID/REPOSITORY/foo:1.0-2019 `
     REGISTRY_REGION-docker.pkg.dev/PROJECT_ID/REPOSITORY/foo:1.0-2022 `
     REGISTRY_REGION-docker.pkg.dev/PROJECT_ID/REPOSITORY/foo:1.0-20h2
   

9. Push the newly created multi-arch image manifest to your Artifact Registry repository:

    docker manifest push `
      REGISTRY_REGION-docker.pkg.dev/PROJECT_ID/REPOSITORY/foo:1.0
   

10. To ensure that your multi-arch image was built and pushed successfully, navigate to REGISTRY_REGION-docker.pkg.dev/PROJECT_ID/REPOSITORY/foo and click on that image. You'll see the 3 images inside:

    • foo:1.0-2019
    • foo:1.0-2022
    • foo:1.0-20h2
    • foo:1.0

Now you can refer to the multi-arch image REGISTRY_REGION-docker.pkg.dev/PROJECT_ID/REPOSITORY/foo:1.0 in your Pod specs. This will let you safely use auto-upgrade for your GKE Windows node pools.

Building multi-arch images using the Cloud Build gke-windows-builder

To ease the effort of the manual build steps, you can use the gke-windows-builder based on the OSS gke-windows-builder. You can use the gke-windows-builder with Cloud Build to build the Windows Server multi-arch images automatically. GKE updates the builder to include new supported Windows SAC and LTSC versions when they are released. Another benefit of using the builder is that you don't have to create your own Windows VM with Powershell to build the images. The Windows VM is replaced by a Docker container that runs the commands for you inside Cloud Build.

To help you understand how the builder works, follow this example to build a "hello world" multi-arch image. These steps can be performed on Linux or Windows servers.

Preparing the environment

To prepare your environment, complete the following steps:

1. Create a workspace directory on your work machine, for example: ~/gke-windows-builder/hello-world.
2. Create or select a project for this tutorial.
3. Make sure that billing is enabled for your project.

4. Enable the Compute Engine, Cloud Build and Artifact Registry APIs for your project. The gke-windows-builder is invoked using Cloud Build, and the resulting multi-arch container images are pushed to Artifact Registry. Compute Engine is required for the builder to create and manage Windows Server VMs.

   gcloud services enable compute.googleapis.com cloudbuild.googleapis.com \
     artifactregistry.googleapis.com cloudbuild.googleapis.com
   

5. Grant the following Identity and Access Management (IAM) roles to your Cloud Build service account by using the Google Cloud CLI:

   1. Set variables:

      export PROJECT=$(gcloud info --format='value(config.project)')
      export MEMBER=$(gcloud projects describe $PROJECT --format 'value(projectNumber)')@cloudbuild.gserviceaccount.com
      

   2. Assign roles. These roles are required for the builder to create the Windows Server VMs, to copy the workspace to a Cloud Storage bucket, to configure the networks to build the Docker image and to push resulting image to Artifact Registry:

      gcloud projects add-iam-policy-binding $PROJECT --member=serviceAccount:$MEMBER --role='roles/compute.instanceAdmin'
      gcloud projects add-iam-policy-binding $PROJECT --member=serviceAccount:$MEMBER --role='roles/iam.serviceAccountUser'
      gcloud projects add-iam-policy-binding $PROJECT --member=serviceAccount:$MEMBER --role='roles/compute.networkViewer'
      gcloud projects add-iam-policy-binding $PROJECT --member=serviceAccount:$MEMBER --role='roles/storage.admin'
      gcloud projects add-iam-policy-binding $PROJECT --member=serviceAccount:$MEMBER --role='roles/artifactregistry.writer'
      

6. Add a firewall rule named allow-winrm-ingress to allow WinRM to connect to Windows Server VMs to run a Docker build:

   gcloud compute firewall-rules create allow-winrm-ingress --allow=tcp:5986 --direction=INGRESS
   

7. Create a Docker repository in Artifact Registry for your project. If you have never used Docker repositories in Artifact Registry before, complete the Quickstart for Docker first. Run this command to create your repository:

   gcloud artifacts repositories create REPOSITORY \
     --repository-format=docker --location=REGISTRY_REGION \
     --description="Docker repository"
   

   Replace the following:

   • REPOSITORY: a name such as windows-multi-arch-images.
   • REGISTRY_REGION: a valid Artifact Registry location.

Creating the hello.exe binary in your workspace

For this tutorial, create a simple "hello world" application, written in Go. The code for the sample app is on GitHub.

1. Clone the repository containing the sample code for this tutorial to your local machine by using the following commands:

    git clone https://github.com/GoogleCloudPlatform/kubernetes-engine-samples
    cd kubernetes-engine-samples/windows/windows-multi-arch
   

2. The hello.go file prints the words "Hello World":

   package main
   
   import "fmt"
   
   func main() {
   	fmt.Println("Hello World!")
   }
   

3. Generate the hello.exe binary:

   GOOS=windows go build hello.go
   

You'll see the hello.exe binary in your workspace.

Creating a Dockerfile and build files in your workspace

In this section you use a Dockerfile to build each single-arch Windows Server image and then use a build file to trigger the Cloud Build. The build combines the single-arch images into a multi-arch image.

1. The Dockerfile is a text document that contains instructions for Docker to build an image. The gke-windows-builder replaces the WINDOWS_VERSION with a specific Windows Server version to build the image for. For example, the builder will run docker build -t multi-arch-helloworld:latest_20h2 --build-arg WINDOWS_VERSION=20H2 . on Windows Server 20H2.

   ARG WINDOWS_VERSION=
   FROM mcr.microsoft.com/windows/servercore:${WINDOWS_VERSION}
   COPY hello.exe /hello.exe
   USER ContainerUser
   ENTRYPOINT ["hello.exe"]

2. In the same directory that contains the Dockerfile, the cloudbuild.yaml file is your build config file. Replace <REPOSITORY> and <REGISTRY_REGION> with the name and region for the Artifact Registry repository that you created in the previous step. At build time, Cloud Build automatically replaces $PROJECT_ID with your project ID.

   timeout: 3600s
   steps:
   - name: 'us-docker.pkg.dev/gke-windows-tools/docker-repo/gke-windows-builder:latest'
     args:
     - --container-image-name
     # Replace <REGISTRY_REGION> and <REPOSITORY>.
     - '<REGISTRY_REGION>-docker.pkg.dev/$PROJECT_ID/<REPOSITORY>/multiarch-helloworld:latest'
     # Specify specific variants of images to be built. Or, remove the following 2 lines to default to all available variants.
     - --versions
     - '20H2,ltsc2019'

Building the image

Now you can build the image and view your logs to verify a successful build.

1. To build the image, run the following command:

   gcloud builds submit --config=cloudbuild.yaml .
   

2. You'll see logs like the following example. The last line in the log shows that the build succeeded:

   Creating temporary tarball archive of 2 file(s) totalling 492 bytes before compression.
   Uploading tarball of [.] to [gs://PROJECT_ID_cloudbuild/source/1600082502.509759-b949721a922d462c94a75da9be9f1181.tgz]
   Created [https://cloudbuild.googleapis.com/v1/projects/PROJECT_ID/builds/ec333452-1301-47e8-90e2-716aeb2f5650].
   Logs are available at [https://console.cloud.google.com/cloud-build/builds/ec333452-1301-47e8-90e2-716aeb2f5650?project=840737568665].
   ------------------------ REMOTE BUILD OUTPUT---------------------------------------
   ...
   ...
   
   Created manifest list REGISTRY_REGION-docker.pkg.dev/PROJECT_ID/REPOSITORY/multiarch-helloworld:latest
   sha256:3ecbbc9f5144f358f81f7c7f1a7e28f069c98423d59c40eaff72bf184af0be02
   2020/09/14 11:34:25 Instance: 35.184.178.49 shut down successfully
   PUSH
   DONE
   -----------------------------------------------------------------------------------
   
   ID                                    CREATE_TIME                DURATION  SOURCE                                                                                      IMAGES  STATUS
   ec333452-1301-47e8-90e2-716aeb2f5650  2020-09-14T11:21:43+00:00  12M43S    gs://PROJECT_ID_cloudbuild/source/1600082502.509759-b949721a922d462c94a75da9be9f1181.tgz  -                 SUCCESS
   

You've just built the image using the build config file and pushed the image to Artifact Registry at REGISTRY_REGION-docker.pkg.dev/PROJECT_ID/REPOSITORY/multiarch-helloworld:latest.

Deploying the image

To deploy the multi-arch Windows image onto a cluster, see Deploying a Windows Server application to learn how to deploy the image.

Advanced gke-windows-builder usage

You can customize the behavior of the gke-windows-builder by adding flags to the args section of the cloudbuild.yaml build config file. Some flags for common behaviors are described in this section, but this is not an exhaustive list; to see the full list of flags that gke-windows-builder supports, run the following command on a Linux server or in Cloud Shell:

docker run -it us-docker.pkg.dev/gke-windows-tools/docker-repo/gke-windows-builder:latest --help

To speed up your builds you can use a larger machine type for the Windows instances:

  - --machineType
  - 'n1-standard-8'

Instead of building the image for all Windows versions that GKE supports, you can choose specific Windows Server versions to build for by using the --versions flag:

  - --versions
  - '20H2,ltsc2019'

If your workspace has many files, your image build will be more reliable if you configure the builder to copy the workspace via Cloud Storage rather than WinRM. Create a bucket in your project, such as gs://{your project}_builder, then set the --workspace-bucket flag:

  - --workspace-bucket
  - '{your project}_builder'

To run the Windows builder instances in a Shared VPC service project, use these flags that control the instance's network setup:

  - --subnetwork-project
  - 'shared-vpc-host-project'
  - --subnetwork
  - 'host-project-subnet-shared-with-service-project'