Build multi-architecture container images for Dataflow
Stay organized with collections
Save and categorize content based on your preferences.
If you use a custom container in Dataflow, the container must
match the architecture of the worker VMs. This document describes how to create
multi-architecture containers that are compatible with both x86 and Arm VMs.
You can use the Docker CLI or Cloud Build to build
the container image.
Use Docker to build the image
Create a Dockerfile. Use the FROM instruction to specify a
multi-architecture base image.
FROM apache/beam_python3.10_sdk:2.50.0
# Make your customizations here, for example:
ENV FOO=/bar
COPY path/to/myfile ./
Install the
Buildx
tool. To check whether the tool is installed, run the following command:
dockerbuildxversion
Run the following command to create a builder instance that uses the
docker-container driver. This driver is required to build
multi-architecture images.
dockerbuildxcreate--driver=docker-container--use
The --use flag sets the new builder instance as the current builder.
Run the following command to configure Docker to authenticate requests to Artifact Registry.
gcloudauthconfigure-dockerREGION-docker.pkg.dev
Replace REGION with the region of the Artifact Registry repository.
Run the following command to build and push the container image to
Artifact Registry:
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-09-04 UTC."],[[["\u003cp\u003eThis document provides instructions on creating multi-architecture container images compatible with both x86 and Arm virtual machines (VMs) for use with Dataflow custom containers.\u003c/p\u003e\n"],["\u003cp\u003eYou can build these multi-architecture images using either the Docker CLI with Buildx or Google Cloud Build, and they should specify a multi-architecture base image within their Dockerfile.\u003c/p\u003e\n"],["\u003cp\u003eWhen using the Docker CLI, the \u003ccode\u003edocker buildx\u003c/code\u003e tool is necessary, employing the \u003ccode\u003edocker-container\u003c/code\u003e driver to build the multi-architecture images.\u003c/p\u003e\n"],["\u003cp\u003eCloud Build utilizes a \u003ccode\u003edocker_buildx.yaml\u003c/code\u003e configuration file to define the build process, which includes creating a builder instance and specifying the target platforms.\u003c/p\u003e\n"],["\u003cp\u003eTo verify that a container image is multi-architecture, check its manifest file in Artifact Registry for \u003ccode\u003earm64\u003c/code\u003e and \u003ccode\u003eamd64\u003c/code\u003e entries within the \u003ccode\u003eplatform\u003c/code\u003e section.\u003c/p\u003e\n"]]],[],null,["# Build multi-architecture container images for Dataflow\n\nIf you use a custom container in Dataflow, the container must\nmatch the architecture of the worker VMs. This document describes how to create\nmulti-architecture containers that are compatible with both x86 and Arm VMs.\n\nYou can use the Docker CLI or [Cloud Build](/build/docs/overview) to build\nthe container image.\n\nUse Docker to build the image\n-----------------------------\n\n1. Create a Dockerfile. Use the `FROM` instruction to specify a\n multi-architecture base image.\n\n FROM apache/beam_python3.10_sdk:2.50.0\n\n # Make your customizations here, for example:\n ENV FOO=/bar\n COPY path/to/myfile ./\n\n2. Install the\n [Buildx](https://docs.docker.com/build/architecture/)\n tool. To check whether the tool is installed, run the following command:\n\n docker buildx version\n\n3. Run the following command to create a builder instance that uses the\n `docker-container` driver. This driver is required to build\n multi-architecture images.\n\n docker buildx create --driver=docker-container --use\n\n The `--use` flag sets the new builder instance as the current builder.\n4. Run the following command to configure Docker to authenticate requests to Artifact Registry.\n\n gcloud auth configure-docker \u003cvar translate=\"no\"\u003eREGION\u003c/var\u003e-docker.pkg.dev\n\n Replace \u003cvar translate=\"no\"\u003eREGION\u003c/var\u003e with the region of the Artifact Registry repository.\n5. Run the following command to build and push the container image to\n Artifact Registry:\n\n docker buildx build \\\n --platform=linux/amd64,linux/arm64 \\\n -t \u003cvar translate=\"no\"\u003eREGISTRY\u003c/var\u003e/\u003cvar translate=\"no\"\u003eIMAGE\u003c/var\u003e:\u003cvar translate=\"no\"\u003eTAG\u003c/var\u003e \\\n --push .\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003eREGISTRY\u003c/var\u003e: the Docker repository\n - \u003cvar translate=\"no\"\u003eIMAGE\u003c/var\u003e: the image name\n - \u003cvar translate=\"no\"\u003eTAG\u003c/var\u003e: the image tag\n\nUse Cloud Build to build the image\n----------------------------------\n\n1. Create a Dockerfile. Use the `FROM` instruction to specify a\n multi-architecture base image.\n\n FROM apache/beam_python3.10_sdk:2.50.0\n\n # Make your customizations here, for example:\n ENV FOO=/bar\n COPY path/to/myfile ./\n\n2. In the same directory that contains the Dockerfile, create a file named\n `docker_buildx.yaml`. Paste in the following text:\n\n steps:\n - name: 'docker'\n args: ['buildx', 'create', '--driver', 'docker-container', '--name', 'container', '--use']\n - name: 'docker'\n args: ['buildx', 'build', '--platform', 'linux/amd64,linux/arm64', '-t', '\u003cvar translate=\"no\"\u003eREGISTRY\u003c/var\u003e/\u003cvar translate=\"no\"\u003eIMAGE\u003c/var\u003e:\u003cvar translate=\"no\"\u003eTAG\u003c/var\u003e', '--push', '.']\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003eREGISTRY\u003c/var\u003e: the Docker repository\n - \u003cvar translate=\"no\"\u003eIMAGE\u003c/var\u003e: the image name\n - \u003cvar translate=\"no\"\u003eTAG\u003c/var\u003e: the image tag\n3. To build and push the image, run the\n [`gcloud builds submit` command](/sdk/gcloud/reference/builds/submit):\n\n gcloud builds submit --region=\u003cvar translate=\"no\"\u003eREGION\u003c/var\u003e --config docker_buildx.yaml\n\n Replace \u003cvar translate=\"no\"\u003eREGION\u003c/var\u003e with the region of the Cloud Build\n service to use.\n\nFor more information, see\n[Build and push a Docker image with Cloud Build](/build/docs/build-push-docker-image).\n\nVerify the container image\n--------------------------\n\n1. Open the **Repositories** page in the Google Cloud console.\n\n [Open the Repositories page](https://console.cloud.google.com/artifacts)\n2. Click the repository with the container image.\n\n3. Click the image to see its versions.\n\n4. Click a version.\n\n5. Click **Manifest**.\n\n6. In the manifest file, the `platform` section should have entries for\n `arm64` and `amd64`. For example:\n\n {\n \"schemaVersion\": 2,\n \"mediaType\": \"application/vnd.docker.distribution.manifest.list.v2+json\",\n \"manifests\": [\n {\n \"mediaType\": \"application/vnd.docker.distribution.manifest.v2+json\",\n \"digest\": \"sha256:441d5438885049e2b388523a8cb5b77ea829c3c3f53326fb221fe185abd67f07\",\n \"size\": 3074,\n \"platform\": {\n \"architecture\": \"amd64\",\n \"os\": \"linux\"\n }\n },\n {\n \"mediaType\": \"application/vnd.docker.distribution.manifest.v2+json\",\n \"digest\": \"sha256:d3b98b0f8f3f555f5453c79b240bd2b862d4f52d853fe81bae55f01a663de29c\",\n \"size\": 3073,\n \"platform\": {\n \"architecture\": \"arm64\",\n \"os\": \"linux\"\n }\n }\n ]\n }\n\nWhat's next\n-----------\n\n- [Use Arm VMs on Dataflow](/dataflow/docs/guides/use-arm-vms)\n- [Run a Dataflow job in a custom container](/dataflow/docs/guides/run-custom-container)"]]