Configure a private container registry

This page explains how to configure an existing container registry server for Google Distributed Cloud (software only) for VMware.

This page is for Admins, Architects, and Operators who set up, monitor, and manage the tech infrastructure. To learn more about common roles and example tasks that we reference in Google Cloud content, see Common GKE Enterprise user roles and tasks.

Overview

By default during cluster creation or upgrade, Google Distributed Cloud pulls system images from gcr.io/gke-on-prem-release using the component access service account. Optionally, you can provide your own container registry server so that system images are pulled from your private registry server instead.

Google Distributed Cloud doesn't support unsecured container registries. When you start your container registry server, you must provide a certificate and a key. The certificate can be signed by a public certificate authority (CA), or it can be self-signed.

Create a container registry server

To learn how to create a container registry server, see Run an externally-accessible registry in the Docker documentation.

Configure the registry

To use a private container registry, fill in the privateRegistry section in the admin cluster configuration file before creating the cluster. When this section is filled in:

  • When you run the gkectl prepare command before cluster creation or upgrade, the command extracts the images from the tar file that is specified in the bundlePath field in your admin cluster configuration file and pushes the images to your private registry server.

  • During cluster creation or upgrade, the system images are pulled from your private registry server.

Limitations with advanced clusters and the full bundle

There are two Google Distributed Cloud bundles available: full and regular. To determine which bundle is on your admin workstation, check the bundlePath field in your admin cluster configuration file. If the filename ends in -full, the full bundle is on your admin workstation. If the filename doesn't end in -full, the regular bundle is on your admin workstation.

If you created your admin workstation using the gkeadm command, the command creates the admin workstation VM with the full bundle on it, and configures the bundlePath field in the admin cluster configuration file.

If advanced cluster is enabled, there are limitations with using the full bundle with a private registry, as follows:

  • Version 1.31: the full bundle isn't supported with a private registry. To use a private registry on an advanced cluster:

    1. Download the regular size bundle to your admin workstation.
    2. Update the filename in the bundlePath field in the admin cluster configuration file.

  • Version 1.32: using the full bundle is supported, but the gkectl prepare command pulls images from gcr.io/gke-on-prem-release instead of the tar file. The command does, however, push the images to your private registry so that the system images are pulled from your private registry during cluster creation or upgrade.

Verify images are pulled from your registry server

How you verify that images are being pulled from your registry server depends on whether advanced cluster is enabled.

  • If advanced cluster isn't enabled, run the following command:

    kubectl --kubeconfig ADMIN_CLUSTER_KUBECONFIG get pods \
    --all-namespaces -o jsonpath="{.items[*].spec['initContainers', 'containers'][*].image}"

    Replace ADMIN_CLUSTER_KUBECONFIG with the path of the kubeconfig file for your admin cluster.

    The output of this command displays all the images in your cluster. You can verify that all the Google Distributed Cloud images are from your own registry server.

  • If advanced cluster is enabled, do the following steps:

    You can determine whether containerd is pulling images from your local registry by examining the contents of a file called config.toml as shown in the following steps:

    1. Log into a node and examine the contents of the file /etc/containerd/config.toml.

    2. Check the pluginsio.containerd.grpc.v1.cri".registry.mirrors field of the config.toml file to see whether your registry server is listed in the endpoint field.

      The following is an excerpt from an example config.toml file.

      version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
...
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."gcr.io"]
[plugins."io.containerd.grpc.v1.cri".registry.configs."privateregistry2.io".tls]
ca_file = '/etc/containerd/certs.d/privateregistry2.io/ca.crt'
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."gcr.io"]
endpoint = ["http://privateregistry.io", "http://privateregistry2.io"]
...

    3. If your registry mirror appears in the endpoint field, then the node is pulling images from your registry mirror rather than from Artifact Registry.