Customize containerd configuration in GKE nodes

This page shows you how to customize the configuration of the containerd container runtime on your Google Kubernetes Engine (GKE) nodes. Before reading this document, ensure that you're familiar with what a container runtime is and why you'd want to customize it.

About containerd configuration in GKE

You can manually configure a set of options in the containerd runtime on GKE nodes that run an operating system like Container-Optimized OS. Customizing the runtime lets you configure special requirements like access to private image registries. To set these options, you create a YAML file called a runtime configuration file and pass the file to GKE when you create or update a cluster.

This method of customizing containerd lets you avoid deploying privileged DaemonSets, which are a security risk. When you provide GKE with a runtime configuration file, GKE recreates your nodes and updates the containerd config.toml file on every node with your configuration. The configuration persists through node termination, upgrades, and recreations.

The runtime configuration file only lets you configure options in containerd. GKE also supports configuring specific kubelet options and low-level Linux kernel options by using a separate file called a node system configuration file. For more details, see Customizing node system configuration.

Limitations

You can't use a runtime configuration file to change containerd settings in Ubuntu node images. Only Container-Optimized OS with containerd is supported. This is the default node image for all GKE clusters.

Available containerd configuration options

The following table describes the options that you can configure using a runtime configuration file:

Runtime configuration file options

Access private image registries with private credentials that you store in Secret Manager.

For instructions, see Access private registries with private CA certificates.

privateRegistryAccessConfig:
  enabled: true
  certificateAuthorityDomainConfig:
  - gcpSecretManagerCertificateConfig:
      secretURI: "SECRET_LOCATION"
    fqdns:
    - "FQDN1"
    - "FQDN2"

This configuration has the following fields:

  • enabled: true: enables private registry configuration. If you set enabled: false, delete any other fields in the privateRegistryAccessConfig item.
  • certificateAuthorityDomainConfig: contains up to five certificate and FQDN definitions.
  • gcpSecretManagerCertificateConfig: contains a certificate stored in Secret Manager and an array of FQDNs.
  • secretURI: the location of the certificate in Secret Manager.
  • fqdns: a list of fully-qualified domain names of private registries. You can also use IPv4 addresses, but we recommend using the FQDN.

Apply containerd configuration to new clusters

This section shows you how to apply a containerd configuration file when you create a new GKE cluster.

Run the following command:

gcloud container clusters create-auto CLUSTER_NAME \
    --location=LOCATION \
    --scopes="cloud-platform" \
    --containerd-config-from-file="PATH_TO_CONFIG_FILE"

Replace the following:

  • CLUSTER_NAME: the name of your new cluster.
  • LOCATION: the Compute Engine location of your new cluster.
  • PATH_TO_CONFIG_FILE: the path to the configuration file that you created, like ~/containerd-configuration.yaml.

You can enable private registry configuration on new Standard clusters by running the gcloud container clusters create command with the same options.

Apply containerd configuration to existing clusters

This section shows you how to apply a containerd configuration to existing clusters and nodes.

Check access scopes

Existing clusters must have the cloud-platform access scope to use this feature. This section shows you how to check your access scopes and update an existing cluster with a new or modified private registry configuration file.

For details about the default access scopes in new clusters, see Access scopes in GKE.

Check Autopilot access scopes

Run the following command:

gcloud container clusters describe CLUSTER_NAME \
    --location=LOCATION \
    --flatten=nodeConfig \
    --format='csv[delimiter="\\n",no-heading](oauthScopes)'

If your cluster doesn't have the https://www.googleapis.com/auth/cloud-platform access scope, create a new cluster with this access scope.

Check Standard access scopes

To check your Standard cluster access scopes, check a node pool:

gcloud container node-pools describe NODE_POOL_NAME \
    --cluster=CLUSTER_NAME \
    --location=LOCATION \
    --flatten=nodeConfig \
    --format='csv[delimiter="\\n",no-heading](oauthScopes)'

Replace NODE_POOL_NAME with the name of the node pool.

If your cluster doesn't have the https://www.googleapis.com/auth/cloud-platform access scope, create a new node pool with the cloud-platform access scope and delete your existing node pool.

Update the cluster to use your configuration file

Run the following command:

gcloud container clusters update CLUSTER_NAME \
    --location=LOCATION \
    --containerd-config-from-file="PATH_TO_CONFIG_FILE"

Recreate nodes in Standard clusters

If your Standard cluster doesn't use automatic upgrades, you must manually recreate your node pools to apply the new configuration. To trigger a manual node recreation, upgrade your cluster to the same GKE version that it already uses.

gcloud container clusters upgrade CLUSTER_NAME \
    --location=LOCATION \
    --cluster-version=VERSION

Replace VERSION with the same GKE patch version that the cluster already uses.

Disable containerd configuration options

To remove your custom configuration, do the following:

  1. Update your configuration file to specify enabled: false in the configuration item that you want to disable and delete any other fields in the item, like in the following example:

    privateRegistryAccessConfig:
  enabled: false
  2. Apply the updated configuration file to your cluster. For instructions, see Apply containerd configuration to existing clusters.