Upgrade overview

This page provides an overview of the upgrade process and version skew information that should help you plan the order in which you upgrade clusters in a multi-cluster environment. For more detailed planning information, including a checklist to help you plan the upgrade, see Upgrade best practices.

Upgrade sequence

In-place upgrades since version 1.7 must always follow a specific upgrade sequence:

  1. Upgrade the admin workstation. We recommend that you do this even if you plan to use the Google Cloud console, Google Cloud CLI, or Terraform to upgrade user clusters.

  2. Upgrade the user clusters, one at a time.

    • In version 1.14 and later, you can optionally upgrade a user cluster's control plane separately from the node pools on the user cluster.

    • In version 1.16 and later, you can optionally skip a minor version when upgrading node pools. For more information, see Skip a version when upgrading node pools.

    After all node pools in a user cluster are at the same version as the user cluster's control plane, the user cluster is completely upgraded.

    An admin cluster can't be at a later minor version than any of the user clusters that it manages. If any of your user clusters are at the same minor version as the admin cluster, then you can't upgrade the admin cluster.

  3. When all the user clusters are at least one minor version later than the admin cluster, you can optionally upgrade the admin cluster.

The version skew and version rules for upgrades have changed in 1.28 and later. For more information, see Version skew.

User cluster upgrades

When upgrading user clusters, you can choose to upgrade the user cluster as a whole (meaning you can upgrade the control plane and all node pools in the cluster), or you can upgrade the user cluster's control plane and leave the node pools at the current version. Which approach you take depends on several factors, such as:

  • The environment (production or non-production) that the cluster is in.
  • The length of your maintenance window.
  • The version of the user cluster.

For example, in a development environment, you might want to keep the process simple and upgrade both the user cluster's control plane and all node pools. But in a production environment with a short maintenance window, you might want to upgrade only the control plane because that takes less time and with high availability (HA) control planes, the control plane upgrade shouldn't disrupt user workloads. When the control plane is at version 1.28 or higher, you can skip a minor version when upgrading node pools.

Upgrading node pools separately from the control plane is supported for Ubuntu and COS node pools, but not for Windows node pools.

Selectively upgrade node pools

In certain situations, you might want to upgrade some, but not all of the node pools in a user cluster. For example, after upgrading the control plane, you could upgrade a node pool that has light traffic or runs your least critical workloads. After you are convinced that your workloads run correctly on the new version, you could upgrade additional node pools, until eventually all the node pools are upgraded. For more information, see Upgrade node pools.

Skip a minor version when upgrading node pools

If your clusters are at version 1.16 or higher, you can skip a minor version when upgrading node pools. Performing a skip-version upgrade halves the time that it would take to sequentially upgrade node pools two versions. Additionally, skip-version upgrades lets you increase the time between upgrades needed to stay on a supported version. Reducing the number of upgrades reduces workload disruptions and verification time. For more information, see Skip a version when upgrading node pools.

Choose a tool to upgrade user clusters

Google Distributed Cloud provides you a choice of tools for upgrading user clusters.

  • The command-line tool gkectl, which you run on your admin workstation. Before the upgrade, you modify your user cluster configuration file to set the target version for the cluster's control plane and optionally, for the node pools. You specify this file on the command line to gkectl.

  • The Google Cloud console, Google Cloud CLI, or Terraform, which you can run from any computer that has network connectivity to the GKE On-Prem API. These standard tools are clients of the GKE On-Prem API, which runs on Google Cloud infrastructure.

    • You can use Terraform for the upgrade only if you created the user cluster using Terraform.

    • If your user cluster was created using gkectl, the cluster must be enrolled in the GKE On-Prem API to use the console or the gcloud CLI for the upgrade. In 1.16 and later, clusters created using gkectl are enrolled in the GKE On-Prem API by default. For clusters created in earlier versions, you can enroll the cluster after it is created.

      Even if you decide to use gkectl for the upgrade, you might want to enroll the cluster in the GKE On-Prem API to get information about the clusters using the console or the gcloud CLI.

The tool that you use depends on how you plan to upgrade user clusters:

  • Upgrade the cluster as a whole: You can use gkectl, the Google Cloud console, the Google Cloud CLI, or Terraform to upgrade a user cluster (the control plane together with all node pools).

  • Upgrade only the control plane: You can use gkectl, the gcloud CLI, or Terraform to upgrade a user cluster's control plane separately from the node pools. The console doesn't support upgrading only the control plane.

  • Selectively upgrade node pools after the control plane is upgraded: You can use gkectl, the gcloud CLI, or Terraform to upgrade specific node pools after the control plane is upgraded.

  • Upgrade the control plane and one or more node pools at the same time: Only gkectl supports this use case.

Admin cluster upgrades

When the control plane and node pools on all user clusters are at least one minor version later than the admin cluster, you can optionally upgrade the admin cluster. Only gkectl supports upgrading admin clusters. The GKE On-Prem API clients don't support upgrading admin clusters.

Version skew

The version skew is the difference in minor versions between an admin cluster and its managed user cluster(s). In the following sections, the user cluster version refers to the version of the control plane and node pools together, as a whole.

Additionally, the version skew is the difference in minor versions between a user cluster's control plane and the node pools on the user cluster.

In a multi-cluster environment, understanding the supported version skew and version rules for upgrades might help you plan the order in which you upgrade clusters.

Admin and user cluster version skew

An admin cluster can manage user clusters that are at different versions. This capability lets you upgrade a fleet of user clusters on a schedule that works for your organization.

1.29 and higher

The version skew is the same as in 1.28. In 1.29, this feature transitioned to the General Availability stage.

In version 1.29 and higher, user clusters can be up to 2 minor versions higher than their admin cluster. For example, if an admin cluster is at 1.28, the user clusters managed by that admin cluster can be at 1.28, 1.29, or 1.30.

In general terms, if 1.n is the admin cluster minor version, then the user clusters can be at 1.n, 1.n+1, or 1.n+2. User clusters at 1.n+2 can't be upgraded to the next minor version until the admin cluster is upgraded at least one minor version.

1.28

In version 1.28, user clusters can be up to 2 minor versions higher than their admin cluster. For example, if an admin cluster is at 1.15, the user clusters managed by that admin cluster can be at 1.15, 1.16, or 1.28. User clusters at 1.28 can't be upgraded to 1.29 until the admin cluster is upgraded to at least 1.16.

1.16 and lower

In version 1.16 and lower, user clusters can only be 1 minor version higher than their admin cluster. For example, if an admin cluster is at 1.15, the user clusters managed by that admin cluster can be at 1.15 or 1.16.

In general terms, if 1.n is the admin cluster minor version, then the user clusters can be at 1.n or 1.n+1. User clusters can't be upgraded to the next minor version until the admin cluster is at the same minor version as the user cluster.

User cluster control plane and node pool version skew

1.29 and higher

The version skew is the same as in 1.28. In 1.29, this feature transitioned to the General Availability stage.

In version 1.29 and higher, a user cluster's control plane can be up to 2 minor versions higher than the node pools in the cluster. For example, if a user cluster's control plane is at 1.30, the node pools in the cluster can be at 1.28, 1.29, or 1.30.

In general terms, if 1.n is the minor version of a user cluster control plane, the node pools in the cluster can be at 1.n, 1.n-1, or 1.n-2. User cluster control planes can't be upgraded to the next minor version until all node pools are at 1.n or 1.n-1.

1.28

In version 1.28, a user cluster's control plane can be up to 2 minor versions higher than the node pools in the cluster. For example, if a user cluster's control plane is at 1.28, the node pools in the cluster can be at 1.15, 1.16, or 1.28. User cluster control planes can't be upgraded to 1.29 until all node pools are at 1.28 or 1.16.

1.16 and lower

In version 1.16 and lower, a user cluster's control plane can only be 1 minor version higher than the node pools in the cluster. For example, if a user cluster's control plane is at 1.16, the node pools in the cluster can be at 1.15 or 1.16.

In general terms, if 1.n is the minor version of a user cluster control plane, the node pools in the cluster can be at 1.n or 1.n-1. User clusters can't be upgraded to the next minor version until all node pools are at the same minor version as the control plane.

Version rules for admin cluster and user cluster control plane upgrades

The version rules for admin clusters and user cluster control plane upgrades are the same. You can upgrade directly to any version that is in the same minor release or the next minor release. For example, you can upgrade from 1.30.0 to 1.30.1, or from 1.29.1 to 1.30.0. The patch versions don't affect the upgrade version rules.

If you are upgrading to a version that is not part of the next minor release, you must upgrade through one version of each minor release between your current version and your target version. Skipping a minor version isn't supported. For example, if you want to upgrade from version 1.28.x to version 1.30.x, you can't upgrade directly. You must first upgrade from 1.28.x to 1.29.x, and then upgrade to 1.30.x.

In general terms, only upgrades from 1.n to 1.n+1 are supported for admin cluster upgrades and user cluster control plane upgrades.

Version rules for node pool upgrades

In version 1.28 and later, you can skip one minor version when upgrading a node pool in a user cluster. For example, if a user cluster control plane is at 1.30 and a node pool is at 1.28, you can skip 1.29 and upgrade the node pool directly to 1.30. The patch versions don't affect the upgrade version rules.

In general terms, if a user cluster control plane is at 1.n, then you can upgrade node pools that are at 1.n-2 directly to 1.n. Skipping one minor version when upgrading node pools might reduce the amount of time than doing two node pool upgrades (to upgrade from 1.n-2 to 1.n-1 then to 1.n). This is another reason why you might prefer to upgrade a user cluster's control plane separately from the node pools that run on the user cluster.

Patch version upgrades

We recommend that you upgrade to the latest patch version whenever possible to ensure your clusters have the latest security fixes. Patch versions don't affect the version skew and upgrade rules. For a given minor version, you can upgrade to any higher patch version. That is, you can upgrade a 1.30.X version cluster to version 1.30.Y as long as Y is greater than X. For example, you can upgrade from 1.29.0 to 1.29.1 and you can upgrade from 1.29.1 to 1.29.3.

What's next

Review Upgrade best practices and create a plan for upgrading your clusters.