Upgrade a cluster

This document explains how to upgrade clusters in Google Distributed Cloud (software only) for VMware. This document provides the steps for upgrading your admin workstation, user clusters, and admin clusters. The steps for upgrading a user cluster show how to upgrade the control plane and all node pools. If you want to upgrade the user cluster control plane and node pools separately, see Upgrade node pools.

This page is for IT administrators and Operators who manage the lifecycle of the underlying tech infrastructure. To learn more about common roles and example tasks that we reference in Google Cloud content, see Common GKE Enterprise user roles and tasks.

Before you proceed, we recommend that you review the following documentation:

• Upgrade overview
  Among other things, this document describes the supported version skew and version rules for upgrades, which have changed for 1.28 and later.

• Upgrade best practices
  This document provides checklists and best practices for upgrading clusters.

Review your firewall rules

In version 1.29 and later, server-side preflight checks are enabled by default. Server-side preflight checks require additional firewall rules. In Firewall rules for admin clusters, search for "Preflight checks" and make sure all required firewall rules are configured.

With server-side preflight checks, when you upgrade a user cluster using gkectl, the preflight checks are run on the admin cluster instead of locally on the admin workstation. Server-side preflight checks are also run on the admin cluster when you use the Google Cloud console, the Google Cloud CLI, or Terraform to upgrade a cluster.

When you upgrade an admin cluster, Google Distributed Cloud deploys a Kubernetes in Docker (kind) cluster to temporarily host the Kubernetes controllers needed to upgrade the admin cluster. This transient cluster is called a bootstrap cluster. Server-side preflight checks are run on the bootstrap cluster when you upgrade an admin cluster.

Google API and IAM requirements

To upgrade a cluster to version 1.28 and later, you must enable kubernetesmetadata.googleapis.com and grant the kubernetesmetadata.publisher IAM role to the logging-monitoring service account. These changes are required to use Cloud Monitoring.

1. Enable kubernetesmetadata.googleapis.com:

   gcloud services enable --project PROJECT_ID  \
       kubernetesmetadata.googleapis.com
   

   Replace PROJECT_ID with the ID of the fleet host project in which the user cluster is a member. This is the project that you specified when the cluster was created. If you created the cluster using gkectl, this is the project ID in the gkeConnect.projectID field in the cluster configuration file.

2. If your organization has set up an allowlist that lets traffic from Google APIs and other addresses pass through your proxy server, add kubernetesmetadata.googleapis.com to the allowlist.

3. Grant the kubernetesmetadata.publisher role to the logging-monitoring service account:

   gcloud projects add-iam-policy-binding PROJECT_ID \
     --member "serviceAccount:SERVICE_ACCOUNT_EMAIL" \
     --role "roles/kubernetesmetadata.publisher"
   

   Replace SERVICE_ACCOUNT_EMAIL with the email address of your logging-monitoring service account.

IAM requirements for upgrading user clusters

Skip this section if you plan to use gkectl for the user cluster upgrade.

If you want to use the Google Cloud console, the Google Cloud CLI, or Terraform to upgrade a user cluster, and you aren't a project owner, you must be granted the Identity and Access Management role roles/gkeonprem.admin on the Google Cloud project that the cluster was created in. For details on the permissions included in this role, see GKE on-prem roles in the IAM documentation.

To use the console to upgrade the cluster, at a minimum, you also need the following:

• roles/container.viewer. This role lets users view the GKE Clusters page and other container resources in the console. For details about the permissions included in this role, or to grant a role with read/write permissions, see Kubernetes Engine roles in the IAM documentation.

• roles/gkehub.viewer. This role lets users view clusters in the console. For details about the permissions included in this role, or to grant a role with read/write permissions,see GKE Hub roles in the IAM documentation.

Make configuration changes either before or after an upgrade

If you need to make configuration changes to your clusters, do the cluster update either before or after the upgrade. The only change to the cluster configuration for an upgrade should be the version. Depending on the cluster version and type, other configuration changes are either silently ignored or cause the upgrade to fail. For more information see Remove unsupported changes to unblock upgrade.

Check available versions for cluster upgrades

Run the following command to see which versions are available for upgrade:

gkectl version --kubeconfig ADMIN_CLUSTER_KUBECONFIG

The output shows the current version and the versions available for upgrade.

If you plan to use the console, the gcloud CLI or Terraform for the upgrade, it takes about 7 to 14 days after a release for the version to be available in the GKE On-Prem API in all Google Cloud regions. The console lists only the available versions for the user cluster upgrade. The steps for upgrading a user cluster using the gcloud CLI or Terraform include a step to run gcloud container vmware clusters query-version-config to get available versions for the upgrade.

Upgrade your admin workstation

You need to upgrade your admin workstation if you plan to use gkectl to upgrade a user cluster.

If you plan to use the console, the gcloud CLI, or Terraform to upgrade a user cluster, you can skip upgrading the admin workstation for now. But you will need to upgrade the admin workstation when you are ready to upgrade the admin cluster because only gkectl supports admin cluster upgrades.

The way you upgrade your admin workstation depends on how you created it: gkeadm or user-managed.

gcloud storage cp gs://gke-on-prem-release/gkectl/VERSION/gkectl ./
chmod +x gkectl

gcloud storage cp gs://gke-on-prem-release/gke-onprem-bundle/VERSION/gke-onprem-vsphere-VERSION.tgz ./

Upgrade a user cluster

You can use gkectl, the console, the gcloud CLI, or Terraform to upgrade a user cluster. For information on deciding which tool to use, see Choose a tool to upgrade user clusters.

gkectl upgrade cluster \
  --kubeconfig ADMIN_CLUSTER_KUBECONFIG \
  --config USER_CLUSTER_CONFIG \
  --dry-run

Warning  PodEvictionTooLong  96s (x2 over 4m7s)  onprem-user-cluster-controller
Waiting too long(>10m0.00000003s) for (kube-system/coredns-856d6dbfdf-dl6nz) eviction.

gkectl upgrade cluster \
  --kubeconfig ADMIN_CLUSTER_KUBECONFIG \
  --config USER_CLUSTER_CONFIG_FILE \
  --skip-validation-all

Upgrade the admin cluster

After you have upgraded your user clusters, you can upgrade your admin cluster.

Before you begin:

1. If you are upgrading to version 1.13 or higher, you must first register the admin cluster by filling out the gkeConnect section in the admin cluster configuration file. Run the gkectl update cluster command with the configuration file changes.

2. Make sure your gkectl and clusters are the appropriate version for an upgrade, and that you have downloaded the appropriate bundle. The version skew between you admin and user clusters depends on the Google Distributed Cloud version. To make sure you can upgrade your admin cluster, see Admin and user cluster version skew.

3. Make sure the bundlepath field in the admin cluster configuration file matches the path of the bundle to which you want to upgrade.

   If you make any other changes to the fields in the admin cluster configuration file, these changes are ignored during the upgrade. To make those changes take effect, you must first upgrade the cluster, and then run an update cluster command with the configuration file changes to make other changes to the cluster.

Run gkectl upgrade admin

Do the steps in this section on your admin workstation. There are two variations of the gkectl upgrade admin command:

• Asynchronous:
  With the asynchronous variation, the command starts the upgrade and then completes. You don't need to watch the output of the command for the entire duration of the upgrade. Instead, you can periodically check on the upgrade progress by running gkectl list admin and gkectl describe admin. To use the asynchronous variation, include the --async flag in the command.

  Requirements for asynchronous upgrade:

  • Supported only for HA admin clusters with version 1.29 or higher.
  • All user clusters must have Controlplane V2 enabled.

• Synchronous:
  With the synchronous variation, the gkectl upgrade admin command outputs status messages to the admin workstation as the upgrade progresses.

Remove the full bundle

If you downloaded a full bundle, and you have successfully run the gkectl prepare and gkectl upgrade admin commands, you should delete the full bundle to save disk space on the admin workstation. Run the following command to delete the full bundle:

rm /var/lib/gke/bundles/gke-onprem-vsphere-${TARGET_VERSION}-full.tgz

Resuming an admin cluster upgrade

If an admin cluster upgrade is interrupted or fails, the upgrade can be resumed if the admin cluster checkpoint contains the state required to restore the state prior to the interruption.

Warning: Don't repair the admin master with gkectl repair admin-master after a failed upgrade attempt. This will cause the admin cluster to get into a bad state.

Follow these steps:

1. Check if the admin control plane is healthy before you begin the initial upgrade attempt. See Diagnosing cluster issues. As discussed in that topic, run the gkectl diagnose cluster command for the admin cluster.

2. If the admin control plane is unhealthy prior to the initial upgrade attempt, repair the admin control plane with the gkectl repair admin-master command.

3. When you rerun the upgrade command after an upgrade has been interrupted or has failed, use the same bundle and target version as you did in the previous upgrade attempt.

When you rerun the upgrade command, the resumed upgrade recreates admin cluster state from the checkpoint and reruns the entire upgrade. Starting from 1.12.0, if the admin control plane is unhealthy, the upgrade process will directly upgrade to the target version without trying to restore the admin cluster at the source version before proceeding to upgrade.

The upgrade will resume from the point where it failed or exited if the admin cluster checkpoint is available. If the checkpoint is unavailable, the upgrade will fall back to relying on the admin control plane, and therefore the admin control plane must be healthy in order to proceed with the upgrade. After a successful upgrade, the checkpoint is regenerated.

If gkectl exits unexpectedly during an admin cluster upgrade, the kind cluster is not cleaned up. Before you rerun the upgrade command to resume the upgrade, delete the kind cluster:

docker stop gkectl-control-plane && docker rm gkectl-control-plane

After deleting the kind cluster, rerun the upgrade command again.

Roll back an admin workstation after an upgrade

You can roll back the admin workstation to the version used before the upgrade.

During the upgrade, gkeadm records the version before it was upgraded in the output information file. During the rollback, gkeadm uses the version listed to download the older file.

To roll back your admin workstation to the previous version:

gkeadm rollback admin-workstation --config=AW_CONFIG_FILE

You can omit --config=AW_CONFIG_FILE if your admin workstation configuration file is the default admin-ws-config.yaml. Otherwise, replace AW_CONFIG_FILE with the path to the admin workstation configuration file.

The rollback command performs these steps:

1. Downloads the rollback version of gkeadm.
2. Backs up the home directory of the current admin workstation.
3. Creates a new admin workstation using the rollback version of gkeadm.
4. Deletes the original admin workstation.

Install bundle with a different version for upgrade

If you upgrade your workstation, a bundle with a corresponding version is installed there for upgrading your clusters. If you want a different version, follow these steps to install a bundle for TARGET_VERSION, which is the version to which you want to upgrade.

1. To check the current gkectl and cluster versions, run this command. Use the flag --details/-d for more detailed information.

   gkectl version --kubeconfig ADMIN_CLUSTER_KUBECONFIG --details
   

   The output provides information about your cluster versions.

2. Based on the output you get, look for the following issues, and fix them as needed.

   • If the current admin cluster version is more than one minor version lower than the TARGET_VERSION, upgrade all your clusters to be one minor version lower than the TARGET_VERSION.

   • If the gkectl version is lower than 1.11, and you want to upgrade to 1.12.x, you will have to perform multiple upgrades. Upgrade one minor version at a time, until you get to 1.11.x, and then proceed with the instructions in this topic.

   • If the gkectl version is lower than the TARGET_VERSION, upgrade the admin workstation to the TARGET_VERSION.

3. When you have determined that your gkectl and cluster versions are appropriate for an upgrade, download the bundle.

   Check whether the bundle tarball already exists on the admin workstation.

   stat /var/lib/gke/bundles/gke-onprem-vsphere-TARGET_VERSION.tgz

   If the bundle is not on the admin workstation, download it.

   gcloud storage cp gs://gke-on-prem-release/gke-onprem-bundle/TARGET_VERSION/gke-onprem-vsphere-TARGET_VERSION.tgz /var/lib/gke/bundles/
   

4. Install the bundle.

   gkectl prepare --bundle-path /var/lib/gke/bundles/gke-onprem-vsphere-TARGET_VERSION.tgz --kubeconfig ADMIN_CLUSTER_KUBECONFIG
   

   Replace ADMIN_CLUSTER_KUBECONFIG with the path of your kubeconfig file. You can omit this flag if the file is in your current directory and has the name kubeconfig.

5. List available cluster versions, and make sure the target version is included in the available user cluster versions.

   gkectl version --kubeconfig ADMIN_CLUSTER_KUBECONFIG --details

You can now create a user cluster at the target version, or upgrade a user cluster to the target version.

Troubleshooting the upgrade process

If you experience an issue when following the recommended upgrade process, follow these recommendations to resolve them. These suggestions assume that you have begun with a version 1.11.x setup, and are proceeding through the recommended upgrade process.

See also: Troubleshooting cluster creation and upgrade

Troubleshooting a user cluster upgrade issue

Suppose you find an issue with the upgrade version when upgrading a user cluster. You determine from Google Support that the issue will be fixed in an upcoming patch release. You can proceed as follows:

1. Continue using the current version for production.
2. Test the patch release in a non-production cluster when it is released.
3. Upgrade all production user clusters to the patch release version when you are confident.
4. Upgrade the admin cluster to the patch release version.

Troubleshooting an admin cluster upgrade issue

If you encounter an issue when upgrading the admin cluster, you must contact Google Support to resolve the issue with the admin cluster.

In the meantime, with the new upgrade flow, you can still benefit from new user cluster features without being blocked by the admin cluster upgrade, which allows you to reduce the upgrade frequency of the admin cluster if you want. Your upgrade process can proceed as follows:

1. Upgrade production user clusters to 1.12.x.
2. Keep the admin cluster at its earlier version and continue receiving security patches.
3. Test admin cluster upgrade from 1.11.x to 1.12.x in a test environment, and report issues if there are any;
4. If your issue is solved by a 1.12.x patch release, you can then choose to upgrade the production admin cluster to this patch release if desired.

Known issues for recent versions

The following known issues might affect upgrades if you are upgrading from version 1.7 or later.

See also: Known issues

Upgrading the admin workstation might fail if the data disk is nearly full

If you upgrade the admin workstation with the gkectl upgrade admin-workstation command, the upgrade might fail if the data disk is nearly full, because the system attempts to back up the current admin workstation locally while upgrading to a new admin workstation. If you cannot clear sufficient space on the data disk, use the gkectl upgrade admin-workstation command with the additional flag --backup-to-local=false to prevent making a local backup of the current admin workstation.

Disruption for workloads with PodDisruptionBudgets

Upgrading clusters can cause disruption or downtime for workloads that use PodDisruptionBudgets (PDBs).

Nodes fail to complete their upgrade process

If you have PodDisruptionBudget objects configured that are unable to allow any additional disruptions, node upgrades might fail to upgrade to the control plane version after repeated attempts. To prevent this failure, we recommend that you scale up the Deployment or HorizontalPodAutoscaler to allow the node to drain while still respecting the PodDisruptionBudget configuration.

To see all PodDisruptionBudget objects that do not allow any disruptions:

kubectl get poddisruptionbudget --all-namespaces -o jsonpath='{range .items[?(@.status.disruptionsAllowed==0)]}{.metadata.name}/{.metadata.namespace}{"\n"}{end}'

Appendix

About VMware DRS rules enabled in version 1.1.0-gke.6

As of version 1.1.0-gke.6, Google Distributed Cloud automatically creates VMware Distributed Resource Scheduler (DRS) anti-affinity rules for your user cluster's nodes, causing them to be spread across at least three physical hosts in your datacenter. As of version 1.1.0-gke.6, this feature is automatically enabled for new clusters and existing clusters.

Before you upgrade, be sure that your vSphere environment meets the following conditions:

• VMware DRS is enabled. VMware DRS requires vSphere Enterprise Plus license edition. To learn how to enable DRS, see Enabling VMware DRS in a cluster

• The vSphere username provided in your credentials configuration file has the Host.Inventory.EditCluster permission.

• There are at least three physical hosts available.

If your vSphere environment does not meet the preceding conditions, you can still upgrade, but for upgrading a user cluster from 1.3.x to 1.4.x, you need to disable anti-affinity groups. For more information, see this known issue in the Google Distributed Cloud release notes.

About downtime during upgrades

Re-create an information file if missing

If the output information file for your admin workstation is missing, you must re-create this file so you can then proceed with the upgrade. This file was created when you initially created your workstation, and if you have since done an upgrade, it was updated with new information.

The output information file has this format:

Admin workstation version: GKEADM_VERSION
Created using gkeadm version: GKEADM_VERSION
VM name: ADMIN_WS_NAME
IP: ADMIN_WS_IP
SSH key used: FULL_PATH_TO_ADMIN_WS_SSH_KEY
To access your admin workstation:
ssh -i FULL-PATH-TO-ADMIN-WS-SSH-KEY ubuntu@ADMIN-WS-IP

Here is a sample output information file:

Admin workstation version: v1.10.3-gke.49
Created using gkeadm version: v1.10.3-gke.49
VM name: admin-ws-janedoe
IP: 172.16.91.21
SSH key used: /usr/local/google/home/janedoe/.ssh/gke-admin-workstation
Upgraded from (rollback version): v1.10.0-gke.194
To access your admin workstation:
ssh -i /usr/local/google/home/janedoe/.ssh/gke-admin-workstation ubuntu@172.16.91.21

Create the file in an editor, substituting the appropriate parameters. Save the file with a filename that is the same as the VM name in the directory from which gkeadm is run. For example, if the VM name is admin-ws-janedoe, save the file as admin-ws-janedoe.

What's next

• gcloud CLI reference documentation

  • gcloud container vmware clusters
  • gcloud container vmware node-pools
  • gcloud container vmware admin-clusters

• Terraform reference documentation

  • google_gkeonprem_vmware_cluster
  • google_gkeonprem_vmware_node_pool