Google Distributed Cloud (software only) for bare metal 1.29 release notes

This document lists production updates to Google Distributed Cloud (software only) for bare metal (formerly known as Google Distributed Cloud Virtual, previously known as Anthos clusters on bare metal). Check this page periodically for any new announcements.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.

October 24, 2024

Release 1.29.700-gke.113

Google Distributed Cloud for bare metal 1.29.700-gke.113 is now available for download. To upgrade, see Upgrade clusters. Google Distributed Cloud for bare metal 1.29.700-gke.113 runs on Kubernetes 1.29.

After a release, it takes approximately 7 to 14 days for the version to become available for installations or upgrades with the GKE On-Prem API clients: the Google Cloud console, the gcloud CLI, and Terraform.

If you use a third-party storage vendor, check the Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of Google Distributed Cloud for bare metal.

Fixes:

  • Fixed an issue where the control plane VIP might become unavailable because Keepalived didn't check correctly that the VIP is on a node with a responsive HAProxy.
  • Fixed an issue where bmctl restore fails due to etcd containers not starting correctly.
  • Fixed an issue where the registry mirror reachability check fails for a single unreachable registry mirror. Now the reachability check applies to configured registry mirrors only, instead of all registry mirrors.

The following container image security vulnerabilities have been fixed in 1.29.700-gke.113:

Known issues:

For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

October 01, 2024

Release 1.29.600-gke.108

Google Distributed Cloud for bare metal 1.29.600-gke.108 is now available for download. To upgrade, see Upgrade clusters. Google Distributed Cloud for bare metal 1.29.600-gke.108 runs on Kubernetes 1.29.

After a release, it takes approximately 7 to 14 days for the version to become available for installations or upgrades with the GKE On-Prem API clients: the Google Cloud console, the gcloud CLI, and Terraform.

If you use a third-party storage vendor, check the Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of Google Distributed Cloud for bare metal.

Fixes:

  • Fixed Cloud Audit Logging failure due to allowlisting issue with multiple project IDs.

Known issues:

For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

September 12, 2024

Release 1.29.500-gke.163

Google Distributed Cloud for bare metal 1.29.500-gke.163 is now available for download. To upgrade, see Upgrade clusters. Google Distributed Cloud for bare metal 1.29.500-gke.163 runs on Kubernetes v1.29.7-gke.1200.

After a release, it takes approximately 7 to 14 days for the version to become available for installations or upgrades with the GKE On-Prem API clients: the Google Cloud console, the gcloud CLI, and Terraform.

If you use a third-party storage vendor, check the Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of Google Distributed Cloud for bare metal.

The following container image security vulnerabilities have been fixed in 1.29.500-gke.163:

Known issues:

For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

August 08, 2024

Release 1.29.400-gke.86

Google Distributed Cloud for bare metal 1.29.400-gke.86 is now available for download. To upgrade, see Upgrade clusters. Google Distributed Cloud for bare metal 1.29.400-gke.86 runs on Kubernetes 1.29.

If you use a third-party storage vendor, check the Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of Google Distributed Cloud for bare metal.

  • GA: Added support in version 1.29.400-gke.86 and higher for Red Hat Enterprise Linux (RHEL) version 9.2. For more information, see Select your operating system.

Known issues:

For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

July 25, 2024

Release 1.29.300-gke.185

Google Distributed Cloud for bare metal 1.29.300-gke.185 is now available for download. To upgrade, see Upgrade clusters. Google Distributed Cloud for bare metal 1.29.300-gke.185 runs on Kubernetes 1.29.

If you use a third-party storage vendor, check the Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of Google Distributed Cloud for bare metal.

Functionality changes:

  • Updated registry mirror support to allow you to specify a port for host addresses.

  • Updated Kubernetes audit logging to include request and response payloads from the Kubernetes API server for bare metal custom resources, such as Cluster, NodePool, BareMetalMachine, and BareMetalCluster.

Fixes:

The following container image security vulnerabilities have been fixed in 1.29.300-gke.185:

Known issues:

For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

July 03, 2024

Security bulletin (all minor versions)

A remote code execution vulnerability, CVE-2024-6387, was recently discovered in OpenSSH. The vulnerability exploits a race condition that can be used to obtain access to a remote shell, enabling attackers to gain root access. At the time of publication, exploitation is believed to be difficult and take several hours per machine being attacked. We are not aware of any exploitation attempts. This vulnerability has a Critical severity.

For mitigation steps and more details, see the GCP-2024-040 security bulletin.

June 27, 2024

Release 1.29.200-gke.243

Google Distributed Cloud for bare metal 1.29.200-gke.243 is now available for download. To upgrade, see Upgrade clusters. Google Distributed Cloud for bare metal 1.29.200-gke.243 runs on Kubernetes 1.29.

If you use a third-party storage vendor, check the Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of Google Distributed Cloud for bare metal.

Functionality changes:

  • Updated registry mirror support to allow you to specify a port for host addresses.

  • Updated the networking preflight check to verify that either the ip_tables or the nf_tables kernel module is available for loading, instead of being explicitly loaded.

  • Added support for Red Hat Enterprise Linux 8.10 for Google Distributed Cloud software version 1.29.200-gke.243 and higher.

Fixes:

  • Fixed an issue where upgraded clusters didn't get label updates that match the labels applied for newly created clusters, for a given version.

  • Fixed an issue where service accounts created by using the --create-service-accounts flag with the bmctl create config command don't have enough permissions.

The following container image security vulnerabilities have been fixed in 1.29.200-gke.243

Known issues:

For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

May 28, 2024

Security bulletin (all minor versions)

A new vulnerability (CVE-2024-4323) has been discovered in Fluent Bit that could result in remote code execution. Fluent Bit versions 2.0.7 through 3.0.3 are affected.

Google Distributed Cloud software doesn't use a vulnerable version of Fluent Bit and is unaffected.

For more information, see the GCP-2024-031 security bulletin.

May 15, 2024

Release 1.29.100-gke.251

GKE on Bare Metal 1.29.100-gke.251 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.29.100-gke.251 runs on Kubernetes 1.29.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

  • Added new API and IAM role requirements for Cloud Monitoring:

    • You must enable the kubernetesmetadata.googleapis.com API for your project and grant the roles/kubernetesmetadata.publisher IAM role to the Logging and Monitoring service account (anthos-baremetal-cloud-ops, when created automatically). Clusters use this API as an endpoint to send Kubernetes metadata to Google Cloud. The metadata is vital for cluster monitoring, debugging, and recovery. If you install your clusters behind a proxy, add kubernetesmetadata.googleapis.com to the list of allowed connections.

    • Due to changes in the way service accounts are checked, you must also grant the following IAM roles to the Logging and Monitoring service account:

      • roles/monitoring.viewer

      • roles/serviceusage.serviceUsageViewer

    These API and IAM role requirements apply to both creating new 1.29 clusters and upgrading existing clusters to 1.29.

Functionality changes:

  • Added checks to validate the SSH client certificate file type before saving the certificate as a Secret.

  • Deprecated the spec.gkeVersion field in Machine and BareMetalMachine custom resources. After GKE on Bare Metal release 1.30, the value of gkeVersion isn't guaranteed to be reliable.

  • Added preflight checks for available disk space in specific directories:

    • During cluster creation, the following directories are checked:

      • / (the root directory) has at least 4 GiB of free space

      • /var/log/fluent-bit-buffers has at least 12 GiB of free space

      • /var/opt/buffered-metrics has at least 10016 MiB of free space

    • During a cluster upgrade, the following directory is checked:

      • / (the root directory) has at least 2 GiB of free space

Fixes:

  • Fixed an issue where the kubelet doesn't honor shortened, 1-second grace period for pod deletion during eviction-based draining.

The following container image security vulnerabilities have been fixed in 1.29.100-gke.251:

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

April 29, 2024

Release 1.29.0-gke.1449

GKE on Bare Metal 1.29.0-gke.1449 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.29.0-gke.1449 runs on Kubernetes 1.29.

If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.

Version 1.15 end of life: In accordance with the Version Support Policy, version 1.15 (all patch releases) of GKE on Bare Metal has reached its end of life and is no longer supported.

  • Added new API and IAM role requirements for Cloud Monitoring:

    • You must enable the kubernetesmetadata.googleapis.com API for your project and grant the roles/kubernetesmetadata.publisher IAM role to the Logging and Monitoring service account (anthos-baremetal-cloud-ops, when created automatically). Clusters use this API as an endpoint to send Kubernetes metadata to Google Cloud. The metadata is vital for cluster monitoring, debugging, and recovery. If you install your clusters behind a proxy, add kubernetesmetadata.googleapis.com to the list of allowed connections.

    • Due to changes in the way service accounts are checked, you must also grant the following IAM roles to the Logging and Monitoring service account:

      • roles/monitoring.viewer

      • roles/serviceusage.serviceUsageViewer

    These API and IAM role requirements apply to both creating new 1.29 clusters and upgrading existing clusters to 1.29.

  • GA: Support GKE Identity Service v2 capability for an improved security flow when you authenticate with third-party identity solutions.

    The GA offering of GKE Identity Service v2 has the following requirements and restrictions:

    • GKE Identity Service v2 now requires ports 11001 and 11002 on the control plane load balancer nodes, instead of 8443 and 8444. Ensure these ports are open and available before you upgrade a cluster to version 1.29.0-gke.1449 and higher. If the ports aren't open, upgrade preflight checks fail.

    • GKE Identity Service v2 requires version 1.5.1 or higher of the Anthos Auth gcloud CLI component. If necessary, update the Anthos Auth component (gcloud components update anthos-auth). If you use the Google Cloud SDK, updating the SDK (gcloud components update) to version 474.0.0 or later also updates the Anthos Auth component to the required version.

    • GKE Identity Service v2 doesn't work with GKE on Bare Metal clusters with the following configurations:

      • Clusters with a single control plane node only.

      • Clusters that use control plane nodes for load balancing. That is, clusters that aren't configured with either a separate load balancing node pool or manual load balancing.

  • GA: Added support for skews of up to two minor versions for selective node pool upgrades.

  • GA: Added capability to pause and resume cluster upgrades.

  • GA: Maintenance mode now uses eviction-based draining for nodes, instead of taint-based draining. Eviction-based draining uses the Eviction API, which honors Pod Disruption Budgets (PDBs). Draining nodes this way provides better protection against workload disruptions.

  • Preview: Added support for node-level private registry configuration for workload images.

  • Preview: Added support for rolling back select node pool upgrades.

  • Preview: Added support for admin and hybrid clusters to manage multiple versions user clusters concurrently.

  • Preview: Added support for using an intermediate Certificate Authority (CA) as the cluster root CA.

  • Preview: Added support to route workload logs to a third-party custom Kafka destination. This capability isn't enabled by default. You enable this capability in the cluster stackdriver resource spec by adding the unmanagedKafkaOutputConfig section. This section lets you specify the IP addresses of Kafka message brokers (brokers), topic names (topics), and keys to map the topics to partitions (topicKeys).

  • Improved command-line interface errors and error documentation.

Functionality changes:

  • GKE Identity Service v2 now sends extra parameters (extraParams) to your OIDC provider.

  • Extra node viewing permissions are added for accounts specified with the spec.clusterSecurity.authorization.clusterViewer.gcpAccounts field in the Cluster resource.

  • Added Status.Available field to BareMetalMachine resources to indicate whether the machine is available.

  • Updated preflight checks add a check for networking kernel modules (ip_tables or np_tables) and remove the iptables package check.

  • The Google plugin for the GKE Identity Service now caches the public keys based on max-age in cache-control header.

Fixes:

  • Fixed an issue where the kubelet doesn't honor shortened, 1-second grace period for pod deletion during eviction-based draining.

  • Fixed a cluster upgrade issue where the lifecycle-controller-deployer Pod was unable to migrate existing GKE on Bare Metal resources to the latest API version. This issue blocked upgrades to earlier version 1.28 releases.

  • Fixed an issue with configuring a proxy for your cluster that required you to manually set HTTPS_PROXY and NO_PROXY environment variables on the admin workstation.

  • Fixed an issue where upgrades are blocked because cluster-operator can't delete stale, failing preflight check resources.

  • Fixed an issue where the network check ConfigMap wasn't updated when nodes were added or removed.

The following container image security vulnerabilities have been fixed in version 1.29.0-gke.1449:

Known issues:

For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.