This document lists production updates to Distributed Cloud. We recommend that Google Distributed Cloud developers periodically check this list for any new announcements.
You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.
To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.
April 03, 2024
A Denial-of-Service (DoS) vulnerability (CVE-2023-45288) was recently discovered in multiple implementations of the HTTP/2 protocol, including the golang HTTP server used by Kubernetes. The vulnerability could lead to a DoS of the Google Kubernetes Engine (GKE) control plane. For more information, see the GCP-2024-022 security bulletin.
March 21, 2024
Release 1.15.11
GKE on Bare Metal 1.15.11 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.15.11 runs on Kubernetes 1.26.
If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.
The following container image security vulnerabilities have been fixed in 1.15.11:
Medium-severity container vulnerabilities:
Low-severity container vulnerabilities:
Known issues:
For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.
March 04, 2024
Release 1.15.10
GKE on Bare Metal 1.15.10 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.15.10 runs on Kubernetes 1.26.
If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.
GKE on Bare Metal version 1.15.10 and later has been qualified on and supports Red Hat Enterprise Linux (RHEL) version 8.9.
Fixes:
The following container image security vulnerabilities have been fixed in 1.15.10:
High-severity container vulnerabilities:
Medium-severity container vulnerabilities:
Known issues:
For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.
February 01, 2024
Release 1.15.9
GKE on Bare Metal 1.15.9 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.15.9 runs on Kubernetes 1.26.
If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.
Fixes:
The following container image security vulnerabilities have been fixed in 1.15.9:
Critical container vulnerabilities:
High-severity container vulnerabilities:
Medium-severity container vulnerabilities:
Low-severity container vulnerabilities:
Known issues:
For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.
January 31, 2024
Security bulletin (all minor versions)
A security vulnerability, CVE-2024-21626, has been discovered in runc
where a user with permission to create Pods might be able to gain full access to the node filesystem.
For instructions and more details, see the GCP-2024-005 security bulletin.
December 13, 2023
Release 1.15.8
GKE on Bare Metal 1.15.8 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.15.8 runs on Kubernetes 1.26.
If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.
Functionality changes:
- Changed upgrade preflight checks behavior to skip kubeadm job creation check to improve upgrade reliability.
Fixes:
- Fixed an issue where the network check ConfigMap wasn't being updated when nodes were added or removed.
Fixes:
The following container image security vulnerabilities have been fixed in 1.15.8:
High-severity container vulnerabilities:
Medium-severity container vulnerabilities:
Low-severity container vulnerabilities:
Known issues:
For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.
November 20, 2023
Release 1.15.7
Anthos clusters on bare metal 1.15.7 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.7 runs on Kubernetes 1.26.
If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.
Fixed an issue where CoreDNS Pods can get stuck in an unready state.
The following container image security vulnerabilities have been fixed in 1.15.7:
Critical container vulnerabilities:
High-severity container vulnerabilities:
Medium-severity container vulnerabilities:
Low-severity container vulnerabilities:
Known issues:
For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.
October 30, 2023
Release 1.15.6
GKE on Bare Metal 1.15.6 is now available for download. To upgrade, see Upgrade clusters. GKE on Bare Metal 1.15.6 runs on Kubernetes 1.26.
If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.
Functionality changes:
- Removed hardcoded timeout value for the
bmctl backup
operation.
Fixes:
Fixed a memory leak in Dataplane V2.
Added direct dependencies on systemd, containerd, and kubelet over their mount point folders in
/var/lib/
.
Fixes:
Fixed the following vulnerabilities:
Critical container vulnerabilities:
High-severity container vulnerabilities:
Medium-severity container vulnerabilities:
Low-severity container vulnerabilities:
Known issues:
For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.
September 25, 2023
Release 1.15.5
Anthos clusters on bare metal 1.15.5 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.5 runs on Kubernetes 1.26.
If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.
Fixed an issue to prevent cluster upgrades from starting on a node before either all Pods have been drained or the Pod draining timeout has been reached.
The following container image security vulnerabilities have been fixed in 1.15.5:
High-severity container vulnerabilities:
Medium-severity container vulnerabilities:
Known issues:
For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.
August 16, 2023
Release 1.15.4
Anthos clusters on bare metal 1.15.4 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.4 runs on Kubernetes 1.26.
If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.
Functionality changes:
Audit logs are compressed on the wire for Cloud Audit Logs consumption, reducing egress bandwidth by approximately 60%.
Upgraded local volume provisioner to v2.5.0.
Upgraded snapshot controller to v5.0.1.
Deprecated v1beta1 volume snapshot custom resources. Anthos clusters on bare metal will stop serving v1beta1 resources in a future release.
Fixes:
- Fixed an issue for clusters configured with manual load balancing where CA rotation reported that there were no (
0
) control plane nodes.
Fixes:
The following container image security vulnerabilities have been fixed:
- High-severity container vulnerabilities:
- Medium-severity container vulnerabilities:
- Low-severity container vulnerabilities:
Known issues:
For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.
July 20, 2023
Release 1.15.3
Anthos clusters on bare metal 1.15.3 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.3 runs on Kubernetes 1.26.
If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.
Anthos clusters on bare metal 1.15.3 supports adding the
gkeOnPremAPI
section to your admin and user cluster configuration files to
enroll the clusters in the Anthos On-Prem API. Enrolling the clusters in the Anthos On-Prem API lets you
upgrade admin and user clusters using the Google Cloud console or the Google Cloud CLI.
Fixes:
Fixed an issue where the apiserver could become responsive during a cluster upgrade for clusters with a single control plane node.
Fixed an issue where cluster installations or upgrades fail when the cluster name has more than 45 characters.
Fixed an issue where node-specific labels set on the node pool were sometimes overwritten.
Fixed an issue where audit logs were duplicated into the offline buffer even when they are sent to Cloud Audit Logs successfully.
The following container image security vulnerabilities have been fixed:
Known issues:
For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.
June 27, 2023
Security bulletin (all minor versions)
A number of vulnerabilities have been discovered in Envoy, which is used in Anthos Service Mesh (ASM). These were reported separately as GCP-2023-002.
For more information, see the GCP-2023-016 security bulletin.
June 22, 2023
Release 1.15.2
Anthos clusters on bare metal 1.15.2 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.2 runs on Kubernetes 1.26.
If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.
Functionality changes:
Added preflight check to make sure control plane and load balancer nodes aren't in maintenance mode before an upgrade.
Upgraded etcd version to v3.4.26-0-gke.0.
Fixes:
Fixed an issue where containerd didn't restart when there was a version mismatch. This issue caused an inconsistent containerd version within the cluster.
Fixed an issue where the
spec.proxy.noProxy
value wasn't used in the Google Cloud connectivity preflight check (bmctl check gcp
).Fixed an issue that caused the logging agent to use continuously increasing amounts of memory. The following container image security vulnerabilities have been fixed:
Known issues:
For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.
June 16, 2023
Security bulletin (all minor versions)
Two new security issues were discovered in Kubernetes where users may be able to launch containers that bypass policy restrictions when using ephemeral containers and either ImagePolicyWebhook (CVE-2023-2727) or the ServiceAccount admission plugin (CVE-2023-2728).
For more information, see the GCP-2023-014 security bulletin.
May 31, 2023
Release 1.15.1
Anthos clusters on bare metal 1.15.1 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.1 runs on Kubernetes 1.26.
If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.
Functionality changes:
Updated the cluster snapshot capability so that information can be captured for the target cluster even when the cluster custom resource is missing or unavailable.
Improved
bmctl
error reporting for failures during the creation of a bootstrap cluster.Added support for using the
baremetal.cluster.gke.io/maintenance-mode-deadline-seconds
cluster annotation to specify the maximum node draining duration, in seconds. By default, a 20-minute (1200 seconds) timeout is enforced. When the timeout elapses, all pods are stopped and the node is put into maintenance mode. For example to change the timeout to 10 minutes, add the annotationbaremetal.cluster.gke.io/maintenance-mode-deadline-seconds: "600"
to your cluster.Added
node_pool_name
to theanthos_baremetal_node_os_count
metric.
Fixes:
Fixed an issue that caused the
bmctl restore
command to stop responding for clusters with manually configured load balancers.Fixed an issue that caused health checks to report failure when they find a Pod with a status of
TaintToleration
even when the replicaset for the Pod has sufficient Pods running.Fixed an issue that prevented Anthos clusters on bare metal from restoring a high-availability quorum for nodes that use
/var/lib/etcd
as a mountpoint.Fixed an issue that caused conflicts with third-party Ansible automation.
Fixed an issue where invalid kubelet image pull settings, such as negative values, resulted in update job failures. Unchecked job failures generate an excessive accumulation of kubelet configuration backup files.
Fixed a cluster upgrade issue that prevented some control plane nodes from rejoining a cluster configured for high availability.
The following container image security vulnerabilities have been fixed:
- CVE-2018-1099
- CVE-2019-19906
- CVE-2020-8032
- CVE-2021-3468
- CVE-2021-43784
- CVE-2022-2097
- CVE-2022-2196
- CVE-2022-3424
- CVE-2022-3707
- CVE-2022-4129
- CVE-2022-4304
- CVE-2022-4379
- CVE-2022-4382
- CVE-2022-4450
- CVE-2022-4904
- CVE-2022-24407
- CVE-2022-29162
- CVE-2022-41723
- CVE-2022-41725
- CVE-2023-0045
- CVE-2023-0215
- CVE-2023-0286
- CVE-2023-0458
- CVE-2023-0461
- CVE-2023-1073
- CVE-2023-1074
- CVE-2023-1076
- CVE-2023-1077
- CVE-2023-1078
- CVE-2023-1079
- CVE-2023-1118
- CVE-2023-1281
- CVE-2023-1513
- CVE-2023-1611
- CVE-2023-1670
- CVE-2023-1829
- CVE-2023-1855
- CVE-2023-1872
- CVE-2023-1989
- CVE-2023-1990
- CVE-2023-1998
- CVE-2023-2162
- CVE-2023-2194
- CVE-2023-21102
- CVE-2023-22998
- CVE-2023-23004
- CVE-2023-23559
- CVE-2023-25012
- CVE-2023-26545
- CVE-2023-27487
- CVE-2023-27488
- CVE-2023-27491
- CVE-2023-27492
- CVE-2023-27493
- CVE-2023-27496
- CVE-2023-28328
- CVE-2023-28466
- CVE-2023-28484
- CVE-2023-29469
- CVE-2023-30456
- CVE-2023-30772
- CVE-2023-32269
Known issues:
For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.
May 10, 2023
CentOS Linux 8 Support Deprecated
CentOS Linux 8 reached its end of life (EOL) on December 31st, 2021. We strongly recommend that you migrate to one of the other supported operating systems from Anthos clusters on bare metal. All support for CentOS is removed from Anthos clusters for bare metal release 1.17 (December 2023) and subsequent releases.
April 27, 2023
Release 1.15.0
Anthos clusters on bare metal 1.15.0 is now available for download. To upgrade, see Upgrading Anthos on bare metal. Anthos clusters on bare metal 1.15.0 runs on Kubernetes 1.26.
If you use a third-party storage vendor, check the GDCV Ready storage partners document to make sure the storage vendor has already passed the qualification for this release of GKE on Bare Metal.
Version 1.12 end of life: In accordance with the Anthos Version Support Policy, version 1.12 (all patch releases) of Anthos clusters on bare metal has reached its end of life and is no longer supported.
Cluster lifecycle:
- Upgraded from Kubernetes version 1.25 to version 1.26.
- GA: Set in-place upgrade (without bootstrap cluster) as the default upgrade method for self-managed clusters.
- GA: Added support for configuring worker node pools for parallel node upgrades to significantly reduce upgrade times. Added a
minimumAvailableNodes
field to specify a minimum number of nodes to keep available for workloads throughout the upgrade. - Preview: Added support for parallel upgrades of worker node pools.
- Added support for Red Hat Enterprise Linux (RHEL) version 8.7.
- Added support for Ubuntu 22.04 LTS.
- GA: Added support for increasing the number of IP addresses for Services after cluster creation. For more information, see Increase service network range.
- Preview: Added ability to configure kubelet image pull settings for node pools. For more information, see Configure kubelet image pull settings.
- Streamlined the snapshot uploading and sharing process.
- GA: Added support of Control group v2 (cgroup v2).
- Preview: Added a separate instance of etcd for the
etcd-events
object. - Updated
cert-manager
to version 1.17.2. - Updated automated API enablement when you run
bmctl create config
with the--enable-apis
flag. The following APIs are added to the enablement list:- Enable
storage.googleapis.com
as a required API. - Enable
gkeonprem.googleapis.com
as a recommended API.
- Enable
- Added a new field
status.failures
to the NodePool custom resource to aggregate failures across machines in the NodePool. - Added a new condition type
PreflightCheckSuccessful
to the NodePool custom resource. This condition type summarizes the preflight check status across machines in the NodePool.
Networking:
- Added support for ClusterDNS to specify order for
upstreamNameServers
with anorderPolicy
. Allowed values fororderPolicy
arerandom
,round_robin
, orsequential
. The default value israndom
.
Observability:
- Added support for filtering application logs. This feature can reduce application logging billing and network traffic from the cluster to Cloud Logging. For more information, see Filter application logs.
GA: Fully managed Cloud Monitoring Integration dashboards:
- In the next Anthos release (version 1.16), the following dashboards in Cloud Monitoring Sample Library are unavailable:
- Anthos cluster control plane uptime
- Anthos cluster node status
- Anthos cluster pod status
- Anthos utilization metering
- GKE on-prem node status
- GKE on-prem control plane uptime
- GKE on-prem pod status
- GKE on-prem vSphere vm health status
- In the next Anthos release (version 1.16), the following customized dashboards aren't created when you create a new cluster:
- Anthos cluster control plane uptime
- Anthos cluster pod status
- Anthos cluster node status
- Anthos cluster VM status
- An added Anthos integration page is available from the Cloud Monitoring Integration page. The Anthos integration includes descriptions and previews for the predefined Anthos dashboards:
- Anthos Cluster Control Plane Uptime
- Anthos Cluster Node Status
- Anthos Cluster Pod Status
- Anthos Cluster KubeVirt VM Status
- Anthos Cluster Utilization Metering
For more information, see Use predefined dashboards.
- In the next Anthos release (version 1.16), the following dashboards in Cloud Monitoring Sample Library are unavailable:
Preview: Added support for system metrics when you use Google Cloud Managed Service for Prometheus.
Security and Identity:
- Preview: Added support for Binary Authorization, a service on Google Cloud that provides software supply-chain security for container-based applications. For more information, see Binary Authorization for Anthos clusters overview.
- Preview: Added support for VPC Service Controls, which provides additional security for your clusters to help mitigate the risk of data exfiltration.
- Improved security by disabling port 10255, the kubelet read-only port, by default. For more information, see Disable kubelet read-only port in Hardening your cluster's security.
Functionality changes:
- Replacing taints and labels. Clusters created and upgraded to Anthos clusters on bare metal version 1.15.0 and higher have
node-role.kubernetes.io/control-plane:*
taints andnode-role.kubernetes.io/control-plane
labels. These new taints and labels replace thenode-role.kubernetes.io/master
label andnode-role.kubernetes.io/master:*
taints on new and upgraded control plane nodes.
Networking changes:
- Replaced the anetd CNI plugin for the bootstrap cluster with kindnet.
- Increased eBPF map limit to 512 K to allow for more load balancer Services.
- Upgraded CoreDNS to version 1.9.4.
Anthos VM Runtime:
- Moved the Anthos VM Runtime release notes to a separate page in the Anthos VM Runtime documentation section.
Fixes:
- Fixed an issue that caused the
bmctl reset nodes
command to fail if thebmctl-workspace
directory was empty. - Fixed an intermittent issue that caused the
bmctl upgrade cluster
command to indicate that the operation was complete before the cluster was in a ready state.
Known issues:
For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.