Google Distributed Cloud air-gapped 1.13.3 release notes

Billing:

• User fails to create BillingAccountBinding due to validation webhook error.

Block storage:

• Grafana pods stuck in Init state due to volume mount errors.
• There is a Trident multi-attach error.

Database Service:

• The dbs-fleet subcomponent has a reconciliation error when upgrading.
• The DBCluster creation fails after upgrade.

Identity and access management:

• The gatekeeper-audit pods in the opa-system namespace frequently restart.

Monitoring:

• The Cortex store gateway pods can crashloop on startup while syncing with the storage backend. The pods exceed their memory limits, causing Kubernetes to terminate them.
• The Kube control-plane metrics proxy pods can crashloop with image pull backoff error.
• A growth in WAL (write-ahead log) causes Prometheus to use a lot of memory. The system control plane VM node reports NodeHasInsufficientMemory and EvictionThresholdMet events because of this issue.

Networking:

• The switch image failed to extract or pull an image.

Object storage:

• Some object storage upgrade warnings can be ignored.

Operating system:

• Pods are stuck in a ContainerCreating state on a single node.

Physical servers:

• The DL380a server fails to provision.

Upgrade:

• A Helm failure during upgrade causes a series of rollbacks.
• When upgrading from HW2.0 and Ubuntu, the node upgrade incorrectly displays RockyLinux.
• The dhcp-tftp-core-server pod is not drained.
• The OrganizationUpgradeis stuck at node upgrade stage.
• Intermittent connectivity failure to external cluster VIP.
• Kernel fails to create container.
• An Incorrect version of Trident error appears during upgrade.
• During user cluster provisioning, some pods fail to be scheduled.
• The tenant organization upgrade fails at the preflight check stage with ErrImagePull.
• The root org upgrade is stuck on a failed signature job.
• During upgrade, the task for a root organization fails due to missing service accounts.
• Upgrade fails on shared-service-cluster upgrade
• The node fails during the user cluster upgrade.
• The root organization upgrade fails for preflight check.
• There is a persistent timeout during the initial root organizationupgrade.
• The obj-syslog-server subcomponent fails reconciliation in the root org.

Virtual machines:

• The NVIDIA device plugin DaemonSet fails with the driver rpc error message on cluster nodes with GPUs. This issue causes GPUs to be unavailable for virtual machines and pods.
• System cluster VM not ready.
• A data volume reports that the scratch space is not found.
• The obj-syslog-server subcomponent fails reconciliation in the root org.

Vertex AI:

• The streaming_recognize pre-trained API function of Speech-to-Text fails because of an issue with the client library.
• Job status polling is not supported for the batchTranslateDocument API.
• batchTranslateDocument requests might cause performance issues.
• The first time you enable pre-trained APIs, the GDC console might show an inconsistent status after a few minutes.
• Translation requests with more than 250 characters can crash translation-prediction-server pods.
• The GPUAllocation for shared service cluster is not configured correctly.
• When upgrading from version 1.9.x to 1.13.3, the Operable Component Lifecycle Management (OCLCM) controller for Vertex AI subcomponents might show errors.
• Translation requests might generate the RESOURCE_EXHAUSTED error code when the system frequency limit has been exceeded.
• batchTranslateDocument requests return error 503 "Batch Document translation is not implemented if the enableRAG operable parameter is not set to true in the cluster.