Google Distributed Cloud air-gapped 1.13.1 release notes

Backup and restore:

• Attempting to restore a backup to a quota-constrained user cluster fails.

Billing:

• Billing metrics are not correctly emitted to the cortex because of the missing MetricsProxySidecar.

Block storage:

• Virtual machine launcher pods fail to map volumes.
• Storage-related failures might make the system unusable.
• Persistent volumes are created with an incorrect size.
• When an organization is deactivated, there might be an issue deleting a StorageVirtualMachine.
• Secrets and certificates are not cleaned up after deactivating an organization.
• A deletion reconciliation failure can occur in the StorageVirtualMachine.
• Ansible jobs get stuck during the bare metal upgrade.

Cluster management:

• The machine-init job fails during cluster provisioning.
• The connection for a database pod running in the service cluster to an object storage bucket in the org admin cluster fails.
• The preflight check fails.

Database Service:

• For user-facing databases, the initial provisioning, resizing, or enabling HA on an existing database cluster takes up to 40 minutes longer than before, and the performance is two to three times slower than before.
• The database service clone does not work for a storage quota-constrained cluster due to a problem with backup and restore.
• The Iops enforcement might impact storage performance.

DNS:

• DNSSEC must be explicitly turned off in resolved.conf.

Infrastructure as code (IAC):

• Excessive GitLab token creation risks filling GitLab databases.

Key Management Service (KMS):

• When the kms-rootkey-controller memory usage exceeds the 600Mi limit, the controller enters a CrashLoopBackOff due to an OOMKilled status.

Monitoring:

• Dashboards don't display Vertex AI metrics.
• The mon-cortex pod has a reconciliation error.
• The metrics-server-exporter pod in the system cluster is crash looping.
• The mon-prober-backend-prometheus-config ConfigMap gets reset to include no probe jobs, and alert MON-A0001 is triggered.
• After configuring the Monitoring service to send alerts, multiple duplicate alerts are automatically created.
• The ObservabilityPipeline object shows Reconciler error logs that you must ignore.

Multi-zone bootstrap:

• There are no specific roles for bootstrapping multi-zone deployments.
• The Bootstrap resource that is created is incompatible with the logic that processes it.
• A required resource is not created during bootstrap, causing components that rely on this resource to not function correctly.

Networking:

• The node is not reachable.
• There are connectivity issues to Database Service instances.
• A PodCIDR is not assigned to nodes even though a ClusterCIDRConfig is created.
• A VM node has drifted or inaccurate time.

Object storage:

• Deleting an organization might not be successful.

Operating system:

• In rare situations, pods are stuck in the init state on a particular node.
• The bm-system-machine-preflight-check Ansible job for a bare metal or VM node fails with Either ip_tables or nf_tables kernel module must be loaded.

Operations Suite Infrastructure (OI):

• For Hardware 3.0, Launch Smart Storage Administration (SSA) is no longer needed.

Perimeter security:

• The org system cluster gets stuck during the organization bootstrap.

• The PANW firewall AddressGroups don't update with the OCITcidr-claim changes, resulting in unresolvable iac.gdch.domain.example domains.

Platform security:

• When PKI BYO SubCA mode generates a new certificate signing request (CSR) while a previously signed certificate is uploaded to the SubCA, the reconciler doesn't check if the new CSR matches the old signed certificate and marks the cert-manager CertificateRequest custom resource (CR) as Ready. This occurs during SubCA certificate renewal or manual rotation.

• A known issue in cert-manager results in unsuccessful issuance of PKI bring-your-own (BYO) certificates with Automated Certificate Management Environment (ACME).

Physical servers:

• The server is stuck at the provisioning state.
• The server bootstrap fails due to POST issues on the HPE server.
• The server is stuck in the provisioning state.

Upgrade:

• The bm-system and other jobs running ansible playbook are stuck at gathering facts.
• The management IP of a server is unreachable during upgrade.
• The upgrade fails in the iac-zoneselection-global subcomponent.

Vertex AI:

• The MonitoringTarget shows a Not Ready status when user clusters are being created, causing pre-trained APIs to continually show an Enabling state in the user interface.
• The Translation frontend pod and service fail to initialize because the ODS system cluster secret is outdated.

Virtual machines:

• The BYO image import fails for qcow2 and raw images.
• Provisioning a disk from a custom image fails.
• The object storage upgrade shows an error during the postflight or preflight check.