User cluster configuration file 1.30 and higher

This document describes the fields in the user cluster configuration file for versions 1.30 and higher Google Distributed Cloud.

Generating a template for your configuration file

If you used gkeadm to create your admin workstation, then gkeadm generated a template for your user cluster configuration file. And gkeadm filled in some of the fields for you.

If you did not use gkeadm to create your admin workstation, you can use gkectl to generate a template for your user cluster configuration file.

To generate a template for your user cluster configuration file:

gkectl create-config cluster --config=OUTPUT_FILENAME --gke-on-prem-version=VERSION

Replace the following:

OUTPUT_FILENAME: a path of your choice for the generated template. If you omit this flag, gkectl names the file user-cluster.yaml and puts it in the current directory.

VERSION: the desired version number. For example: gkectl create-config cluster --gke-on-prem-version=1.10.0-gke.8.

Template

Required fields and default values

If a field is marked as Required, then the completed configuration file must have a value filled in for the field.

Some fields are conditionally required. For example, loadBalancer.metalLB.addressPools is required if loadBalancer.kind is equal to MetalLB.

If a Default value is given for a field, then the cluster will use that value if you don't enter anything for the field. You can override a default value by entering a value.

If a field is not marked as Required and no Default value is given, then the field is optional. You can fill it in if it is relevant for you, but you don't have to fill it in.

Filling in your configuration file

In your configuration file, enter field values as described in the following sections.

enableAdvancedCluster

1.31 and higher

Preview
Optional
Immutable
Boolean
Prepopulated: false
Default: false

Set enableAdvancedClusters to true to enable advanced clusters. When this flag is enabled, the underlying Google Distributed Cloud software deploys controllers that allow for a more extensible architecture. Enabling advanced clusters gives you access to new features and capabilities, such as topology domains.

1.30 and lower

Not available.

name

Required
String

A name of your choice for your user cluster. The name must:

  • contain at most 40 characters
  • contain only lowercase alphanumeric characters or a hyphen (-)
  • start with an alphabetic character
  • end with an alphanumeric character

Example:

name: "my-user-cluster"

gkeOnPremVersion

Required
Mutable
String

The Google Distributed Cloud version for your user cluster.

Example:

gkeOnPremVersion: "1.29.0-gke.1456"

preparedSecrets.namespace

If infraConfigFilePath is configured in the admin cluster configuration file, remove this field. Otherwise, if you want to use prepared credentials, fill in this field.

Immutable
String
Possible values: A string that begins with "gke-onprem-secrets-"

The name of a Kubernetes namespace in the admin cluster where prepared Secrets are kept for this user cluster.

Example:

preparedSecrets:
  namespace: "gke-onprem-secrets-alice"

enableControlplaneV2

1.30 and higher

Mutable (from false to true is allowed)
Boolean
Prepopulated: true
Default: true

In version 1.30 and higher, Controlplane V2 is required for new user clusters. If you include this field in your configuration file, it must be set to true.

To migrate an existing cluster to Controlplane V2, see Migrate a user cluster to recommended features. After migrating to Controlplane V2, disabling Controlplane V2 isn't allowed.

With Controlplane V2, the control plane for a user cluster runs on one or more nodes in the user cluster itself. The benefits of Controlplane V2 include:

  • Architectural consistency between admin and user clusters.

  • Failure isolation. An admin cluster failure doesn't affect user clusters.

  • Operational separation. An admin cluster upgrade doesn't cause downtime for user clusters.

  • Deployment separation. You can put the admin and user clusters in different failure domains or geographical sites. For example, a user cluster in an edge location could be in a different geographical site from the admin cluster.

1.29 and lower

Immutable
Boolean
Prepopulated: true
Default: true

To enable Controlplane V2, set enableControlplaneV2 to true or remove the setting from your user cluster configuration file. Otherwise, set it to false. With Controlplane V2, the control plane for a user cluster runs on one or more nodes in the user cluster itself. When Controlplane V2 isn't enabled, the user cluster control plane runs in the admin cluster.

We recommend that you enable Controlplane V2.

Example:

enableControlplaneV2: true

vCenter

This section holds information about your vSphere environment and your connection to vCenter Server.

If infraConfigFilePath is configured in the admin cluster configuration file, remove this entire section.

If you included the vCenter section in the admin cluster configuration file:

  • If you want a field in this section to be the same as what you specified for your admin cluster, remove the field or leave it commented out.

  • If you want a field to be different from what you specified for your admin cluster, fill it in here. Any fields that you fill in here in the vCenter section override the corresponding fields in your admin cluster configuration file.

vCenter.address

Immutable
String
Default: Inherit from the admin cluster

The IP address or the hostname of the instance of vCenter Server that you want to use for this user cluster.

For more information, see Finding your vCenter server address.

If you specify a value that is different from the instance of vCenter Server used by the admin cluster, then Controlplane V2 must be enabled, and you must provide values for network.vCenter.networkName and all the required fields in the section vCenter.

Examples:

vCenter:
  address: "203.0.113.101"

vCenter:
  address: "my-vcenter-server-2.my-domain.example"

vCenter.datacenter

Immutable
String
Default: Inherit from the admin cluster

The relative path of a vSphere data center.

If you specify a value for this field, then you must also specify:

  • vCenter.networkName
  • Either vCenter.datastore or vCenter.storagePolicyName
  • Either vCenter.cluster or vCenter.resourcePool

The value you specify is relative to the root folder named /.

If your data center is in the root folder, the value is the name of the data center.

Example:

vCenter:
  datacenter: "my-uc-data-center"

Otherwise, the value is a relative path that includes one or more folders along with the name of the data center.

Example:

vCenter:
  datacenter: "data-centers/data-center-2"

vCenter.cluster

Immutable
String
Default: Inherit from the admin cluster

The relative path of a vSphere cluster that represents the ESXi hosts where your user cluster VMs will run. This vSphere cluster represents a subset of the physical ESXi hosts in your vCenter data center.

If you specify a value for this field, then you must also specify:

  • vCenter.networkName
  • vCenter.datacenter
  • Either vCenter.datastore or vCenter.storagePolicyName

The value you specify is relative to /.../DATA_CENTER/vm/.

If your vSphere cluster is in the /.../DATA_CENTER/vm/ folder, the value is the name of the vSphere cluster

Example:

vCenter:
  cluster: "my-uc-vsphere-cluster"

Otherwise, the value is a relative path that includes one or more folders along with the name of the vSphere cluster.

Example:

vCenter:
  cluster: "clusters/vsphere-cluster-2"

vCenter.resourcePool

Immutable
String
Default: Inherit from the admin cluster

A vCenter resource pool for your user cluster VMs.

If you want to use the default resource pool, set this to VSPHERE_CLUSTER/Resources.

Example:

vCenter:
  resourcePool: "my-uc-vsphere-cluster/Resources"

The value you specify is relative to /.../DATA_CENTER/host/.../VSPHERE_CLUSTER/Resources/

If your resource pool is a direct child of /.../DATA_CENTER/host/.../VSPHERE_CLUSTER/Resources/ the value is the name of the resource pool.

Example:

vCenter:
  resourcePool: "my-uc-resource-pool"

Otherwise, the value is a relative path that has two or more resource pools.

Example:

vCenter:
  resourcePool: "uc-resource-pool-1/uc-resource-pool-2"

vCenter.datastore

Optional
Immutable
String
Default: Inherit from the admin cluster

The name of a vSphere datastore for your user cluster.

The value you specify must be a name, not a path. Don't include any folders in the value.

Example:

vCenter:
  datastore: "my-datastore"

If you specify a value for this field, don't specify a value for vCenter.storagePolicyName. The vCenter.datastore field is immutable except when you set the field to an empty string when you migrate a datastore to Storage Policy Based Management (SPBM).

vCenter.storagePolicyName

Optional
Immutable
String

The name of a VM storage policy for the cluster nodes.

You can also specify a VM storage policy for the nodes in a particular node pool. But the policy specified here applies to any node pool that doesn't have its own policy.

For more information, see Configure a storage policy.

If you specify a value for this field, don't specify a value for vCenter.datastore.

vCenter.caCertPath

Mutable
String
Default: Inherit from the admin cluster

The path of the CA certificate for your vCenter server. For more information, see Getting your vCenter CA certificate.

Example:

vCenter:
  caCertPath: "/usr/local/google/home/me/certs/vcenter-ca-cert-2.pem"

For information on how to update this field, see Update vCenter certificate references.

vCenter.credentials.fileRef.path

Mutable
String
Default: Inherit from the admin cluster

If you provide a value for preparedSecrets.namespace, do not provide a value for this field. Instead, provide a value for vCenter.credentials.secretRef.version.

The path of a credentials configuration file that holds the username and password of your vCenter user account. The user account must have the Administrator role or equivalent privileges. See vSphere requirements.

You can use gkectl update credentials to update this field in an existing cluster. For more information, see Rotating service account keys.

For information about updating your vCenter credentials, see Updating cluster credentials.

Example:

vCenter:
  credentials:
    fileRef:
      path: "my-config-directory/my-credentials-2.yaml"

vCenter.credentials.fileRef.entry

Mutable
String
Default: Inherit from the admin cluster

The name of the credentials block, in your credentials configuration file, that holds the username and password of your vCenter user account.

Example:

vCenter:
  credentials:
    fileRef:
      entry: "vcenter-credentials-2"

vCenter.credentials.secretRef.version

String
Possible values: An integer string or "latest"
Default value: "latest"

If you provide a value for preparedSecrets.namespace, fill in this field. Otherwise, remove this field or leave it commented out.

The version of a prepared Secret in the admin cluster that holds the username and password for the instance of vCenter Server that you intend to use for this user cluster.

Example:

vCenter:
  credentials:
    secretRef:
      version: "1"

vCenter.folder

Immutable
String
Default: Inherit from the admin cluster

The relative path of a vSphere folder that you have already created. This folder will hold your user cluster VMs.

If you do not specify a value, your user cluster VMs will be put in /.../DATA_CENTER/vm/.

If you specify a value, it is relative to /.../DATA_CENTER/vm/.

The value can be the name of a folder.

Example:

vCenter:
  folder: "my-uc-folder"

Or the value can be a relative path that includes more than one folder.

Example:

vCenter:
  folder: "folders/folder-2"

componentAccessServiceAccountKeyPath

Mutable
String
Default: Inherit from the admin cluster

The path of the JSON key file for your component access service account.

Example:

componentAccessServiceAccountKeyPath: "my-key-folder/uc-access-key.json"

componentAccessServiceAccountKey.secretRef.version

Mutable
String
Possible values: An integer string or "latest"
Default value: "latest"

The version of a prepared Secret in the admin cluster that holds a JSON key for your component access service account.

Example:

componentAccessServiceAccountKey:
  secretRef:
    version: "1"

privateRegistry

Fill in this section only if your admin cluster is using a private registry and you want your user cluster to use a different private registry. Otherwise, remove this section or leave it commented out.

If you want to have some different settings from admin cluster settings:

  • Different username, password, certificate

  • (Only available for Control Plane V2) Different private registry address

Fill the privateRegistry fields here. The new settings are picked up during cluster creation and update.

privateRegistry.address

Immutable
String
Default: Inherit from the admin cluster

The IP address or FQDN (Fully Qualified Domain Name) of the machine that runs your private Docker registry.

Examples:

privateRegistry:
  address: "203.0.113.10"

privateRegistry:
  address: "fqdn.example.com"

privateRegistry.credentials.fileRef.path

Mutable
String
Default: Inherit from the admin cluster

If you are planning to use prepared secret for the private registry, do not provide a value for this field. Instead, provide a value for privateRegistry.credentials.secretRef.version.

The path of a credentials configuration file that holds the username and password of an account that Google Distributed Cloud can use to access your private Docker registry.

Example:

privateRegistry:
  credentials:
    fileRef:
      path: "my-config-folder/admin-creds.yaml"

privateRegistry.credentials.fileRef.entry

Mutable
String
Default: Inherit from the admin cluster

The name of the credentials block, in your credentials configuration file, that holds the username and password of your private Docker registry account.

privateRegistry:
  credentials:
    fileRef:
      entry: "private-registry-creds"

privateRegistry.credentials.secretRef.version

String
Possible values: An integer string or "latest"
Default value: "latest"

If you provide a value for preparedSecrets.namespace and want to use prepared secret for a private registry, fill in this field. Otherwise, remove this field or leave it commented out.

The version of a prepared Secret in the admin cluster that holds the username and password for the instance of vCenter Server that you intend to use for this user cluster.

Example:

privateRegistry:
  credentials:
    secretRef:
      version: "1"

privateRegistry.caCertPath

Mutable
String
Default: Inherit from the admin cluster

When Docker pulls an image from your private registry, the registry must prove its identity by presenting a certificate. The registry's certificate is signed by a certificate authority (CA). Docker uses the CA's certificate to validate the registry's certificate.

Set this field to the path of the CA's certificate.

Example:

privateRegistry:
  caCertPath: "my-cert-folder/registry-ca.crt"

network

This section holds information about your user cluster network.

network.hostConfig

1.30 and higher

This section holds information about NTP servers, DNS servers, and DNS search domains used by the VMs that are your cluster nodes.

If infraConfigFilePath is configured in the admin cluster configuration file (available in 1.31 and higher), remove this entire section. Otherwise, this section is required if one or more of the following are true:

  • network.ipMode.type is set to static.
  • enableControlplaneV2 is set to true or allowed to default to true. Note that Conrolplane V2 is required for new user clusters.

1.29 and lower

This section holds information about NTP servers, DNS servers, and DNS search domains used by the VMs that are your cluster nodes. If you are using the Seesaw load balancer, this information also applies to your Seesaw VMs.

This section is required if one or more of the following are true:

  • network.ipMode.type is set to static.
  • enableControlplaneV2 is set to true or allowed to default to true. The Seesaw load balancer isn't supported on clusters with Controlplane V2 enabled.
  • loadBalancer.kind is set to "Seesaw".

network.hostConfig.dnsServers

Required
Immutable
Array of strings.
The maximum number of elements in the array is three.

The addresses of DNS servers for the VMs.

Example:

network:
  hostConfig:
    dnsServers:
    - "172.16.255.1"
    - "172.16.255.2"

network.hostConfig.ntpServers

Required
Immutable
Array of strings

The addresses of time servers for the VMs to use.

Example:

network:
  hostConfig:
    ntpServers:
    - "216.239.35.0"

network.hostConfig.searchDomainsForDNS

Immutable
Array of strings

DNS search domains for the hosts to use. These domains are used as part of a domain search list.

Example:

network:
  hostConfig:
    searchDomainsForDNS:
    - "my.local.com"

network.ipMode.type

Immutable
String
Prepopulated: "static"
Default: "dhcp"

If you want your cluster nodes to get their IP address from a DHCP server, set this to "dhcp". If you want your cluster nodes to have static IP addresses chosen from a list that you provide, set this to "static".

Example:

network:
  ipMode:
    type: "static"

network.ipMode.ipBlockFilePath

This field is required if network.ipMode.type = static or if infraConfigFilePath is configured in the admin cluster configuration file.

Immutable
String

The absolute or relative path of the IP block file for your cluster.

Example:

network:
  ipMode:
    ipBlockFilePath: "/my-config-folder/user-cluster-ipblock.yaml"

network.serviceCIDR

Immutable
String
Prepopulated: "10.96.0.0/20"
>

A range of IP addresses, in CIDR format, to be used for Services in your cluster. Must be at least a /24 range.

Example:

network:
  serviceCIDR: "10.96.0.0/20"

network.podCIDR

Immutable
String
Prepopulated: "192.168.0.0/16"
Default: "192.168.0.0/16"

A range of IP addresses, in CIDR format, to be used for Pods in your cluster. Must be at least a /18 range.

Example:

network:
  podCIDR: "192.168.0.0/16"

The Service range must not overlap with the Pod range.

The Service and Pod ranges must not overlap with any address outside the cluster that you want to reach from inside the cluster.

For example, suppose your Service range is 10.96.232.0/24, and your Pod range is 192.168.0.0/16. Any traffic sent from a Pod to an address in either of those ranges will be treated as in-cluster and will not reach any destination outside the cluster.

In particular, the Service and Pod ranges must not overlap with:

  • IP addresses of nodes in any cluster

  • IP addresses used by load balancer machines

  • VIPs used by control-plane nodes and load balancers

  • IP address of vCenter servers, DNS servers, and NTP servers

We recommend that your Service and Pod ranges be in the RFC 1918 address space.

Here is one reason for the recommendation to use RFC 1918 addresses. Suppose your Pod or Service range contains external IP addresses. Any traffic sent from a Pod to one of those external addresses will be treated as in-cluster traffic and will not reach the external destination.

network.vCenter.networkName

If infraConfigFilePath is configured in the admin cluster configuration file, remove this field. Otherwise, this field is required if vCenter.address is different from the vCenter address you are using for the admin cluster.

Immutable
String

The name of the vSphere network for your user cluster nodes.

Example:

network:
  vCenter:
    networkName: "my-network"

If the name contains a special character, you must use an escape sequence for it.

Special characters Escape sequence
Slash (/) %2f
Backslash (\) %5c
Percent sign (%) %25

If the network name is not unique in your data center, you can specify a full path.

Example:

network:
  vCenter:
    networkName: "/my-uc-datacenter/network/my-network"

network.additionalNodeInterfaces

If you set multipleNetworkInterfaces to true, fill in this section. Otherwise, remove this section or leave it commented out.

Immutable

An array of objects, each of which describes a network interface that can be used on all or some of your cluster nodes.

network.additionalNodeInterfaces[i].networkName

Required if multipleNetworkInterfaces = true
Immutable
String

The name of the vSphere network that this additional node interface will connect to.

Example:

network:
  additionalNodeInterfaces:
  - networkName: "my-node-interface-1"

network.additionalNodeInterfaces[i].type

Required if multipleNetworkInterfaces = true
Immutable
String

Example:

network:
  additionalNodeInterfaces:
  - name: "my-node-interface-1"
    type: "static"

network.additionalNodeInterfaces[i].ipBlockFilePath

Required if network.additionalNodeInterfaces[i].type = static
Immutable
String

The path of an IP block file that holds IP addresses to be used for this network interface on the nodes that have this network interface.

Example:

network:
  additionalNodeInterfaces:
  - name: "my-node-interface-1"
    type: "static"
    ipBlockFilePath: "my-ipblock-file-1"

network.controlPlaneIPBlock

If infraConfigFilePath is configured in the admin cluster configuration file, remove this entire section.

Otherwise, whether you include this section depends on the value you set in enableControlplaneV2:

  • If enableControlplaneV2 is true, fill in this section.
  • If enableControlplaneV2 is false, remove this section.

network.controlPlaneIPBlock.netmask

Required if enableControlplaneV2 = true
Immutable
String

Example:

network:
  controlPlaneIPBlock:
    netmask: "255.255.255.0"

network.controlPlaneIPBlock.gateway

Required if enableControlplaneV2 = true
Immutable
String

Example:

network:
  controlPlaneIPBlock:
    gateway: "172.16.21.1"

network.controlPlaneIPBlock.ips

Required if enableControlplaneV2 = true
Immutable
Array of objects, each of which has an IP address and an optional hostname.

For a high-availability (HA) user cluster, the array has three elements. For a non-HA user cluster, the array has one element.

Example:

network:
  controlPlaneIPBlock:
    ips:
    - ip: "172.16.21.6"
      hostname: "cp-vm-1"
    - ip: "172.16.21.7"
      hostname: "cp-vm-2"
    - ip: "172.16.21.8"
      hostname: "cp-vm-3"

loadBalancer

This section holds information about the load balancer for your user cluster.

loadBalancer.vips.controlPlaneVIP

Required
Immutable
String

The IP address that you have chosen to configure on the load balancer for the Kubernetes API server of the user cluster.

Example:

loadBalancer:
  vips:
    controlplaneVIP: "203.0.113.3"

loadBalancer.vips.ingressVIP

Required
Immutable
String

The IP address that you have chosen to configure on the load balancer for the ingress proxy.

Example:

loadBalancer:
  vips:
    ingressVIP: "203.0.113.4"

loadBalancer.kind

1.30 and higher

Required
Immutable
String
Prepopulated: "MetalLB"

If infraConfigFilePath is configured in the admin cluster configuration file (available in 1.31 and higher), set this to "ManualLB". Otherwise, set this to "ManualLB" or "MetalLB".

Use "ManualLB" if you have a third-party load balancer (such as F5 BIG-IP or Citrix) or "MetalLB" for our bundled solution.

Although you can upgrade a cluster that has kind set to "F5BigIP" or "Seesaw", you can't use Dataplane V2 or Controlplane V2 on existing clusters with either of these load balancer configurations. For information on migrating load balancing configurations, see the following:

Example:

loadBalancer:
  kind: "MetalLB"

1.29 and earlier

Required
Immutable
String
Prepopulated: "MetalLB"

Set this to "ManualLB", "F5BigIP", "Seesaw", or "MetalLB".

To enable Dataplane V2 and Controlplane V2, we recommend that you use "ManualLB" if you have a third-party load balancer (such as F5 BIG-IP or Citrix) or "MetalLB" for our bundled solution.

Example:

loadBalancer:
  kind: "MetalLB"

loadBalancer.manualLB

If you set loadbalancer.kind to "manualLB", fill in this section. Otherwise, remove this section or leave it commented out.

loadBalancer.manualLB.ingressHTTPNodePort

Required if loadBalancer.kind = ManualLB (see Version note)
Immutable
Integer
Prepopulated: 30243

The ingress proxy in a user cluster is exposed by a Kubernetes Services of type LoadBalancer. The Service has a ServicePort for HTTP. Choose a nodePort value for the HTTP ServicePort, and set this field to the nodePort value.

Version note: In version 1.30 and higher, ingress node ports are optional for clusters using Controlplane V2.

Example:

loadBalancer:
  manualLB:
    ingressHTTPNodePort: 32527

loadBalancer.manualLB.ingressHTTPSNodePort

Required if loadBalancer.kind = ManualLB (see Version note)
Immutable
Integer
Prepopulated: 30879

The ingress proxy in a user cluster is exposed by a Service of type LoadBalancer. The Service has a ServicePort for HTTPS. Choose a nodePort value for the HTTPS ServicePort, and set this field to the nodePort value.

Version note: In version 1.30 and higher, ingress node ports are optional for clusters using Controlplane V2.

Example:

loadBalancer:
  manualLB:
    ingressHTTPSNodePort: 30139

loadBalancer.manualLB.controlPlaneNodePort

Required if loadBalancer.kind = ManualLB and enableControlplaneV2 = false
Immutable
Integer
Prepopulated: 30562

The Kubernetes API server of a user cluster runs in the admin cluster, and is exposed by a Service of type LoadBalancer. You must choose a nodePort value for the Service.

Set this field to the nodePort value.

Example:

loadBalancer:
  manualLB:
    controlPlaneNodePort: 30968

loadBalancer.manualLB.konnectivityServerNodePort

Required if loadBalancer.kind = ManualLB and enableControlplaneV2 = false
Immutable
Integer
Prepopulated: 30563

The Kubernetes API server of a user cluster that uses kubeception runs in the admin cluster, and is exposed by a Service of type LoadBalancer. The Konnectivity server reuses this service with a different nodePort value. You must choose a nodePort value for the Konnectivity server.

Set this field to the nodePort value for the Konnectivity server.

Example:

loadBalancer:
  manualLB:
    konnectivityServerNodePort: 30969

loadBalancer.f5BigIP

1.30 and higher

In 1.30 and higher, the value "F5BigIP" isn't allowed for loadbalancer.kind for new user clusters. If the loadBalancer.f5BigIP section is in your configuration file, remove it or comment it out.

You can still use your F5 BIG-IP load balancer with new user clusters, but the configuration is different. For configuration details, see Enabling manual load balancing mode.

1.29 and lower

If you set loadbalancer.kind to "f5BigIP", fill in this section. Otherwise, remove this section or leave it commented out.

To enable new and advanced features, we recommend that you configure manual load balancing for your F5 BIG-IP load balancer. To enable manual load balancing, set loadbalancer.kind to"ManualLB" and fill in the loadBalancer.manualLB section. For more information, see Enabling manual load balancing mode.

If you have an existing F5-BIG-IP load balancer and the cluster configuration uses this section, after you have upgraded to 1.29 or higher, we recommend that you migrate to manual load balancing.

loadBalancer.f5BigIP.address

1.30 and higher

Not allowed for new clusters
Required if loadBalancer.kind = "f5BigIP"
String

The address of your F5 BIG-IP load balancer.

Example:

loadBalancer:
  f5BigIP:
      address: "203.0.113.2"

1.29 and lower

Required if loadBalancer.kind = "f5BigIP"
String

The address of your F5 BIG-IP load balancer.

Example:

loadBalancer:
  f5BigIP:
      address: "203.0.113.2"

loadBalancer.f5BigIP.credentials.fileRef.path

1.30 and higher

Not allowed for new clusters
Required if loadBalancer.kind = "f5BigIP"
String

The path of a credentials file that holds the username and password of an account that Google Distributed Cloud can use to connect to your F5 BIG-IP load balancer.

The user account must have a user role that has sufficient permissions to set up and manage the load balancer. Either the Administrator role or the Resource Administrator role is sufficient.

For information about updating your F5 BIG-IP credentials, see Updating cluster credentials.

Example:

loadBalancer:
  f5BigIP:
    credentials:
      fileRef:
        path: "my-config-folder/user-creds.yaml"

1.29 and lower

Required if loadBalancer.kind = "f5BigIP"
String

The path of a credentials file that holds the username and password of an account that Google Distributed Cloud can use to connect to your F5 BIG-IP load balancer.

The user account must have a user role that has sufficient permissions to set up and manage the load balancer. Either the Administrator role or the Resource Administrator role is sufficient.

For information about updating your F5 BIG-IP credentials, see Updating cluster credentials.

Example:

loadBalancer:
  f5BigIP:
    credentials:
      fileRef:
        path: "my-config-folder/user-creds.yaml"

loadBalancer.f5BigIP.credentials.fileRef.entry

1.30 and higher

Not allowed for new clusters
Required if loadBalancer.kind = "f5BigIP"
String

The name of the credentials block, in your credentials configuration file, that holds the username and password of your F5 BIG-IP account.

Example:

loadBalancer:
  f5BigIP:
    credentials:
      fileRef:
        entry: "f5-creds"

1.29 and lower

Required if loadBalancer.kind = "f5BigIP"
String

The name of the credentials block, in your credentials configuration file, that holds the username and password of your F5 BIG-IP account.

Example:

loadBalancer:
  f5BigIP:
    credentials:
      fileRef:
        entry: "f5-creds"

loadBalancer.f5BigIP.partition

1.30 and higher

Not allowed for new clusters
Required if loadBalancer.kind = "f5BigIP"
String

The name of a BIG-IP partition that you created for your user cluster.

Example:

loadBalancer:
  f5BigIP:
    partition: "my-f5-user-partition"

1.29 and lower

Required if loadBalancer.kind = "f5BigIP"
String

The name of a BIG-IP partition that you created for your user cluster.

Example:

loadBalancer:
  f5BigIP:
    partition: "my-f5-user-partition"

loadBalancer.f5BigIP.snatPoolName

1.30 and higher

Not allowed for new clusters
Required if loadBalancer.kind = "f5BigIP" and you are using SNAT
String

The name of your SNAT pool.

Example:

loadBalancer:
  f5BigIP:
    snatPoolName: "my-snat-pool"

1.29 and lower

Required if loadBalancer.kind = "f5BigIP" and you are using SNAT
String

The name of your SNAT pool.

Example:

loadBalancer:
  f5BigIP:
    snatPoolName: "my-snat-pool"

loadBalancer.seesaw

1.30 and higher

In 1.30 and higher, the value "Seesaw" isn't allowed for loadbalancer.kind for new user clusters because this load balancer isn't supported with new and advanced features. If the loadBalancer.seesaw section is in your configuration file, remove it or comment it out.

Instead, you can configure the bundled MetalLB load balancer. To enable the MetalLB load balancer, set loadbalancer.kind to"MetalLB" and fill in the loadBalancer.metalLB section. For more information, see Bundled load balancing with MetalLB.

1.29 and lower

If you set loadbalancer.kind to "Seesaw", fill in this section. Otherwise, remove this section or leave it commented out.

Note the following limitations with the SeeSaw load balancer:

To use these features, we recommend that you configure the MetalLB load balancer. To enable the MetalLB load balancer, set loadbalancer.kind to"MetalLB" and fill in the loadBalancer.metalLB section. For more information, see Bundled load balancing with MetalLB.

loadBalancer.seesaw.ipBlockFilePath

1.30 and higher

Not allowed for new clusters
Required if loadBalancer.kind = Seesaw
Immutable
String

The path of the IP block file for your Seesaw VMs.

Example:

loadBalancer:
  seesaw:
    ipBlockFilePath: "config-folder/user-seesaw-ipblock.yaml"

1.29 and lower

Required if loadBalancer.kind = Seesaw
Immutable
String

The path of the IP block file for your Seesaw VMs.

Example:

loadBalancer:
  seesaw:
    ipBlockFilePath: "config-folder/user-seesaw-ipblock.yaml"

loadBalancer.seesaw.vrid

1.30 and higher

Not allowed for new clusters
Required if loadBalancer.kind = Seesaw
Immutable
Integer
Possible values: 1 - 255
Prepopulated: 0

The virtual router identifier of your Seesaw VM. This identifier, which is an integer of your choice, must be unique in a VLAN.

Example:

loadBalancer:
  seesaw:
    vrid: 125

1.29 and lower

Required if loadBalancer.kind = Seesaw
Immutable
Integer
Possible values: 1 - 255
Prepopulated: 0

The virtual router identifier of your Seesaw VM. This identifier, which is an integer of your choice, must be unique in a VLAN.

Example:

loadBalancer:
  seesaw:
    vrid: 125

loadBalancer.seesaw.masterIP

1.30 and higher

Not allowed for new clusters
Required if loadBalancer.kind = Seesaw
Immutable
String

The virtual IP address configured on your Master Seesaw VM.

Example:

loadBalancer:
  seesaw:
    masterIP: 172.16.20.21

1.29 and lower

Required if loadBalancer.kind = Seesaw
Immutable
String

The virtual IP address configured on your Master Seesaw VM.

Example:

loadBalancer:
  seesaw:
    masterIP: 172.16.20.21

loadBalancer.seesaw.cpus

1.30 and higher

Not allowed for new clusters
Required if loadBalancer.kind = Seesaw
Mutable
Integer
Prepopulated: 4

The number of CPUs for each of your Seesaw VMs.

Example:

loadBalancer:
  seesaw:
    cpus: 8

1.29 and lower

Required if loadBalancer.kind = Seesaw
Mutable
Integer
Prepopulated: 4

The number of CPUs for each of your Seesaw VMs.

Example:

loadBalancer:
  seesaw:
    cpus: 8

loadBalancer.seesaw.memoryMB

1.30 and higher

Not allowed for new clusters
Required if loadBalancer.kind = Seesaw
Mutable
Integer
Prepopulated: 3072

The number of mebibytes of memory for each of your Seesaw VMs.

Example:

loadBalancer:
  seesaw:
    memoryMB: 8192

Note: This field specifies the number of mebibytes of memory, not the number of megabytes. One mebibyte is 220 = 1,048,576 bytes. One megabyte is 106 = 1,000,000 bytes.

1.29 and lower

Required if loadBalancer.kind = Seesaw
Mutable
Integer
Prepopulated: 3072

The number of mebibytes of memory for each of your Seesaw VMs.

Example:

loadBalancer:
  seesaw:
    memoryMB: 8192

Note: This field specifies the number of mebibytes of memory, not the number of megabytes. One mebibyte is 220 = 1,048,576 bytes. One megabyte is 106 = 1,000,000 bytes.

loadBalancer.seesaw.vCenter.networkName

1.30 and higher

Not allowed for new clusters
Immutable
String
Default: Same as the cluster nodes

The name of the vCenter network that contains your Seesaw VMs.

Example:

loadBalancer:
  seesaw:
    vCenter:
      networkName: "my-seesaw-network"

1.29 and lower

Immutable
String
Default: Same as the cluster nodes

The name of the vCenter network that contains your Seesaw VMs.

Example:

loadBalancer:
  seesaw:
    vCenter:
      networkName: "my-seesaw-network"

loadBalancer.seesaw.enableHA

1.30 and higher

Not allowed for new clusters
Immutable
Relevant if loadBalancer.kind = Seesaw
Boolean
Prepopulated: false
Default: false

If you want to create a high-availability (HA) Seesaw load balancer, set this to true. Otherwise set this to false. An HA Seesaw load balancer uses a (Master, Backup) pair of VMs.

Example:

loadBalancer:
  seesaw:
    enableHA: true

1.29 and lower

Immutable
Relevant if loadBalancer.kind = Seesaw
Boolean
Prepopulated: false
Default: false

If you want to create a high-availability (HA) Seesaw load balancer, set this to true. Otherwise set this to false. An HA Seesaw load balancer uses a (Master, Backup) pair of VMs.

Example:

loadBalancer:
  seesaw:
    enableHA: true

loadBalancer.seesaw.disableVRRPMAC

1.30 and higher

Not allowed for new clusters
Immutable
Relevant if loadBalancer.kind = Seesaw
Boolean
Prepopulated: true
Default: true

If set to true (recommended), the Seesaw load balancer doesn't use MAC learning for failover. Instead, it uses gratuitous ARP. If this field is set this to false, the Seesaw load balancer uses MAC learning. If you are using vSphere 7 or later, and you have a high-availability Seesaw load balancer, then this field must be set to true.

Example:

loadBalancer:
  seesaw:
    disableVRRPMAC: true

1.29 and lower

Immutable
Relevant if loadBalancer.kind = Seesaw
Boolean
Prepopulated: true
Default: true

If you set this to true (recommended), the Seesaw load balancer doesn't use MAC learning for failover. Instead, it uses gratuitous ARP. If you set this to false, the Seesaw load balancer uses MAC learning. If you are using vSphere 7 or later, and you have a high-availability Seesaw load balancer, then you must set this to true.

Example:

loadBalancer:
  seesaw:
    disableVRRPMAC: true

loadBalancer.metalLB

If you set loadbalancer.kind to "MetalLB", fill in this section. Otherwise, remove this section or leave it commented out.

For more information, see Bundled load balancing with MetalLB.

loadBalancer.metalLB.addressPools

Required if loadBalancer.kind = 'MetalLB`
Mutable (see exception)

Array of objects, each of which holds information about an address pool to be used by the MetalLB load balancer.

If you will be creating the cluster with enableAdvancedCluster set to true, the array isn't mutable. You can add address pool(s), but removing an address pool (or pools) from an existing address pool array isn't supported.

loadBalancer.metalLB.addressPools[i].name

Required if loadBalancer.kind = MetalLB
Mutable (see exception)
String

A name of your choice for the address pool.

If you will be creating the cluster with enableAdvancedCluster set to true, the name isn't mutable. Changing the name after the cluster is created isn't supported.

Example:

loadBalancer:
  metalLB:
    addressPools:
    - name: "my-address-pool-1"
loadBalancer.metalLB.addressPools[i].addresses

Required if loadBalancer.kind = MetalLB
Mutable (see exception)

Array of strings, each of which is a range of addresses. Each range must be in CIDR format or hyphenated range format.

If you will be creating the cluster with enableAdvancedCluster set to true, the array isn't mutable. Removing an address (or addresses) from an existing address pool isn't supported, but you can add new address(s).

Example:

loadBalancer:
  metalLB:
    addressPools:
    - name: "my-address-pool-1"
      addresses:
      - "192.0.2.0/26"
      - "192.0.2.64-192.0.2.72"
loadBalancer.metalLB.addressPools[i].avoidBuggyIPs

Relevant if loadBalancer.kind = MetalLB
Mutable
Boolean
Prepopulated: false
Default: false

If you set this to true, the MetalLB controller will not assign IP addresses ending in .0 or .255 to Services. This avoids the problem of buggy consumer devices mistakenly dropping traffic sent to those special IP addresses.

Example:

loadBalancer:
  metalLB:
    addressPools:
    - name: "my-address-pool-1"
      - "192.0.2.0/26"
      - "192.0.2.64-192.0.2.72"
      avoidBuggyIPs: true
loadBalancer.metalLB.addressPools[i].manualAssign

Relevant if loadBalancer.kind = MetalLB
Mutable
Boolean
Prepopulated: false
Default: false

If you do not want the MetalLB controller to automatically assign IP addresses from this pool to Services, set this to true. Then a developer can create a Service of type LoadBalancer and manually specify one of the addresses from the pool.

Otherwise, set this to false.

Example:

loadBalancer:
  metalLB:
    addressPools:
    - name: "my-address-pool-1"
      manualAssign: true

enableDataplaneV2

1.30 and higher

Mutable (from false to true is allowed)
Boolean
Prepopulated: true
Default: true

This field controls the Container Network Interface (CNI) that the cluster uses. Enabling Dataplane V2 is required to use Controlplane V2. You can change the field from false to true to enable Dataplane V2, but disabling Dataplane V2 isn't allowed.

The benefits of Dataplane V2 include the following:

  • Dataplane V2 provides you with a more advanced and capable Container Network Interface (CNI) when compared with the previous CNI option, Calico. Calico is in maintenance mode, which means it receives only critical bug fixes and security updates, but no new feature development. In contrast, Dataplane V2 is actively developed and enhanced, ensuring that you have access to the latest networking innovations and capabilities.

  • Dataplane V2 is the preferred CNI for GKE and other Google Distributed Cloud products.

1.29 and lower

Immutable
Boolean
Prepopulated: true
Default: false

If you want to enable Dataplane V2, or if you want to use Controlplane V2, set this to true. Otherwise set this to false.

We recommend that you enable Dataplane V2.

Example:

  enableDataplaneV2: true

See Troubleshoot Google Distributed Cloud NFS and DataPlane v2 issues for troubleshooting steps.

dataplaneV2.forwardMode

Immutable
String
Possible values: "snat", "dsr"
Prepopulated: "snat"
Default: "snat"

The forwarding mode for a cluster that has Dataplane V2 enabled.

With source network address translation (SNAT) mode, a packet is SNAT-translated when it is forwarded from a load balancer node to a backend Pod. The Pod cannot see the original source IP address, and the return packet must pass through the load balancer node.

With direct server return (DSR) mode, a packet retains its original source IP address when it is forwarded from a load balancer node to a backend Pod. The Pod can see the original source IP address, and the return packet goes directly to the client without passing through the load balancer node.

Example:

dataplaneV2:
  forwardMode: "dsr"

multipleNetworkInterfaces

Immutable
Boolean
Prepopulated: false
Default: false

If infraConfigFilePath is configured in the admin cluster configuration file, this field must be false.

If you want to enable multiple network interfaces for Pods, set this to true.

For more information, see Configure multiple network interfaces for Pods.

Example:

multipleNetworkInterfaces: true

advancedNetworking

Immutable
boolean
Prepopulated: false
Default: false

If you plan to create an egress NAT gateway, set this to true. Otherwise set it to false.

If you set this field to true, you must also set enableDataplaneV2 to true.

Example:

advancedNetworking: true

disableBundledIngress

Set this to true if you want to disable bundled ingress for the cluster. Otherwise, set it to false.

Boolean
Mutable
Prepopulated: false
Default: false

Example:

disableBundledIngress: true

storage.vSphereCSIDisabled

If you want to disable the deployment of vSphere CSI components, set this to true. Otherwise, set it to false.

If infraConfigFilePath is configured in the admin cluster configuration file, this field must be true.

Mutable
Boolean
Prepopulated: false
Default: false

Example:

storage:
  vSphereCSIDisabled: false

masterNode

This section holds information about the nodes that serve as control-plane nodes for this user cluster.

masterNode.cpus

Mutable
Integer
Prepopulated: 4
Default: 4

The number of CPUs for each node that serves as a control plane for this user cluster.

Example:

masterNode:
  cpus: 8

masterNode.memoryMB

Mutable
Integer
Prepopulated: 8192
Default: 8192

The mebibytes of memory for each node that serves as a control plane for this user cluster. Must be a multiple of 4.

Example:

masterNode:
  memoryMB: 8192

Note: This field specifies the number of mebibytes of memory, not the number of megabytes. One mebibyte is 2^20 = 1,048,576 bytes. One megabyte is 10^6 = 1,000,000 bytes.

masterNode.replicas

Immutable
Integer
Possible values: 1 or 3
Prepopulated: 1
Default: 1

The number of control-plane nodes for this user cluster. This number cannot be changed once you have created the cluster. If you want to update the number of replicas later, you must recreate the user cluster.

If infraConfigFilePath is configured, then you must set this field to 3. Only HA user clusters are supported with topology domains.

Example:

masterNode:
  replicas: 3

masterNode.autoResize.enabled

Mutable
Boolean
Prepopulated: false
Default: false

Set this to true to enable automatic resizing of the control-plane nodes for the user cluster. Otherwise, set this to false.

For more information, see Enable node resizing for the control-plane nodes of a user cluster.

Example:

masterNode:
  autoResize:
    enabled: true

masterNode.vsphere

If infraConfigFilePath is configured in the admin cluster configuration file, remove this entire section.

masterNode.vsphere.datastore

Optional
Immutable
String
Default: The value of vCenter.datastore

The datastore where the control-plane nodes for this user cluster will be created.

The value you specify must be a name, not a path. Don't include any folders in the value.

Example:

masterNode:
  vSphere:
    datastore: "control-plane-datastore"

If you specify a value for this field, don't specify a value for masterNode.vsphere.storagePolicyName. The masterNode.vsphere.datastore field is immutable except when you set the field to an empty string when you migrate a datastore to Storage Policy Based Management (SPBM).

masterNode.vsphere.storagePolicyName

Optional
Immutable
String
Default: The value of vCenter.storagePolicyName

The name of a VM storage policy for the control-plane nodes.

For more information, see Configure a storage policy.

Example:

masterNode:
  vSphere:
    storagePolicyName: "control-plane-storage-policy"

If you specify a value for this field, don't specify a value for masterNode.vsphere.datastore.

masterNode.topologyDomains

1.31 and higher

If infraConfigFilePath is configured in the admin cluster configuration file (which enables topology domains), optionally include this field.

Array of strings | But only one element is supported
Immutable
Default: vSphereInfraConfig.defaultTopologyDomain if specified in the vSphere infrastructure configuration file

User cluster control plane nodes are spread across the topology domains specified in this field.

1.30 and lower

Not available.

nodePools

Required
Mutable
Array of objects, each of which describes a node pool.

For more information, see Creating and managing node pools.

nodePools[i].name

Required
Immutable
String

A name of your choice for the node pool. The name must:

  • contain at most 40 characters
  • contain only lowercase alphanumeric characters or a hyphen (-)
  • start with an alphabetic character
  • end with an alphanumeric character

Example:

nodePools:
- name: "my-node-pool-1"

nodePools[i].gkeOnPremVersion

When you upgrade a user cluster, you can specify that selected node pools remain at the previous version.

If you want this node pool to remain at the previous version, set this to the previous version. Otherwise, remove this field or set it to the empty string. For more information, see Upgrade a user cluster.

Mutable
String
Default: The cluster gkeOnPremVersion

Example:

nodePools:
- name: "my-node-pool"
  gkeOnPremVersion: "1.13.0-gke.16"

nodePools[i].cpus

Required
Mutable
Integer
Prepopulated: 4

The number of vCPUs for each node in the pool.

Example:

nodePools"
- name: "my-node-pool"
  cpus: 8

nodePools[i].memoryMB

Required
Mutable
Integer
Prepopulated: 8192

The mebibytes of memory for each node in the pool. Must be a multiple of 4.

Example:

nodePools"
- name: "my-node-pool"
  memoryMB: 8192

Note: This field specifies the number of mebibytes of memory, not the number of megabytes. One mebibyte is 2^20 = 1,048,576 bytes. One megabyte is 10^6 = 1,000,000 bytes.

nodePools[i].replicas

Required
Mutable
Integer
Prepopulated: 3
Possible values: The total number of untainted nodes across all node pools in the cluster must be at least 3.

The number of nodes in the pool.

Example:

nodePools:
- name: "my-node-pool"
  replicas: 5

nodePools[i].bootDiskSizeGB

Mutable
Integer
Prepopulated: 40
Default: 40

The size of the boot disk in gibibytes for each node in the pool.

Example:

nodePools
- name: "my-node-pool"
  bootDiskSizeGB: 40

nodePools[i].osImageType

Mutable
String
Possible values: "ubuntu_containerd", "cos", "ubuntu_cgv2", "cos_cgv2", "windows"
Prepopulated: "ubuntu_cgv2"
Default: "ubuntu_containerd"

The type of OS image to run on the VMs in the node pool.

If you will be creating the cluster with enableAdvancedCluster set to true (which is required for setting up topology domains), only ubuntu-cgroupv2 and ubuntu_containerd are supported.

Example:

nodePools
- name: "my-node-pool"
  osImageType: "ubuntu_cgv2"

nodePools[i].osImage

Required if nodePools[i].osImageType = windows
Mutable
String

The name of a Windows VM template. The template must be based on a Windows Server 2019 ISO that has the en-US language/region tag.

For more information, see User guide for Windows Server OS node pools.

nodePools[i].labels

Mutable
Mapping

Labels to apply to each Kubernetes Node object in the pool.

Example:

nodePools:
- name: "my-node-pool"
  labels:
    environment: "production"
    tier: "cache"

The kubelet can't apply labels to itself in certain namespaces for security reasons.

The reserved node label namespaces are : kubernetes.io, k8s.io, googleapis.com.

nodePools[i].taints

Mutable
Array of objects, each of which describes a Kubernetes taint that is applied to each node in the pool. Taints are key-value pairs associated with an effect. Taints are used with tolerations for Pod scheduling. Specify one of the following for the effect: NoSchedule, PreferNoSchedule, NoExecute.

Example:

nodePools:
- name: "my-node-pool"
  taints:
  - key: "staging"
    value: "true"
    effect: "NoSchedule"

nodePools[i].vsphere

If infraConfigFilePath is configured in the admin cluster configuration file, remove all fields in the nodePools[i].vsphere section except for nodePools[i].vsphere.tags.

nodePools[i].vsphere.datastore

Mutable
String
Default: The value of vCenter.datastore

The name of the vCenter datastore where the nodes will be created.

Example:

nodePools:
- name: "my-node-pool"
  vsphere:
    datastore: "my-datastore"

If you specify a value for this field, don't specify a value for nodePools[i].vsphere.storagePolicyName.

nodePools[i].vsphere.storagePolicyName

Mutable
String
Default: The value of vCenter.storagePolicyName

The name of a VM storage policy for the nodes in the pool.

For more information, see Configure a storage policy.

Example:

nodePools:
- name: "my-node-pool"
  vsphere:
    storagePolicyName: "my-storage-policy"

If you specify a value for this field, don't specify a value for nodePools[i].vsphere.datastore.

nodePools[i].vsphere.hostgroups

Fill in this section if you want to configure your cluster to use VM-Host affinity.

Mutable
Array of strings, each of which is the name of a host DRS group.

The array can have only one element.

Example:

nodePools:
- name: "my-node-pool"
  vsphere:
    hostgroups:
    - "my-hostgroup"

nodePools[i].vsphere.tags

Mutable
Array of objects, each of which describes a vSphere tag to be placed on the VMs in the node pool.

Each tag has a category and a name.

Example:

nodePools:
- name: "my-node-pool"
  vsphere:
    tags:
    - category: "purpose"
      name: "testing"

If you want to attach tags to all VMs in a node pool, your vCenter user account must have these vSphere tagging privileges:

  • vSphere Tagging.Assign or Unassign vSphere Tag
  • vSphere Tagging.Assign or Unassign vSphere Tag on Object (vSphere 7)

nodePools[i].autoscaling

Mutable

If you want to enable automatic scaling for the node pool, fill in this section. Otherwise, remove this section or leave it commented out.

nodePools[i].autoscaling.minReplicas

Mutable
Integer
Possible values: Greater than or equal to 1
Prepopulated: 0

The minimum number of nodes that the autoscaler can set for the pool.

Example:

nodePools:
- name: "my-node-pool"
  autoscaling:
    minReplicas: 5

nodePools[i].autoscaling.maxReplicas

Mutable
Integer
Prepopulated: 0

The maximum number of nodes that the autoscaler can set for the pool.

Example:

nodePools:
- name: "my-node-pool"
  autoscaling:
    maxReplicas: 10

nodePools[i].enableLoadBalancer

Relevant if loadBalancer.kind = "MetalLB"
Mutable
Boolean
Prepopulated: true
Default: false

Set this to true if you want to allow the MetalLB speaker to run on the nodes in the pool. Otherwise, set it to false.

If you will be creating the cluster with enableAdvancedCluster set to true, this field has no effect because the MetalLB speaker will always run on the user cluster's control-plane nodes.

For more information, see Bundled load balancing with MetalLB.

Example:

nodePools:
- name: "my-node-pool"
  enableLoadBalancer: true

nodePools[i].updateStrategy.rollingUpdate.maxSurge

Preview
Mutable
Integer
Default: 1

The maximum number of machines in the node pool that can be updated simultaneously during an update or upgrade.

nodePools:
- name: "my-node-pool"
  updateStrategy:
    rollingUpdate:
      maxSurge: 3

nodePools[i].topologyDomains

1.31 and higher

If infraConfigFilePath is configured in the admin cluster configuration file (which enables topology domains), optionally include this field.

Array of strings | But only one element is supported
Immutable
Default: vSphereInfraConfig.defaultTopologyDomain if specified in the vSphere infrastructure configuration file

Nodes in this node pool will be put into the topology domain specified by this field. For each node pool, only one topology domain is allowed. All nodes in a node pool will be put into a single topology domain.

1.30 and lower

Not available.

antiAffinityGroups.enabled

Mutable
Boolean
Prepopulated: true
Default: true

If enableAdvancedCluster is set to true (which is required for topology domains), set antiAffinityGroups.enabled to false.

If this field is true Google Distributed Cloud creates VMware Distributed Resource Scheduler (DRS) anti-affinity rules for your user cluster's nodes, causing them to be spread across at least three physical ESXi hosts in your datacenter

This feature requires that your vSphere environment meets the following conditions:

  • VMware DRS is enabled. VMware DRS requires vSphere Enterprise Plus license edition.

  • Your vSphere user account has the Host.Inventory.Modify cluster privilege.

  • There are at least four ESXi hosts available.

Even though the rule requires that the cluster nodes are spread across three ESXi hosts, we strongly recommend that you have at least four ESXi hosts available.

Recall that if you have a vSphere Standard license, you cannot enable VMware DRS.

If you do not have DRS enabled, or if you do not have at least four hosts where vSphere VMs can be scheduled, set antiAffinityGroups.enabled to false.

Example:

antiAffinityGroups:
  enabled: false

enableVMTracking

Preview
Immutable
Prepopulated: false

Set this to true to enable VM tracking with vSphere tags. Otherwise, set it to false.

For more information, see Enable VM tracking.

Example:

enableVMTracking: true

nodePoolUpdatePolicy.maximumConcurrentNodePoolUpdate

Mutable
Integer
Possible values: 0 or 1
Prepopulated: 0 Default: TODO: REVIEWERS: DOES THIS HAVE A DEFAULT VALUE?

The number of node pools to update at a time. A value of 1 specifies that one node pool can be updated at a time. A value of 0 specifies that an unlimited number of node pools can be updated at a time.

Example:

nodePoolUpdatePolicy:
  maximumConcurrentNodePoolUpdate: 1

nodePoolUpdatePolicy.updateStrategy.rollingUpdate.maxSurge

Preview
Mutable
Integer
Default: 1

The maximum number of machines in a node pool that can be updated simultaneously during an update or upgrade. Applies to any node pool that doesn't specify its own update strategy.

nodePools:
- name: "my-node-pool"
  updateStrategy:
    rollingUpdate:
      maxSurge: 3

authentication

This section holds information about how cluster users are authenticated and authorized.

authentication.oidc

Do not use this section. Instead, after cluster creation, edit the ClientConfig custom resource as described in Configuring clusters for Anthos Identity Service with OIDC .

authentication.sni

If you want to provide an additional serving certificate for the cluster's Kubernetes API server, fill in this section. Otherwise, remove this section or leave it commented out.

authentication.sni.certPath

String

The path of a serving certificate for the Kubernetes API server.

Example:

authentication:
  sni:
    certPath: "my-cert-folder/example.com.crt"

authentication.sni.keyPath

Optional
String

The path of the certificate's private key file.

Example:

authentication:
  sni:
    keyPath: "my-cert-folder/example.com.key"

gkeConnect

Required
Immutable

This section holds information about the Google Cloud project and service account you want to use to register your cluster to a Google Cloud fleet.

gkeConnect.projectID

Required
Immutable
String

The ID of your fleet host project. For new clusters, this project ID must be the same as the ID set in stackdriver.projectID and cloudAuditLogging.projectID. If the project IDs aren't the same, cluster creation fails. This requirement isn't applied to existing clusters.

Example:

gkeConnect:
  projectID: "my-fleet-host-project"

gkeConnect.location

Immutable
String
Default: global

Each cluster's fleet membership is managed by the Fleet service (gkehub.googleapis.com) and the Connect service (gkeconnect.googleapis.com). The location of the services can be global or regional. In 1.28 and later, you can optionally specify the Google Cloud region in which the Fleet and the Connect services run. If not specified, the global instances of the services are used. Note the following:

  • User clusters created prior to 1.28 are managed by the global Fleet and Connect services.

  • New clusters created using the GKE On-Prem API clients— the Google Cloud console, the Google Cloud CLI, or Terraform—use the same region that you specify for the GKE On-Prem API.

  • For new clusters, if you include this field, the region that you specify must be the same as the region configured in cloudAuditLogging.clusterLocation, stackdriver.clusterLocation, and gkeOnPremAPI.location. If the regions aren't the same, cluster creation fails.

Example:

gkeConnect:
  location: "us-central1"

gkeConnect.registerServiceAccountKeyPath

Required
Mutable
String

The path of the JSON key file for your connect-register service account.

To update the value of this field, use gkectl update cluster.

For information about changing your connect-register service account key, see Rotating service account keys.

Example:

gkeConnect:
  registerServiceAccountKeyPath: "my-key-folder/connect-register-key.json"

gkeConnect.registerServiceAccountKey.secretRef.version

Mutable
String
Possible values: An integer string or "latest"
Default value: "latest"

The version of a prepared Secret in the admin cluster that holds a JSON key for your connect-register service account.

Example:

gkeConnect:
  registerServiceAccountKey:
    secretRef:
      version: "1"

gkeOnPremAPI

In 1.16 and later, if the GKE On-Prem API is enabled in your Google Cloud project, all clusters in the project are enrolled in the GKE On-Prem API automatically in the region configured in stackdriver.clusterLocation.

  • If you want to enroll all clusters in the project in the GKE On-Prem API, be sure to do the steps in Before you begin to activate and use the GKE On-Prem API in the project.

  • If you don't want to enroll the cluster in the GKE On-Prem API, include this section and set gkeOnPremAPI.enabled to false. If you don't want to enroll any clusters in the project, disable gkeonprem.googleapis.com (the service name for the GKE On-Prem API) in the project. For instructions, see Disabling services.

Enrolling your user cluster in the GKE On-Prem API lets you use the standard tools—the Google Cloud console, the Google Cloud CLI, and Terraform— to manage the lifecycle of the cluster. Additionally, enrolling the cluster lets you use the console or the gcloud CLI to view cluster details. For example, you run can gcloud commands to get information about your user cluster.

After you add this section and create or update the cluster, if subsequently you remove the section and update the cluster, the update will fail.

gkeOnPremAPI.enabled

Mutable
Boolean
Default: true

By default, the cluster is enrolled in the GKE On-Prem API if the GKE On-Prem API is enabled in your project. Set to false if you don't want to enroll the cluster.

After the cluster is enrolled in the GKE On-Prem API, if you need to unenroll the cluster, make the following change and then update the cluster:

gkeOnPremAPI:
  enabled: false

gkeOnPremAPI.location

Immutable
String
Default: stackdriver.clusterLocation

The Google Cloud region where the GKE On-Prem API runs and stores cluster metadata. Choose one of the supported regions. You must use the same region that is configured in gkeConnect.location, stackdriver.clusterLocation and cloudAuditLogging.clusterLocation. If gkeOnPremAPI.enabled is false, don't include this field.

stackdriver

Required by default
Mutable

If you want to enable Cloud Logging and Cloud Monitoring for your cluster, fill in this section. Otherwise remove this section or leave it commented out.

This section is required by default. That is, if you don't include this section, you must include the --skip-validation-stackdriver flag when you run gkectl create cluster.

This section is required if you want to manage the lifecycle of the user cluster using GKE On-Prem API clients.

stackdriver.projectID

Required for Logging and Monitoring
Immutable
String

The ID of your fleet host project. For new clusters, this project ID must be the same as the ID set in gkeConnect.projectID and cloudAuditLogging.projectID. If the project IDs aren't the same, cluster creation fails. This requirement isn't applied to existing clusters.

If needed, you can configure a log router in this project to route logs into log buckets in another project. For information on how to configure the log router, see Supported destinations.

Example:

stackdriver:
  projectID: "my-fleet-host-project"

stackdriver.clusterLocation

Required for Logging and Monitoring
Immutable
String
Prepopulated: "us-central1"

The Google Cloud region where you want to store logs. It is a good idea to choose a region that is near your on-premises data center.

For new clusters, if you include the gkeOnPremAPI and cloudAuditLogging sections in the configuration file, the region that you set here must be the same region that you set in gkeConnect.location, gkeOnPremAPI.location, and cloudAuditLogging.clusterLocation. If the regions aren't the same, cluster creation fails.

Example:

stackdriver:
  clusterLocation: "us-central1"

stackdriver.enableVPC

Immutable
Boolean
Prepopulated: false

If your cluster's network is controlled by a VPC, set this to true. This ensures that all telemetry flows through Google's restricted IP addresses. Otherwise, set this to false.

Example:

stackdriver:
  enableVPC: false

stackdriver.serviceAccountKeyPath

Required for Logging and Monitoring
Mutable
String

The path of the JSON key file for your logging-monitoring service account.

To update the value of this field, use gkectl update cluster.

For information about changing your logging-monitoring service account key, see Rotating service account keys.

Example:

stackdriver:
  serviceAccountKeyPath: "my-key-folder/log-mon-key.json"

stackdriver.serviceAccountKey.secretRef.version

Mutable
String
Possible values: An integer string or "latest"
Default value: "latest"

The version of a prepared Secret in the admin cluster that holds a JSON key for your logging monitoring service account.

Example:

stackdriver:
  serviceAccountKey:
    secretRef:
      version: "1"

stackdriver.disableVsphereResourceMetrics

Mutable
Boolean
Prepopulated: false
Default: false

Set this to true to disable the collection of metrics from vSphere. Otherwise, set it to false.

Example:

stackdriver:
  disableVsphereResourceMetrics: true

usageMetering

1.30 and higher

This preview feature is deprecated, and you shouldn't use this for new clusters. Instead, we recommend that you use the predefined dashboard, Anthos Cluster Utilization Metering, to understand resource usage at different levels.

Although existing clusters using this feature will continue to function, you can also use Anthos Cluster Utilization Metering.

1.29 and lower

Preview
Immutable

This preview feature is deprecated in version 1.30, and we recommend that you don't use this for new clusters. Instead, we recommend that you use the predefined dashboard, Anthos Cluster Utilization Metering, to understand resource usage at different levels.

Although existing clusters using this feature will continue to function, you can also use Anthos Cluster Utilization Metering.

usageMetering.bigQueryProjectID

Preview
Required for usage metering
Immutable
String

The ID of the Google Cloud project where you want to store usage metering data.

Example:

usageMetering:
bigQueryProjectID: "my-bq-project"

usageMetering.bigQueryDatasetID

Preview
Required for usage metering
Immutable
String

The ID of the BigQuery dataset where you want to store usage metering data. Example:

usageMetering:
bigQueryDatasetID: "my-bq-dataset"

usageMetering.bigQueryServiceAccountKeyPath

Preview
Required for usage metering
Immutable
String.

The path of the JSON key file for your BigQuery service account.

To update the value of this field, use gkectl update cluster.

For information about changing your BigQuery service account key, see Rotating service account keys.

Example:

usageMetering:
bigQueryServiceAccountKeyPath: "my-key-folder/bq-key.json"

usageMetering.bigQueryServiceAccountKey.secretRef.version

Mutable
String
Possible values: An integer string or "latest"
Default value: "latest"

The version of a prepared Secret in the admin cluster that holds a JSON key for your BigQuery service account.

Example:

gkeConnect:
bigQueryServiceAccountKey:
  secretRef:
    version: "1"

usageMetering.enableConsumptionMetering

Preview
Required for usage metering
Immutable
Boolean
Prepopulated: false

Set this to true if you want to enable consumption-based metering. Otherwise set this to false.

Example:

usageMetering:
enableConsumptionMetering: true

cloudAuditLogging

If you want to integrate the audit logs from your cluster's Kubernetes API server with Cloud Audit Logs, fill in this section. Otherwise, remove this section or leave it commented out.

cloudAuditLogging.projectID

Required for Cloud Audit Logs
Immutable
String

The ID of your fleet host project. For new clusters, this project ID must be the same as the ID set in gkeConnect.projecID and stackdriver.projectID. If the project IDs aren't the same, cluster creation fails. This requirement isn't applied to existing clusters.

If needed, you can configure a log router in this project to route logs into log buckets in another project. For information on how to configure the log router, see Supported destinations.

Example:

cloudAuditLogging:
  projectID: "my-fleet-host-project"

cloudAuditLogging.clusterLocation

Required for Cloud Audit Logs
Immutable
String

The Google Cloud region where you want to store audit logs. It is a good idea to choose a region that is near your on-premises data center

For new clusters, if you include the gkeOnPremAPI and stackdriver sections in the configuration file, the region that you set here must be the same region that you set in gkeConnect.location, gkeOnPremAPI.location, and stackdriver.clusterLocation. If the regions aren't the same, cluster creation fails.

Example:

cloudAuditLogging:
  clusterLocation: "us-central1"

cloudAuditLogging.serviceAccountKeyPath

Required for Cloud Audit Logs
Mutable
String

The path of the JSON key file for your audit-logging service account.

If you will be creating the cluster with enableAdvancedCluster set to true (which is required for setting up topology domains), then set cloudAuditLogging.serviceAccountKeyPath to the same path as stackdriver.serviceAccountKeyPath.

To update the value of this field, use gkectl update cluster.

For information about changing your audit-logging service account key, see Rotating service account keys.

Example:

cloudAuditLogging:
  serviceAccountKeyPath: "my-key-folder/audit-log-key.json"

cloudAuditLogging.serviceAccountKey.secretRef.version

Mutable
String
Possible values: An integer string or "latest"
Default value: "latest"

The version of a prepared Secret in the admin cluster that holds a JSON key for your audit logging service account.

Example:

cloudAuditLogging:
  serviceAccountKey:
    secretRef:
      version: "1"

autoRepair.enabled

Optional
Mutable
Boolean
Prepopulated: true

Set this to true to enable node auto repair. Otherwise, set it to false.

Example:

autoRepair:
  enabled: true

secretsEncryption

If you want to encrypt Secrets without the need for an external KMS (Key Management Service), or any other dependencies, fill in this section. Otherwise, remove this section or leave it commented out.

If you will be creating the cluster with enableAdvancedCluster set to true (which is required for setting up topology domains), then remove this section. This feature isn't supported with advanced clusters.

secretsEncryption.mode

Required for Secrets encryption
Immutable
String
Possible value: "GeneratedKey"
Prepopulated: "GeneratedKey"

The Secret encryption mode.

secretsEncryption:
  mode: "GeneratedKey"

secretsEncryption.generatedKey.keyVersion

Required for Secrets encryption
Mutable
Integer
Prepopulated: 1

An integer of your choice to use for the key version number. We recommend that you start with 1.

For information about updating this field for an existing cluster, see Always-on Secrets encryption.

Example:

secretsEncryption:
  generatedKey:
    keyVersion: 1

secretsEncryption.generatedKey.disabled

Mutable
Boolean
Prepopulated: false

Set this to true to disable Secrets encryption. Otherwise set it to false.

For information about updating this field for an existing cluster, see Always-on Secrets encryption.

Example:

secretsEncryption:
  generatedKey:
    disabled: false