Admin cluster configuration file 1.30 and higher

This document describes the fields in the admin cluster configuration file for versions 1.30 and higher Google Distributed Cloud.

Generating a template for your configuration file

If you used gkeadm to create your admin workstation, then gkeadm generated a template for your admin cluster configuration file. Also, gkeadm filled in some of the fields for you.

If you did not use gkeadm to create your admin workstation, you can use gkectl to generate a template for your admin cluster configuration file.

To generate a template for your admin cluster configuration file:

gkectl create-config admin --config=OUTPUT_FILENAME --gke-on-prem-version=VERSION

Replace OUTPUT_FILENAME with a path of your choice for the generated template. If you omit this flag, gkectl names the file admin-cluster.yaml and puts it in the current directory.

Replace VERSION with the full version of a 1.29 patch release. For example: gkectl create-config admin --gke-on-prem-version=1.31.0-gke.889. This version must be be equal to or less than your gkectl version. If you omit this flag, the generated config template is populated with values based on the latest 1.31 patch.

Template

Filling in your configuration file

In your configuration file, enter field values as described in the following sections.

enableAdvancedCluster

1.31 and higher

Available only for new clusters
Preview
Optional
Immutable
Boolean
Prepopulated: false
Default: false

Set enableAdvancedClusters to true to enable advanced clusters. When this flag is enabled, the underlying Google Distributed Cloud software deploys controllers that allow for a more extensible architecture. Enabling advanced clusters gives you access to new features and capabilities, such as topology domains. This field is required if infraConfigFilePath is configured.

1.30 and lower

Not available.

name

Optional
String
Default: A random name with prefix "gke-admin-"

A name of your choice for the cluster.

Example:

name: "my-admin-cluster"

bundlePath

Required
Mutable
String

The path of your Google Distributed Cloud bundle file.

The Google Distributed Cloud full bundle file contains all of the components in a particular release of Google Distributed Cloud. When you create an admin workstation, it comes with a full bundle at:

/var/lib/gke/bundles/gke-onprem-vsphere-VERSION-full.tgz

Example:

bundlePath: "/var/lib/gke/bundles/gke-onprem-vsphere-1.10.0-gke.8.full.tgz"

preparedSecrets.enabled

If infraConfigFilePath is configured, remove this field.

Preview
Optional
Immutable
Boolean
Prepopulated: false
Default: false

Set this to true if you want to use prepared credentials in the admin cluster. Otherwise, set it to false or omit the field.

Example:

preparedSecrets:
  enabled: true

vCenter

This sections holds information about your vSphere environment and your connection to vCenter Server.

If infraConfigFilePath is configured, remove this entire section. Otherwise, this section is required and immutable.

vCenter.address

Required
Immutable
String

The IP address or the hostname of your vCenter server.

For more information, see Finding your vCenter server address.

Examples:

vCenter:
  address: "203.0.113.100"

vCenter:
  address: "my-vcenter-server.my-domain.example"

vCenter.datacenter

Required
Immutable
String

The relative path of a vSphere data center.

The value you specify is relative to the root folder named /.

If your data center is in the root folder, the value is the name of the data center.

Example:

vCenter:
  datacenter: "my-data-center"

Otherwise, the value is a relative path that includes one or more folders along with the name of the data center.

Example:

vCenter:
  datacenter: "data-centers/data-center-1"

vCenter.cluster

Required
Immutable
String

The relative path of a vSphere cluster that represents the ESXi hosts where your admin cluster VMs will run. This vSphere cluster represents a subset of the physical ESXi hosts in your vCenter data center.

The value you specify is relative to /.../DATA_CENTER/vm/.

If your vSphere cluster is in the /.../DATA_CENTER/vm/ folder, the value is the name of the vSphere cluster.

Example:

vCenter:
  cluster: "my-vsphere-cluster"

Otherwise, the value is a relative path that includes one or more folders along with the name of the vSphere cluster.

Example:

vCenter:
  cluster: "clusters/vsphere-cluster-1"

vCenter.resourcePool

Required
Immutable
String

A vCenter resource pool for your admin cluster VMs.

If you want to use the default resource pool, set this to VSPHERE_CLUSTER/Resources.

Example:

vCenter:
  resourcePool: "my-vsphere-cluster/Resources"

If you want to use a resource pool that you have already created, set this to the relative path of your resource pool.

The value you specify is relative to /.../DATA_CENTER/host/.../VSPHERE_CLUSTER/Resources/

If your resource pool is a direct child of /.../DATA_CENTER/host/.../VSPHERE_CLUSTER/Resources/ the value is the name of the resource pool.

Example:

vCenter:
  resourcePool: "my-resource-pool"

Otherwise, the value is a relative path that has two or more resource pools.

Example:

vCenter:
  resourcePool: "resource-pool-1/resource-pool-2"

vCenter.datastore

Required if vCenter.storagePolicyName isn't specified
Immutable
String

The name of a vSphere datastore for your admin cluster.

The value you specify must be a name, not a path. Don't include any folders in the value.

Example:

vCenter:
  datastore: "my-datastore"

You must specify a value for either vCenter.datastore or vCenter.storagePolicyName, but not both. If you specify a value for this field, don't specify a value for vCenter.storagePolicyName. The vCenter.datastore field is immutable except when you set the field to an empty string when you migrate a datastore to Storage Policy Based Management (SPBM).

vCenter.storagePolicyName

Required if vCenter.datastore isn't specified
Immutable
String

The name of a VM storage policy for the cluster nodes.

The admin cluster must be highly-available (HA) to work with the storage policy.

For more information, see Configure a storage policy.

You must specify a value for either vCenter.datastore or vCenter.storagePolicyName, but not both. If you specify a value for this field, don't specify a value for vCenter.datastore.

vCenter.caCertPath

Required
Mutable
String

The path of the CA certificate for your vCenter server.

For more information, see Getting your vCenter CA certificate.

For information about updating this field for an existing cluster, see Update vCenter certificate references.

Example:

vCenter:
  caCertPath: "/usr/local/google/home/me/certs/vcenter-ca-cert.pem"

vCenter.credentials.fileRef.path

Required
String

The path of a credentials configuration file that holds the username and password of your vCenter user account. The user account should have the Administrator role or equivalent privileges. See vSphere requirements.

You can use gkectl update to update this field in an existing cluster.

For information on how to update your vCenter credentials, see Updating cluster credentials.

Example:

vCenter:
  credentials:
    fileRef:
      path: "my-config-folder/admin-creds.yaml"

vCenter.credentials.fileRef.entry

Required
String

The name of the credentials block, in your credentials configuration file, that holds the username and password of your vCenter user account.

You can use gkectl update to update this field in an existing cluster.

For information on how to update your vCenter credentials, see Updating cluster credentials.

Example:

vCenter:
  credentials:
    fileRef:
      entry: "vcenter-creds"

vCenter.folder

Optional
Immutable
String
Default: The datacenter-wide folder

The relative path of a vSphere folder that you have already created. This folder will hold your admin cluster VMs.

If you do not specify a value, your admin cluster VMs will be put in /.../DATA_CENTER/vm/.

If you specify a value, it is relative to /.../DATA_CENTER/vm/.

The value can be the name of a folder.

Example:

vCenter:
  folder: "my-folder"

Or the value can be a relative path that includes more than one folder.

Example:

vCenter:
  folder: "folders/folder-1"

vCenter.dataDisk

Don't provide a value for this field. Either delete the field or leave it commented out.

network

Required
Immutable

This section holds information about your admin cluster network.

network.hostConfig

This section holds information about NTP servers, DNS servers, and DNS search domains used by the VMs that are your cluster nodes.

If infraConfigFilePath is configured, remove this entire section. Otherwise, this section is required and immutable.

network.hostConfig.dnsServers

Required if network.hostConfig section is filled in.
Immutable
Array of strings.
The maximum number of elements in the array is three.

The addresses of DNS servers for the VMs.

Example:

network:
  hostConfig:
    dnsServers:
    - "172.16.255.1"
    - "172.16.255.2"

network.hostConfig.ntpServers

Required if network.hostConfig section is filled in.
Immutable
Array of strings

The addresses of time servers for the VMs to use.

Example:

network:
  hostConfig:
    ntpServers:
    - "216.239.35.0"

network.hostConfig.searchDomainsForDNS

Optional
Immutable
Array of strings

DNS search domains for the VMs to use. These domains are used as part of a domain search list.

Example:

network:
  hostConfig:
    searchDomainsForDNS:
    - "my.local.com"

network.ipMode.type

Required
Immutable
String
Prepopulated: "static"
Default: "dhcp"

If you want your cluster nodes to get their IP address from a DHCP server, set this to "dhcp". If you want your cluster nodes to have static IP addresses chosen from a list that you provide, set this to "static".

Example:

network:
  ipMode:
    type: "static"

network.ipMode.ipBlockFilePath

The absolute or relative path of the IP block file for your cluster.

If infraConfigFilePath is configured, this field is required.

This field is also required in the following cases:

  • 1.16 and lower non-HA admin cluster: list the IP addresses for the control-plane node and add-on nodes in an IP block file.

  • 1.16 and lower HA admin cluster: list the IP addresses for add-on nodes in an IP block file.

  • 1.29 and lower kubeception user clusters: list the IP addresses for the user cluster control-plane nodes in the admin cluster IP block file.

Immutable
String

Example:

network:
  ipMode:
    ipBlockFilePath: "/my-config-folder/admin-cluster-ipblock.yaml"

network.serviceCIDR

Required
Immutable
String
Smallest possible range: /24
Largest possible range: /12
Prepopulated: "10.96.232.0/24"
Default: "10.96.232.0/24"

A range of IP addresses, in CIDR format, to be used for Services in your cluster.

Example:

network:
  serviceCIDR: "10.96.232.0/24"

network.podCIDR

Required
Immutable
String
Smallest possible range: /18
Largest possible range: /8
Prepopulated: "192.168.0.0/16"
Default: "192.168.0.0/16"

A range of IP addresses, in CIDR format, to be used for Pods in your cluster.

Example:

network:
  podCIDR: "192.168.0.0/16"

The Service range must not overlap with the Pod range.

The Service and Pod ranges must not overlap with any address outside the cluster that you want to reach from inside the cluster.

For example, suppose your Service range is 10.96.232.0/24, and your Pod range is 192.168.0.0/16. Any traffic sent from a Pod to an address in either of those ranges will be treated as in-cluster and will not reach any destination outside the cluster.

In particular, the Service and Pod ranges must not overlap with:

  • IP addresses of nodes in any cluster

  • IP addresses used by load balancer machines

  • VIPs used by control-plane nodes and load balancers

  • IP address of vCenter servers, DNS servers, and NTP servers

We recommend that your Service and Pod ranges be in the RFC 1918 address space.

Here is one reason for the recommendation to use RFC 1918 addresses. Suppose your Pod or Service range contains external IP addresses. Any traffic sent from a Pod to one of those external addresses will be treated as in-cluster traffic and will not reach the external destination.

network.vCenter.networkName

The name of the vSphere network for your cluster nodes.

If infraConfigFilePath is configured, remove this field. Otherwise, this field is required and immutable.

Example:

network:
  vCenter:
    networkName: "my-network"

If the name contains a special character, you must use an escape sequence for it.

Special characters Escape sequence
Slash (/) %2f
Backslash (\) %5c
Percent sign (%) %25

If the network name is not unique in your data center, you can specify a full path.

Example:

network:
  vCenter:
    networkName: "/data-centers/data-center-1/network/my-network"

network.controlPlaneIPBlock

Network information for the admin cluster's control-plane nodes.

If infraConfigFilePath is configured, remove this entire section. Otherwise, this section is required and immutable.

network.controlPlaneIPBlock.netmask

Required
Immutable
String

The netmask for the network that has your control-plane nodes.

Example:

network:
  controlPlaneIPBlock:
    netmask: "255.255.255.0"

network.controlPlaneIPBlock.gateway

Required
Immutable
String

The IP address of the default gateway for your control-plane nodes.

Example:

network:
  controlPlaneIPBlock:
    gateway: "172.16.22.1"

network.controlPlaneIPBlock.ips

Required
Immutable
Array of three objects, each of which has an IP address and an optional hostname.

These are the IP addresses that will be assigned to your control-plane nodes.

Example:

network:
  controlPlaneIPBlock:
    ips:
    - ip: "172.16.22.6"
      hostname: "admin-cp-vm-1"
    - ip: "172.16.22.7"
      hostname: "admin-cp-vm-2"
    - ip: "172.16.22.8"
      hostname: "admin-cp-vm-3"

infraConfigFilePath

1.31

Specifying a path in this field enables topology domains.

Preview
Optional
String
Mutable

Specify the full path of the vSphere infrastructure configuration file.

Example:

infraConfigFilePath: "/my-config-folder/vsphere-infrastructure.yaml"

In addition to adding the path, copy the vSphere infrastructure configuration file to the applicable directory.

If you add a path to infraConfigFilePath, you need to make the following changes to your admin cluster and user cluster configuration files.

Admin cluster configuration file changes

  • Remove the following from your admin cluster configuration file. You configure this information in the vSphere infrastructure configuration file per topology domain.

    • vCenter: remove the entire vCenter section.
    • network.hostConfig: remove the entire network.hostConfig section.
    • network.vCenter.networkName: remove this field.

  • Make the following changes in your admin cluster configuration file:

    • network.controlPlaneIPBlock: remove the entire section. Instead, specify the IP addresses for the admin cluster control-plane nodes in an IP block file.

    • preparedSecrets: remove this field. Prepared credentials aren't supported when topology domains is enabled. Remove this field.

  • The following fields are required:

User cluster configuration file changes

  • Remove the following from your user cluster configuration file. You configure this information in the vSphere infrastructure configuration file per topology domain.

    • vCenter: remove the entire vCenter section.
    • network.hostConfig: remove the entire network.hostConfig section.
    • network.vCenter.networkName: remove this field.
    • masterNode.vsphere: remove the entire section.
    • nodePools[i].vsphere.datastore and nodePools[i].vsphere.storagePolicyName: remove these fields.

  • Make the following changes in your user cluster configuration file:

    • network.controlPlaneIPBlock: remove the entire section. Instead, specify the IP addresses for the admin cluster control-plane nodes in an IP block file.

    • preparedSecrets: remove this field. Prepared credentials aren't supported when topology domains is enabled.

    • nodePools[i].vsphere.hostgroups: remove this field. VM-Host affinity aren't supported when topology domains is enabled.

    • multipleNetworkInterfaces: set this field to false. Multiple network interfaces for Pods aren't supported when topology domains is enabled.

    • storage.vSphereCSIDisabled: set this field to true to disable the deployment of vSphere CSI components

  • The following fields are required:

1.30 and lower

Not available.

loadBalancer

This section holds information about the load balancer for your admin cluster.

loadBalancer.vips.controlPlaneVIP

Required
Immutable
String

The IP address that you have chosen to configure on the load balancer for the Kubernetes API server of the admin cluster.

Example:

loadBalancer:
  vips:
    controlplaneVIP: "203.0.113.3"

loadBalancer.kind

Specify the kind of load balancer to use.

1.30 and higher

Required
Immutable
String
Prepopulated: "MetalLB"

If infraConfigFilePath is configured (available in 1.31 and higher), set this to "ManualLB". Otherwise, set this to "ManualLB" or "MetalLB".

To enable new and advanced features, use "ManualLB" if you have a third-party load balancer (such as F5 BIG-IP or Citrix) or "MetalLB" for our bundled solution.

Note the following differences from previous versions:

  • In 1.30 and higher, the value "F5BigIP" isn't allowed for new admin clusters.

  • In 1.28 and higher, the value "Seesaw" isn't allowed for new admin clusters.

If you have an existing cluster that uses these load balancers, see the following migration guides:

Example:

loadBalancer:
  kind: "MetalLB"

1.28 and 1.29

Required
Immutable
String
Prepopulated: "MetalLB"

String. Set this to "ManualLB", "F5BigIP", or "MetalLB"

To enable new and advanced features, we recommend that you use "ManualLB" if you have a third-party load balancer (such as F5 BIG-IP or Citrix) or "MetalLB" for our bundled solution.

In 1.28 and higher, the value "Seesaw" isn't allowed for new admin clusters. If you have an existing cluster that uses Seesaw load balancer, see Migrate from the Seesaw load balancer to MetalLB.

Example:

loadBalancer:
  kind: "MetalLB"

1.16 and lower

Required
Immutable
String
Prepopulated: "MetalLB"

String. Set this to "ManualLB", "F5BigIP", "Seesaw", or "MetalLB"

If you set adminMaster.replicas to 3, then you can't use the Seesaw load balancer.

Example:

loadBalancer:
  kind: "MetalLB"

loadBalancer.manualLB

If you set loadbalancer.kind to "ManualLB", fill in this section. Otherwise, remove this section.
Immutable

loadBalancer.manualLB.ingressHTTPNodePort

Remove this field from your configuration file. It is not used in an admin cluster.

loadBalancer.manualLB.ingressHTTPSNodePort

Remove this field from your configuration file. It is not used in an admin cluster.

loadBalancer.manualLB.konnectivityServerNodePort

Remove this field from your configuration file. It is not used in an admin cluster.

loadBalancer.f5BigIP

1.30 and higher

In 1.30 and higher, the value "F5BigIP" isn't allowed for loadbalancer.kind for new admin clusters. If the loadBalancer.f5BigIP section is in your configuration file, remove it or comment it out.

You can still use your F5 BIG-IP load balancer with new admin clusters, but the configuration is different. For configuration details, see Enabling manual load balancing mode.

If you have an existing F5-BIG-IP load balancer and the cluster configuration uses this section, after you have upgraded to 1.29 or higher, we recommend that you migrate to manual load balancing.

1.29

If you set loadbalancer.kind to "f5BigIP", fill in this section. Otherwise, remove this section.

To enable new and advanced features, we recommend that you configure manual load balancing for your F5 BIG-IP load balancer. To enable manual load balancing, set loadbalancer.kind to"ManualLB" and fill in the loadBalancer.manualLB section. For more information, see Enabling manual load balancing mode.

If you have an existing F5-BIG-IP load balancer and the cluster configuration uses this section, after you have upgraded to 1.29 or higher, we recommend that you migrate to manual load balancing.

1.28 and lower

If you set loadbalancer.kind to "f5BigIP", fill in this section. Otherwise, remove this section.

To enable new and advanced features, we recommend that you configure manual load balancing for your F5 BIG-IP load balancer. To enable manual load balancing, set loadbalancer.kind to"ManualLB" and fill in the loadBalancer.manualLB section. For more information, see Enabling manual load balancing mode.

loadBalancer.f5BigIP.address

1.30 and higher

Not allowed for new clusters
Required if loadBalancer.kind = "f5BigIp"
Immutable
String

The address of your F5 BIG-IP load balancer. For example:

loadBalancer:
  f5BigIP:
    address: "203.0.113.2"

1.29 and lower

Required if loadBalancer.kind = "f5BigIp"
Immutable
String

The address of your F5 BIG-IP load balancer. For example:

loadBalancer:
  f5BigIP:
    address: "203.0.113.2"

loadBalancer.f5BigIP.credentials.fileRef.path

1.30 and higher

Not allowed for new clusters
Required if loadBalancer.kind = "f5BigIp"
Mutable
String

The path of a credentials configuration file that holds the username and password of an account that Google Distributed Cloud can use to connect to your F5 BIG-IP load balancer.

The user account must have a user role that has sufficient permissions to set up and manage the load balancer. Either the Administrator role or the Resource Administrator role is sufficient.

You can use gkectl update to update this field in an existing cluster.

For information on how to update your F5 BIG-IP credentials, see Updating cluster credentials.

Example:

loadBalancer:
  f5BigIP:
    credentials:
      fileRef:
        path: "my-config-folder/admin-creds.yaml"

1.29 and lower

Required if loadBalancer.kind = "f5BigIp"
Mutable
String

The path of a credentials configuration file that holds the username and password of an account that Google Distributed Cloud can use to connect to your F5 BIG-IP load balancer.

The user account must have a user role that has sufficient permissions to set up and manage the load balancer. Either the Administrator role or the Resource Administrator role is sufficient.

You can use gkectl update to update this field in an existing cluster.

For information on how to update your F5 BIG-IP credentials, see Updating cluster credentials.

Example:

loadBalancer:
  f5BigIP:
    credentials:
      fileRef:
        path: "my-config-folder/admin-creds.yaml"

loadBalancer.f5BigIP.credentialsfileRef.entry

1.30 and higher

Not allowed for new clusters
Required if loadBalancer.kind = "f5BigIp"
Mutable
String

The name of the credentials block, in your credentials configuration file, that holds the username and password of your F5 BIG-IP account.

You can use gkectl update to update this field in an existing cluster.

For information on how to update your F5 BIG-IP credentials, see Updating cluster credentials.

Example:

loadBalancer:
  f5BigIP:
    credentials:
      fileRef:
        entry: "f5-creds"

1.29 and lower

Required if loadBalancer.kind = "f5BigIp"
Mutable
String

The name of the credentials block, in your credentials configuration file, that holds the username and password of your F5 BIG-IP account.

You can use gkectl update to update this field in an existing cluster.

For information on how to update your F5 BIG-IP credentials, see Updating cluster credentials.

Example:

loadBalancer:
  f5BigIP:
    credentials:
      fileRef:
        entry: "f5-creds"

loadBalancer.f5BigIP.partition

1.30 and higher

Not allowed for new clusters
Required if loadBalancer.kind = "f5BigIp"
Immutable
String

The name of a BIG-IP partition that you created for your admin cluster.

Example:

loadBalancer:
  f5BigIP:
    partition: "my-f5-admin-partition"

1.29 and lower

Required if loadBalancer.kind = "f5BigIp"
Immutable
String

The name of a BIG-IP partition that you created for your admin cluster.

Example:

loadBalancer:
  f5BigIP:
    partition: "my-f5-admin-partition"

loadBalancer.f5BigIP.snatPoolName

1.30 and higher

Not allowed for new clusters
Optional
Relevant if loadBalancer.kind = "f5BigIp"
Immutable
String

If you are using SNAT, the name of your SNAT pool. If you are not using SNAT, remove this field.

Example:

loadBalancer:
  f5BigIP:
    snatPoolName: "my-snat-pool"

1.29 and lower

Optional
Relevant if loadBalancer.kind = "f5BigIp"
Immutable
String

If you are using SNAT, the name of your SNAT pool. If you are not using SNAT, remove this field.

Example:

loadBalancer:
  f5BigIP:
    snatPoolName: "my-snat-pool"

loadBalancer.seesaw

Do not use this section. The Seesaw load balancer is not supported for new admin clusters in version 1.28 and higher. Instead, we recommend that you configure the MetalLB load balancer for new admin clusters. For more information on configuring MetalLB, see Bundled load balancing with MetalLB.

Although we still support Seesaw for non-HA admin clusters that have been upgraded, we recommend that you migrate to MetalLB.

antiAffinityGroups.enabled

Optional
Mutable
Boolean
Prepopulated: true

Set this to true to enable DRS rule creation. Otherwise, set this to false.

Example:

antiAffinityGroups:
  enabled: true

If this field is true, Google Distributed Cloud creates VMware Distributed Resource Scheduler (DRS) anti-affinity rules for your admin cluster nodes, causing them to be spread across at least three physical ESXi hosts in your datacenter.

This feature requires that your vSphere environment meets the following conditions:

  • VMware DRS is enabled. VMware DRS requires vSphere Enterprise Plus license edition.

  • Your vSphere user account has the Host.Inventory.Modify cluster privilege.

  • There are at least four ESXi hosts available.

Even though the rule requires that the cluster nodes are spread across three ESXi hosts, we strongly recommend that you have at least four ESXi hosts available. This protects you from losing your admin cluster control plane. For example, suppose you have only three ESXi hosts, and your admin cluster control-plane node is on an ESXi host that fails. The DRS rule will prevent the control-plane node from being placed on one of the remaining two ESXi hosts.

Recall that if you have a vSphere Standard license, you cannot enable VMware DRS.

If you do not have DRS enabled, or if you do not have at least four hosts where vSphere VMs can be scheduled, set antiAffinityGroups.enabled to false.

adminMaster

Immutable

Configuration settings for the control-plane nodes in the admin cluster.

adminMaster.cpus

Preview
Optional
Immutable
Integer
Prepopulated: 4
Default: 4

The number of vCPUs for each control-plane node in the admin cluster.

Example:

adminMaster:
  cpus: 4

adminMaster.memoryMB

Preview
Optional
Immutable
Integer
Prepopulated: 16384
Default: 16384

The number of mebibytes of memory for each control-plane node in the admin cluster.

Example:

adminMaster:
  memoryMB: 16384

adminMaster.replicas

1.28 and higher

Required for new clusters
Immutable
Integer
Possible values: 3

The number of control-plane nodes in the admin cluster. In 1.28 and higher, new admin clusters must be highly available (HA). Set this field to 3 to create an HA admin cluster with 3 control plane nodes.

Example:

adminMaster:
  replicas: 3

1.16 and lower

Optional
Immutable
Integer
Possible values: 1 or 3
Prepopulated: 1
Default: 1

The number of control-plane nodes in the admin cluster. Set this to 3 if you want to create a high-availability admin cluster. Otherwise, set it to 1.

If you set this to 3, then you must also fill in the network.controlPlaneIPBlock section and the network.hostConfig section.

If you set this to 3, then you cannot use the Seesaw load balancer.

Example:

adminMaster:
  replicas: 3

adminMaster.topologyDomains

1.31 and higher

If infraConfigFilePath is configured (which enables topology domains), optionally include this field.

Array of strings | But only one element is supported
Immutable
Default: vSphereInfraConfig.defaultTopologyDomain if specified in the vSphere infrastructure configuration file

Admin cluster control-plane nodes will spread across the topology domains specified in this field.

1.30 and lower

Not available.

proxy

If your network is behind a proxy server, fill in this section. Otherwise, remove this section or leave it commented out. The proxy server you specify here is used by the user clusters that this admin cluster manages.
Immutable

proxy.url

Required if proxy section is filled in.
Immutable
String

The HTTP address of your proxy server. Include the port number even if it's the same as the scheme's default port.

Example:

proxy:
  url: "http://my-proxy.example.local:80"

The proxy server you specify here is used by your Google Distributed Cloud clusters. Also, your admin workstation is automatically configured to use this same proxy server unless you set the HTTPS_PROXY environment variable on your admin workstation.

If you specify proxy.url, you must also specify proxy.noProxy.

After the proxy configuration for the admin cluster has been set, it cannot be modified or deleted, unless the cluster is rebuilt.

proxy.noProxy

Optional
Immutable
String

A comma-separated list of IP addresses, IP address ranges, host names, and domain names that should not go through the proxy server. When Google Distributed Cloud sends a request to one of these addresses, hosts, or domains, the request is sent directly.

Example:

proxy:
  noProxy: "10.151.222.0/24,my-host.example.local,10.151.2.1"

privateRegistry

If you have a private container registry, fill in this section. Otherwise, remove this section or leave it commented out.

privateRegistry.address

Required for private registry
Immutable
String

The IP address or FQDN (Fully Qualified Domain Name) of the machine that runs your private registry.

Examples:

privateRegistry:
  address: "203.0.113.10"

privateRegistry:
  address: "fqdn.example.com"

privateRegistry.credentials.fileRef.path

Required for private registry
Mutable
String

The path of a credentials configuration file that holds the username and password of an account that Google Distributed Cloud can use to access your private registry.

Example:

privateRegistry:
  credentials:
    fileRef:
      path: "my-config-folder/admin-creds.yaml"

privateRegistry.credentials.fileRef.entry

Required for private registry
Mutable
String

The name of the credentials block, in your credentials configuration file, that holds the username and password of your private registry account.

privateRegistry:
  credentials:
    fileRef:
      entry: "private-registry-creds"

privateRegistry.caCertPath

Required for private registry
Mutable
String

When the container runtime pulls an image from your private registry, the registry must prove its identity by presenting a certificate. The registry's certificate is signed by a certificate authority (CA). The container runtime uses the CA's certificate to validate the registry's certificate.

Set this field to the path of the CA's certificate.

Example:

privateRegistry:
  caCertPath: "my-cert-folder/registry-ca.crt"

componentAccessServiceAccountKeyPath

Required
Mutable
String

The path of the JSON key file for your component access service account.

Example:

componentAccessServiceAccountKeyPath: "my-key-folder/access-key.json"

gkeConnect

Required
Mutable

When you fill in the gkeConnect section, the admin cluster is automatically registered to a fleet after it is created. This section holds information about the Google Cloud project and service account needed to register the cluster.

During cluster creation or update, several RBAC policies are configured on the admin cluster. These RBAC policies are needed so that you can create user clusters in the Google Cloud console.

gkeConnect.projectID

Required
Immutable
String

The ID of your fleet host project. For new clusters, this project ID must be the same as the ID set in stackdriver.projectID and cloudAuditLogging.projectID. If the project IDs aren't the same, cluster creation fails. This requirement isn't applied to existing clusters.

Example:

gkeConnect:
  projectID: "my-fleet-host-project"

gkeConnect.location

Optional
Immutable
String
Default: global

Each cluster's fleet membership is managed by the Fleet service (gkehub.googleapis.com) and the Connect service (gkeconnect.googleapis.com). The location of the services can be global or regional. In 1.28 and later, you can optionally specify the Google Cloud region in which the Fleet and the Connect services run. If not specified, the global instances of the services are used. Note the following:

  • Admin clusters created prior to 1.28 are managed by the global Fleet and Connect services.

  • For new clusters, if you include this field, the region that you specify must be the same as the region configured in cloudAuditLogging.clusterLocation, stackdriver.clusterLocation, and gkeOnPremAPI.location. If the regions aren't the same, cluster creation fails.

Example:

gkeConnect:
  location: "us-central1"

gkeConnect.registerServiceAccountKeyPath

Required
Mutable
String

The path of the JSON key file for your connect-register service account.

Example:

gkeConnect:
  registerServiceAccountKeyPath: "my-key-folder/connect-register-key.json"

gkeOnPremAPI

Optional

In 1.16 and later, if the GKE On-Prem API is enabled in your Google Cloud project, all clusters in the project are enrolled in the GKE On-Prem API automatically in the region configured in stackdriver.clusterLocation.

  • If you want to enroll all clusters in the project in the GKE On-Prem API, be sure to do the steps in Before you begin to activate and use the GKE On-Prem API in the project.

  • If you don't want to enroll the cluster in the GKE On-Prem API, include this section and set gkeOnPremAPI.enabled to false. If you don't want to enroll any clusters in the project, disable gkeonprem.googleapis.com (the service name for the GKE On-Prem API) in the project. For instructions, see Disabling services.

Enrolling your admin cluster in the GKE On-Prem API lets you use standard tools—the Google Cloud console, the Google Cloud CLI, or Terraform—to upgrade user clusters that the admin cluster manages. Enrolling your cluster also lets you run gcloud commands to get information about your clusters.

After you add this section and create or update the admin cluster, if subsequently you remove the section and update the cluster, the update will fail.

gkeOnPremAPI.enabled

Required if the gkeOnPremAPI section is included.
Mutable
Boolean
Default: true

By default, the cluster is enrolled in the GKE On-Prem API if the GKE On-Prem API is enabled in your project. Set to false if you don't want to enroll the cluster.

After the cluster is enrolled in the GKE On-Prem API, if you need to unenroll the cluster, make the following change and then update the cluster:

gkeOnPremAPI:
  enabled: false

gkeOnPremAPI.location

Immutable
String
Default:stackdriver.clusterLocation

The Google Cloud region where the GKE On-Prem API runs and stores cluster metadata. Choose one of the supported regions. You must use the same region that is configured in cloudAuditLogging.clusterLocation, gkeConnect.location, and stackdriver.clusterLocation. If gkeOnPremAPI.enabled is false, don't include this field.

stackdriver

Required by default
Mutable

If you want to enable Cloud Logging and Cloud Monitoring for your cluster, fill in this section. Otherwise remove this section.

This section is required by default. That is, if you don't include this section, you must include the --skip-validation-stackdriver flag when you run gkectl create admin.

This section is required in the admin cluster if you want to manage the lifecycle of user clusters using GKE On-Prem API clients.

stackdriver.projectID

Required for Logging and Monitoring
Immutable
String

The ID of your fleet host project. For new clusters, this project ID must be the same as the ID set in gkeConnect.projectID and cloudAuditLogging.projectID. If the project IDs aren't the same, cluster creation fails. This requirement isn't applied to existing clusters.

If needed, you can configure a log router in this project to route logs into log buckets in another project. For information on how to configure the log router, see Supported destinations.

Example:

stackdriver:
  projectID: "my-fleet-host-project"

stackdriver.clusterLocation

Required for Logging and Monitoring
Immutable
String
Prepopulated: "us-central1"

The Google Cloud region where you want to store logs. It is a good idea to choose a region that is near your on-premises data center.

For new clusters, if you include the gkeOnPremAPI and cloudAuditLogging sections in the configuration file, the region that you set here must be the same region that you set in cloudAuditLogging.clusterLocation, gkeConnect.location, and gkeOnPremAPI.location. If the regions aren't the same, cluster creation fails.

Example:

stackdriver:
  clusterLocation: "us-central1"

stackdriver.enableVPC

Optional
Immutable
Boolean
Prepopulated: false

If your cluster's network is controlled by a VPC, set this totrue. This ensures that all telemetry flows through Google's restricted IP addresses. Otherwise, set this to false.

Example:

stackdriver:
  enableVPC: false

stackdriver.serviceAccountKeyPath

Required for Logging and Monitoring
Mutable
String

The path of the JSON key file for your logging-monitoring service account.

If you will be creating the cluster with enableAdvancedCluster set to true (which is required for setting up topology domains), then stackdriver.serviceAccountKeyPath must be the same as cloudAuditLogging.serviceAccountKeyPath.

For information on how to update this field in an existing cluster, see Rotating service account keys.

Example:

stackdriver:
  serviceAccountKeyPath: "my-key-folder/log-mon-key.json"

stackdriver.disableVsphereResourceMetrics

Optional
Mutable
Relevant for Logging and Monitoring
Boolean
Prepopulated: false
Default: false

Set this to true to disable the collection of metrics from vSphere. Otherwise, set it to false.

This section is required in the admin cluster if you want to manage the lifecycle of user clusters in the Google Cloud console.

Example:

stackdriver:
  disableVsphereResourceMetrics: true

cloudAuditLogging

If you want to integrate the audit logs from your cluster's Kubernetes API server with Cloud Audit Logs, fill in this section. Otherwise, remove this section or leave it commented out.
Mutable

This section is required in the admin cluster if you want to manage the lifecycle of user clusters in using GKE On-Prem API clients.

cloudAuditLogging.projectID

Required for Cloud Audit Logs
Immutable
String

The ID of your fleet host project. For new clusters, this project ID must be the same as the ID set in gkeConnect.projecID and stackdriver.projectID. If the project IDs aren't the same, cluster creation fails. This requirement isn't applied to existing clusters.

If needed, you can configure a log router in this project to route logs into log buckets in another project. For information on how to configure the log router, see Supported destinations.

Example:

cloudAuditLogging:
  projectID: "my-fleet-host-project"

cloudAuditLogging.clusterLocation

Required for Cloud Audit Logs
Immutable
String

The Google Cloud region where you want to store audit logs. It is a good idea to choose a region that is near your on-premises data center

For new clusters, if you include the gkeOnPremAPI and stackdriver sections in the configuration file, the region that you set here must be the same region that you set in gkeConnect.location, gkeOnPremAPI.location, and stackdriver.clusterLocation. If the regions aren't the same, cluster creation fails.

Example:

cloudAuditLogging:
  clusterLocation: "us-central1"

cloudAuditLogging.serviceAccountKeyPath

Required for Cloud Audit Logs
Mutable
String

The path of the JSON key file for your audit-logging service account.

If you will be creating the cluster with enableAdvancedCluster set to true (which is required for setting up topology domains), then cloudAuditLogging.serviceAccountKeyPath must be the same as stackdriver.serviceAccountKeyPath.

For information on how to update this field in an existing cluster, see Rotating service account keys.

Example:

cloudAuditLogging:
  serviceAccountKeyPath: "my-key-folder/audit-log-key.json"

clusterBackup.datastore

Preview
Optional
Mutable
String

If you want to enable backing up of the admin cluster, set this to the vSphere datastore where you want to save cluster backups.

If you will be creating the cluster with enableAdvancedCluster set to true (which is required for setting up topology domains), then remove this section. Backing up the admin cluster to a vSphere datastore isn't supported.

Example:

clusterBackup:
  datastore: "my-datastore"

autoRepair.enabled

Optional
Mutable
Boolean
Prepopulated: true

Set this to true to enable node auto repair. Otherwise, set this to false.

To update the value of this field, use gkectl update admin.

Example:

autoRepair:
  enabled: true

secretsEncryption

If you want to encrypt Secrets without the need for an external KMS (Key Management Service), or any other dependencies, fill in this section. Otherwise, remove this section or leave it commented out.
Immutable

If you will be creating the cluster with enableAdvancedCluster set to true (which is required for setting up topology domains), then remove this section. This feature isn't supported with advanced clusters.

secretsEncryption.mode

Required for Secrets encryption
Immutable
String
Possible value: "GeneratedKey"
Prepopulated: "GeneratedKey"

The Secret encryption mode.

secretsEncryption:
  mode: "GeneratedKey"

secretsEncryption.generatedKey.keyVersion

Required for Secrets encryption
Mutable
Integer
Prepopulated: 1

An integer of your choice to use for the key version number. We recommended that you start with 1.

Example:

secretsEncryption:
  generatedKey:
    keyVersion: 1

secretsEncryption.generatedKey.disabled

Optional for Secrets encryption
Mutable
Boolean
Prepopulated: false

Set this to true to disable Secrets encryption. Otherwise set it to false.

Example:

secretsEncryption:
  generatedKey:
    disabled: false

osImageType

Optional
Mutable
String
Possible values: "ubuntu_containerd", "cos", "ubuntu_cgv2", "cos_cgv2"
Prepopulated: "ubuntu_cgv2"
Default: "ubuntu_containerd"

The type of OS image to run on your admin cluster nodes.

If you will be creating the cluster with enableAdvancedCluster set to true (which is required for setting up topology domains), only ubuntu-cgroupv2 and ubuntu_containerd are supported.

Example:

osImageType: "cos"