vSphere infrastructure configuration file

This page describes the fields in a vSphere infrastructure configuration file, which is used in the setup of topology domains. This file is required if infraConfigFilePath in the admin cluster configuration file is set.

The following sections provide descriptions of the objects and fields in the Secret and the VSphereInfraConfig custom resources. See the Example configuration file section that follows that shows example values for the fields.

All fields are required in the configuration file unless noted as "Optional".

Secret

Describes the vSphere credentials Secret which stores credentials of each vCenter Server. Add or delete Secrets as needed according to how many vCenter Server instances you are planning to use for the admin cluster and user clusters. Each object must be separated by ---. You can remove all Secrets from this configuration file after the admin cluster is created successfully.

Secret.name

Immutable
String

The name of the Secret. This name must match one of the names in the credentials.vCenters[i].secretName field in the VSphereInfraConfig custom resource.

Secret.namespace

Immutable
String
Prepopulated: kube-system

The namespace that the Secret is in. This namespace must match the corresponding credentials.vCenters[i].secretNamespace field in the VSphereInfraConfig custom resource.

stringData.admin-username and stringData.admin-password

The credentials of the vCenter user account that has the vCenter Server Administrator role or equivalent privileges. For more information, see vCenter user account privileges.

stringData.ca-crt

Mutable
Multiline string

The Base64-encoded CA certificate of the vCenter Server.

Follow the steps in Getting your vCenter CA root certificate until you run the command that shows the Base64-encoded certificate. Copy and paste the entire certificate into this field.

VSphereInfraConfig

The vSphere infrastructure configuration. This configuration will be stored in the admin cluster and shared by the admin cluster and all its managing user clusters. Each admin cluster should have only one vSphere infrastructure configuration.

VSphereInfraConfig.name

Immutable
String
Prepopulated: default

The name of the vSphere infrastructure configuration.

credentials

The credentials used by the vSphere infrastructure configuration.

credentials.vCenters

An array of objects, each of which contains the Secret references for each vCenter Server. Add or delete entries as needed. After the admin cluster is created, you can add a new credentials.vCenters object to the configuration file if needed, and then run gkectl update admin to update the configuration in the cluster.

credentials.vCenters[i].address

Immutable
String

The IP address or the hostname of your vCenter Server.

For more information, see Finding your vCenter Server address.

credentials.vCenters[i].secretName

Immutable
String

The Secret name. The name must match a name in the Secret custom resource.

credentials.vCenters[i].secretNamespace

Immutable
String
Prepopulated kube-system

The namespace that the Secret is in. The namespace must match the corresponding namespace in the Secret custom resource.

topologyDomains

An array of objects, each of which describes a topology domain configuration. Add or delete entries as needed. After the admin cluster is created, you can add a new topologyDomains object to the configuration file if needed, and then run gkectl update admin to update the configuration in the cluster.

topologyDomains[i].name

Immutable
String

The name of the topology domain.

topologyDomains[i].topologyLabels

A list of labels applied to the nodes within this topology domain that is used by the Kubernetes scheduler. Each label is a "key":"value" pair. We recommend that you specify only one label per topology domain.

Example:

topologyDomains:
- name: "td-01"
  topologyLabel: "topology.examplepetstore.com/zone": "zone-1"

For the key, you can use the Kubernetes cluster-level default constraint, "topology.kubernetes.io/zone", as described in Built-in default constraints.

After the cluster is created, the topology label that you specified is populated to labels of nodes in the topology domain. If you don't use "topology.kubernetes.io/zone" as the key, you will need to configure the topology key in the PodTemplate in your Deployment, StatefulSet, or ReplicaSet, as applicable.

For example, suppose you defined the key in the topology label as "topology.examplepetstore.com/zone". In the PodTemplate, you specify the key as the value for the topologySpreadConstraints.topologyKey field. This lets the Kubernetes scheduler distribute Pods across the topology domain to ensure high availability and prevent over-concentration in any single area in case of failure.

For more information on configuring topologySpreadConstraints, see Pod Topology Spread Constraints in the Kubernetes documentation.

topologyDomains[i].compute

Compute resources allocated for this topology domain.

topologyDomains[i].compute.vcenter

Immutable
String

The IP address or the hostname of your vCenter Server.

For more information, see Finding your vCenter Server address.

topologyDomains[i].compute.datacenter

Immutable
String

The relative path of a vSphere data center.

The value you specify is relative to the root folder named /.

If your data center is in the root folder, the value is the name of the data center.

topologyDomains[i].compute.cluster

Immutable
String

The relative path of a vSphere cluster that represents the ESXi hosts where your cluster VMs in this topology domain will run. This vSphere cluster represents a subset of the physical ESXi hosts in your vCenter data center.

The value you specify is relative to /.../DATA_CENTER/vm/.

If your vSphere cluster is in the /.../DATA_CENTER/vm/ folder, the value is the name of the vSphere cluster.

Otherwise, the value is a relative path that includes one or more folders along with the name of the vSphere cluster.

topologyDomains[i].compute.resourcePool

Immutable
String

A vCenter resource pool for your VMs in this topology domain.

If you want to use the default resource pool, set this to VSPHERE_CLUSTER/Resources.

Example:

topologyDomains:
- name: "td-01"
  compute:
    resourcePool: "my-vsphere-cluster/Resources"

If you want to use a resource pool that you have already created, set this to the relative path of your resource pool.

The value you specify is relative to /.../DATA_CENTER/host/.../VSPHERE_CLUSTER/Resources/

If your resource pool is a direct child of /.../DATA_CENTER/host/.../VSPHERE_CLUSTER/Resources/ the value is the name of the resource pool.

Example:

topologyDomains:
- name: "td-01"
  compute:
    resourcePool: "my-resource-pool"

Otherwise, the value is a relative path that has two or more resource pools.

Example:

topologyDomains:
- name: "td-01"
  compute:
    resourcePool: "resource-pool-1/resource-pool-2"

topologyDomains[i].compute.folder

Optional
Immutable
String

The relative path of a vSphere folder that you have already created. This folder will hold all VMs in this topology domain.

If you don't specify a value, all VMs in this topology domain will be put in /.../DATA_CENTER/vm/.

If you specify a value, it is relative to /.../DATA_CENTER/vm/.

The value can be the name of a folder.

Example:

topologyDomains:
- name: "td-01"
  compute:
    folder: "my-folder"

Or the value can be a relative path that includes more than one folder.

Example:

topologyDomains:
- name: "td-01"
  compute:
    folder: "folders/folder-1"

topologyDomains[i].storage

Storage configurations of this topology domain. It is used by virtual machine data disks and boot disks only. CNS volumes (provisioned by PVCs and PVs) aren't included.

Specify a value for either storage.policyName or storage.datastore, but not both.

topologyDomains[i].storage.policyName

Required if storage.datastore isn't specified
Immutable
String

The name of a VM storage policy for the cluster nodes.

For more information, see Configure a storage policy.

You must specify a value for either storage.datastore or storage.policyName, but not both. If you specify a value for this field, don't specify a value for storage.datastore.

topologyDomains[i].storage.datastore

Required if storage.policyName isn't specified
Immutable
String

The name of a vSphere datastore for your admin cluster.

The value you specify must be a name, not a path. Don't include any folders in the value.

Example:

topologyDomains:
- name: "td-01"
  storage:
    datastore: "my-datastore"

You must specify a value for either storage.datastore or storage.PolicyName, but not both. If you specify a value for this field, don't specify a value for storage.PolicyName.

topologyDomains[i].network

Network configurations of this topology domain.

topologyDomains[i].network.vsphereNetwork

Immutable
String

The name of the vSphere network for your cluster nodes.

Example:

topologyDomains:
- name: "td-01"
  network:
    vsphereNetwork: "my-network"

If the name contains a special character, you must use an escape sequence for it.

Special characters Escape sequence
Slash (/) %2f
Backslash (\) %5c
Percent sign (%) %25

If the network name is not unique in your data center, you can specify a full path.

Example:

topologyDomains:
- name: "td-01"
  network:
    vsphereNetwork: "/data-centers/data-center-1/network/my-network"

topologyDomains[i].network.gateway

The gateway address of the subnet for cluster nodes in this topology domain. The IP address must be the same as the gateway address in the IP block file. For an example, see the example IP block file for topology domains.

topologyDomains[i].network.dnsServers

Immutable
Array of strings.
The maximum number of elements in the array is three.

The IP addresses of DNS servers for the VMs.

topologyDomains[i].network.ntpServers

Immutable
Array of strings

The IP addresses of time servers for the VMs to use.

topologyDomains[i].network.searchDomains

Optional
Immutable
Array of strings

DNS search domains for the VMs to use. These domains are used as part of a domain search list.

Example:

topologyDomains:
- name: "td-01"
  network:
    hostConfig:
      searchDomainsForDNS:
      - "my.local.com"

topologyDomains[i].defaultTopologyDomain

Immutable
Optional
String

The name of the default topology domain. This name must match one of the names in topologyDomains[i].name. If you don't specify a default name at cluster creation time, you can update the configuration file with a name if needed, and then run gkectl update admin to update the configuration in the cluster.

Nodes will be allocated into the default topology domain unless you set a topology domain for control-plane nodes or nodes in a node pool.

Example configuration file

The following example shows a configuration file with values filled in.

apiVersion: v1
kind: Secret
metadata:
  namespace: kube-system
  name: "vsphere-auth-0"
stringData:
  admin-username: "example-admi1n@vsphere.local"
  admin-password: "GltFl4@5f"
  ca-crt: |
    -----BEGIN CERTIFICATE-----
    MIIEITCCAwmgAwIBAgIJAPFVaTIKdyCbMA0GCSqGSIb3DQEBCwUAMIGbMQswCQYD
    VQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZ
    FgVsb2NhbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExHTAbBgNV
    BAoMFGF0bC1xdWFsLXZjMDMuYW50aG9zMRswGQYDVQQLDBJWTXdhcmUgRW5naW5l
    ZXJpbmcwHhcNMjIwODA3MDIzNTQ4WhcNMzIwODA0MDIzNTQ4WjCBmzELMAkGA1UE
    AwwCQ0ExFzAVBgoJkiaJk/IsZAEZFgd2c3BoZXJlMRUwEwYKCZImiZPyLGQBGRYF
    bG9jYWwxCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMR0wGwYDVQQK
    DBRhdGwtcXVhbC12YzAzLmFudGhvczEbMBkGA1UECwwSVk13YXJlIEVuZ2luZWVy
    aW5nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAz5MOjTQ4FHcIYjzo
    bcIFJmnVDjSnLGYDdVayhmUo1+CC5RqJRuAnJmDYt9PY/lJhVDVUSFHWJdvRjeAP
    0+xjRs33VJctzjDqwy2xRMVxoA+c8DfvALHIK+rHwIidGNt5eb2HOrB1MSCB5KFi
    gkXhJf/2kiEPiTMqbu2nWySdBfZRVSYs1jSiAuz/cgyglfVUcRFn4oRhR35QZ8gj
    t54IGY9atzKCYuvFAgeTCGBxI30cZRQDCtkWyWHPdK3GniNW/qh4C7gC+VI+Pnh+
    fUGtAx3Px/O5h5RpHDmvuSqMNoWEsChKGgbrbkZGB9OWL8OsVAUnz8TXC6s4/AG1
    0GGDGQIDAQABo2YwZDAdBgNVHQ4EFgQUM8RXjNIlar+S6uZBSH+3oVkUdGwwHwYD
    VR0RBBgwFoEOZW1haWxAYWNtZS5jb22HBH8AAAEwDgYDVR0PAQH/BAQDAgEGMBIG
    A1UdEwEB/wQIMAYBAf8CAQAwDQYJKoZIhvcNAQELBQADggEBABIO4RO3Jk6PaDyt
    +PGP1ng5fpcKbwCi90KfEkfAd/HV3YM6suEhCar6azKXJutL9gGzidi9NvKv43Pg
    hOOuTQWbdwuhqgF3YPKZ1HJWSV0NR5J9WFjp9rk1EdM6NApUwmy7EIbo8TY7M6SL
    jaR8VgaelG2PSf2S4ybxKdJ2gx2pPTcoYpd/+d9qaDFLNWa4irzB6sqBUVlkAMX+
    JW4FmMs8ym4aLzlFBzG5gjK1PCsBugbVRSAbF+Y8LwnMoe2WMiF1nNXoKSiaQm0A
    Z0psC1b37G9dKzr9soLS5OENFrAEClfVD5luETvUHo51VhanetWbZmL5dK8iGAuj
    otykS2w=
    -----END CERTIFICATE-----

---
apiVersion: v1
kind: Secret
metadata:
  namespace: kube-system
  name: "vsphere-auth-1"
stringData:
  admin-username: "example-admin2@vsphere.local"
  admin-password: "xuBAcK5lT@uq2Ix"
  ca-crt: |
    -----BEGIN CERTIFICATE-----
    MIIEITCCAwmgAwIBAgIJAMJPzGX27yKBMA0GCSqGSIb3DQEBCwUAMIGbMQswCQYD
    VQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZ
    FgVsb2NhbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExHTAbBgNV
    BAoMFGF0bC1xdWFsLXZjMTMuYW50aG9zMRswGQYDVQQLDBJWTXdhcmUgRW5naW5l
    ZXJpbmcwHhcNMjMxMjAzMjM0NDM5WhcNMzMxMTMwMjM0NDM5WjCBmzELMAkGA1UE
    AwwCQ0ExFzAVBgoJkiaJk/IsZAEZFgd2c3BoZXJlMRUwEwYKCZImiZPyLGQBGRYF
    bG9jYWwxCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMR0wGwYDVQQK
    DBRhdGwtcXVhbC12YzEzLmFudGhvczEbMBkGA1UECwwSVk13YXJlIEVuZ2luZWVy
    aW5nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6d2PidzORMEOuIKa
    XOXB4nB7ukYGOeju1nKJ90KnvKfZBNjN1JKuNPQcZelFyEcUEHnI18cA2VGbpXbF
    O9jMlxu5VlhPepjEPAgtcAuCZNlxzRDip9u9ar3ma2SxMCogz91qDZKSE/PQpqNI
    Ozpus6YsHmK3jzAUq/Zs1+wl9HFot0e5kPf79oqQC8gPxJqmFQyGm50SdOMwj5sD
    pYm/3TZZ1gw4WftBhDecO9hKwcReCVjeVgp5omMcFHwIhqUBqZM3qX5gJaxE7ELO
    xu66VibA1g70Xzso2hGVQ13ruGjgUAjDsICOTpHHcT33KoMRzn2BWlKtmsnThIrE
    nbR9QQIDAQABo2YwZDAdBgNVHQ4EFgQUI84+93Iniliyzs3HSDatbYGsiJ4wHwYD
    VR0RBBgwFoEOZW1haWxAYWNtZS5jb22HBH8AAAEwDgYDVR0PAQH/BAQDAgEGMBIG
    A1UdEwEB/wQIMAYBAf8CAQAwDQYJKoZIhvcNAQELBQADggEBAIIihWWIcl+sKRmS
    QB4wkjR9UA7QV96lbm+HFE5qvP5xVXss+PmSjLjeNZFfX65tGlKXWDmYstcw/jon
    hnJVe/jxjIZ3GddrVfHdwOEQ+FORQyfxkLAQuHKvm/2PobGNsU0rOw14Ur4Eea6P
    pIl70eJhRX22dkOhb+y6jz+k+L52HbzMz/X6yUnTFAJzobJD0C1vGAx3FgPdbUqF
    fXYO/Q8NNrk+4561eT3x9T4RIj/0ZkBal0yQEdYnd+lOIkEmk7vmO1/eyb6+66rs
    3wz4DIxtfhJIUlj0mBL0KkyDDDoc0oFTf5V/xrYls1dJ35DNFoG/KBJGgGbaW41t
    vTz+0JA=
    -----END CERTIFICATE-----

---
apiVersion: v1
kind: Secret
metadata:
  namespace: kube-system
  name: "vsphere-auth-2"
stringData:
  admin-username: "example-admin3@vsphere.local"
  admin-password: "nWKCITwqKTaY5@M5"
  ca-crt: |
    -----BEGIN CERTIFICATE-----
    MIIEITCCAwmgAwIBAgIJAN4alqihWQukMA0GCSqGSIb3DQEBCwUAMIGbMQswCQYD
    VQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZ
    FgVsb2NhbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExHTAbBgNV
    BAoMFGF0bC1xdWFsLXZjMDcuYW50aG9zMRswGQYDVQQLDBJWTXdhcmUgRW5naW5l
    ZXJpbmcwHhcNMjMwMzI2MTc0NjE1WhcNMzMwMzIzMTc0NjE1WjCBmzELMAkGA1UE
    AwwCQ0ExFzAVBgoJkiaJk/IsZAEZFgd2c3BoZXJlMRUwEwYKCZImiZPyLGQBGRYF
    bG9jYWwxCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMR0wGwYDVQQK
    DBRhdGwtcXVhbC12YzA3LmFudGhvczEbMBkGA1UECwwSVk13YXJlIEVuZ2luZWVy
    aW5nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuvM3aSemSrBxWVei
    6pYlRvLWaCva2Q0MQvmUdVISa9uul/PHE2lwAcTAAr3Sayat2D5IRjd7NWbzaLVQ
    Sxel0TINpMfITDT0x6lZ8FfKgrRpY30CPZQlgVo16rGKE0V+wHTkGYIzRITETzuu
    z6nkpzvryh6JadSf4wrGj1TmrYpqYEmdzf7bLmz4Rql5PgVco7z+pqo0BA+UU/Uc
    /OtEcVKAyaifHzZwPlscOpKpskQrP4Ke6i3iuyrEmaDH+4sQ1TDrcYpUBfqkl+xT
    0G6ZKHs6KP18x0Xg4ifeC8x7sgajJ20TpbkDY4KtBdraj3HB4gZCxg7qY/0G75yF
    C2bZ4wIDAQABo2YwZDAdBgNVHQ4EFgQUeE+R+gRo1oU578dyiyq4tPYefpIwHwYD
    VR0RBBgwFoEOZW1haWxAYWNtZS5jb22HBH8AAAEwDgYDVR0PAQH/BAQDAgEGMBIG
    A1UdEwEB/wQIMAYBAf8CAQAwDQYJKoZIhvcNAQELBQADggEBADzBUXsxICPGxt1o
    wPrbmWhkrPy2pNr9unP/d+Py20groZ/KQ/d6dzDN7pxc6BtQbMfho7pij+mf3aHq
    oBo+J80Ha/vZExMMwYrp/OTlxy2Jdb5pzxZQtWqNVS5SkzNtDs7DhXDTY19TBZhZ
    4d+/g2eSq2/dRT7zF8q3LH6mmq0EbSRSDOm8r6AUdWt+Ov7YKTYk9c+5UTL9M9u4
    1+VBkF34hB/xc1fsFSPHTQQDOlAy5HTZgM3ygtuhmAY+UUQVLZf+8sJh+5zqc6gw
    Ecnz7HRcRPUitKACsxUm4f5VHiKUMrIBP5U1dovl4kUN8Ue0e+KnIjIGKjp4wrHi
    UtQboUw=
    -----END CERTIFICATE-----

---
apiVersion: vmware.cluster.gke.io/v1alpha1
kind: VSphereInfraConfig
metadata:
  name: default
credentials:
  vCenters:
  - address: "vc03.example"
    secretName: "vsphere-auth-0"
    secretNamespace: "kube-system"
  - address: "vc13.example"
    secretName: "vsphere-auth-1"
    secretNamespace: "kube-system"
  - address: "vc07.example"
    secretName: "vsphere-auth-2"
    secretNamespace: "kube-system"
topologyDomains:
- name: "topology-domain-0"
  topologyLabels:
    "topology.kubernetes.io/zone": "zone-0"
  compute:
    vcenter: "vc03.example"
    datacenter: "vc03"
    cluster: "workloads3"
    resourcePool: "vc03-default-pool"
  storage:
    datastore: "vsanDatastore"
  network:
    vsphereNetwork: "qual-343"
    gateway: "203.0.113.1"
    dnsServers:
    - "192.0.2.1"
    - "192.0.2.2"
    ntpServers:
    - "203.0.113.50"
- name: "topology-domain-1"
  topologyLabels:
    "topology.kubernetes.io/zone": "zone-1"
  compute:
    vcenter: "vc13.example"
    datacenter: "vc13-dc1"
    cluster: "workloads13-1"
    resourcePool: "vc13-dc1-default-pool"
  storage:
    datastore: "vsanDatastore-1c"
  network:
    vsphereNetwork: "scale-332"
    gateway: "203.0.113.2"
    dnsServers:
    - "192.0.2.10"
    - "192.0.2.11"
    ntpServers:
    - "203.0.113.51"
- name: "topology-domain-2"
  topologyLabels:
    "topology.kubernetes.io/zone": "zone-2"
  compute:
    vcenter: "vc07.example"
    datacenter: "vc07"
    cluster: "workloads7"
    resourcePool: "vc07-default-pool"
  storage:
    datastore: "vsanDatastore"
  network:
    vsphereNetwork: "qual-323"
    gateway: "203.0.113.3"
    dnsServers:
    - "192.0.2.21"
    - "192.0.2.22"
    ntpServers:
    - "203.0.113.52"