User cluster configuration file

This page describes the fields in the user cluster configuration file.

Generating a template for your configuration file

If you used gkeadm to create your admin workstation, then gkeadm generated a template for your user cluster configuration file. And gkdadm filled in some of the fields for you.

If you did not use gkeadm to create your admin workstation, you can use gkectl to generate a template for your user cluster configuration file.

To generate a template for your user cluster configuration file:

gkectl create-config cluster --config=[OUTPUT_FILENAME]

where [OUTPUT_FILENAME] is a path of your choice for the generated template. If you omit this flag, gkectl names the file user-cluster.yaml and puts it in the current directory.

Filling in your configuration file

In your configuration file, enter field values as described in the following sections.

name

String. A name of your choice for your user cluster. For example:

name: "my-user-cluster"

gkeOnPremVersion

String. The GKE on-prem version for your user cluster. For example:

gkeOnPremVersion: 1.4.3-gke.3

vCenter

If you want all aspects of your vCenter environment to be the same as what you specified for your admin cluster, remove this section or leave it commented out.

If you want some aspects of your vCenter environment to be different from what you specified for your admin cluster, fill in the relevant fields in this section. Any fields that you set here in the vCenter section override the corresponding fields in your admin cluster configuration file.

vCenter.credentials.address

String. The IP address or the hostname of the vCenter server for your user cluster.

Before you fill in the address field, download and inspect the serving certificate of your vCenter server. Enter the following command to download the certificate and save it to a file named vcenter.pem.

true | openssl s_client -connect [VCENTER_IP]:443 -showcerts 2>/dev/null | sed -ne '/-BEGIN/,/-END/p' > vcenter.pem

where [VCENTER_IP] is the IP address of your vCenter Server.

Open the certificate file to see the Subject Common Name and the Subject Alternative Name:

openssl x509 -in vcenter.pem -text -noout

The output shows the Subject Common Name (CN). This might be an IP address, or it might be a hostname. For example:

Subject: ... CN = 203.0.113.101
 
Subject: ... CN = my-user-vc-server.my-domain.example

The output might also include one or more DNS names under Subject Alternative Name:

X509v3 Subject Alternative Name:
    DNS:vcenter.my-domain.example

Choose the Subject Common Name or one of the DNS names under Subject Alternative Name to use as the value of vcenter.credentials.address in your configuration file. For example:

vCenter:
  credentials:
    address: "203.0.113.101"
    ...

vCenter:
  credentials:
    address: "my-user-vc-server.my-domain.example"
    ...

vCenter.credentials.username

String. A vCenter Server user account for accessing your user cluster's vCenter server. The user account should have the Administrator role or equivalent privileges. See vSphere requirements.

For example:

vCenter:
  credentials:
    username: "my-user-account@vsphere.local"

vCenter.credentials.password

String. The password for the vCenter Server user account. For example:

vCenter:
  credentials:
    password: "#STyZ2T#Ko2o"

vCenter.datastore

String. The name of the vCenter datastore for your user cluster. For example:

vCenter:
  datastore: "MY-USER-DATASTORE"

vCenter.resourcePool

String. The name of the vCenter resource pool for your user cluster. If you are using a non-default resource pool, provide the name of your vCenter resource pool.

For example:

vCenter:
  resourcePool: "MY-USER-POOL"

If you are using the default resource pool, provide the following value:

vCenter:
  resourcePool: "[VCENTER_CLUSTER]/Resources"

where [VCENTER_CLUSTER] is the name of your vCenter cluster.

See Specifying the root resource pool for a standalone host.

vCenter.caCertPath

String. When a client, like GKE on-prem, sends a request to your vCenter server, the server must prove its identity to the client by presenting a certificate or a certificate bundle. To verify the certificate or bundle, GKE on-prem must have the root certificate in the chain of trust.

Set vCenter.caCertPath to the path of the root certificate. For example:

vCenter:
  caCertPath: "/usr/local/google/home/me/certs/user-vcenter-ca-cert.pem"

Your VMware installation has a certificate authority (CA) that issues a certificate to your vCenter server. The root certificate in the chain of trust is a self-signed certificate created by VMware.

If you do not want to use the VMWare CA, which is the default, you can configure VMware to use a different certificate authority.

If your vCenter server uses a certificate issued by the default VMware CA, download the certificate as follows:

curl -k "https://[SERVER_ADDRESS]/certs/download.zip" > download.zip

where [SERVER_ADDRESS] is the address of your vCenter server.

Install the unzip command and unzip the certificate file:

sudo apt-get install unzip
unzip downloads.zip

If the unzip command doesn't work the first time, enter the command again.

Find the certificate file in certs/lin.

network

This section holds information about your user cluster network.

network.ipMode.type

String. If you want your cluster nodes to get their IP address from a DHCP server, set this to "dhcp". If you want your cluster nodes to have static IP addresses chosen from a list that you provide, set this to"static". For example:

network:
  ipMode:
    type: "static"

network.ipBlockFilePath

If you set ipMode.type to "static", fill in this field.

If you set ipMode.type to "dhcp", remove this field or leave it commented out.

String. The path of the hostconfig file for your cluster. For example:

network:
  ipBlockFilePath: "/my-config-directory/user-hostconfig.yaml"

network.serviceCIDR and network.podCiDR

Strings. Your user cluster must have a range of IP addresses to use for Services and a range of IP addresses to use for Pods. These ranges are specified by the network.serviceCIDR and network.podCIDR fields. These fields are populated with default values. If you like, you can change the populated values to values of your choice.

The Service and Pod ranges must not overlap. Also, the Service and Pod ranges must not overlap with IP addresses that are used for nodes in any cluster.

Example:

network:
  serviceCIDR: "10.96.232.0/24"
  podCIDR: "192.168.0.0/16"

network.vCenter.networkName

String. The name of the vSphere network for your user cluster nodes.

If the name contains a special character, you must use an escape sequence for it.

Special characters Escape sequence
Slash (/) %2f
Backslash (\) %5c
Percent sign (%) %25

If the network name is not unique, it is possible to specify a path to the network, such as /DATACENTER/network/NETWORK_NAME.

For example:

network:
  vCenter:
    networkName: "MY-USER-CLUSTER-NETWORK"

loadBalancer

This section holds information about the load balancer for your user cluster.

loadBalancer.vips.controlPlaneVIP

The IP address that you have chosen to configure on the load balancer for the Kubernetes API server of the admin cluster. For example:

loadBalancer:
  vips:
    controlplaneVIP: "203.0.113.3"

loadBalancer.vips.ingressVIP

The IP address that you have chosen to configure on the load balancer for ingress traffic. For example:

loadBalancer:
  vips:
    ingressVIP: "203.0.113.4"

loadBalancer.kind

String. Set this to "Seesaw", "F5BigIP", or "ManualLB" For example:

loadBalancer:
  kind: "Seesaw"

loadBalancer.manualLB

If you set loadbalancer.kind to "manualLB", fill in this section. Otherwise, remove this section or leave it commented out.

loadBalancer.manualLB.ingressHTTPNodePort

Integer. The ingress service in the admin cluster is implemented as a Service of type NodePort. The Service has a ServicePort for HTTP. You must choose a nodePort value for the HTTP ServicePorts.

Set this field to the nodePort value. For example:

loadBalancer:
  manualLB:
    ingressHTTPNodePort: 32527

loadBalancer.manualLB.ingressHTTPSNodePort

Integer. The ingress service in the admin cluster is implemented as a Service of type NodePort. The Service has a ServicePort for HTTPS. You must choose a nodePort value for the HTTPS ServicePort.

Set this field to the nodePort value. For example:

loadBalancer:
  manualLB:
    ingressHTTPSNodePort: 30139

loadBalancer.manualLB.controlPlaneNodePort

Integer. The Kubernetes API server in the admin cluster is implemented as a Service of type NodePort. You must choose a nodePort value for the Service.

Set this field to the nodePort value. For example:

loadBalancer:
  manualLB:
    contolPLaneNodePort: 30968

loadBalancer.manualLB.addonsNodePort

Integer. The add-ons server is implemented as a Service of type NodePort. You must choose a nodePort value for the Service.

Set this field to the nodePort value. For example:

loadBalancer:
  manualLB:
    addonsNodePort: 31405

loadBalancer.f5BigIP

If you set loadbalancer.kind to "f5BigIP", fill in this section. Otherwise, remove this section or leave it commented out.

loadBalancer.f5BigIP.credentials.address

String. The address of your F5 BIG-IP load balancer. For example:

loadBalancer:
  f5BigIP:
    credentials:
      address: "203.0.113.2"

loadBalancer.f5BigIP.credentials.username

String. The username of an account that GKE on-prem can use to connect to your F5 BIG-IP load balancer. For example:

loadBalancer:
  f5BigIP:
    credentials:
      username: "my-admin-f5-name"

loadBalancer.f5BigIP.credentials.password

String. The password that GKE on-prem can use to connect to your F5 BIG-IP load balancer. For example:

loadBalancer:
  f5BigIP:
    credentials:
      password: "rJDlm^%7aOzw"

loadBalancer.f5BigIP.partition

String. The name of a BIG-IP partition that you created for your admin cluster. For example:

loadBalancer:
  f5BigIP:
    partition: "my-f5-admin-partition"

loadBalancer.f5BigIP.snatPoolName

String. If you are using SNAT, the name of your SNAT pool. If you are not using SNAT, remove this field or leave it commented out. For example:

loadBalancer:
  f5BigIP:
    snatPoolName: "my-snat-pool"

loadBalancer.seesaw

If you set loadbalancer.kind to "Seesaw", fill in this section. Otherwise, remove this section or leave it commented out.

loadBalancer.seesaw.ipBlockFilePath

String. Set this to the path of the hostconfig file for your Seesaw VM. For example:

loadbalancer:
  seesaw:
    ipBlockFilePath: "admin-seesaw-hostconfig.yaml"

loadBalancer.seesaw.vird

Integer. The virtual router identifier of your Seesaw VM or VM pair. This identifier must be unique in a VLAN. Valid range is 1-255. For example:

loadBalancer:
  seesaw:
    vrid: 125

loadBalancer.seesaw.masterIP

String. A VIP of your choice that your control plane Seesaw VM will advertise. For example:

loadBalancer:
  seesaw:
    masterIP: 172.16.20.21

loadBalancer.seesaw.cpus

Integer. The number of CPUs for each of your Seesaw VMs. For example:

loadBalancer:.
  seesaw:
    cpus: 8

loadBalancer.seesaw.memoryMB

Integer. The number of megabytes of memory for each of your Seesaw VMs. For example:

loadBalancer:.
  seesaw:
    memoryMB: 8192

loadBalancer.seesaw.vCenter.networkName

String. The name of the network that contains your Seesaw VMs. For example:

loadBalancer:
  seesaw:
    vCenter:
      networkName: "my-seesaw-network"

loadBalancer.seesaw.enableHA

Boolean. If you want to create a highly-available Seesaw load balancer, set this to true. Otherwise set this to false. For example:

loadBalancer:.
  seesaw:
    enableHA: true

masterNode

This section holds information about the nodes, in the admin cluster, that serve as control plane nodes for your user cluster.

masterNode.cpus

Integer. The number of CPUs for each admin cluster node that serve as control planes for this user cluster. For example:

masterNode:
  cpus: 8

masterNode.memoryMB

Integer. The megabytes of memory for each admin cluster node that serves as a control plane for this user cluster. For example:

masterNode:
  memoryMB: 8192

masterNode.replicas

Integer. The number of control plane nodes for this user cluster. Set this field to 1 or 3. For example:

masterNode:
  replicas: 3

nodePools

Array of objects, each of which describes a node pool.

nodePools[i].name

String. A name of your choice for the node pool. For example:

nodePools:
- name: "my-node-pool"

noodePools[i].labels

Mapping. Labels to apply to each node in the pool. For example:

nodePools:
- name: "my-node-pool"
  labels:
    environment: "production"
    tier: "cache"

nodePools[i].taints

Array of objects, each of which describes a taint. For example:

nodePools:
- name: "my-node-pool"
  taints:
  - key: "staging"
    value: "true"
    effect: "NoSchedule"

nodePools[i].cpus

Integer. The number of CPUs for each node in the pool. For example:

nodePools"
- name: "my-node-pool"
  cpus: 8

nodePools[i].memoryMB

Integer. The megabytes of memory for each node in the pool. For example:

nodePools"
- name: "my-node-pool"
  memoryMB: 8192

nodePools[i].replicas

Integer. The number of nodes in the pool. For example:

nodePools:
- name: "my-node-pool"
  replicas: 5

nodePools[i].vsphere.datastore

String. The name of the vCenter datastore on which each node in the pool will be created. For example:

nodePools:
- name: "my-node-pool"
  vsphere:
    datastore: "my-datastore"

antiAffinityGroups.enabled

Boolean. Set this to true to enable DRS rule creation. Otherwise, set this to false. For example:

antiAffinityGroups:
  enabled: true

authentication

This section holds information about how cluster users are authenticated and authorized.

authentication.oidc

If you want to use OpenID Connect (OIDC) to manage access to this cluster, fill in this section. Otherwise, remove this section or leave it commented out.

authentication.oidc.issuerURL

String. The URL of your OpenID provider. Client applications, like the gcloud CLI and Google Cloud console, send authorization requests to this URL. The Kubernetes API server uses this URL to discover public keys for verifying tokens. Must use HTTPS. For example:

authentication:
  oidc:
    issuerURL: "https://example.com/adfs"

authentication.oidc.kubectlRedirectURL

String. The redirect URL for the gcloud CLI. For example:

authentication:
  oidc:
    kubectlRedirectURL: "https://localhost:1025/callback"

authentication.oidc.clientID

String. ID for the client application that makes authentication requests to the OpenID provider. Both the gcloud CLI and Google Cloud console use this ID. For example:

authentication:
  oidc:
    clientID: "my-big-hex-string"

authentication.oidc.clientSecret

String. Secret for the client application. Both the gcloud CLI and Google Cloud console use this secret. For example:

authentication:
  oidc:
    clientSecret: "N3i&JlLZoD!W"

authentication.oidc.username

String. The JWT claim to use as the user's name. The default is sub, which is expected to be a unique identifier of the end user. You can choose other claims, such as email or name, depending on the OpenID provider. However, claims other than email are prefixed with the issuer URL to prevent naming clashes with other plugins. For example:

authentication:
  oidc:
    username: "sub"

authentication.oidc.usernamePrefix

String. Prefix prepended to username claims to prevent clashes with existing names. If you do not provide this field, and username is a value other than email, the prefix defaults to issuerurl#. You can use the value - to disable all prefixing. For example:

authentication:
  oidc:
    usernamePrefix: "my-prefix"

authentication.oidc.group

String. The JWT claim that the provider will use to return your security groups. For example:

authentication:
  oidc:
    group: "sec-groups"

authentication.oidc.groupPrefix

String. Prefix prepended to group claims to prevent clashes with existing names. For example, given a group foobar and a prefix gid-, gid-foobar. By default, this value is empty, and there is no prefix. For example:

authentication:
  oidc:
    groupPrefix: "gid-"

authentication.oidc.scopes

String. A comma-delimited list of additional scopes to send to the OpenID provider. For example:

authentication:
  oidc:
    scopes: "offline-access"

For authentication with Microsoft Azure or Okta, set this to offline_access.

authentication.oidc.extraParams

String. A comma-delimited list of additional key-value parameters to send to the OpenID provider.

For example:

authentication:
  oidc:
    extraparams: "prompt=consent"

authentication.oidc.deployCloudConsoleProxy

String. Specifies whether to deploy a reverse proxy in the cluster to allow Google Cloud console access to the on-premises OIDC provider for authenticating users. Mmust be a string: "true" or "false". If your identity provider is not reachable over the public internet, and you wish to authenticate using Google Cloud console, then you must set this field to "true". If left blank, this field defaults to "false".

authentication.oidc.caPath

String. Path to the certificate for the certificate authority (CA) that issued your identity provider's web certificate. This value might not be necessary. For example, if your identity provider's certificate was issued by a well-known public CA, then you would not need to provide a value here. However, if deployCloudConsoleProxy is "true", then you must provide this value, even for a well-known public CA.

For example:

authentication:
  oidc:
    caPath: "my-cert-folder/provider-root-cert.pem"

authentication.sni

If you want to provide an additional serving certificate for the cluster's Kubernetes API server, fill in this section. Otherwise, remove this section or leave it commented out.

authentication.sni.certPath

String. The path to a serving certificate for the Kubernetes API server. For example:

authentication:
  sni:
    certPath: "my-cert-folder/example.com.crt"

authentication.sni.keyPath

String. Path to the certificate's private key file. For example:

authentication:
  sni:
    keyPath: "my-cert-folder/example.com.key"

stackdriver

This section holds information about the Google Cloud project and service account you want to use for storing logs and metrics.

stackdriver.projectID

String. The ID of the Google Cloud project where you want to view logs. For example:

stackdriver:
  projectID: "my-logs-project"

stackdriver.clusterLocation

String. The Google Cloud region where you want to store logs. It is a good idea to choose a region that is near your on-premises data center. For example:

stackdriver:
  clusterLocation: "us-central1"

stackdriver.enableVPC

Boolean. If your cluster's network is controlled by a VPC, set this to this field totrue. This ensures that all telemetry flows through Google's restricted IP addresses. Otherwise, set this field to false. For example:

stackdriver:
  enableVPC: false

stackdriver.serviceAccountKeyPath

String. The path of the JSON key file for your logging-monitoring service account. For example:

stackdriver:
  serviceAccountKeyPath: "my-key-folder/log-mon-key.json"

cloudAuditLogging

If you want to integrate the audit logs from your cluster's Kubernetes API server with Cloud Audit Logs, fill in this section. Otherwise, remove this section or leave it commented out.

cloudAuditLogging.projectid`

String. The project ID of the Google Cloud project where you want to store audit logs. For example:

cloudAuditLogging:
  projectid: "my-audit-project"

cloudAuditLogging.clusterlocation

String. The Google Cloud region where you want to store audit logs. It is a good idea to choose a region that is near your on-premises data center. For example:

cloudAuditLogging:
  clusterlocation: "us-central1"

cloudAuditLogging.serviceaccountkeypath

String. The path of the JSON key file for your audit-logging service account. For example:

cloudAuditLogging:
  serviceaccountkeypath: "my-key-folder/audit-log-key.json"

gkeConnect

This section holds information about the Google Cloud project and service accounts you want to use to connect your cluster to Google Cloud.

gkeConnect.projectID

String. The ID of the Google Cloud project that you want to use for connecting your cluster to Google Cloud. For example:

gkeConnect:
  projectID: "my-connect-project-123"

gkeConnect.registerServiceAccountKeyPath

String. The path of the JSON key file for your connect-register service account. For example:

gkeConnect:
  registerServiceAccountKeyPath: "my-key-folder/connect-register-key.json"

gkeConnect.agentServiceAccountKeyPath

String. The path of the JSON key file for your connect-agent service account. For example:

gkeConnect:
  agentServiceAccountKeyPath: "my-key-folder/connect-agent-key.json"

usageMetering

If you want to enable usage metering for your cluster, then fill in this section. Otherwise, remove this section or leave it commented out.

usageMetering.bigQueryProjectID

String. The ID of the Google Cloud project where you want to store usage metering data. For example:

usageMetering:
  bigQueryProjectID: "my-bq-project"

usageMetering.bigQueryDatasetID

String. The ID of the BigQuery dataset where you want to store usage metering data. For example:

usageMetering:
  bigQueryDatasetID: "my-bq-dataset"

usageMetering.bigQueryServiceAccountKeyPath

String. The path of the JSON key file for your BigQuery service account. For example:

usageMetering:
  bigQueryServiceAccountKeyPath: "my-key-folder/bq-key.json"

usageMetering.enableConsumptionMetering

Boolean. Set this to true if you want to enable consumption-based metering. Otherwise set this to false. For example:

usageMetering:
  enableConsumptionMetering: true

cloudRun.enabled

Boolean. Set this to true if you want to enable Cloud Run. Otherwise set this to false. For example:

cloudRun:
  enabled: true