Create basic clusters

Linux and macOS

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member="user:ACCOUNT" \
    --role="roles/resourcemanager.projectIamAdmin"

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member="user:ACCOUNT" \
    --role="roles/serviceusage.serviceUsageAdmin"

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member="user:ACCOUNT" \
    --role="roles/iam.serviceAccountCreator"

gcloud projects add-iam-policy-binding PROJECT_ID \
    --member="user:ACCOUNT" \
    --role="roles/iam.serviceAccountKeyAdmin"

Windows

gcloud projects add-iam-policy-binding PROJECT_ID ^
    --member="user:ACCOUNT" ^
    --role="roles/resourcemanager.projectIamAdmin"

gcloud projects add-iam-policy-binding PROJECT_ID ^
    --member="user:ACCOUNT" ^
    --role="roles/serviceusage.serviceUsageAdmin"

gcloud projects add-iam-policy-binding PROJECT_ID ^
    --member="user:ACCOUNT" ^
    --role="roles/iam.serviceAccountCreator"

gcloud projects add-iam-policy-binding PROJECT_ID ^
    --member="user:ACCOUNT" ^
    --role="roles/iam.serviceAccountKeyAdmin"

Console

This section shows how to create a user cluster configured for the minimal infrastructure with MetalLB as the load balancer.

When you create a user cluster, the Google Cloud console automatically enables the GKE On-Prem API in the Google Cloud project that you select for the cluster creation. The GKE On-Prem API runs in Google Cloud's infrastructure, and the Google Cloud console uses this API to create the cluster in your vSphere data center. To manage your clusters, the GKE-On-Prem API must store metadata about your cluster's state in the Google Cloud region that you specify when creating the cluster. This metadata lets the GKE On-Prem API manage the user cluster lifecycle and doesn't include workload-specific data.

1. In the Google Cloud console, go to the GKE Enterprise clusters page.

   Go to the GKE Enterprise clusters page

2. Select the Google Cloud project that you want to create the cluster in. The selected project is also used as the fleet host project. This must be the same project that the admin cluster is registered to. After the user cluster is created, it is automatically registered to the selected project's fleet.

3. Click Create Cluster.

4. In the dialog box, click On-premises.

5. Next to VMware vSphere, click Configure.

6. Review the prerequisites, and click Continue.

Cluster basics

1. Enter a Name for the user cluster.

2. Under Admin cluster, select the admin cluster from the list. If the admin cluster that you want to use isn't displayed, see the troubleshooting section The admin cluster isn't displayed on the Cluster basics drop-down list.

3. In the GCP API Location field, select the Google Cloud region from the list. In addition to controlling the region where the GKE On-Prem API runs, this setting controls the region in which the following is stored:

   • The user cluster metadata that the GKE On-Prem API needs to manage the cluster lifecycle
   • The Cloud Logging and Cloud Monitoring data of system components
   • The Admin Audit log created by Cloud Audit Logs

4. Select the Google Distributed Cloud version for your user cluster.

5. Click Continue to go to the Networking section.

Networking

In this section, you specify the IP addresses for your cluster's nodes, Pods, and Services. A user cluster needs to have one IP address for each node and an additional IP address to be used for a temporary node during upgrade of the user cluster. For more information, see How many IP addresses does a user cluster need?.

1. In the Node IPs section, select the IP mode for the user cluster. Select one of the following:

   • DHCP: Choose DHCP if you want your cluster nodes to get their IP address from a DHCP server.

   • Static: Choose Static if you want to provide static IP addresses for your cluster nodes.

2. If you selected DHCP, skip to the next step to specify Service and Pod CIDRs. For Static IP mode, provide the following information:

   1. Enter the IP address of the Gateway for the user cluster.

   2. Enter the Subnet mask for the user cluster nodes.

   3. In the IP Addresses section, enter the IP addresses and optionally, the hostnames for the nodes in the user cluster. You can enter either an individual IP v4 address (such as 192.0.2.1) or a CIDR block of IPv4 addresses (such as 192.0.2.0/24).

      • If you enter a CIDR block, don't enter a hostname.
      • If you enter an individual IP address, you can optionally enter a hostname. If you don't enter a hostname, Google Distributed Cloud uses the VM's name from vSphere as the hostname.

   4. Click + Add IP Address as needed to enter more IP addresses.

3. In the Service and Pod CIDRs section, the Google Cloud console provides the following address ranges for your Kubernetes Service and Pod:

   • Service CIDR: 10.96.0.0/20
   • Pod CIDR: 192.168.0.0/16

   If you prefer to enter your own address ranges, see IP addresses for Pods and Services for best practices.

4. If you selected Static IP mode, specify the following information in Host config section:

   1. Enter the IP addresses of the DNS servers.
   2. Enter the IP addresses of the NTP servers.
   3. Optionally, enter DNS search domains.

5. Click Continue to go to the Load balancer section.

Load balancer

Configure MetalLB as the load balancer.

1. In the Load balancer type, leave MetalLB selected.

2. In the Virtual IPs section, enter the following:

   • Control plane VIP: The destination IP address to be used for traffic sent to the Kubernetes API server server of the user cluster. This IP address must be in the same L2 domain as the admin cluster nodes. Don't add this address in the Address pools section.

   • Ingress VIP: The IP address to be configured on the load balancer for the ingress proxy. You must add this to an address pool in the Address pools section.

3. In the Address pools section, specify an address pool for the ingress VIP.

   1. Click + Add IP Address Range.
   2. Enter a name for the address pool.
   3. Enter an IP address range in either CIDR notation (such as 192.0.2.0/26) or range notation (such as 192.0.2.64-192.0.2.72). The IP addresses in each pool cannot overlap, and must be in the same subnet as the cluster nodes.

   4. Under Assignment of IP addresses, select one of the following:

      • Automatic: Choose this option if you want the MetalLB controller to automatically assign IP addresses from the address pool to Services of type LoadBalancer.
      • Manual: Choose this option if you intend to use addresses from the pool to manually specify addresses for Services of type LoadBalancer

   5. Click Avoid buggy IP addresses if you want the MetalLB controller to not use addresses from the pool that end in .0 or .255. This avoids the problem of buggy consumer devices mistakenly dropping traffic sent to those special IP addresses.

   6. Repeat as needed. When you're finished click Done.

4. Click Continue.

Control Plane

1. Review the default values configured for the nodes in the admin cluster that run the control-plane components for your user cluster. The default values are sufficient for the minimal infrastructure, but you can adjust the values as needed.

2. Click Continue.

Features

1. Clear Enable anti-affinity groups. When you set up the minimal infrastructure, you only created one ESXi host, so you shouldn't enable anti-affinity groups.

2. Click Continue.

Node pools

Your cluster will be created with at least one node pool. A node pool is a template for the groups of nodes created in this cluster. For more information, see Creating and managing node pools .

Review the default values configured for the node pool. The default values are sufficient for the minimal infrastructure, but you can adjust the values as needed.

Verify and Complete

Click Verify and Complete to create the user cluster. It takes 10 minutes or more to create the user cluster. The Google Cloud console displays status messages as it verifies the settings and creates the cluster in your data center.

The operator who creates the user cluster in the Google Cloud console is automatically logged in to the cluster after it is created. Other operators need to follow the steps in Logging in to a cluster from the Cloud Console to gain access to the cluster.

If an error is encountered verifying the settings, the Google Cloud console displays an error message that should be clear enough for you to fix the configuration issue and try again to create the cluster.

For more information about possible errors and how to fix them, see Troubleshoot user cluster creation in the Google Cloud console.

Create the user cluster kubeconfig file

To access the user cluster in your data center from the command line, you need to get a kubeconfig file from the admin cluster. After the Google Cloud console indicates that user cluster status is healthy, run the following command on the admin workstation to save a new kubeconfig file for the user cluster locally. Replace the following:

• CLUSTER_NAME: the name of the newly-created user cluster
• ADMIN_CLUSTER_KUBECONFIG: the path to the admin cluster kubeconfig file
• USER_CLUSTER_KUBECONFIG: the name of the user cluster kubeconfig file that the command outputs

kubectl get secret admin \
--kubeconfig ADMIN_CLUSTER_KUBECONFIG \
-n CLUSTER_NAME \
-o=jsonpath='{.data.admin\.conf}' | base64 -d > USER_CLUSTER_KUBECONFIG

After the file has been saved, you can begin accessing the user cluster using kubectl, as in the following example:

kubectl --kubeconfig USER_CLUSTER_KUBECONFIG get namespaces

Command line

1. Get an SSH connection to your admin workstation.

2. Specify static IPs for your user cluster.

   To specify the static IP addresses that you want to use for your user cluster, create an IP block file named user-cluster-ipblock.yaml. You need three IP addresses for your user cluster nodes and an additional address to be used temporarily during upgrades. Here is an example of an IP block file with four addresses:

   blocks:
     - netmask: 255.255.255.0
       gateway: 172.16.20.1
       ips:
       - ip: 172.16.20.55
         hostname: user-vm-1
       - ip: 172.16.20.56
         hostname: user-vm-2
       - ip: 172.16.20.57
         hostname: user-vm-3
       - ip: 172.16.20.58
         hostname: user-vm-4
   

Fill in the user cluster configuration file

When gkeadm created your admin workstation, it generated a configuration file named user-cluster.yaml. This configuration file is for creating your user cluster.

1. Familiarize yourself with the user cluster configuration file.

   Scan the documentation for the user cluster configuration file to get a general sense of what's in the file. You might want to keep the page open in a separate tab or window so you can refer to it as you fill in values for the fields.

2. Open user-cluster.yaml for editing.

   Several fields in your user cluster configuration file are already filled in with default values, generated values, or values that you provided when you created your admin workstation. You can leave those fields unchanged.

3. Set name to a name of your choice for the user cluster.

Fill in fields in the network section

1. Enter IP addresses for the following fields in the network.hostConfig section:

   dnsServers:
   ntpServers:
   

2. Set network.ipMode.type to "static".

3. Set network.ipMode.ipBlockFilePath to "user-cluster-ipblock.yaml"

4. Fill in or keep the default values for network.serviceCIDR and network.podCIDR.

Fill in fields in the load balancer section

1. Fill in the following fields in the loadBalancer.vips section:

   controlPlaneVIP:
   ingressVIP:
   

2. Set loadBalancer.kind to "MetalLB".

3. Fill in the loadBalancer.metalLB section. For more information, see Bundled load balancing with MetalLB.

4. For at least one of your node pools, set enableLoadBalancer to true. For more information, see Bundled load balancing with MetalLB.

5. Set antiAffinityGroups.enabled to false.

Example of a user cluster configuration file

Here's an example of a filled-in user cluster configuration file:

apiVersion: v1
kind: UserCluster
name: "my-user-cluster"
gkeOnPremVersion: "1.11.0-gke.543"
network:
  hostConfig:
    dnsServers:
    - "203.0.113.1"
    - "198.51.100.1"
    ntpServers:
    - "216.239.35.4"
  ipMode:
    type: "static"
  serviceCIDR: "10.96.0.0/20"
  podCIDR: "192.168.0.0/16"
loadBalancer:
  vips:
    controlPlaneVIP: "172.16.20.61"
    ingressVIP: "172.16.20.62"
  kind: "MetalLB"
  metalLB:
    addressPools:
    - name: "uc-address-pool"
      addresses:
      - "172.16.20.62-172.16.20.72"
nodePools:
- name: "uc-node-pool"
  cpus: 4
  memoryMB: 8192
  replicas: 3
  enableLoadBalancer: true
antiAffinityGroups:
  enabled: false
gkeConnect:
  projectID: "my-project-123"
  registerServiceAccountKeyPath: "connect-register-sa-2203040617.json"
stackdriver:
  projectID: "my-project-123"
  clusterLocation: "us-central1"
  enableVPC: false
  serviceAccountKeyPath: "log-mon-sa-2203040617.json"
  disableVsphereResourceMetrics: false
autoRepair:
  enabled: true

Validate the configuration and create the cluster

1. Verify that the your user cluster configuration file is valid and can be used for cluster creation:

   gkectl check-config --kubeconfig kubeconfig --config user-cluster.yaml
   

2. Create the user cluster:

   gkectl create cluster --kubeconfig kubeconfig --config user-cluster.yaml
   

   Cluster creation takes approximately 30 minutes.

Locate the user cluster kubeconfig file

The gkectl create cluster command creates a kubeconfig file named USER_CLUSTER_NAME-kubeconfig in the current directory. You will need this kubeconfig file later to interact with your user cluster.

Verify that your user cluster is running

Verify that your user cluster is running:

kubectl get nodes --kubeconfig USER_CLUSTER_KUBECONFIG

Replace USER_CLUSTER_KUBECONFIG with the path of your kubeconfig file.

The output shows the user cluster nodes. For example:

my-user-cluster-node-pool-69-d46d77885-7b7tx   Ready ...
my-user-cluster-node-pool-69-d46d77885-lsvzk   Ready ...
my-user-cluster-node-pool-69-d46d77885-sswjk   Ready ...