This document shows how to create basic clusters for a small proof-of-concept installation of Google Distributed Cloud.
These are the primary steps:
Create an admin workstation.
From your admin workstation, create an admin cluster.
From your admin workstation, create a user cluster.
Before you begin
Set up your vSphere and Google Cloud environments as described in Set up minimal infrastructure.
Gather information
To prepare for filling in an admin workstation configuration file, gather the following information:
The username and password of your vCenter account
The root CA certificate for your vCenter Server
The name of your vSphere data center
The name of your vSphere cluster
The name of your vSphere datastore
The name of your vSphere network
The IP addresses described in Plan your IP addresses
The ID of your Google Cloud project
The path of the JSON key file for your component access service account
Log in
The gkeadm
command-line tool uses your SDK account
property to create service accounts. So it is important
that you set your SDK account
property before you run gkeadm
to create an
admin workstation.
Log in with any Google Account. This sets your SDK account
property:
gcloud auth login
Verify that your SDK account
property is set correctly:
gcloud config list
The output shows the value of your SDK account
property.
For example:
[core] account = my-name@google.com disable_usage_reporting = False Your active configuration is: [default]
Grant roles to your SDK account
The Google Account that is set as your SDK account
property must have these
IAM roles so that gkeadm
can create and
manage service accounts for you:
resourcemanager.projectIamAdmin
serviceusage.serviceUsageAdmin
iam.serviceAccountCreator
iam.serviceAccountKeyAdmin
To grant roles, you must have certain permissions on your Google Cloud project. For details, see Granting, changing, and revoking access to resources.
If you have the required permissions, you can grant the roles yourself. Otherwise, someone else in your organization must grant the roles for you.
To grant the roles:
Linux and macOS
gcloud projects add-iam-policy-binding PROJECT_ID \ --member="user:ACCOUNT" \ --role="roles/resourcemanager.projectIamAdmin" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="user:ACCOUNT" \ --role="roles/serviceusage.serviceUsageAdmin" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="user:ACCOUNT" \ --role="roles/iam.serviceAccountCreator" gcloud projects add-iam-policy-binding PROJECT_ID \ --member="user:ACCOUNT" \ --role="roles/iam.serviceAccountKeyAdmin"
Windows
gcloud projects add-iam-policy-binding PROJECT_ID ^ --member="user:ACCOUNT" ^ --role="roles/resourcemanager.projectIamAdmin" gcloud projects add-iam-policy-binding PROJECT_ID ^ --member="user:ACCOUNT" ^ --role="roles/serviceusage.serviceUsageAdmin" gcloud projects add-iam-policy-binding PROJECT_ID ^ --member="user:ACCOUNT" ^ --role="roles/iam.serviceAccountCreator" gcloud projects add-iam-policy-binding PROJECT_ID ^ --member="user:ACCOUNT" ^ --role="roles/iam.serviceAccountKeyAdmin"
Replace the following:
PROJECT_ID
: the ID of your Google Cloud projectACCOUNT
: the value of your SDKaccount
property
Create an admin workstation.
The steps in this section use the gkeadm
command-line tool, which is
available for 64-bit Linux, Windows 10, Windows Server 2019, and macOS 10.15
and higher.
Download gkeadm
to your current directory.
Generate templates
./gkeadm create config
The preceding command created these files in your current directory:
credential.yaml
admin-ws-config.yaml
Fill in credential.yaml
In credential.yaml
, fill in your vCenter username and password. For example:
kind: CredentialFile items: - name: vCenter username: "my-account-name" password: "AadmpqGPqq!a"
Familiarize yourself with the admin workstation configuration file
Scan the documentation for the admin workstation configuration file to get a general sense of what's in the file. You might want to keep the page open in a separate tab or window so you can refer to it as you fill in values for the fields.
Fill in admin-ws-config.yaml
Open admin-ws-config.yaml
for editing.
The following fields are filled in for you with default or generated values:
vCenter: credentials: fileRef: credentials: entry: adminWorkstation: name: cpus: memoryMB: diskGB: dataDiskName: dataDiskMB: ntpServer:
Fill in the gcp.componentAccessServiceAccountKeyPath field.
Fill in the following fields in the vCenter section.
credentials: address: datacenter: datastore: cluster: network: caCertPath:
Set vCenter.resourcePool to CLUSTER_NAME/Resources. Replace CLUSTER_NAME with the name of your vSphere cluster.
If necessary, fill in the proxyURL field.
If necessary, fill in the adminWorkstation.proxyURL field.
Set
adminWorkstation.network.ipAllocationMode
to "static"
.
Fill in the following fields in the adminWorkstation.network.hostConfig section:
ip: gateway: netmask: dns:
Example admin workstation configuration file
Here's an example of a filled-in admin workstation configuration file:
gcp: componentAccessServiceAccountKeyPath: "/usr/local/google/home/me/keys/component-access-key.json" vCenter: credentials: address: "vc01.example" fileRef: path: "credential.yaml" entry: "vCenter" datacenter: "vc01" datastore: "vc01-datastore-1" cluster: "vc01-workloads-1" network: "vc01-net-1" resourcePool: "vc01-workloads-1/Resources" caCertPath: "/usr/local/google/home/stevepe/certs/vc01-cert.pem" proxyUrl: "" adminWorkstation: name: gke-admin-ws-220304-014925 cpus: 4 memoryMB: 8192 diskGB: 50 dataDiskName: gke-on-prem-admin-workstation-data-disk/gke-admin-ws-220304-014925-data-disk.vmdk dataDiskMB: 512 network: ipAllocationMode: "static" hostConfig: ip: "172.16.20.49" gateway: "172.16.20.1" netmask: "255.255.255.0" dns: - "203.0.113.1" proxyUrl: "" ntpServer: ntp.ubuntu.com
Create your admin workstation
Create your admin workstation, and automatically create service accounts:
./gkeadm create admin-workstation --auto-create-service-accounts
The output gives detailed information about the creation of your admin workstation and provides a command that you can use to get an SSH connection to your admin workstation:
... Admin workstation is ready to use. Admin workstation information saved to /usr/local/google/home/me/my-admin-workstation This file is required for future upgrades SSH into the admin workstation with the following command: ssh -i /usr/local/google/home/me/.ssh/gke-admin-workstation ubuntu@172.16.20.49 ********************************************************************
For more detailed information about creating an admin workstation, see Create an admin workstation.
View the files on your admin workstation
Use the command displayed in the preceding output to get an SSH connection to your admin workstation. For example:
ssh -i /usr/local/google/home/me/.ssh/gke-admin-workstation ubuntu@172.16.20.49
List the files in the home directory:
ls -1
The output includes the JSON key files for two service accounts that gkeadm
created for you: a connect-register service account and a logging-monitoring
service account. Make a note of the name of the JSON key files for your
connect-register service account. You will need it later when you create your
clusters. For example:
admin-cluster.yaml admin-ws-config.yaml sa-key.json connect-register-sa-2203040617.json credential.yaml log-mon-sa-2203040617.json logs vc01-cert.pem user-cluster.yaml
Create an admin cluster
Get an SSH connection to your admin workstation.
Specify static IP addresses for your admin cluster
To specify the static IP addresses that you want to use for your admin cluster
nodes, create an
IP block file
named admin-cluster-ipblock.yaml
.
You need five IP addresses for the following nodes in your admin cluster:
Three nodes to run the admin cluster control plane and add-ons
An additional node to be used temporarily during upgrades
One node to run the control plane for the user cluster that you will create later
Here is an example of an IP block file that has addresses for five nodes:
blocks: - netmask: 255.255.255.0 gateway: 172.16.20.1 ips: - ip: 172.16.20.50 hostname: admin-vm-1 - ip: 172.16.20.51 hostname: admin-vm-2 - ip: 172.16.20.52 hostname: admin-vm-3 - ip: 172.16.20.53 hostname: admin-vm-4 - ip: 172.16.20.54 hostname: admin-vm-5
The ips
field is an array of IP addresses and hostnames. These are the IP
addresses and hostnames that Google Distributed Cloud will assign to
your admin cluster nodes.
In the IP block file, you also specify a subnet mask and a default gateway for the admin cluster nodes.
Familiarize yourself with the admin cluster configuration file
Scan the documentation for the admin cluster configuration file to get a general sense of what's in the file. You might want to keep the page open in a separate tab or window so you can refer to it as you fill in values for the fields.
Fill in the admin cluster configuration file
When gkeadm
created your admin workstation, it generated a configuration file
named admin-cluster.yaml
. This configuration file is for creating your admin
cluster.
Open admin-cluster.yaml
for editing.
Several fields in your admin cluster configuration file are already filled in
with default values, generated values, or values that you provided when you
created your admin workstation. You can leave those fields unchanged.
Fill in the vCenter.dataDisk field.
Fill in the following fields in the network.hostConfig section:
dnsServers: ntpServers:
Set
network.ipMode.type
to "static"
.
Set
network.ipMode.ipBlockFilePath
to "admin-cluster-ipblock.yaml"
Fill in or keep the default values for network.serviceCIDR and network.podCIDR.
Fill in the following fields in the loadBalancer.vips section:
controlPlaneVIP: addonsVIP:
Set
loadBalancer.kind
to "MetalLB"
.
Set
antiAffinityGroups.enabled
to false
.
Set gkeConnect.projectID to the ID of your Google Cloud project.
Set gkeConnect.registerServiceAccountKeyPath to the path of the JSON key file for your connect-register service account.
Example of an admin cluster configuration file
Here's an example of a filled-in admin cluster configuration file:
apiVersion: v1 kind: AdminCluster name: "gke-admin-01" bundlePath: "/var/lib/gke/bundles/gke-onprem-vsphere-1.11.0-gke.543-full.tgz" vCenter: address: "vc01.example" datacenter: "vc-01" cluster: "vc01-workloads-1" resourcePool: "my-cluster/Resources" datastore: "vc01-datastore-1" caCertPath: "/usr/local/google/home/me/certs/vc01-cert.pem"" credentials: fileRef: path: "credential.yaml" entry: "vCenter" dataDisk: "vc01-admin-disk.vmdk" network: hostConfig: dnsServers: - "203.0.113.1" - "198.51.100.1" ntpServers: - "216.239.35.4" ipMode: type: "static" serviceCIDR: "10.96.232.0/24" podCIDR: "192.168.0.0/16" vCenter: networkName: "vc01-net-1" loadBalancer: vips: controlPlaneVIP: "172.16.20.59" addonsVIP: "172.16.20.60" kind: "MetalLB" antiAffinityGroups: enabled: false componentAccessServiceAccountKeyPath: "sa-key.json" gkeConnect: projectID: "my-project-123" registerServiceAccountKeyPath: "connect-register-sa-2203040617.json" stackdriver: projectID: "my-project-123" clusterLocation: "us-central1" enableVPC: false serviceAccountKeyPath: "log-mon-sa-2203040617.json" disableVsphereResourceMetrics: false
Validate the admin cluster configuration file
Verify that the your admin cluster configuration file is valid and can be used for cluster creation:
gkectl check-config --config admin-cluster.yaml
Prepare your vSphere environment
Run gkectl prepare
to initialize your vSphere environment:
gkectl prepare --config admin-cluster.yaml --skip-validation-all
Create the admin cluster
Create the admin cluster:
gkectl create admin --config admin-cluster.yaml
Resume admin cluster creation after a failure
If the admin cluster creation fails or is canceled, you can run the create
command again:
gkectl create admin --config admin-cluster.yaml
Locate the admin cluster kubeconfig file
The gkectl create admin
command creates a kubeconfig file named
kubeconfig
in the current directory. You will need this kubeconfig file
later to interact with your admin cluster.
Verify that your admin cluster is running
Verify that your admin cluster is running:
kubectl get nodes --kubeconfig kubeconfig
The output shows the admin cluster nodes. For example:
gke-admin-master-hdn4z Ready control-plane,master ... gke-admin-node-7f46cc8c47-g7w2c Ready ... gke-admin-node-7f46cc8c47-kwlrs Ready ...
Create a user cluster
You can create a user cluster either in the Google Cloud console or from the command line on your admin workstation.
Console
This section shows how to create a user cluster configured for the minimal infrastructure with MetalLB as the load balancer.
When you create a user cluster, the Google Cloud console automatically enables the GKE On-Prem API in the Google Cloud project that you select for the cluster creation. The GKE On-Prem API runs in Google Cloud's infrastructure, and the Google Cloud console uses this API to create the cluster in your vSphere data center. To manage your clusters, the GKE-On-Prem API must store metadata about your cluster's state in the Google Cloud region that you specify when creating the cluster. This metadata lets the GKE On-Prem API manage the user cluster lifecycle and doesn't include workload-specific data.
In the Google Cloud console, go to the GKE Enterprise clusters page.
Select the Google Cloud project that you want to create the cluster in. The selected project is also used as the fleet host project. This must be the same project that the admin cluster is registered to. After the user cluster is created, it is automatically registered to the selected project's fleet.
Click Create Cluster.
In the dialog box, click On-premises.
Next to VMware vSphere, click Configure.
Review the prerequisites, and click Continue.
Cluster basics
- Enter a Name for the user cluster.
Under Admin cluster, select the admin cluster from the list. If the admin cluster that you want to use isn't displayed, see the troubleshooting section The admin cluster isn't displayed on the Cluster basics drop-down list.
In the GCP API Location field, select the Google Cloud region from the list. In addition to controlling the region where the GKE On-Prem API runs, this setting controls the region in which the following is stored:
- The user cluster metadata that the GKE On-Prem API needs to manage the cluster lifecycle
- The Cloud Logging and Cloud Monitoring data of system components
- The Admin Audit log created by Cloud Audit Logs
Select the Google Distributed Cloud version for your user cluster.
Click Continue to go to the Networking section.
Networking
In this section, you specify the IP addresses for your cluster's nodes, Pods, and Services. A user cluster needs to have one IP address for each node and an additional IP address to be used for a temporary node during upgrade of the user cluster. For more information, see How many IP addresses does a user cluster need?.
In the Node IPs section, select the IP mode for the user cluster. Select one of the following:
DHCP: Choose DHCP if you want your cluster nodes to get their IP address from a DHCP server.
Static: Choose Static if you want to provide static IP addresses for your cluster nodes.
If you selected DHCP, skip to the next step to specify Service and Pod CIDRs. For Static IP mode, provide the following information:
- Enter the IP address of the Gateway for the user cluster.
Enter the Subnet mask for the user cluster nodes.
In the IP Addresses section, enter the IP addresses and optionally, the hostnames for the nodes in the user cluster. You can enter either an individual IP v4 address (such as 192.0.2.1) or a CIDR block of IPv4 addresses (such as 192.0.2.0/24).
- If you enter a CIDR block, don't enter a hostname.
- If you enter an individual IP address, you can optionally enter a hostname. If you don't enter a hostname, Google Distributed Cloud uses the VM's name from vSphere as the hostname.
Click + Add IP Address as needed to enter more IP addresses.
In the Service and Pod CIDRs section, the Google Cloud console provides the following address ranges for your Kubernetes Service and Pod:
- Service CIDR: 10.96.0.0/20
- Pod CIDR: 192.168.0.0/16
If you prefer to enter your own address ranges, see IP addresses for Pods and Services for best practices.
If you selected Static IP mode, specify the following information in Host config section:
- Enter the IP addresses of the DNS servers.
- Enter the IP addresses of the NTP servers.
- Optionally, enter DNS search domains.
Click Continue to go to the Load balancer section.
Load balancer
Configure MetalLB as the load balancer.
In the Load balancer type, leave MetalLB selected.
In the Virtual IPs section, enter the following:
Control plane VIP: The destination IP address to be used for traffic sent to the Kubernetes API server server of the user cluster. This IP address must be in the same L2 domain as the admin cluster nodes. Don't add this address in the Address pools section.
Ingress VIP: The IP address to be configured on the load balancer for the ingress proxy. You must add this to an address pool in the Address pools section.
In the Address pools section, specify an address pool for the ingress VIP.
- Click + Add IP Address Range.
- Enter a name for the address pool.
- Enter an IP address range in either CIDR notation (such as 192.0.2.0/26) or range notation (such as 192.0.2.64-192.0.2.72). The IP addresses in each pool cannot overlap, and must be in the same subnet as the cluster nodes.
Under Assignment of IP addresses, select one of the following:
- Automatic: Choose this option if you want the MetalLB controller
to automatically assign IP addresses from the address pool to Services
of type
LoadBalancer
. - Manual: Choose this option if you intend to use addresses from
the pool to manually specify addresses for Services of type
LoadBalancer
- Automatic: Choose this option if you want the MetalLB controller
to automatically assign IP addresses from the address pool to Services
of type
Click Avoid buggy IP addresses if you want the MetalLB controller to not use addresses from the pool that end in .0 or .255. This avoids the problem of buggy consumer devices mistakenly dropping traffic sent to those special IP addresses.
Repeat as needed. When you're finished click Done.
Click Continue.
Control Plane
Review the default values configured for the nodes in the admin cluster that run the control-plane components for your user cluster. The default values are sufficient for the minimal infrastructure, but you can adjust the values as needed.
Click Continue.
Features
Clear Enable anti-affinity groups. When you set up the minimal infrastructure, you only created one ESXi host, so you shouldn't enable anti-affinity groups.
Click Continue.
Node pools
Your cluster will be created with at least one node pool. A node pool is a template for the groups of nodes created in this cluster. For more information, see Creating and managing node pools .
Review the default values configured for the node pool. The default values are sufficient for the minimal infrastructure, but you can adjust the values as needed.
Verify and Complete
Click Verify and Complete to create the user cluster. It takes 10 minutes or more to create the user cluster. The Google Cloud console displays status messages as it verifies the settings and creates the cluster in your data center.
The operator who creates the user cluster in the Google Cloud console is automatically logged in to the cluster after it is created. Other operators need to follow the steps in Logging in to a cluster from the Cloud Console to gain access to the cluster.
If an error is encountered verifying the settings, the Google Cloud console displays an error message that should be clear enough for you to fix the configuration issue and try again to create the cluster.
For more information about possible errors and how to fix them, see Troubleshoot user cluster creation in the Google Cloud console.
Create the user cluster kubeconfig file
To access the user cluster in your data center from the command line, you need
to get a kubeconfig
file from the admin cluster. After the
Google Cloud console indicates that user cluster status is healthy, run the
following command on the admin workstation to save a new kubeconfig
file for
the user cluster locally. Replace the following:
- CLUSTER_NAME: the name of the newly-created user cluster
- ADMIN_CLUSTER_KUBECONFIG: the path to the admin cluster
kubeconfig
file - USER_CLUSTER_KUBECONFIG: the name of the user cluster
kubeconfig
file that the command outputs
kubectl get secret admin \ --kubeconfig ADMIN_CLUSTER_KUBECONFIG \ -n CLUSTER_NAME \ -o=jsonpath='{.data.admin\.conf}' | base64 -d > USER_CLUSTER_KUBECONFIG
After the file has been saved, you can begin accessing the user cluster using
kubectl
, as in the following example:
kubectl --kubeconfig USER_CLUSTER_KUBECONFIG get namespaces
Command line
Get an SSH connection to your admin workstation.
Specify static IPs for your user cluster.
To specify the static IP addresses that you want to use for your user cluster, create an IP block file named
user-cluster-ipblock.yaml
. You need three IP addresses for your user cluster nodes and an additional address to be used temporarily during upgrades. Here is an example of an IP block file with four addresses:blocks: - netmask: 255.255.255.0 gateway: 172.16.20.1 ips: - ip: 172.16.20.55 hostname: user-vm-1 - ip: 172.16.20.56 hostname: user-vm-2 - ip: 172.16.20.57 hostname: user-vm-3 - ip: 172.16.20.58 hostname: user-vm-4
Fill in the user cluster configuration file
When gkeadm
created your admin workstation, it generated a configuration file
named user-cluster.yaml
. This configuration file is for creating your user
cluster.
Familiarize yourself with the user cluster configuration file.
Scan the documentation for the user cluster configuration file to get a general sense of what's in the file. You might want to keep the page open in a separate tab or window so you can refer to it as you fill in values for the fields.
Open
user-cluster.yaml
for editing.Several fields in your user cluster configuration file are already filled in with default values, generated values, or values that you provided when you created your admin workstation. You can leave those fields unchanged.
Set name to a name of your choice for the user cluster.
Fill in fields in the network section
Enter IP addresses for the following fields in the network.hostConfig section:
dnsServers: ntpServers:
Set network.ipMode.type to
"static"
.Set network.ipMode.ipBlockFilePath to
"user-cluster-ipblock.yaml"
Fill in or keep the default values for network.serviceCIDR and network.podCIDR.
Fill in fields in the load balancer section
Fill in the following fields in the loadBalancer.vips section:
controlPlaneVIP: ingressVIP:
Set loadBalancer.kind to
"MetalLB"
.Fill in the loadBalancer.metalLB section. For more information, see Bundled load balancing with MetalLB.
For at least one of your node pools, set enableLoadBalancer to
true
. For more information, see Bundled load balancing with MetalLB.Set antiAffinityGroups.enabled to
false
.
Example of a user cluster configuration file
Here's an example of a filled-in user cluster configuration file:
apiVersion: v1 kind: UserCluster name: "my-user-cluster" gkeOnPremVersion: "1.11.0-gke.543" network: hostConfig: dnsServers: - "203.0.113.1" - "198.51.100.1" ntpServers: - "216.239.35.4" ipMode: type: "static" serviceCIDR: "10.96.0.0/20" podCIDR: "192.168.0.0/16" loadBalancer: vips: controlPlaneVIP: "172.16.20.61" ingressVIP: "172.16.20.62" kind: "MetalLB" metalLB: addressPools: - name: "uc-address-pool" addresses: - "172.16.20.62-172.16.20.72" nodePools: - name: "uc-node-pool" cpus: 4 memoryMB: 8192 replicas: 3 enableLoadBalancer: true antiAffinityGroups: enabled: false gkeConnect: projectID: "my-project-123" registerServiceAccountKeyPath: "connect-register-sa-2203040617.json" stackdriver: projectID: "my-project-123" clusterLocation: "us-central1" enableVPC: false serviceAccountKeyPath: "log-mon-sa-2203040617.json" disableVsphereResourceMetrics: false autoRepair: enabled: true
Validate the configuration and create the cluster
Verify that the your user cluster configuration file is valid and can be used for cluster creation:
gkectl check-config --kubeconfig kubeconfig --config user-cluster.yaml
Create the user cluster:
gkectl create cluster --kubeconfig kubeconfig --config user-cluster.yaml
Cluster creation takes approximately 30 minutes.
Locate the user cluster kubeconfig file
The gkectl create cluster
command creates a kubeconfig file named
USER_CLUSTER_NAME-kubeconfig
in the current directory. You will need this
kubeconfig file later to interact with your user cluster.
Verify that your user cluster is running
Verify that your user cluster is running:
kubectl get nodes --kubeconfig USER_CLUSTER_KUBECONFIG
Replace USER_CLUSTER_KUBECONFIG with the path of your kubeconfig file.
The output shows the user cluster nodes. For example:
my-user-cluster-node-pool-69-d46d77885-7b7tx Ready ... my-user-cluster-node-pool-69-d46d77885-lsvzk Ready ... my-user-cluster-node-pool-69-d46d77885-sswjk Ready ...