Create a user cluster using Anthos On-Prem API clients

Console

Most of the settings in the console correspond to the fields in the cluster configuration file.

1. In the Google Cloud console, go to the GKE Enterprise clusters page.

   Go to the GKE Enterprise clusters page

2. Select the Google Cloud project that you want to create the cluster in. The selected project is also used as the fleet host project. This must be the same project that the admin cluster is registered to. After the user cluster is created, it is automatically registered to the selected project's fleet.

3. Click Create Cluster.

4. In the dialog box, click On-premises.

5. Next to Bare metal, click Configure.

6. Review the prerequisites and the sample architecture. When you're ready, click Next to begin configuring the cluster.

The following sections guide you through configuring the user cluster.

Cluster basics

Enter basic information about the cluster.

1. Enter a Name for the user cluster.

2. Under Admin cluster, select the admin cluster from the list.

3. In the Google Cloud API Location field, select the Google Cloud region from the list. This setting specifies the region where the Anthos On-Prem API runs, and the region in which the following are stored:

   • The user cluster metadata that the Anthos On-Prem API needs to manage the cluster lifecycle
   • The Cloud Logging and Cloud Monitoring data of system components
   • The Admin Audit log created by Cloud Audit Logs

   The cluster name, project, and location together uniquely identify the cluster in Google Cloud.

4. Select the Google Distributed Cloud version for your user cluster. User clusters must be either the same minor version as the admin cluster or one minor version lower than the admin cluster.

5. As the cluster creator, you are granted cluster admin privileges to the cluster. Optionally, enter the email address of another user who will administer the cluster in the Admin user field.

   When the cluster is created, the Anthos On-Prem API applies the Kubernetes role-based access control (RBAC) policies to the cluster to grant you and other admin users the Kubernetes clusterrole/cluster-admin role, which provides full access to every resource in the cluster in all namespaces.

6. In the Node configuration section, specify the following:

   • Maximum Pods per node: Enter the maximum number of Pods that can be run on a single node. Allowable values are between 32–250, inclusive. Kubernetes assigns a Classless Inter-Domain Routing (CIDR) block to each node so that each pod can have a unique IP address. The size of the CIDR block corresponds to the maximum number of pods per node. For more information about setting the maximum number of pods per node, Pod networking.

   • Container runtime: containerd is the only available container runtime for your cluster.

7. Click Next to go to the Networking section.

Networking

In this section, you specify the IP addresses for your cluster's nodes, Pods, and Services. If you are using bundled load balancing with MetalLB, you configure that as well.

1. In the Control plane node section, enter the IPv4 address of each control plane node. Control plane nodes run the system workload. Typically, this is either a single machine if using a minimum deployment or three machines if using a high availability (HA) deployment. Specify an odd number of nodes to have a majority quorum for HA. This field can be changed whenever you update or upgrade a cluster.

   Click + Add IP Address as needed to enter more IP addresses.

2. In the Load balancer section, select the load balancer from the Mode list to set up for your cluster. See Overview of load balancers for more information.

   Bundled with MetalLB

   Configure load balancing with the bundled MetalLB load balancer. With this option, Google Distributed Cloud deploys Layer 4 load balancers that run on either a dedicated pool of worker nodes or on the same nodes as the control plane.

   1. In the Load balancer node pools section, select one of the following:

      • Use control plane nodes: Choose this option to run the load balancers on the same nodes as the control plane.

      • Create load balancer node pool: Choose this advanced option if you need to run the load balancers on a dedicated pool of worker nodes. All nodes in the load balancer node pool must be in the same Layer 2 subnet as the load balancer virtual IPs (VIPs) that you configure in the Load balancer address pools section.

        1. In the Load balancer node pool IP 1 field, enter an IPv4 address for a node in your load-balancer node pool.

        2. Click + Add IP address as needed to enter additional IP addresses.

   2. In the Load balancer address pools section, add one or more address pools for the MetalLB controller to choose from and assign to Services of type LoadBalancer. The ingress VIP, which you specify in the Virtual IPs section, must be in one of these pools.

      1. Enter a name for the address pool.

      2. Enter an IP address range in either CIDR notation (for example: 192.0.2.0/26) or range notation (for example: 192.0.2.64-192.0.2.72). To specify a single IP address in a pool, use /32 in the CIDR notation (for example: 192.0.2.1/32).

      3. If the ingress VIP isn't in the address range, select + Add IP Address Range and enter another address range that includes the ingress VIP.

         The IP addresses in each pool cannot overlap, and must be in the same subnet as the cluster nodes.

      4. Under Assignment of IP addresses, select one of the following:

         • Automatic: Choose this option if you want the MetalLB controller to automatically assign IP addresses from the address pool to Services of type LoadBalancer.
         • Manual: Choose this option if you intend to use addresses from the pool to manually specify addresses for Services of type LoadBalancer.

      5. Click Avoid buggy IP addresses if you want the MetalLB controller to not use addresses from the pool that end in .0 or .255. This avoids the problem of buggy consumer devices mistakenly dropping traffic sent to those special IP addresses.

      6. When you're finished click Done.

      7. If needed, click Add Address Pool.

   Manual load balancer

   With manual load balancing, you configure your own load-balancing solutions for control plane and data plane traffic. You must configure your control plane VIP on the external load balancer before creating a cluster. The external control plane load balancer can also be used for data plane traffic, or you can set up a separate load balancer for the data plane. For more information, see Configure manual load balancing.

3. In the Virtual IPs section, enter the following:

   • Control plane VIP: The destination IP address to be used for traffic sent to the user cluster's Kubernetes API server. The control plane VIP must be in the same subnet as the load balancer nodes and must not be in any of the address ranges used for the load balancer address pools.

   • Port: The destination port used for traffic sent to the Kubernetes API server. The default is 443.

   • Ingress VIP: The IP address to be configured on the load balancer for the ingress proxy. Enter an address from one of the load balancer address pools.

4. In the Service and Pod CIDRs section, specify the Kubernetes Service and Pod IP address ranges in CIDR notation. They must not overlap with each other, nor with any address outside the cluster that you want to reach from inside the cluster. We recommend that you use the private IP address ranges defined by RFC 1918. The console provides the following default address ranges, but you can change them:

   • Service CIDR: 10.96.0.0/20 If you don't accept the default, enter a CIDR range between /24 and /12, where /12 provides the most IP addresses.

   • Pod CIDR: 192.168.0.0/16 If you don't accept the default, enter a CIDR range between between /18 and /8, where /8 provides the most IP addresses.

5. In the Advanced attributes attributes section, optionally specify the following:

   • Proxy URL: The HTTP address of your proxy server. Include the port number even if it's the same as the scheme's default port, for example: http://my-proxy.example.local:80

   • URLs: A comma-separated list of IP addresses, IP address ranges, host names, and domain names that should not go through the proxy server. When Google Distributed Cloud sends a request to one of these addresses, hosts, or domains, the request is sent directly.

6. Click Next.

Storage

Google Distributed Cloud provides block and file storage interfaces. They have default options, but you can customize the configurations. For more information, see Configure local storage.

1. Optionally, you can configure the following:

   • Local volume provisioner node mounts: Specifies the configuration for local PersistentVolumes (PVs) backed by mounted disks. You need to format and mount these disks, which you can do before or after cluster creation.

   • Local volume provisioner share: Specifies the configuration for local PersistentVolumes backed by subdirectories in a shared file system. These subdirectories are automatically created during cluster creation.

2. Click Next.

Features

To help you monitor, troubleshoot, and operate your cluster, the following are enabled automatically and can't be disabled:

• Cloud Logging of system services
• Cloud Monitoring of system services
• The Admin Activity audit log

Create a node pool

Your cluster must have at least one node pool for worker nodes. A node pool is a template for the groups of worker nodes created in this cluster.

In the console, you configure at least one node pool (or accept the default values) and then create the cluster. You can add additional node pools after the cluster is created. With the gcloud CLI, you create the cluster first and then add one or more node pools to the newly-created cluster.

1. Click default pool in the left navigation bar.

2. In the Node pool defaults section, enter the Node pool name or accept "default-pool" as the name.

3. In the Worker nodes section, enter the IP addresses of machines for the cluster to run on.

4. In the Node pool metadata (optional) section, if you want to add Kubernetes labels and taints do the following:

   1. Click + Add Kubernetes Labels. Enter the Key and Value for the label. Repeat as needed.
   2. Click + Add Taint. Enter the Key, Value, and Effect for the taint. Repeat as needed.

5. Click Verify and Complete to create the user cluster. It takes 15 minutes or more to create the user cluster. The console displays status messages as it verifies the settings and creates the cluster in your data center.

   If there is a problem with the configuration, the console displays an error message that should be clear enough for you to fix the configuration issue and try again to create the cluster.

gcloud CLI

You use the following command to create a user cluster:

gcloud container bare-metal clusters create

After creating the cluster, you need to create at least one node pool using the following command:

gcloud container bare-metal node-pools create

Most of the flags for creating the cluster and the node pool correspond to the fields in the user cluster configuration file. To help you get started, you can test the complete command in the Examples section. For information about the flags, see the sections that follow the examples, or refer to the gcloud CLI reference.

Before you begin

The Google Distributed Cloud version that you select when creating a user cluster must be a version that your admin cluster supports. Additionally, the latest minor or patch versions aren't available in the Anthos On-Prem API until 7 to 10 days after the release. You can run a gcloud command to get a list of supported versions that you can install on the user cluster.

1. Make sure to update components:

   gcloud components update
   

2. Get a list of available versions to install on the user cluster:

   gcloud container bare-metal clusters query-version-config \
     --admin-cluster-membership=ADMIN_CLUSTER_NAME \
     --admin-cluster-membership-project=FLEET_HOST_PROJECT_ID \
     --admin-cluster-membership-location=global \
     --location=REGION
   

   Replace the following:

   • ADMIN_CLUSTER_NAME: The name of the admin cluster.

   • FLEET_HOST_PROJECT_ID: The ID of the project that the admin cluster is registered to.

   • REGION: The Google Cloud region that you specified when you enrolled the cluster in the Anthos On-Prem API.

We suggest that you use the highest supported version to get the latest fixes and improvements.

Examples

This section provides an example of a command that creates a cluster using the MetalLB load balancer and an example using a manual load balancer. The information that you specify varies depending on the type of load balancer you will be using. See Overview of load balancers for more information.

The examples create the cluster without any node pools. After the cluster is running, you must add a node pool before deploying workloads.

gcloud container bare-metal clusters create USER_CLUSTER_NAME \
  --project=FLEET_HOST_PROJECT_ID \
  --admin-cluster-membership=projects/FLEET_HOST_PROJECT_ID/locations/global/memberships/ADMIN_CLUSTER_NAME \
  --location=REGION \
  --version=VERSION \
  --admin-users=YOUR_EMAIL_ADDRESS \
  --admin-users=ANOTHER_EMAIL_ADDRESS \
  --metal-lb-address-pools='pool=NAME,avoid-buggy-ips=True|False,manual-assign=True|False,addresses=IP_ADDRESS_RANGE_1;IP_ADDRESS_RANGE_2;...' \
  --control-plane-node-configs='node-ip=CP_IP_ADDRESS_1,labels=CP_KEY_1.1=CP_VALUE_1.1;CP_KEY_1.2=CP_VALUE_1.2;...' \
  --control-plane-vip=CONTROL_PLANE_VIP \
  --control-plane-load-balancer-port=CONTROL_PLANE_LB_PORT \
  --ingress-vip=INGRESS_VIP \
  --island-mode-service-address-cidr-blocks=SERVICE_CIDR_BLOCK \
  --island-mode-pod-address-cidr-blocks=POD_CIDR_BLOCK \
  --lvp-share-path=/mnt/localpv-share \
  --lvp-share-storage-class=local-shared \
  --lvp-node-mounts-config-path=/mnt/localpv-disk \
  --lvp-node-mounts-config-storage-class=local-disks

'pool=NAME,avoid-buggy-ips=True|False,manual-assign=True|False,addresses=IP_ADDRESS_RANGE_1;IP_ADDRESS_RANGE_2;...' \

--metal-lb-address-pools='pool=pool1,avoid-buggy-ips=False,manual-assign=True,addresses=192.0.2.0/26;192.0.2.64-192.0.2.72'
--metal-lb-address-pools='pool=pool2,avoid-buggy-ips=True,manual-assign=True,addresses=10.251.133.0/24;10.251.134.80/32'

gcloud container bare-metal clusters create USER_CLUSTER_NAME \
  --project=FLEET_HOST_PROJECT_ID \
  --admin-cluster-membership=projects/FLEET_HOST_PROJECT_ID/locations/global/memberships/ADMIN_CLUSTER_NAME \
  --location=REGION \
  --version=VERSION \
  --admin-users=YOUR_EMAIL_ADDRESS \
  --admin-users=ANOTHER_EMAIL_ADDRESS \
  --enable-manual-lb \
  --control-plane-node-configs='node-ip=CP_IP_ADDRESS_1,labels=CP_KEY_1.1=CP_VALUE_1.1;CP_KEY_1.2=CP_VALUE_1.2;...' \
  --control-plane-vip=CONTROL_PLANE_VIP \
  --control-plane-load-balancer-port=CONTROL_PLANE_LB_PORT \
  --ingress-vip=INGRESS_VIP \
  --island-mode-service-address-cidr-blocks=SERVICE_CIDR_BLOCK \
  --island-mode-pod-address-cidr-blocks=POD_CIDR_BLOCK \
  --lvp-share-path=/mnt/localpv-share \
  --lvp-share-storage-class=local-shared \
  --lvp-node-mounts-config-path=/mnt/localpv-disk \
  --lvp-node-mounts-config-storage-class=local-disks

Before running the gcloud command to create the cluster, you might want to include --validate-only to validate the configuration that you specified in the flags to the gcloud command. When you are ready to create the cluster, remove this flag and run the command.

It takes 15 minutes or more to create the user cluster. You can view the cluster in the Google Cloud console on the Anthos clusters page.

For a complete list of the flags and their descriptions, see the gcloud CLI reference.

Create a node pool

After the cluster is created, you need to create at least one node pool before deploying workloads. A node pool is a template for the groups of worker nodes created in this cluster. With the gcloud CLI, you create the cluster first and then add one or more node pools to the newly-created cluster.

gcloud container bare-metal node-pools create NODE_POOL_NAME \
  --cluster=USER_CLUSTER_NAME \
  --project=FLEET_HOST_PROJECT_ID \
  --location=REGION \
  --node-configs='node-ip=NP_IP_ADDRESS_1,labels=NP_KEY_1.1=NP_VALUE_1.1;NP_KEY_1.2=NP_VALUE_1.2;...'

Replace the following:

• NODE_POOL_NAME: A name of your choice for the node pool. The name must:

  • contain at most 40 characters
  • contain only lowercase alphanumeric characters or a hyphen (-)
  • start with an alphabetic character
  • end with an alphanumeric character

• USER_CLUSTER_NAME: The name of the newly-created user cluster.

• FLEET_HOST_PROJECT_ID: The ID of the project that the cluster is registered to.

• REGION: The Google Cloud region that you specified when you created the cluster.

• --node-configs: The IPv4 address of a worker node machine. Specify this flag for each node. The value for the flag has the following format:

  'node-ip=NP_IP_ADDRESS_1,labels=NP_KEY_1.1=NP_VALUE_1.1;NP_KEY_1.2=NP_VALUE_1.2;...' \
  

  The value has segments that start with the keywords node-ip and labels. Separate each segment with a comma.

  • node-ip: The IP address of a worker node. You can specify only one node-ip per flag. Add this flag again for each node in the node pool.

  • labels: One or more key=value pairs attached to the node.

  Note the following syntax rules:

  • Surround the entire value in single quotes.
  • Whitespace isn't allowed.
  • Separate each key=value pair in the labels segment with a semicolon.

  Optionally, you can specify the following:

  • --node-labels=KEY=VALUE,...: A comma-separated list of Kubernetes labels (key=value pairs) applied to each node in the pool.

  • --node-taints=KEY=VALUE:EFFECT,... A comma-separated list of Kubernetes taints applied to each node in the pool. Taints are key=value pairs associated with an effect. Taints are used with tolerations for Pod scheduling. Specify one of the following for EFFECT: NoSchedule, PreferNoSchedule, NoExecute.

The following example creates a node pool called default-pool on user-cluster-, and adds two nodes to the node pool. All both nodes are labeled with node-pool-key=node-pool-value and have the taint dedicated=experimental:PreferNoSchedule,

gcloud container bare-metal node-pools create default-pool \
  --cluster=user-cluster-1  \
  --project=example-project-12345 \
  --location=us-west1 \
  --node-configs='node-ip=10.200.0.10' \
  --node-configs='node-ip=10.200.0.11,labels=key2.1=value2.1' \
  --node-labels=node-pool-key=node-pool-value \
  --node-taints=dedicated=experimental:PreferNoSchedule

For more information, see the gcloud CLI reference.

Terraform

Before you begin

The Google Distributed Cloud version that you select when creating a user cluster must be a version that your admin cluster supports. Additionally, the latest minor or patch versions aren't available in the Anthos On-Prem API until 7 to 10 days after the release. You can run a gcloud command to get a list of supported versions that you can install on the user cluster.

1. Make sure to update components:

   gcloud components update
   

2. Get a list of available versions to install on the user cluster:

   gcloud container bare-metal clusters query-version-config \
     --admin-cluster-membership=ADMIN_CLUSTER_NAME \
     --admin-cluster-membership-project=FLEET_HOST_PROJECT_ID \
     --admin-cluster-membership-location=global \
     --location=REGION
   

   Replace the following:

   • ADMIN_CLUSTER_NAME: The name of the admin cluster.

   • FLEET_HOST_PROJECT_ID: The ID of the project that the admin cluster is registered to.

   • REGION: The Google Cloud region that you specified when you enrolled the cluster in the Anthos On-Prem API.

We suggest that you use the highest supported version to get the latest fixes and improvements.

Example

You can use the following basic configuration sample to create a user cluster with bundled MetalLB load balancer. For more information, see the google_gkeonprem_bare_metal_cluster reference documentation.

1. Clone the anthos-samples repository and change to the directory where the Terraform sample is located:

   git clone https://github.com/GoogleCloudPlatform/anthos-samples
   cd anthos-samples/anthos-onprem-terraform/abm_user_cluster_metallb
   

   The sample provides an example variables file to pass in to main.tf.

2. Make a copy of the terraform.tfvars.sample file:

   cp terraform.tfvars.sample terraform.tfvars
   

   
   project_id          = "PROJECT_ID"
   region              = "ON_PREM_API_REGION"
   admin_cluster_name  = "ADMIN_CLUSTER_NAME"
   bare_metal_version  = "VERSION"
   admin_user_emails   = ["YOUR_EMAIL_ADDRESS", "ADMIN_2_EMAIL_ADDRESS"]
   cluster_name        = "abm-user-cluster-metallb"
   control_plane_ips   = ["10.200.0.4"]
   worker_node_ips     = ["10.200.0.5", "10.200.0.6"]
   control_plane_vip   = "10.200.0.50"
   ingress_vip         = "10.200.0.51"
   lb_address_pools    = [
       { name = "lbpool_1", addresses = ["10.200.0.51-10.200.0.70"] }
   ]
   

3. Modify the parameters values in terraform.tfvars and save the file.

   The following list describes the variables:

   • project_id: The ID of the project that you want to create the cluster in. The specified project is also used as the fleet host project. This must be the same project that the admin cluster is registered to. After the user cluster is created, it is automatically registered to the selected project's fleet. The fleet host project can't be changed after the cluster is created.

   • region: The Google Cloud region in which the Anthos On-Prem API runs. Specify us-west1 or another supported region.

   • admin_cluster_name: The name of the admin cluster that manages the user cluster.

   • bare_metal_version: The Google Distributed Cloud version for your user cluster. Specify either the same version as the admin cluster, or a version that is no more than one minor version lower than the admin cluster.

   • cluster_name: A name of your choice for your user cluster. The name can't be changed after the cluster is created. The name must:

     • contain at most 40 characters
     • contain only lowercase alphanumeric characters or a hyphen (-)
     • start with an alphabetic character
     • end with an alphanumeric character

   • control_plane_ips: A list of one or more IPv4 addresses for the control plane nodes. Control plane nodes run the system workload. Typically, you have a single machine if using a minimum deployment, or three machines if using a high availability (HA) deployment. Specify an odd number of nodes to have a majority quorum for HA. You can change these addresses whenever you update or upgrade a cluster.

   • worker_node_ips: A list of one or more IPv4 addresses for the worker node machines.

   • control_plane_vip: The virtual IP address (VIP) that you have chosen to configure on the load balancer for the Kubernetes API server of the user cluster.

   • ingress_vip: The IP address that you have chosen to configure on the load balancer for the ingress proxy.

   • lb_address_pools: A list of maps that define the address pools to be used by the MetalLB load balancer. The ingress VIP must be in one of these pools.

   • admin_user_emails: A list of email addresses of the users to be granted administrative privileges on the cluster. Be sure to add your email address if you intend to administer the cluster.

   When the cluster is created, the Anthos On-Prem API applies the Kubernetes role-based access control (RBAC) policies to the cluster to grant the admin users the Kubernetes clusterrole/cluster-admin role, which provides full access to every resource in the cluster in all namespaces. This also lets users log on to the console using their Google identity.

4. Save the changes in terraform.tfvars.

5. Initialize and create the Terraform plan:

   terraform init
   

   Terraform installs any needed libraries, such as the Google Cloud provider.

6. Review the configuration and make changes if needed:

   terraform plan
   

7. Apply the Terraform plan to create the user cluster:

   terraform apply
   

   It takes 15 minutes or more to create the user cluster. You can view the cluster in the Google Cloud console on the Anthos clusters page.

Connect gateway

1. Either initialize the gcloud CLI for use with the fleet host project, or run the following commands to log in with your Google account, set your fleet host project as the default, and update components:

   gcloud auth login
   gcloud config set project PROJECT_ID
   gcloud components update
   

2. Fetch the cluster credentials used to interact with Connect gateway. In the following command, replaceMEMBERSHIP_NAME with your cluster's name. In Google Distributed Cloud, the membership name is the same as the cluster name.

   gcloud container fleet memberships get-credentials MEMBERSHIP_NAME
   

   This command returns a special Connect gateway-specific kubeconfig that lets you connect to the cluster through the gateway.

After you have the necessary credentials, you can run commands using kubectl as you normally would for any Kubernetes cluster, and you don't need to specify the name of the kubeconfig file, for example:

kubectl get namespaces

Admin workstation

Use the bmctl get credentials command to retrieve a kubeconfig file for the newly created user cluster.

bmctl get credentials --cluster CLUSTER_NAME --admin-kubeconfig ADMIN_KUBECONFIG_PATH

Replace the following:

• CLUSTER_NAME: the name of the target user cluster.

• ADMIN_KUBECONFIG_PATH: the path to the admin cluster kubeconfig file.

A kubeconfig with the user cluster's credentials are written to a file, bmctl-workspace/CLUSTER_NAME/CLUSTER_NAME-TIMESTAMP-kubeconfig. The TIMESTAMP in the file name indicates the date and time the file was created.

Because this file contains authentication credentials for your cluster, you should store it in a secure location with restricted access.