Create basic clusters

bmctl

When the GKE On-Prem API (gkeonprem.googleapis.com) is enabled for your project, newly created clusters are enrolled with the API automatically. Enrollment occurs even when spec.gkeOnPremAPI is unspecified in the cluster configuration file. When your cluster is enrolled with this Google Cloud-hosted API, you can use the Google Cloud console or the gcloud CLI to manage the cluster lifecycle.

To disable automatic enrollment, uncomment the spec.gkeOnPremAPI section in the cluster configuration file and set spec.gkeOnPremAPI.enabled to false before you create the cluster. The spec.gkeOnPremAPI.enabled field is mutable, so you can unenroll your cluster at any time after it's been created.

1. Generate a cluster configuration file:

   bmctl create config -c USER_CLUSTER_NAME \
     --project-id=PROJECT_ID

2. Verify the user cluster configuration file:

   The following cluster configuration file is filled in with the values that you entered in the planning table earlier. In addition to the values you entered, note the following differences from the generated configuration file:

   • Comments have been removed from this sample to improve readability.
   • The cluster type, spec.type, has been set to user.
   • The spec.clusterSecurity.authorization.clusterAdmin.gcpAccounts field was added to grant the clusterrole/cluster-admin to your account. Among other things, this field lets you sign in to your cluster in the Google Cloud console to see more cluster details.

   ---
   apiVersion: v1
   kind: Namespace
   metadata:
     name: cluster-USER_CLUSTER_NAME
   ---
   apiVersion: baremetal.cluster.gke.io/v1
   kind: Cluster
   metadata:
     name: USER_CLUSTER_NAME
     namespace: cluster-USER_CLUSTER_NAME
   spec:
     type: user
     profile: default
     anthosBareMetalVersion: CLUSTER_VERSION
     gkeConnect:
       projectID: PROJECT_ID
     controlPlane:
       nodePoolSpec:
         nodes:
         - address: USER_CP_NODE_IP
     clusterNetwork:
       pods:
         cidrBlocks:
         - 192.168.0.0/16
       services:
         cidrBlocks:
         - 10.96.0.0/20
     loadBalancer:
       mode: bundled
       ports:
         controlPlaneLBPort: 443
       vips:
         controlPlaneVIP: USER_CP_VIP
         ingressVIP: USER_INGRESS_VIP
       addressPools:
       - name: pool1
         addresses:
         - START_IP-END_IP
     clusterOperations:
       projectID: PROJECT_ID
       location: us-central1
     clusterSecurity:
       authorization:
         clusterAdmin:
           gcpAccounts:
           - GOOGLE_ACCOUNT_EMAIL
     storage:
       lvpNodeMounts:
         path: /mnt/localpv-disk
         storageClassName: local-disks
       lvpShare:
         path: /mnt/localpv-share
         storageClassName: local-shared
         numPVUnderSharedPath: 5
     nodeConfig:
       podDensity:
         maxPodsPerNode: 250
     # GKEOnPremAPI (Optional) Specify if you wish to explicitly enable/disable the cloud hosted gkeonprem
     # API to enable/disable cluster lifecycle management from gcloud UI and Terraform.
     # gkeOnPremAPI:
       # enabled: false
       # location is the Cloud location for the cluster resource metadata where the cluster will be enrolled.
       # location: us-central1
   ---
   apiVersion: baremetal.cluster.gke.io/v1
   kind: NodePool
   metadata:
     name: node-pool-1
     namespace: cluster-USER_CLUSTER_NAME
   spec:
     clusterName: USER_CLUSTER_NAME
     nodes:
     - address: USER_WORKER_NODE_IP
   

3. Replace the contents of the generated configuration file on your admin workstation with the contents from the preceding sample.

   Open the generated file, bmctl-workspace/USER_CLUSTER_NAME/USER_CLUSTER_NAME.yaml and replace its contents with the contents of the sample you verified in the preceding step.

4. Create the user cluster:

   bmctl create cluster -c USER_CLUSTER_NAME \
   --kubeconfig bmctl-workspace/ADMIN_CLUSTER_NAME/ADMIN_CLUSTER_NAME-kubeconfig

   The bmctl command displays output to the screen as it runs preflight checks and creates the cluster. Verbose information is written to logs in the baremetal/bmctl-workspace/USER_CLUSTER_NAME/log folder on the admin workstation.

   Cluster creation can take several minutes to finish.

5. Verify that your cluster was created and is running:

   kubectl --kubeconfig bmctl-workspace/USER_CLUSTER_NAME/USER_CLUSTER_NAME-kubeconfig \
     get nodes

   If your cluster is running, the response looks something like this:

   NAME    STATUS   ROLES           AGE     VERSION
   nuc-3   Ready    control-plane   4m32s   v1.26.2-gke.1001
   nuc-4   Ready    worker          2m2s    v1.26.2-gke.1001

Console

Do the following steps to create a user cluster in the console:

1. In the console, go to the Create a bare metal cluster page.

   Go to Create a bare metal cluster

2. Make sure that the Google Cloud project list has PROJECT_ID selected.

3. Click Create Cluster.

4. In the dialog, click On-premises.

5. Next to Bare metal, click Configure. The Prerequisites page displays.

6. Under Choose your cluster type, select Create a user cluster for an existing admin cluster

7. Click Next.

Cluster basics

1. Enter USER_CLUSTER_NAME as the name of the user cluster or use the default.

2. Make sure that the newly created admin cluster is selected.

3. In GCP API Location, select us-central1.

4. In Google Distributed Cloud version, select CLUSTER_VERSION or use the default. You can use the defaults for the rest of the settings on this page.

5. Click Networking in the left-navigation bar.

Networking

1. In the Control plane section, enter the following in the Control plane node IP 1 field:

   USER_CP_NODE_IP
   

2. In the Load balancer section, use the default load balancer, Bundled with MetalLB.

3. Go to the New address pool section. Enter the IP address range in the IP address range 1 field:

   10.200.0.51-10.200.0.70
   

4. Click Done.

5. In the Virtual IPs section, enter the IP address for the Control Plane VIP:

   USER_CP_VIP
   

6. Enter the IP address for the Ingress VIP:

   USER_INGRESS_VIP
   

7. In the Service and Pod CIDRs section, if you didn't change the Service and Pod CIDRs in the planner, you can use the defaults.

   If you changed the CIDRs, then enter the CIDRs you want to use:

   • Service CIDR:

   10.96.0.0/20
   

   • Pod CIDR:

   192.168.0.0/16
   

8. Click default pool in the left-navigation bar.

Create a node pool

Your cluster must have at least one node pool for worker nodes. A node pool is a template for the groups of worker nodes created in this cluster.

Enter the IP address of the user cluster worker node in the Nodes address 1 field:

USER_WORKER_NODE_IP

Create the cluster

1. Click Verify and Create to create the user cluster.

   It takes 15 minutes or more to create the user cluster. The console displays status messages as it verifies the settings and creates the cluster.

   If there is a problem with the configuration, the console displays an error message that should be clear enough for you to fix the configuration issue and try again to create the cluster.

   To see additional information about the creation process, click Show details to display a side panel. Click close to close the details panel.

   When the cluster is created, Cluster status: running is displayed.

2. After the cluster is created, click arrow_back Clusters to go back to the Clusters page.

gcloud CLI

This section shows you how to create a user cluster and a node pool using the gcloud CLI.

1. Run the following command to create a user cluster:

   gcloud container bare-metal clusters create USER_CLUSTER_NAME \
     --project=PROJECT_ID \
     --location=us-central1 \
     --admin-cluster-membership=ADMIN_CLUSTER_NAME \
     --admin-cluster-membership-project=PROJECT_ID \
     --admin-cluster-membership-location=global \
     --version=CLUSTER_VERSION \
     --admin-users=GOOGLE_ACCOUNT_EMAIL \
     --island-mode-service-address-cidr-blocks=10.96.0.0/20 \
     --island-mode-pod-address-cidr-blocks=192.168.0.0/16 \
     --metal-lb-address-pools='pool=lb-pool-1,manual-assign=True,addresses=START_IP-END_IP' \
     --control-plane-node-configs='node-ip=USER_CP_NODE_IP' \
     --control-plane-vip=USER_CP_VIP \
     --control-plane-load-balancer-port=443 \
     --ingress-vip=USER_INGRESS_VIP \
     --lvp-share-path=/mnt/localpv-share \
     --lvp-share-storage-class=local-shared \
     --lvp-node-mounts-config-path=/mnt/localpv-disk \
     --lvp-node-mounts-config-storage-class=local-disks
   

   The output from the command is similar to the following:

   Waiting for operation [projects/example-project-12345/locations/us-west1/operations/operation-1679543737105-5f7893fd5bae9-942b3f97-75e59179] to complete.
   

   In the example output, the string operation-1679543737105-5f7893fd5bae9-942b3f97-75e59179 is the OPERATION_ID of the long-running operation. You can find out the status of the operation with the following command:

   gcloud container bare-metal operations describe OPERATION_ID \
     --project=PROJECT_ID \
     --location=us-central1
   

   It takes about 15 minutes or more to create the cluster. As the cluster is being created, you can run the previous command every so often to get the current status.

   When the cluster is created, you see output like the following:

   Created Anthos cluster on bare metal [https://gkeonprem.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/bareMetalClusters/USER_CLUSTER_NAME].
   

2. After the cluster is created, run the following command to create a node pool on the newly created cluster.

   gcloud container bare-metal node-pools create node-pool-1 \
     --cluster=USER_CLUSTER_NAME \
     --project=PROJECT_ID \
     --location=us-central1 \
     --node-configs node-ip=USER_WORKER_NODE_IP

For more information and other examples, see the following:

• gcloud container bare-metal clusters

• gcloud container bare-metal node-pools

• Create a user cluster

Terraform

This section shows you how to create a user cluster and a node pool using Terraform.

1. Create a directory and a new file within that directory. The filename must have the .tf extension. In this guide, the file is called main.tf.

   mkdir DIRECTORY && cd DIRECTORY && touch main.tf
   

2. Verify the user cluster Terraform resource:

   The following Terraform resource example is filled in with the values that you entered in the planning table in the preceding section.

   resource "google_gkeonprem_bare_metal_cluster"  "cluster-basic" {
     provider = google-beta
     name = "USER_CLUSTER_NAME"
     project = "PROJECT_ID"
     location = "us-central1"
     admin_cluster_membership = "projects/PROJECT_ID/locations/global/memberships/ADMIN_CLUSTER_NAME"
     bare_metal_version = "CLUSTER_VERSION"
     network_config {
       island_mode_cidr {
         service_address_cidr_blocks = ["10.96.0.0/20"]
         pod_address_cidr_blocks = ["192.168.0.0/16"]
       }
     }
     control_plane {
       control_plane_node_pool_config {
         node_pool_config {
           labels = {}
           operating_system = "LINUX"
           node_configs {
             labels = {}
             node_ip = "USER_CP_NODE_IP"
           }
         }
       }
     }
     load_balancer {
       port_config {
         control_plane_load_balancer_port = 443
       }
       vip_config {
         control_plane_vip = "USER_CP_VIP"
         ingress_vip = "USER_INGRESS_VIP"
       }
       metal_lb_config {
         address_pools {
           pool = "pool1"
           addresses = [
             "START_IP-END_IP"
           ]
           avoid_buggy_ips = true
           manual_assign = true
         }
       }
     }
     storage {
       lvp_share_config {
         lvp_config {
           path = "/mnt/localpv-share"
           storage_class = "local-shared"
         }
         shared_path_pv_count = 5
       }
       lvp_node_mounts_config {
         path = "/mnt/localpv-disk"
         storage_class = "local-disks"
       }
     }
     security_config {
       authorization {
         admin_users {
           username = "GOOGLE_ACCOUNT_EMAIL"
         }
       }
     }
   }
   
   resource "google_gkeonprem_bare_metal_node_pool" "node-pool-default" {
     provider = google-beta
     name =  "node-pool-1"
     bare_metal_cluster = google_gkeonprem_bare_metal_cluster.cluster-basic.name
     project = "PROJECT_ID"
     location = "us-central1"
     node_pool_config {
       operating_system = "LINUX"
       node_configs {
         node_ip = "USER_WORKER_NODE_IP"
       }
     }
   }
   

3. Copy the Terraform resource to main.tf and save the file.

4. Initialize and create the Terraform plan:

   terraform init
   

   Terraform installs any needed libraries, such as the Google Cloud provider.

5. Review the configuration and make changes if needed:

   terraform plan
   

6. Apply the Terraform plan to create the user cluster:

   terraform apply
   

   When prompted, enter yes.

   It takes about 15 minutes (or more depending on your network) to create the basic user cluster and node pool.

7. To create a kubeconfig file for the user cluster, run the following command on the admin workstation:

   bmctl get credentials --cluster USER_CLUSTER_NAME \
   --admin-kubeconfig bmctl-workspace/ADMIN_CLUSTER_NAME/ADMIN_CLUSTER_NAME-kubeconfig

   The output from the command displays the name of the kubeconfig file for the user cluster, for example:

   bmctl-workspace/USER_CLUSTER_NAME/USER_CLUSTER_NAME-TIMESTAMP-kubeconfig

   The TIMESTAMP in the filename indicates the date and time the file was created.

   Because this file contains authentication credentials for your cluster, you should store it in a secure location with restricted access.

8. Verify that your cluster was created and is running:

   kubectl --kubeconfig bmctl-workspace/USER_CLUSTER_NAME/USER_CLUSTER_NAME-TIMESTAMP-kubeconfig \
       get nodes

   If your cluster is running, the response looks something like this:

   NAME    STATUS   ROLES           AGE     VERSION
   nuc-3   Ready    control-plane   4m32s   v1.26.2-gke.1001
   nuc-4   Ready    worker          2m2s    v1.26.2-gke.1001

For more information and other examples, see the following:

• google_gkeonprem_bare_metal_cluster

• google_gkeonprem_bare_metal_node_pool

• terraform.tfvars.sample

• Create a user cluster