Update clusters

bmctl

1. Change the values of the applicable fields in the cluster's configuration file, which by default is located here:
   bmctl-workspace/CLUSTER-NAME/CLUSTER-NAME.yaml

2. Update the cluster by running the bmctl update command:

   bmctl update cluster -c CLUSTER_NAME --kubeconfig=KUBECONFIG
   

   Replace the following:

   • CLUSTER_NAME: the name of the cluster you want to update.
   • KUBECONFIG: for admin, hybrid, or standalone clusters, enter the path to the cluster's kubeconfig file. For a user cluster, enter the path to the admin cluster's kubeconfig file.

gcloud CLI

1. Specify only the flags for the configuration that you want to modify.

2. Run the applicable update command:

   • Admin clusters: gcloud container bare-metal admin-clusters update

   • User clusters: gcloud container bare-metal clusters update.

   • Node pools on a user cluster: gcloud container bare-metal node-pools update

Terraform

1. Change the values of the applicable fields in the Terraform configuration file that you used to create the cluster or node pool. See the Terraform reference documentation for detailed field descriptions:

   • google_gkeonprem_bare_metal_admin_cluster

   • google_gkeonprem_bare_metal_cluster

   • google_gkeonprem_bare_metal_node_pool

2. Update the configuration by running the terraform apply command.

bmctl

You add or remove a node from a node pool by adding or removing the IP address of the node in a specific section of the cluster configuration file. The following list shows which section to edit for a given node pool:

• Worker node pool: add or remove the IP address of the node in the spec.nodes section of the NodePool spec.
• Control plane node pool: add or remove the IP address of the node in the spec.controlPlane.nodePoolSpec.nodes section of the Cluster spec.
• Load balancer node pool: add or remove the IP address of the node in the spec.loadBalancer.nodePoolSpec.nodes section of the Cluster spec.

Example: remove a worker node

Here's a sample cluster configuration file that shows the specifications of two worker nodes:

---
apiVersion: baremetal.cluster.gke.io/v1
kind: NodePool
metadata:
  name: nodepool1
  namespace: cluster-cluster1
spec:
  clusterName: cluster1
  nodes:
  - address: 192.0.2.1
  - address: 192.0.2.2

To remove a node:

1. (Optional) If the node that you're removing is running critical pods, first put the node into maintenance mode.

   You can monitor the node draining process for worker nodes by viewing the status.nodesDrained and status.nodesDraining fields on the NodePool resource.

2. Edit the cluster configuration file to delete the IP address entry for the node.

3. Update the cluster:

   bmctl update cluster1 \
       --kubeconfig=ADMIN_KUBECONFIG
   

gcloud CLI

You use an update command to add or remove nodes. The update command that you use and the flag in which you specify the IP address depends on the type of node pool that you want to update:

• Worker node pool: Run gcloud container bare-metal node-pools update and specify the IP address in the flag --node-configs 'node-ip=IP_ADDRESS'.

• Control plane node pool on an admin cluster: Run gcloud container bare-metal admin-clusters update and specify the IP address in the flag --control-plane-node-configs 'node-ip=IP_ADDRESS'.

• Control plane node pool on a user cluster: Run gcloud container bare-metal clusters update and specify the IP address in the flag --control-plane-node-configs 'node-ip=IP_ADDRESS'.

• Load balancer node pool: Run gcloud container bare-metal clusters update and specify the IP address in the flag --metal-lb-load-balancer-node-configs 'node-ip=IP_ADDRESS' or
  --bgp-load-balancer-node-configs 'node-ip=IP_ADDRESS'

The flag in which you specify the IP address only accepts one node-ip. You include the flag for each IP address in the node pool.

The update commands replace all IP addresses with the IP addresses that you specify. To add a node, include the IP addresses of the existing nodes and the IP address of the new node in the update command. Similarly, you remove nodes by only including the IP addresses of the nodes that you want to keep.

Example: remove a worker node

This section shows how to remove a worker node from a node pool using example data. Additional gcloud CLI commands that you might find useful are also included in the following steps.

1. (Optional) If the node that you're removing is running critical pods, first put the node into maintenance mode.

   You can monitor the node draining process for worker nodes by viewing the status.nodesDrained and status.nodesDraining fields on the NodePool resource.

2. Run the list command to list all the node pools in the cluster:

   gcloud container bare-metal node-pools list \
       --cluster=abm-user-cluster1 \
       --project=example-project-12345 \
       --location=us-central1
   

   The output is similar to the following:

   NAME         LOCATION     STATE
   node-pool-1  us-central1  RUNNING
   node-pool-2  asia-east1   RUNNING
   

3. Run the describe command to list all the IP addresses in the node pool:

   gcloud container bare-metal node-pools describe node-pool-1 \
       --cluster=abm-user-cluster1 \
       --project=example-project-12345 \
       --location=us-central1
   

   The following example output is truncated for readability:

   annotations:
     ...
     baremetal.cluster.gke.io/version: 1.31
   ...
   name: projects/example-project-12345/locations/us-central1/bareMetalClusters/abm-user-cluster1/bareMetalNodePools/node-pool-1
   nodePoolConfig:
     nodeConfigs:
     - nodeIp: 192.0.2.1
     - nodeIp: 192.0.2.2
     operatingSystem: LINUX
   state: RUNNING
   ...
   

   Note the following in the example output:

   • The name field contains the fully-qualified name of the node pool. When specifying the node pool name in a command, you can either specify the fully-qualified name, or the node pool name, for example, node-pool-1, along with the --cluster, --project, and the --location flags.

   • The nodeConfigs section contains two nodeIp fields with the IP addresses of the nodes.

4. Run the following command to remove the node with the IP address 192.0.2.1:

   gcloud container bare-metal node-pools update node-pool-1 \
       --cluster=abm-user-cluster1 \
       --project=example-project-12345 \
       --location=us-central1 \
       --node-configs='node-ip=192.0.2.2'
   

   The update command replaces all IP addresses with the IP addresses that you specify. Because 192.0.2.1 isn't included, the node is removed.

   The output of the command is similar to the following:

   Waiting for operation [projects/example-project-12345/locations/us-central1/operations/operation-1697154681749-6078d9def4030-76686d6e-9fcb1de9] to complete
   

   In the example output, the string operation-1697154681749-6078d9def4030-76686d6e-9fcb1de9 is the OPERATION_ID of the long-running operation. You can find out the status of the operation by running the following command in another terminal window:

   gcloud container bare-metal operations describe operation-1697154681749-6078d9def4030-76686d6e-9fcb1de9 \
       --project= example-project-12345 \
       --location=us-central1
   

   You can rerun the command every so often to check the status.

bmctl

You can use bmctl to replace high availability (HA) control plane nodes in all cluster types.

You replace a node in a cluster by performing the following steps:

1. Remove the node's IP address from the cluster configuration file.
2. Update the cluster.
3. Check the status of nodes in the cluster.
4. Add a new node's IP address to the same cluster configuration file.
5. Update the cluster.

Example: replace a HA control plane node

Here's a sample cluster configuration file that shows three control plane nodes in a user cluster:

---
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
name: user-cluster
namespace: cluster-user-cluster
spec:
  controlPlane:
  nodePoolSpec:
    nodes:
    - address: 192.0.2.11
    - address: 192.0.2.12
    - address: 192.0.2.13

To replace the last node listed in the spec.controlPlane.nodePoolSpec.nodes section, perform the following steps:

1. Remove the node by deleting its IP address entry in the cluster configuration file. After making this change, the cluster configuration file should look something like this:

   ---
   apiVersion: baremetal.cluster.gke.io/v1
   kind: Cluster
   metadata:
   name: user-cluster
   namespace: cluster-user-cluster
   spec:
     controlPlane:
     nodePoolSpec:
       nodes:
       - address: 192.0.2.11
       - address: 192.0.2.12
   

2. Update the cluster by running the following command:

   bmctl update cluster -c CLUSTER_NAME \
     --kubeconfig=KUBECONFIG
   

   Make the following changes:

   • Replace CLUSTER_NAME with the name of the cluster you want to update.
   • If the cluster is a self-managing cluster (such as admin or standalone cluster), replace KUBECONFIG with the path to the cluster's kubeconfig file. If the cluster is a user cluster, as it is in this example, replace KUBECONFIG with the path to the admin cluster's kubeconfig file.

3. After the bmctl update command has executed successfully, it takes some time to complete the machine-preflight and machine-init jobs. You can view the status of nodes and their respective node pools by running the commands described in the Verify your updates section of this document. After the node pool and nodes are in a ready state, you can proceed to the next step.

4. Add a new control plane node to the node pool by adding the IP address of the new control plane node to the spec.controlPlane.nodePoolSpec.nodes section of the cluster configuration file. After making this change, the cluster configuration file should look something like this:

   ---
   apiVersion: baremetal.cluster.gke.io/v1
   kind: Cluster
   metadata:
   name: user-cluster
   namespace: cluster-user-cluster
   spec:
     controlPlane:
     nodePoolSpec:
       nodes:
       - address: 192.0.2.11
       - address: 192.0.2.12
       - address: 192.0.2.14
   

5. Update the cluster by running the following command:

   bmctl update cluster -c CLUSTER_NAME \
     --kubeconfig=KUBECONFIG
   

gcloud CLI

You can use the gcloud CLI to replace high availability (HA) control plane nodes in admin and user clusters.

You replace a node in a cluster by performing the following steps:

1. Remove the node's IP address by running the applicable update command:

   • User cluster: gcloud container bare-metal clusters update
   • Admin cluster: gcloud container bare-metal admin-clusters update

2. Check the status of the node removal in the cluster by running gcloud container bare-metal operations describe OPERATION_ID.

3. Add the new node's IP address by running the applicable update command.

Example: replace a HA control plane node

This section shows how to replace a control plane from a cluster using example data. Additional gcloud CLI commands that you might find useful are also included in the following steps.

1. Run the list command to list all the user clusters in a Google Cloud project:

   gcloud container bare-metal clusters list \
       --project=example-project-12345 \
       --location=-
   

   When you set --location=-, that means to list all clusters in all regions. If you need to scope down the list, set --location to a specific region.

   The output is similar to the following:

   NAME                 LOCATION      VERSION   ADMIN_CLUSTER        STATE
   abm-user-cluster1a   us-central1   1.31      abm-admin-cluster1   RUNNING
   abm-user-cluster1b   europe-west1  1.31      abm-admin-cluster1   RUNNING
   

2. Run the describe command on the cluster:

   gcloud container bare-metal clusters describe abm-user-cluster1  \
       --project=example-project-12345 \
       --location=us-central1
   

   The example output is truncated for readability:

   ...
   controlPlane:
     controlPlaneNodePoolConfig:
       nodePoolConfig:
         nodeConfigs:
         - nodeIp: 192.0.2.11
         - nodeIp: 192.0.2.12
         - nodeIp: 192.0.2.13
         operatingSystem: LINUX
   ...
   name: projects/example-project-1234567/locations/us-central1/bareMetalClusters/abm-user-cluster1a
   ...
   

   Note the following in the example output:

   • The name field contains the fully-qualified name of the cluster. When specifying the cluster name in a command, you can either specify the fully-qualified name, or the cluster name, for example, abm-user-cluster1a, along with the --project and the --location flags.

   • The nodeConfigs section contains three nodeIp fields with the IP addresses of the control plane nodes.

3. Remove the node with the IP address 192.0.2.13:

   gcloud container bare-metal cluster update abm-user-cluster1a \
       --project=example-project-12345 \
       --location=us-central1 \
       --control-plane-node-configs 'node-ip=192.0.2.11'
       --control-plane-node-configs 'node-ip=192.0.2.12'
   

   The output of the command is similar to the following:

   Waiting for operation [projects/example-project-12345/locations/us-central1/operations/operation-1956154681749-6078d9def4030-76686d6e-9fcb1d7] to complete
   

   In the example output, the string operation-1956154681749-6078d9def4030-76686d6e-9fcb1de7 is the OPERATION_ID of the long-running operation. You can find out the status of the operation by running the following command in another terminal window:

   gcloud container bare-metal operations describe operation-1956154681749-6078d9def4030-76686d6e-9fcb1de7 \
       --project= example-project-12345 \
       --location=us-central1
   

   You can rerun the command every so often to check the status.

4. Add the new node with the IP address 192.0.2.14:

   gcloud container bare-metal cluster update abm-user-cluster1a \
       --project=example-project-12345 \
       --location=us-central1 \
       --control-plane-node-configs 'node-ip=192.0.2.11'
       --control-plane-node-configs 'node-ip=192.0.2.12'
       --control-plane-node-configs 'node-ip=192.0.2.14'
   

kubectl

You can view the status of nodes and their respective node pools with the kubectl get command.

For example, the following command shows the status of the node pools in the cluster namespace cluster-my-cluster:

kubectl -n cluster-my-cluster get nodepools.baremetal.cluster.gke.io

The system returns results similar to the following:

NAME                    READY   RECONCILING   STALLED   UNDERMAINTENANCE   UNKNOWN
cluster-my-cluster      3       0             0         0                  0
cluster-my-cluster-lb   2       0             0         0                  0
np1                     3       0             0         0                  0

Reconciling=1 means that the reconciliation step is still in progress. You should wait until the status changes to Reconciling=0.

You can also check status of nodes in the a cluster by running following command:

kubectl get nodes --kubeconfig=KUBECONFIG

gcloud CLI

As described previously, after running an update command, you can check the status of the operation using gcloud container bare-metal operations describe OPERATIONS_ID. The output of the command gives the status of the nodes, for example:

  ...
  metrics:
  - intValue: '1'
    metric: NODES_RECONCILING
  - intValue: '2'
    metric: NODES_HEALTHY
  - intValue: '0'
    metric: NODES_FAILED
  - intValue: '0'
    metric: NODES_IN_MAINTENANCE
  - intValue: '3'
    metric: NODES_TOTAL
  stage: HEALTH_CHECK
...

No matter which tool you use to update a node pool, you can get the current status of a node pool by running the applicable describe command as shown previously.

bmctl

The addressPools section contains fields for specifying load-balancing pools for the MetalLB and Border Gateway Protocol (BGP) bundled load balancers. You can add more load-balancing address pools at any time, but you can't remove any existing address pools. Starting with Google Distributed Cloud version 1.16.0, you can modify the values for addressPools.avoidBuggyIPs and addressPools.manualAssign at any time.

addressPools:
- name: pool1
  addresses:
  - 198.51.100.0-198.51.100.4
  - 198.51.100.240/28
- name: pool2
  addresses:
  - 198.51.100.224/28

gcloud CLI

You can add more load-balancing address pools at any time for the bundled load balancers, but you can't remove any existing address pools. The flag that you specify in gcloud container bare-metal clusters update to add an address pool depends on the type of bundled load balancer:

• MetalLB (layer 2): Use the --metal-lb-address-pools flag.
• Border Gateway Protocol (BGP): Use the --bgp-address-pools flag.

The value for the flags have the following format:

'pool=NAME,avoid-buggy-ips=True|False,manual-assign=True|False,addresses=IP_ADDRESS_RANGE_1;IP_ADDRESS_RANGE_2;...' \

The value has segments that start with the keywords pool, avoid-buggy-ip, manual-assign, and addresses. Separate each segment with a comma.

• pool: A name of your choice for the pool.

• avoid-buggy-ips: If you set this to True, the IP address management (IPAM) controller won't assign IP addresses ending in .0 or .255 to Services. This avoids the problem of buggy consumer devices mistakenly dropping traffic sent to those special IP addresses. If not specified, defaults to False. Starting with Google Distributed Cloud version 1.16.0, you can modify this value in an existing address pool.

• manual-assign: If you don't want the IPAM controller to automatically assign IP addresses from this pool to Services, set this to True. Then a developer can create a Service of type LoadBalancer and manually specify one of the addresses from the pool. If not specified, manual-assign is set to False. Starting with Google Distributed Cloud version 1.16.0, you can modify this value in an existing address pool.

• In the list of addresses: Each address must be a range either in CIDR or hyphenated-range format. To specify a single IP address in a pool (such as for the ingress VIP), use /32 in CIDR notation (for example, 192.0.2.1/32).

Note the following syntax rules:

• Surround the entire value in single quotes.
• Whitespace isn't allowed.
• Separate each IP address range with a semicolon.

You can specify more than one instance of the flag, as shown in the following example:

--metal-lb-address-pools='pool=pool2,avoid-buggy-ips=False,manual-assign=True,addresses=198.51.100.0/30;198.51.100.64-198.51.100.72'
--metal-lb-address-pools='pool=pool3,avoid-buggy-ips=True,manual-assign=True,addresses=203.0.113.0/28'

bmctl

If you add the baremetal.cluster.gke.io/prevent-deletion: "true" annotation to your cluster configuration file, you are prevented from deleting the cluster. For example, running kubectl delete cluster or bmctl reset cluster produces an error.

apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: ci-10c3c6f4d9c698e
  namespace: cluster-ci-10c3c6f4d9c698e
  annotations:
    baremetal.cluster.gke.io/prevent-deletion: "true"
spec:
  clusterNetwork:

gcloud CLI

If you specify the --add-annotations flag with the value baremetal.cluster.gke.io/prevent-deletion="true", you are prevented from deleting the cluster. For example:

1. Add the annotation to prevent accidental cluster deletion:

   gcloud container bare-metal clusters update abm-user-cluster1a \
       --project=example-project-12345 \
       --location=us-central1 \
       --add-annotations=baremetal.cluster.gke.io/prevent-deletion="true"
   

2. Attempt to delete the user cluster:

   gcloud container bare-metal clusters delete abm-user-cluster1a \
       --project=example-project-12345 \
       --location=us-central1 \
       --force \
       --allow-missing
   

   The response from the command is similar to the following:

   ERROR: (gcloud.container.bare-metal.clusters.delete) INVALID_ARGUMENT:
   invalid request: admission webhook "vcluster.kb.io" denied the request:
   annotations[baremetal.cluster.gke.io/prevent-deletion]: Invalid value:
   "true": Annotation "baremetal.cluster.gke.io/prevent-deletion" should be
   removed in order to delete this cluster
   

   To remove the annotation, specify --remove-annotations=baremetal.cluster.gke.io/prevent-deletion="true" in the update command.

bmctl

You can add or remove a user or service account as a cluster administrator for a user cluster by specifying email addresses in the clusterSecurity.authorization.clusterAdmin.gcpAccounts section of the cluster configuration file. The accounts are granted the cluster-admin role on the cluster, providing full access to the cluster.

apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: cluster1
  namespace: cluster-cluster1
spec:
  clusterSecurity:
    authorization:
      clusterAdmin:
        gcpAccounts:
        - alex@example.com
        - hao@example.com
        - my-sa@example-project-12345.iam.gserviceaccount.com

When updating a user cluster to add an account, be sure to include all accounts in the list (both existing and new accounts) because bmctl update overwrites the list with what you specify in the configuration file. To remove an account, remove it from the cluster configuration file and run bmctl update.

gcloud CLI

You can add or remove a user or service account as a cluster administrator by specifying an email addresses in the --admin-users flag. The flag accepts only one email address. To add multiple users, specify one account in each flag, for example:

gcloud container bare-metal clusters update abm-user-cluster1a \
    --project=example-project-12345 \
    --location=us-central1 \
    --admin-users=alex@example.com \
    --admin-users=hao@example.com
    --admin-users=my-sa@example-project-12345.iam.gserviceaccount.com

The update command overwrites the whole grant list. Specify all existing and new users who you want to be cluster administrators.

bmctl

apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: cluster1
  namespace: cluster-cluster1
  annotations:
    baremetal.cluster.gke.io/private-mode: "true"
spec:
  nodeAccess:
    loginUser: abm

gcloud CLI

You specify the user that you want to use to access node machines in the --login-user flag, for example:

gcloud container bare-metal clusters update abm-user-cluster1a \
    --project=example-project-12345 \
    --location=us-central1 \
    --login-user=abm

bmctl

Set clusterNetwork.advancedNetworking to truein the cluster configuration when you create your cluster:

apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: cluster1
  namespace: cluster-cluster1
spec:
  clusterNetwork:
    ...
    advancedNetworking: true
    ...

gcloud CLI

Include the --enable-advanced-networking flag in the gcloud container bare-metal clusters create command when you create your cluster.

bmctl

apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: cluster1
  namespace: cluster-cluster1
spec:
  ...
  clusterNetwork:
    services:
      cidrBlocks:
        - 192.0.2.0/14
  ...

gcloud CLI

To increase the range of the IPv4 service CIDR on a user cluster, specify the new range in the --island-mode-service-address-cidr-blocks flag.

gcloud container bare-metal clusters update cluster1 \
    --project=example-project-12345 \
    --location=us-central1 \
    --island-mode-service-address-cidr-blocks=192.0.2.0/14

bmctl

The following supported fields control Container Registry pull operations for kubelet:

• registryBurst (default: 10)
• registryPullQPS (default: 5)
• serializeImagePulls (default: true)

For more information about each of the kubelet configuration fields, see the Cluster configuration field reference.

You can specify these fields in kubeletConfig sections of the Cluster spec and the NodePool spec for the following pools of nodes:

• Cluster spec:
• Control plane nodes spec.controlPlane.nodePoolSpec.kubeletConfig
• Load balancer nodes spec.loadBalancer.nodePoolSpec.kubeletConfig
• NodePool spec:
• Worker nodes spec.kubeletConfig

The following example shows the added fields with their default values in the cluster configuration file. Note that the annotation preview.baremetal.cluster.gke.io/custom-kubelet: "enable" is required.

apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
  name: cluster1
  namespace: cluster-cluster1
  annotations:
    preview.baremetal.cluster.gke.io/custom-kubelet: "enable"
spec:
  ...
  controlPlane:
    nodePoolSpec:
      kubeletConfig:
        registryBurst: 10
        registryPullQPS: 5
        serializeImagePulls: true
  ...
  loadBalancer:
    nodePoolSpec:
      kubeletConfig:
        registryBurst: 10
        registryPullQPS: 5
        serializeImagePulls: true
  ...
apiVersion: baremetal.cluster.gke.io/v1
kind: NodePool
metadata:
  name: node-pool-new
  namespace: cluster-cluster1
spec:
  clusterName: cluster1
  ...
  kubeletConfig:
    registryBurst: 10
    registryPullQPS: 5
    serializeImagePulls: true

In each case, the setting applies to all nodes in the pool.

gcloud CLI

The following flags control Container Registry pull operations for kubelet:

• Control plane nodes

  • --control-plane-registry-burst
  • --control-plane-registry-pull-qps
  • --disable-control-plane-serialize-image-pulls
  • --enable-control-plane-serialize-image-pulls

• Load balancer nodes

  • --bgp-load-balancer-registry-burst
  • --bgp-load-balancer-registry-pull-qps
  • --disable-bgp-load-balancer-serialize-image-pulls
  • --enable-bgp-load-balancer-serialize-image-pulls
  • --metal-lb-load-balancer-registry-burst
  • --metal-lb-load-balancer-registry-pull-qps
  • --disable-metal-lb-load-balancer-serialize-image-pull
  • --enable-metal-lb-load-balancer-serialize-image-pulls

• Worker nodes

  • --registry-burst
  • --registry-pull-qps
  • --disable-serialize-image-pulls
  • --enable-serialize-image-pulls

bmctl

• bmctl command reference

• Cluster configuration field reference

gcloud CLI

• gcloud container bare-metal clusters

• gcloud container bare-metal node-pools

• gcloud container bare-metal admin-clusters

Terraform

• google_gkeonprem_bare_metal_cluster

• google_gkeonprem_bare_metal_node_pool

• google_gkeonprem_bare_metal_admin_cluster