Version 1.3. This version is no longer supported. For more information see the version support policy.

Bundled load balancing with Seesaw

GKE on-prem can run in one of three load balancing modes: integrated, manual, or bundled. This topic shows how to configure GKE on-prem to run in bundled load balancing mode.

In bundled load balancing mode, GKE on-prem provides and manages the load balancer. You do not have to get a license for a load balancer, and the amount of setup that you have to do is minimal.

The bundled load balancer that GKE on-prem provides is the Seesaw load balancer.

Advantages of bundled load balancing mode

Bundled load balancing mode provides these advantages compared to manual load balancing mode:

A single team can be in charge of both cluster creation and load balancer configuration. For example, a cluster administration team would not have to rely on a separate networking team to acquire, run, and configure the load balancer ahead of time.
GKE on-prem automatically configures virtual IP addresses (VIPs) on the load balancer. At cluster creation time, GKE on-prem configures the load balancer with VIPs for the Kubernetes API server, the ingress service, and the cluster add-ons. As clients create Services of type LoadBalancer, GKE on-prem automatically configures the Service VIPs on the load balancer.
Dependencies among organizations, groups, and administrators are reduced. In particular, the group that manages a cluster is less dependent on the group that manages the network.

Recommended versions

We strongly recommend that you use vSphere 6.7 and Virtual Distributed Switch (VDS) 6.6 for bundled load balancing mode.

If you prefer, you can use earlier versions, but your installation will be less secure. The remaining sections in this topic give more detail about the security advantages of using vSphere 6.7 and VDS 6.6.

Planning your VLANs

A GKE on-prem installation has an admin cluster and one or more user clusters. With bundled load balancing mode, we strongly recommend that you have your clusters on separate VLANs. It is particularly important that your admin cluster is on its own VLAN.

Setting aside virtual IP addresses

Regardless of your choice of load balancing mode, you must set aside several virtual IP addresses (VIPs) that you intend to use for load balancing. These VIPs allow external clients to reach your Kubernetes API servers, your ingress services, and your addon services.

You must set aside a set of VIPs for your admin cluster and a set of VIPs for each user cluster that you intend to create. For a given cluster, these VIPs must be on the same VLAN as the cluster nodes and the Seesaw VMs for that cluster.

For instructions on setting aside VIPs, see Setting aside virtual IP addresses.

Setting aside node IP addresses

With bundled load balancing mode, you can specify static IP addresses for your cluster nodes, or your cluster nodes can get their IP addresses from a DHCP server.

If you want your cluster nodes to have static IP addresses, set aside enough addresses for the nodes in the admin cluster and the nodes in all the user clusters you intend to create. For details about how many node IP addresses to set aside, see Configuring static IP addresses.

Setting aside IP addresses and VIPs for Seesaw VMs

Next, set aside IP addresses and VIPs for the VMs that will run your Seesaw load balancers.

The number of addresses you set aside depends on whether you want to create highly available (HA) Seesaw load balancers or non-HA Seesaw load balancers.

Case 1: HA Seesaw load balancers

For your admin cluster, set aside two IP addresses for a pair of Seesaw VMs. Also for your admin cluster, set aside a single VIP for the pair of Seesaw VMs. All three of these addresses must be on the same VLAN as your admin cluster nodes.

For each user cluster that you intend to create, set aside two IP addresses for a pair of Seesaw VMs. Also for each user cluster, set aside a single VIP for the pair of Seesaw VM. For a given user cluster, all three of these addresses must be on the same VLAN as the user cluster nodes.

Case 2: Non-HA Seesaw load balancers

For your admin cluster, set aside one IP address for a Seesaw VM. Also for your admin cluster, set aside a VIP for the Seesaw load balancer. Both of these addresses must be on the same VLAN as your admin cluster nodes.

For each user cluster that you intend to create, set aside one IP address for a Seesaw VM. Also for each user cluster, set aside a VIP for the Seesaw load balancer. Both of these addresses must be on the same VLAN as the user cluster nodes.

Planning your port groups

Each of your Seesaw VMs has two network interfaces. One of those network interfaces is configured with the VIP of your Seesaw load balancer. The other network interface is configured with the IP address of the VM itself.

For an individual Seesaw VM, the two network interfaces can be connected to the same vSphere port group, or they can be connected to separate port groups. If the port groups are separate, they must be on the same VLAN.

This topic refers to two port groups:

load-balancer port group: For a Seesaw VM, the network interface that is configured with the VIP of the Seesaw load balancer is connected to this port group.
cluster-node port group: For a Seesaw VM, the network interface that is configured with the IP address of the VM itself is connected to this port group. Your GKE on-prem cluster nodes are also connected to this port group.

The load-balancer port group and the container node port group can be one and the same. But we strongly recommend that they are separate.

Creating hostconfig files

For each cluster that you intend to create, specify the addresses you have chosen for your Seesaw VMs in a hostconfig file. This hostconfig file is for your load balancer VMs, not your cluster nodes. If you intend to use static IP addresses for your cluster nodes, you must create a separate hostconfig file for those addresses. Here's an example of a hostconfig file that specifies two IP addresses for Seesaw VMs:

hostconfig:
  dns: "172.16.255.1"
  tod: "192.138.210.214"
  otherdns:
  - "8.8.8.8"
  - "8.8.4.4"
  othertod:
  - "ntp.ubuntu.com"
  searchdomainsfordns:
  - "my.local.com"
blocks:
  - netmask: "255.255.252.0"
    gateway: "110.116.232.1"
    ips:
    - ip: "172.16.20.18"
      hostname: "seesaw-1"
    - ip: "172.16.20.19"
      hostname: "seesaw-2"

Filling in your GKE on-prem configuration file

Before you create a cluster, you prepare a GKE on-prem configuration file.

In your configuration file, set lbmode to "Bundled".

A configuration file can hold the specification for a single cluster, or it can hold the specifications for several clusters. For each cluster in your configuration file, fill in the loadbalancerconfig section.

loadbalancerconfig:
  ipblockfilepath:
  vrid:
  vip:
  cpus:
  memorymb:
  enableha:
  antiaffinitygroups:
    enabled:
  network:

`loadbalancerconfig.ipblockfilepath`

Set loadbalancerconfig.ipfileblockpath to the path of the hostconfig file for your load balancer. For example:

ipblockfilepath: "config-file-directory/my-user-hostconfig.yaml"

`loadbalancerconfig.vrid`

Set loadbalancerconfig.vrid to the virtual router identifier of your Seesaw group. This identifier must be unique in a VLAN. Valid range is 1-255. For example:

vrid: 125

`loadbalancerconfig.vip`

Set loadbalancerconfig.vip to the VIP of your Seesaw group. For example:

vip: "172.16.20.21"

`loadbalancerconfig.cpus`

Set loadbalancerconfig.cpus to number of CPUs for each of your Seesaw VMs. For example:

cpus: 4

`loadbalancerconfig.memorymb`

Set loadbalancerconfig.memorymb to the number of megabytes of memory for each of your Seesaw VMs. For example:

memorymb: 3072

`loadbalancerconfig.antiaffinitygroups.enabled`

If you want to apply an anti-affinity rule to your Seesaw VMs, set the value of loadbalancerconfig.antiaffinitygroups.enabled to true. Otherwise set the value to false. The default value is true. The recommended value is true, so that your Seesaw VMs are put on different physical hosts whenever possible. For example:

antiaffinitygroups:
  enabled: true

`loadbalancerconfig.enableha`

If you want an HA Seesaw load balancer, set the value of loadbalancerconfig.enableha to true. Otherwise set the value to false. The default value is false.

enableha: true

If you set enableha to true, you must enable MAC learning.

`loadbalancerconfig.network`

Set loadbalancerconfig.network to the name of your load-balancer port group. If you leave this field blank, then for each Seesaw VM, the network interface that is configured with the VIP of your Seesaw load balancer will be connected to your cluster-node port group.

For example:

network: "USER_SEESAW_VIP_PORT_GROUP"

Example from a GKE on-prem configuration file

Here's an example that shows portions of a GKE on-prem configuration file. The configuration file describes two clusters: an admin cluster and a user cluster. Each cluster has a loadbalancerconfig section:

lbmode: "Bundled"
...

admincluster:
  loadbalancerconfig:
    ipblockfilepath: "admin-hostconfig.yaml"
    vrid: 124
    vip: 172.16.20.20
    cpus: 2
    memorymb: 3072
    enableha: true
    antiaffinitygroups:
      enabled: true
    network: "ADMIN_LOAD_BALANCER_PORT_GROUP"
...

usercluster:
  loadbalancerconfig:
    ipblockfilepath: "user-hostconfig.yaml"
    vrid: 125
    vip: 172.16.20.21
    cpus: 4
    memorymb: 3072
    enableha: true
    antiaffinitygroups:
      enabled: true
    network: "USER_LOAD_BALANCER_PORT_GROUP"

Enabling MAC learning or promiscuous mode (HA only)

If you are setting up a non-HA Seesaw load balancer, you can skip this section.

If you are setting up an HA Seesaw load balancer, you must enable some combination of MAC learning, forged transmits, and promiscuous mode on your load-balancer port group.

How you enable these features varies according to the type of switch you have:

Switch type	Enabling features	Security impact
vSphere 6.7 with VDS 6.6	Enable MAC learning and forged transmits for your load balancer by running this command: `gkectl prepare network --config [CONFIG_FILE]`, where `[CONFIG_FILE]` is the path of your GKE on-prem configuration file. You need dvPort group.Modify permission to do this.	Minimal. If your load-balancer port group is connected only to your Seesaw VMs, then you can limit MAC learning to your trusted Seesaw VMs.
vSphere 6.5 or vSphere 6.7 with a version of VDS lower than 6.6	Enable promiscuous mode and forged transmits for your load-balancer port group. Use the vSphere user interface on the port group page in Networking tab: Edit Settings -> Security.	All VMs on your load-balancer port group are in promiscuous mode. So any VM on your load-balancer port group can see all traffic. If your load-balancer port group is connected only to your Seesaw VMs, then it is only those VMs that can see all traffic.
NSX-T logical switch	Enable MAC learning on the logical switch.	vSphere does not support creating two logical switches in the same layer-2 domain. So the Seesaw VMs and the cluster nodes must be on the same logical switch. This means that MAC learning is enabled for all cluster nodes. An attacker might be able to achieve a MAC spoof by running privileged Pods in the cluster.
vSphere Standard Switch	Enable promiscuous mode and forged transmits for your load-balancer port group. Use the vSphere user interface on each ESXI host: Configure -> Virtual switches -> Standard Switch -> Edit Setting on the port group -> Security.	All VMs on your load-balancer port group are in promiscuous mode. So any VM on your load-balancer port group can see all traffic. If your load balancer-port group is connected only to your Seesaw VMs, then it is only those VMs that can see all traffic.

Running a preflight check on your configuration file

After you create your hostconfig files and your GKE on-prem configuration file, run a preflight check on your configuration file:

gkectl check-config --config [CONFIG_FILE]

where [CONFIG_FILE] is the path of your GKE on-prem configuration file.

If you already have an admin cluster, and your configuration file contains only usercluster specifications, then you need to include the kubeconfig file of your admin cluster in the command:

gkectl --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] check-config --config [CONFIG_FILE]

where [ADMIN_CLUSTER_KUBECONFIG] is the path of the kubeconfig file of your admin cluster.

If the preflight check fails, make adjustments to your GKE on-prem configuration file and your hostconfig files as needed. Then run the preflight check again.

Uploading OS images

Run this command to upload OS images to your vSphere environment:

gkectl prepare --config [CONFIG_FILE]

where [CONFIG_FILE] is the path of your GKE on-prem configuration file.

Creating your load balancer

Create and configure the VM(s) for your load balancer:

gkectl create loadbalancer --config [CONFIG_FILE]

where [CONFIG_FILE] is the path of your GKE on-prem configuration file.

Creating a cluster that uses bundled load balancing mode

Create your cluster(s):

gkectl create cluster --config [CONFIG_FILE]

where [CONFIG_FILE] is the path of your GKE on-prem configuration file.

If you already have an admin cluster, and your configuration file contains only usercluster specifications, then you need to include the kubeconfig file of your admin cluster in the command:

gkectl --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] check-config --config [CONFIG_FILE]

where [ADMIN_CLUSTER_KUBECONFIG] is the path of the kubeconfig file for your admin cluster.

Performance and load testing

The download throughput of your application scales linearly with the number of backends. This is because the backends send responses directly to the clients, bypassing the load balancer, using Direct Server Return.

In contrast, the upload throughput of your application is limited by the capacity of the one Seesaw VM that performs the load balancing.

Applications vary in the amount of CPU and memory that they require, so it is critically important that you do a load test before you start serving a large number of clients.

Testing indicates that a single Seesaw VM with 8 CPUs and 8 GB of memory can handle 10 GB/s (line rate) uploading traffic with 10 K concurrent TCP connections. However, it is important that you run your own load test if you plan to support a large number of concurrent TCP connections.

Scaling limits

With bundled load balancing, there are limits to how much your cluster can scale. There is a limit on the number of nodes in your cluster, and there is a limit on the number of Services that can be configured on your load balancer. There is also a limit on health checks. The number of health checks depends on both the number of nodes and the number of Services.

Starting with version 1.3.1, the number of health checks depend on the number of nodes and the number of traffic local Services. A traffic local Service is a Service that has its externalTrafficPolicy set to "Local".

	Version 1.3.0	Version 1.3.1 and later
Max Services (S)	100	500
Max nodes (N)	100	100
Max health checks	S * N <= 10K	N + L * N <= 10K, where L is number of traffic local services

Example: In version 1.3.1, suppose you have 100 nodes and 99 traffic local Services. Then the number of health checks is 100 + 99 * 100 = 10,000, which is within the 10K limit.

Upgrading the load balancer for your admin cluster

Starting with v1.4, load balancers are upgraded when upgrading the cluster. You don't need to run any other command to ugprade load balancers separately. But you can still use gkectl upgrade loadbalancer below to update some parameters.

To upgrade/update the load balancer separately, use the same configuration file that you intend to use to upgrade the cluster.

In the configuration file, update the bundlepath field, and verify that lbmode is set to "Bundled".

If you want to change the CPUs and memory for your Seesaw VMs, provide values for cpus and memorymb. Otherwise leave those fields empty.

For example:

lbmode: "Bundled"
bundlepath: "/var/lib/gke/bundles/gke-onprem-vsphere-1.3.2-gke.1-full.tgz"
admincluster:
  loadbalancerconfig:
    cpus: 6
    memorymb: 4096

Then run this command to upgrade your load balancer:

gkectl upgrade loadbalancer --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] --config [CLUSTER_CONFIG] --name gke-admin

where:

[ADMIN_CLUSTER_KUBECONFIG] is the kubeconfig file for your admin cluster.
[CLUSTER_CONFIG] is the configuration file for the cluster you intend to upgrade.

During an upgrade of a load balancer, there will be some down time. If HA is enabled for the load balancer, the maximum down time is two seconds.

Upgrading the load balancer for a user cluster

To upgrade/update the load balancer for a user cluster separately, follow the steps for upgrading the load balancer for your admin cluster with these two changes:

In the configuration file, verify that usercluster.clustername is set to the name of the cluster you want to update. For example:

lbmode: "Bundled"
bundlepath: "/var/lib/gke/bundles/gke-onprem-vsphere-1.3.2-gke.1-full.tgz"
usercluster:
  clustername: "my-cluster"
  loadbalancerconfig:
    cpus: 6
    memorymb: 4096

In the gkectl upgrade loadbalancer command, set --name to the name of the user cluster.

Viewing Seesaw logs

The Seesaw bundled load balancer stores log files on the Seesaw VMs in /var/log/seesaw/. The most important log file is seesaw_engine.INFO.

Viewing information about your Seesaw VMs

You can get information about your Seesaw VMs for a cluster from the SeesawGroup custom resource.

View the SeesawGroup custom resource for a cluster:

kubectl --kubeconfig [CLUSTER_KUBECONFIG] get seesawgroups -n kube-system -o yaml

where [CLUSTER_KUBECONFIG] is the path of the kubeconfig file for the cluster.

The output has an isReady field that shows whether the VMs are ready to handle traffic. The output also shows the names and IP addresses of the Seesaw VMs, and which VM is the control plane:

apiVersion: seesaw.gke.io/v1alpha1
kind: SeesawGroup
metadata:
  ...
  name: seesaw-for-cluster-1
  namespace: kube-system
  ...
spec: {}
status:
  machines:
  - hostname: cluster-1-seesaw-1
    ip: 172.16.20.18
    isReady: true
    lastCheckTime: "2020-02-25T00:47:37Z"
    role: Master
  - hostname: cluster-1-seesaw-2
    ip: 172.16.20.19
    isReady: true
    lastCheckTime: "2020-02-25T00:47:37Z"
    role: Backup

Viewing Seesaw metrics

The Seesaw bundled load balancer provides the following metrics:

Throughput per Service or node
Packet rate per Service or node
Active connections per Service or node
CPU and memory usage
Number of healthy backend Pods per Service
Which VM is the control plane and which is the backup
Uptime

You can use any monitoring and dashboarding solutions of your choice, as long as they support the Prometheus format. One possibility is to use the Prometheus and Grafana add-ons that are integrated with GKE on-prem.

Using the integrated Prometheus and Grafana add-ons

Enable Prometheus and Grafana for your cluster.

The next step is to create a Service object and an Endpoints object so that the Prometheus add-on knows about your Seesaw VMs.

Save the following configuration as seesaw-metrics.yaml. The configuration includes a Service manifest and an Endpoints manifest:

apiVersion: v1
kind: Service
metadata:
   name: seesaw-metrics
    annotations:
      monitoring.gke.io/scrape: 'true'
      monitoring.gke.io/scheme: 'https'
      monitoring.gke.io/tls_config: 'seesaw-ca'
spec:
    type: ClusterIP
    clusterIP: "None"
    ports:
    - name: metrics
      port: 20257
      protocol: TCP
      targetPort: 20257
---
kind: Endpoints
apiVersion: v1
metadata:
  name: seesaw-metrics
subsets:
 - addresses:
     - ip: [SEESAW_VM1_IP]
     - ip: [SEESAW_VM2_IP]
   ports:
     - name: metrics
       port: 20257

where:

[SEESAW_VM1_IP] is the IP address of one of your Seesaw VMs.
[SEESAW_VM2_IP] is the IP address of your other Seesaw VM.

Create the Service and Endpoints objects:

kubectl --kubeconfig [CLUSTER_KUBECONFIG] apply seesaw-metrics.yaml

Now you can create Grafana dashboards and charts. to view the metrics.

Deleting a load balancer

If you delete a cluster that uses bundled load balancing, you should then delete the Seesaw VMs for that cluster. You can do this by deleting the Seesaw VMs in the vSphere user interface.

As an alternative, you can run this command, which deletes the Seesaw VMs and the Seesaw group file:

gkectl delete loadbalance --config vsphere.yaml --seesaw-group-file [GROUP_FILE]

where

[GROUP_FILE] is the Seesaw group file. The group file is on your admin workstation next to config.yaml. The name of the group file has the form seesaw-for-[CLUSTER_NAME]-[IDENTIFIER].yaml.
vsphere.yaml is a file that contains the following information about your vCenter server:

vcenter:
  credentials:
    address:
    username:
    password:
  datacenter:
  cacertpath:

Troubleshooting

Getting an SSH connection to a Seesaw VM

Occasionally you might want to SSH into a Seesaw VM for troubleshooting or debugging.

Getting the SSH key

If you have already created your cluster, use the following steps to get the SSH key:

Get the seesaw-ssh Secret from the cluster. Get the SSH key from the Secret and base64 decode it. Save the decoded key in a temporary file:
```
kubectl --kubeconfig [CLUSTER_KUBECONFIG] get -n  kube-system secret seesaw-ssh -o \
jsonpath='{@.data.seesaw_ssh}' | base64 -d | base64 -d > /tmp/seesaw-ssh-key
```
where [CLUSTER_KUBECONFIG] is the kubeconfig file for your cluster.
Set the appropriate permissions for the key file:
```
chmod 0600 /tmp/seesaw-ssh-key
```

If you have not already created your cluster, use the following steps to get the SSH key:

Locate the file named seesaw-for-[CLUSTER_NAME]-[IDENTIFIER].yaml.

The file is called the group file and is located next to config.yaml.

Also, gkectl create loadbalancer prints the location of the group file.

In the file, get the value of credentials.ssh.privateKey, and base64 decode it. Save the decoded key in a temporary file:

cat seesaw-for-[CLUSTER_NAME]-[IDENTIFIER].yaml  | grep privatekey | sed 's/    privatekey: //g' \
| base64 -d > /tmp/seesaw-ssh-key

Set the appropriate permissions for the key file:
```
chmod 0600 /tmp/seesaw-ssh-key
```

Now you can SSH into the Seesaw VM:

ssh -i /tmp/seesaw-ssh-key ubuntu@[SEESAW_IP]

where [SEESAW_IP] is the IP address of the Seesaw VM.

Getting snapshots

You can capture snapshots for Seesaw VMs by using the gkectl diagnose snapshot command along with the --scenario flag.

If you set --scenario to all or all-with-logs, the output includes Seesaw snapshots along with other snapshots.

If you set --scenario to seesaw, the output includes only Seesaw snapshots.

Examples:

gkectl diagnose snapshot --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] --scenario seesaw

where [ADMIN_CLUSTER_KUBECONFIG] is the kubeconfig file for your admin cluster.

gkectl diagnose snapshot --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] --cluster-name [CLUSTER_NAME] --scenario seesaw

gkectl diagnose snapshot --seesaw-group-file [GROUP_FILE] --scenario seesaw

where [GROUP_FILE] is the path of the group file for the cluster: for example:seesaw-for-gke-admin-xxxxxx.yaml.

Known issues

Unable to upgrade load balancer of v1.3.x

It's known that if antiaffinitygroups is disabled for a Seesaw load balancer, upgrading the load balancer from v1.3.x to v1.3.x+ will fail with error:

updated SeesawGroup is not valid: SeesawConfig is invalid: AntiAffinityGroups must be set to default value if user doesn't provide it.

It has been fixed in v1.4 so you can choose to skip v1.3.x+ and upgrade to v1.4.