Creating an admin workstation

This page explains how to create the admin workstation virtual machine (VM) for the first time. This page also explains how to configure the workstation's Docker daemon to work with a proxy.

See also Upgrading an existing admin workstation.

Overview

You use HashiCorp Terraform version 0.11 to provision an admin workstation virtual machine (VM) in vSphere, and you use the admin workstation to install GKE On-Prem. You download the admin workstation Open Virtual Appliance (OVA) file and use Terraform to deploy the VM to vSphere.

Before you begin

  1. Download the admin workstation OVA.
  2. Read the admin workstation overview.

Deploying the admin workstation OVA

To deploy the OVA, perform the following steps:

  1. Save one of the following Terraform configurations, depending on whether you're using DHCP or static IPs. Be sure to save both the TF and TFVARS files. The TF file is the Terraform HCL config, and the TFVARS file declares the variables to use in the configuration:

    dhcp.tf

    #########################
####### VARIABLES #######
#########################

# vSphere username
variable "vsphere_user" { }
# vSphere password
variable "vsphere_password" { }
# vSphere server address
variable "vsphere_server" { }
# Install this public key in the created VM
variable "ssh_public_key_path" { default = "~/.ssh/vsphere_workstation.pub" }
# vSphere network to use for the VM
variable "network" { default = "VM Network"}
# Hostname for the VM
variable "vm_name" { default = "vsphere-workstation" }
# vSphere datacenter to create this VM in
variable "datacenter" { }
# vSphere datastore to create this VM in
variable "datastore" { }
# vSphere cluster to create this VM in
variable "cluster" { }
# vSphere resource pool to create this VM in
variable "resource_pool" { }
# Number of CPUs for this VM. Recommended minimum 4.
variable "num_cpus" { default = 4 }
# Memory in MB for this VM. Recommended minimum 8192.
variable "memory" { default = 8192 }
# The VM template to clone
variable "vm_template" { }
# Enable the provided Docker registry. If you use your own registry, set to "false"
variable "registry_enable" { default = "true" }
# Username to set for the Docker registry
variable "registry_username" { default = "gke" }
# Password to set for the Docker registry
variable "registry_password" { default = "password" }
# Optional DNS hostname for the registry's certificate
variable "registry_dns_hostname" { default = "" }

#########################
##### FOR UPGRADING #####
#########################
# Path on disk to the htpasswd file
variable "registry_htpasswd" { default = "" } // filepath
# Path on disk to the certificate for the Docker registry in the admin workstation
variable "registry_cert" { default = "" } // filepath
# Path on disk to the registry's CA
variable "registry_ca" { default = "" } // filepath
# Path on disk to the registry's private key
variable "registry_private_key" { default = "" } // filepath


##########################
##########################

provider "vsphere" {
  version        = "~> 1.5"
  user           = "${var.vsphere_user}"
  password       = "${var.vsphere_password}"
  vsphere_server = "${var.vsphere_server}"

  # if you have a self-signed cert
  allow_unverified_ssl = true
}

### vSphere Data ###

data "vsphere_datastore" "datastore" {
  name          = "${var.datastore}"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

data "vsphere_datacenter" "dc" {
  name = "${var.datacenter}"
}

data "vsphere_compute_cluster" "cluster" {
  name          = "${var.cluster}"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

data "vsphere_resource_pool" "pool" {
  name          = "${var.resource_pool}"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

data "vsphere_network" "network" {
  name          = "${var.network}"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

data "vsphere_virtual_machine" "template_from_ovf" {
  name          = "${var.vm_template}"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

data "template_file" "dhcp_ip_config" {
  template = <<EOF
network:
  version: 2
  ethernets:
    ens192:
      dhcp4: true
EOF
}

data "template_file" "user_data" {
  template = <<EOF
#cloud-config
apt:
  primary:
    - arches: [default]
      uri: http://us-west1.gce.archive.ubuntu.com/ubuntu/
write_files:
  - path: /etc/netplan/99-dhcp.yaml
    permissions: '0644'
    encoding: base64
    content: |
      $${dhcp_ip_config}
runcmd:
  - netplan apply
  - /var/lib/gke/guest-startup.sh $${reg_enable} $${reg_username} $${reg_password} $${reg_dns_hostname} $${reg_htpasswd} $${reg_cert} $${reg_private_key} $${reg_ca}
EOF
  vars = {
    dhcp_ip_config = "${base64encode(data.template_file.dhcp_ip_config.rendered)}"

    reg_enable = "${var.registry_enable}"
    reg_username = "${var.registry_username}"
    reg_password = "${var.registry_password}"
    reg_dns_hostname = "${var.registry_dns_hostname}"

    reg_htpasswd = ""
    reg_cert = ""
    reg_private_key = ""
    reg_ca = ""

    #########################
    ##### FOR UPGRADING #####
    # reg_htpasswd = "${file(var.registry_htpasswd)}"
    # reg_cert = "${file(var.registry_cert)}"
    # reg_private_key = "${file(var.registry_private_key)}"
    # reg_ca = "${file(var.registry_ca)}"
    #########################
  }
}

### vSphere Resources ###

resource "vsphere_virtual_machine" "vm" {
  name             = "${var.vm_name}"
  resource_pool_id = "${data.vsphere_resource_pool.pool.id}"
  datastore_id     = "${data.vsphere_datastore.datastore.id}"
  num_cpus         = "${var.num_cpus}"
  memory           = "${var.memory}"
  guest_id         = "${data.vsphere_virtual_machine.template_from_ovf.guest_id}"
  enable_disk_uuid = "true"
  scsi_type = "${data.vsphere_virtual_machine.template_from_ovf.scsi_type}"
  network_interface {
    network_id   = "${data.vsphere_network.network.id}"
    adapter_type = "${data.vsphere_virtual_machine.template_from_ovf.network_interface_types[0]}"
  }

  wait_for_guest_net_timeout = 15

  nested_hv_enabled = false
  cpu_performance_counters_enabled = false

  disk {
    label            = "disk0"
    size             = "${max(50, data.vsphere_virtual_machine.template_from_ovf.disks.0.size)}"
    eagerly_scrub    = "${data.vsphere_virtual_machine.template_from_ovf.disks.0.eagerly_scrub}"
    thin_provisioned = "${data.vsphere_virtual_machine.template_from_ovf.disks.0.thin_provisioned}"
  }

  cdrom {
    client_device = true
  }

  vapp {
    properties = {
      hostname    = "${var.vm_name}"
      public-keys = "${file(var.ssh_public_key_path)}"
      user-data   = "${base64encode(data.template_file.user_data.rendered)}"
    }
  }

  clone {
    template_uuid = "${data.vsphere_virtual_machine.template_from_ovf.id}"
  }
}

output "ip_address" {
  value = "${vsphere_virtual_machine.vm.default_ip_address}"
}

    dhcp.tfvars

    vsphere_user = "administrator@vsphere.local"
vsphere_password = ""
# vSphere server IP or DNS name
vsphere_server = ""
ssh_public_key_path = "~/.ssh/vsphere_workstation.pub"

vm_name = "admin-workstation"

datastore = ""
datacenter = ""
cluster = ""
resource_pool = ""
network = "VM Network"

num_cpus = 4
memory = 8192
vm_template = "gke-on-prem-admin-appliance-vsphere-1.0.2-gke.3"

# docker registry credentials
registry_enable = "true"
registry_username = "gke"
registry_password = "password"
registry_dns_hostname = ""

# only needed when re-creating the workstation and you want to
# reuse registry certs. Each value is a filepath relative to this file.
# registry_htpasswd=""
# registry_cert=""
# registry_ca=""
# registry_private_key=""

    static-ip.tf

    #########################
####### VARIABLES #######
#########################

# vSphere username
variable "vsphere_user" { }
# vSphere password
variable "vsphere_password" { }
# vSphere server address
variable "vsphere_server" { }
# Install this public key in the created VM
variable "ssh_public_key_path" { default = "~/.ssh/vsphere_workstation.pub" }
# vSphere network to use for the VM
variable "network" { default = "VM Network"}
# Hostname for the VM
variable "vm_name" { default = "vsphere-workstation" }
# vSphere datacenter to create this VM in
variable "datacenter" { }
# vSphere datastore to create this VM in
variable "datastore" { }
# vSphere cluster to create this VM in
variable "cluster" { }
# vSphere resource pool to create this VM in
variable "resource_pool" { }
# Number of CPUs for this VM. Recommended minimum 4.
variable "num_cpus" { default = 4 }
# Memory in MB for this VM. Recommended minimum 8192.
variable "memory" { default = 8192 }
# The VM template to clone
variable "vm_template" { }
# The IP address to assign this this VM
variable "ipv4_address" { }
# Netmask prefix length
variable "ipv4_netmask_prefix_length" { }
# Default gateway to use
variable "ipv4_gateway" { }
# DNS resolvers to use
variable "dns_nameservers" { }
# Enable the provided Docker registry. If you use your own registry, set to "false"
variable "registry_enable" { default = "true" }
# Username to set for the Docker registry
variable "registry_username" { default = "gke" }
# Password to set for the Docker registry
variable "registry_password" { default = "password" }
# Optional DNS hostname for the registry's certificate
variable "registry_dns_hostname" { default = "" }

#########################
##### FOR UPGRADING #####
#########################
# Path on disk to the htpasswd file
variable "registry_htpasswd" { default = "" } // filepath
# Path on disk to the certificate for the Docker registry in the admin workstation
variable "registry_cert" { default = "" } // filepath
# Path on disk to the registry's CA
variable "registry_ca" { default = "" } // filepath
# Path on disk to the registry's private key
variable "registry_private_key" { default = "" } // filepath


##########################
##########################

provider "vsphere" {
  version        = "~> 1.5"
  user           = "${var.vsphere_user}"
  password       = "${var.vsphere_password}"
  vsphere_server = "${var.vsphere_server}"

  # if you have a self-signed cert
  allow_unverified_ssl = true
}

### vSphere Data ###

data "vsphere_datastore" "datastore" {
  name          = "${var.datastore}"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

data "vsphere_datacenter" "dc" {
  name = "${var.datacenter}"
}

data "vsphere_compute_cluster" "cluster" {
  name          = "${var.cluster}"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

data "vsphere_resource_pool" "pool" {
  name          = "${var.resource_pool}"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

data "vsphere_network" "network" {
  name          = "${var.network}"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

data "vsphere_virtual_machine" "template_from_ovf" {
  name          = "${var.vm_template}"
  datacenter_id = "${data.vsphere_datacenter.dc.id}"
}

##########################
### IF USING STATIC IP ###
##########################
data "template_file" "static_ip_config" {
  template = <<EOF
network:
  version: 2
  ethernets:
    ens192:
      dhcp4: no
      dhcp6: no
      addresses: ["${var.ipv4_address}/${var.ipv4_netmask_prefix_length}"]
      gateway4: ${var.ipv4_gateway}
      nameservers:
        addresses: [${var.dns_nameservers}]
EOF
}

data "template_file" "user_data" {
  template = <<EOF
#cloud-config
apt:
  primary:
    - arches: [default]
      uri: http://us-west1.gce.archive.ubuntu.com/ubuntu/
write_files:
  - path: /tmp/static-ip.yaml
    permissions: '0644'
    encoding: base64
    content: |
      $${static_ip_config}
runcmd:
  - /var/lib/gke/guest-startup.sh $${reg_enable} $${reg_username} $${reg_password} $${reg_dns_hostname} $${reg_htpasswd} $${reg_cert} $${reg_private_key} $${reg_ca}
EOF
  vars = {
    static_ip_config = "${base64encode(data.template_file.static_ip_config.rendered)}"

    reg_enable = "${var.registry_enable}"
    reg_username = "${var.registry_username}"
    reg_password = "${var.registry_password}"
    reg_dns_hostname = "${var.registry_dns_hostname}"

    reg_htpasswd = ""
    reg_cert = ""
    reg_private_key = ""
    reg_ca = ""

    ########################
    #### FOR UPGRADING #####
    # reg_htpasswd = "${file(var.registry_htpasswd)}"
    # reg_cert = "${file(var.registry_cert)}"
    # reg_private_key = "${file(var.registry_private_key)}"
    # reg_ca = "${file(var.registry_ca)}"
    ########################
  }
}
##########################
### IF USING STATIC IP ###
##########################

### vSphere Resources ###

resource "vsphere_virtual_machine" "vm" {
  name             = "${var.vm_name}"
  resource_pool_id = "${data.vsphere_resource_pool.pool.id}"
  datastore_id     = "${data.vsphere_datastore.datastore.id}"
  num_cpus         = "${var.num_cpus}"
  memory           = "${var.memory}"
  guest_id         = "${data.vsphere_virtual_machine.template_from_ovf.guest_id}"
  enable_disk_uuid = "true"
  scsi_type = "${data.vsphere_virtual_machine.template_from_ovf.scsi_type}"
  network_interface {
    network_id   = "${data.vsphere_network.network.id}"
    adapter_type = "${data.vsphere_virtual_machine.template_from_ovf.network_interface_types[0]}"
  }

  wait_for_guest_net_timeout = 15

  nested_hv_enabled = false
  cpu_performance_counters_enabled = false

  disk {
    label            = "disk0"
    size             = "${max(50, data.vsphere_virtual_machine.template_from_ovf.disks.0.size)}"
    eagerly_scrub    = "${data.vsphere_virtual_machine.template_from_ovf.disks.0.eagerly_scrub}"
    thin_provisioned = "${data.vsphere_virtual_machine.template_from_ovf.disks.0.thin_provisioned}"
  }

  cdrom {
    client_device = true
  }

  vapp {
    properties = {
      hostname    = "${var.vm_name}"
      public-keys = "${file(var.ssh_public_key_path)}"
      user-data   = "${base64encode(data.template_file.user_data.rendered)}"
    }
  }

  clone {
    template_uuid = "${data.vsphere_virtual_machine.template_from_ovf.id}"
  }
}

output "ip_address" {
  value = "${vsphere_virtual_machine.vm.default_ip_address}"
}

    static-ip.tfvars

    vsphere_user = "administrator@vsphere.local"
vsphere_password = ""
# vSphere server IP or DNS name
vsphere_server = ""
ssh_public_key_path = "~/.ssh/vsphere_workstation.pub"

vm_name = "admin-workstation"

datastore = ""
datacenter = ""
cluster = ""
resource_pool = ""
network = "VM Network"

num_cpus = 4
memory = 8192
vm_template = "gke-on-prem-admin-appliance-vsphere-1.0.2-gke.3"

# XXX.XXX.XXX.XXX
ipv4_address = "100.115.250.100"
# Eg 22
ipv4_netmask_prefix_length = "22"
# XXX.XXX.XXX.XXX
ipv4_gateway = "100.115.251.254"
# comma separated DNS servers
dns_nameservers = "8.8.8.8,8.8.4.4"

# docker registry credentials
registry_enable = "true"
registry_username = "gke"
registry_password = "password"
registry_dns_hostname = ""

# only needed when re-creating the workstation and you want to
# reuse registry certs. Each value is a filepath relative to this file.
# registry_htpasswd=""
# registry_cert=""
# registry_ca=""
# registry_private_key=""

  2. Create an SSH key-value pair so that you can authenticate to the admin workstation VM:

    ssh-keygen -t rsa -f ~/.ssh/vsphere_workstation -N ""

  3. Create a directory for your terraform files:

    mkdir [TERRAFORM_DIR]

    where [TERRAFORM_DIR] is the path of a directory where you want to keep your Terraform files.

  4. Copy the terraform config matching your host network configuration to [TERRAFORM_DIR]/terraform.tf and [TERRAFORM_DIR]/terraform.tfvars.

  5. In terraform.tfvars enter values for the following variables, which specify elements of your vSphere environment:

    • vsphere_server
    • vspher_user
    • vsphere_password
    • datastore
    • datacenter
    • cluster
    • resource_pool
    • network
    • vm_template (You created this in a previous step by running govc markastemplate.)

    Enter a value for vm_name. This is the name of the admin workstation you are about to create. It can be any name of your choice.

    If you are using static IP addresses, enter values for the following variables:

    • ipv4_address
    • ipv4_netmask_prefix_length
    • ipv4_gateway
    • dns_nameservers

    If you want to use the Docker registry that runs on the admin workstation VM, set registry_enable to "true". Also enter values for these variables:

    • registry_username
    • registry_password

    If you want to have a DNS hostname for the Docker registry that runs on the admin workstation VM, enter a value for registry_dns_hostname. This is useful if your admin workstation VM does not have a static IP address.

    If you want to use your own private Docker registry, set registry_enable to "false".

  6. Create the admin workstation VM. This might take a few minutes:

    cd [TERRAFORM_DIR]
terraform init && terraform apply -auto-approve -input=false

SSH in to your admin workstation

  1. Change to the directory containing the Terraform configuration.

  2. Retrieve the IP address of the admin workstation VM:

    terraform output ip_address

  3. SSH in to the admin workstation VM by using the generated key and IP address:

    ssh -i ~/.ssh/vsphere_workstation ubuntu@$(terraform output ip_address)

    or, if you want to just use its address:

    ssh -i ~/.ssh/vsphere_workstation ubuntu@[IP_ADDRESS]

Verifying that the admin workstation is set up correctly

Verify that gkectl and docker are installed:

gkectl version
docker version

If you set registry_enable to "true", you can verify that Docker images for the GKE On-Prem components have been loaded into the Docker daemon of your admin workstation VM:

docker images

It can take fifteen minutes or more for all of the Docker images to be loaded.

Eventually, you can verify that your admin workstation VM is running a Docker registry:

docker ps

The output shows the container that is running the Docker registry:

IMAGE ...                      COMMAND ...              NAMES
docker-registry:v2.7.1-gke.0   "registry serve /e..."   registry2

Configuring Docker to pull through your proxy

If your network runs behind a proxy, and your admin workstation VM is running a Docker registry, you need to configure the Docker daemon running on your admin workstation to pull images through your proxy:

  1. Gather the addresses of your HTTP and HTTPS proxies.

  2. Gather IP addresses and hostnames of every host that you need to contact without proxying, including:

    • The vCenter server IP address.
    • The IP addresses of all ESXi hosts.
    • IP addresses that you intend to configure on your load balancer.
    • The 192.168.0.0/16 range.

    In your admin workstation, add these addresses to the no_proxy variable:

    printf -v no_proxy '%s,' [ADDRESSES];

    Optionally, you can export the range to an environment variable for later reference. Keep in mind that applications and processes might use this variable:

    export no_proxy="${no_proxy%,}";

  3. Open Docker's configuration file, stored at /root/.docker/config.json, /home/[USER]/.docker/config.json, or elsewhere depending on your setup.

  4. Within config.json, add the following lines:

    "proxies": {

"default": {
       "httpProxy": "[HTTP_PROXY]",
       "httpsProxy": "[HTTPS_PROXY]",
       "noProxy": "[ADDRESSES]"
           }
}

    where:

    • [HTTP_PROXY] is your HTTP proxy, if you have one.
    • [HTTPS_PROXY] is your HTTPS proxy, if you have one.
    • [ADDRESSES] is a comma-delimited list of addresses and hostnames that you need to contact without proxying.

  5. Restart Docker for the changes to take effect:

    sudo systemctl restart docker

Troubleshooting

For more information, refer to Troubleshooting.

Terraform vSphere provider session limit

GKE On-Prem uses Terraform's vSphere provider to bring up VMs in your vSphere environment. The provider's session limit is 1000 sessions. The current implementation doesn't close active sessions after use. You might encounter 503 errors if you have too many sessions running.

Sessions are automatically closed after 300 seconds.

Symptoms

If you have too many sessions running, you might encounter the following error:

Error connecting to CIS REST endpoint: Login failed: body:
  {"type":"com.vmware.vapi.std.errors.service_unavailable","value":
  {"messages":[{"args":["1000","1000"],"default_message":"Sessions count is
  limited to 1000. Existing sessions are 1000.",
  "id":"com.vmware.vapi.endpoint.failedToLoginMaxSessionCountReached"}]}},
  status: 503 Service Unavailable
Potential causes

There are too many Terraform provider sessions running in your environment.

Resolution

Currently, this is working as intended. Sessions are automatically closed after 300 seconds. For more information, refer to to GitHub issue #618.

Using a proxy for Docker: oauth2: cannot fetch token

Symptoms

While using a proxy, you encounter the following error:

oauth2: cannot fetch token: Post https://oauth2.googleapis.com/token: proxyconnect tcp: tls: oversized record received with length 20527
Potential causes

You might have provided a HTTPS proxy instead of HTTP.

Resolution

In your Docker configuration, change the proxy address to http:// instead of https://.

Verifying that licenses are valid

Remember to verify that your licenses is valid, especially if you are using trial licenses. You might encounter unexpected failures if your F5, ESXi host, or vCenter licenses have expired.

openssl can't validate admin workstation OVA

Symptoms

Running openssl dgst against the admin workstation OVA file doesn't return Verified OK

Potential causes

An issue is present in the OVA file that prevents successful validation.

Resolution

Try downloading and deploying the admin workstation OVA again, as instructed in Download the admin workstation OVA . If the issue persists, reach out to Google for assistance.

What's next

Before installing GKE On-Prem, here are a few optional configuration options:

Previous
Getting started