Creating an admin workstation

This document shows how to create an admin workstation, for Google Distributed Cloud. The admin workstation hosts command-line interface (CLI) tools and configuration files to provision clusters during installation, and CLI tools for interacting with provisioned clusters post-installation.

This page is for Admins, Architects, and Operators who set up, monitor, and manage the tech infrastructure. To learn more about common roles and example tasks that we reference in Google Cloud content, see Common GKE Enterprise user roles and tasks.

The instructions here are complete. For a shorter introduction to creating an admin workstation, see Create an admin workstation in the Create basic clusters guide.

There are two ways you can create an admin workstation:

  • Use gkeadm to create an admin workstation VM in your vSphere environment.

  • Create a user-managed admin workstation on any computer of your choice.

gkeadm

Before you begin

Know your vCenter server address.

Know your CA cert path.

Install the Google Cloud CLI.

Create one or more Google Cloud projects as described in Using multiple Google Cloud projects.

Planning your service accounts

When you use gkeadm to create an admin workstation, you have the option of letting gkeadm create some of your service accounts and keys for you. In that case, gkeadm also grants the appropriate Identity and Access Management roles to the service accounts.

As an alternative, you can create your service accounts and keys manually. In that case, you must manually grant IAM roles to your service accounts.

Creating service accounts manually gives you more flexibility than having gkeadm create them for you:

  • Automatically created service accounts all have the same parent Google Cloud project as your component access service account. When you create a service account manually, you can choose the parent Google Cloud project.

  • Automatically created service accounts are all granted IAM roles on the parent Google Cloud project of your component access service account. This is fine if that is the only Google Cloud project associated with your clusters. But if you want to associate your clusters with multiple Google Cloud projects, then you need the flexibility to grant roles to a service account on a Google Cloud project of your choice.

If you decide to create your own service accounts, follow the instructions in Service accounts and keys.

Regardless of whether you have gkeadm create service accounts for you automatically, there is one service account that you must create manually: your component access service account. For instructions on how to create your component access service account and grant it the appropriate IAM roles, see Component access service account.

There is one other service account that you might need to create manually: your audit logging service account. If you want to use a GKE On-Prem API client to manage your user clusters, then you must enable audit logging in your admin cluster.

Generating templates for your configuration files

Download gkeadm to your current directory.

Generate templates:

./gkeadm create config

The preceding command created these files in your current directory:

  • credential.yaml
  • admin-ws-config.yaml

Filling in credential.yaml

In credential.yaml, fill in your vCenter username and password. For example:

kind: CredentialFile
items:
- name: vCenter
username: "my-account-name"
password: "AadmpqGPqq!a"

Filling in admin-ws-config.yaml

Several fields in admin-ws-config.yaml are already filled in with default or generated values. You can keep the populated values or make changes as you prefer.

Fields that you must fill in

Fill in the following required fields. For information on how to fill in the fields, Admin workstation configuration file.

gcp:
 componentAccessServiceAccountKeyPath: "Fill in"
vCenter:
credentials:
  address: "Fill in"
datacenter: "Fill in"
datastore: "Fill in"
cluster: "Fill in"
network: "Fill in"
resourcePool: "Fill in"
caCertPath: "Fill in"

If you want to create your admin workstation inside a vSphere VM folder, fill in the vCenter.folder field:

vCenter:
folder: "Fill in"

If your admin workstation will be behind a proxy server, fill in the proxyURL field:

adminWorkstation:
proxyURL: "Fill in"

If you want your admin workstation to get its IP address from a DHCP server, set ipAllocationMode to "dhcp", and remove the hostconfig section:

adminWorkstation:
network:
  ipAllocationMode: "dhcp"

If you want to specify a static IP address for your admin workstation, set ipAllocationMode to "static", and fill in the hostconfig section:

adminWorkstation:
network:
  ipAllocationMode: "static"
  hostconfig:
    ip: "Fill in"
    gateway: "Fill in"
    netmask: "Fill in"
    dns:
    - "Fill in"

Logging in

  1. Log in with any Google Account. This sets your SDK account property:
gcloud auth login
  1. Verify that your SDK account property is set correctly:
gcloud config list
  1. The Google Account that is set as your SDK account property is called your SDK account. The gkeadm command-line tool uses your SDK account to download the admin workstation OVA and enable services in your Google Cloud project.

If you choose to have gkeadm automatically create service accounts for you, then gkeadm also uses your SDK account to create service accounts and keys, and to grant roles to service accounts. So it is important that you set your SDK account property before you run gkeadm to create an admin workstation.

The output shows the values of your SDK account property. For example:

[core]
account = my-name@google.com
disable_usage_reporting = False
Your active configuration is: [default]

Grant roles to your SDK account

Your SDK account must have the following IAM role on the parent Google Cloud project of your component access service account. This is so that gkeadm can enable services on the Google Cloud project.

  • serviceUsage.serviceUsageAdmin

If you choose to have gkeadm automatically create service accounts for you, then your SDK account must also have the following roles on the parent project of your component access service account. This is so that gkeadm can create service accounts and keys.

  • resourcemanager.projectIamAdmin
  • iam.serviceAccountCreator
  • iam.serviceAccountKeyAdmin

To grant roles on a Google Cloud project, you must have certain permissions on the Google Cloud project. For details, see Granting, changing, and revoking access to resources.

If you have the required permissions, you can grant the roles yourself. Otherwise, someone else in your organization must grant the roles for you.

To grant the required role to your SDK account:

Linux and macOS

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="user:ACCOUNT" \
  --role="roles/serviceusage.serviceUsageAdmin"

Windows

gcloud projects add-iam-policy-binding PROJECT_ID ^
  --member="user:ACCOUNT" ^
  --role="roles/serviceusage.serviceUsageAdmin"

Replace the following:

  • PROJECT_ID: the ID of the parent Google Cloud project of your component access service account

  • ACCOUNT: your SDK account

To grant additional roles in case you want gkeadm to automatically create service accounts:

Linux and macOS

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="user:ACCOUNT" \
  --role="roles/resourcemanager.projectIamAdmin"

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="user:ACCOUNT" \
  --role="roles/iam.serviceAccountCreator"

gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="user:ACCOUNT" \
  --role="roles/iam.serviceAccountKeyAdmin"

Windows

gcloud projects add-iam-policy-binding PROJECT_ID ^
  --member="user:ACCOUNT" ^
  --role="roles/resourcemanager.projectIamAdmin"

gcloud projects add-iam-policy-binding PROJECT_ID ^
  --member="user:ACCOUNT" ^
  --role="roles/iam.serviceAccountCreator"

gcloud projects add-iam-policy-binding PROJECT_ID ^
  --member="user:ACCOUNT" ^
  --role="roles/iam.serviceAccountKeyAdmin"

Replace the following:

  • PROJECT_ID: the ID of the parent project of your component access service account

  • ACCOUNT: your SDK account

Creating your admin workstation

Enter this command to create your admin workstation. If you want gkeadm to create the connect-register and logging-monitoring service accounts for you, include the --auto-create-service-accounts flag. If you want to create those service service accounts manually, omit the flag.

./gkeadm create admin-workstation [--auto-create-service-accounts]

The output gives detailed information about the creation of your admin workstation:

...
Getting ... service account...
...
********************************************************************
Admin workstation is ready to use.

Admin workstation information saved to /usr/local/google/home/me/my-admin-workstation
This file is required for future upgrades
SSH into the admin workstation with the following command:
ssh -i /usr/local/google/home/me/.ssh/gke-admin-workstation ubuntu@172.16.5.1
********************************************************************

Getting an SSH connection to your admin workstation

Near the end of the preceding output there is a command you can use to get an SSH connection to your admin workstation. Enter that command now. For example:

ssh -i /usr/local/google/home/me/.ssh/gke-admin-workstation ubuntu@172.16.5.1

List the files on your admin workstation:

ls -1

In the output, you can see two cluster configuration files, your CA certificate file, and the JSON key file for your component access service account. If gkeadm created service accounts for you, you can also see the JSON key files for those service accounts. For example:

admin-cluster.yaml
user-cluster.yaml
vcenter-ca-cert.pem
component-access-key.json

Verify that gkeadm activated your component access service account on your admin workstation:

gcloud config get-value account

Copying JSON key files to your admin workstation

Before you create a cluster, the JSON key files for your service accounts must be on your admin workstation in the home directory.

The key for your component access service account is already on your admin workstation.

If you included the --auto-create-service-accounts flag when you ran gkeadm create admin-workstation, then then the keys for the following service accounts are already on your admin workstation in the home directory. Otherwise you must manually copy the keys to the home directory of your admin workstation:

  • Connect-register service account
  • Logging-monitoring service account

If you created any of the following service accounts, you must manually copy the keys for those service accounts to the home directory of your admin workstation:

  • Usage metering service account
  • Audit logging service account
  • Binary authorization service account

Restoring the admin workstation from a backup file

When you upgrade an admin workstation, the gkeadm upgrade command saves a backup file. Later, if you no longer have an admin workstation, or if you have lost some of the files that were on your upgraded admin workstation, you can use this backup file to create an admin workstation that is restored to what was in place immediately after the upgrade.

To create an admin workstation from a backup file, run this command:

gkeadm create admin-workstation --restore-from-backup ADMIN_WORKSTATION_NAME-backup.tar.gz

Replace ADMIN_WORKSTATION_NAME with the name of the admin workstation.

User-managed

Choose a computer to serve as your admin workstation. You can use either Ubuntu or Red Hat Enterprise Linux (RHEL). These are the requirements:

  • Ubuntu 20.04 LTS or 22.04 LTS

    • 4 CPU cores
    • 8 GiB RAM
    • 100 GiB of storage

  • RHEL 8.6, 8.7, or 8.8

    • 4 CPU cores
    • 12 GB RAM
    • 256 GiB of storage

Access to Google Cloud

Your admin workstation must have access to Google Cloud to download and install tools, process authorization requests, create service accounts, and more.

For information on different ways to connect to Google Cloud, see Connect to Google.

Access to Google Cloud can be direct or via a proxy server. For information on firewall rules and configuring a proxy server, see Proxy and firewall rules.

Access to vCenter Server

To create and manage clusters from your admin workstation, you need access to your instance of vCenter Server. For information, see:

Set up an NTP server

When your cluster is configured to use an NTP server, you need to set up a time sync service on your admin workstation to ensure that timedatectl reports time that is in sync with the cluster. This is required to avoid large clock drift issues, which might cause certificate verification failures due to unmatched expiration dates.

Ubuntu

We recommend that you use the chrony time server.

To install chrony:

sudo apt-get update
sudo apt install chrony

This provides two binaries:

  • chronyd - the daemon to sync and serve via the Network Time Protocol
  • chronyc - the command-line interface for the chrony daemon

To configure chronyd:

Edit /etc/chrony/chrony.conf to add or remove server lines. Then restart chrony:

sudo systemctl restart chrony.service

RHEL

We recommend that you use the chrony time server.

For installation instructions, see How to configure chrony.

Passwordless sudo

If your security policy allows it, enable passwordless sudo for the current user. This allows gkectl to prepare a private registry if one is provided, setup proxy for docker if your network is behind a proxy server, and force delete the bootstrap cluster that is used for admin cluster lifecycle operations in case the deletion fails.

If you choose not to enable passwordless sudo, and you intend to use a private registry for your admin cluster, then do the following manual configuration before you create the admin cluster:

If you choose not to enable passwordless sudo, you might need to manually delete the kind cluster after you create an admin cluster. For more information, see Kind cluster does not get deleted in the troubleshooting documentation.

Install software

Ubuntu

Install the following software:

sudo apt-get update
sudo apt-get -y install kubectl

RHEL

Install the following software:

Docker 19.03 or later

  1. Remove any previous Docker version:

    sudo dnf remove docker \
  docker-client \
  docker-client-latest \
  docker-common \
  docker-latest \
  docker-latest-logrotate \
  docker-logrotate \
  docker-engine

  2. Remove podman-manpages:

    sudo dnf remove podman-manpages

  3. Install Docker 19.03+:

    sudo dnf install -y yum-utils

sudo yum-config-manager \
  --add-repo \
  https://download.docker.com/linux/centos/docker-ce.repo

sudo dnf install -y --allowerasing docker-ce docker-ce-cli containerd.io

sudo systemctl start docker

  4. Verify that you are now running version 19.03+:

    sudo docker version

  5. Compare your output with the following example to ensure the Client and Server versions are 19.03+:

    Client: Docker Engine - Community
Version:           19.03.13
...
Server: Docker Engine - Community
Engine:
Version:          19.03.13

  6. Verify Docker runs:

    docker run hello-world

    You should see something similar to this:

    Hello from Docker!

    This message shows that your installation appears to be working correctly.

Google Cloud CLI latest version:

See Install the gcloud CLI.

kubectl

Run gcloud components install kubectl.

Log in

The Google Account that is set as your SDK account property is called your SDK account. The gkectl command-line tool uses your SDK account to download cluster node OVAs, pull container images and more. So it is important that you set your SDK account property before you run gkectl commands.

Log in with any Google Account. This sets your SDK account property:

gcloud auth login

Verify that your SDK account property is set correctly:

gcloud config list

The output shows the values of your SDK account property. For example:

[core]
account = my-name@google.com
disable_usage_reporting = False
Your active configuration is: [default]

Download gkectl and the bundle

Navigate to a directory where you want to install gkectl.

Download gkectl:

gcloud storage cp gs://gke-on-prem-release/gkectl/VERSION/gkectl ./
chmod +x gkectl

Replace VERSION with the version of Google Distributed Cloud. For example: 1.16.0-gke.1.

Download the Google Distributed Cloud bundle. Make sure the version matches the one you used to download gkectl:

gcloud storage cp gs://gke-on-prem-release/gke-onprem-bundle/VERSION/gke-onprem-vsphere-VERSION.tgz ./

Service accounts and keys

Make sure you have created the following required service accounts and JSON key files:

Also Make sure you have created any optional service service accounts and JSON key files that you need:

Place all of your JSON key files in the home directory of your admin workstation.

Troubleshooting

The following section re-enables SSH access to the admin workstation in the event the SSH key is lost or corrupted.

SSH key recovery

Use a temporary VM to perform the following steps:

  1. To generate a new set of SSH keys, follow the instructions in Create SSH keys in the Compute Engine documentation.

  2. Ensure that the temporary VM and the admin workstation are in a Powered Off state.

  3. Within vSphere, attach the boot disk of the admin workstation to the temporary VM.

    The boot disk has the label Hard disk 1.

  4. Mount the boot disk inside the VM by running the following commands:

    sudo mkdir -p /mnt/boot-disk
sudo mount DISK_ID /mnt/boot-disk

    Replace DISK_ID with your boot disk identifier, which should have a format similar to dev/sdc1.

  5. Edit the authorized_keys file on the boot disk to append the contents of the public key file generated in the first step:

    vi /mnt/boot-disk/.ssh/authorized_keys

  6. Shut down the temporary VM.

  7. Power on the admin workstation.

  8. Use the newly generated private key to access the admin workstation.

    ssh -i ~/.ssh/new-admin-ws.key ubuntu@"${ADMIN_WS_IP}"

Use the newly generated private key to continue accessing your admin workstation.

What's next

Create an admin cluster