Authenticating with OIDC and AD FS

This page shows how to use OpenID Connect (OIDC) with Active Directory Federation Services (AD FS) to configure authentication for GKE on-prem user clusters. Note that AD FS only supports OIDC in versions 2016 and above.

For an overview of the authentication flow, see Authentication. For information about using OpenID providers other than AD FS, see Authenticating with OpenID Connect.

Overview

GKE on-prem supports OpenID Connect (OIDC) as one of the authentication mechanisms for interacting with a user cluster's Kubernetes API server. With OIDC, you can manage access to Kubernetes clusters by using the standard procedures in your organization for creating, enabling, and disabling employee accounts.

There are two ways an employee can use the OIDC authentication flow:

  • An employee can use kubectl to initiate an OIDC flow. To make this flow automatic, GKE on-prem provides the Anthos Plugin for Kubectl, a kubectl plugin.
  • An employee can use Google Cloud console to initiate an OIDC authentication flow.

In this exercise, you configure both options: kubectl and Google Cloud console. You use a set of AD FS management wizards to configure your AD FS server, and your AD employee database.

Before you begin

This topic assumes you are familiar with OAuth 2.0 and OpenID Connect. This topic assumes you are familiar with OpenID scopes and claims.

This topic applies to enterprises that have the following infrastructure:

  • The enterprise uses Active Directory (AD) for its employee database.
  • The enterprise runs an Active Directory Federation Services (AD FS) server.
  • The AD FS server acts as an OpenID provider.

Personas

This topic refers to three personas:

  • Organization administrator: This person chooses an OpenID provider and registers client applications with the provider.

  • Cluster administrator: This person creates one or more user clusters and creates authentication configuration files for developers who use the clusters.

  • Developer: This person runs workloads on one or more clusters and uses OIDC to authenticate.

Creating a redirect URL for the Anthos Plugin for Kubectl

This section is for organization administrators.

As part of establishing a relationship with your AD FS server, you must specify a redirect URL that the AD FS server can use to return ID tokens to Anthos Plugin for Kubectl. The Anthos Plugin for Kubectl runs on each developer's local machine and listens on a port of your choice. Choose a port number greater than 1024 that is suitable for this purpose. Then, the redirect URL is:

http://localhost:[PORT]/callback

where [PORT] is your port number.

When you configure your AD FS server, specify http://localhost:[PORT]/callback as one of your redirect URLs.

Configuring a redirect URL for Google Cloud console

This section is for organization administrators.

In addition to having a redirect URL for the Anthos Plugin for Kubectl, you need a redirect URL for Google Cloud console. The redirect URL for Google Cloud console is:

https://console.cloud.google.com/kubernetes/oidc

When you configure your AD FS server, specify https://console.cloud.google.com/kubernetes/oidc as one of your redirect URLs.

Configuring AD FS

The following sections explain how to configure AD FS for GKE on-prem.

Setting the redirect URLs

This section is for organization administrators.

  1. Open the AD FS management pane.

  2. Select Application Groups > Actions > Add an Application Group.

  3. Select Server Application. Enter a name and description of your choice. Click Next.

  4. Enter your two redirect URLs. You are given a client ID. This is how the AD FS server identifies the Anthos Plugin for Kubectl and Google Cloud console. Save the client ID for later.

  5. Select Generate a shared secret. The Anthos Plugin for Kubectl and Google Cloud console use this secret to authenticate to the AD FS server. Save the secret for later.

Configuring security groups (optional)

This section is for organization administrators.

  1. In AD FS management, select Relying party trusts > Add a new relying party trust.

  2. Select Claims aware, and click Start.

  3. Select Enter data about relying party manually.

  4. Enter a display name.

  5. Skip the next two steps.

  6. Enter a Relying party trust identifier. Suggestion: token-groups-claim.

  7. For Access control policy, select Permit everyone. This means that all employees share their security group information with the Anthos Plugin for Kubectl and Google Cloud console.

  8. Click Finish.

Mapping LDAP attributes to claim names

This section is for organization administrators.

  1. In AD FS management, select Relying party trusts > Edit claim issuance policy.

  2. Select Send LDAP Attributes as Claims, and click Next.

  3. For Claim rule name, enter groups.

  4. For Attribute store, select Active Directory.

  5. In the table, for LDAP Attribute, select:

    • AD FS version 5.0 and later: Token-Groups Qualified by Domain name
    • AD FS versions before 5.0: Token Groups - Qualified Names

  6. For Outgoing Claim Type, select:

    • AD FS version 5.0 and later: Group
    • AD FS versions before 5.0: groups

  7. Click Finish, and click Apply.

Registering the Anthos Plugin for Kubectl and Google Cloud console with AD FS

This section is for organization administrators.

Open a PowerShell window in Administrator mode, and enter this command:

Grant-AD FSApplicationPermission `
    -ClientRoleIdentifier "[CLIENT_ID]" `
    -ServerRoleIdentifier [SERVER_ROLE_IDENTIFIER] `
    -ScopeName "allatclaims", "openid"

where:

  • [CLIENT_ID] is the client ID that you obtained previously.

  • [SERVER_ROLE_IDENTIFIER] is the claim identifier you entered previously. Recall that the suggested identifier was token-groups-claim.

Populating the oidc specification in GKE on-prem configuration file

This section is for cluster administrators.

Before you create a user cluster, you generate a GKE on-prem configuration file using gkectl create-config. The configuration includes the following oidc specification. You populate oidc with values specific to your provider:

oidc:
  issuerurl:
  kubectlredirecturl:
  clientid:
  clientsecret:
  username:
  usernameprefix:
  group:
  groupprefix:
  scopes:
  extraparams:
  usehttpproxy:
  capath:
  • issuerurl: Required. URL of your OpenID provider, such as https://example.com/adfs. Client applications, like the Anthos Plugin for Kubectl and Google Cloud console, send authorization requests to this URL. The Kubernetes API server uses this URL to discover public keys for verifying tokens. Must use HTTPS.
  • kubectlredirecturl: Required. The redirect URL configured previously for the Anthos Plugin for Kubectl.
  • clientid: Required. ID for the client application that makes authentication requests to the OpenID provider. Both the Anthos Plugin for Kubectl and Google Cloud console use this ID.
  • clientsecret: Optional. Secret for the client application. Both the Anthos Plugin for Kubectl and Google Cloud console use this secret.
  • username: Optional. JWT claim to use as the username. The default is sub, which is expected to be a unique identifier of the end user. You can choose other claims, such as email or name, depending on the OpenID provider. However, claims other than email are prefixed with the issuer URL to prevent naming clashes with other plugins.
  • usernameprefix: Optional. Prefix prepended to username claims to prevent clashes with existing names. If you do not provide this field, and username is a value other than email, the prefix defaults to issuerurl#. You can use the value - to disable all prefixing.
  • group: Optional. JWT claim that the provider will use to return your security groups.
  • groupprefix: Optional. Prefix prepended to group claims to prevent clashes with existing names. For example, given a group foobar and a prefix gid-, gid-foobar. By default, this value is empty, and there is no prefix.
  • scopes: Optional. Additional scopes to send to the OpenID provider as a comma-delimited list.
    • For authentication with Microsoft Azure, pass in offline_access.

  • extraparams: Optional. Additional key-value parameters to send to the OpenID provider as a comma-delimited list.
  • usehttpproxy: Optional. Specifies whether to deploy a reverse proxy in the cluster to allow Google Cloud console access to the on-premises OIDC provider for authenticating users. Value must be a string: "true" or "false". If your identity provider is not reachable over the public internet, and you wish to authenticate using Google Cloud console, then this field must be set to "true".

  • capath: Optional. Path to the certificate for the certificate authority (CA) that issued your identity provider's web certificate. This value might not be necessary. For example, if your identity provider's certificate was issued by a well-known public CA, then you would not need to provide a value here. However, if usehttpproxy is "true", then this value must be provided, even for a well-known public CA.

Example: Authorizing users and groups

Many providers encode user-identifying properties, such as email and user IDs, in a token. However, these properties have implicit risks for authentication policies:

  • User IDs can make policies difficult to read and audit.
  • Emails can create both an availability risk (if a user changes their primary email) and potentially a security risk (if an email can be re-assigned).

Therefore, it's a best practice to use group policies, as a group ID can be both persistent and easier to audit.

Suppose your provider creates identity tokens that include the following fields:

{
  'iss': 'https://server.example.com'
  'sub': 'u98523-4509823'
  'groupList': ['developers@example.corp', 'us-east1-cluster-admins@example.corp']
  ...
}

Given this token format, you'd populate your configuration file's oidc specification like so:

issueruri: 'https://server.example.com'
username: 'sub'
usernameprefix: 'uid-'
group: 'groupList'
groupprefix: 'gid-'
extraparams: 'resource=token-groups-claim'
...

After you've created the user cluster, you could then use Kubernetes role-based access control (RBAC) to grant privileged access to the authenticated users. For example, you could create a ClusterRole that grants its users read-only access to the cluster's Secrets, and create a ClusterRoleBinding resource to bind the role to the authenticated group:

ClusterRole

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: secret-reader
rules:
- apiGroups: [""]
  # The resource type for which access is granted
  resources: ["secrets"]
  # The permissions granted by the ClusterRole
  verbs: ["get", "watch", "list"]

ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: read-secrets-admins
subjects:
  # Allows anyone in the "us-east1-cluster-admins" group to
  # read Secrets in any namespace within this cluster.
- kind: Group
  name: gid-us-east1-cluster-admins # Name is case sensitive
  apiGroup: rbac.authorization.k8s.io
  # Allows this specific user to read Secrets in any
  # namespace within this cluster
- kind: User
  name: uid-u98523-4509823
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io

Creating your first authentication configuration file

This section is for cluster administrators.

After you create a user cluster, create an authentication configuration file for your cluster:

gkectl create-login-config --kubeconfig [USER_CLUSTER_KUBECONFIG]

where [USER_CLUSTER_KUBECONFIG] is the path of your user cluster's kubeconfig file. When you created your user cluster, gkectl create cluster generated a kubeconfig file for the cluster.

The preceding command creates a file named kubectl-anthos-config.yaml in the current directory.

Adding additional clusters to your authentication configuration file

This section is for cluster administrators.

You can store the authentication configuration for several clusters in a single file. Then you can distribute that one file to the developers who should have access to all of those clusters.

Start with an existing authentication configuration file. Then use the following command to merge that file with the configuration for an additional cluster:

gkectl create-login-config --kubeconfig [USER_CLUSTER_KUBECONFIG] \
    --merge-from [IN_AUTH_CONFIG_FILE] --output [OUT_AUTH_CONFIG_FILE]

where

  • [USER_CLUSTER_KUBECONFIG] is the kubeconfig file of the additional cluster.

  • [IN_AUTH_CONFIG_FILE] is the path of the existing authentication configuration file.

  • [OUT_AUTH_CONFIG_FILE] is the path where you want to save the file that holds the merged configuration. This can be the same as [IN_AUTH_CONFIG_FILE].

Distributing the authentication configuration file

This section is for cluster administrators.

The authentication configuration file that you distribute to your developers must be namedkubectl-anthos-config.yaml. You can distribute the authentication configuration file in a number of ways:

  • Manually give the file to each developer.

  • Host the file at a URL, so developers can download it.

  • Push the file to each developer's machine.

Regardless of your distribution method, the authentication configuration file must be placed at this location on each developer's machine:

Linux

$HOME/.config/google/anthos/kubectl-anthos-config.yaml

macOS

$HOME/Library/Preferences/google/anthos/kubectl-anthos-config.yaml

Windows

%APPDATA%\google\anthos\kubectl-anthos-config.yaml

Placing the authentication configuration file

This section is for developers.

The authentication configuration file contains the details of all clusters you can authenticate to. Do not modify the contents of this file.

If your cluster administrator has provided you with an authentication configuration file, place the file in the correct directory. If your cluster administer has already placed the configuration on your machine, you can skip this section.

Copy your authentication configuration file to its default location:

Linux

mkdir -p  $HOME/.config/google/anthos/
cp [AUTH_CONFIG_FILE] $HOME/.config/google/anthos/kubectl-anthos-config.yaml

where [AUTH_CONFIG_FILE] is the name of your authentication configuration file.

macOS

mkdir -p  $HOME/Library/Preferences/google/anthos/
cp [AUTH_CONFIG_FILE] $HOME/Library/Preferences/google/anthos/kubectl-anthos-config.yaml

where [AUTH_CONFIG_FILE] is the name of your authentication configuration file.

Windows

md "%APPDATA%\google\anthos"
copy [AUTH_CONFIG_FILE] "%APPDATA%\google\anthos\kubectl-anthos-config.yaml"

where [AUTH_CONFIG_FILE] is the name of your authentication configuration file.

Getting the Anthos Plugin for Kubectl

This section is for cluster administrators.

The Anthos Plugin for Kubectl is included in your admin workstation at:

Linux

/usr/bin/kubectl_anthos/v1.0beta/linux_amd64/kubectl-anthos

macOS

/usr/bin/kubectl_anthos/v1.0beta/darwin_amd64/kubectl-anthos

Windows

/usr/bin/kubectl_anthos/v1.0beta/windows_amd64/kubectl-anthos

You can distribute the plugin to your developers, or you can have them download the plugin directly.

Downloading the Anthos Plugin for Kubectl

This section is for cluster administrators and developers.

Each developer needs to have the Anthos Plugin for Kubectl on their own machine. Developers can download the plugin individually, or the cluster administrator can download the plugin and then distribute it to the developers.

Note for developers: Ask your cluster administrator what version of the plugin you should use.

Download the Anthos Plugin for Kubectl:

Linux

gcloud storage cp gs://gke-on-prem-release/kubectl-anthos/v1.0beta/linux_amd64/kubectl-anthos ./
chmod +x kubectl-anthos

macOS

gcloud storage cp gs://gke-on-prem-release/kubectl-anthos/v1.0beta/darwin_amd64/kubectl-anthos ./
chmod +x kubectl-anthos

Windows

gcloud storage cp gs://gke-on-prem-release/kubectl-anthos/v1.0beta/windows_amd64/kubectl-anthos ./
rename kubectl-anthos kubectl-anthos.exe.

Installing the Anthos Plugin for Kubectl

This section is for developers.

Your cluster administrator might provide you with the Anthos Plugin for Kubectl. Or your cluster administrator might tell you to download the plugin yourself.

To install the Anthos Plugin for Kubectl, move the executable file to any location on your PATH. The executable file must be named kubectl-anthos. To learn more, see the Installing kubectl plugins.

Verify that the Anthos Plugin for Kubectl is installed:

kubectl plugin list
kubectl anthos version

Listing available clusters

This section is for developers.

If your authentication configuration file is at the default path, enter this command to list the clusters that you can authenticate to:

kubectl anthos listclusters

If your authentication configuration file is not at the default path, enter this command to list clusters:

kubectl anthos listclusters --loginconfig [AUTH_CONFIG_FILE]

where [AUTH_CONFIG_FILE] is the path to your authentication configuration file.

Using the Anthos Plugin for Kubectl to authenticate

This section is for developers.

Log in to a user cluster:

kubectl anthos login --cluster [CLUSTER_NAME] --user [USER_NAME] \
    --loginconfig [AUTH_CONFIG_FILE] --kubeconfig [USER_CLUSTER_KUBECONFIG]

where:

  • [CLUSTER_NAME] is the name of your user cluster. This name must be selected from the list provided by the kubectl anthos listclusters command.

  • [USER_NAME] an optional parameter that specifies the username for the credentials stored in the kubeconfig file. The default value is [CLUSTER_NAME]-anthos-default-user.

  • [AUTH_CONFIG_FILE] is the path of your authentication configuration file. If your authentication configuration file is in the default location, you can omit this parameter.

  • [USER_CLUSTER_KUBECONFIG] is the path of your user cluster's kubeconfig file. If a kubeconfig file does not exist, the command generates a new kubeconfig file from the authentication configuration file, and adds the cluster details and tokens to the kubeconfig file.

kubectl anthos login launches a browser where you can enter your credentials.

The kubeconfig file provided now contains an ID token that kubectl can use to authenticate to the Kubernetes API server on the user cluster.

The kubectl anthos login command has an optional --dry-run flag. If you include the --dry-run flag, the command prints the kubectl commands that would add tokens to the kubeconfig file, but does not actually save the tokens in the kubeconfig file.

Verify that authentication was successful by entering a kubectl command. For example:

kubectl get nodes --kubeconfig [USER_CLUSTER_KUBECONFIG]

Using OIDC with Google Cloud console

This section is for developers.

  1. Verify that your cluster is configured for OIDC.

  2. Verify that your cluster has been registered with Google Cloud, either automatically during cluster creation or manually.

  3. Visit the Kubernetes clusters page in Google Cloud console.

    Visit the Kubernetes clusters page

  4. In the list of clusters, locate your GKE On-Prem cluster, and click Login.

  5. Select Authenticate with the Identity Provider configured for the cluster, and click LOGIN.

    You will be redirected to your identity provider, where you might need to log in or consent to Google Cloud console accessing your account. Then you will be redirected back to the Kubernetes clusters page in Google Cloud console.

Summary

Your enterprise runs an AD FS server that acts as your OpenID provider. Your OpenID provider knows about your two client applications: the Anthos Plugin for Kubectl and Google Cloud console. Your OpenID provider knows that your client applications can request the openid and allatclaims scopes.

In AD FS version before 5.0, Token-Groups Qualified Names LDAP attribute in your AD database is mapped to the groups claim in your OpenID provider. In 5.0 and later, the attribute is Token-Groups Qualified by Domain name. The provider returns tokens that include the employee's ID, the issuer ID, the openid claim and groups claim. The groups (Group in 5.0) claim lists the security groups that an employee belongs to.

Custom configuration

By default, the Anthos Plugin for Kubectl expects the authentication configuration file to be at a certain location. If you do not want to put the authentication configuration file at the default location, you can manually pass the path of the authentication configuration file to the plugin commands by using the --login-config flag. For example, see Using the Anthos Plugin for Kubectl to authenticate.

Troubleshooting OIDC in GKE on-prem

Invalid configuration

If Google Cloud console cannot read the OIDC configuration from your cluster, the LOGIN button will be disabled.

Invalid provider configuration

If your identity provider configuration is invalid, you will see an error screen from your identity provider after you click LOGIN. Follow the provider-specific instructions to correctly configure the provider or your cluster.

Invalid permissions

If you complete the authentication flow, but still don't see the details of the cluster, make sure you granted the correct RBAC permissions to the account that you used with OIDC. Note that this might be a different account from the one you use to access Google Cloud console.

Error: missing 'RefreshToken' field in 'OAuth2Token' in credentials struct

You might get this error if the authorization server prompts for consent, but the required authentication parameter wasn't provided. Provide the prompt=consent parameter to GKE on-prem configuration file's oidc: extraparams field, and regenerate the client authentication file with the --extra-params prompt=consent flag.

What's next