Using the Connect gateway

This page tells you how to use the Connect gateway to connect to a registered cluster. Before reading this page, ensure that you're familiar with the concepts in our overview. The guide assumes that your project administrator has already set up the gateway, and given you the necessary roles and permissions.

Before you begin

  • Ensure that you have the following command line tools installed:

    If you are using Cloud Shell as your shell environment for interacting with Google Cloud, these tools are installed for you.

  • Ensure that you have initialized the gcloud CLI for use with your project.

Log in to your Google Cloud account

You can use your own Google Cloud account or a Google Cloud service account to interact with connected clusters via the gateway API.

Follow the instructions in Authorizing Google Cloud CLI tools to log in to your user account. The Connect gateway supports service account impersonation, so even if you're logged in to your own user account you can use a service account to interact with clusters, as you'll see in the following sections.

Select a registered cluster

If you don't know the name of the cluster you want to access, you can see all your current fleet's registered clusters by running the following command:

gcloud container fleet memberships list

This lists all your fleet's clusters, including their membership names and external IDs. Each cluster in a fleet has a unique membership name. For GKE clusters, the membership name generally matches the name you gave it when you created the cluster, unless the cluster's name was not unique within its project at registration.

Get the cluster's gateway kubeconfig

Use the following command to get the kubeconfig you need to interact with your specified cluster:

gcloud container fleet memberships get-credentials MEMBERSHIP_NAME

Replace MEMBERSHIP_NAME with your cluster's fleet membership name.

This command returns a special Connect gateway-specific kubeconfig that lets you connect to the cluster through the Connect gateway.

If you want to use a service account rather than your own Google Cloud account, use gcloud config to set auth/impersonate_service_account to the service account email address.

To fetch the cluster credential used to interact with the Connect gateway using a service account, run the following commands: Note the following:

  • Google Distributed Cloud (software only) clusters on bare metal and VMware: The membership name is the same as the cluster name.

  • GKE on AWS: Use gcloud container aws clusters get-credentials.

  • GKE on Azure: Use gcloud container azure clusters get-credentials.

You can find out more about letting users impersonate a service account in Manage access to service accounts.

gcloud config set auth/impersonate_service_account SA_EMAIL_ADDRESS
gcloud container fleet memberships get-credentials MEMBERSHIP_NAME

Replace SA_EMAIL_ADDRESS with the email address of the service account. You can find out more about letting users impersonate a service account in Manage access to service accounts.

Run commands against the cluster

Once you have the necessary credentials, you can run commands using kubectl or a go-client as you normally would for any Kubernetes cluster. Your output should look something like the following:

# Get namespaces in the Cluster.
kubectl get namespaces
NAME              STATUS   AGE
default           Active   59d
gke-connect       Active   4d

kubectl exec/cp/attach/port-forward commands

The following kubectl commands are streaming commands and have additional requirements:

  • attach
  • cp
  • exec
  • port-forward

Note the following requirements:

  • Clusters must be at version 1.30 or later for the attach, cp, and exec commands, and at version 1.31 or later for the port-forward command.

  • The kubectl client must be at version 1.31 or later.

    To check your client version, look at the output of the kubectl version command. To install a newer version of kubectl see Install tools.

Users and service accounts with the roles/gkehub.gatewayAdmin IAM role and the cluster-admin ClusterRole have the necessary permissions to run the attach, cp, exec, and port-forward commands. If users and service accounts have been granted a custom IAM role or custom RBAC role, you might need to grant additional permissions. See the following sections for additional information.

Grant an additional permission if needed

The gkehub.gateway.stream IAM permission is required to run the attach, cp, exec, and port-forward commands through the Connect gateway. This permission is included in roles/gkehub.gatewayAdmin.

For users who aren't project owners, or for users or service accounts who haven't been granted roles/gkehub.gatewayAdmin in the project, you must either grant them roles/gkehub.gatewayAdmin or create a custom role that includes the other required roles and the gkehub.gateway.stream permission. For information on how to create a custom role, see Create and manage custom roles in the IAM documentation.

Create and apply additional RBAC policies if needed

Users and service accounts with the cluster-admin ClusterRole have the necessary permissions to run the attach, cp, exec, and port-forward commands.

At a minimum, the following rules are needed to run the commands:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: stream-role
  namespace: NAMESPACE  # Specify the namespace
rules:
- apiGroups: ["*"]
  resources: ["pods/exec", "pods/attach", "pods/portforward"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: stream-rolebinding
  namespace: NAMESPACE  # Specify the namespace
roleRef:
   apiGroup: "rbac.authorization.k8s.io"
   kind: Role
   name: stream-role
subjects:
- kind: User
  name: EMAIL # Specify the user that should have stream access
  namespace: NAMESPACE  # Specify the namespace

Troubleshooting

If you're having problems connecting to a cluster through the gateway, you or your administrator can check for the following common issues.

  • The server doesn't have a resource type: You may see this error message when the command kubectl get ns fails. There are multiple potential reasons for this error. Run your kubectl commands in verbose mode to see more details, for example kubectl get ns -v 10.
  • Cannot find active connections for cluster(project: 12345, membership: my-cluster): You may see this error when Connect Agent loses connectivity or isn't installed properly (clusters outside Google Cloud only). To resolve this issue, you need to verify if gke-connect namespace exists on the cluster. If the gke-connect namespace exists on the cluster, see the Connect Troubleshooting page to fix the connectivity issues.
  • The requested URL was not found on this server: You may see this error when the kubeconfig contains an incorrect server address. Make sure the Google Cloud CLI version you're using is the latest version and retry to generate the gateway kubeconfig. Don't manually edit the kubeconfig file, which will cause unexpected errors.
  • The user identity does not have sufficient permissions to use the gateway API: You need the roles/gkehub.gatewayAdmin roles/gkehub.gatewayReader or roles/gkehub.gatewayEditor role to use the API. See Grant IAM roles to users in the gateway setup guide for more details.
  • The Connect agent is not authorized to send the user’s requests: The Connect agent needs to be allowed to forward requests on your behalf, which is specified using an impersonation policy on the cluster. See Configure RBAC authorization in the gateway setup guide for an example of adding a user to the gateway-impersonate role.
  • The user identity does not have sufficient RBAC permissions to perform the operation: You must have appropriate permissions on the cluster to run your chosen operations. See Configure RBAC authorization in the gateway setup guide for an example of adding a user to the appropriate ClusterRole.
  • The user identity does not have sufficient permissions to perform the operation when using Google Groups or third-party support: See Collecting GKE Identity Service logs for instructions on how to inspect logs related to identity information.
  • The Connect agent is unhealthy: See the Connect Troubleshooting page to make sure your cluster is connected.
  • executable gke-gcloud-auth-plugin not found or no Auth Provider found for name gcp: kubectl versions 1.26 and later may display this error due to changes to kubectl authentication starting with GKE v1.26. Install gke-gcloud-auth-plugin and re-run gcloud container fleet memberships get-credentials MEMBERSHIP_NAME with the latest version of Google Cloud CLI.
  • Connections to the gateway fail with older versions of the Google Cloud CLI: For GKE clusters, the Connect agent is no longer required for the gateway to function so it does not get installed by default during membership registration. Earlier versions of the Google Cloud CLI (399.0.0 and below) assume the existence of the Connect agent on the cluster. Attempting to use the gateway with these earlier versions may fail on clusters registered with a newer version of the Google Cloud CLI. To resolve this, either upgrade your Google Cloud CLI client to a newer version or re-run the membership registration command with the --install-connect-agent flag.

kubectl exec/cp/attach/port-forward troubleshooting

The error returned from running the command is often a generic 400 Bad Request error which is not clear enough to debug the issue. To return more detailed error messages, use the kubectl client version 1.32 or later to run the command with a verbosity of level 4 or higher, for example: kubectl exec -v 4 ....

In the returned logs, search for the log that contains the following responses:

  • For the kubectl exec/cp/attach command: RemoteCommand fallback:
  • For the kubectl port-forward command: fallback to secondary dialer from primary dialer err:

For troubleshooting some of the common error messages that you might receive from the kubectl exec -v 4 ... command, see the following section.

Missing IAM permissions

If the error message contains generic::permission_denied: Permission'gkehub.gateway.stream' denied on resource, this could indicate that you haven't been granted the required IAM permissions to run the command. This feature requires users to have the gkehub.gateway.stream IAM permission, which is included by default in the roles/gkehub.gatewayAdmin role. See the IAM permission section for instructions.

Missing required RBAC permissions

If the error message contains ...generic::failed_precondition: failed to connect to the cluster's API Server with response (status=403 Forbidden..., it indicates that you're missing RBAC permissions. You need a set of RBAC permissions in the cluster to run these kubectl commands. For more information about setting up the required RBAC permissions, see Create and apply additional RBAC policies if needed.

Error message generic::resource_exhausted: Gateway's active_streams quota exhausted

There is a quota limit of 10 active streams per fleet host project. This is defined under the connectgateway.googleapis.com/active_streams quota. See View and manage quotas for instructions on managing your quotas.

Error message generic::failed_precondition: error encountered within the cluster

If you get the error generic::failed_precondition: error encountered within the cluster, check the Connect Agent logs in the cluster to identify the underlying cause:

kubectl logs -n gke-connect -l app=gke-connect-agent --tail -1

The log to look for in the Connect Agent is failed to create the websocket connection....

Error message generic::failed_precondition: connection to Agent failed/terminated

If you encounter this error immediately when running the command, there is an issue with the cluster's connection to Google. See the general troubleshooting guide for more information.

If you encounter this error after the session is active for about 20 to 30 minutes, it is an expected limitation due to security reasons. The connection needs to be re-established.

What's next?