Install in-cluster Cloud Service Mesh on GKE
This page explains how to install unmanaged, in-cluster Cloud Service Mesh on GKE:
- Run
asmcli
to do a new installation of Cloud Service Mesh 1.23.4-asm.1. - Optionally, deploy an ingress gateway.
- Deploy or redeploy your workloads to inject sidecar proxies.
Note that for Kubernetes workloads on Google Cloud, we recommend provisioning a managed control plane
For instructions to prepare an offline installation of Cloud Service Mesh,
see
Prepare an offline installation of Cloud Service Mesh
You will need to specify the --offline
and --output_dir
options when running
asmcli install
.
Limitations
Note the following limitations:
All Cloud Service Mesh clusters for one mesh must be registered to the same fleet at all times to use Cloud Service Mesh. Other clusters in the project of a Cloud Service Mesh cluster must not be registered to a different fleet.
For GKE Autopilot clusters, only managed Cloud Service Mesh is supported. Cloud Service Mesh support for GKE Autopilot is only available with versions 1.21.3+.
The
asmcli
tool must have access to the Google Kubernetes Engine (GKE) endpoint. You can configure access through a "jump" server, such as a Compute Engine VM within the Virtual Private Cloud (VPC) giving specific access.
Before you begin
Before you begin, make sure that you:
- Review the prerequisites.
- Review the information in Plan the install.
- Install the required tools.
- Download
asmcli
. - Grant cluster admin permissions.
- Validate project and cluster.
Roles required to install in-cluster Cloud Service Mesh
The following table describes the roles that are required to install in-cluster Cloud Service Mesh.
Role name | Role ID | Grant location | Description |
---|---|---|---|
GKE Hub Admin | roles/gkehub.admin | Fleet project | Full access to GKE Hubs and related resources. |
Kubernetes Engine Admin | roles/container.admin | Cluster project. Note that this role must be granted in both Fleet and cluster project for cross-project bindings. | Provides access to full management of Container Clusters and their Kubernetes API objects. |
Mesh Config Admin | roles/meshconfig.admin | Fleet and cluster project | Provides permissions required to initialize managed components of Cloud Service Mesh, such as managed control plane and backend permission that allows workloads to talk to Stackdriver without each being individually authorized (for both managed and in-cluster control planes). |
Project IAM Admin | roles/resourcemanager.projectIamAdmin | Cluster project | Provides permissions to administer IAM policies on projects. |
Service Account Admin | roles/iam.serviceAccountAdmin | Fleet project | Authenticate as a service account. |
Service Management Admin | roles/servicemanagement.admin | Fleet project | Full control of Google Service Management resources. |
Service Usage Admin | roles/serviceusage.serviceUsageAdmin | Fleet project | Ability to enable, disable, and inspect service states, inspect operations, and consume quota and billing for a consumer project.(Note 1) |
CA Service Admin Beta | roles/privateca.admin | Fleet project | Full access to all CA Service resources. (Note 2) |
Notes:
- Service Usage Admin - This role is necessary
as a prerequisite to enable the
mesh.googleapis.com
API when initially provisioning managed Cloud Service Mesh. - CA Service Admin - This role is only required if you are integrating with CA Service.
Install Cloud Service Mesh
The following outlines how to install Cloud Service Mesh:
Run
asmcli install
to install the in-cluster control plane on a single cluster. See the following sections for command line examples. The examples contain both required arguments and optional arguments that you might find useful. We recommend that you always specify theoutput_dir
argument so that you can locate sample gateways and tools such asistioctl
. See the navigation bar on the right for a list of the examples.Private GKE clusters need an additional firewall configuration step to allow traffic to
istiod
.Optionally, install an ingress gateway. By default,
asmcli
doesn't install theistio-ingressgateway
. We recommend that you deploy and manage the control plane and gateways separately. If you need the defaultistio-ingressgateway
installed with the in-cluster control plane, include the--option legacy-default-ingressgateway
argument.To complete setting up Cloud Service Mesh, you need to enable automatic sidecar injection and deploy or redeploy workloads.
If you are installing Cloud Service Mesh on more than one cluster, run
asmcli install
on each cluster. When you runasmcli install
, be sure to use the sameFLEET_PROJECT_ID
for each cluster. After Cloud Service Mesh is installed, see the instructions to set up a multi-cluster mesh on GKE.If your clusters are on different networks (as they are in island mode then you should pass a unique network name to
asmcli
using the--network_id
flag.
Install default features and Mesh CA
This section shows how to run asmcli
to install Cloud Service Mesh with the
default supported features for your platform and enable
Cloud Service Mesh certificate authority as the certificate authority.
Run the following command to install the control plane with default features and Cloud Service Mesh certificate authority. Enter your values in the provided placeholders.
./asmcli install \
--project_id PROJECT_ID \
--cluster_name CLUSTER_NAME \
--cluster_location CLUSTER_LOCATION \
--fleet_id FLEET_PROJECT_ID \
--output_dir DIR_PATH \
--enable_all \
--ca mesh_ca
--project_id
,--cluster_name
, and--cluster_location
Specify the project ID that the cluster is in, the cluster name, and either the cluster zone or region.--fleet_id
The project ID of the fleet host project. If you don't include this option,asmcli
uses the project that the cluster was created in when registering the cluster.--output_dir
Include this option to specify a directory whereasmcli
downloads theanthos-service-mesh
package and extracts the installation file, which containsistioctl
, samples, and manifests. Otherwiseasmcli
downloads the files to atmp
directory. You can specify either a relative path or a full path. The environment variable$PWD
doesn't work here.-
--enable_all
Allows the script to:- Grant required IAM permissions.
- Enable the required Google APIs.
- Set a label on the cluster that identifies the mesh.
- Register the cluster to the fleet if it isn't already registered.
--ca mesh_ca
Use Cloud Service Mesh certificate authority as the certificate authority.asmcli
configures Cloud Service Mesh certificate authority to use fleet workload identity
Install default features and Certificate Authority (CA) Service
This section shows how to run asmcli
to install Cloud Service Mesh with the default
supported features for your platform and
enable CA Service as the certificate authority.
In addition to Cloud Service Mesh certificate authority, you can configure Cloud Service Mesh to use Certificate Authority Service. This guide provides you an opportunity to integrate with CA Service, which is recommended for the following use cases:
- If you need different certificate authorities to sign workload certificates on different clusters.
- If you need to back your signing keys in a managed HSM.
- If you are in a highly regulated industry and are subject to compliance.
- If you want to chain up your Cloud Service Mesh CA to a custom enterprise root certificate to sign workload certificates.
The cost of Cloud Service Mesh certificate authority is included in the Cloud Service Mesh pricing. The CA Service isn't included in the base Cloud Service Mesh price and is charged separately. Additionally, CA Service comes with an explicit SLA, but the Cloud Service Mesh certificate authority does not.
Configure CA Service
- Create the CA pool
in the tier
DevOps
and in the same region as the cluster that it serves to avoid excessive latency issues or potential cross-region outages. For more information, see Workload-optimized tiers. - Create the CA to have at least one active certificate authority in the CA pool in the same project as the GKE cluster. Use subordinate CA's to sign Cloud Service Mesh workload certificates. Note down the CA pool corresponding to the subordinate CA.
If it is meant to only service certificates for Cloud Service Mesh workloads, set up the following issuance policy for the CA pool:
policy.yaml
baselineValues: keyUsage: baseKeyUsage: digitalSignature: true keyEncipherment: true extendedKeyUsage: serverAuth: true clientAuth: true caOptions: isCa: false identityConstraints: allowSubjectPassthrough: false allowSubjectAltNamesPassthrough: true celExpression: expression: subject_alt_names.all(san, san.type == URI && san.value.startsWith("spiffe://PROJECT_ID.svc.id.goog/ns/") )
To update the CA pool's issuance policy, use the following command:
gcloud privateca pools update CA_POOL --location ca_region --issuance-policy policy.yaml
For information on setting a policy on a pool, see Using a certificate issuance policy.
If you are using a certificate template, then configure it now. For more information, follow the CA Service guide for workload identity certificates. Ensure the certificate template is created in the same region as the CA pool. If there are multiple regions for CA pools, then create a certificate template per region.
Configure Cloud Service Mesh to use CA Service
Run the following commands on Google Distributed Cloud (software only) for VMware or Google Distributed Cloud (software only) for bare metal to install the control plane with default features and Certificate Authority Service. Enter your values in the provided placeholders.
Set the current context to your user cluster:
kubectl config use-context CLUSTER_NAME
Run
asmcli install
:./asmcli install \ --kubeconfig KUBECONFIG_FILE \ --fleet_id FLEET_PROJECT_ID \ --output_dir DIR_PATH \ --enable_all \ --ca gcp_cas \ --platform multicloud \ --ca_pool projects/CA_POOL_PROJECT_ID/locations/ca_region/caPools/CA_POOL
--fleet_id
The project ID of the fleet host project.--kubeconfig
The full path to thekubeconfig
file. The environment variable$PWD
doesn't work here. Additionally, relativekubeconfig
file locations that use a `~` will not work.--output_dir
Include this option to specify a directory whereasmcli
downloads theanthos-service-mesh
package and extracts the installation file, which containsistioctl
, samples, and manifests. Otherwiseasmcli
downloads the files to atmp
directory. You can specify either a relative path or a full path. The environment variable$PWD
doesn't work here.--platform multicloud
Specifies that the platform is something other than Google Cloud, such as on-premises or multi-cloud.-
--enable_all
Allows the script to:- Grant required IAM permissions.
- Enable the required Google APIs.
- Set a label on the cluster that identifies the mesh.
- Register the cluster to the fleet if it isn't already registered.
--ca gcp_cas
Use Certificate Authority Service as the certificate authority. Changing certificate authorities during an upgrade causes downtime.asmcli
configures Certificate Authority Service to use fleet workload identity--ca_pool
The full identifier for the Certificate Authority Service CA Pool. If you are using a certificate template, then append the template id separated by:
. For example:--ca_pool projects/CA_POOL_PROJECT_ID/locations/ca_region/caPools/CA_POOL:projects/CA_POOL_PROJECT_ID/locations/ca_region/certificateTemplates/CERT_TEMPLATE_ID
To view SLOs and infrastructure metrics in the Cloud Service Mesh UI, you must also perform the first three steps in Enable application logging and monitoring. If logging and monitoring are not enabled and do not receive custom logs and metrics, the Cloud Service Mesh dashboard will not display SLOs, error logs, or CPU and memory metrics.
Install default features with Istio CA
This section explains how to:
- Generate certificates and keys for the Istio CA that Cloud Service Mesh uses to sign your workloads.
- Run
asmcli
to install Cloud Service Mesh with default features and enable Istio CA.
By default environments that install Cloud Service Mesh with Istio CA report metrics to Prometheus. If you want to use the Cloud Service Mesh dashboards, you must enable Stackdriver. For more information, see Install with optional features.
For the best security, we highly recommend maintaining an offline root CA and using the subordinate CAs to issue certificates for each cluster. For more information, see Plug in CA Certificates. In this configuration, all workloads in the service mesh use the same root certificate authority (CA). Each Cloud Service Mesh CA uses an intermediate CA signing key and certificate, signed by the root CA. When multiple CAs exist within a mesh, this establishes a hierarchy of trust among the CAs. You can repeat these steps to provision certificates and keys for any number of certificate authorities.
The Makefile to generate the certificates is located in the
istio-1.23.4-asm.1
subdirectory in the --output_dir
directory that you
specified in the asmcli validate
command. If you didn't run asmcli validate
,
or you don't have the downloaded directory locally, you can get the Makefile by
downloading the Cloud Service Mesh installation file
and extracting the contents.
Change to the
istio-1.23.4-asm.1
directory.Create a directory for the certificates and keys:
mkdir -p certs && \ pushd certs
Generate a root certificate and key:
make -f ../tools/certs/Makefile.selfsigned.mk root-ca
This generates these files:
- root-cert.pem: the root certificate
- root-key.pem: the root key
- root-ca.conf: the configuration for openssl to generate the root certificate
- root-cert.csr: the CSR for the root certificate
Generate an intermediate certificate and key:
make -f ../tools/certs/Makefile.selfsigned.mk cluster1-cacerts
This generates these files in a directory named
cluster1
:- ca-cert.pem: the intermediate certificates
- ca-key.pem: the intermediate key
- cert-chain.pem: the certificate chain which
istiod
uses - root-cert.pem: the root certificate
If you perform these steps using an offline computer, copy the generated directory to a computer with access to the clusters.
Return to the previous directory:
popd
Run the following command to install the control plane with default features and Istio CA. Enter your values in the provided placeholders.
./asmcli install \
--project_id PROJECT_ID \
--cluster_name CLUSTER_NAME \
--cluster_location CLUSTER_LOCATION \
--fleet_id FLEET_PROJECT_ID \
--output_dir DIR_PATH \
--enable_all \
--ca citadel \
--ca_cert CA_CERT_FILE_PATH \
--ca_key CA_KEY_FILE_PATH \
--root_cert ROOT_CERT_FILE_PATH \
--cert_chain CERT_CHAIN_FILE_PATH
--project_id
,--cluster_name
, and--cluster_location
Specify the project ID that the cluster is in, the cluster name, and either the cluster zone or region.--fleet_id
The project ID of the fleet host project. If you don't include this option,asmcli
uses the project that the cluster was created in when registering the cluster.--output_dir
Include this option to specify a directory whereasmcli
downloads theanthos-service-mesh
package and extracts the installation file, which containsistioctl
, samples, and manifests. Otherwiseasmcli
downloads the files to atmp
directory. You can specify either a relative path or a full path. The environment variable$PWD
doesn't work here.-
--enable_all
Allows the script to:- Grant required IAM permissions.
- Enable the required Google APIs.
- Set a label on the cluster that identifies the mesh.
- Register the cluster to the fleet if it isn't already registered.
-ca citadel
Use Istio CA as the certificate authority.--ca_cert
The intermediate certificate--ca_key
The key for the intermediate certificate--root_cert
The root certificate--cert_chain
The certificate chain
Install with Istio CA with Google Cloud Observability enabled
If you want to use Cloud Service Mesh dashboards, you must enable Stackdriver.
Run the following command to install the control plane with default features and Istio CA. Enter your values in the provided placeholders.
./asmcli install \
--project_id PROJECT_ID \
--cluster_name CLUSTER_NAME \
--cluster_location CLUSTER_LOCATION \
--fleet_id FLEET_PROJECT_ID \
--output_dir DIR_PATH \
--enable_all \
--ca citadel \
--ca_cert CA_CERT_FILE_PATH \
--ca_key CA_KEY_FILE_PATH \
--root_cert ROOT_CERT_FILE_PATH \
--cert_chain CERT_CHAIN_FILE_PATH
--project_id
,--cluster_name
, and--cluster_location
Specify the project ID that the cluster is in, the cluster name, and either the cluster zone or region.--fleet_id
The project ID of the fleet host project. If you don't include this option,asmcli
uses the project that the cluster was created in when registering the cluster.--output_dir
Include this option to specify a directory whereasmcli
downloads theanthos-service-mesh
package and extracts the installation file, which containsistioctl
, samples, and manifests. Otherwiseasmcli
downloads the files to atmp
directory. You can specify either a relative path or a full path. The environment variable$PWD
doesn't work here.-
--enable_all
Allows the script to:- Grant required IAM permissions.
- Enable the required Google APIs.
- Set a label on the cluster that identifies the mesh.
- Register the cluster to the fleet if it isn't already registered.
-ca citadel
Use Istio CA as the certificate authority.--ca_cert
The intermediate certificate--ca_key
The key for the intermediate certificate--root_cert
The root certificate--cert_chain
The certificate chain--option stackdriver
Enables Stackdriver option. Note that you can also enable both Stackdriver and Prometheus by using--option prometheus-and-stackdriver
.
Install with optional features
An overlay file is a YAML file containing an IstioOperator
custom resource
(CR) that you pass to asmcli
to configure the control plane. You can
override the default control plane configuration and
enable an optional feature by passing
the YAML file to asmcli
. You can layer on more overlays, and each overlay
file overrides the configuration on the previous layers. As a best practice,
we recommend that you save the overlay files in your version control system.
There are two options to enable optional features:
--option
and
--custom_overlay
.
Use --option
to if you don't need to change the overlay
file. With this method asmcli
fetches the file from the
GitHub repository
for you.
Use --custom_overlay
when you need to customize the overlay file.
For more information, see Enabling optional features on the in-cluster control plane.
Run the following command to install the control plane with an optional
feature. To add multiple files, specify --custom_overlay
and the filename,
for example: --custom_overlay overlay_file1.yaml
--custom_overlay overlay_file2.yaml --custom_overlay overlay_file3.yaml
Enter your values in the provided placeholders.
./asmcli install \
--project_id PROJECT_ID \
--cluster_name CLUSTER_NAME \
--cluster_location CLUSTER_LOCATION \
--fleet_id FLEET_PROJECT_ID \
--output_dir DIR_PATH \
--enable_all \
--ca mesh_ca \
--custom_overlay OVERLAY_FILE
--project_id
,--cluster_name
, and--cluster_location
Specify the project ID that the cluster is in, the cluster name, and either the cluster zone or region.--fleet_id
The project ID of the fleet host project. If you don't include this option,asmcli
uses the project that the cluster was created in when registering the cluster.--output_dir
Include this option to specify a directory whereasmcli
downloads theanthos-service-mesh
package and extracts the installation file, which containsistioctl
, samples, and manifests. Otherwiseasmcli
downloads the files to atmp
directory. You can specify either a relative path or a full path. The environment variable$PWD
doesn't work here.-
--enable_all
Allows the script to:- Grant required IAM permissions.
- Enable the required Google APIs.
- Set a label on the cluster that identifies the mesh.
- Register the cluster to the fleet if it isn't already registered.
--ca mesh_ca
Use Cloud Service Mesh certificate authority as the certificate authority. Note thatasmcli
configures Cloud Service Mesh certificate authority to use fleet workload identity--custom_overlay
Specify the name of the overlay file.
Install gateways
Cloud Service Mesh gives you the option to deploy and manage gateways as part of your service mesh. A gateway describes a load balancer operating at the edge of the mesh receiving incoming or outgoing HTTP/TCP connections. Gateways are Envoy proxies that provide you with fine-grained control over traffic entering and leaving the mesh.
Create a namespace for the ingress gateway if you don't already have one. Gateways are user workloads, and as a best practice, they shouldn't be deployed in the control plane namespace. Replace
GATEWAY_NAMESPACE
with the name of your namespace.kubectl create namespace GATEWAY_NAMESPACE
Expected output:
namespace/GATEWAY_NAMESPACE created
Enable auto-injection on the gateway. The steps required depend on whether you want to use default injection labels (for example,
istio-injection=enabled
) or the revision label on the gateway namespace. The default revision tag and revision label are used by the sidecar injector webhook to associate injected proxies with a particular control plane revision.Default injection labels
Apply the default injection labels to the namespace.
kubectl label namespace GATEWAY_NAMESPACE istio-injection=enabled istio.io/rev-
Revision label
Use the following command to locate the revision label on
istiod
:kubectl get deploy -n istio-system -l app=istiod -o \ "jsonpath={.items[*].metadata.labels['istio\.io/rev']}{'\n'}"
The command outputs the revision label that corresponds to the Cloud Service Mesh version, for example:
asm-1234-1
Apply the revision label to the namespace. In the following command,
REVISION
is the value of theistiod
revision label that you noted in the previous step.kubectl label namespace GATEWAY_NAMESPACE \ istio.io/rev=REVISION --overwrite
Expected output:
namespace/GATEWAY_NAMESPACE labeled
You can ignore the message
"istio.io/rev" not found
in the output. That means that the namespace didn't previously have theistio.io/rev
label, which you should expect in new installations of Cloud Service Mesh or new deployments. Because auto-injection fails if a namespace has both theistio.io/rev
and theistio-injection
label, allkubectl label
commands in the Cloud Service Mesh documentation explicitly specify both labels.If the gateway namespace is not labelled, the
istio-ingressgateway
pods will fail with anImagePullBackOff
error when the gateway attempts to pull and theauto
image. This image should be replaced by the webhook.Download the example ingress gateway .yaml configuration file from the
anthos-service-mesh-packages
repository.Apply the example ingress gateway .yaml configuration as is, or modify as needed.
kubectl apply -n GATEWAY_NAMESPACE \ -f CONFIG_PATH/istio-ingressgateway
Expected output:
deployment.apps/istio-ingressgateway created poddisruptionbudget.policy/istio-ingressgateway created horizontalpodautoscaler.autoscaling/istio-ingressgateway created role.rbac.authorization.k8s.io/istio-ingressgateway created rolebinding.rbac.authorization.k8s.io/istio-ingressgateway created service/istio-ingressgateway created serviceaccount/istio-ingressgateway created
Learn more about best practices for gateways.
Deploy and redeploy workloads
Cloud Service Mesh uses sidecar proxies to enhance network security, reliability, and observability. With Cloud Service Mesh, these functions are abstracted away from the application's primary container and implemented in a common out-of-process proxy delivered as a separate container in the same Pod.
Your installation isn't complete until you enable automatic sidecar proxy injection (auto-injection) and restart the Pods for any workloads that were running on your cluster before you installed Cloud Service Mesh.
To enable auto-injection, you label your namespaces with the
default injection labels
if the default tag is set up, or a revision label
that was set on istiod
when you installed Cloud Service Mesh. The default revision
tag and revision label are used by the sidecar injector webhook to associate
injected sidecars with an istiod
revision. After adding the label, any
existing Pods in the namespace must be restarted for sidecars to be injected.
Before you deploy new workloads in a new namespace, make sure to configure auto-injection so that Cloud Service Mesh can monitor and secure traffic.
The steps required to enable auto-injection depend on whether you want to use default injection labels or the revision label:
Default (Recommended)
If you used a default tag revision to enable auto-injection on the gateway, verify that the default tag exists in the directory that you specified in
--output_dir
and that it is pointing to the newly installed revision.DIR_PATH/istioctl tag list
Run the following command.
NAMESPACE
is the name of the namespace where you want to enable auto-injection.kubectl label namespace NAMESPACE istio-injection=enabled istio.io/rev-
The default injection labels inject the revision the default tag is pointing to.
Revision label
Use the following command to locate the revision label on
istiod
:kubectl -n istio-system get pods -l app=istiod --show-labels
The output looks similar to the following:
NAME READY STATUS RESTARTS AGE LABELS istiod-asm-1234-1-5788d57586-bljj4 1/1 Running 0 23h app=istiod,istio.io/rev=asm-1234-1,istio=istiod,pod-template-hash=5788d57586 istiod-asm-1234-1-5788d57586-vsklm 1/1 Running 1 23h app=istiod,istio.io/rev=asm-1234-1,istio=istiod,pod-template-hash=5788d57586
In the output, under the
LABELS
column, note the value of theistiod
revision label, which follows the prefixistio.io/rev=
. In this example, the value isasm-1234-1
.Apply the revision label and remove the
istio-injection
label if it exists. In the following command,NAMESPACE
is the name of the namespace where you want to enable auto-injection, andREVISION
is the revision label you noted in the previous step.kubectl label namespace NAMESPACE istio-injection- istio.io/rev=REVISION --overwrite
You can ignore the message
"istio-injection not found"
in the output. That means that the namespace didn't previously have theistio-injection
label, which you should expect in new installations of Cloud Service Mesh or new deployments. Because auto-injection behavior is undefined when a namespace has both theistio-injection
and the revision label, allkubectl label
commands in the Cloud Service Mesh documentation explicitly ensure that only one is set.
If workloads were running on your cluster before you installed Cloud Service Mesh, restart the Pods to trigger re-injection.
How you restart Pods depends on your application and the environment the cluster is in. For example, in your staging environment, you might simply delete all the Pods, which causes them to restart. But in your production environment, you might have a process that implements a blue-green deployment so that you can safely restart Pods to avoid traffic interruption.
You can use
kubectl
to perform a rolling restart:kubectl rollout restart deployment -n NAMESPACE
What's next?
If your mesh consists entirely of GKE clusters, see Set up a multi-cluster mesh on GKE.