Supported features using Istio APIs (managed control plane)

This page describes the supported features and limitations for Cloud Service Mesh using Cloud Service Mesh or istiod as the control plane and the differences between each implementation. Note that these are not options you can choose. The istiod implementation is only available for existing users. New installations use the Cloud Service Mesh implementation.

For the list of Cloud Service Mesh supported features for an in-cluster control plane, see Using Istio APIs (in-cluster istiod control plane). If you're not sure which Cloud Service Mesh control plane you're using, you can check your control plane implementation using the instructions in Identify control plane implementation

Limitations

The following limitations apply:

  • GKE clusters must be in one of the supported regions.
  • GKE version must be a supported version.
  • Only the platforms listed in Environments are supported.
  • Changing release channels is not supported.
  • Migrations from managed Cloud Service Mesh with asmcli to Cloud Service Mesh with fleet API are not supported. Similarly, provisioning managed Cloud Service Mesh with fleet API from --management manual to --management automatic is not supported.
  • Migrations and upgrades are supported only from in-cluster Cloud Service Mesh versions 1.9+ installed with Mesh CA. Installations with Istio CA (previously known as Citadel) must first migrate to Mesh CA.
  • Scale is limited to 1000 services and 5000 workloads per cluster.
  • Only multi-primary deployment option for multi-cluster is supported: primary-remote deployment option for multi-cluster is not.
  • istioctl ps is not supported. Instead you can use the gcloud beta container fleet mesh debug commands as described in Troubleshooting.

  • Unsupported APIs:

    • EnvoyFilter API

    • WasmPlugin API

    • IstioOperator API

    • Kubernetes Ingress API

    • Kubernetes Gateway API

  • You can use the managed control plane without an GKE Enterprise subscription, but certain UI elements and features in Google Cloud console are only available to GKE Enterprise subscribers. For information about what is available to subscribers and non-subscribers, see GKE Enterprise and Cloud Service Mesh UI differences.

  • During the provisioning process for a managed control plane, Istio CRDs corresponding to the selected channel are installed in the specified cluster. If there are existing Istio CRDs in the cluster, they will be overwritten

  • Managed Cloud Service Mesh only supports the default DNS domain .cluster.local.

  • As of November 14th, 2023, new installations of managed Cloud Service Mesh on the rapid release channel fetch JWKS only using Envoys. This is equivalent to the PILOT_JWT_ENABLE_REMOTE_JWKS=envoy Istio option. Compared to installations on the regular and stable release channels, or installations on the rapid release channel before November 14th, 2023, you might need extra ServiceEntry and DestinationRule configurations. For an example, see the requestauthn-with-se.yaml.tmpl.

Control plane differences

There are differences in supported features between the istiod and Cloud Service Mesh control plane implementations. To check which implementation you are using, see Identify control plane implementation.

  • – indicates the feature is available and enabled by default.
  • † - indicates that feature APIs may have differences between various platforms.
  • * – indicates the feature is supported for the platform and can be enabled, as described in Enable optional features or the feature guide linked in the feature table.
  • § – indicates that the feature is supported by allowlist. Contact Google Cloud Support to request access.
  • – indicates either the feature isn't available or it isn't supported.

The default and optional features are fully supported by Google Cloud Support. Features not explicitly listed in the tables receive best-effort support.

What determines control plane implementation

When you provision managed Cloud Service Mesh the first time in a fleet, we determine which control plane implementation to use. The same implementation is used for all clusters that provision managed Cloud Service Mesh in that fleet.

New fleets that onboard to managed Cloud Service Mesh receive the TRAFFIC_DIRECTOR control plane implementation, with certain exceptions:

  • If you are an existing managed Cloud Service Mesh user, you receive the ISTIOD control plane implementation when you onboard a new fleet in the same Google Cloud Organization to managed Cloud Service Mesh, until at least June 24, 2024. If you are one of these users, you can contact Support to fine-tune this behavior.
  • If any cluster in your fleet uses Certificate Authority Service when you provision managed Cloud Service Mesh, you receive the ISTIOD control plane implementation.
  • If any cluster in your fleet contains an in-cluster Cloud Service Mesh control plane when you provision managed Cloud Service Mesh, you will receive the ISTIOD control plane implementation.
  • If any cluster in your fleet uses GKE Sandbox, when you provision managed Cloud Service Mesh, you receive the ISTIOD control plane implementation.

Identify control plane implementation

Run the following command to identify your control plane implementation:

  gcloud container fleet mesh describe --project FLEET_PROJECT_ID

Note that if you are using Cloud Service Mesh with Google Cloud APIs (see About Cloud Service Mesh), then this command won't work.

The output is similar to the following:

  ...
  membershipSpecs:
    projects/746296320118/locations/us-central1/memberships/demo-cluster-1:
      mesh:
        management: MANAGEMENT_AUTOMATIC
  membershipStates:
    projects/746296320118/locations/us-central1/memberships/demo-cluster-1:
      servicemesh:
        controlPlaneManagement:
          details:
          - code: REVISION_READY
            details: 'Ready: asm-managed'
          state: ACTIVE
          implementation: TRAFFIC_DIRECTOR
  ...

The possible values implementation are:

  • TRAFFIC_DIRECTOR: The core infrastructure of Google Cloud serves as the Cloud Service Mesh control plane.
  • ISTIOD: managed instance of istiod serves as the Cloud Service Mesh control plane.
  • UPDATING: The cluster is being migrated between implementations. Soon you will have the TRAFFIC_DIRECTOR implementation.

Managed control plane supported features

Install, upgrade, and rollback

Feature Managed (TD) Managed (istiod)
Installation on GKE clusters using fleet feature API
Upgrades from ASM 1.9 versions that use Mesh CA
Direct (skip-level) upgrades from Cloud Service Mesh versions prior to 1.9 (see notes for indirect upgrades)
Direct (skip-level) upgrades from Istio OSS (see notes for indirect upgrades)
Direct (skip-level) upgrades from Istio-on-GKE add-on (see notes for indirect upgrades)
Enabling optional features

Environments

Feature Managed (TD) Managed (istiod)
GKE 1.25-1.27 in one of the supported regions
GKE 1.25-1.27 clusters with Autopilot
Environments outside of Google Cloud (GKE Enterprise on-premises, GKE Enterprise on other public clouds, Amazon EKS, Microsoft AKS, or other Kubernetes clusters)

Scale

Feature Managed (TD) Managed (istiod)
1000 services and 5000 workloads per cluster
50 ServicePorts per mesh and 36 pods per ServicePort

Platform environment

Feature Managed (TD) Managed (istiod)
Single network
Multi-network
Single-project
Multi-project with Shared VPC

Multi-cluster deployment

Feature Managed (TD) Managed (istiod)
Multi-primary
Primary-remote
Multi-cluster endpoint discovery with declarative API
Multi-cluster endpoint discovery with remote secrets

Notes on terminology

  • A multi-primary configuration means that the configuration must be replicated in all clusters.
  • A primary-remote configuration means that a single cluster contains the configuration and is considered the source of truth.
  • Cloud Service Mesh uses a simplified definition of network based on general connectivity. Workload instances are on the same network if they are able to communicate directly, without a gateway.

Base Images

Feature Managed (TD) Managed (istiod)
Distroless proxy image †

† Cloud Service Mesh with a managed (TD) control plane only supports the distroless image type. You cannot change it.

Note that distroless images have minimal binaries, so you cannot exec the usual commands like bash or curl because they are not present in the distroless image. However, you can use ephemeral containers to attach to a running workload Pod to be able to inspect it and run custom commands. For example, see Collecting Cloud Service Mesh logs.

Security

VPC Service Controls

Feature Managed (TD) Managed (istiod)
VPC Service Controls preview
VPC Service Controls GA

Certificate distribution and rotation mechanisms

Feature Managed (TD) Managed (istiod)
Workload certificate management
External certificate management on ingress and egress gateways.

Certificate authority (CA) support

Feature Managed (TD) Managed (istiod)
Cloud Service Mesh certificate authority
Certificate Authority Service
Istio CA
Integration with custom CAs

Security features

In addition to supporting Istio security features, Cloud Service Mesh provides even more capabilities to help you secure your applications.

Feature Managed (TD) Managed (istiod)
IAP integration
End-user authentication
Dry-run mode
Denial logging
Audit policies (not supported)

Authorization policy

Feature Managed (TD) Managed (istiod)
Authorization v1beta1 policy †
CUSTOM Authorization Policy §

† The Cloud Service Mesh control plane implementation does not support the rules.from.source.RemoteIp and rules.from.source.NotRemoteIp fields.

Peer authentication

Feature Managed (TD) Managed (istiod)
Auto-mTLS
mTLS PERMISSIVE mode
mTLS STRICT mode * *
mTLS DISABLE mode

Request authentication

Feature Managed (TD) Managed (istiod)
JWT authentication(Note 1)
JWT Claim Based Routing
JWT Copy Claim to Headers

Notes:

  1. Third-party JWT is enabled by default.
  2. Add the full fqdn/hostname in JWKSURI while defining RequestAuthentication API.
  3. Managed control plane enforces Envoy to fetch JWKS when specifying JWKS URI.

Telemetry

Metrics

Feature Managed (TD) Managed (istiod)
Cloud Monitoring (HTTP in-proxy metrics)
Cloud Monitoring (TCP in-proxy metrics)
Prometheus metrics export to Grafana (Envoy metrics only) * *
Prometheus metrics export to Kiali
Google Cloud Managed Service for Prometheus, not including the Cloud Service Mesh dashboard * *
Istio Telemetry API †
Custom adapters/backends, in or out of process
Arbitrary telemetry and logging backends

† The Cloud Service Mesh control plane supports a subset of Istio telemetry API used to configure access logs and trace.

Proxy request logging

Feature Managed (TD) Managed (istiod)
Traffic logs
Access logs * *

Tracing

Feature Managed (TD) Managed (istiod)
Cloud Trace * *
Jaeger tracing (allows use of customer-managed Jaeger) Compatible
Zipkin tracing (allows use of customer-managed Zipkin) Compatible

Networking

Traffic interception and redirection mechanisms

Feature Managed (TD) Managed (istiod)
Traditional use of iptables using init containers with CAP_NET_ADMIN
Istio Container Network Interface (CNI)
Whitebox sidecar

Protocol support

Feature Managed (TD) Managed (istiod)
IPv4
HTTP/1.1
HTTP/2
TCP byte streams (Note 1)
gRPC
IPv6

Notes:

  1. Although TCP is a supported protocol for networking and TCP metrics are collected, they are not reported. Metrics are displayed only for HTTP services in the Google Cloud console.
  2. Services that are configured with Layer 7 capabilities for the following protocols are not supported: WebSocket, MongoDB, Redis, Kafka, Cassandra, RabbitMQ, Cloud SQL. You might be able to make the protocol work by using TCP byte stream support. If TCP byte stream cannot support the protocol (for example, Kafka sends a redirect address in a protocol-specific reply and this redirect is incompatible with Cloud Service Mesh's routing logic), then the protocol isn't supported.

Envoy deployments

Feature Managed (TD) Managed (istiod)
Sidecars
Ingress gateway
Egress directly out from sidecars
Egress using egress gateways * *

CRD support

Feature Managed (TD) Managed (istiod)
Sidecar resource
Service entry resource
Percentage, fault injection, path matching, redirects, retries, rewriting, timeout, retry, mirroring, header manipulation, and CORS routing rules
`EnvoyFilter` API §
`WasmPlugin` API
Istio Operator

Load balancer for the Istio ingress gateway

Feature Managed (TD) Managed (istiod)
Third-party external load balancer
Google Cloud Internal load balancer * *

Service mesh cloud gateway

Feature Managed (TD) Managed (istiod)
Service mesh cloud gateway

Kubernetes Gateway API

Feature Managed (TD) Managed (istiod)
Kubernetes Gateway API

Load balancing policies

Feature Managed (TD) Managed (istiod)
Round robin
Least connections
Random
Passthrough
Consistent hash
Locality

Service entry

Feature Managed (TD) Managed (istiod)
ServiceEntry v1beta1 †

† The Cloud Service Mesh control plane implementation does not support following fields and values in fields:

  • workloadSelector field
  • endpoints[].network field
  • endpoints[].locality field
  • endpoints[].weight field
  • endpoints[].serviceAccount field
  • DNS_ROUND_ROBIN value in resolution field
  • MESH_INTERNAL value in location field
  • Unix domain socket address in endpoints[].address field

Destination rule

Feature Managed (TD) Managed (istiod)
DestinationRule v1beta1 †

† The Cloud Service Mesh control plane implementation does not support trafficPolicy.loadBalancer.localityLbSetting field and trafficPolicy.tunnel field. Additionally, the Cloud Service Mesh control plane implementation requires that the destination rule defining subsets is in the same namespace and cluster with the Kubernetes service or ServiceEntry.

Sidecar

Feature Managed (TD) Managed (istiod)
Sidecar v1beta1 †

† The Cloud Service Mesh control plane implementation does not support following fields and values in fields:

  • ingress field
  • egress.port field
  • egress.bind field
  • egress.captureMode field
  • inboundConnectionPool field

MeshConfig

Feature Managed (TD) Managed (istiod)
LocalityLB §
ExtensionProviders §
CACert
ImageType - distroless §
OutboundTrafficPolicy §
defaultProviders.accessLogging
defaultProviders.tracing
defaultConfig.tracing.stackdriver §
accessLogFile §

ProxyConfig

Feature Managed (TD) Managed (istiod)
DNS proxy (ISTIO_META_DNS_CAPTURE, ISTIO_META_DNS_AUTO_ALLOCATE)
HTTP/1.0 support (ISTIO_META_NETWORK)
Image selection (distroless or base image) †

† Distroless image is used for the injection.

Regions

GKE clusters must be in one of the following regions or any zone within the following regions.

Region Location
asia-east1 Taiwan
asia-east2 Hong Kong
asia-northeast1 Tokyo, Japan
asia-northeast2 Osaka, Japan
asia-northeast3 South Korea
asia-south1 Mumbai, India
asia-south2 Delhi, India
asia-southeast1 Singapore
asia-southeast2 Jakarta
australia-southeast1 Sydney, Australia
australia-southeast2 Melbourne, Australia
europe-central2 Poland
europe-north1 Finland
europe-southwest1 Spain
europe-west1 Belgium
europe-west2 England
europe-west3 Frankfurt, Germany
europe-west4 Netherlands
europe-west6 Switzerland
europe-west8 Milan, Italy
europe-west9 France
europe-west10 Berlin, Germany
europe-west12 Turin, Italy
me-central1 Doha
me-central2 Dammam, Saudi Arabia
me-west1 Tel Aviv
northamerica-northeast1 Montreal, Canada
northamerica-northeast2 Toronto, Canada
southamerica-east1 Brazil
southamerica-west1 Chile
us-central1 Iowa
us-east1 South Carolina
us-east4 Northern Virginia
us-east5 Ohio
us-south1 Dallas
us-west1 Oregon
us-west2 Los Angeles
us-west3 Salt Lake City
us-west4 Las Vegas

User interface

Feature Managed (TD) Managed (istiod)
Cloud Service Mesh dashboards in the Google Cloud console
Cloud Monitoring
Cloud Logging

Tooling

Feature Managed (TD) Managed (istiod)
gcloud beta container fleet mesh debug tool