This document describes how to configure and use canary deployments to deploy your applications to GKE or GKE Enterprise using Cloud Deploy with Kubernetes Gateway API service mesh.
A canary deployment is a progressive rollout of a new version of your application, where you gradually increase the percentage of traffic sent to the new version, while monitoring the application's performance. This helps you to catch potential problems early and minimize the impact on your users.
How canary deployments work for GKE and GKE Enterprise using Gateway API
In addition to the Deployment and Service references, you provide an HTTPRoute resource, with a
backendRefs
rule that references the Service.Cloud Deploy creates a new Deployment, with the name of your original Deployment plus
-canary
, and a new Service with the original Service name plus-canary
.Secrets, ConfigMaps, and Horizontal Pod Autoscalers are also copied and renamed with
-canary
.For each canary phase Cloud Deploy modifies the HTTPRoute to update the weighting between the original Deployment's pods and the canary Deployment's pods, based on the percentage for that phase.
Because there can be a delay propagating changes to
HTTPRoute
resources, you can include therouteUpdateWaitTime
property in your configuration, so the system waits a specified amount of time for this propagation.During the
stable
phase, the-canary
Deployment is scaled down to zero, and the original Deployment is updated to use the new release's Deployment.Also, the HTTPRoute is now reverted to the original which you supplied.
Cloud Deploy doesn't modify the original Deployment or Service until the
stable
phase.
Using Cloud Deploy, you can configure canary deployments to GKE and GKE Enterprise in a single stage or in multiple stages.
The instructions here include only what is specific to canary configuration. The document Deploy to a Google Kubernetes Engine cluster has the general instructions for configuring and executing your deployment pipeline.
Make sure you have the required permissions
In addition to other Identity and Access Management permissions you need for using Cloud Deploy, you need the following permissions in order to perform additional actions that might be needed for a canary deployment:
clouddeploy.rollouts.advance
clouddeploy.rollouts.ignoreJob
clouddeploy.rollouts.cancel
clouddeploy.rollouts.retryJob
clouddeploy.jobRuns.get
clouddeploy.jobRuns.list
clouddeploy.jobRuns.terminate
See IAM roles and permissions for more information about what available roles include these permissions.
Prepare your skaffold.yaml
Your skaffold.yaml
file defines how your Kubernetes manifests are rendered and deployed. For a canary deployment to GKE/GKE Enterprise, ensure it correctly points to your manifests and defines any necessary build artifacts. No special canary-specific configuration is required within skaffold.yaml
itself beyond what's needed for a standard deployment. You might use Skaffold profiles to manage different manifest variations for custom canary phases.
Prepare your Kubernetes manifests
Your Kubernetes manifests must include both a Deployment
resource and a Service
resource.
The Service
must define a selector
that matches the labels of the pods managed by the Deployment
.
The default label Cloud Deploy looks for is app
, but this can be configured in the pipeline.
In addition to the Deployment
and Service
, your manifests must include an HTTPRoute
resource configured for traffic splitting, referencing the Service
and associated Gateway.
Configure an automated canary
Use Kubernetes Gateway API (with Istio or any supported implementation) for precise, percentage-based traffic splitting managed by the mesh/gateway, orchestrated by Cloud Deploy.
Set up Gateway API Resources: Ensure your Gateway, and underlying service mesh (e.g., Istio) or Gateway controller are correctly configured in your cluster(s).
In your Kubernetes manifest, provided to Cloud Deploy when you created the release, include the following:
An
HTTPRoute
that references your Gateway resourceA Deployment
A Service
Configure your delivery pipeline and the target that you will canary-deploy to:
Configuration for the target is the same as for any target.
The delivery pipeline configuration, in the progression sequence for the specific target, includes a
gatewayServiceMesh
stanza to reference your Kubernetes Gateway APIHTTPRoute
configuration, as well as your Deployment and Service.strategy: canary: runtimeConfig: kubernetes: gatewayServiceMesh: httpRoute: "ROUTE" service: "SERVICE" deployment: "DEPLOYMENT" routeUpdateWaitTime: "WAIT_TIME" podSelectorLabel: "LABEL" canaryDeployment: percentages: - 50
Where...
ROUTE is your httpRoute configuration that defines the routing behavior you want.
SERVICE is your Service configuration, which Cloud Deploy requires for canary deployments to GKE and GKE Enterprise.
DEPLOYMENT is your Deployment configuration, which Cloud Deploy requires for canary deployments to GKE and GKE Enterprise.
WAIT_TIME is an amount of time for Cloud Deploy to wait for changes to the
HTTPRoute
resource to finish propagating, to avoid dropped requests. For example:routeUpdateWaitTime: 60s
.If you're running canary using Gateway API without Istio, and the Gateway API is connected to a Google Cloud load balancer, a small amount of traffic may be lost when the canary instance is scaled down. You can configure this setting if you observe this behavior.
LABEL is a pod selector label. This must match the label selector in the Kubernetes Service and Deployment defined in your manifest. This is optional. The default is
app
.
Configure a custom canary
You can configure your canary manually instead of relying fully on the automation provided by Cloud Deploy. With custom canary configuration, you specify the following, in your delivery pipeline definition:
Rollout phase names
In a fully-automated canary, Cloud Deploy names the phases for you (
canary-25
,canary-75
,stable
, for example). With a custom canary, however, you can give each phase any name, as long as it's unique among all phases for this canary stage, and it honors resource ID restrictions. But the final (100%) phase name must bestable
.Percentage goals for each phase
Specify the percentages separately, per phase.
Skaffold profiles to use for the phase
You can use a separate Skaffold profile for each phase, or the same profile, or any combination. And each profile can use a different Kubernetes manifest. You can also use more than one profile for a given phase. Cloud Deploy combines them.
Whether there is a verify job for the phase
Remember that if you're enabling verify, you need to configure your
skaffold.yaml
for verification also.Whether there are predeploy or postdeploy jobs for the phase
If you're enabling predeploy or postdeploy jobs, you need to configure your
skaffold.yaml
for those jobs.
All target types are supported for custom canary.
Custom canary configuration elements
The following YAML shows the configuration for the phases of fully custom canary deployment:
strategy:
canary:
# Custom configuration for each canary phase
customCanaryDeployment:
phaseConfigs:
- phaseId: "PHASE1_NAME"
percentage: PERCENTAGE1
profiles: [ "PROFILE_NAME" ]
verify: true | false
predeploy:
actions: "PREDEPLOY_ACTION"
postdeploy:
actions: "POSTDEPLOY_ACTION"
- …
- phaseId: "stable"
percentage: 100
profiles: [ "LAST_PROFILE_NAME" ]
verify: true|false
predeploy:
actions: "PREDEPLOY_ACTION"
postdeploy:
actions: "POSTDEPLOY_ACTION"
In this YAML
PHASE1_NAME
Is the name of the phase. Each phase name must be unique.
[ "PROFILE_NAME" ]
Is the name of the profile to use for the phase. You can use the same profile for each phase, or a different one for each, or any combination. Also, you can specify more than one profile. Cloud Deploy uses all of the profiles you specify, plus the profile or manifest used by the overall stage.
stable
The final phase must be named
stable
.PERCENTAGE1
Is the percentage to deploy for the first phase. Each phase must have a unique percentage value, and that value must be a whole percentage (not
10.5
, for example), and the phases must be in ascending order.verify: true|false
Tells Cloud Deploy whether to include a verify job for the phase. Note that for each phase to use verify, Skaffold uses the same profile for verify that is specified for render and deploy for that phase.
PREDEPLOY_ACTION
Is the same as the ACTION_NAME that you used in your
skaffold.yaml
to define the custom action you want to run before deploying.POSTDEPLOY_ACTION
Is the same as the ACTION_NAME that you used in your
skaffold.yaml
to define the custom action you want to run after deploying.
The percentage for the last phase must be 100
. Phases are executed according
in the order you configure them in this customCanaryDeployment
stanza, but if
the percentage values are not in ascending order, the command to
register the delivery pipeline
fails with an error.
Note that the configuration for a Gateway API custom canary doesn't include a
runtimeConfig
stanza. If you include runtimeConfig
, it's considered a
custom-service-based canary.
Configure a custom-automated canary
This combines custom phase definition (names, percentages, profiles, verify, hooks) with Cloud Deploy's automatic traffic management for GKE or GKE Enterprise. You define the phases, but Cloud Deploy handles the underlying resource manipulation based on the percentages and the chosen runtimeConfig
.
To configure this, include both a runtimeConfig
section with serviceNetworking
and the customCanaryDeployment
section (defining phaseConfigs) within the strategy.canary block. Cloud Deploy will use the specified Skaffold profiles for rendering but will automatically adjust traffic according to the runtimeConfig
and phase percentages.
serialPipeline:
stages:
- targetId: gke-prod
profiles: []
strategy:
canary:
# Include runtimeConfig for automatic traffic management
runtimeConfig:
kubernetes:
gatewayServiceMesh:
httpRoute: "my-route"
service: "my-app"
deployment: "my-deployment"
# Include customCanaryDeployment for phase customization
customCanaryDeployment:
phaseConfigs:
- phaseId: "warmup"
percentage: 10
profiles: ["profile-a"] # Profile used for rendering this phase
verify: true
- phaseId: "scaling"
percentage: 50
profiles: ["profile-b"] # Different profile for this phase
verify: true
- phaseId: "stable"
percentage: 100
profiles: ["profile-b"] # Can reuse profiles
verify: true
Deploy an HTTPRoute to a different cluster
When you have a canary configured using Gateway API service mesh, you can specify an alternate, non-target cluster on which to deploy the HTTPRoute.
To do so, you use a routeDestinations
stanza, in your canary strategy
configuration, to identify the destination cluster or clusters for the
HTTPRoute, and a boolean setting to propagate the Service to the same non-target
cluster. And you create an associatedEntities
stanza in your target
configuration to identify the clusters.
Configure
associatedEntities
on your target.Each entity is a cluster where Cloud Deploy will deploy the HTTPRoute and, optionally, the Kubernetes Service. In your target definition, include an
associatedEntities
stanza:associatedEntities: [KEY]: gkeClusters: - cluster: [PATH] dnsEndpoint: [true|false] internalIp: [true|false] proxyUrl:
Where:
KEY
is an arbitrary name for this group of associated entities. You'll use this name to reference the entities from therouteDestinations
in your canary config.PATH
is the resource path identifying the GKE cluster where your HTTPRoute (and optionally the Service) will be deployed.dnsEndpoint
indicates whether or not to use the cluster's DNS Endpoint if multiple endpoints are configured. The default isfalse
.internalIp
indicates whether or not to use the cluster's internal IP (private IP) if multiple endpoints are configured. The default isfalse
.
You can include any number of clusters, with or without
internalIp
.Configure
routeDestinations
in your canary config.Each route destination references an
associatedEntities
stanza, and indicates whether or not to also deploy the Service to the alternate cluster. Add the following insidegatewayServiceMesh
stanza in your canary config:routeDestinations: destinationIds: ["KEY"] propagateService: [true|false]
Where:
KEY
is the name you configured in the target, inassociatedEntities
. Use this name to reference the entities from therouteDestinations
in your canary config.You can also provide the value
@self
to deploy the HTTPRoute to the target cluster in addition to the associated destination.propagateService
indicates whether or not you want to deploy the Service to the associated cluster, in addition to the HTTPRoute. The default isfalse
.
Execute the GKE or GKE Enterprise canary
Register Pipeline and Targets: Apply your delivery pipeline and GKE or GKE Enterprise target configuration files.
gcloud deploy apply --file=delivery-pipeline.yaml --region=REGION gcloud deploy apply --file=gke-targets.yaml --region=REGION
The delivery pipeline includes the automated or custom canary configuration, for your chosen runtime.
Create a Release: Start the deployment, providing the image name.
gcloud deploy releases create RELEASE_NAME \ --delivery-pipeline=PIPELINE_NAME \ --region=REGION # e.g., --images=my-cloudrun-service=gcr.io/my-project/my-app:v1.1 # Add --skaffold-file or --source if not using default Skaffold config discovery
The delivery pipeline identified by
PIPELINE_NAME
contains the automated or custom canary configuration described in this document.Advance the canary:
gcloud CLI
gcloud deploy rollouts advance ROLLOUT_NAME \ --release=RELEASE_NAME \ --delivery-pipeline=PIPELINE_NAME \ --region=REGION
Where:
ROLLOUT_NAME
is the name of the current rollout which you're advancing to the next phase.RELEASE_NAME
is the name of the release that this rollout is part of.PIPELINE_NAME
is the name of the delivery pipeline you're using to manage deployment of this release.REGION
is the name of the region in which the release was created, for exampleus-central1
. This is required.See the Google Cloud SDK reference for more information about the
gcloud deploy rollouts advance
command.Google Cloud console
Click your pipeline shown in the list of delivery pipelines.
The Delivery pipeline details page shows a graphical representation of your delivery pipeline's progress.
On the Rollouts tab, under Delivery pipeline details, click the name of your rollout.
The rollout details page is shown, for that rollout.
Notice that in this example, the rollout has a
canary-50
phase and astable
phase. Your rollout might have more phases or different phases.Click Advance rollout.
The rollout is advanced to the next phase.
Skipped phases
If you deploy a canary and your application has not been deployed to that runtime yet, Cloud Deploy skips the canary phase and runs the stable phase. See Skipping phases the first time to find out why this happens.
What's next
Try the canary deployment quickstart.
Find out how to manage the lifecycle of your canary's rollouts.
Learn more about parallel deployment.
Learn more about Cloud Deploy deployment strategies.