Use a canary deployment strategy

This document describes how to configure and use a canary deployment strategy.

What is a canary deployment?

A canary deployment is a progressive rollout of an application that splits traffic between an already-deployed version and a new version, rolling it out to a subset of users before rolling out fully.

Supported target types

Canary deployment in Cloud Deploy supports all target types, including the following:

• Google Kubernetes Engine
• Cloud Run (services only—not jobs.)
• GKE Enterprise

Canary also works with multi-targets.

Why use a canary deployment strategy?

A canary deployment gives you a chance to partially release your application. In this way, you can ensure the new version of your application is reliable before you deliver it to all users.

If you're deploying to GKE or GKE Enterprise, for example, you would deploy the new version of your application to a limited number of pods. The old version would continue to run, but with more of the traffic being sent to the new pods.

If you're deploying to Cloud Run, Cloud Run itself splits traffic between the old and new revisions, according to the percentages you configure.

Types of canary

Cloud Deploy lets you configure the following types of canary deployment:

• Automated

  With an automated canary deployment, you configure Cloud Deploy with a series of percentages that express a progressive deployment. Cloud Deploy performs additional operations on your behalf, to apportion traffic percentages between the old and new versions.

• Custom-automated

  For a custom-automated canary, you can provide the following:

  • The phase name
  • The percentage goal
  • The Skaffold profile to use for the phase
  • Whether or not to include a verify job
  • Whether or not to include a predeploy or postdeploy job, or both

  But you don't need to provide traffic-balancing information; Cloud Deploy creates the necessary resources.

• Custom

  With a custom canary, you configure each canary phase separately, including the following:

  • The phase name
  • The percentage goal
  • The Skaffold profile to use for the phase
  • Whether or not to include a verify job
  • Whether or not to include a predeploy or postdeploy job, or both

  Additionally for a fully custom canary, you provide all of the traffic-balancing configuration, as described here.

Phases of a canary deployment

When you create a release for a canary deployment, the rollout is created with a phase for each canary increment, plus a final stable phase for 100%.

For example, if you configure a canary for 25%, 50%, and 75% increments, the rollout will have the following phases:

• canary-25
• canary-50
• canary-75
• stable

You can read more about rollout phases, jobs, and job runs in Manage rollouts.

What happens during an automated or custom-automated canary

To support your canary deployment, Cloud Deploy includes special processing steps when rendering your Kubernetes manifest or Cloud Run service configuration:

If you want to see the differences among phases of a canary deployment, you can view changes in the per-phase rendered manifest available in the release inspector. You can do this even before the rollout has started. Also, if you're using parallel deployment, you can also inspect each child's rendered manifest.

Configure a canary deployment

This section describes how to configure your delivery pipeline and targets for a canary deployment.

The instructions here include only what is specific to canary configuration. The document Deploy your application has the general instructions for configuring and executing your deployment pipeline.

Make sure you have the required permissions

In addition to other Identity and Access Management permissions you need for using Cloud Deploy, you need the following permissions in order to perform additional actions that might be needed for a canary deployment:

• clouddeploy.rollouts.advance
• clouddeploy.rollouts.ignoreJob
• clouddeploy.rollouts.cancel
• clouddeploy.rollouts.retryJob
• clouddeploy.jobRuns.get
• clouddeploy.jobRuns.list
• clouddeploy.jobRuns.terminate

See IAM roles and permissions for more information about what available roles include these permissions.

Prepare your skaffold.yaml

As with a standard deployment, your canary needs a skaffold.yaml file, which defines how manifests and service definitions are rendered and deployed.

The skaffold.yaml you create for a canary deployment does not have any special requirements beyond what's needed for standard deployment.

Prepare your manifest or service definition

As with a standard deployment, your canary needs a Kubernetes manifest or a Cloud Run service definition.

Configure an automated canary

The following instructions are for Cloud Run and GKE and GKE Enterprise service-based networking targets. If you're using Kubernetes Gateway API with GKE or GKE Enterprise, see this documentation.

You configure the automated canary in your delivery pipeline definition:

serialPipeline:
  stages:
  - targetId: prod
    profiles: []
    strategy:
      canary:
        runtimeConfig:
          kubernetes:
            serviceNetworking:
              service: "SERVICE_NAME"
              deployment: "DEPLOYMENT_NAME"
              podSelectorLabel: "LABEL"
        canaryDeployment:
          percentages: [PERCENTAGES]
          verify: true|false
          predeploy:
            actions: "PREDEPLOY_ACTION"
          postdeploy:
            actions: "POSTDEPLOY_ACTION"

serialPipeline:
  stages:
  - targetId: prod
    profiles: []
    strategy:
      canary:
        runtimeConfig:
          cloudRun:
            automaticTrafficControl: true
        canaryDeployment:
          percentages: [PERCENTAGES]
          verify: true|false
          predeploy:
            actions: "PREDEPLOY_ACTION"
          postdeploy:
            actions: "POSTDEPLOY_ACTION"

Configure a custom canary

You can configure your canary manually instead of relying fully on the automation provided by Cloud Deploy. With custom canary configuration, you specify the following, in your delivery pipeline definition:

• Rollout phase names

  In a fully-automated canary, Cloud Deploy names the phases for you (canary-25, canary-75, stable, for example). With a custom canary, however, you can give each phase any name, as long as it's unique among all phases for this canary stage, and it honors resource name restrictions. But the final (100%) phase name must be stable.

• Percentage goals for each phase

  Specify the percentages separately, per phase.

• Skaffold profiles to use for the phase

  You can use a separate Skaffold profile for each phase, or the same profile, or any combination. And each profile can use a different Kubernetes manifest or Cloud Run service definition. You can also use more than one profile for a given phase. Cloud Deploy combines them.

• Whether there is a verify job for the phase

  Remember that if you're enabling verify, you need to configure your skaffold.yaml for verification also.

• Whether there are predeploy or postdeploy jobs for the phase

  If you're enabling predeploy or postdeploy jobs, you need to configure your skaffold.yaml for those jobs.

All target types are supported for custom canary.

Custom canary configuration elements

The following YAML shows the configuration for the phases of fully custom canary deployment:

strategy:
  canary:
    # Custom configuration for each canary phase
    customCanaryDeployment:
      phaseConfigs:
      - phaseId: "PHASE1_NAME"
        percentage: PERCENTAGE1
        profiles: [ "PROFILE_NAME" ]
        verify: true | false
        predeploy:
          actions: "PREDEPLOY_ACTION"
        postdeploy:
          actions: "POSTDEPLOY_ACTION"
      - …
      - phaseId: "stable"
        percentage: 100
        profiles: [ "LAST_PROFILE_NAME" ]
        verify: true|false
        predeploy:
          actions: "PREDEPLOY_ACTION"
        postdeploy:
          actions: "POSTDEPLOY_ACTION"

In this YAML

• PHASE1_NAME

  Is the name of the phase. Each phase name must be unique.

• [ "PROFILE_NAME" ]

  Is the name of the profile to use for the phase. You can use the same profile for each phase, or a different one for each, or any combination. Also, you can specify more than one profile. Cloud Deploy uses all of the profiles you specify, plus the profile or manifest used by the overall stage.

• stable

  The final phase must be named stable.

• PERCENTAGE1

  Is the percentage to deploy for the first phase. Each phase must have a unique percentage value, and that value must be a whole percentage (not 10.5, for example), and the phases must be in ascending order.

• verify: true|false

  Tells Cloud Deploy whether to include a verify job for the phase. Note that for each phase to use verify, Skaffold uses the same profile for verify that is specified for render and deploy for that phase.

• PREDEPLOY_ACTION

  Is the same as the ACTION_NAME that you used in your skaffold.yaml to define the custom action you want to run before deploying.

• POSTDEPLOY_ACTION

  Is the same as the ACTION_NAME that you used in your skaffold.yaml to define the custom action you want to run after deploying.

The percentage for the last phase must be 100. Phases are executed according in the order you configure them in this ​customCanaryDeployment stanza, but if the percentage values are not in ascending order, the command to register the delivery pipeline fails with an error.

Note that the configuration for a custom canary doesn't include a runtimeConfig stanza. If you include runtimeConfig, it's considered a custom-automated canary.

Configure a custom-automated canary

A custom-automated canary is similar to a custom canary because you specify the separate canary phases, with custom phase names, percentage values, Skaffold profiles, verify jobs, and predeploy and postdeploy jobs. But with a custom canary, you don't provide the configurations that define the traffic apportionment—Cloud Deploy does that for you, but you still provide the Skaffold profiles to be used for each stage.

To configure a custom-automated canary, include a runtimeConfig stanza, as shown here, and include the customCanaryDeployment stanza, as shown here.

Configure a canary deployment using Kubernetes Gateway API service mesh

Although you can use a Cloud Deploy canary deployment to deploy your application to Kubernetes service-based networking, An alternative is to use the Kubernetes Gateway API service mesh. This section describes how to do so.

You can use Gateway API with Istio or any supported implementation.

1. Set up your Gateway API resources:

   • Using Istio

   • GKE Gateway controller

   These are examples only.

2. In your Kubernetes manifest, provided to Cloud Deploy when you created the release, include the following:

   • An HTTPRoute that references your Gateway resource

   • A Deployment

   • A Service

3. Configure your delivery pipeline and the target that you will canary-deploy to:

   • Configuration for the target is the same as for any target.

   • The delivery pipeline configuration, in the progression sequence for the specific target, includes a gatewayServiceMesh stanza to reference your Kubernetes Gateway API HTTPRoute configuration, as well as your Deployment and Service.

     strategy:
      canary:
        runtimeConfig:
          kubernetes:
            gatewayServiceMesh:
              httpRoute: "ROUTE"
              service: "SERVICE"
              deployment: "DEPLOYMENT"
              routeUpdateWaitTime: "WAIT_TIME"
              podSelectorLabel: "LABEL"
        canaryDeployment:
          percentages:
          - 50
     

     Where...

     • ROUTE is your httpRoute configuration that defines the routing behavior you want.

     • SERVICE is your Service configuration, which Cloud Deploy requires for canary deployments to GKE and GKE Enterprise.

     • DEPLOYMENT is your Deployment configuration, which Cloud Deploy requires for canary deployments to GKE and GKE Enterprise.

     • WAIT_TIME is an amount of time for Cloud Deploy to wait for changes to the HTTPRoute resource to finish propagating, to avoid dropped requests. For example: routeUpdateWaitTime: 60s.

       If you're running canary using Gateway API without Istio, and the Gateway API is connected to a Google Cloud load balancer, a small amount of traffic may be lost when the canary instance is scaled down. You can configure this setting if you observe this behavior.

     • LABEL is a pod selector label. This must match the label selector in the Kubernetes Service and Deployment defined in your manifest. This is optional. The default is app.

Use parallel deployment with a canary deployment strategy

You can run a canary deployment using parallel deployment. This means the target you're progressively deploying to can comprise two or more child targets. For example, you can deploy progressively to clusters in separate regions, at the same time.

How is a parallel canary different from single-target canaries

• As with single-target canary deployment, if you're deploying to GKE targets, you need a Kubernetes Deployment configuration and a Kubernetes Service configuration in your manifest.

• As with single-target canary deployment, your delivery pipeline configuration must include a strategy.canary stanza inside the stage definition for the applicable stage.

• Additionally, you need to configure a multi-target, and you need to configure the child targets which that multi-target references.

• When you create a release, a controller rollout and the child rollouts are created.

  Both types of rollout—controller and child—have separate phases for all of the configured canary percentages, and a stable phase for the canary 100%.

• You can't advance a child rollout.

  You can advance controller rollouts only. When you advance the controller rollout to the next stage, the child rollouts are advanced too, by Cloud Deploy.

• You can't retry failed jobs in the controller rollout.

  You can retry a job in child rollouts only.

• You can't ignore failed jobs in the controller rollout.

  You can ignore failed jobs in child rollouts only.

• You can cancel a controller rollout, but you can't cancel child rollouts.

• You can terminate job runs under a child rollout only, not a controller rollout.

What to do if a parallel rollout fails in canary

When a child rollout fails, the controller rollout can transition to different states, depending on what happens with the child rollouts:

• If one or more child rollouts fail, but at least one child rollout is still IN_PROGRESS, the controller rollout remains IN_PROGRESS.

• If one or more child rollouts fail, but at least one child rollout succeeds, the controller rollout is HALTED if there are more phases after the current one.

  If this is the stable phase, the controller rollout is FAILED.

  HALTED gives you a chance to either ignore, retry failed jobs within the failed child rollout, or cancel the controller rollout and prevent further actions on the child rollouts.

• If the controller rollout is in a HALTED state because of a failed child rollout, and you ignore the failed job in the child rollout, the controller rollout reverts to an IN_PROGRESS state.

Deploy an HTTPRoute to a different cluster

When you have a canary configured using Gateway API service mesh, you can specify an alternate, non-target cluster on which to deploy the HTTPRoute.

To do so, you use a routeDestinations stanza, in your canary strategy configuration, to identify the destination cluster or clusters for the HTTPRoute, and a boolean setting to propagate the Service to the same non-target cluster. And you create an associatedEntities stanza in your target configuration to identify the clusters.

1. Configure associatedEntities on your target.

   Each entity is a cluster where Cloud Deploy will deploy the HTTPRoute and, optionally, the Kubernetes Service. In your target definition, include an associatedEntities stanza:

   associatedEntities:
     [KEY]:
       gkeClusters:
       - cluster: [PATH]
         internalIp: [true|false]
         proxyUrl:
   

   Where:

   • KEY is an arbitrary name for this group of associated entities. You'll use this name to reference the entities from the routeDestinations in your canary config.

   • PATH is the resource path identifying the GKE cluster where your HTTPRoute (and optionally the Service) will be deployed.

   • internalIp indicates whether or not you want use the internal IP (private IP) if the cluster has both an internal IP and a public IP configured. The default is false.

   You can include any number of clusters, with or without internalIp.

2. Configure routeDestinations in your canary config.

   Each route destination references an associatedEntities stanza, and indicates whether or not to also deploy the Service to the alternate cluster. Add the following inside gatewayServiceMesh stanza in your canary config:

   routeDestinations:
     destinationIds: ["KEY"]
     propagateService: [true|false]
   

   Where:

   • KEY is the name you configured in the target, in associatedEntities. Use this name to reference the entities from the routeDestinations in your canary config.

     You can also provide the value @self to deploy the HTTPRoute to the target cluster in addition to the associated destination.

   • propagateService indicates whether or not you want to deploy the Service to the associated cluster, in addition to the HTTPRoute. The default is false.

Execute the configured canary

To run the canary deployment:

1. Register the configured delivery pipeline and targets.

   gcloud deploy apply --file=PIPELINE
   

   The delivery pipeline includes the automated or custom canary configuration, for your chosen runtime.

   This command assumes your targets are defined in the same file or have otherwise already been registered. If not, be sure to register your targets too.

2. Create a release:

   gcloud deploy releases create RELEASE_NAME \
                                 --delivery-pipeline=PIPELINE_NAME \
                                 --region=REGION
   

   The delivery pipeline identified by PIPELINE_NAME contains the automated or custom canary configuration described in this document.

3. Advance the canary:

   gcloud CLI

   gcloud deploy rollouts advance ROLLOUT_NAME \
                               --release=RELEASE_NAME \
                               --delivery-pipeline=PIPELINE_NAME \
                               --region=REGION
   

   Where:

   ROLLOUT_NAME is the name of the current rollout which you're advancing to the next phase.

   RELEASE_NAME is the name of the release that this rollout is part of.

   PIPELINE_NAME is the name of the delivery pipeline you're using to manage deployment of this release.

   REGION is the name of the region in which the release was created, for example us-central1. This is required.

   See the Google Cloud SDK reference for more information about the gcloud deploy rollouts advance command.

   Google Cloud console

   1. Open the Delivery pipelines page.

   2. Click your pipeline shown in the list of delivery pipelines.

      The Delivery pipeline details page shows a graphical representation of your delivery pipeline's progress.

   3. On the Rollouts tab, under Delivery pipeline details, click the name of your rollout.

      The rollout details page is shown, for that rollout.

      

      Notice that in this example, the rollout has a canary-50 phase and a stable phase. Your rollout might have more phases or different phases.

   4. Click Advance rollout.

      The rollout is advanced to the next phase.

Skipped phases

If you deploy a canary and your application has not been deployed to that runtime yet, Cloud Deploy skips the canary phase and runs the stable phase. See Skipping phases the first time to find out why this happens.

What's next

• Try the canary deployment quickstart.

• Find out how to manage the lifecycle of your canary's rollouts.

• Learn more about parallel deployment.