Verify your deployment

This document describes how to verify a Cloud Deploy deployment.

You can configure Cloud Deploy and Skaffold to verify that an application that you have deployed to any target is working correctly. Verification is done using your own testing image, and you configure Cloud Deploy and Skaffold to run those tests after deployment finishes.

By default, deployment verification runs in the Cloud Deploy execution environment, but you can also configure it to run on the same cluster where the application is running.

How does deployment verification work?

  1. You configure Skaffold for verification.

    This configuration identifies the container image or images to use to run tests, and the specific commands (script, for example) to run from that container image.

  2. You configure one or more targets in your delivery pipeline for deployment verification.

    This configuration enables verification for workloads deployed to that target.

  3. After a rollout is deployed (skaffold apply), Cloud Deploy runs the skaffold verify command in the Cloud Deploy execution environment.

    For deployments to Google Kubernetes Engine and GKE Enterprise, you can optionally run the verification container on the same cluster where the application container is running.

  4. Skaffold invokes the test or tests specified in the verify stanza of your skaffold.yaml to run against the deployed application.

  5. Success or failure of the tests executed indicates success or failure of the verification.

    • Verification success is determined by the executed container's exit code.

      0 indicates success. A non-zero exit code indicates failure. To generate the desired verification result, make sure the container exits with the appropriate exit code. If more than one container is executed as part of verification, they must all succeed for verification to succeed.

    • If the verification fails, the rollout also fails.

    • If a deployment fails during verification, you can see that by inspecting the rollout:

      Details in Google Cloud console for rollout, including verification status

  6. You can ignore or retry a failed verification.

    You can also terminate an in-progress verify job.

Components used for verification

The rollout resource includes the following objects, which support deployment verification:

  • Phase

    The collection of operations (jobs) in a rollout that are logically grouped together, for example a deploy or a deploy and verify.

  • Job

    The specific operation to be performed on a rollout, such as deploy or verify.

  • Job run

    A child of the rollout resource, the job run is an instance of a job, for example an attempt to deploy.

For more about Cloud Deploy resources, see Cloud Deploy service architecture

Notifications generated by deployment verification

Cloud Deploy generates Pub/Sub messages and publishes them for the following events:

  • Job run create, update, and delete

    These notifications are published to the clouddeploy-resources topic, and contain the following attributes:

    • Resource
    • ResourceType (JobRun)
    • Action (Create, Update, Delete)
    • ProjectNumber
    • Location
    • TargetId
    • DeliveryPipelineId
    • ReleaseId
    • RolloutId
    • JobRunId

The following is an example Pub/Sub message for a job run create, published to the clouddeploy-resources topic:

{
    "ackId": "UAYWLF1GSFE3GQhoUQ5PXiM_NSAoRRAGAE8CKF15MFcrQVh9Dz4NGXJ9YXRiWRIJBkUHeF9cEQ1iXE5EB0nq0KDVV1dKXxYGAExQeVhbHQVoWVh0Bnn7h5nK-8HjYwk9OqKarPdtO4PY2fNHZiI9XhJLLD5-My5FQV5AEkw4G0RJUytDCypYEU4EISE-MD5FU0Q",
    "message": {
      "attributes": {
        "Action": "Create",
        "DeliveryPipelineId": "dv-pipeline",
        "JobRunId": "634f8c6f-30c3-49ca-af80-68dc24d4cc5d",
        "Location": "us-central1",
        "ProjectNumber": "253401481285",
        "ReleaseId": "test-release-100",
        "Resource": "projects/253401481285/locations/us-central1/deliveryPipelines/dv-pipeline/releases/test-release-100/rollouts/test-release-100-to-dev-0001/jobRuns/634f8c6f-30c3-49ca-af80-68dc24d4cc5d",
        "ResourceType": "JobRun",
        "RolloutId": "test-release-100-to-dev-0001"
      },
      "messageId": "5572937706805411",
      "publishTime": "2022-09-07T14:00:46.040Z"
    }
  },

  • Job run start, succeed, and fail

    These notifications are published to the clouddeploy-operations topic, and contain the following attributes:

    • Resource
    • ResourceType (JobRun)
    • Action (Start, Succeed, Failure)
    • ProjectNumber
    • Location
    • TargetId
    • DeliveryPipelineId
    • ReleaseId
    • RolloutId
    • JobRunId
    • PhaseId
    • JobId
    • JobType (Deploy or Verify)

The following is an example Pub/Sub message for a failed job run, published to the clouddeploy-operations topic:

{
    "ackId": "RFAGFixdRkhRNxkIaFEOT14jPzUgKEUUBAgUBXx9cEFPdVhec2hRDRlyfWB9aVsbCAUXU3cJURsHaE5tdR-6xcvaS0NVb18UAgRFWndfXhMEblhfcy-fkK3HwvT9U0AvOemNgdZpe6jHiulvZiM9XxJLLD5-My5FQV5AEkw4G0RJUytDCypYEU4EISE-MD5FUw",
    "message": {
      "attributes": {
        "Action": "Failure",
        "DeliveryPipelineId": "dv-pipeline",
        "JobId": "verify",
        "JobRunId": "b389224a-c259-4a00-ab75-c22e48bc3136",
        "JobType": "Verify",
        "Location": "us-central1",
        "PhaseId": "stable",
        "ProjectNumber": "253401481285",
        "ReleaseId": "test-release-101",
        "Resource": "projects/253401481285/locations/us-central1/deliveryPipelines/dv-pipeline/releases/test-release-101/rollouts/test-release-101-to-dev-0001/jobRuns/b389224a-c259-4a00-ab75-c22e48bc3136",
        "ResourceType": "JobRun",
        "RolloutId": "test-release-101-to-dev-0001",
        "TargetId": "dev"
      },
      "messageId": "5573609905896436",
      "publishTime": "2022-09-07T15:35:37.906Z"
    }
  },

Configure Cloud Deploy for deployment verification

Enabling deployment verification for a Cloud Deploy target consists of adding a verify: true property to a given target (or targets) in a delivery pipeline progression, as shown in this example:

apiVersion: deploy.cloud.google.com/v1
kind: DeliveryPipeline
metadata:
 name: my-demo-app
description: main application pipeline
serialPipeline:
 stages:
 - targetId: dev
   profiles: []
   strategy:
     standard:
       verify: true
 - targetId: prod
   profiles: []
   strategy:
     standard:
       verify: false

In this configuration, deployment verification is enabled on the dev target, but not on the prod target. verify: false is equivalent to omitting the verify property or the entire strategy stanza.

The verify operation is executed within its own execution environment. This execution environment can be configured for VERIFY the same as it can be for RENDER and DEPLOY.

Configure Skaffold for deployment verification

Enabling deployment verification for a target requires a verify stanza in the skaffold.yaml configuration file for your deployment. This configuration can be for a specific Skaffold profile, if you're using separate profiles per target.

This verify stanza identifies a container to run to do the verification— for example, an integration test.

The following is an example skaffold.yaml that includes a verify stanza:

apiVersion: skaffold/v4beta7
kind: Config
build:
  artifacts:
    - image: integration-test
      context: integration-test
manifests:
  rawYaml:
  - kubernetes.yaml
deploy:
  kubectl: {}
verify:
- name: verify-integration-test
  container:
    name: integration-test
    image: integration-test
    command: ["./test-systems.sh"]
- name: verify-endpoint-test
  container:
    name: alpine
    image: alpine
    command: ["/bin/sh"]
    args: ["-c", "wget #ENDPOINT_URL"]

This simple example shows a verify stanza identifying a container to use and a test script to run in that container. #ENDPOINT_URL in this example is just a placeholder for your applications URL, and is not an available Cloud Deploy environment variable.

Run the verification container on the application cluster

By default, deployment verification runs in the Cloud Deploy execution environment. You can also configure Skaffold to run the verification container on the same cluster where your application is running. When you configure in-cluster verification in skaffold.yaml and enable verification on a target, the verification runs automatically in that target's cluster.

This ability is available for deployments to GKE and GKE Enterprise only, not for Cloud Run. Deployments to Cloud Run can only run verification in the Cloud Deploy execution environment.

In-cluster verification requires Skaffold version 2.3 or higher.

To run your verification container on the cluster, include an executionMode.kubernetesCluster stanza in your skaffold.yaml configuration file, inside the verify stanza for the specific verification container:

verify:
- name: 
  container:
    name: 
    image: 
    command: 
    args: 
  executionMode:
    kubernetesCluster:

The following is an example verify stanza that includes executionMode to invoke the verification container on the application cluster:

verify:
- name: integration-test-container
  container:
    name: integration-test-container
    image: integration-test-container
  executionMode:
    kubernetesCluster: {}

The executionMode stanza is optional, and if you omit it, Skaffold runs the verification container in the Cloud Deploy execution environment.

Retry the verification

When a verify job fails, you can retry the verification, creating a new job run:

gcloud deploy rollouts retry-job ROLLOUT_NAME \
             --job-id=JOB_ID \
             --phase-id=PHASE_ID \
             --delivery-pipeline=PIPELINE_NAME \
             --release=RELEASE_NAME \
             --region=REGION

Retrying the verification changes the state of the rollout from FAILED to IN_PROGRESS.

You can only retry a verification for a rollout whose verify job failed.

Available environment variables

Cloud Deploy provides and populates the following environment variables in the VERIFY execution environment, which you can use for your tests:

  • ANTHOS_MEMBERSHIP

    For targets of type ANTHOS, the fully specified resource name of the Anthos membership.

  • CLOUD_RUN_LOCATION

    For targets of type RUN, the region the Cloud Run service is deployed in.

  • CLOUD_RUN_PROJECT

    For targets of type RUN, the project in which the Cloud Run service was created.

  • CLOUD_RUN_SERVICE

    For targets of type RUN, the fully-specified resource name of the deployed Cloud Run service, for example projects/p/locations/us-central1/services/dev.

  • CLOUD_RUN_SERVICE_URLS

    For targets of type RUN, the URL or URLs (comma-separated list) that end users will use to access your service. You can find these in the Cloud Run service details for your service, in the Google Cloud console.

  • CLOUD_RUN_REVISION

    For targets of type RUN, the specific revision of the Cloud Run service.

  • GKE_CLUSTER

    For targets of type GKE, the fully specified resource name of the Google Kubernetes Engine cluster, for example projects/p/locations/us-central1/clusters/dev.

  • TARGET_TYPE

    The specific runtime type of the target. Either GKE, ANTHOS, or RUN.

  • CLOUD_DEPLOY_LOCATION

    The region in which the execution environment is running.

  • CLOUD_DEPLOY_DELIVERY_PIPELINE

    The ID delivery pipeline the execution environment is running.

  • CLOUD_DEPLOY_TARGET

    The ID of the target the execution environment is running.

  • CLOUD_DEPLOY_PROJECT

    The project number of the Google Cloud project in which the execution environment is running.

  • CLOUD_DEPLOY_RELEASE

    The ID of the release in which the verify will run.

  • CLOUD_DEPLOY_ROLLOUT

    The ID of the rollout that contains the jobs for verify.

  • CLOUD_DEPLOY_JOB_RUN

    The ID of the job run that represents the current execution of the job.

  • CLOUD_DEPLOY_PHASE

    The phase in the rollout that contains the job for verify.

Deploy parameters as environment variables

In addition to the environment variables listed in this section, Cloud Deploy can pass to your custom containers any deploy parameters you've set.

Learn more.