Migrate from Cloud Run to Google Kubernetes Engine

Cloud Run and Kubernetes both use standard container images as deployment artifacts, they both use a declarative API model with resources that can be represented in YAML files with the same standard structure.

Introduction

The Cloud Run Admin API v1 is designed to maximize portability with Kubernetes, for example, the Cloud Run Admin API resources share the same structure conventions and attribute names as Kubernetes resources. See the Cloud Run service YAML reference.

The Cloud Run Admin API v1 implements the Knative Serving API specification, but you do not need to migrate to Knative to move your Cloud Run workloads to a Kubernetes cluster like GKE.

Quickstart

This quickstart is an example of a simple migration.

Simple resource comparison

Compare the following simple Cloud Run service named my-app with the equivalent Kubernetes deployment. Note how the YAML files are almost identical.

However, the parts in blue are different and need to be changed. The parts in green should be added.

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: my-app
  namespace: 'PROJECT_NUMBER'
spec:
  template:
    spec:
      containers:
      - image: gcr.io/cloudrun/hello
        env:
        - name: HELLO
          value: world

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: default
  labels:
    app: my-app
spec:
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - image: gcr.io/cloudrun/hello
        env:
        - name: HELLO
          value: world
  replicas: 3
  selector:
    matchLabels:
      app: my-app

Migrate a simple Cloud Run service to GKE

1. Download service YAML file into the current directory:

   gcloud run services describe my-app --format export > my-app.yaml

2. Modify the YAML to match a Kubernetes deployment:

   • For the "kind" attribute: replace the value "Service" with "Deployment"
   • For the "apiVersion" attribute: replace the value "serving.knative.dev/v1" with "apps/v1"
   • Replace metadata.namespace with the namespace of the GKE cluster you want to deploy to, for example default.
   • Add a new label to under metadata and spec.template.metadata.
   • Set a fixed number of instances ("replicas") using spec.template.spec.replicas and set a label selector in spec.template.spec.selector.

3. Install and use the kubectl command line tool to deploy the my-app.yaml file to your GKE cluster:

   kubectl apply -f ./my-app.yaml

4. Expose the deployment as a service:

   kubectl expose deployment my-app --type LoadBalancer --port 80 --target-port 8080

Considerations when migrating from Cloud Run to GKE

Cluster:

• Cloud Run is a fully managed platform, while GKE requires more platform management. If you have not yet created a GKE cluster, use GKE Autopilot.

• The scalability of GKE workloads is restricted by the size of the cluster. If you are not using an Autopilot cluster, consider using node auto-provisioning and a cluster autoscaler to resize your cluster.

• Cloud Run has a built-in zonal redundancy, so migrate to a regional cluster and provision enough replicas to ensure your service is resilient to a zonal outage in the selected Google Cloud region.

Pricing

Cloud Run charges for used resources, while GKE charges for provisioned resources.

Security:

• Contrary to Cloud Run, invoking a GKE service is not subject to an IAM invoker permission.

• Because GKE does not provide strong isolation between containers, consider using GKE Sandbox if you need to execute unknown or untrusted code.

Networking

Cloud Run requires a Serverless VPC Access connector to access other resources in a VPC. GKE workloads are directly in a VPC and do not need a connector.

Features not supported by Google Kubernetes Engine

The following Cloud Run features are not available on GKE:

• Configuration versioning via revisions
• Traffic splitting for gradual rollouts and rollbacks of revisions
• Request-based billing
• CPU boost
• Session affinity
• Labels on GKE workloads are not Google Cloud labels
• Tags on workloads

Migrate Cloud Run resources

The following sections describe migrating resources used in Cloud Run, such as Cloud Run services, jobs, and secrets.

Migrate Cloud Run services

You can migrate a Cloud Run service to the following resources on GKE:

1. Kubernetes deployment to create instances (called "pods" in Kubernetes).
2. Kubernetes services to expose the deployment at a specific endpoint.
3. Kubernetes Horizontal Pod Autoscaler: to automatically scale the deployment.

The attributes of a Kubernetes deployment are a superset of the attributes of a Cloud Run services. As shown in the quickstart, after you change the apiVersion and kind attributes to apps/v1 and Deployment, you also need to change the following:

• Replace namespace with the GKE cluster namespace to deploy to, for example default.
• serviceAccountName should reference a Kubernetes service account, which can optionally act as an IAM service account with Workload Identity Federation for GKE.
• Add a LABEL at metadata.labels and spec.template.metadata.labels that will be used to select the deployment and pods. For example: app: NAME
• Under spec.template:
  • Add a replicas attribute to specify a number of "instances".
  • Add a selector.matchLabels attribute that selects on the label LABEL.
• If your Cloud Run service mounts secrets, see Migrate secrets.
• If the migrated Cloud Run service was accessing resources on a Virtual Private Cloud, you do not need to use a Serverless VPC Access connector.

After creating the Kubernetes deployment, create a Kubernetes services to expose it:

apiVersion: v1
kind: Service
metadata:
  name: NAME
spec:
  selector:
    LABEL
  ports:
    - protocol: TCP
      port: 80
      targetPort: PORT

Replace:

• NAME: with the name of your service.
• LABEL: with the label defined in your deployment. For example app: NAME.
• PORT: with the containerPort of the container receiving requests in the Cloud Run service, which defaults to 8080.

You can then optionally create a Kubernetes Horizontal Pod Autoscaler in order to automatically scale the number of pods. Follow the Kubernetes Horizontal Pod Autoscaling documentation to create an HorizontalPodAutoscaler. Use the minimum instances (autoscaling.knative.dev/minScale) and maximum instances (autoscaling.knative.dev/maxScale) values of your Cloud Run service as values for the minReplicas and maxReplicas attributes HorizontalPodAutoscaler.

Migrate Cloud Run jobs

You can migrate a Cloud Run job to a Kubernetes job on GKE.

Contrary to Cloud Run jobs, Kubernetes jobs are executed when they are created. If you want to execute the job again, you need to create a new job.

The following samples show the structural difference between a Cloud Run job and a Kubernetes job:

apiVersion: run.googleapis.com/v1
kind: Job
metadata:
  name: my-job
spec:
  template:
    spec:
      template:
        spec:
          containers:
          - image: us-docker.pkg.dev/cloudrun/container/job

apiVersion: batch/v1
kind: Job
metadata:
  name: my-job
spec:
  template:
    spec:
      containers:
      - image: us-docker.pkg.dev/cloudrun/container/job

Migrate secrets

You can keep existing secrets in Secret Manager or you can migrate them to Kubernetes secrets.

If you choose to keep secrets in Secret Manager, you will need to update how you use them on GKE

If you choose to migrate from Secret Manager to Kubernetes secrets, consider these differences between Secret Manager secrets and Kubernetes secrets:

• Allowed characters in names:
  • Kubernetes secrets: [a-z0-9-.]{1,253}
  • Secret Manager secrets: [a-zA-Z0-9_-]{1,255}
• Versioning: Secrets from Secret Manager are versioned, while Kubernetes secrets aren't.
• Payload: Secrets from Secret Manager contain a single []byte, while Kubernetes secrets contain a map<string, string>.

Migration strategy

After you create the equivalent resources, exposing external endpoints behind a global external Application Load Balancer enables you to gradually migrate traffic between Cloud Run and Google Kubernetes Engine (GKE).