GPU-accelerated video transcoding with FFmpeg on Cloud Run jobs

This tutorial describes how to transcode low-priority offline videos using Cloud Run jobs.

Objectives

In this tutorial, you will do the following:

Costs

In this document, you use the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator.

New Google Cloud users might be eligible for a free trial.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

  2. Install the Google Cloud CLI.

  3. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  4. To initialize the gcloud CLI, run the following command: 

    gcloud init

  5. Create or select a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

  6. Verify that billing is enabled for your Google Cloud project.

  7. Install the Google Cloud CLI.

  8. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  9. To initialize the gcloud CLI, run the following command: 

    gcloud init

  10. Create or select a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    • Create a Google Cloud project:

      gcloud projects create PROJECT_ID

      Replace PROJECT_ID with a name for the Google Cloud project you are creating.

    • Select the Google Cloud project that you created:

      gcloud config set project PROJECT_ID

      Replace PROJECT_ID with your Google Cloud project name.

  11. Verify that billing is enabled for your Google Cloud project.

  12. Enable the Cloud Run, Artifact Registry, and Cloud Build APIs:

    Roles required to enable APIs

    To enable APIs, you need the Service Usage Admin IAM role (roles/serviceusage.serviceUsageAdmin), which contains the serviceusage.services.enable permission. Learn how to grant roles. 

    gcloud services enable run.googleapis.com cloudbuild.googleapis.com artifactregistry.googleapis.com
  13. Set your region as an environment variable: 
    export REGION=REGION
  14. Create a service account: 
    gcloud iam service-accounts create video-encoding
  15. Request Total Nvidia L4 GPU allocation without zonal redundancy, per project per region, under Cloud Run Admin API in the Quotas and system limits page to complete this tutorial. Alternatively, you can deploy a Cloud Run service to automatically receive a grant of 3 nvidia-l4 GPU quota (zonal redundancy off) for a region.

Required roles

To get the permissions that you need to complete the tutorial, ask your administrator to grant you the following IAM roles on your project:

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Grant the roles

Console

  1. In the Google Cloud console, go to the IAM page.

    Go to IAM
  2. Select the project.
  3. Click Grant access.

  4. In the New principals field, enter your user identifier. This is typically the Google Account email address that is used to deploy the Cloud Run service.

  5. In the Select a role list, select a role.
  6. To grant additional roles, click Add another role and add each additional role.
  7. Click Save.

gcloud

To grant the required IAM roles to your account on your project:

     gcloud projects add-iam-policy-binding PROJECT_ID \
         --member=PRINCIPAL \
         --role=ROLE

Replace:

  • PROJECT_NUMBER with your Google Cloud project number.
  • PROJECT_ID with your Google Cloud project ID.
  • PRINCIPAL with the account you are adding the binding for. This is typically the Google Account email address that is used to deploy the Cloud Run service.
  • ROLE with the role you are adding to the deployer account.

Prepare your application

To retrieve the code sample for use:

  1. Clone the sample repository to your local machine:

    git clone https://github.com/GoogleCloudPlatform/cloud-run-samples

  2. Change to the directory that contains the Cloud Run sample code:

    cd cloud-run-samples/jobs-video-encoding

Create Cloud Storage buckets

To store the videos for processing, and to save the results of encoding, create the following two Cloud Storage buckets:

  1. Create a bucket to store videos before processing:

    gcloud storage buckets create gs://preprocessing-PROJECT_ID \
  --location LOCATION

    Replace the following:

    • PROJECT_ID: your project ID.
    • LOCATION: the Cloud Storage location.

  2. Grant the service account access to read from this bucket:

    gcloud storage buckets add-iam-policy-binding gs://preprocessing-PROJECT_ID \
  --member="serviceAccount:video-encoding@PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/storage.objectViewer"

    Replace PROJECT_ID with your project ID.

  3. Create a bucket to store transcoded videos after processing:

    gcloud storage buckets create gs://transcoded-PROJECT_ID \
  --location LOCATION

    Replace the following:

    • PROJECT_ID: your project ID.
    • LOCATION: the Cloud Storage location.

  4. Grant the service account access to read from and write to this bucket:

    gcloud storage buckets add-iam-policy-binding gs://transcoded-PROJECT_ID \
  --member="serviceAccount:video-encoding@PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/storage.objectAdmin"

    Replace PROJECT_ID with your project ID.

Deploy a Cloud Run job

Create a Cloud Run job by using the Dockerfile in the sample repository and mounting the buckets that you created:

  1. Navigate to the sample directory:

    cd cloud-run-samples/jobs-video-encoding

  1. Create an Artifact Registry if the default Cloud Run registry doesn't already exist:

    gcloud artifacts repositories create cloud-run-source-deploy \
  --repository-format=docker \
  --location LOCATION

    Replace LOCATION with the name of the location of the registry.

  2. Build the container image:

    gcloud builds submit \
  --tag LOCATION-docker.pkg.dev/PROJECT_ID/cloud-run-source-deploy/IMAGE_NAME \
  --machine-type E2-HIGHCPU-32

    Replace the following:

    • PROJECT_ID: your project ID.
    • LOCATION:name of the location of the registry.
    • IMAGE_NAME: name for the container image, for example: ffmpeg-image.

    Cloud Run uses a larger machine type to reduce build time.

  3. Deploy the job:

    gcloud beta run jobs create video-encoding-job \
    --image LOCATION-docker.pkg.dev/PROJECT_ID/cloud-run-source-deploy/IMAGE_NAME \
    --region REGION \
    --memory 32Gi \
    --cpu 8 \
    --gpu 1 \
    --gpu-type nvidia-l4 \
    --no-gpu-zonal-redundancy \
    --max-retries 1 \
    --service-account video-encoding@PROJECT_ID.iam.gserviceaccount.com \
    --add-volume=name=input-volume,type=cloud-storage,bucket=preprocessing-PROJECT_ID,readonly=true \
    --add-volume-mount=volume=input-volume,mount-path=/inputs \
    --add-volume=name=output-volume,type=cloud-storage,bucket=transcoded-PROJECT_ID \
    --add-volume-mount=volume=output-volume,mount-path=/outputs

    Replace the following:

    • PROJECT_ID: your project ID.
    • REGION: the name of the region. Note: This must be the same region that you have GPU quota for.
    • IMAGE_NAME: name for the container image, for example, ffmpeg-image.

    If this is the first time you deployed from source in this project, Cloud Run prompts you to create a default Artifact Registry repository.

Run the job

To run the job, follow these steps:

  1. Upload an example video to encode:

    gcloud storage cp gs://cloud-samples-data/video/cat.mp4 gs://preprocessing-PROJECT_ID

  2. Run the job:

    gcloud run jobs execute video-encoding-job  \
    --region REGION \
    --wait \
    --args="cat.mp4,encoded_cat.mp4,-vcodec,h264_nvenc,-cq,21,-movflags,+faststart"

    The entrypoint.sh file requires an input file, output file, and any arguments to send to FFmpeg.

  3. Review the Cloud Run logs to make sure the video trancoded:

    gcloud run jobs logs read video-encoding-job --region REGION

  4. Download the transcoded video:

    gcloud storage cp gs://transcoded-PROJECT_ID/encoded_cat.mp4 .

Clean up

To avoid additional charges to your Google Cloud account, delete all the resources you deployed with this quickstart.

Delete your repository

Cloud Run only charges for the time your job executes. However, you might still be charged for storing the container image in Artifact Registry. To delete Artifact Registry repositories, follow the steps in Delete repositories in the Artifact Registry documentation.

Delete your job

Cloud Run jobs only incur cost when a job task is executing. To delete your Cloud Run job, follow one of these steps:

Console

To delete a job:

  1. In the Google Cloud console, go to Cloud Run:

    Go to Cloud Run

  2. Locate the job you want to delete in the jobs list, and click its checkbox to select it.

  3. Click Delete. This terminates all the job executions in progress and all running container instances.

gcloud

To delete a job, run the following command:

gcloud run jobs delete JOB_NAME

Replace JOB_NAME with the name of the job.

Delete your test project

Deleting your Google Cloud project stops billing for all resources in that project. To release all Google Cloud resources in your project, follow these steps:

    Delete a Google Cloud project:

    gcloud projects delete PROJECT_ID

What's next