Deploy image processing using microservices and asynchronous messaging

Last reviewed 2023-07-17 UTC

This document describes how to implement the reference architecture that's described in Integrate microservices with Pub/Sub and GKE. The architecture is designed to handle long-running processes by using containers and asynchronous messaging.

The document uses an example photo-sharing application that generates photo thumbnails. You deploy the application using Google Kubernetes Engine (GKE) and use Pub/Sub to invoke long-running processes asynchronously. You also use Pub/Sub notifications for Cloud Storage to add side jobs without modifying the application's code.

The application is containerized by Cloud Build and stored in Artifact Registry. It uses Cloud Vision to detect inappropriate images.

Architecture

The following diagram illustrates the design of the example photo album application that implements the reference architecture.

Architecture of the photo album application.

Figure 1. Architecture for image processing that's based on using containers and asynchronous messaging.

The preceding diagram illustrates how the thumbnail is generated:

  1. A client uploads an image to the application.
  2. The application stores the image in Cloud Storage.
  3. A request is generated for the thumbnail.
  4. The thumbnail generator generates the thumbnail.
  5. The successful response is sent to the photo album application.
  6. The successful response is sent to the client and you can find the thumbnail in Cloud Storage.

The following diagram shows how the application implements thumbnail generation as a separate service in an asynchronous manner.

Architecture of the thumbnail extraction process.

Figure 2. Architecture of the thumbnail extraction process.

You use Pub/Sub to send service requests to the thumbnail generation service. This new architecture makes the service call asynchronous so that a thumbnail is created in the background after the application sends the response back to a client. This design also allows the thumbnail generation service to scale so that multiple jobs can run in parallel.

Objectives

  • Deploy an example photo album application on GKE.
  • Make asynchronous service calls from the application.
  • Use Pub/Sub notifications for Cloud Storage to trigger the application when a new file is uploaded to the Cloud Storage bucket.
  • Use Pub/Sub to perform more tasks without modifying the application.

Costs

In this document, you use the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish building the example application, you can avoid continued billing by deleting the resources you created. For more information, see Clean up.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Google Cloud project.

  4. Enable the GKE, Cloud SQL, Cloud Build, Artifact Registry, and Cloud Vision APIs.

    Enable the APIs

  5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  6. Make sure that billing is enabled for your Google Cloud project.

  7. Enable the GKE, Cloud SQL, Cloud Build, Artifact Registry, and Cloud Vision APIs.

    Enable the APIs

  8. In the Google Cloud console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

Set up the environment

In this section, you assign default settings for values that are used throughout the document. If you close your Cloud Shell session, you lose these environment settings.

  1. In Cloud Shell, set your default Google Cloud project:

    gcloud config set project PROJECT_ID
    

    Replace PROJECT_ID with your Google Cloud project ID.

  2. Set your default Compute Engine region:

    gcloud config set compute/region REGION
    export REGION=REGION
    

    Replace REGION with a region that is close to you. For more information, see Regions and zones.

  3. Set your default Compute Engine zone:

    gcloud config set compute/zone ZONE
    export ZONE=ZONE
    

    Replace ZONE with a zone that is close to you.

  4. Download the example application files and set your current directory:

    git clone https://github.com/GoogleCloudPlatform/gke-photoalbum-example
    cd gke-photoalbum-example
    

Create a Cloud Storage bucket and upload the default thumbnail image

  1. In Cloud Shell, create a Cloud Storage bucket to store the original images and thumbnails:

    export PROJECT_ID=$(gcloud config get-value project)
    gsutil mb -c regional -l ${REGION} gs://${PROJECT_ID}-photostore
    
  2. Upload the default thumbnail file:

    gsutil cp ./application/photoalbum/images/default.png \
        gs://${PROJECT_ID}-photostore/thumbnails/default.png
    
    • The uploaded images are stored in the following format: gs://PROJECT_ID-photostore/FILENAME, where FILENAME represents the name of the image file that is uploaded.
    • The generated thumbnails are stored in the following format: gs://PROJECT_ID-photostore/thumbnails/FILENAME.
    • The original image and the corresponding thumbnail have the same FILENAME value, but the thumbnail is stored in the thumbnails bucket.
    • While the thumbnail is being created, the following default.png placeholder thumbnail image is displayed in the photo album application.

      The default placeholder thumbnail image.

  3. Make the thumbnail file public:

    gsutil acl ch -u AllUsers:R \
        gs://${PROJECT_ID}-photostore/thumbnails/default.png
    

Create a Cloud SQL instance and a MySQL database

  1. In Cloud Shell, create the Cloud SQL instance:

    gcloud sql instances create photoalbum-db --region=${REGION} \
        --database-version=MYSQL_8_0
    
  2. Retrieve the connection name:

    gcloud sql instances describe photoalbum-db \
        --format="value(connectionName)"
    

    Make a note of the name because you'll use it later.

  3. Set the password for the root@% MySQL user:

    gcloud sql users set-password root --host=% --instance=photoalbum-db \
        --password=PASSWORD
    

    Replace PASSWORD with a secure password for the root@%user.

  4. Connect to the Cloud SQL instance:

    gcloud sql connect photoalbum-db --user=root --quiet
    

    When you're prompted, enter the password that you set up in the preceding step.

  5. Create a database called photo_db, where the user is appuser and the password is pas4appuser:

    create database photo_db;
    create user 'appuser'@'%' identified by 'pas4appuser';
    grant all on photo_db.* to 'appuser'@'%' with grant option;
    flush privileges;
    
  6. Confirm the result and exit from MySQL:

    show databases;
    select user from mysql.user;
    exit
    

    In the output, confirm that the photo_db database and the appuser user are created:

    mysql> show databases;
    +--------------------+
    | Database           |
    +--------------------+
    | information_schema |
    | mysql              |
    | performance_schema |
    | photo_db           |
    | sys                |
    +--------------------+
    5 rows in set (0.16 sec)
    
    mysql> \t
    Outfile disabled.
    mysql> select user from mysql.user;
    +-------------------+
    | user              |
    +-------------------+
    | appuser           |
    | cloudsqlreplica   |
    | cloudsqlsuperuser |
    | root              |
    | cloudsqlexport    |
    | cloudsqlimport    |
    | cloudsqloneshot   |
    | root              |
    | cloudsqlexport    |
    | cloudsqlimport    |
    | cloudsqloneshot   |
    | root              |
    | cloudsqlapplier   |
    | cloudsqlimport    |
    | mysql.infoschema  |
    | mysql.session     |
    | mysql.sys         |
    | root              |
    +-------------------+
    18 rows in set (0.16 sec)
    
    mysql> exit
    Bye
    

Create a Pub/Sub topic and a subscription

  1. In Cloud Shell, create a Pub/Sub topic called thumbnail-service:

    gcloud pubsub topics create thumbnail-service
    

    The photo album application sends requests to the thumbnail generation service by publishing a message on the thumbnail-service topic.

  2. Create a Pub/Sub subscription called thumbnail-workers:

    gcloud pubsub subscriptions create --topic thumbnail-service thumbnail-workers
    

    The thumbnail generation service receives requests from the thumbnail-workers subscription.

Create a GKE cluster

  1. In Cloud Shell, create a GKE cluster that has permission to call APIs:

    gcloud container clusters create "photoalbum-cluster" \
        --scopes "https://www.googleapis.com/auth/cloud-platform" \
        --num-nodes "5"
    
  2. Get access credentials configured so that you can manage the cluster using the kubectl command in later steps:

    gcloud container clusters get-credentials photoalbum-cluster
    
  3. Show the list of nodes:

    kubectl get nodes
    

    In the output, confirm that there are five nodes whose STATUS value is Ready:

    NAME                                                STATUS   ROLES    AGE     VERSION
    gke-photoalbum-cluster-default-pool-d637570a-2pfh   Ready    <none>   2m55s   v1.24.10-gke.2300
    gke-photoalbum-cluster-default-pool-d637570a-3rm4   Ready    <none>   2m55s   v1.24.10-gke.2300
    gke-photoalbum-cluster-default-pool-d637570a-f7l4   Ready    <none>   2m55s   v1.24.10-gke.2300
    gke-photoalbum-cluster-default-pool-d637570a-qb2z   Ready    <none>   2m53s   v1.24.10-gke.2300
    gke-photoalbum-cluster-default-pool-d637570a-rvnp   Ready    <none>   2m54s   v1.24.10-gke.2300
    

Create the Artifact Registry repository

  • In Cloud Shell, create a repository to store container images:

    gcloud artifacts repositories create photoalbum-repo \
        --repository-format=docker \
        --location=us-central1 \
        --description="Docker repository"
    

Build images for the application

  1. In a text editor, open the application/photoalbum/src/auth_decorator.py file and update the username and password:

    USERNAME = 'username'
    PASSWORD = 'passw0rd'
    
  2. In Cloud Shell, build an image for the photo album application by using the Cloud Build service:

    gcloud builds submit ./application/photoalbum -t \
        us-central1-docker.pkg.dev/${PROJECT_ID}/photoalbum-repo/photoalbum-app
    
  3. Build an image for the thumbnail-worker thumbnail generation service by using the Cloud Build service:

    gcloud builds submit ./application/thumbnail -t \
        us-central1-docker.pkg.dev/${PROJECT_ID}/photoalbum-repo/thumbnail-worker
    

Deploy the photo album application

  1. In Cloud Shell, update the Kubernetes Deployment manifests for the photo album and the thumbnail generator with values from your environment:

    connection_name=$(gcloud sql instances describe photoalbum-db \
        --format "value(connectionName)")
    
    digest_photoalbum=$(gcloud container images describe \
        us-central1-docker.pkg.dev/${PROJECT_ID}/photoalbum-repo/photoalbum-app:latest --format \
        "value(image_summary.digest)")
    
    sed -i.bak -e "s/\[PROJECT_ID\]/${PROJECT_ID}/" \
        -e "s/\[CONNECTION_NAME\]/${connection_name}/" \
        -e "s/\[DIGEST\]/${digest_photoalbum}/" \
        config/photoalbum-deployment.yaml
    
    digest_thumbnail=$(gcloud container images describe \
        us-central1-docker.pkg.dev/${PROJECT_ID}/photoalbum-repo/thumbnail-worker:latest --format \
        "value(image_summary.digest)")
    
    sed -i.bak -e "s/\[PROJECT_ID\]/${PROJECT_ID}/" \
        -e "s/\[CONNECTION_NAME\]/${connection_name}/" \
        -e "s/\[DIGEST\]/${digest_thumbnail}/" \
            config/thumbnail-deployment.yaml
    
  2. Create deployment resources to launch the photo album application and the thumbnail generation service:

    kubectl create -f config/photoalbum-deployment.yaml
    kubectl create -f config/thumbnail-deployment.yaml
    
  3. Create a service resource to assign an external IP address to the application:

    kubectl create -f config/photoalbum-service.yaml
    
  4. Check the results for the pods:

    kubectl get pods
    

    In the output, confirm that there are three pods for each photoalbum-app and thumbail-worker pod and that their STATUS value is Running:

    NAME                                READY     STATUS    RESTARTS   AGE
    photoalbum-app-555f7cbdb7-cp8nw     2/2       Running   0          2m
    photoalbum-app-555f7cbdb7-ftlc6     2/2       Running   0          2m
    photoalbum-app-555f7cbdb7-xsr4b     2/2       Running   0          2m
    thumbnail-worker-86bd95cd68-728k5   2/2       Running   0          2m
    thumbnail-worker-86bd95cd68-hqxqr   2/2       Running   0          2m
    thumbnail-worker-86bd95cd68-xnxhc   2/2       Running   0          2m
    

    The thumbnail-worker pods subscribe thumbnail generation requests from the thumbnail-workers subscription. For more information, see how the callback function is used in the source code.

  5. Check the results for the services:

    kubectl get services
    

    In the output, confirm that there is an external IP address in the EXTERNAL-IP column for the photoalbum-service service. It might take a few minutes until the services are all running.

    NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)        AGE
    kubernetes           ClusterIP      10.23.240.1     <none>            443/TCP        20m
    photoalbum-service   LoadBalancer   10.23.253.241   146.148.111.115   80:32657/TCP   2m
    

    Make a note of the external IP address because it's used later. In this example, it's 146.148.111.115.

Test the photo album application

  1. To access the deployed application in a web browser, go to the following URL and enter the username and password that you previously set up:

    http://EXTERNAL_IP
    

    Replace EXTERNAL_IP with the IP address that you copied in the previous step.

  2. To upload an image file, click Upload.

    The placeholder thumbnail that's displayed while you wait for the service to generate a unique thumbnail.

    The thumbnail placeholder appears on the screen.

    In the background, the thumbnail generation service creates a thumbnail of the uploaded image. To see the generated thumbnail, click Refresh. The Cloud Vision API adds image labels that it detects.

    The thumbnail with associated image labels.

    To see the original image, click the thumbnail.

Add an inappropriate-image detection feature

The following image illustrates how you can use Pub/Sub notifications for Cloud Storage to trigger a service that detects inappropriate content. This feature blurs the image when a new file with inappropriate content is stored in the Cloud Storage bucket.

Architecture of the inappropriate-content feature.

In the preceding image, the service uses the Safe Search Detection feature from Vision API to detect inappropriate content within an image.

The photo application triggers the thumbnail generator and the image checker asynchronously. As a result, there is no guarantee that they will be executed in a specific order. If the thumbnail generation occurs before the image is blurred, you might see an inappropriate thumbnail for a short while. However, the image checker eventually blurs both the inappropriate image and the inappropriate thumbnail.

Create a Pub/Sub topic, subscription, and notification

  1. In Cloud Shell, create a Pub/Sub topic called safeimage-service:

    gcloud pubsub topics create safeimage-service
    
  2. Create a Pub/Sub subscription called safeimage-workers:

    gcloud pubsub subscriptions create --topic safeimage-service \
        safeimage-workers
    
  3. Configure a Pub/Sub notification so that a message is sent to the safeimage-service topic when a new file is uploaded to the Cloud Storage bucket:

    gsutil notification create -t safeimage-service -f json \
        gs://${PROJECT_ID}-photostore
    

Build and deploy the worker image

  1. In Cloud Shell, build a container image for the safeimage-workers subscription by using Cloud Build:

    gcloud builds submit ./application/safeimage \
        -t us-central1-docker.pkg.dev/${PROJECT_ID}/photoalbum-repo/safeimage-worker
    
  2. Update the Kubernetes Deployment manifests for the safe-image service with your Google Cloud project ID, Cloud SQL connection name, and container image digests:

    digest_safeimage=$(gcloud container im