Setting up a Pub/Sub proxy for mobile clients on GKE


This tutorial shows you how to publish messages from mobile or client-side apps to Pub/Sub by using a proxy that handles authentication and authorization logic instead of client-side credentials.

While you can authenticate messages from the client to Pub/Sub by using Identity and Access Management (IAM), such long-lived credentials never expire. In client-side apps, these credentials can be discovered through techniques like app decompiling and reverse engineering.

Instead, you can offload authentication and authorization logic to a proxy that performs the following tasks:

  • Authenticates incoming requests to validate the user.
  • Forwards requests to Pub/Sub along with appropriate IAM permissions.

This tutorial shows how to implement a Pub/Sub proxy on Google Kubernetes Engine (GKE). This tutorial is intended for application developers and system architects who define and implement the design for mobile or client-side applications. It assumes you understand fundamental Kubernetes concepts and are familiar with Cloud Endpoints.

Request flow for this tutorial

To understand how Pub/Sub fits into a streaming pipeline, consider a clickstream analysis. In this use case, you might want to understand how users interact with your mobile app. To gain these insights, you capture user activity in real time. The following diagram shows the flow of data.

Pub/Sub proxy receives messages from client before data is aggregated.

The data captured by the app is pushed to Pub/Sub through a proxy. Pub/Sub can have subscribers downstream, such as Dataflow or Dataproc, which aggregate the data so that you can perform meaningful analysis.

The following diagram shows a detailed view of the request flow that this tutorial follows.

How pipeline components interact in a user request.

The next sections explain how the various components in this diagram interact.

User authentication

Mobile apps can use various methods to authenticate users. The authentication flow is specific to your app. This tutorial shows one such solution for authenticating users. An implementation of this solution accompanies this tutorial.

Requests from the client app to the Pub/Sub proxy

The app backend generates a short-lived authentication token that the client stores locally (for example, by using the Android Keystore system or iOS keychain services). This tutorial uses OpenID Connect (OIDC) ID tokens to authenticate the client app. Google issues and signs the OIDC ID token.

The client-side app sends a request to the Pub/Sub proxy by using the OIDC ID token. The Pub/Sub proxy validates the token and forwards the request to Pub/Sub along with the appropriate IAM credentials.

Publishing messages

After the client app is successfully authenticated, the Pub/Sub proxy sends a publish request to Pub/Sub. Using IAM, Pub/Sub helps ensure that the caller (the Pub/Sub proxy) has the right permissions to send publish requests. In this tutorial, the Pub/Sub proxy uses the Compute Engine default service account to authenticate with Pub/Sub. The Compute Engine default service account has the Editor IAM role (roles/editor), which provides publisher access to the Pub/Sub proxy.

Objectives

  • Create a GKE cluster to run a Pub/Sub proxy.
  • Create a Pub/Sub topic.
  • Deploy the Pub/Sub proxy.
  • Configure Endpoints to authenticate requests to the Pub/Sub proxy.
  • Verify that messages are published to Pub/Sub.

Costs

In this document, you use the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

  1. In the Google Cloud console, go to the project selector page.

    Go to project selector

  2. Select or create a Google Cloud project.

  3. Make sure that billing is enabled for your Google Cloud project.

  4. In the Google Cloud console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

  5. Define the environment variables that you need for this tutorial:
        export PROJECT=$(gcloud config get-value project)
        export REGION=us-central1
        export ZONE=${REGION}-b
        export CLUSTER=pubsub-proxy
        export TOPIC=proxy-test
        export SERVICE_ACCOUNT=publish-test
        export ENDPOINTS_SERVICE="pubtest.endpoints.${PROJECT}.cloud.goog"
        export GENERATE_TOKEN="https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts"
  6. Enable the APIs for Cloud Build, Compute Engine, Google Kubernetes Engine, Artifact Analysis, Container Registry, Endpoints, Service Management, Service Control, and Pub/Sub:
        gcloud services enable \
            cloudbuild.googleapis.com \
            compute.googleapis.com \
            container.googleapis.com \
            containeranalysis.googleapis.com \
            containerregistry.googleapis.com \
            endpoints.googleapis.com \
            servicemanagement.googleapis.com \
            servicecontrol.googleapis.com \
            pubsub.googleapis.com

Creating a Pub/Sub topic

  • In Cloud Shell, create a Pub/Sub topic that you publish messages to:

    gcloud pubsub topics create $TOPIC
    

Creating a GKE cluster

  1. In Cloud Shell, create a GKE cluster:

    gcloud container clusters create $CLUSTER \
        --zone $ZONE \
        --scopes "https://www.googleapis.com/auth/cloud-platform"
    
  2. Get credentials for the running cluster:

    gcloud container clusters get-credentials $CLUSTER \
        --zone $ZONE \
        --project $PROJECT
    

Building a container image

  1. In Cloud Shell, clone the code repository:

    git clone https://github.com/GoogleCloudPlatform/solutions-pubsub-proxy-rest
    
  2. Use Cloud Build to build a container image from source, and then save it in Container Registry:

    cd solutions-pubsub-proxy-rest && \
        gcloud builds submit --tag gcr.io/$PROJECT/pubsub-proxy:v1
    

Creating a static external IP address

  1. In Cloud Shell, create a static external IP address that is later assigned to the Pub/Sub proxy load balancer:

    gcloud compute addresses create service-ip --region $REGION
    
  2. Store the static IP address in an environment variable, PROXY_IP:

    PROXY_IP=$(gcloud compute addresses describe service-ip \
        --region $REGION --format='value(address)')
    

Deploying Endpoints

The Pub/Sub proxy uses Endpoints to authenticate requests from users. Endpoints uses the Extensible Service Proxy (ESP) to provide API management features such as authentication, monitoring, tracing, and API lifecycle management. This tutorial uses Endpoints only for authenticating incoming requests to the Pub/Sub proxy.

In this tutorial, you deploy ESP as a sidecar with the Pub/Sub proxy. ESP intercepts and authenticates incoming requests before it forwards them to the Pub/Sub proxy.

  1. In Cloud Shell, replace the [PROJECT_ID] placeholder with your Google Cloud project ID in the openapi.yaml file:

    sed -i -e "s/\[PROJECT_ID\]/$PROJECT/g" openapi.yaml
    
  2. In the OpenAPI manifest file, replace the [IP_ADDRESS] placeholder with the value of PROXY_IP:

    sed -i -e "s/\[IP_ADDRESS\]/$PROXY_IP/g" openapi.yaml
    
  3. Deploy the OpenAPI service definition to Endpoints:

    gcloud endpoints services deploy openapi.yaml
    

    The preceding command creates the following:

    • A managed service with the name that you have specified in the host field of the openapi.yaml file (pubtest.endpoints.project-id.cloud.goog), where project-id is the ID of your Google Cloud project.
    • A DNS A record that uses the service name and the Pub/Sub proxy load balancer IP address mapping that is defined in the x-google-endpoints extension in the openapi.yaml file.

    During deployment, you see a warning that you can ignore because this tutorial uses OIDC ID tokens for authentication instead of API keys.

    WARNING: openapi.yaml: Operation 'post' in path '/publish': Operation does
    not require an API key; callers may invoke the method without specifying an
    associated API-consuming project. To enable API key all the
    SecurityRequirement Objects
    (https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md#security-requirement-object)
    inside security definition must reference at least one SecurityDefinition
    of type : 'apiKey'.
    
  4. Check to see if the service is deployed correctly:

    gcloud endpoints services describe ${ENDPOINTS_SERVICE}
    

    The output is similar to the following:

    [...]
    producerProjectId: project-id
    serviceConfig:
      documentation:
        summary: Pub/Sub proxy exposed as an Endpoint API
    [...]
      name: pubtest.endpoints.project-id.cloud.goog
      title: PubSub Proxy
      usage: {}
    serviceName: pubtest.endpoints.project-id.cloud.goog
    

    In the output:

    • project-id: The ID of your Google Cloud project.

Deploying a proxy

  1. In Cloud Shell, generate a self-signed SSL certificate to allow HTTPS connections to the proxy.

    openssl req -x509 -nodes -days 365 \
        -newkey rsa:2048 -keyout ./nginx.key \
        -out ./nginx.crt \
        -subj "/CN=${ENDPOINTS_SERVICE}"
    
  2. Create a Kubernetes secret by using the SSL certificate and private key:

    kubectl create secret generic nginx-ssl \
        --from-file=./nginx.crt \
        --from-file=./nginx.key
    
  3. Replace the [PROJECT_ID] placeholder in the deployment manifest file with your Google Cloud project ID:

    sed -i -e "s/\[PROJECT_ID\]/$PROJECT/g" kube/deployment.yaml
    
  4. Replace the [IP_ADDRESS] placeholder in the service manifest file with the value of PROXY_IP:

    sed -i -e "s/\[IP_ADDRESS\]/$PROXY_IP/g" kube/service.yaml
    
  5. Deploy the proxy:

    kubectl apply -f kube/
    
  6. Verify that the deployment is successful:

    kubectl rollout status deployment/pubsub-proxy
    

    The output is similar to the following:

    [...]
    deployment "pubsub-proxy" successfully rolled out
    
  7. Ensure that two containers (ESP and Pub/Sub proxy) are running in the Pod:

    kubectl get pods $(kubectl get pod \
        -l app=pubsub-proxy \
        -o jsonpath="{.items[0].metadata.name}") \
        -o jsonpath={.spec.containers[*].name}
    

    The output is similar to the following:

    esp  pubsub-proxy
    
  8. Watch for the value of EXTERNAL-IP to change from <pending> to the static external IP address that you created earlier:

    kubectl get svc pubsub-proxy -w
    

    The output is similar to the following:

    NAME          TYPE          CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
    pubsub-proxy  LoadBalancer  10.7.247.212  <pending>     443:31104/TCP  6m32s
    pubsub-proxy  LoadBalancer  10.7.247.212  <PROXY_IP>    443:31104/TCP  6m5s
    

    To stop watching, press CTRL+C.

    After the Pub/Sub proxy is successfully deployed, it is exposed at https://${ENDPOINTS_SERVICE}/publish. It might take a few minutes for the new DNS configuration to propagate.

  9. Verify the DNS configuration:

    watch nslookup ${ENDPOINTS_SERVICE}
    

    The output is similar to the following:

    Server:   169.254.169.254
    Address:  169.254.169.254#53
    
    Non-authoritative answer:
    Name: pubtest.endpoints.project-id.cloud.goog
    Address: gke-load-balancer-ip
    

    In the output:

    • gke-load-balancer-ip: The IP address of your GKE load balancer (proxy IP).

    To stop watching, press CTRL+C.

If any of the preceding steps results in an error, see the troubleshooting steps.

Generating an authentication token

The following procedure for generating an authentication token is intended as an example. For your production environment, you need a way for users to generate their own authentication tokens. For example, you can find sample code for obtaining an OIDC ID token programmatically in the Identity-Aware Proxy documentation.

To generate an authentication token, do the following:

  1. Create a Google Cloud service account for which you generate an OIDC ID token:

    gcloud iam service-accounts create \
        $SERVICE_ACCOUNT \
        --display-name $SERVICE_ACCOUNT
    
  2. Get the email identity of the service account:

    SA_EMAIL=${SERVICE_ACCOUNT}@${PROJECT}.iam.gserviceaccount.com
    
  3. Grant the Service Account Token Creator IAM role (roles/iam.serviceAccountTokenCreator) for the service account:

    gcloud iam service-accounts add-iam-policy-binding $SA_EMAIL \
        --member user:$(gcloud config get-value account) \
        --role roles/iam.serviceAccountTokenCreator
    
  4. Using the IAM credentials API, generate an OIDC ID token.

    TOKEN=$(curl -s ${GENERATE_TOKEN}/${SA_EMAIL}:generateIdToken \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
        -d '{"audience": "'${ENDPOINTS_SERVICE}'", "includeEmail": "true"}' | jq -r ".token")
    

    The Endpoints service name is specified in the audience field. The audience claim identifies the recipient that the token is intended for.

  5. Verify that the token is successfully created:

    echo $TOKEN
    

    The JSON Web Token (JWT) is similar to the following:

    eyJhbGciOiJSUzI1NiIsImtpZCI6IjY4NjQyODlm[...].eyJhdWQiOiJwdWJ0ZXN0LmVuZHBvaW50cy52aXR
    hbC1vY3RhZ29uLTEwOTYxMi5jbG91ZC5nb[...].SjBI4TZjZAlYo6lFKkrvfAcVUp_AJzFKoSsjNbmD_n[...]
    

Calling Pub/Sub by using a proxy

  1. In Cloud Shell, publish a test message:

    curl -i -k -X POST https://${ENDPOINTS_SERVICE}/publish \
        -H "Authorization: Bearer $TOKEN" \
        -H "Content-Type: application/json" \
        -d '{"topic": "'$TOPIC'", "messages": [ {"attributes": {"key1": "value1", "key2" : "value2"}, "data": "test data"}]}'
    

    The output is similar to the following:

    HTTP/2 200
    server: nginx
    date: Sun, 02 Jun 2019 03:53:46 GMT
    ...
    
  2. Check whether the message successfully published to the Pub/Sub topic:

    kubectl logs -f --tail=5 deployment/pubsub-proxy -c pubsub-proxy
    

    The Pub/Sub proxy deployment logs display the message Successfully published:

    2019-06-02 03:49:39.723:INFO:oejs.Server:main: Started @2554ms
    Jun 02, 2019 3:53:44 AM com.google.pubsub.proxy.publish.PublishMessage
    getPublisher
    INFO: Creating new publisher for: proxy-test
    Jun 02, 2019 3:53:47 AM
    com.google.pubsub.proxy.publish.PublishMessage$1 onSuccess
    INFO: Successfully published: 569006136173844
    

Troubleshooting

  1. In Cloud Shell, check the state of both containers in the Pub/Sub proxy Pod:

    kubectl describe pods $(kubectl get pod -l app=pubsub-proxy \
        -o jsonpath="{.items[0].metadata.name}")
    

    In the log output, the status of the containers is Running:

    [...]
    Containers:
      esp:
    [...]
      State:  Running
        Started:  Fri, 21 Jun 2019 16:41:30 +0530
      Ready:  True
      Restart Count:  0
    [...]
      pubsub-proxy:
        State:  Running
          Start