This tutorial teaches you how to troubleshoot runtime errors encountered when you use Eventarc to route events from Cloud Storage to an unauthenticated Cloud Run service using Cloud Audit Logs.
Objectives
This tutorial shows you how to complete the following tasks:
- Create an Artifact Registry standard repository to store your container image.
- Create a Cloud Storage bucket to be the event source.
- Build, upload, and deploy a container image to Cloud Run.
- Create Eventarc triggers.
- Upload a file to the Cloud Storage bucket.
- Troubleshoot and fix the runtime errors.
Costs
In this document, you use the following billable components of Google Cloud:
To generate a cost estimate based on your projected usage,
use the pricing calculator.
Before you begin
Security constraints defined by your organization might prevent you from completing the following steps. For troubleshooting information, see Develop applications in a constrained Google Cloud environment.
- Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
- Install the Google Cloud CLI.
-
To initialize the gcloud CLI, run the following command:
gcloud init
-
Create or select a Google Cloud project.
-
Create a Google Cloud project:
gcloud projects create PROJECT_ID
Replace
PROJECT_ID
with a name for the Google Cloud project you are creating. -
Select the Google Cloud project that you created:
gcloud config set project PROJECT_ID
Replace
PROJECT_ID
with your Google Cloud project name.
-
-
Make sure that billing is enabled for your Google Cloud project.
-
Enable the Artifact Registry, Cloud Build, Cloud Logging, Cloud Run, Cloud Storage, Eventarc, and Pub/Sub APIs:
gcloud services enable artifactregistry.googleapis.com
cloudbuild.googleapis.com eventarc.googleapis.com logging.googleapis.com pubsub.googleapis.com run.googleapis.com storage.googleapis.com - Install the Google Cloud CLI.
-
To initialize the gcloud CLI, run the following command:
gcloud init
-
Create or select a Google Cloud project.
-
Create a Google Cloud project:
gcloud projects create PROJECT_ID
Replace
PROJECT_ID
with a name for the Google Cloud project you are creating. -
Select the Google Cloud project that you created:
gcloud config set project PROJECT_ID
Replace
PROJECT_ID
with your Google Cloud project name.
-
-
Make sure that billing is enabled for your Google Cloud project.
-
Enable the Artifact Registry, Cloud Build, Cloud Logging, Cloud Run, Cloud Storage, Eventarc, and Pub/Sub APIs:
gcloud services enable artifactregistry.googleapis.com
cloudbuild.googleapis.com eventarc.googleapis.com logging.googleapis.com pubsub.googleapis.com run.googleapis.com storage.googleapis.com -
If you are the project creator, you are granted the basic Owner role (
roles/owner
). By default, this Identity and Access Management (IAM) role includes the permissions necessary for full access to most Google Cloud resources and you can skip this step.If you are not the project creator, required permissions must be granted on the project to the appropriate principal. For example, a principal can be a Google Account (for end users) or a service account (for applications and compute workloads). For more information, see the Roles and permissions page for your event destination.
Note that by default, Cloud Build permissions include permissions to upload and download Artifact Registry artifacts.
Required permissions
To get the permissions that you need to complete this tutorial, ask your administrator to grant you the following IAM roles on your project:
-
Cloud Build Editor (
roles/cloudbuild.builds.editor
) -
Cloud Run Admin (
roles/run.admin
) -
Eventarc Admin (
roles/eventarc.admin
) -
Logs View Accessor (
roles/logging.viewAccessor
) -
Project IAM Admin (
roles/resourcemanager.projectIamAdmin
) -
Service Account Admin (
roles/iam.serviceAccountAdmin
) -
Service Account User (
roles/iam.serviceAccountUser
) -
Service Usage Admin (
roles/serviceusage.serviceUsageAdmin
) -
Storage Admin (
roles/storage.admin
)
For more information about granting roles, see Manage access to projects, folders, and organizations.
You might also be able to get the required permissions through custom roles or other predefined roles.
-
Cloud Build Editor (
- For Cloud Storage, enable audit logging for the
ADMIN_READ
,DATA_WRITE
, andDATA_READ
data access types.- Read the Identity and Access Management (IAM) policy associated with your
Google Cloud project, folder, or organization and store it in a temporary file:
gcloud projects get-iam-policy PROJECT_ID > /tmp/policy.yaml
- In a text editor, open
/tmp/policy.yaml
, and add or change only the audit log configuration in theauditConfigs
section:auditConfigs: - auditLogConfigs: - logType: ADMIN_READ - logType: DATA_WRITE - logType: DATA_READ service: storage.googleapis.com bindings: - members: [...] etag: BwW_bHKTV5U= version: 1
- Write your new IAM policy:
gcloud projects set-iam-policy PROJECT_ID /tmp/policy.yaml
If the preceding command reports a conflict with another change, then repeat these steps, starting with reading the IAM policy. For more information, see Configure Data Access audit logs with the API.
- Read the Identity and Access Management (IAM) policy associated with your
Google Cloud project, folder, or organization and store it in a temporary file:
- Grant the
eventarc.eventReceiver
role to the Compute Engine service account:export PROJECT_NUMBER="$(gcloud projects describe $(gcloud config get-value project) --format='value(projectNumber)')" gcloud projects add-iam-policy-binding $(gcloud config get-value project) \ --member=serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com \ --role='roles/eventarc.eventReceiver'
- If you enabled the Pub/Sub service account on or before April 8,
2021, grant the
iam.serviceAccountTokenCreator
role to the Pub/Sub service account:gcloud projects add-iam-policy-binding $(gcloud config get-value project) \ --member="serviceAccount:service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com"\ --role='roles/iam.serviceAccountTokenCreator'
- Set the defaults used in this tutorial:
export REGION=us-central1 gcloud config set run/region ${REGION} gcloud config set run/platform managed gcloud config set eventarc/location ${REGION}
Create an Artifact Registry standard repository
Create an Artifact Registry standard repository to store your container image:
gcloud artifacts repositories create REPOSITORY \ --repository-format=docker \ --location=$REGION
Replace REPOSITORY
with a unique name for the
repository.
Create a Cloud Storage bucket
Create a Cloud Storage bucket in each of two regions as the event source for the Cloud Run service:
Create a bucket in
us-east1
:export BUCKET1="troubleshoot-bucket1-PROJECT_ID" gsutil mb -l us-east1 gs://${BUCKET1}
Create a bucket in
us-west1
:export BUCKET2="troubleshoot-bucket2-PROJECT_ID" gsutil mb -l us-west1 gs://${BUCKET2}
After the event source is created, deploy the event receiver service on Cloud Run.
Deploy the event receiver
Deploy a Cloud Run service that receives and logs events.
Retrieve the code sample by cloning the GitHub repository:
Go
git clone https://github.com/GoogleCloudPlatform/golang-samples.git cd golang-samples/eventarc/audit_storage
Java
git clone https://github.com/GoogleCloudPlatform/java-docs-samples.git cd java-docs-samples/eventarc/audit-storage
.NET
git clone https://github.com/GoogleCloudPlatform/dotnet-docs-samples.git cd dotnet-docs-samples/eventarc/audit-storage
Node.js
git clone https://github.com/GoogleCloudPlatform/nodejs-docs-samples.git cd nodejs-docs-samples/eventarc/audit-storage
Python
git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git cd python-docs-samples/eventarc/audit-storage
Review the code for this tutorial which consists of the following:
An event handler that receives the incoming event as a CloudEvent within the HTTP
POST
request:Go
Java
.NET
Node.js
Python
A server that uses the event handler:
Go
Java
.NET
Node.js
Python
A Dockerfile that defines the operating environment for the service. The contents of the Dockerfile vary by language:
Go
Java
.NET
Node.js
Python
Build your container image with Cloud Build and upload the image to Artifact Registry:
export PROJECT_ID=$(gcloud config get-value project) export SERVICE_NAME=troubleshoot-service gcloud builds submit --tag $REGION-docker.pkg.dev/${PROJECT_ID}/REPOSITORY/${SERVICE_NAME}:v1
Deploy the container image to Cloud Run:
gcloud run deploy ${SERVICE_NAME} \ --image $REGION-docker.pkg.dev/${PROJECT_ID}/REPOSITORY/${SERVICE_NAME}:v1 \ --allow-unauthenticated
When the deployment succeeds, the command line displays the service URL.
Create a trigger
After deploying a Cloud Run service, set up a trigger to listen for events from Cloud Storage through audit logs.
Create an Eventarc trigger to listen for Cloud Storage events that are routed using Cloud Audit Logs:
gcloud eventarc triggers create troubleshoot-trigger \ --destination-run-service=troubleshoot-service \ --event-filters="type=google.cloud.audit.log.v1.written" \ --event-filters="serviceName=storage.googleapis.com" \ --event-filters="methodName=storage.objects.create" \ --service-account=${PROJECT_NUMBER}-compute@developer.gserviceaccount.com
This creates a trigger called
troubleshoot-trigger
.To confirm
troubleshoot-trigger
has been created, run:gcloud eventarc triggers list
The output should be similar to the following:
NAME: troubleshoot-trigger TYPE: google.cloud.audit.log.v1.written DESTINATION: Cloud Run service: troubleshoot-service ACTIVE: By 20:03:37 LOCATION: us-central1
Generate and view an event
Confirm that you have successfully deployed the service and can receive events from Cloud Storage.
Create and upload a file to the
BUCKET1
storage bucket:echo "Hello World" > random.txt gsutil cp random.txt gs://${BUCKET1}/random.txt
Monitor the logs to check if the service received an event. To view the log entry, complete the following steps:
Filter the log entries and return the output in JSON format:
gcloud logging read "resource.labels.service_name=troubleshoot-service \ AND textPayload:random.txt" \ --format=json
Look for a log entry similar to:
"textPayload": "Detected change in Cloud Storage bucket: ..."
Note that, initially, no log entry is returned. This indicates that there is a problem in the setup that you must investigate.
Investigate the problem
Go through the process of investigating why the service is not receiving events.
Initialization time
Although your trigger is created immediately, it can take up to two minutes for a trigger to propagate and filter events. Run the following command to confirm that a trigger is active:
gcloud eventarc triggers list
The output indicates the status of the trigger. In the following example,
troubleshoot-trigger
will be active by 14:16:56:
NAME TYPE DESTINATION_RUN_SERVICE ACTIVE
troubleshoot-trigger google.cloud.audit.log.v1.written troubleshoot-service By 14:16:56
Once the trigger is active, upload a file again to the storage bucket. Events are written in the Cloud Run service logs. If the service does not receive events, it could be related to the size of events.
Audit logs
In this tutorial, Cloud Storage events are routed using Cloud Audit Logs and sent to Cloud Run. Confirm that the audit logs are enabled for Cloud Storage.
In the Google Cloud console, go to the Audit logs page.
- Select the Google Cloud Storage checkbox.
- Ensure that the Admin Read, Data Read, and Data Write log types are selected.
Once you have enabled Cloud Audit Logs, upload the file again to the storage bucket and check the logs. If the service still does not receive events, this could be related to the trigger location.
Trigger location
There could be multiple resources in different locations and you must filter for events from sources that are in the same region as the Cloud Run target. For more information, see the locations supported by Eventarc and Understand Eventarc locations.
In this tutorial, you deployed the Cloud Run service to
us-central1
. Because you set eventarc/location
to us-central1
, you also
created a trigger in the same location.
However, you created two Cloud Storage buckets in us-east1
and
us-west1
locations. To receive events from those locations, you must create
Eventarc triggers in those locations.
Create an Eventarc trigger located in us-east1
:
Confirm the location of the existing trigger:
gcloud eventarc triggers describe troubleshoot-trigger
Set the location and region to
us-east1
:gcloud config set eventarc/location us-east1 gcloud config set run/region us-east1
Deploy the event receiver again by building and deploying the container image to Cloud Run.
Create a new trigger located in
us-east1
:gcloud eventarc triggers create troubleshoot-trigger-new \ --destination-run-service=troubleshoot-service \ --event-filters="type=google.cloud.audit.log.v1.written" \ --event-filters="serviceName=storage.googleapis.com" \ --event-filters="methodName=storage.objects.create" \ --service-account=${PROJECT_NUMBER}-compute@developer.gserviceaccount.com
Check that the trigger is created:
gcloud eventarc triggers list
A trigger can take up to two minutes to initialize before it starts to route events.
To confirm that the trigger is now deployed correctly, generate and view an event.
Other issues you might encounter
You might encounter other issues when using Eventarc.
Event size
The events you send must not exceed the limits on event size.
A trigger that previously delivered events has stopped working
Verify that the source is generating events. Check the Cloud Audit Logs and make sure the monitored service is emitting logs. If logs are recorded but events are not delivered, contact support.
Verify that a Pub/Sub topic with the same trigger name exists. Eventarc uses Pub/Sub as its transport layer and will either use an existing Pub/Sub topic or will automatically create a topic and manage it for you.
- To list triggers, see
gcloud eventarc triggers list
. To list the Pub/Sub topics, run:
gcloud pubsub topics list
Verify that the Pub/Sub topic name includes the name of the created trigger. For example:
name: projects/PROJECT_ID/topics/eventarc-us-east1-troubleshoot-trigger-new-123
If the Pub/Sub topic is missing, create the trigger again for a specific provider, event type, and Cloud Run destination.
- To list triggers, see
Verify that the trigger has been configured for the service.
In the Google Cloud console, go to the Services page.
Click the name of the service to open its Service details page.
Click the Triggers tab.
The Eventarc trigger associated with the service should be listed.
Verify the health of the Pub/Sub topic and subscription using Pub/Sub metric types.
You can monitor forwarded undeliverable messages using the
subscription/dead_letter_message_count
metric. This metric shows the number of undeliverable messages that Pub/Sub forwards from a subscription.If messages are not published to the topic, check Cloud Audit Logs and make sure the monitored service is emitting logs. If logs are recorded but events are not delivered, contact support.
You can monitor push subscriptions using the
subscription/push_request_count
metric and grouping the metric byresponse_code
andsubcription_id
.If push errors are reported, check the Cloud Run service logs. If the receiving endpoint returns a non-OK status code, it indicates that the Cloud Run code is not working as expected and you must contact support.
For more information, see Create metric-threshold alerting policies.
Clean up
If you created a new project for this tutorial, delete the project. If you used an existing project and want to keep it without the changes added in this tutorial, delete the resources created for the tutorial.
Delete the project
The easiest way to eliminate billing is to delete the project that you created for the tutorial.
To delete the project:
- In the Google Cloud console, go to the Manage resources page.
- In the project list, select the project that you want to delete, and then click Delete.
- In the dialog, type the project ID, and then click Shut down to delete the project.
Delete tutorial resources
Delete the Cloud Run service you deployed in this tutorial:
gcloud run services delete SERVICE_NAME
Where
SERVICE_NAME
is your chosen service name.You can also delete Cloud Run services from the Google Cloud console.
Remove any gcloud CLI default configurations you added during the tutorial setup.
For example:
gcloud config unset run/region
or
gcloud config unset project
Delete other Google Cloud resources created in this tutorial:
- Delete the Eventarc trigger:
Replacegcloud eventarc triggers delete TRIGGER_NAME
TRIGGER_NAME
with the name of your trigger.
- Delete the Eventarc trigger: