Troubleshoot issues

This page shows you how to resolve issues that you might encounter when using Eventarc Advanced.

Permission denied while using Eventarc service agent

If you encounter the following error while trying to create an Eventarc Advanced resource, wait a few minutes (potentially, seven), and then try creating the resource again:

Permission denied while using the Eventarc Service Agent. If you recently started to use
Eventarc, it may take a few minutes before all necessary permissions are propagated to the
Service Agent. Otherwise, verify that it has Eventarc Service Agent role.

A service agent acts as the identity of a given Google Cloud service for a particular project. For more information, see Service agents and view the Identity and Access Management (IAM) permissions for the Eventarc Service Agent role (roles/eventarc.serviceAgent).

If you are still running into the previous error after attempting to create your resource again, complete the following steps to verify that the Eventarc service agent exists in your Google Cloud project and has the necessary role:

  1. In the Google Cloud console, go to the IAM page.

    Go to IAM

  2. On the View by principals tab, select the Include Google-provided role grants checkbox.

  3. In the list of principals, locate the Eventarc service agent, which uses the following format:

    service-PROJECT_NUMBER@gcp-sa-eventarc.iam.gserviceaccount.com

  4. Verify that the service agent has the Eventarc Service Agent role. If the service agent doesn't have the role, then grant the role.

HTTP 503 Service Unavailable errors

If you encounter an HTTP 503 Service Unavailable error for a pipeline that routes messages to a Google destination using a DNS address—for example, Cloud Run—make sure that Private Google Access is enabled on the subnet used in the network attachment; otherwise, the DNS address can't be resolved.

CMEK issues

You can use customer-managed encryption keys (CMEK) to protect Eventarc. The keys are created and managed through Cloud Key Management Service (Cloud KMS). The following table describes different CMEK issues and how to resolve them when using Cloud KMS with Eventarc.

Issues that occur when creating or updating Eventarc resources

CMEK issue Error message Description
Disabled key $KEY is not enabled, current state is: DISABLED

The provided Cloud KMS key has been disabled for an Eventarc resource. Events or messages associated with the resource are no longer protected.

Solution:

  1. Display the key used for a resource:
  2. Re-enable the Cloud KMS key.
Exceeded quota Quota exceeded for limit

Your quota limit for Cloud KMS requests has been reached.

Solution:

  • Limit the number of Cloud KMS calls.
  • Increase the quota.
For more information, see Monitor and adjust Cloud KMS quotas.
Mismatched region Key region $REGION must match the resource to be protected

The provided KMS key region is different from the region of the channel.

Solution:

Use a Cloud KMS key from the same region. Note that for channels in multi-region eu, you should protect it using a Cloud KMS key in multi-region europe. For more information, see Cloud KMS locations and Eventarc multi-region locations.

Organization policy constraint project/PROJECT_ID violated org policy constraint

Eventarc is integrated with the following two organization policy constraints to help ensure CMEK usage across an organization. Any existing Eventarc resource isn't subject to a policy that is set after the resource is created; however, updating the resource might fail.

  • constraints/gcp.restrictNonCmekServices causes all resource creation requests without a specified Cloud KMS key to fail.

    Solution:

    Specify a Cloud KMS key for the Eventarc resource. For more information, see Require CMEKs for new Eventarc resources.

  • constraints/gcp.restrictCmekCryptoKeyProjects restricts the Cloud KMS keys that you can use to protect an Eventarc resource.

    Solution:

    Use a supported Cloud KMS key from an allowed Eventarc project, folder, or organization. For more information, see Restrict Cloud KMS keys for an Eventarc project.

Issues that occur during event delivery

CMEK issue Error message Description
Disabled key $KEY is not enabled, current state is: DISABLED

The provided Cloud KMS key has been disabled for an Eventarc resource. Events or messages associated with the resource are no longer protected.

Solution:

  1. Display the key used for a resource:
  2. Re-enable the Cloud KMS key.
Exceeded quota Quota exceeded for limit

Your quota limit for Cloud KMS requests has been reached.

Solution:

  • Limit the number of Cloud KMS calls.
  • Increase the quota.
For more information, see Monitor and adjust Cloud KMS quotas.
Permission error Permission 'cloudkms.cryptoKeyVersions.useToEncrypt' denied on resource $KEY (or it may not exist)

Either the provided Cloud KMS key doesn't exist or the Identity and Access Management (IAM) permission is not properly configured.

Solution:

To resolve issues that you might encounter when using externally managed keys through Cloud External Key Manager (Cloud EKM), see Cloud EKM error reference.

What's next