Troubleshoot issues

This page shows you how to resolve issues that you might encounter when using Eventarc Advanced.

HTTP 503 Service Unavailable errors

If you encounter an HTTP 503 Service Unavailable error for a pipeline that routes messages to a Google destination using a DNS address—for example, Cloud Run, Pub/Sub, Workflows, or another Eventarc Advanced bus—make sure that Private Google Access is enabled on the subnet used in the network attachment; otherwise, the DNS address can't be resolved.

CMEK issues

You can use customer-managed encryption keys (CMEK) to protect Eventarc. The keys are created and managed through Cloud Key Management Service (Cloud KMS). The following table describes different CMEK issues and how to resolve them when using Cloud KMS with Eventarc.

Issues that occur when creating or updating Eventarc resources

CMEK issue Error message Description
Disabled key $KEY is not enabled, current state is: DISABLED

The provided Cloud KMS key has been disabled for an Eventarc resource. Events or messages associated with the resource are no longer protected.

Solution:

  1. Display the key used for a resource:
  2. Re-enable the Cloud KMS key.
Exceeded quota Quota exceeded for limit

Your quota limit for Cloud KMS requests has been reached.

Solution:

  • Limit the number of Cloud KMS calls.
  • Increase the quota.
For more information, see Monitor and adjust Cloud KMS quotas.
Mismatched region Key region $REGION must match the resource to be protected

The provided KMS key region is different from the region of the channel.

Solution:

Use a Cloud KMS key from the same region. Note that for channels in multi-region eu, you should protect it using a Cloud KMS key in multi-region europe. For more information, see Cloud KMS locations and Eventarc multi-region locations.

Organization policy constraint project/PROJECT_ID violated org policy constraint

Eventarc is integrated with the following two organization policy constraints to help ensure CMEK usage across an organization. Any existing Eventarc resource isn't subject to a policy that is set after the resource is created; however, updating the resource might fail.

  • constraints/gcp.restrictNonCmekServices causes all resource creation requests without a specified Cloud KMS key to fail.

    Solution:

    Specify a Cloud KMS key for the Eventarc resource. For more information, see Require CMEKs for new Eventarc resources.

  • constraints/gcp.restrictCmekCryptoKeyProjects restricts the Cloud KMS keys that you can use to protect an Eventarc resource.

    Solution:

    Use a supported Cloud KMS key from an allowed Eventarc project, folder, or organization. For more information, see Restrict Cloud KMS keys for an Eventarc project.

Issues that occur during event delivery

CMEK issue Error message Description
Disabled key $KEY is not enabled, current state is: DISABLED

The provided Cloud KMS key has been disabled for an Eventarc resource. Events or messages associated with the resource are no longer protected.

Solution:

  1. Display the key used for a resource:
  2. Re-enable the Cloud KMS key.
Exceeded quota Quota exceeded for limit

Your quota limit for Cloud KMS requests has been reached.

Solution:

  • Limit the number of Cloud KMS calls.
  • Increase the quota.
For more information, see Monitor and adjust Cloud KMS quotas.
Permission error Permission 'cloudkms.cryptoKeyVersions.useToEncrypt' denied on resource $KEY (or it may not exist)

Either the provided Cloud KMS key doesn't exist or the Identity and Access Management (IAM) permission is not properly configured.

Solution:

To resolve issues that you might encounter when using externally managed keys through Cloud External Key Manager (Cloud EKM), see Cloud EKM error reference.

What's next