An enrollment identifies a subscription to a particular bus. The enrollment defines the matching criteria that determines which messages are routed to a destination. It also specifies the pipeline that matched messages should be routed through. A pipeline lets you configure a target destination and also provides the option of transforming any matched events prior to delivering them to the destination.
Note the following:
- A pipeline and enrollment must be in the same Google Cloud project. (The bus can be in the same or a different project.)
- One pipeline can be used for multiple enrollments.
- Only one destination can be the target of messages routed by a pipeline.
- Before configuring either a pipeline or an enrollment, you should have already created an Eventarc Advanced bus.
Required roles
An Identity and Access Management (IAM) role contains a set of permissions that lets you perform specific actions on Google Cloud resources. The following roles and permissions are required when creating a pipeline and an enrollment to route messages:
- To get the permission that you need to create a pipeline, ask your
administrator to grant you the
Eventarc Developer
(
roles/eventarc.developer
) IAM role on your pipeline project. This predefined role contains theeventarc.pipelines.create
permission, which is required to create a pipeline. - To get the permission that you need to create an enrollment, ask your
administrator to grant you the
Eventarc Developer
(
roles/eventarc.developer
) IAM role on your enrollment project. This predefined role contains theeventarc.enrollments.create
permission, which is required to create an enrollment. - To get the permission that you need to use a bus, ask your
administrator to grant you the
Eventarc Message Bus User
(
roles/eventarc.messageBusUser
) IAM role on your bus project. This predefined role contains theeventarc.buses.use
permission, which is required to use a bus.
For more information about granting roles, see Manage access. You might also be able to get these permissions with custom roles or other predefined roles.
Enroll to receive events
You can create a pipeline and an enrollment in the Google Cloud console or by using the Google Cloud CLI.
Console
In the Google Cloud console, you can create both the pipeline and enrollment at the same time from the Pipelines page.
To create an enrollment, in the Google Cloud console, go to the Eventarc > Pipelines page.
Click
Create pipeline.In the Pipeline details pane, do the following:
- Enter a Pipeline name. This is the ID for your pipeline.
- In the Region list, select a region to deploy your pipeline in. The pipeline must be created in the same region as the bus. For more information, see Eventarc Advanced locations.
- Optional: You can modify the default configuration to retry events.
- For Encryption, accept the default Google-managed encryption key or select Cloud KMS key. For more information, see Use customer-managed encryption keys (CMEK).
If you select Cloud KMS key, do the following:
In the Key type list, select a method to manage your keys.
You can manage your keys manually or you can use Autokey which lets you generate key rings and keys on-demand. If the Autokey option is disabled, it isn't yet integrated with the current resource type.
In the Select a customer-managed key, select a key.
Note that you must select a region before you can view your customer-managed keys.
Optional: To manually enter the resource name of the key, in the Select a customer-managed key list, click Enter key manually, and enter the key name in the specified format.
If prompted, grant the
cloudkms.cryptoKeyEncrypterDecrypter
role to the Eventarc Service Agent.
Optional: To add labels, click What are labels?
Add label. Labels are key-value pairs that help you organize your Google Cloud resources. For more information, seeClick Continue.
In the Enrollments pane, do the following:
- Click Add an enrollment.
- Enter an Enrollment name.
- In the Bus list, select a bus to subscribe to.
In the CEL expression field, write an evaluation expression using CEL. For example:
message.type == "google.cloud.dataflow.job.v1beta3.statusChanged"
Note that the default expression, true, indicates that all messages are routed, unfiltered.
Click Done.
You can add another enrollment or click Continue.
Optional: In the Event mediation pane, do the following or click Continue:
- Select the Apply a transformation checkbox.
In the Inbound format list, select the applicable format.
Note that when applying a transformation, you must specify an inbound data format for a pipeline, and that all events must match that format.
For Avro or Protobuf formats, you must specify an inbound schema. Optionally, you can upload an inbound schema. For more information, see Format received events.
In the CEL expression field, write a transformation expression using CEL.
Click Continue.
In the Destination pane, do the following:
In the Destination type list, select a destination type to route messages to. Depending on the destination type, do the following:
HTTP endpoint: Specify the destination URI. The host can be either a static IP addressable from a Virtual Private Cloud (VPC) network, or the internal Domain Name System (DNS) hostname of a service resolvable using Cloud DNS.
Note that you can use this destination type to target a Cloud Run function.
Eventarc Advanced bus: Select an Eventarc Advanced bus.
Workflows workflow: Select a Workflows workflow.
Pub/Sub topic: Select or create a Pub/Sub topic.
Cloud Run service (via HTTP): Select a Cloud Run service that will receive events as HTTP
POST
requests sent to its root URL path (/
); optionally, you can specify a relative URL path on the destination service to which events should be sent.Cloud Run job (via HTTP): Select a Cloud Run job that will receive events as HTTP
POST
requests.
Specify a network attachment.
A network attachment is a resource that lets a producer VPC network initiate connections to a consumer VPC network. To publish events, Eventarc Advanced uses the network attachment to establish a connection to the endpoint hosted in a VPC network.
You can create a network attachment that automatically accepts connections from any Private Service Connect interface that refers to the network attachment. Create the network attachment in the same network and region containing the destination endpoint.
If you are routing messages to a Google destination using a DNS address—for example, Cloud Run, Pub/Sub, Workflows, or another Eventarc Advanced bus—make sure that Private Google Access is enabled on the subnet used in the network attachment; otherwise, the DNS address can't be resolved.
If applicable, in the Outbound format list, select a format.
Note that if an inbound data format is not specified for a pipeline, an outbound format can't be set.
If applicable, apply a Message binding. For more information, see Define a message binding.
To enable authentication, select the Enable authentication checkbox.
In the Auth header list, select the type of token to generate and attach as an
Authorization
header in the HTTP request:An OAuth token should generally be used only when calling Google APIs hosted on
*.googleapis.com
. Optionally, specify the Scope of this token; otherwise, it defaults tohttps://www.googleapis.com/auth/cloud-platform
. For Google Cloud services, it's a best practice to use thehttps://www.googleapis.com/auth/cloud-platform
scope, which includes all Google Cloud APIs, together with Identity and Access Management (IAM), which provides fine-grained access control.Note that all requests to another Eventarc Advanced bus, Pub/Sub, or Workflows must have an HTTP Authorization header containing an OAuth token signed by Google for one of the authorized service accounts.
An OIDC token can be used for many scenarios, including endpoints where you intend to validate the token yourself. Additionally, specify the Audience that this token is intended for. Generally, it should match the URL of the target pipeline. If not specified, the entire URL is used, including any request parameters.
Note that Cloud Run performs an IAM check on every request and you can use the
run.routes.invoke
permission to configure who can access your Cloud Run service in either of the following ways:Grant the permission to select service accounts or groups to allow access to the service. All requests must have an HTTP Authorization header containing an OpenID Connect token signed by Google for one of the authorized service accounts.
Grant the permission to allUsers to allow unauthenticated access.
For more information, see Access control for Cloud Run.
Learn more about token types.
In the Service account list, select the service account that will invoke your destination service. Or, you can create a new service account.
This specifies the IAM service account email associated with the pipeline and to which you previously granted specific roles required by Eventarc Advanced.
Click Create.
gcloud
When using the gcloud CLI, first create the pipeline, and then create the enrollment using the appropriate commands.
Pipeline
Open a terminal.
Create a pipeline using the
gcloud beta eventarc pipelines create
command:gcloud beta eventarc pipelines create PIPELINE_NAME \ --destinations=DESTINATION_KEY \ --location=REGION
Replace the following:
PIPELINE_NAME
: the ID of the pipeline or a fully qualified nameDESTINATION_KEY
: one or more key-value pairs to configure a destination for the pipelineYou must set only one of the following keys:
http_endpoint_uri
—a destination URI. The host can be either a static IP addressable from a Virtual Private Cloud (VPC) network, or the internal Domain Name System (DNS) hostname of a service resolvable using Cloud DNS. Optionally, you can also sethttp_endpoint_message_binding_template
to specify a message binding.Note that you can use this key to target a Cloud Run function, a Cloud Run job, or a Cloud Run service that will receive events as HTTP
POST
requests.message_bus
—the ID of an Eventarc Advanced bus.pubsub_topic
—the ID of a Pub/Sub topic.workflow
—the ID of a Workflows workflow.
You must set the following key:
network_attachment
—a resource that lets a producer VPC network initiate connections to a consumer VPC network. To publish events, Eventarc Advanced uses the network attachment to establish a connection to the endpoint hosted in a VPC network.You can create a network attachment that automatically accepts connections from any Private Service Connect interface that refers to the network attachment. Create the network attachment in the same network and region containing the destination resource.
If you are routing messages to a Google destination using a DNS address—for example, Cloud Run, Pub/Sub, Workflows, or another Eventarc Advanced bus—make sure that Private Google Access is enabled on the subnet used in the network attachment; otherwise, the DNS address can't be resolved.
To enable authentication, you can set one of the following keys:
google_oidc_authentication_service_account
—the service account email used to generate an an OIDC token which can be used for many scenarios, including endpoints where you intend to validate the token yourself. Optionally, you can setgoogle_oidc_authentication_audience
to specify the audience that this token is intended for. Generally, it should match the URL of the target pipeline. If not specified, the entire URL is used, including any request parameters.Note that Cloud Run performs an IAM check on every request and you can use the
run.routes.invoke
permission to configure who can access your Cloud Run service in either of the following ways:Grant the permission to select service accounts or groups to allow access to the service. All requests must have an HTTP Authorization header containing an OpenID Connect token signed by Google for one of the authorized service accounts.
Grant the permission to allUsers to allow unauthenticated access.
For more information, see Access control for Cloud Run .
oauth_token_authentication_service_account
—the service account email used to generate an OAuth token which should generally be used only when calling Google APIs hosted on*.googleapis.com
. Optionally, you can setoauth_token_authentication_scope
to specify the scope of this token; otherwise, it defaults tohttps://www.googleapis.com/auth/cloud-platform
. For Google Cloud services, it's a best practice to use thehttps://www.googleapis.com/auth/cloud-platform
scope, which includes all Google Cloud APIs, together with Identity and Access Management (IAM), which provides fine-grained access control.Note that all requests to another Eventarc Advanced bus, Pub/Sub, or Workflows must have an HTTP Authorization header containing an OAuth token signed by Google for one of the authorized service accounts.
Learn more about token types.
Optional: You can set one of the following keys:
output_payload_format_avro_schema_definition
output_payload_format_json
output_payload_format_protobuf_schema_definition
Note that if you set an output format, you must also specify an input format (see the following
input-payload-format-*
flags).
Optional: If
http_endpoint_uri
is not used as a destination key, you can set the following keys:project
—the Google Cloud project ID of the destination resource; by default, the project ID for the pipeline is used.location
—the location of the destination resource; by default, the location for the pipeline is used.
REGION
: a supported Eventarc Advanced locationAlternatively, you can set the gcloud CLI location property:
gcloud config set eventarc/location REGION
Optional: You can use the following flags:
--async
to return from the command immediately, without waiting for the operation in progress to complete.--crypto-key
to specify the fully qualified name of a customer-managed encryption key; if unspecified, Google-owned and managed keys are used.--logging-config
to configure the logging level which must be one of the following:NONE
,DEBUG
,INFO
,NOTICE
,WARNING
,ERROR
,CRITICAL
,ALERT
,EMERGENCY
.--mediations
to apply a transformation;transformation_template
is the only supported template and only one mediation per pipeline is supported; for example:--mediations=transformation_template='message.removeFields(["id\ ","credit_card_number","age"])'
Note that if you are applying a transformation, you must use one of the following flags to specify an input format.
One of the following to specify an input format:
--input-payload-format-avro-schema-definition
--input-payload-format-json
--input-payload-format-protobuf-schema-definition
--max-retry-attempts
,--max-retry-delay
, and--min-retry-delay
to retry events
For example:
gcloud beta eventarc pipelines create my-pipeline \ --destinations=http_endpoint_uri='https://example-endpoint.com',network_attachment=my-network-attachment,oauth_token_authentication_service_account=example-service-account@example-project.gserviceaccount.iam.com,oauth_token_authentication_scope='https://www.googleapis.com/auth/cloud-platform',output_payload_format_avro_schema_definition='{"type": "record","name": "my_record", "fields": [{"name": "my_field", "type":"string"}]}' \ --input-payload-format-avro-schema-definition='{"type":"record", "name": "my_record", "fields": [{"name": "my_field","type": "string"}]}' \ --location=us-central1 \ --async
For more details and examples, see the gcloud CLI documentation.
Enrollment
Open a terminal.
Create an enrollment using the
gcloud beta eventarc enrollments create
command:gcloud beta eventarc enrollments create ENROLLMENT_NAME \ --cel-match=MATCH_EXPRESSION \ --destination-pipeline=PIPELINE_NAME \ --message-bus=BUS_NAME \ --message-bus-project=PROJECT_ID \ --location=REGION
Replace the following:
ENROLLMENT_NAME
: the ID of the enrollment or a fully qualified nameMATCH_EXPRESSION
: the matching expression for this enrollment using CEL—for example,"message.type == 'google.cloud.dataflow.job.v1beta3.statusChanged'"
PIPELINE_NAME
: the target pipeline ID or its fully qualified name for this enrollmentBUS_NAME
: the ID of the Eventarc Advanced bus or its fully qualified namePROJECT_ID
: the Google Cloud project ID for the busREGION
: a supported Eventarc Advanced locationAlternatively, you can set the gcloud CLI location property:
gcloud config set eventarc/location REGION
Optional: You can use the following flag:
--async
to return from the command immediately, without waiting for the operation in progress to complete
For example:
gcloud beta eventarc enrollments create my-enrollment \ --cel-match="message.type == 'google.cloud.dataflow.job.v1beta3.statusChanged'" \ --destination-pipeline=my-pipeline \ --message-bus=my-message-bus \ --message-bus-project=another-google-cloud-project \ --location=us-central1 \ --async
Delete an enrollment
You can delete an enrollment in the Google Cloud console or by using the Google Cloud CLI.
Console
To delete an enrollment, in the Google Cloud console, go to the Eventarc > Pipelines page.
Click the name of the pipeline from which you want to delete the enrollment.
The Pipeline details pane opens.
Click Continue.
The Enrollments pane opens.
For the enrollment that you want to delete, click the
delete icon.Click Save.
gcloud
Open a terminal.
Delete an enrollment using the
gcloud beta eventarc enrollments delete
command:gcloud beta eventarc enrollments delete ENROLLMENT_NAME \ --location=REGION
Replace the following:
ENROLLMENT_NAME
: the ID of the enrollment or a fully qualified nameREGION
: the supported Eventarc Advanced location of the enrollment
Delete a pipeline
You can delete a pipeline in the Google Cloud console or by using the Google Cloud CLI.
Note that deleting a pipeline might take more than 10 minutes.
Console
To delete a pipeline, in the Google Cloud console, go to the Eventarc > Pipelines page.
In the list of pipelines, select the checkbox next to the name of the pipeline that you want to delete.
Click
Delete.Confirm the deletion by entering
Delete
.Click Delete.
gcloud
Open a terminal.
Delete a pipeline using the
gcloud beta eventarc pipelines delete
command:gcloud beta eventarc pipelines delete PIPELINE_NAME \ --location=REGION
Replace the following:
PIPELINE_NAME
: the ID of the pipeline or a fully qualified nameREGION
: the supported Eventarc Advanced location of the pipeline