- Resource: Pipeline
- Destination
- NetworkConfig
- HttpEndpoint
- AuthenticationConfig
- OidcToken
- OAuthToken
- MessagePayloadFormat
- ProtobufFormat
- AvroFormat
- JsonFormat
- Mediation
- Transformation
- RetryPolicy
- Methods
Resource: Pipeline
A representation of the Pipeline resource.
JSON representation |
---|
{ "name": string, "createTime": string, "updateTime": string, "labels": { string: string, ... }, "uid": string, "annotations": { string: string, ... }, "displayName": string, "destinations": [ { object ( |
Fields | |
---|---|
name |
Identifier. The resource name of the Pipeline. Must be unique within the location of the project and must be in |
create |
Output only. The creation time. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z" and "2014-10-02T15:01:23.045123456Z". A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
update |
Output only. The last-modified time. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z" and "2014-10-02T15:01:23.045123456Z". A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
labels |
Optional. User labels attached to the Pipeline that can be used to group resources. An object containing a list of "key": value pairs. Example: { "name": "wrench", "mass": "1.3kg", "count": "3" }. An object containing a list of |
uid |
Output only. Server-assigned unique identifier for the Pipeline. The value is a UUID4 string and guaranteed to remain unchanged until the resource is deleted. |
annotations |
Optional. User-defined annotations. See https://google.aip.dev/128#annotations. An object containing a list of |
display |
Optional. Display name of resource. |
destinations[] |
Required. List of destinations to which messages will be forwarded. Currently, exactly one destination is supported per Pipeline. |
mediations[] |
Optional. List of mediation operations to be performed on the message. Currently, only one Transformation operation is allowed in each Pipeline. |
crypto |
Optional. Resource name of a KMS crypto key (managed by the user) used to encrypt/decrypt the event data. If not set, an internal Google-owned key will be used to encrypt messages. It must match the pattern "projects/{project}/locations/{location}/keyRings/{keyring}/cryptoKeys/{key}". |
input |
Optional. The payload format expected for the messages received by the Pipeline. If inputPayloadFormat is set then any messages not matching this format will be treated as persistent errors. If inputPayloadFormat is not set, then the message data will be treated as an opaque binary and no output format can be set on the Pipeline through the Pipeline.Destination.output_payload_format field. Any Mediations on the Pipeline that involve access to the data field will fail as persistent errors. |
logging |
Optional. Config to control Platform Logging for Pipelines. |
retry |
Optional. The retry policy to use in the pipeline. |
etag |
Output only. This checksum is computed by the server based on the value of other fields, and might be sent only on create requests to ensure that the client has an up-to-date value before proceeding. |
Destination
Represents a target of an invocation over HTTP.
JSON representation |
---|
{ "networkConfig": { object ( |
Fields | |
---|---|
network |
Optional. Network config is used to configure how Pipeline resolves and connects to a destination. |
authentication |
Optional. An authentication config used to authenticate message requests, such that destinations can verify the source. For example, this can be used with private GCP destinations that require GCP credentials to access like Cloud Run. This field is optional and should be set only by users interested in authenticated push |
output |
Optional. The message format before it is delivered to the destination. If not set, the message will be delivered in the format it was originally delivered to the Pipeline. This field can only be set if Pipeline.input_payload_format is also set. |
Union field destination_descriptor . The destination identifier to which the request should be routed to. destination_descriptor can be only one of the following: |
|
http |
Optional. An HTTP endpoint destination described by an URI. If a DNS FQDN is provided as the endpoint, Pipeline will create a peering zone to the consumer VPC and forward DNS requests to the VPC specified by network config to resolve the service endpoint. See: https://cloud.google.com/dns/docs/zones/zones-overview#peering_zones |
workflow |
Optional. The resource name of the Workflow whose Executions are triggered by the events. The Workflow resource should be deployed in the same project as the Pipeline. Format: |
message |
Optional. The resource name of the Message Bus to which events should be published. The Message Bus resource should exist in the same project as the Pipeline. Format: |
topic |
Optional. The resource name of the Pub/Sub topic to which events should be published. Format: |
NetworkConfig
Represents a network config to be used for destination resolution and connectivity.
JSON representation |
---|
{ "networkAttachment": string } |
Fields | |
---|---|
network |
Required. Name of the NetworkAttachment that allows access to the consumer VPC. Format: |
HttpEndpoint
Represents a HTTP endpoint destination.
JSON representation |
---|
{ "uri": string, "messageBindingTemplate": string } |
Fields | |
---|---|
uri |
Required. The URI of the HTTP enpdoint. The value must be a RFC2396 URI string. Examples: |
message |
Optional. The CEL expression used to modify how the destination-bound HTTP request is constructed. If a binding expression is not specified here, the message is treated as a CloudEvent and is mapped to the HTTP request according to the CloudEvent HTTP Protocol Binding Binary Content Mode. In this representation, all fields except the To construct the HTTP request payload and the value of the content-type HTTP header, the payload format is defined as follows: 1) Use the output_payload_format_type on the Pipeline.Destination if it is set, else: 2) Use the input_payload_format_type on the Pipeline if it is set, else: 3) Treat the payload as opaque binary data. The If a non-empty binding expression is specified then this expression is used to modify the default CloudEvent HTTP Protocol Binding Binary Content representation. The result of the CEL expression must be a map of key/value pairs which is used as follows: - If a map named The CEL expression may access the incoming CloudEvent message in its definition, as follows: - The
Additionally, the following CEL extension functions are provided for use in this CEL expression: - toBase64Url: map.toBase64Url() -> string - Converts a CelValue to a base64url encoded string - toJsonString: map.toJsonString() -> string - Converts a CelValue to a JSON string - merge: map1.merge(map2) -> map3 - Merges the passed CEL map with the existing CEL map the function is applied to. - If the same key exists in both maps, if the key's value is type map both maps are merged else the value from the passed map is used. - denormalize: map.denormalize() -> map - Denormalizes a CEL map such that every value of type map or key in the map is expanded to return a single level map. - The resulting keys are "." separated indices of the map keys. - For example: { "a": 1, "b": { "c": 2, "d": 3 } "e": [4, 5] } .denormalize() -> { "a": 1, "b.c": 2, "b.d": 3, "e.0": 4, "e.1": 5 } - setField: map.setField(key, value) -> message - Sets the field of the message with the given key to the given value. - If the field is not present it will be added. - If the field is present it will be overwritten. - The key can be a dot separated path to set a field in a nested message. - Key must be of type string. - Value may be any valid type. - removeFields: map.removeFields([key1, key2, ...]) -> message - Removes the fields of the map with the given keys. - The keys can be a dot separated path to remove a field in a nested message. - If a key is not found it will be ignored. - Keys must be of type string. - toMap: [map1, map2, ...].toMap() -> map - Converts a CEL list of CEL maps to a single CEL map - toDestinationPayloadFormat(): message.data.toDestinationPayloadFormat() -> string or bytes - Converts the message data to the destination payload format specified in Pipeline.Destination.output_payload_format - This function is meant to be applied to the message.data field. - If the destination payload format is not set, the function will return the message data unchanged. - toCloudEventJsonWithPayloadFormat: message.toCloudEventJsonWithPayloadFormat() -> map - Converts a message to the corresponding structure of JSON format for CloudEvents - This function applies toDestinationPayloadFormat() to the message data. It also sets the corresponding datacontenttype of the CloudEvent, as indicated by Pipeline.Destination.output_payload_format. If no outputPayloadFormat is set it will use the existing datacontenttype on the CloudEvent if present, else leave datacontenttype absent. - This function expects that the content of the message will adhere to the standard CloudEvent format. If it doesn't then this function will fail. - The result is a CEL map that corresponds to the JSON representation of the CloudEvent. To convert that data to a JSON string it can be chained with the toJsonString function. The Pipeline expects that the message it receives adheres to the standard CloudEvent format. If it doesn't then the outgoing message request may fail with a persistent error. |
AuthenticationConfig
Represents a config used to authenticate message requests.
JSON representation |
---|
{ // Union field |
Fields | |
---|---|
Union field authentication_method_descriptor . The type of authentication method. authentication_method_descriptor can be only one of the following: |
|
google |
Optional. This authenticate method will apply Google OIDC tokens signed by a GCP service account to the requests. |
oauth |
Optional. If specified, an OAuth token will be generated and attached as an This type of authorization should generally only be used when calling Google APIs hosted on *.googleapis.com. |
OidcToken
Represents a config used to authenticate with a Google OIDC token using a GCP service account. Use this authentication method to invoke your Cloud Run and Cloud Functions destinations or HTTP endpoints that support Google OIDC.
JSON representation |
---|
{ "serviceAccount": string, "audience": string } |
Fields | |
---|---|
service |
Required. Service account email used to generate the OIDC Token. The principal who calls this API must have iam.serviceAccounts.actAs permission in the service account. See https://cloud.google.com/iam/docs/understanding-service-accounts for more information. Eventarc service agents must have roles/roles/iam.serviceAccountTokenCreator role to allow the Pipeline to create OpenID tokens for authenticated requests. |
audience |
Optional. Audience to be used to generate the OIDC Token. The audience claim identifies the recipient that the JWT is intended for. If unspecified, the destination URI will be used. |
OAuthToken
Contains information needed for generating an OAuth token. This type of authorization should generally only be used when calling Google APIs hosted on *.googleapis.com.
JSON representation |
---|
{ "serviceAccount": string, "scope": string } |
Fields | |
---|---|
service |
Required. Service account email used to generate the OAuth token. The principal who calls this API must have iam.serviceAccounts.actAs permission in the service account. See https://cloud.google.com/iam/docs/understanding-service-accounts for more information. Eventarc service agents must have roles/roles/iam.serviceAccountTokenCreator role to allow Pipeline to create OAuth2 tokens for authenticated requests. |
scope |
Optional. OAuth scope to be used for generating OAuth access token. If not specified, "https://www.googleapis.com/auth/cloud-platform" will be used. |
MessagePayloadFormat
Represents the format of message data.
JSON representation |
---|
{ // Union field |
Fields | |
---|---|
Union field kind . The kind of message format. One of Protobuf, Avro, and JSON supported. This allows specification of what specific format messages are sent and received. kind can be only one of the following: |
|
protobuf |
Optional. Protobuf format. |
avro |
Optional. Avro format. |
json |
Optional. JSON format. |
ProtobufFormat
The format of a Protobuf message payload.
JSON representation |
---|
{ "schemaDefinition": string } |
Fields | |
---|---|
schema |
Optional. The entire schema definition is stored in this field. |
AvroFormat
The format of an AVRO message payload.
JSON representation |
---|
{ "schemaDefinition": string } |
Fields | |
---|---|
schema |
Optional. The entire schema definition is stored in this field. |
JsonFormat
This type has no fields.
The format of a JSON message payload.
Mediation
Mediation defines different ways to modify the Pipeline.
JSON representation |
---|
{ // Union field |
Fields | |
---|---|
Union field mediation_descriptor . The config of mediation. mediation_descriptor can be only one of the following: |
|
transformation |
Optional. How the Pipeline is to transform messages |
Transformation
Transformation defines the way to transform an incoming message.
JSON representation |
---|
{ "transformationTemplate": string } |
Fields | |
---|---|
transformation |
Optional. The CEL expression template to apply to transform messages. The following CEL extension functions are provided for use in this CEL expression: - merge: map1.merge(map2) -> map3 - Merges the passed CEL map with the existing CEL map the function is applied to. - If the same key exists in both maps, if the key's value is type map both maps are merged else the value from the passed map is used. - denormalize: map.denormalize() -> map - Denormalizes a CEL map such that every value of type map or key in the map is expanded to return a single level map. - The resulting keys are "." separated indices of the map keys. - For example: { "a": 1, "b": { "c": 2, "d": 3 } "e": [4, 5] } .denormalize() -> { "a": 1, "b.c": 2, "b.d": 3, "e.0": 4, "e.1": 5 } - setField: map.setField(key, value) -> message - Sets the field of the message with the given key to the given value. - If the field is not present it will be added. - If the field is present it will be overwritten. - The key can be a dot separated path to set a field in a nested message. - Key must be of type string. - Value may be any valid type. - removeFields: map.removeFields([key1, key2, ...]) -> message - Removes the fields of the map with the given keys. - The keys can be a dot separated path to remove a field in a nested message. - If a key is not found it will be ignored. - Keys must be of type string. - toMap: [map1, map2, ...].toMap() -> map - Converts a CEL list of CEL maps to a single CEL map - toDestinationPayloadFormat(): message.data.toDestinationPayloadFormat() -> string or bytes - Converts the message data to the destination payload format specified in Pipeline.Destination.output_payload_format - This function is meant to be applied to the message.data field. - If the destination payload format is not set, the function will return the message data unchanged. - toCloudEventJsonWithPayloadFormat: message.toCloudEventJsonWithPayloadFormat() -> map - Converts a message to the corresponding structure of JSON format for CloudEvents - This function applies toDestinationPayloadFormat() to the message data. It also sets the corresponding datacontenttype of the CloudEvent, as indicated by Pipeline.Destination.output_payload_format. If no outputPayloadFormat is set it will use the existing datacontenttype on the CloudEvent if present, else leave datacontenttype absent. - This function expects that the content of the message will adhere to the standard CloudEvent format. If it doesn't then this function will fail. - The result is a CEL map that corresponds to the JSON representation of the CloudEvent. To convert that data to a JSON string it can be chained with the toJsonString function. |
RetryPolicy
The retry policy configuration for the Pipeline. The pipeline exponentially backs off in case the destination is non responsive or returns a retryable error code. The default semantics are as follows: The backoff starts with a 5 second delay and doubles the delay after each failed attempt (10 seconds, 20 seconds, 40 seconds, etc.). The delay is capped at 60 seconds by default. Please note that if you set the minRetryDelay and maxRetryDelay fields to the same value this will make the duration between retries constant.
JSON representation |
---|
{ "maxAttempts": integer, "minRetryDelay": string, "maxRetryDelay": string } |
Fields | |
---|---|
max |
Optional. The maximum number of delivery attempts for any message. The value must be between 1 and 100. The default value for this field is 5. |
min |
Optional. The minimum amount of seconds to wait between retry attempts. The value must be between 1 and 600. The default value for this field is 5. A duration in seconds with up to nine fractional digits, ending with ' |
max |
Optional. The maximum amount of seconds to wait between retry attempts. The value must be between 1 and 600. The default value for this field is 60. A duration in seconds with up to nine fractional digits, ending with ' |
Methods |
|
---|---|
|
Create a new Pipeline in a particular project and location. |
|
Delete a single pipeline. |
|
Get a single Pipeline. |
|
Gets the access control policy for a resource. |
|
List pipelines. |
|
Update a single pipeline. |
|
Sets the access control policy on the specified resource. |
|
Returns permissions that a caller has on the specified resource. |