This document explains how to create a Pub/Sub subscription with Single Message Transforms (SMTs).
Subscription SMTs allow for lightweight modifications to message data and attributes directly within Pub/Sub. This feature enables data cleaning, filtering, or format conversion before the messages are delivered to a subscriber client.
To create a subscription with SMTs, you can use the Google Cloud console, the Google Cloud CLI, the client library, or the Pub/Sub API.
Before you begin
Learn about the Pub/Sub service and its terminology.
Learn about SMTs.
Required roles and permissions
To get the permissions that
you need to create a subscription with SMTs,
ask your administrator to grant you the
Pub/Sub Editor (roles/pubsub.editor
)
IAM role on your project.
For more information about granting roles, see Manage access to projects, folders, and organizations.
This predefined role contains the permissions required to create a subscription with SMTs. To see the exact permissions that are required, expand the Required permissions section:
Required permissions
The following permissions are required to create a subscription with SMTs:
-
Grant the create a subscription permission on the project:
pubsub.subscriptions.create
You might also be able to get these permissions with custom roles or other predefined roles.
Depending on the subscription type, you might need additional permissions. To find out the exact list of permissions, refer to the document that discusses creating the specific subscription. For example, if you are creating a BigQuery subscription with SMTs, see the Create BigQuery subscriptions page.
If you create a subscription in a different project than the topic,
you must grant the roles/pubsub.subscriber
role to the principal of
the project containing the subscription in the project containing the topic.
You can configure access control at the project level and at the individual resource level.
Create a subscription with SMTs
Before you create a subscription with SMTs, review the documentation for Properties of a subscription.
The following samples assume that you want to create a subscription with this User Defined Function (UDF) SMT. For more information about UDFs, see the UDFs overview.
function redactSSN(message, metadata) {
const data = JSON.parse(message.data);
delete data['ssn'];
message.data = JSON.stringify(data);
return message;
}
Console
In the Google Cloud console, go to the Pub/Sub Subscriptions page.
-
Click Create subscription.
The Create subscription page opens.
-
In the Subscription ID field, enter an ID for your subscription. For more information about naming subscriptions, see the naming guidelines.
-
Under Transforms, click Add a transform.
-
Enter a function name. For example:
redactSSN
. -
If you don't want to use the SMT with your subscription immediately, click the Disable transform option. This will still save the SMT, but it will not be executed as messages flow through your subscription.
-
Enter a new transform. For example:
function redactSSN(message, metadata) { const data = JSON.parse(message.data); delete data['ssn']; message.data = JSON.stringify(data); return message; }
-
Pub/Sub provides a validate function that lets you validate an SMT. Click Validate to validate the transform.
-
If you want to add another transform, click Add a transform.
- To arrange all the SMTs in a specific order, you can use the up and down arrows. To remove an SMT, click the delete button.
-
Pub/Sub provides a test function that lets you check the result of running the SMT on a sample message. To test the SMTs, click Test transform.
-
In the Test transform window, select the function that you want to test.
-
In the Input message window, enter a sample message.
-
If you want to add message attributes, click Add an attribute and enter one or more key-value pairs.
-
Click Test. The result of applying the SMTs on the message is displayed.
-
Close the window to stop testing SMTs on sample messages.
-
Click Create to create the subscription.
gcloud
-
In the Google Cloud console, activate Cloud Shell.
At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.
-
Pub/Sub provides a validate function that lets you validate an SMT. Run the
gcloud pubsub message-transforms validate
command:gcloud pubsub message-transforms validate --message-transform-file=TRANSFORM_FILE
Replace the following:
-
TRANSFORM_FILE: The path to the YAML or JSON file containing a single SMT.
Here is an example of a YAML transform file:
javascriptUdf: code: > function redactSSN(message, metadata) { const data = JSON.parse(message.data); delete data['ssn']; message.data = JSON.stringify(data); return message; } functionName: redactSSN
-
-
Pub/Sub provides a test function that lets you check the result of running one or more SMTs on a sample message. Run the
gcloud pubsub message-transforms test
command:gcloud pubsub message-transforms test --message-transforms-file=TRANSFORMS_FILE
Replace the following:
-
TRANSFORMS_FILE: The path to the YAML or JSON file containing one or more SMTs.
Here is an example of a YAML transforms file:
- javascriptUdf: code: > function redactSSN(message, metadata) { const data = JSON.parse(message.data); delete data['ssn']; message.data = JSON.stringify(data); return message; } functionName: redactSSN
-
-
To create the subscription, run the
gcloud pubsub subscriptions create
command:gcloud pubsub subscriptions create SUBSCRIPTION_ID \ --topic=TOPIC_NAME \ --message-transforms-file=TRANSFORMS_FILE
Replace the following:
-
SUBSCRIPTION_ID: The ID or name of the subscription you want to create. For guidelines on how to name a subscription, see Resource names. The name of a subscription is immutable.
-
TOPIC_NAME: The name of the topic to subscribe to, in the format
projects/PROJECT_ID/topics/TOPIC_ID
. -
TRANSFORMS_FILE: The path to the YAML or JSON file containing one or more SMTs.
Here is an example of a YAML transforms file:
- javascriptUdf: code: > function redactSSN(message, metadata) { const data = JSON.parse(message.data); delete data['ssn']; message.data = JSON.stringify(data); return message; } functionName: redactSSN
-
Java
Before trying this sample, follow the Java setup instructions in the Pub/Sub quickstart using client libraries. For more information, see the Pub/Sub Java API reference documentation.
To authenticate to Pub/Sub, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Python
Before trying this sample, follow the Python setup instructions in the Pub/Sub quickstart using client libraries. For more information, see the Pub/Sub Python API reference documentation.
To authenticate to Pub/Sub, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Go
Before trying this sample, follow the Go setup instructions in the Pub/Sub quickstart using client libraries. For more information, see the Pub/Sub Go API reference documentation.
To authenticate to Pub/Sub, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.