[[["易于理解","easyToUnderstand","thumb-up"],["解决了我的问题","solvedMyProblem","thumb-up"],["其他","otherUp","thumb-up"]],[["很难理解","hardToUnderstand","thumb-down"],["信息或示例代码不正确","incorrectInformationOrSampleCode","thumb-down"],["没有我需要的信息/示例","missingTheInformationSamplesINeed","thumb-down"],["翻译问题","translationIssue","thumb-down"],["其他","otherDown","thumb-down"]],["最后更新时间 (UTC)：2025-08-19。"],[],[],null,["# Single Message Transforms (SMTs) overview\n\nMost streaming data pipelines require data transformations. Some users prefer\ntransforming data after it reaches its destination in an extract, load,\ntransform (ELT) pipeline, while others opt for transforming data before its\ningestion in an extract, transform, and load (ETL) pipeline. Traditionally, this\narchitecture required complex pipelines with tools like Dataflow or\nApache Flink to perform data transformations.\n\nPub/Sub offers **Single Message Transforms** (SMTs) to simplify\ndata transformations for streaming pipelines. SMTs enable lightweight\nmodifications to message data and attributes directly within\nPub/Sub. SMTs eliminate the need for additional data processing\nsteps or separate data transformation products.\n\nSMTs use cases\n--------------\n\nConsider designing an online store that wants to give customers personalized\nproduct recommendations as they browse the website. To do this, you can use\nPub/Sub to collect real-time data about customer activity on the\nsite. This includes data about the products viewed, the products added to the\ncart, and the ratings given to products.\n\nHowever, this raw data often needs some adjustments before it can be used to\ngenerate recommendations. For example, the raw data might contain extraneous\ndetails which are irrelevant for your use case. Examples of such details are the\ncustomer's browser type or the time they visited the site. The data might also\nnot be in the required format for the recommendation system. For example,\ntimestamps might be in different formats, or product IDs might need to be\nconverted to a different type.\n\nYou can use Pub/Sub SMTs to make data transformations\nsuch as the following:\n\n- Remove personally identifiable information (PII), such as full names and\n addresses, to protect customer privacy.\n\n- Retain only recommendation-relevant events, such as product views and\n purchases, and discard others, such as customer profile changes.\n\n- Ensure all timestamps, currency values, and product IDs adhere to a\n consistent format and type compatible with the recommendation system.\n\n- Generate new data fields from raw data, such as shopping cart total value or\n product page dwell time.\n\nIn summary, SMTs enable a wide range of use cases, including the following:\n\n- **Data masking and redaction**: Protect sensitive data by masking or\n redacting fields like credit card numbers or PII, aiding compliance with\n data privacy regulations.\n\n- **Data format conversion**: Transform data between different formats to\n ensure compatibility with downstream systems.\n\n- **Message filtering** : Process only relevant messages by filtering out\n unwanted messages based on content or attributes. SMTs allow for more\n complex filtering conditions than Pub/Sub's\n [built-in filters](/pubsub/docs/subscription-message-filter).\n\n- **Simple data transformations**: Perform basic data manipulation tasks, such\n as string manipulation, date formatting, or mathematical operations.\n\nSample message flow for SMTs\n----------------------------\n\nThe image shows an example Pub/Sub system with SMTs applied at both the\ntopic and subscription levels.\n**Figure 1** How messages are transformed with SMTs.\n\n\u003cbr /\u003e\n\nThe following procedure shows how the messages flow in the Pub/Sub\nsystem:\n\n1. The publisher applications **Publisher 1** and **Publisher 2** publish\n messages **A** and **B** respectively to the Pub/Sub topic.\n\n2. The topic's SMTs transform messages **A** and **B** into messages **A'** and\n **B'**, respectively.\n\n3. If a schema is attached to the topic, the transformed messages **A'** and\n **B'** are validated against the schema. If, for example, **A'** does not\n match the schema, the publish of message **A** fails with an error.\n\n4. The transformed messages **A'** and **B'** are written to\n Pub/Sub storage.\n\n5. Pub/Sub delivers messages **A'** and **B'** to all attached\n subscriptions, which are **Subscription 1** and **Subscription 2** as shown\n in the image.\n\n6. If **Subscription 1** has a filter configured, messages\n **A'** and **B'** are evaluated against the filter. Only messages matching\n the filter proceed to the next step. Other messages are automatically\n acknowledged by Pub/Sub.\n\n7. If **Subscription 2** has a filter configured, messages\n **A'** and **B'** are evaluated against the filter. Only messages matching\n the filter proceed to the next step. Other messages are automatically\n acknowledged by Pub/Sub.\n\n8. **Subscription 1** 's SMTs transform messages **A'** and **B'** . **A'**\n becomes **A''** and **B'** becomes **B''**.\n\n9. **Subscription 2** 's SMTs transform messages **A'** and **B'** . **A'**\n remains as **A'** and **B'** is filtered out.\n\n10. If **Subscription 1** is a push subscription with payload unwrapping\n enabled, messages **A''** and **B''** are unwrapped. If\n **Subscription 2** is a push subscription with payload unwrapping\n enabled, **A'** is unwrapped.\n\n11. **Subscriber 1** receives message **B''** , **Subscriber 2** receives\n message **A''** , and **Subscriber 3** receives message **A'**.\n\n12. Subscribers acknowledge the received messages.\n\n13. Pub/Sub deletes the acknowledged messages from storage.\n\nImportant information about SMTs\n--------------------------------\n\n- SMTs are integrated into the Pub/Sub\n API, allowing you to manage them as part of your topic or subscription\n configurations.\n\n- Up to 5 SMTs can be enabled on a topic or subscription.\n\n- SMTs operate on a single Pub/Sub message. They\n cannot aggregate multiple Pub/Sub messages.\n\n- When a SMT is run, it takes as input the Pub/Sub message,\n including its data and attributes. The output is a transformed\n Pub/Sub message, with modifications to its data or attributes.\n\n- If you have an SMT defined on a subscription that has ordering enabled and\n executing the SMT on any message throws an error, the subsequent messages\n for the same ordering key are not delivered to the subscriber.\n [Set up a dead-letter topic on the subscription](/pubsub/docs/handling-failures)\n to remove such a message that throws an error from the backlog of messages\n so subsequent messages can be delivered.\n\nTypes of SMTs\n-------------\n\n- [User Defined Functions](/pubsub/docs/smts/udfs-overview)\n\nWhat's next\n-----------\n\n- [Choose between topic and subscription SMTs](/pubsub/docs/smts/choose-smts)\n\n- [Learn about User Defined Functions](/pubsub/docs/smts/udfs-overview)\n\n- [Create a topic with SMTs](/pubsub/docs/smts/create-topic-smt)\n\n- [Create a subscription with SMTs](/pubsub/docs/smts/create-subscription-smt)"]]
