A JavaScript User-Defined Function (UDF) is a type of Single Message Transform (SMT). UDFs provide a flexible way to implement custom transformation logic within Pub/Sub, similar to BigQuery JavaScript UDFs.
UDFs accept a single message as input, perform the defined actions on the input, and return the result of the process.
UDFs have the following key properties:
Function name: The name of the JavaScript function within the provided code that Pub/Sub applies to messages.
Code: The JavaScript code that defines the transformation logic. This code must contain a function with the following signature:
/** * Transforms a Pub/Sub message. * @return {(Object<string, (string | Object<string, string>)>|* null)} - To * filter a message, return `null`. To transform a message, return a map with * the following keys: * - (required) 'data' : {string} * - (optional) 'attributes' : {Object<string, string>} * Returning empty `attributes` will remove all attributes from the message. * * @param {(Object<string, (string | Object<string, string>)>} - Pub/Sub * message. Keys: * - (required) 'data' : {string} * - (required) 'attributes' : {Object<string, string>} * * @param {Object<string, any>} metadata - Pub/Sub message metadata. * Keys: * - (optional) 'message_id' : {string} * - (optional) 'publish_time': {string} YYYY-MM-DDTHH:MM:SSZ format * - (optional) 'ordering_key': {string} */ function <function_name>(message, metadata) { // Perform custom transformation logic return message; // to filter a message instead, return `null` }
Inputs
message
argument: A JavaScript object representing the Pub/Sub message. It contains the following properties:data
: (String
, required) The message payload.attributes
: (Object<String, String>
, optional) A map of key-value pairs representing message attributes.
metadata
argument: A JavaScript object containing immutable metadata about the Pub/Sub message:message_id
: (String
, optional) The unique ID of the message.publish_time
: (String
, optional) The message's publish time in RFC 3339 format (YYYY-MM-DDTHH:mm:ssZ).ordering_key
: (String
, optional) The message's ordering key, if applicable.
Outputs
To transform a message, edit the contents of
message.data
andmessage.attributes
, and return the alteredmessage
object.To filter a message, return
null
.
How UDFs transform a message
The result of running a UDF on a message can be one of the following:
The UDF transforms a message.
The UDF returns
null
.Topic SMTs: Pub/Sub returns success to the publisher and includes a message ID in the response for the filtered messages. Pub/Sub does not store the message or send it to any subscribers.
Subscription SMTs: Pub/Sub acknowledges the message delivery without sending the message to a subscriber.
The UDF throws an error.
Topic SMTs: Pub/Sub returns the error to the publisher and does not publish any of the messages.
Subscription SMTs: Pub/Sub negatively acknowledges the message.
Resource Limits
Pub/Sub enforces resource limits on UDFs to ensure efficient transformation operations. The limitations include:
- Maximum 20 KB of code per UDF
- Maximum 500 ms of execution per message
- No calls to external APIs
- No imports of external libraries
Sample UDFs
Here are some sample UDFs for publishing and subscribing.
Function: Convert a day of the week integer to the corresponding string
When you add the following UDF to a topic or a subscription, the following changes take place during message publish or delivery:
Pub/Sub applies the function to the message. If the message does not have a JSON payload, the UDF throws an error.
The UDF looks for a field called
dayOfWeek
and if the value of this field is a number between 0 and 6, converts it to a corresponding day of the week such asMonday
. If the field does not exist or the number is not in the range of 0 to 6, the code sets thedayOfWeek
field toUnknown
.The UDF serializes the modified payload back into the message.
Pub/Sub passes the updated message to the next step in your pipeline.
function intToString(message, metadata) {
const data = JSON.parse(message.data);
switch(`data["dayOfWeek"]`) {
case 0:
data["dayOfWeek"] = "Sunday";
break;
case 1:
data["dayOfWeek"] = "Monday";
break;
case 2:
data["dayOfWeek"] = "Tuesday";
break;
case 3:
data["dayOfWeek"] = "Wednesday";
break;
case 4:
data["dayOfWeek"] = "Thursday";
break;
case 5:
data["dayOfWeek"] = "Friday";
break;
case 6:
data["dayOfWeek"] = "Saturday";
break;
default:
data["dayOfWeek"] = "Unknown";
}
message.data = JSON.stringify(data);
return message;
}
Function: Redact a social security number
When you add the following UDF to a topic or a subscription, the following changes take place during message publish or delivery:
Pub/Sub applies the function to the message. If the message does not have a JSON payload, the UDF throws an error.
The UDF removes the field
ssn
from the message payload (if it exists).The UDF serializes the modified payload back into the message.
Pub/Sub passes the updated message to the next step in your pipeline.
function redactSSN(message, metadata) {
const data = JSON.parse(message.data);
delete data['ssn'];
message.data = JSON.stringify(data);
return message;
}
Function: Filter out and auto-ack specific messages
When you add the following UDF to a topic or a subscription, the following changes take place during message publish or delivery:
Pub/Sub applies the function to the message. If the message does not have a JSON payload, the UDF throws an error.
The UDF checks if the payload contains a field called
region
.If the value of the
region
field is notUS
, the function returns null, causing Pub/Sub to filter the message.If the value of the
region
field isUS
, Pub/Sub passes the original message to the next step in your pipeline.
function filterForUSRegion(message, metadata) {
const data = JSON.parse(message.data);
if (data["region"] !== "US") {
return null;
}
return message;
}
Function: Validate message content to ensure the amount is not greater than 100
When you add the following UDF to a topic or a subscription, the following changes take place during message publish or delivery:
Pub/Sub applies the function to the message. If the message does not have a JSON payload, the UDF throws an error.
The UDF checks if the message contains a field called
amount
.If the value of the
amount
field is greater than100
, the function throws an error.If the value of the
amount
field is not greater than100
, the function returns the original message.Pub/Sub then either marks the message as failed, or passes the original message to the next step in your pipeline.
function validateAmount(message, metadata) {
const data = JSON.parse(message.data);
if (data["amount"] > 100) {
throw new Error("Amount is invalid");
}
return message;
}