User Defined Functions (UDFs) overview

A JavaScript User-Defined Function (UDF) is a type of Single Message Transform (SMT). UDFs provide a flexible way to implement custom transformation logic within Pub/Sub, similar to BigQuery JavaScript UDFs.

UDFs accept a single message as input, perform the defined actions on the input, and return the result of the process.

UDFs have the following key properties:

• Function name: The name of the JavaScript function within the provided code that Pub/Sub applies to messages.

• Code: The JavaScript code that defines the transformation logic. This code must contain a function with the following signature:

  /**
  * Transforms a Pub/Sub message.
  * @return {(Object<string, (string | Object<string, string>)>|* null)} - To
  * filter a message, return `null`. To transform a message, return a map with
  * the following keys:
  *   - (required) 'data' : {string}
  *   - (optional) 'attributes' : {Object<string, string>}
  * Returning empty `attributes` will remove all attributes from the message.
  *
  * @param  {(Object<string, (string | Object<string, string>)>} - Pub/Sub
  * message. Keys:
  *   - (required) 'data' : {string}
  *   - (required) 'attributes' : {Object<string, string>}
  *
  * @param  {Object<string, any>} metadata - Pub/Sub message metadata.
  * Keys:
  *   - (optional) 'message_id'  : {string}
  *   - (optional) 'publish_time': {string} YYYY-MM-DDTHH:MM:SSZ format
  *   - (optional) 'ordering_key': {string}
  */
  function <function_name>(message, metadata) {
    // Perform custom transformation logic
    return message; // to filter a message instead, return `null`
  }
  

Inputs

• message argument: A JavaScript object representing the Pub/Sub message. It contains the following properties:

  • data: (String, required) The message payload.

  • attributes: (Object<String, String>, optional) A map of key-value pairs representing message attributes.

• metadata argument: A JavaScript object containing immutable metadata about the Pub/Sub message:

  • message_id: (String, optional) The unique ID of the message.

  • publish_time: (String, optional) The message's publish time in RFC 3339 format (YYYY-MM-DDTHH:mm:ssZ).

  • ordering_key: (String, optional) The message's ordering key, if applicable.

Outputs

• To transform a message, edit the contents of message.data and message.attributes, and return the altered message object.

• To filter a message, return null.

How UDFs transform a message

The result of running a UDF on a message can be one of the following:

• The UDF transforms a message.

• The UDF returns null.

  • Topic SMTs: Pub/Sub returns success to the publisher and includes a message ID in the response for the filtered messages. Pub/Sub does not store the message or send it to any subscribers.

  • Subscription SMTs: Pub/Sub acknowledges the message delivery without sending the message to a subscriber.

• The UDF throws an error.

  • Topic SMTs: Pub/Sub returns the error to the publisher and does not publish any of the messages.

  • Subscription SMTs: Pub/Sub negatively acknowledges the message.

Limitations

Pub/Sub enforces resource limits on UDFs to ensure efficient transformation operations. The limitations include:

• A maximum of 20 KB of code per UDF
• A maximum of 500 ms of execution time per message
• Support for only ECMAScript standard built-ins
• No calls to external APIs
• No imports of external libraries

Sample UDFs

Here are some sample UDFs for publishing and subscribing.

Function: Convert a day of the week integer to the corresponding string

When you add the following UDF to a topic or a subscription, the following changes take place during message publish or delivery:

1. Pub/Sub applies the function to the message. If the message does not have a JSON payload, the UDF throws an error.

2. The UDF looks for a field called dayOfWeek and if the value of this field is a number between 0 and 6, converts it to a corresponding day of the week such as Monday. If the field does not exist or the number is not in the range of 0 to 6, the code sets the dayOfWeek field to Unknown.

3. The UDF serializes the modified payload back into the message.

4. Pub/Sub passes the updated message to the next step in your pipeline.

function intToString(message, metadata) {
  const data = JSON.parse(message.data);
  switch(`data["dayOfWeek"]`) {
    case 0:
      data["dayOfWeek"] = "Sunday";
      break;
    case 1:
      data["dayOfWeek"] = "Monday";
      break;
    case 2:
      data["dayOfWeek"] = "Tuesday";
      break;
    case 3:
      data["dayOfWeek"] = "Wednesday";
      break;
    case 4:
      data["dayOfWeek"] = "Thursday";
      break;
    case 5:
      data["dayOfWeek"] = "Friday";
      break;
    case 6:
      data["dayOfWeek"] = "Saturday";
      break;
    default:
      data["dayOfWeek"] = "Unknown";
  }
  message.data = JSON.stringify(data);
  return message;
}

Function: Redact a social security number

When you add the following UDF to a topic or a subscription, the following changes take place during message publish or delivery:

1. Pub/Sub applies the function to the message. If the message does not have a JSON payload, the UDF throws an error.

2. The UDF removes the field ssn from the message payload (if it exists).

3. The UDF serializes the modified payload back into the message.

4. Pub/Sub passes the updated message to the next step in your pipeline.

function redactSSN(message, metadata) {
  const data = JSON.parse(message.data);
  delete data['ssn'];
  message.data = JSON.stringify(data);
  return message;
}

Function: Filter out and auto-ack specific messages

When you add the following UDF to a topic or a subscription, the following changes take place during message publish or delivery:

1. Pub/Sub applies the function to the message. If the message does not have a JSON payload, the UDF throws an error.

2. The UDF checks if the payload contains a field called region.

3. If the value of the region field is not US, the function returns null, causing Pub/Sub to filter the message.

4. If the value of the region field is US, Pub/Sub passes the original message to the next step in your pipeline.

function filterForUSRegion(message, metadata) {
  const data = JSON.parse(message.data);
  if (data["region"] !== "US") {
    return null;
  }
  return message;
}

Function: Validate message content to ensure the amount is not greater than 100

When you add the following UDF to a topic or a subscription, the following changes take place during message publish or delivery:

1. Pub/Sub applies the function to the message. If the message does not have a JSON payload, the UDF throws an error.

2. The UDF checks if the message contains a field called amount.

3. If the value of the amount field is greater than 100, the function throws an error.

4. If the value of the amount field is not greater than 100, the function returns the original message.

5. Pub/Sub then either marks the message as failed, or passes the original message to the next step in your pipeline.

function validateAmount(message, metadata) {
  const data = JSON.parse(message.data);
  if (data["amount"] > 100) {
    throw new Error("Amount is invalid");
  }
  return message;
}

What's next

• Choose between topic and subscription SMTs

• Create a topic with SMTs

• Create a subscription with SMTs