Migrate from Object change notifications to Pub/Sub notifications

This guide is for users of the Cloud Storage's deprecated Object change notifications feature. Pub/Sub notifications for Cloud Storage is the recommended tool for generating notifications that track changes to objects in your Cloud Storage buckets. Pub/Sub notifications offer improved speed, flexibility, setup, and cost-effectiveness. This guide explains the differences between Object change notifications and Pub/Sub notifications for Cloud Storage and provides migration steps from Object change notifications to Pub/Sub notifications.

Overview of Object change notifications

Object change notifications are a legacy mechanism in Cloud Storage for notifying an application about changes to objects within a bucket. When an Object change notification is set up, Cloud Storage sends HTTP POST requests (webhooks) to a specified application URL whenever an object is added, updated, or deleted. Object change notifications are established by sending a watchAll request to Cloud Storage, using the JSON API, client libraries, or the gsutil notification watchbucket command. There is no pull mechanism, and you must have a publicly accessible domain name backed by an HTTP server to receive the webhook messages. We recommend using Pub/Sub notifications for Cloud Storage for new implementations due to its reliability, scalability, and flexibility.

For more detailed information, see Object change notification.

Overview of Pub/Sub notifications

Pub/Sub notifications for Cloud Storage provide a modern, scalable, and reliable way to trigger actions in response to changes in your Cloud Storage buckets by sending event information to a Pub/Sub topic. Pub/Sub offers push-based delivery of messages as HTTP POST requests to webhooks. When objects are created, updated, or deleted, Cloud Storage publishes messages containing object metadata to a specified Pub/Sub topic, which can then be consumed by various subscribers such as Cloud Run functions, data pipelines, or microservices, enabling flexible and event-driven architectures with at-least-once delivery and robust error handling features.

For more detailed information, see Pub/Sub notifications for Cloud Storage.

Compare Object change notifications with Pub/Sub notifications

The following table compares Object change notifications with Pub/Sub notifications features:

Feature Object change notifications Pub/Sub notifications
Purpose Notifies an application directly through HTTP POST requests (webhooks) when objects in a bucket change. Sends information about changes to objects in Cloud Storage buckets to a Pub/Sub topic.
Delivery mechanism Direct HTTP POST (webhook) to a specified application URL. Messages published to a Pub/Sub topic can then be consumed by various subscribers such as Cloud Run functions, other applications, and data pipelines.
Reliability Attempts reliable delivery but timeliness is not guaranteed. Notifications can be delayed indefinitely. Offers at-least-once delivery, meaning messages might be delivered multiple times, but are not lost. Pub/Sub handles message persistence and retries.
Scalability Less scalable, as it relies on direct webhooks that your application needs to handle. Highly scalable and designed for large-scale event processing.
Flexibility Limited to direct webhook integration. Highly flexible. Pub/Sub messages can trigger Cloud Run functions, feed into data pipelines (Dataflow), and be consumed by other microservices.
Filtering None Robust filtering options available at the Pub/Sub subscription level, allowing subscribers to only receive messages that meet specific criteria.
Security Requires your application endpoint to be publicly accessible (HTTPS). Pub/Sub offers IAM for fine-grained access control to topics and subscriptions. Pub/Sub helps with secure message delivery, whether you're pulling notifications directly or having them pushed to a public endpoint.
Complexity Can be simpler to set up for basic use cases, but managing reliable delivery and scaling can become complex. Requires understanding of Pub/Sub concepts (topics, subscriptions), but offers a more robust and manageable solution for event-driven architectures.
Deprecation status Deprecated. We recommend using Pub/Sub notifications for new implementations. Active. This is the primary and actively developed method for Cloud Storage notifications.
Recommended use Not recommended for new projects. Primarily for less complex legacy integrations that you cannot migrate. Highly recommended for building robust, scalable, and event-driven architectures that react to Cloud Storage changes.

Why migrate to Pub/Sub notifications?

Migrating from the legacy Object change notifications to Pub/Sub for Cloud Storage notifications is an important step for robust event management. Pub/Sub is recommended for event-driven architectures within Google Cloud, offering significant technical and operational advantages over Object change notifications.

The following are the advantages of migrating to Pub/Sub notifications:

  • Reliable delivery: Pub/Sub delivers each published message at least once for every subscription, validating that events reach your consumers. Reliable delivery minimizes data loss and improves the dependability of your workflows compared to Object change notifications less robust delivery model.
  • Scalability: Designed for high throughput, Pub/Sub notifications can handle large volumes of events automatically. Using Pub/Sub notifications, you can eliminate the performance bottlenecks you might encounter with Object change notifications as your data or event frequency increases.
  • Integration with Google Cloud services: Pub/Sub seamlessly integrates with multiple Google Cloud services, providing flexibility for building automated workflows using Cloud Run functions, Cloud Run, Dataflow, and enhanced observability through Cloud Logging and Cloud Monitoring.
  • Granular control: With Pub/Sub you can filter messages at the subscription level. This enables consumers to receive only relevant events, reducing unnecessary processing and network traffic.
  • Platform support: Pub/Sub notifications is the supported messaging service. Migrating helps you use a technology that receives ongoing enhancements, security updates, and comprehensive documentation, unlike the deprecated Object change notifications.

Migration steps

Both Object change notifications and Pub/Sub notifications for Cloud Storage can occasionally send duplicate messages. Therefore, your consuming code must be designed to safely handle duplicate messages.

To migrate from Object change notifications to Pub/Sub notifications, follow these steps:

  1. Start using Pub/Sub notifications for Cloud Storage in addition to your existing Object change notifications setup. For information about how to configure Pub/Sub notifications, see Configure Pub/Sub notifications for Cloud Storage.

  2. Test and verify that your application's Pub/Sub notifications processing workflow is functioning correctly. For information about how to monitor Pub/Sub subscription, see Monitor subscriptions within Pub/Sub.

  3. Stop processing messages received from an Object change notifications channel. To stop a notifications channel, make a stop request.

Considerations for Pub/Sub push subscription

While Pub/Sub pull subscriptions offer enhanced flexibility and control, Pub/Sub push subscriptions closely resemble Object change notification messages. As a result, the push subscriptions become a faster migration path for existing Object change notification users. For a detailed comparison of push and pull subscriptions, see Choose a subscription type.

If you plan to use push subscription and similarly plan to reuse existing notification handling code, you'll need to consider the differences between Object change notifications and Pub/Sub notifications push request formats and response code interpretations. The differences are described in the following sections.

Push request format

This section describes the push request format differences between Object change notifications and Pub/Sub notifications.

  • Object change notifications: An Object change notification is delivered as an HTTP POST request to your application URL, in the following format:

    POST /ApplicationUrlPath
Accept: * / *
Content-Length: 1097
Content-Type: application/json; charset="utf-8"
Host: ApplicationUrlHost
X-Goog-Channel-Id: ChannelId
X-Goog-Channel-Token: ClientToken
X-Goog-Resource-Id: ResourceId
X-Goog-Resource-State: ResourceState
X-Goog-Resource-Uri: https://storage.googleapis.com/storage/v1/b/BucketName/o?alt=json

{
"kind": "storage#object",
"id": "BucketName/ObjectName",
"selfLink": "https://www.googleapis.com/storage/v1/b/BucketName/o/ObjectName",
"name": "ObjectName",
"bucket": "BucketName",
"generation": "1367014943964000",
"metageneration": "1",
"contentType": "application/octet-stream",
"updated": "2013-04-26T22:22:23.832Z",
"size": "10",
"md5Hash": "xHZY0QLVuYng2gnOQD90Yw==",
"mediaLink": "https://content-storage.googleapis.com/storage/v1/b/BucketName/o/ObjectName?generation=1367014943964000&alt=media",
"owner": {
  "entity": "user-jeffersonloveshiking@gmail.com"
},
"crc32c": "C7+82w==",
"etag": "COD2jMGv6bYCEAE="
}

  • Pub/Sub notifications: A Pub/Sub notification, when configured for push delivery, is delivered as an HTTP POST request. The data field contains the base64-encoded Cloud Storage event payload. When the data field is decoded, it matches the Object change notifications message body.

    POST /YourSpecifiedURL
Accept: * / *
Content-Length: 1097
Content-Type: application/json; charset="utf-8"
Host: ApplicationUrlHost
{
  "deliveryAttempt": 5,
  "message":
    {"attributes":
      {"notificationConfig":"projects/_/buckets/foo/notificationConfigs/3",
        "eventType": "OBJECT_FINALIZE",
        "payloadFormat": "JSON_API_V1",
        "bucketId": "foo",
        "objectId": "bar",
        "objectGeneration": 123456,
        "eventTime": "2021-01-15T01:30:15.01Z"
      },
    "data": "ewogImtpbm",
    "messageId": "2070443601311540",
    "message_id": "2070443601311540",
    "orderingKey": "key",
    "publishTime": "2021-02-26T19:13:55.749Z",
    "publish_time": "2021-02-26T19:13:55.749Z"
    },
  "subscription": "projects/myproject/subscriptions/mysubscription"
}

Response code

The following table describes the response code interpretation differences between Object change notifications and Pub/Sub notifications:

Feature Response code Interpretation Action
Object change notifications
102, 200, 201, 202, 204 Success Message processed
500, 502, 503, 504 Failed to process Retry later
Timeouts, failed connections, no response Failed to process Retry later
Any other HTTP code. For example, 404 Permanent failure Don't retry message
Pub/Sub notifications
102, 200, 201, 202, 204 Success Message acknowledged
All other response codes, timeouts, and connection failures Failure Retry later

What's next