Schema overview

A Pub/Sub schema is an optional feature that you can use to enforce the format of the data field in a Pub/Sub message.

A schema creates a contract between the publisher and subscriber about the format of the messages. Pub/Sub enforces this format. Schemas facilitate inter-team consumption of data streams in your organization by creating a central authority for message types and permissions. A Pub/Sub message schema defines the names and data types for the fields in a message.

You can create a schema and associate it with a topic to enforce the schema for published messages. If a specific message does not conform to the schema, the message is not published. You can also create additional revisions for a schema.

Types of schemas

You can create a schema in Pub/Sub by using one of the following frameworks:

For example, the following schema defines the inventory of a warehouse, first in the Avro format and then in the Protocol Buffer format.

Apache Avro format

{
 "type" : "record",
 "name" : "Avro",
 "fields" : [
   {
     "name" : "ProductName",
     "type" : "string",
     "default": ""
   },
   {
     "name" : "SKU",
     "type" : "int",
     "default": 0
   },
   {
     "name" : "InStock",
     "type" : "boolean",
     "default": false
   }
 ]
}

Protocol Buffer format

syntax = "proto3";
message ProtocolBuffer {
  string product_name = 1;
  int32 SKU = 2;
  bool in_stock = 3;
}

Important information about schemas

The following is some important information regarding creating and managing Pub/Sub schemas.

  • For a schema to work with Pub/Sub, you must define only one top-level type. Import statements that reference other types are not supported.

  • You can associate the same schema to multiple topics.

  • You can manually test whether a message validates against a schema.

  • Don't include sensitive information such as personally identifiable information (PII) or security data in your schema field names.

Quotas and limits for schemas

Schemas have the following limits:

  • The schema definition field cannot exceed 300 KB in size.

  • A project can have a maximum of 10,000 schemas.

  • A single schema cannot have more than 20 revisions.

    For information about schemas and limits related to Pub/Sub, see Pub/Sub quotas and limits.

What's next