Metadata bucket versioning

This guide describes how to create a new metadata bucket version for an existing bucket in Manufacturing Data Engine (MDE).

Metadata bucket versions allow you to evolve the bucket schema. Types and parsers always reference a specific bucket version, so when you create a new version, it doesn't affect the existing data processing pipelines. You must upgrade your types and parsers or create new types and parsers to use the new bucket version. You can create a metadata bucket version using the Configuration API or MDE UI.

Before you begin

This guide requires you have already created a metadata bucket for the MDE.

Versioning restrictions

When creating new versions of a bucket, consider the following limitations:

New versions may:

  • Add new optional fields.
  • Mark a required field optional.

New versions may not:

  • Remove fields.
  • Change data type of existing fields.
  • Mark an an optional attribute required.

Create a new metadata bucket version

You can create a metadata bucket version using the terminal or the console.

REST

Execute the following REST API request:

POST /metadata/v1/buckets/BUCKET_NAME/versions

{
  "schema": "BUCKET_SCHEMA",
  "provider": "local"
}

Replace the following:

  • BUCKET_NAME: the name of the bucket to be created.
  • BUCKET_TYPE: the type of the bucket to be created. Must be one of TAG, RECORD, or LOOKUP. The natural_key of a metadata instance in a TAG bucket must be a tag name. The natural_key of a metadata instance in a RECORD bucket can be any key. The natural_key of a metadata instance in a LOOKUP bucket can be any key, but records can't reference instances in a lookup bucket, and the instances are not written to any data sink. Lookup buckets only serve as source of reference data that can be retrieved by a parser using the Whistle function lookupByKey and used to enrich records.
  • BUCKET_SCHEMA: the JSON schema of the bucket. The schema constrains the instances that may be created in the bucket.
  • INSTANCE_OVERWRITE_MODE: Either TRUE or FALSE. Determines whether new instances are created by merging the provided instance with the latest instance for a given natural key or if new instances are created as provided.

Console

  1. To create a New Metadata Bucket Version select the 'Create New Version' from the METADATA section of the Cloud Management MC UI:

    Types section - list of Types

  2. A configuration panel is displayed with a form to update the parameters of the new Metadata Bucket Version:

    Types section - list of Types

    Only backward compatible changes are allowed for new versions. New, breaking, and changes requires creating a new Bucket. In the panel you can update the following:

    • Instance Update Mode: Change the instance ingestion behavior of the Bucket.
    • Metadata Schema: Add new attributes to the existing schema of the Bucket.

  3. After the changes are implemented, click 'CREATE' to save the new version of the Metadata Bucket.

  4. An error message is displayed if the change is unsuccessful. A success message is shown if the change is successful. The new Version number is sequential and is automatically assigned to the Bucket.