Types creation

This guide describes how to create a new type in Manufacturing Data Engine (MDE). You can create a type using the Configuration API or MDE interface.

Before you begin

Among other configuration settings, types allow you to define the contextualization scope for records of that type - the metadata buckets that can or must be used to contextualize records. If you want to associate a metadata bucket to a type, you first need to create the bucket. See how to create a metadata bucket for instructions.

Types naming

When you create a type, the name must be unique in an MDE deployment. Type names must not contain spaces or special characters such as &, @, or %.

Create a type

To create a type with one metadata bucket association and typical storage specifications you can use either the terminal or the console interface.

REST

Execute the following REST API request:

POST /configuration/v1/types

{
  "archetype": "ARCHETYPE NAME",
  "name": "TYPE_NAME",
  "metadataBuckets": [
    {
      "bucketName": "BUCKET_NAME",
      "version": "BUCKET_VERSION"
    }
  ],
  "storageSpecs": [
    {
      "sink": "BIG_QUERY",
      "disabled": "false",
      "materializeCloudMetadata": "true"
    },
    {
      "sink": "BIG_TABLE",
      "disabled": "false",
      "materializeCloudMetadata": "false"
    },
    {
      "sink": "GCS",
      "disabled": "false",
      "materializeCloudMetadata": "false"
    },
    {
      "sink": "PUBSUB_PROTO",
      "disabled": "false",
      "materializeCloudMetadata": "false"
    },
    {
      "sink": "PUBSUB_JSON",
      "disabled": "false",
      "materializeCloudMetadata": "false"
    }
  ]
}

Replace the following:

  • ARCHETYPE: name of the archetype. One of DISCRETE_DATA_SERIES, CLUSTERED_DISCRETE_DATA_SERIES, NUMERIC_DATA_SERIES, CLUSTERED_NUMERIC_DATA_SERIES, CONTINUOUS_DATA_SERIES, CLUSTERED_CONTINUOUS_DATA_SERIES.
  • TYPE_NAME: the name of the type to be created.
  • BUCKET_NAME: name of the bucket to be associated to this type.
  • BUCKET_VERSION: the version of the bucket to be associated to this type.

Console

  1. To create a new Type using the Console open the 'Type' section Tab.

    Types section - list of Types

  2. All available Types in the MDE instance are listed in this section. The list exposes the basic configuration of each Type, including:

  • Name: unique name of the Type. All Types must have different unique names.
  • Storage: displays the current Storage Specs associated to the Type.
  • Archetype: show the name of the Archetype defining the Type. Each Type is associated to a unique Archetype. The Archetype association can't be modified after the type is created.
  • Version: the version number of the Type. A Type can be versioned. The default view in this list only displays the latest version available for the Type. It is possible to show older version by un-checking the 'Show Latest' check-box.
  • Metadata: renders a color-coded badge for each Metadata Bucket associated to the Type. Tag Metadata Buckets are differentiated from Record Metadata Buckets using a specific icon inside the badge.
    • Transformations: displays a badge for each transformation type associated to the Type.
  • Enabled: displays a true value if the Type is enabled and a false value if it is not. Users can enable or disable Types at any time.
  • Active: when Types are created take a certain time to provision. While Types are being provisioned their active status is false. When they are ready they show as true.

Each new Type provisions its own infrastructure to store its Tags and Records for all Data Sinks when it is created. For example, a new Type will create a new Table in BigQuery with the new of the Type and will automatically populate the Schema corresponding to its Archetype in it. While the Type is provisioned it appears as Non Active.

  1. To create a new Type, click 'Add New Type'. The following form will appear:

    Types section - list of Types

  2. To define a new Type provide its Name and its Archetype. Once the Type is created, its name and archetype can't be changed. All Types must have different unique names. Archetypes must be selected from the available Archetype list. You can't create or modify existing Archetypes as they are part of the system configuration.

  3. Open the 'Data Schema' section to define, if needed, the JSON schema defining the Data Field of the mew Type:

    Types section - list of Types

  4. The Data Schema needs to be a valid JSON Schema or won't be accepted. This Data Schema is not enforced as part of the ingestion pipelines. In future MDE versions, the data object in the incoming messages will be validated against the Data Schema of its Type. The Data Schema provides visibility over the structure of the Data Field available in the different Data Sinks. Each Data Schema specification needs also to be defined according to the characteristics of the Archetype.

  5. Complete the definition of the new Type by defining the Metadata Buckets that are associated with the Type opening the 'Metadata Buckets' section:

    Types section - list of Types

  6. To add a Metadata Bucket to the Type click "ADD".

  7. Pick from the list of available Metadata Buckets the ones to add. Only the latest version of those Buckets is displayed. Metadata Bucket Versions can be updated later creating a new version of the Type. A new badge appears in the Metadata section after the Buckets are added.

    Types section - list of Types

  8. Define the default Storage Settings by opening the 'Storage Settings' section of the form. Inside this section there is a toggle button for each Data Sink available in this instance of MDE.

    Types section - list of Types

  9. Select each sink where the Tags belonging to this Type are be stored by default. The Storage Specification is inherited by the Tags when they are created. The Storage Specification can be modified at the Tag level at any time.

  10. To complete the definition of a new Type select which Transformations are associated to this Type by opening the 'Transform & Connect' section.

    Types section - list of Types

  11. Select the Transformations to be associated to the Type and define its parameters as per how to configure a Transformation

  12. Once all the parameters have been added, the new Type must be saved clicking in the 'CREATE' button. If any of the parameters is incorrect an error message is displayed. If the operation is successful, a confirmation message is displayed.