Release notes

• Improvement (376024419): Enable automatic reconciliation of the missing infrastructure elements and for MDE types. When the user creates a new type, multiple infrastructure components are created in the background. To accommodate cases like type activation or upgrade issues, MDE now checks for missing infrastructure components in the startup of the bigquery-sink service and makes sure that it is aligned with the types configurations in MDE.

Helm charts - MDE services containers image tag: 3f285a5. - MDE UI image tag: 2.22.0-dev.0.123.4.

• Improvement (330297418): Enable the three-projects topology for bigquery-sink: the service to be deployed in a project, the Pub/Sub in another project, and the BigQuery datasets in a third project. Find these as dedicated properties in the BigQuery sink helm chart.
• Improvement (320427338): Creating automatic BigQuery views based on the type buckets schema. This provides a stable column-based schema for accessing the types as well as its metadata in a single view. For more information, see Analyticis view.
• Improvement (320427684): Types data field schema validation. The non-numeric types can have a JSON schema that governs the data field of its records, and this schema is now enforced during the processing. For more information, see Data field.
• Improvement (320427624): New real-time stats service for the ingested tags. For more information, see Real-Time service.
• Improvement (324029584): Automatic deletion for excessive metadata instances to prevent the system performance degradation. For more information, see Metadata automatic deletion.
• Improvement (353244779): New Notification API endpoint to display time-sensitive messages to the user such as the metadata instances approach the maximum size limit.
• Improvement (353247809): New API to stop and start the processing of messages.

Helm charts

• MDE services containers image tag: a04af0d.
• MDE UI image tag: 2.22.0-dev.0.123.4.

• Improvement (324563314): Enabling the Dataflow Streaming Engine for the Cloud Storage Flows to improve the scalability behaviour of the Cloud Storage jobs.

Helm charts

• MDE services containers image tag: 09497d7.
• MDE UI image tag: 2.15.0.0.109.2.

• Improvement (320427011): All the services now use full names for Pub/Sub subscriptions and topics (For example: projects/PROJECT_ID/subscriptions/configuration-processing-subscription). This allows having the topics or subscriptions in a different project that the one the services are deployed to.
• Improvement (320443808): All dataflow jobs now use private IP by default. For more information, see Routes Firewall.

Helm charts

• MDE services containers image tag: 4b79010.
• MDE UI image tag: 2.15.0.0.109.2.

• Improvement (311599436): Added a new field to the output of the Group Event Change Transformation to include the message that triggered the change in the transformation
• Improvement (316554360): When parsing a message, add the output of the headers field to the Pub/Sub headers and preserve through the entire pipeline.

Helm charts

• MDE services containers image tag: 8199764.
• MDE UI image tag: 2.15.0.0.109.2.

Helm charts

• MDE services containers image tag: 7a994ee.
• MDE UI image tag: 2.15.0.0.109.2.

The following Dataflow jobs need to be stopped before upgrading:

• eventchange-dataflow-flex-job
• gcs-writer-dataflow-flex-job
• window-dataflow-flex-job

• Improvement (301217870): Improved validation messages in the parser test API to better indicate how the input messages timestamps and extra fields are handled.
• Improvement (300220191): Enabling more special characters in the tag name to allow ., $, !, and *.
• Improvement (302490275): The mde::sanitizeTagName() Whistle function is now supporting the allowed special characters in the tag name.
• Multiple documentation updates including specifying the recommended number of types for the small size deployments.

Note: Only GKE services are affected by this release.

Helm charts

• MDE services containers image tag: e29b2aa.
• MDE UI image tag: "2.15.0.0.104.0-noauth.

Infrastructure changes: MDE version variable mde_version updated from 1.3.3 to 1.3.4:

    tf
    variable "mde_version" {
      type    = string
      default = "1.3.4"
    }

• Fix (300404657): Malformed input to ops-log not stored in BigQuery due serialization issues.
• Fix (300030725): Bucket references and instances should be optional when parsing messages.

Note: Only GKE services are affected by this release.

Helm charts

• MDE services containers image tag: cdcb206.
• MDE UI image tag: 2.13.2.0.99.7-noauth (same as 1.3.2).

Infrastructure changes: MDE version variable mde_version updated from 1.3.2 to 1.3.3.

    tf
    variable "mde_version" {
      type    = string
      default = "1.3.3"
    }

• Fix (296532636): Pub/Sub Sink did not set a data field for numeric records.
• Fix (297851980): Some parsers failed to save when using specific MDE Whistle functions.
• Fix (298031330): Cloud Storage Reader was deployed with incorrect input subscription.
• Fix (296385152): Additional documentation guidance regarding MDE naming limitations.
• Fix (298579836): Cloud Storage ingestion specs mismatch between cols in spec and CSV file could cause endless retries.
• Fix (299418960): Documentation deployment guide added.
• Fix (298907994): Pub/Sub topics not used for 30 days would expire and cause issues if needed later. Expiration removed on all topics.
• Fix (299275899): MC and MDE web interface unable to create new types if system had not config added using API first.
• Fix (299229076, 298609054, 298609075): Ingestion pipeline retry logic updated to better handle misconfigurations.

Note: All changes are either in Dataflow jobs or GKE services.

Helm charts

• MDE services containers image tag: 3bf1635.
• MDE UI image tag: 2.13.2.0.99.7-noauth.

Infrastructure changes

• You must update the file: deployment/terraform/modules/deployment/dataflow-gcs-reader.tf if you have modified the Terraform. The specific change is related to the inputSubscription.
• MDE version variable mde_version updated from 1.3.1 to 1.3.2.
• All Pub/Sub topics no longer have any expiration time set See the Terraform for details. If you have your own Terraform you should also remove the expiration from all relevant topics.
• As a result of Fix (298907994) you may see Terraform plan report that a number of Pub/Sub topics would be created. This only happens if they have expired.

• Fix (294525580): Disable message class doesn't actually disable the message class in the system.
• Fix (294809257, 294995452, 295141703): Missing bucket or natural key references can cause Metadata Manager service to spam the Operations Log tables in BigQuery and potentially overload the ingestion pipeline.
• Fix (294819318): When cloning LookML repository, it could fail.
• Fix (294981315): Helm chart for adding LB to MC update to use the correct resource label.
• Fix (295163432): When deploying MDE in some Google Cloud project, the deployment with terraform apply would recreate the GKE cluster with each apply.
• Fix (295548129): Old validation script from v.1.2.0 removed from package.
• Fix (294807712): MDE web interface had a wrong name in the title.

Helm charts

• Containers image tag: 456bde9.

Infrastructure changes

• Terraform IDs are use for reference not the actual names of the resources. For the full set of changes make sure the review the complete terraform code in the package.
• MDE version variable mde_version updated from 1.3.0 to 1.3.1.
• Dead letter topic added to the following subscriptions and topics.
• Pub/Sub subscriptions mentioned earlier have dead letter policy applied.
• GKE lifecycle has the following ignore changes added.

• Refined the metadata bucket configuration entity by distinguishing between Tag and Record buckets and allow for data contextualization at different aggregation levels.
• Improved query experience and performance in BigQuery by normalizing metadata, allowing efficient retrieval of metadata without needing to unnest records, and enabling new query patterns.
• Improved support for Bigtable sink, allowing all archetypes to be stored in Bigtable.
• Improved query performance by associating each type with a dedicated table in BigQuery to better take advantage of table partitioning.
• Tag names are now global, and parsers targeting different types can emit records referencing the same tag name.
• Improved Federation API surface that enables new data access patterns for all archetypes.
• The number of required fields in a parser has been reduced.
• Improved logging and introduced the ability to set logging levels.
• Improved observability and monitoring by exposing new Cloud Monitoring metrics.
• Internal performance improvements.
• Improved documentation, deployment and configuration guides, as well as improved access and distribution to documentation content.
• Reduced the cost by 20% for smaller instances of MDE by migrating key data processing workloads from Dataflow to GKE.
• Improved the default configuration package by providing better integration with MCe and enabling the ingestion of any data stream with minimal structural requirements. The configuration package now supports the ingestion of data arrays and complex value objects from MCe.
• Upgraded to Whistle v1.4.0.
• Improved transformation model: added the capability to apply several versions of the same transformation to the same Type.

• Metadata buckets support only a single schema and the schema is a property of the bucket.
• Configuration Manager API has been refactored into configuration and Metadata management API, and the API surface has been simplified to align with Google API design and general REST API best practices.
• BigQuery schema changes:
  • Removed default archetype tables.
  • Removed sfp_data dataset.
  • Improved storage footprint by removing fields from abstract base schemas of types (archetype schemas). See the schema changes in the v1.3 reference.
  • Changed field naming style from camel case to snakecase.
  • Configuration model concepts have been refined and renamed.
  • Simplified proto record schemas by removing redundant fields
  • AVRO files containing processed data stored in the Cloud Storage data sink are stored by type.