Deploy an enterprise data management and analytics platform

Stay organized with collections Save and categorize content based on your preferences.

Resource hierarchy

The architecture uses the resource hierarchy from the enterprise foundations blueprint.

Networking

The architecture includes an example data transfer service that uses Workload Identity Federation and a Tink library.

Roles and IAM permissions

The architecture includes segmented data producer roles, data consumer roles, data governance roles, and data platform roles.

Metadata

The architecture uses Data Catalog to manage data metadata.

Central policy management

To manage policies, the architecture uses Google Cloud's implementation of the CDMC framework.

Data access management

To control access to data, the architecture includes an independent process that requires data consumers to request access to data assets from the data owner.

Data quality

The architecture uses the Cloud Data Quality Engine to define and run data quality rules on specified table columns, measuring data quality based on metrics like correctness and completeness.

Data security

The architecture uses tagging, encryption, masking, tokenization, and IAM controls to provide data security.

Data environments

The architecture includes three environments. Two environments (non-production and production) are operational environments that are driven by pipelines. One environment (development) is an interactive environment.

Data owners

Data owners ingest, process, expose, and grant access to data assets.

Data consumers

Data consumers request access to data assets.

Pipelines

The architecture uses the following pipelines to deploy resources:

• Foundation pipeline
• Infrastructure pipeline
• Artifact pipelines
• Service Catalog pipeline

Repositories

Each pipeline uses a separate repository to enable segregation of responsibility.

Process flow

The process requires that changes to the production environment include a submitter and an approver.

Data product scorecards

The Report Engine generates data product scorecards.

Cloud Logging

The architecture uses the logging infrastructure from the enterprise foundations blueprint.

Cloud Monitoring

The architecture uses the monitoring infrastructure from the enterprise foundations blueprint.

data-mesh-ops@example.com

Overall administrators of the data mesh

roles/owner (data platform)

gcp-dm-governance-admins@example.com

Administrators of the data governance project

roles/owner on the data governance project

gcp-dm-governance-developers@example.com

Developers who build and maintain the data governance components

Multiple roles on the data governance project, including roles/viewer, BigQuery roles, and Data Catalog roles

gcp-dm-governance-data-readers@example.com

Readers of data governance information

roles/viewer

gcp-dm-governance-security-administrator@example.com

Security administrators of the governance project

roles/orgpolicy.policyAdmin and roles/iam.securityReviewer

gcp-dm-governance-tag-template-users@example.com

Group with permission to use tag templates

roles/datacatalog.tagTemplateUser

gcp-dm-governance-tag-users@example.com

Group with permission to use tag templates and add tags

roles/datacatalog.tagTemplateUser and roles/datacatalog.tagEditor

gcp-dm-governance-scc-notifications@example.com

Service account group for Security Command Center notifications

None. This is a group for membership, and a service account is created with this name, which has the necessary permissions.

gcp-dm-{data_domain_name}-admins@example.com

Administrators of a specific data domain

roles/owner on the data domain project

gcp-dm-{data_domain_name}-developers@example.com

Developers who build and maintain data products within a data domain

Multiple roles on the data domain project, including roles/viewer, BigQuery roles, and Cloud Storage roles

gcp-dm-{data_domain_name}-data-readers@example.com

Readers of the data domain information

roles/viewer

gcp-dm-{data_domain_name}-metadata-editors@{var.domain}

Editors of Data Catalog entries

Roles to edit Data Catalog entries

gcp-dm-{data_domain_name}-data-stewards@example.com

Data stewards for the data domain

Roles to manage metadata and data governance aspects

gcp-dm-consumer-{project_name}-admins@example.com

Administrators of a specific consumer project

roles/owner on consumer project

gcp-dm-consumer-{project_name}-developers@example.com

Developers working within a consumer project

Multiple roles on the consumer project, including roles/viewer and BigQuery roles

gcp-dm-consumer-{project_name}-data-readers@example.com

Readers of the consumer project information

roles/viewer

common

prj-c-artifact-pipeline

Contains the deployment pipeline that's used to build out the code artifacts of the architecture.

prj-c-service-catalog

Contains the infrastructure used by the Service Catalog to deploy resources in the interactive environment.

prj-c-datagovernance

Contains all the resources used by Google Cloud's implementation of the CDMC framework.

development

fldr-d-dataplatform

Contains the projects and resources of the data platform for developing use cases in interactive mode.

non-production

fldr-n-dataplatform

Contains the projects and resources of the data platform for testing use cases that you want to deploy in an operational environment.

production

fldr-p-dataplatform

Contains the projects and resources of the data platform for deployment into production.

Producers

Contains the data domains.

Consumers

Contains the consumer projects.

Data domain

Contains the projects associated with a particular domain.

Ingestion

The ingestion project ingests data into the data domain. The architecture provides examples of how you can stream data into BigQuery, Cloud Storage, and Pub/Sub. The ingestion project also contains examples of Dataflow and Cloud Composer that you can use to orchestrate the transformation and movement of ingested data.

Non-confidential

The non-confidential project contains data that has been de-identified. You can mask, containerize, encrypt, tokenize, or obfuscate data. Use policy tags to control how the data is presented.

Confidential

The confidential project contains plaintext data. You can control access through IAM permissions.

Key management

Cloud KMS

A service that securely manages encryption keys that protect sensitive data.

Record Manager

Cloud Run

An application that maintains comprehensive logs and records of data processing activities, ensuring organizations can track and audit data usage.

Archiving policy

BigQuery

A BigQuery table that contains the storage policy for data.

Entitlements

BigQuery

A BigQuery table that stores information about who can access sensitive data. This table ensures that only authorized users can access specific data based on their roles and privileges.

Data loss

Sensitive Data Protection

Service used to inspect assets for sensitive data.

DLP findings

BigQuery

A BigQuery table that catalogs data classifications within the data platform.

Policies

BigQuery

A BigQuery table that contains consistent data governance practices (for example, data access types).

Billing export

BigQuery

A table that stores cost information that is exported from Cloud Billing to enable the analysis of cost metrics that are associated with data assets.

Cloud Data Quality Engine

Cloud Run

An application that runs data quality checks for tables and columns.

Data quality findings

BigQuery

A BigQuery table that records identified discrepancies between the defined data quality rules and the actual quality of the data assets.

Scheduler

Cloud Scheduler

A service that controls when the Cloud Data Quality Engine runs and when the Sensitive Data Protection inspection occurs.

Report Engine

Cloud Run

An application that generates reports that help track and measure adherence to the CDMC framework's controls.

Findings and assets

BigQuery and Pub/Sub

A BigQuery report of discrepancies or inconsistencies in data management controls, such as missing tags, incorrect classifications, or non-compliant storage locations.

Tag exports

BigQuery

A BigQuery table that contains extracted tag information from Data Catalog.

Policy management

Organization Policy Service

A service that defines and enforces restrictions on where data can be stored geographically.

Attribute-based access policies

Access Context Manager

A service that defines and enforces granular, attribute-based access policies so that only authorized users from permitted locations and devices can access sensitive information.

Metadata

Data Catalog

A service that stores metadata information about the tables that are in use in the data mesh.

Tag Engine

Cloud Run

An application that adds tags to data in BigQuery tables.

CDMC reports

Looker Studio

Dashboards that let your analysts view reports that were generated by the CDMC architecture engines.

Data control compliance

The Report Engine detects non-compliant data assets through and publishes findings to a Pub/Sub topic. These findings are also loaded into BigQuery for reporting using Looker Studio.

Data ownership is established for both migrated and cloud-generated data

Data Catalog automatically captures technical metadata from BigQuery. Tag Engine applies business metadata tags like owner name and sensitivity level from a reference table, which helps ensure that all sensitive data is tagged with owner information for compliance. This automated tagging process helps provide data governance and compliance by identifying and labeling sensitive data with the appropriate owner information.

Data sourcing and consumption are governed and supported by automation

Data Catalog classifies data assets by tagging them with an is_authoritative flag when they are an authoritative source. Data Catalog automatically stores the information, along with the technical metadata, in a data register. The Report Engine and the Tag Engine can validate and report the data register of authoritative sources using Pub/Sub.

Data sovereignty and cross-border data movement are managed

Organization Policy Service defines permitted storage regions for data assets and Access Context Manager restricts access based on user location. Data Catalog stores the approved storage locations as metadata tags. Report Engine compares these tags against the actual location of the data assets in BigQuery and publishes any discrepancies as findings using Pub/Sub. Security Command Center provides an additional layer of monitoring by generating vulnerability findings if data is stored or accessed outside the defined policies.

Data catalogs are implemented, used, and interoperable

Data Catalog stores and updates the technical metadata for all BigQuery data assets, effectively creating a continuously synchronized Data Catalog. Data Catalog ensures that any new or modified tables and views are immediately added to the catalog, maintaining an up-to-date inventory of data assets.

Data classifications are defined and used

Sensitive Data Protection inspects BigQuery data and identifies sensitive information types. These findings are then ranked based on a classification reference table, and the highest sensitivity level is assigned as a tag in Data Catalog at the column and table levels. Tag Engine manages this process by updating the Data Catalog with sensitivity tags whenever new data assets are added or existing ones are modified. This process ensures a constantly updated classification of data based on sensitivity, which you can monitor and report on using Pub/Sub and integrated reporting tools.

Data entitlements are managed, enforced, and tracked

BigQuery policy tags control access to sensitive data at the column level, ensuring only authorized users can access specific data based on their assigned policy tag. IAM manages overall access to the data warehouse, while Data Catalog stores sensitivity classifications. Regular checks are performed to ensure all sensitive data has corresponding policy tags, with any discrepancies reported using Pub/Sub for remediation.

Ethical access, use, and outcomes of data are managed

Data sharing agreements for both providers and consumers are stored in a dedicated BigQuery data warehouse to control consumption purposes. Data Catalog labels data assets with the provider agreement information, while consumer agreements are linked to IAM bindings for access control. Query labels enforce consumption purposes, requiring consumers to specify a valid purpose when querying sensitive data, which is validated against their entitlements in BigQuery. An audit trail in BigQuery tracks all data access and ensures compliance with the data sharing agreements.

Data is secured, and controls are evidenced

Google's default encryption at rest helps protect data that is stored on disk. Cloud KMS supports customer-managed encryption keys (CMEK) for enhanced key management. BigQuery implements column-level dynamic data masking for de-identification and supports application-level de-identification during data ingestion. Data Catalog stores metadata tags for encryption and de-identification techniques that are applied to data assets. Automated checks ensure that the encryption and de-identification methods align with predefined security policies, with any discrepancies that are reported as findings using Pub/Sub.

A data privacy framework is defined and operational

Data Catalog tags sensitive data assets with relevant information for impact assessment, such as subject location and assessment report links. Tag Engine applies these tags based on data sensitivity and a policy table in BigQuery, which defines the assessment requirements based on data and subject residency. This automated tagging process allows for continuous monitoring and reporting of compliance with impact assessment requirements, ensuring that data protection impact assessments (DPIAs) or protection impact assessment (PIAs) are conducted when necessary.

The data lifecycle is planned and managed

Data Catalog labels data assets with retention policies, specifying retention periods and expiration actions (such as archive or purge). Record Manager automates the enforcement of these policies by purging or archiving BigQuery tables based on the defined tags. This enforcement ensures adherence to the data lifecycle policies and maintains compliance with data retention requirements, with any discrepancies detected and reported using Pub/Sub.

Data quality is managed

The Cloud Data Quality Engine defines and runs data quality rules on specified table columns, measuring data quality based on metrics like correctness and completeness. Results from these checks, including success percentages and thresholds, are stored as tags in Data Catalog. Storing these results allows for continuous monitoring and reporting of data quality, with any issues or deviations from acceptable thresholds published as findings using Pub/Sub.

Cost management principles are established and applied

Data Catalog stores cost-related metrics for data assets, such as query costs, storage costs, and data egress costs, which are calculated using billing information exported from Cloud Billing to BigQuery. Storing cost-related metrics allows for comprehensive cost tracking and analysis, ensuring adherence to cost policies and efficient resource utilization, with any anomalies reported using Pub/Sub.

Data provenance and lineage are understood

Data Catalog's built-in data lineage features track the provenance and lineage of data assets, visually representing the flow of data. Additionally, data ingestion scripts identify and tag the original source of the data in Data Catalog, enhancing the traceability of data back to its origin.

Foundation pipeline

Bootstrap

• Data platform folder and subfolders
• Common projects
• Infrastructure pipeline service account
• Cloud Build trigger for the Infrastructure pipeline
• Shared VPC
• VPC Service Control perimeter

Infrastructure pipeline

Foundation pipeline

• Consumer projects
• Service Catalog service account
• The Cloud Build trigger for the Service Catalog pipeline
• Artifact pipeline service account
• The Cloud Build trigger for the artifact pipeline

Service Catalog pipeline

Infrastructure pipeline

• Resources deployed in the Service Catalog bucket

Artifact pipelines

Infrastructure pipeline

Artifact pipelines produce the various containers and other components of the codebase used by the data mesh.

CDMC framework

Google Cloud CDMC implementation

Provides a governance framework that helps secure, manage and control your data assets. See CDMC Key Controls Framework for more information.

Deployment

Infrastructure pipeline

Provides a series of pipelines that deploy infrastructure, build containers, and create data pipelines. The use of pipelines allows for auditability, traceability, and repeatability.

Artifact pipeline

Deploys various components not deployed by the infrastructure pipeline.

Terraform templates

Builds out the system infrastructure.

Open Policy Agent

Helps ensure that the platform conforms to selected policies.

Network

Private Service Connect

Provides data exfiltration protections around the architecture resources at the API layer and the IP layer. Lets you communicate with Google Cloud APIs using private IP addresses so that you can avoid exposing traffic to the internet.

VPC network with private IP addresses

Helps remove exposure to internet-facing threats.

VPC Service Controls

Helps protect sensitive resources against data exfiltration.

Firewall

Helps protect the VPC network against unauthorized access.

Access management

Access Context Manager

Controls who can access what resources and helps prevent unauthorized use of your resources.

Workload Identity Federation

Removes the need for external credentials to transfer data onto the platform from on-premises environments.

Data Catalog

Provides an index of assets available to users.

IAM

Provides fine-grained access.

Encryption

Cloud KMS

Lets you manage your encryption keys and secrets, and help protect your data through encryption at rest and encryption in transit.

Secrets Manager

Provides a secret store for pipelines that are controlled by IAM.

Encryption at rest

By default, Google Cloud encrypts data at rest.

Encryption in transit

By default, Google Cloud encrypts data in transit.

Detective

Security Command Center

Helps you to detect misconfigurations and malicious activity in your Google Cloud organization.

Continuous architecture

Continually checks your Google Cloud organization against a series of OPA policies that you have defined.

IAM Recommender

Analyzes user permissions and provides suggestions about reducing permissions to help enforce the principle of least privilege.

Firewall Insights

Analyzes firewall rules, identifies overly-permissive firewall rules, and suggests more restrictive firewalls to help strengthen your overall security posture.

Cloud Logging

Provides visibility into system activity and helps enable the detection of anomalies and malicious activity.

Cloud Monitoring

Tracks key signals and events that can help identify suspicious activity.

Preventative

Organization Policy

Lets you control and restrict actions within your Google Cloud organization.