Cortex for Meridian

This page details the process of data reparation and cloud automation for Google Meridian. Cortex Framework for Meridian streamlines the open-source marketing mix modeling (MMM) with cross-media and sales data. Cortex Framework simplifies this process by providing pre-configured data models and automating Meridian open source model execution using Google Cloud services like Colab Enterprise and Workflows.

One of the key value propositions of Google Cloud Cortex Framework is to provide a data and Artificial Intelligence (AI) foundation for next generation enterprise intelligence that enables analytics spanning across key areas such as sales, marketing, order fulfillment, and inventory management.

Cortex Framework for marketing provides cross media platform key performance indicators (KPIs) and metrics. These metrics are a significant part of the pre-modeling data preparation step for running Google's latest open source MMM called Meridian. Advertisers, agencies, and partners can accelerate the pre-modeling data preparation process by leveraging Google Cloud Cortex Framework Data Foundation.

Cortex for Meridian simplifies the pre-modeling process by efficiently gathering and transforming data from core Cortex Framework data sources, including:

• Cross Media & Product Connected Insights.
• SAP (ECC and S/4 HANA)
• Oracle EBS

For more information, see Meridian documentation.

Configuration file

During notebook execution, the system retrieves configuration parameters from the cortex_meridian_config.json file located in the configuration folder within Cloud Storage.

The following section shares different configuration YAML files examples for Meridian execution:

{
  "cortex_bq_project_id": "PROJECT_ID",
  "cortex_meridian_marketing_data_set_id": "K9_REPORTING",
  "cortex_meridian_marketing_view_name": "CrossMediaSalesInsightsWeeklyAgg",
  "column_mappings": {
      "controls": [],
      "geo": "geo",
      "kpi": "number_of_sales_orders",
      "media": [
          "Tiktok_impression",
          "Meta_impression",
          "YouTube_impression",
          "GoogleAds_impression"
      ],
      "media_spend": [
          "Tiktok_spend",
          "Meta_spend",
          "YouTube_spend",
          "GoogleAds_spend"
      ],
      "population": "population",
      "revenue_per_kpi": "average_revenue_per_sales_order",
      "time": "time"
  },
  "channel_names": [
      "TikTok",
      "Meta",
      "YouTube",
      "GoogleAds"
  ],
  "data_processing": {
      "kpi_type": "{USE_CASE_SPECIFIC}",
      "roi_mu": {USE_CASE_SPECIFIC},
      "roi_sigma": {USE_CASE_SPECIFIC},
      "sample": {
          "prior": {USE_CASE_SPECIFIC},
          "posterior": {
              "n_chains": {USE_CASE_SPECIFIC},
              "n_adapt": {USE_CASE_SPECIFIC},
              "n_burnin": {USE_CASE_SPECIFIC},
              "n_keep": {USE_CASE_SPECIFIC}
          }
      }
  }
}

...
    "kpi": "conversions",
    "revenue_per_kpi": "",
...

The following table describes the value for each configuration parameter from the cortex_meridian_config.json file:

Compatibility with Meridian

Cortex Framework Data Foundation and Meridian are released separately. The Cortex Framework release notes provide an overview of its releases and versions. On Meridian's GitHub repository you can see the latest available Meridian versions. Meridian prerequisites and system recommendations are available on Meridian's user guide.

Cortex Framework Data Foundation releases are tested with a specific version of Meridian. You can find the compatible Meridian within the Jupyter notebook, as following image shows:

To update to a newer Meridian version, modify the corresponding line in the notebook. Consider that additional code adjustments may be required in the notebook.

Data Model

This section describes the CrossMediaSalesInsightsWeeklyAgg Data Model using the Entity Relationship Diagram (ERD).

Cortex for Meridian relies on a single view, CrossMediaSalesInsightsWeeklyAgg, to operate. The data source for this view is determined by the k9.Meridian.salesDataSourceType configuration setting, which can be:

• BYOD (Bring Your Own Data): Custom data integration.
• SAP_SALES: Sales data from SAP systems.
• ORACLE_SALES: Sales data from Oracle EBS systems.

The following section shares the Entity Relationship Diagrams for CrossMediaForMeridian:

The following table shows the detailed schema for the CrossMediaSalesInsightsWeeklyAgg view which is part of Cortex for Meridian:

Deployment

This page outlines the steps to deploy Cortex Framework for Meridian, enabling the best-in-class MMM within your Google Cloud environment.

For quickstart demo, see Quickstart demo for Meridian.

Architecture

Cortex for Meridian uses Cortex Framework for marketing and Cross Media data combined with sales data. You can bring sales data from either Oracle EBS, SAP, or another source system.

The following diagram describes the key components of Cortex for Meridian:

Meridian components and services

During the deployment of Cortex Framework Data Foundation (see deployment prerequisites), you can enable Cortex for Meridian by setting deployMeridian to true in the config.json file. This option initiates an additional Cloud Build pipeline, which installs the following components and services required for Meridian:

• BigQuery view: A view is created in the K9 reporting dataset called CrossMediaSalesInsightsWeeklyAgg. This enables querying marketing data and sales data from Cortex Framework. The actual implementation of the view and the underlying sources are dependent on the sales data source you select during deployment.

• Cloud Storage bucket: The PROJECT_ID-cortex-meridian bucket contains all the artifacts needed by and produced by Cortex for Meridian in the following folders:

  • configuration: Define the settings and parameters for Cortex for Meridian. It's used by the Colab Enterprise notebook during notebook execution.
  • csv: The raw data output from running Meridian will be saved as CSV files here.
  • models: The generated model from running Meridian will be saved here.
  • notebook-run-logs: Notebook copies for each executions and logs will be saved here.
  • notebooks: Contains the main notebook with code and logic for running Cortex for Meridian. This notebook is intended for further customization to support your specific needs and requirements.
  • reporting: This is the folder where reports from Meridian executions will be saved. Also contains an HTML template for generating an overview report with links to report output from Meridian.

• Colab Enterprise: Colab Enterprise is a managed service on Google Cloud that provides a secure and collaborative environment for data science and machine learning workflows using Jupyter notebooks. It offers features like managed infrastructure, enterprise-level security controls, and integration with other Google Cloud services, making it suitable for teams working with sensitive data and requiring robust governance. A managed environment for running the Jupyter notebook.

Cortex for Meridian uses Colab Enterprise to define a runtime template with the required infrastructure to automate Meridian runs.

When triggering the end to end pipeline, using workflow, an Execution is created. This will run a copy of the current Jupyter notebook from Cloud Storage with the latest configuration.

• Workflow: A Cloud Workflow called cortex-meridian-execute-notebook orchestrates the execution of the full Cortex for Meridian pipeline. The workflow will call the Colab Enterprise API that creates a runtime based on the runtime template and executes a notebook run with the current configurations, and finally saves all the results into Cloud Storage.

  

  You have two optional configuration options available for the workflow:

  1. If you can provide a new Cortex for Meridian JSON config as input to the workflow. If you do so the flow will take a backup of the old config and update the config with your input. See REPLACE for more information.

  

  1. The pre_notebook_execution step provides an excellent place for you to initiate any additional tasks you need to automate before running the notebook. For example loading data from sources outside of Cortex.

• Service Account: A dedicated Service Account is required to be provided during deployment. This is needed for executing the workflow and the notebook in Colab Enterprise.

Additional deployment parameters for Meridian

The config.json file configures the settings required to run Meridian with Cortex Framework. This file contains the following parameters for Cortex for Meridian:

   "k9": {
...
        "deployMeridian": false,
...

    "Meridian":{
            "salesDataSourceType": "",
            "salesDatasetID":"",
            "deploymentType": "",
            "defaultNotebookFile":"meridian_cortex_marketing.ipynb",
            "defaultConfigFile":"cortex_meridian_config.json",
            "gcsBucketNameSuffix": "cortex-meridian",
            "workflow": {
                "template": "create_notebook_execution_run.yaml",
                "name": "cortex-meridian-execute-notebook",
                "region": "us-central1"
            },
            "runnerServiceAccount": "cortex-meridian-colab-runner",
            "colabEnterprise": {
                "region": "us-central1",
                "runtimeTemplateName": "cortex-meridian-template",
                "runtimeMachine_type": "n1-highmem-32",
                "runtimeAcceleratorCoreCount": 1,
                "runtimeAcceleratorType": "NVIDIA_TESLA_T4",
                "executionName": "cortex-meridian-execution",
                "notebookRunLogsFolder": "notebook-run-logs"
            }
        }
}

The following table describes the value and description for each Meridian parameter:

Workflow

Workflows serve as the primary interface for initiating Cortex for Meridian executions. A default workflow called cortex-meridian-execute-notebook is deployed as part of Cortex for Meridian.

Notebook execution

To start a new Cortex for Meridian execution follow these steps:

1. Go to the cortex-meridian-execute-notebook notebook in Workflows.
2. Click Execute to start a new execution.
3. For initial runs, leave the input field empty to use the default configuration stored in the cortex_meridian_config.json configuration file in Cloud Storage.
4. Click Execute again to proceed.

5. After a brief delay, the workflow execution status will be displayed:

   

6. Track the notebook execution progress in Colab Enterprise.

Workflow steps

The cortex-meridian-execute-notebook workflow contains the following steps:

Customize Meridian execution workflow

You can customize the Meridian execution by providing your own configuration JSON file in the Workflows input field:

1. Input the full JSON of the modified configuration into the input field.
2. The workflow will then:
   1. Replace the existing cortex_meridian_config.json file in Cloud Storage with the provided JSON.
   2. Create a backup of the original configuration file in the Cloud Storage/configuration directory.
   3. The backup filename will follow the format cortex_meridian_config_workflow_backup_workflow_execution_id.json, where workflow_execution_id is an uniquely identifier for the current workflow execution (for example, cortex_meridian_config_workflow_backup_3e3a5290-fac0-4d51-be5a-19b55b2545de.json)

Jupyter notebook overview

The core functionality of loading input data to run and execute Meridian model is handled by the Python notebook meridian_cortex_marketing.ipynb, located in the notebooks folder of your Cloud Storage bucket.

The notebook's execution flow consists of the following steps:

1. Install necessary packages (including Meridian) and imports required libraries.
2. Load helper functions for interacting with Cloud Storage and BigQuery.
3. Retrieves execution configuration from the configuration/cortex_meridian_config.json file in Cloud Storage.
4. Load Cortex Framework data from the Cortex Framework Data Foundation view within BigQuery.
5. Configure the Meridian model specification and map Cortex Framework Data Foundation data models for marketing and sales to Meridian model input schema.
6. Execute Meridian sampling and generates a summary report, saved to Cloud Storage (/reporting).
7. Run the budget optimizer for default scenario and output summary report to Cloud Storage (/reporting).
8. Save model to Cloud Storage (/models).
9. Save CSV results to Cloud Storage (/csv).
10. Generate an overview report and save it to Cloud Storage (/reporting).

Import notebook for manual execution and editing

To customize or manually execute the notebook, import it from Cloud Storage:

1. Go to Colab Enterprise.
2. Click My notebooks.
3. Click Import.
4. Select Cloud Storage as import source and select the notebook from Cloud Storage.
5. Click Import.

6. The notebook will load and open.

Notebook executions results

To review notebook run results, open a full copy of the notebook with all cell outputs:

1. Go to Executions in Colab Enterprise.
2. Select the relevant region from the drop-down.
3. Next to the notebook run that you want to view results for, click View result.
4. Colab Enterprise will open the result of the notebook run in a new tab.
5. To view the result, click the new tab.

Runtime template

Google Cloud Colab Enterprise uses runtime templates to define pre-configured execution environments. A predefined runtime template, suitable for running the Meridian notebook, is included with the Cortex for Meridian deployment. This template is automatically used to create runtime environments for notebook executions.

If needed, you can manually create additional runtime templates.