Release notes

Release 6.0.1

• Quickstart demo updates for permissions with legacy accounts.
• CATGAP Python library vulnerability updates.

GitHub Submodules

Submodules are officially removed. Workloads have been migrated as follows:

Note: As announced in release 5.4, workload specific repositories will continue to receive code updates for at least one more release.

SAP

• SAP Raw to CDC DAG: Rewrote logic and became less costly with about 25% improvement in performance.
• Fiscal dimension: All views now use the new Fiscal Dimension instead of function. ECC/S4 specific Fiscal Functions are now removed as announced in v3.1. Fiscal functions will be removed in the next version.
• Future deprecation: Legacy Currency_Conversion and Currency_Decimal functions will be deprecated in the next version. Join the views currency_conversion and currency_decimal instead.
• Finance: New views for FSV, Cost, and Profit center hierarchy that supersedes the old hier_reader local k9, which will be removed in the next version. For more information, see Cortex Framework: integration with SAP.
• Improved SalesStatus_Items and S/4 Billing logic.

• Inventory: Moved the following views and functions from src/SAP/SAP_REPORTING/local_k9/inventory_snapshots to src/SAP/SAP_REPORTING/{SAP_FLAVOR}; and renamed them to make the structure more logical and succinct. Also added notes and instructions on this module in the Cortex Framework: integration with SAP . There isn't any content or logic change.

  • From 00_stock_characteristics_config to StockCharacteristicsConfig.
  • From 00_slow_moving_threshold to SlowMovingThreshold.
  • From 0_{weekly|monthly}_inventory_aggregation.sql to Aggregate{Weekly|Monthly}Inventory.
  • From stock_weekly_snapshots_inventory_aggregation_update to UpdateMonthlyInventoryAggregation.
  • From 01_stock_{weekly|monthly}_snapshots to Update{Weekly|Monthly}StockSnapshots.

Data Mesh

• Added support for annotations on nested fields.
• Removed inherited default values for lake regions and zone location types. These values need to be provided if deploying lakes and zones.

Release 5.4.2

This release changes the Cloud Build default service account behavior, and now it has one of the two types of service accounts by default:

• The legacy Cloud Build service account (project-number@cloudbuild.gserviceaccount.com).
• The project's compute service account (project-number-compute@developer.gserviceaccount.com).

Consequently, the Quickstart demo deployment is updated to use either default service account. For more information, see Cloud Build Service Account Change.

Release 5.4.1

• Realigned table schema directory location for Google Ads, CM360 and TikTok, by moving it from src/marketing/src/SOURCE/src/table_schema to src/marketing/src/SOURCE/config/table_schema matching other marketing sources.
• Reporting table refresh operation is now atomic, which ensures data is available in reporting during table refresh.

Google Ads

• Renamed Keywords view to KeywordStats.
• Added new reporting views to support campaign level metrics by country (CampaignStats, CampaignStatsByUserCountry, GeoTargetConstant), and new experimental views (CampaignDailyAgg, CampaignDailyAggByUserCountry).
• Enabled auto discovery for all accessible customer IDs. Therefore login_customer_id no longer needs to be explicitly specified in the Secret Manager configuration.
• Improved readability for raw view queries.

Meta

• Updated source raw request specification format to YAML. Required fields can now be specified in a readable format.
• Maximum load days and per request delay can be configured in src/marketing/src/Meta/src/raw/pipelines/config.ini.
• Added table annotations for reporting views.
• Fixed a bug for source raw load date range, causing data to be backfilled multiple times in edge cases.
• Included performance optimizations of the hierarchy flattener code for SAP Finance Financial Statement Version, Profit Center, and Cost Center hierarchies.

K9

• Converted scores to INTEGER type from STRING based on source definition. Sustainability views are now GA.
• Holiday Calendar updated to correctly load data for most recent years.
• Added support for polymorphic fields to Salesforce Sales Cloud (SFDC).
• SFDC encoding text is now configurable in src/SFDC/src/raw_dag_generator/dependencies/config.ini. Default value is UTF-8.

Repository READMEs

• Moved SAP and Salesforce Sales Cloud (SFDC) content from main README to their respective README files.
• Added Google BigQuery Connector for SAP configuration considerations to the README_SAP.

Release 5.4

Plan migration

• Cortex Framework employs multiple submodule repositories, however the Cortex Data Foundation repository would no longer contain submodules.
• Cloud Composer 1 is entering in post-maintenance mode, and Cortex support for Airflow 1 and Cloud Composer 1 is ending. Plan to upgrade to Airflow 2 and Cloud Composer 2. For more information, see Migrate environments to Cloud Composer 2.

Marketing

• New Meta data source: Measure and understand Meta (Facebook and Instagram) paid media campaign performance with new BigQuery integration templates, data models, and a Looker Semantic Model with sample dashboards for further analytics.
• New Salesforce Marketing Cloud (SFMC) data source: Measure and understand SFMC email marketing campaign performance with new BigQuery integration templates, data models, and Looker Semantic Model with sample dashboards for further analytics.
• Support for Google Ads API updated to v15.

SAP Finance

• Added New Profit and Loss view (ProfitAndLoss) that shows company's revenues, expenses, and profits. Also demonstrating Financial Statement Version (FSV), cost-center, and profit-center hierarchy.
• Added sample Financial Income Statement Dashboards for Profit and Loss view to the Looker Semantic Model for Cortex SAP Framework.
• Hierarchy flattener for profit-center and cost-center hierarchy.
• Improved performance and optimization to the financial_statements code, which significantly decreases execution time and resources required for FinancialStatement and BalanceSheet views.

K9 - Sustainability

• Added Intelligent ESG data for Sustainability analysis utilizing Dun and Bradstreet sourced data with Sustainability Looker Studio Dashboard samples. This feature is experimental. The data model is subject to change in subsequent releases.

Other changes

• Now obtain and share a Credly Digital Badge after a successful deployment of Cortex Framework.
• Salesforce (SFDC): Added Line of Business Catalog tags for Data Mesh annotations.
• Updated Cortex Data Foundation principal README. This simplifies searches for deployment content and data sources separately. SAP, SFDC, and Marketing workloads have separate pages now.
• Updated DAG ApacheBeam[gcp] version requirements to 2.53.0. New test harness regions added: me-central1 and me-west1. Addressed a variety of vulnerabilities in Python library versions.
• Known Issues And Limitations from previous release are still in effect.
• It's expected that only the company code C006 balances in the SAP Finance test harness data.

Release 5.3

• New experimental functionality to deploy Data Mesh definitions leveraging Data Catalog, Dataplex and BigQuery. Current functionality includes definitions for lakes, zones, access policies, and other exciting features. By default, configuration only creates descriptions in BigQuery and doesn't require any additional configuration.
• Experimental Balance Sheet report with financial statement version flattening is available. Due to potential performance issues, the underlying data processing pipeline is under development for optimization and would be improved in a future release based on user input. Review CDC settings, as ACDOCA, BSEG and BKPF have modified clustering and partition settings.
• New persisted and optimized fiscal time dimension.
• Google Ads ingestion pipeline upgrade to Google Ads API version 15.
• Minor modifications to marketing ingestion pipelines for LiveRamp: streamlined imports for Python libraries.
• Directory external_dags for SAP was converted into local K9.
• Modification on AccountsPayable for S4 to match invoice based on reference.

Known issues and limitations

• Local K9 for SAP produces a temporary folder (tmp) in the target bucket for DAGS. Remove it manually. There is no impact in the execution.
• Nested fields aren't supported yet by Data Mesh annotations. They would be incorporated in a follow-up release.
• It's expected that data in the test harness for finance doesn't zero out in the balance sheet because it is incomplete.

Release 5.2

• LiveRamp for Marketing: New integration templates and data models to enable identity resolution by integrating with LiveRamp.
• SAP Reusable DAGs are now deployed via the K9 deployment mechanism so they now behave like other cross-workload DAGs. This applies to all DAGs in the external_dag directory. These DAGs no longer search for SQL files from bq_data_replication directory. If you have modified the SQLs for them, follow these steps to migrate:
  1. Back up any SQL file(s) you have modified from gs://<your_airflow_bucket>/data/bq_data_replication.
  2. If any of the relevant DAG(s) are running, pause them in Airflow.
  3. Deploy Cortex Data Foundations v5.2.
  4. Copy your modified SQL file into the deployment directory, which by default is gs://<your_deployment_bucket>/dags/sap/reporting/<dag_name>.
  5. Copy /dags directory from the deployment bucket into your Airflow bucket. Then, wait for Airflow to refresh, and unpause the DAGs in Airflow.
  6. Remove migrated files from gs://<your_airflow_bucket>/data/bq_data_replication.
• Metadata for SAP and TikTok: New experimental annotations feature for SAP artifacts. The deployer now creates views with descriptions for views and their fields.
• Support for new regions: New supported regions incorporated into the test harness.
• Fix for set hierarchy reader in SAP, generating a node with no associated main data for the header.

Release 5.1

• New TikTok for Marketing: New integration templates and data models for TikTok to measure paid media campaign performance insights, through impressions and reach.
• Placeholders for K9 views are now created for each workload.
• New experimental feature to help gather necessary data to request support after a failed deployment. This feature gathers the logs and the configuration to be sent to cortex-support@google.com.
• Added option to calculate doubtful receivables and days in arrear as positive amount in AccountingDocuments.
• Pull request #9 from SAP Reporting, adding safe divide for Fill rate Percent.

Release 5.0

• Cortex for Marketing, has been added to the Data Foundation. For more information, see the Entity Relationship Diagram (ERD) in the docs folder.
• Created quickstart demo deployment.
• K9 cross-workload and source reusable models.
• Optional materialization and performance optimization enabled for all modules and views. This impacts the SQL templates previously containing a CREATE statement. Check the new default configurations as this attempts to replace some views with tables.
• CATGAP has moved to K9. The deployment is disabled by default.
• The test harness data has now moved from buckets into BigQuery datasets available in all relevant regions. Test harness data isn't provided with any warranty of quality, but this change simplifies and speeds up the deployment for those exploring the framework.
• Substitutions are removed, except for Logs in Cloud Storage bucket. Configuration files are now normalized into config.json. Legacy Dotenv files have been removed.
• DeployCDC is now specific to each workload.
• GenerateExtraData is now removed, the control of this deployment is owned by K9.

SAP

• SAP-specific DAGs, like currency conversion, inventory management or hierarchy reader, were moved to the reporting module and are executed by default.
• Configured reporting configuration for materialization to improve match customer expectations, but it might not apply to your business at all.
• UNION deployment logic has been removed, but supported views continue to have the UNION template until further notice.
• The reporting module relies on the time dimension deployment in K9, it can be executed manually if deploying from the cortex-reporting submodule.

Salesforce

• New currency conversion for Salesforce. Since these are optional components for Salesforce deployments, they can be created through the placeholder settings. If you don't have the currency types and conversion in Salesforce, comment out or remove the calls in SFDC/config/ingestion_settings.yaml. The configuration in cdc_placeholder_settings.yaml now takes care of the dependencies in the reporting views.
• The reporting module relies on the time dimension deployment in K9, it can be executed manually if deploying from the Salesforce submodule.

Known issues and limitations

• SAP Reporting and Salesforce Reporting requires execution of K9 pre-processing if used as a separate submodule.
• ML Models repository still allows for substitutions.
• CM360 and Ads integration scripts are in Experimental mode while we gather early feedback from customers on usage.
• Google Ads DAG is supported on Airflow v2 only.

Release 4.2

• New Supply Chain and Finance views: These views support KPIs for Inventory management, Vendor Performance, Accounts Payable and Spend Analysis. The new objects are:
  • Reusable DAG scripts for inventory snapshots (SAP_CDC/external_dag/inventory_snapshots): Initializer and periodic update for weekly and monthly inventory snapshots and aggregations, slow moving threshold, and stock characteristic configuration. Review for CORTEX-CUSTOMER comments for potential updates needed for custom thresholds and material movements. These scripts are in PREVIEW and would be moved into another structure in deployment in the next major release. These scripts fill the base tables and need to be scheduled for the reporting views to function properly.
  • New views for reporting: AccountsPayableOverview, DaysPayableOutstanding, InventoryByPlant, InventoryKeyMetrics, MaterialLedger, MaterialMovementTypesMD MaterialsBatchMD, POScheduleLine, PurchaseDocumentsHistory, SalesOrderStatus, SlowMovingThreshold, StockCharacteristicsConfig, StockInHand, StockMonthlySnapshots, and StockWeeklySnapshots.
  • New overview views: VendorLeadTimeOverview and VendorPerformanceOverview. These views show the reporting logic used in Looker in case you want to replicate them in another tool or a microservice. These views aren't deployed by default.
• Materializer preview : By default, the new views that require a lot of computation are now deployed as materializing DAGs.
• Cortex Analytics Templates - Google Ads Pipelines (CATGAP) : This new experimental feature uses Natural Language Processing Machine Learning models to intelligently map product categories from Google Ads to SAP's product hierarchy. For more information, see CATGAP for details and further setup.
• Reporting views that expect parameters from currencies in config/config.json now produce the same result as many times as currencies are set as targets. Check for CORTEX-CUSTOMER comments for specific guidance if you deploy the data foundation with more than one currency.
• Parameters that control runtime (for example, DEPLOY_SAP, DEPLOY_SFDC, DEPLOY_CDC, and TEST_DATA) are now read from the config file. These are still defaulted in cloudbuild.yaml substitutions. If you want to use the values in the file, the substitutions section in cloudbuild.yaml needs to be commented out. Substitutions from the command line would be phased out.
• Compatibility for Airflow v1 and v2 updated for currency_conversion DAG.
• Addressing SFDC Issue #2.

Known Issues and Limitations

• If using the SAP Reporting submodule as standalone for deployment, sap_config.json needs to be manually generated from config.env for the materializer to read as input. Dotenv files would be removed in the next major release, and this would be addressed for SAP submodules.
• New views aren't released for UNION mode.

Release 4.1

• More options for Salesforce integration with provided scripts or external tools: End-to-end integration from Salesforce APIs (provided out-of-the-box), structure mapping with no change data capture processing and optional CDC and mapping generation scripts.
• Option for sequential deployment (TURBO=false) for SFDC incorporated.
• Salesforce integration now updates the RAW tables with changes, merging changes on landing. This removes the need for additional CDC processing. Deltas are captured using SystemModstamp provided by Salesforce APIs.
• IsArchived flag is removed from CDC processing for Salesforce.
• Errors originating from gsutil steps in cloudbuild.sfdc.yaml not finding files to copy are now caught and surfaced gracefully.
• Removing some substitution defaults (for example, LOCATION) from cloudbuild.yaml file so all configurations are either passed from the command line or read from config/config.json. These parameters would be removed from substitution defaults in future releases.
• Detecting version for Airflow in DAG templates to use updated libraries for Airflow v2 in SAP and Salesforce. This removes some deprecation warnings, but it might need additional libraries installed in your Airflow instance.
• Fix for test harness data not loading in an intended location when the location isn't passed as a substitution.
• Checking existence of DAG-generated files before attempting to copy with gsutil to avoid errors.
• Structure of RAW landed tables has changed to not require additional DAG processing. Review the documentation about mapping and use of the new extraction process before upgrading to avoid disruption.

Release 4.0

• New Salesforce data source. The module includes optional integration and CDC scripts and reporting views. For more information, see the Entity Relationship Diagram for a list of tables and views and the Looker repository for sample dashboards.
• New configuration file (config/config.json) for deployment parameters. We maintain backward compatibility with the existing gcloud builds submit command parameters. Enhanced parameters like SAP UNION datasets and those for Salesforce, can be configured in config.json. This file format replaces the .env format for SAP_REPORTING in the next releases.
• New preview models for SAP Finance: AccountingDocumentsReceivable, AccountsPayable, AccountsPayableTurnover, CashDiscountUtilization, CurrencyConversion (exposing table processed by DAG), VendorPerformance, and VendorConfig (dependant on TVARVC values).
• New split views for SAP Order-To-Cash: SalesOrderPricing and PricingConditions. For more information, see the updated Looker reports to make use of the split models.

Known issues and limitations

• Salesforce integration DAGs have been tested in Airflow 1.0 and Airflow 2.0. They might require library updates or use of backwards compatible libraries to be tested and confirmed in the next release.
• Finance views for SAP are good candidates for partial or total materialization. Check BigQuery's execution details to identify opportunities to create materialization processes and further optimizations that fit your data best.

Release 3.1

• New configurable partitioning and clustering on deployment of CDC landing tables and scripts. See example in cdc_settings.yaml.
• New date dimension generated from external DAGs through _GEN_EXT=true to allow for more flexibility in reporting. This table has been incorporated into views POSchedule, MaterialsValuation, PurchaseDocuments, Deliveries, Billing, SalesOrderScheduleLine, AccountingDocuments, InvoiceDocuments_Flow, POOrderHistory, and SalesOrders_V2. The structure of the table is generated without data if _GEN_EXT=false. If you don't want to execute the generation of other DAGs for external sources, remove them from the list in the generation script.
• Performance improvements to AccountingDocuments, InvoiceDocuments_Flow,Billing, Deliveries, MaterialsValuation, PurchaseDocuments, POOrderHistory, and SalesOrders_V2 when using currency conversion generated from DAG materializing results (see CORTEX-CUSTOMER tags and differences when merging).
• New preview views modularizing functionality from SalesOrders_V2: OneTouchOrder, SalesOrderHeaderStatus, SalesOrderScheduleLine, SalesOrderPartnerFunction, and SalesOrderDetails_SAMPLE.
• Replacing ECC or S4 specific functions with non-specific (for example, fiscal_Case_ecc or fiscal_case_s4 now is fiscal_case). SQL flavor-specific functions would be commented out from generation in the next release.

Release 3.0

• New submodule structure. All SAP-related submodules have been moved under the directory src/SAP/.
• Submodules now execute independently. SAP's ML Models and SAP Reporting submodules now have their own YAML file for independent execution.
• New experimental UNION logic available if your SAP ECC and S/4 systems don't share the same MANDT. All views are now moved to /ecc or /s4 respectively. This alters the templates of all of the views and requires two sets of datasets (for example, one raw landing for ECC, one for S/4). These datasets can be passed to the deployment in the file sap_config.env.
• SAP Reporting can now be deployed using a configuration file (sap_config.env) instead of parameters from the gcloud builds submit --substitutions command. Parameters received from the substitutions override configurations in the file to maintain backwards compatibility. This affects all the cloudbuild.yaml files.
• New simplified view for Sales Orders (_SalesOrders_V2_) with less underlying joins for use cases that don't require them.
• New experimental views for a collaboration with C3.ai. These views are considered experimental as we increase the functionality around and on top of them in collaboration with C3.ai. Experimental views are flagged with an ## EXPERIMENTAL comment.
• Turbo mode for SAP Reporting: The file sap_config.env contains a variable called TURBO. If set to true, the cloudbuild.yaml file is generated dynamically from the dependencies file. This reduces the deployment time.
• Candidates for enhancement points within the code, that is portions where customers should consider incorporating changes or applying extensions, are flagged with a comment ## CORTEX-CUSTOMER.
• Templatized currency and language for views that require the values to be hardcoded for better performance.
• New non-changing views created as tables by default. Sources that are unlikely to ever change (for example, material document types, valuation areas) are now created as tables, instead of views, by default.
• Added the following new views: WeatherDaily, SalesOrders_V2, DeliveryBlockingReasonsMD, BillingBlockingReasonsMD, StorageLocationsMD, PurchaseDocumentTypesMD, MovementTypesMD, ReasonForMovementTypesMD, SpecialStocksMD, PurchasingOrganizationsMD, BatchesMD, MaterialPlantsMD, ProductionOrders, PurchasingGroupsMD, MaterialsValuation, MaterialsMovement, MaterialTypesMD, BillOfMaterialsMD, DivisionsMD, and ValuationAreasMD.
• Added new DAG for currency conversion and currency decimal shifting.
• When test data is set to true, the dynamically generated MERGE operation now copies the test data from the raw landing dataset into the CDC dataset. Test data in CDC dataset isn't copied as it is from the raw landing. Test data is still meant to provide a base for specific demo scenarios and isn't guaranteed to be a reliable source with any quality or referential integrity.
• Started marking releases with tags. This release is v3.0
• Views for ECC are now matched in S4.
• New guide with recommendations for upgrades.

Release 2.2

• New external datasets available: Holidays, Weather, Trends and product hierarchy flattener to feed the Trends search terms.
• Region-specific deployments with test data are now possible. For more information about supported regions and limitations of regional datasets in BigQuery, see BigQuery locations.
• Addressing records with the same key and operation flag sent more than once in different chunks into raw landing.
• Escaping of table and column names in DAG and real-time view generation to avoid issues with reserved words (Issue #7).
• Minor tweak to CustomersMD: Coalesce address between customer main data and latest record in ADRC.
• Minor correction to OrderToCash: Use CDC dataset as source for VBRP.

Known issues

• Analytics hub supports regions in the EU and US . For more information, see Analytics hub introduction. If deploying the Weather DAG in a specific location where the source linked datasets aren't available, you might need to resort to another source for weather data or create a scheduled query to feed a copy in the same location, and use a transfer service to copy the records into a table in the desired location.

Release 2.1.1

• Fixed AddressesMD date format to 9999-12-31.
• Merged original 800 and 200 demo client tables into test harness for ECC (now it has both 800 and 100). This client was mostly used for demos.
• Commenting out deployment of Product Recommender ML model as it requires specific reservations and a sample dataset that is only offered in the US region.
• Addressing Issue #4.
• Ignoring .INCLUDE when generating DAGs and Runtime views.
• Fixed RUNTIME view logic generation affecting keys with deletion.
• New Looker blocks available replacing the older blocks.

Release 2.1

Features and improvements

• Templatized deployment and SQL Flavour for views, setting.yaml file and test harness to deploy either ECC or S4:
  1. New split source of data with MANDT = 050 for ECC and MANDT = 100 for S/4. These populates automatically based on the _SQL_FLAVOUR parameter in the build (default to ECC).
  2. Original dataset from ECC and S/4 mixed still present in the source bucket (MANDT = 800).
  3. The following views would be adapted to S/4 with source tables in a next minor release: FixedAssets, SDStatus_Items, GLDocumentsHdr, BSID and BSAD implementations, InvoiceDocuments_flow,DeliveriesStatus_PerSalesOrg, SalesFulfillment_perOrder, SalesFulfillment, and UoMUsage_SAMPLE.
• Additional views for ECC and S/4 with Accounts Receivables and On-Time/In-Full analysis: Billing, OrderToCas, and AccountingDocumentsReceivable.
• Additional helper functions for currency conversion.
• Additional function for Due Date calculation for discounts.
• New helper function for Fiscal period.
• Renamed CDC parameters to match Reporting and ML Models.
• SAP Reporting now has a SQL Flavour switch to correctly deploy S4 or ECC based on the _SQL_FLAVOUR parameter. Possible values are ECC or S4.
• Enhanced reporting and calculation logics in: Deliveries, PurchaseDocuments and SalesOrders.
• Reduced CDC deployment copy step by 80% implementing multithreaded copy.
• Optimized overall build time when Cloud Build has the container cached, improving overall deployment time by 50%.

Bug Fixes

• Fix for Runtime views not generated from settings.yaml.
• Deployment dependencies causing errors on view creation.

Known issues

• Hierarchy flattener not configured to find sets in new test data.
• .INCLUDE in some tables in some versions of S/4 is known to be marked as key, which results in invalid SQLs in DAG generation. Recommendation as of now is to clear the keyflag field in the offending .INCLUDE entry in the replicated DD03L table in BigQuery when generating the DAG.

New

• Cortex Application Layer: Google Marketplace offering showcasing how to consume the Cortex Data Foundation in your own cloud-native applications.
• Looker Blocks dashboards for Orders Fulfillment, Order Snapshot, Order Details, Sales Performance, Billing, and Pricing.