Step 6: Execute deployment

This page describes the sixth step to deploy Cortex Data Foundation, the core of Cortex Framework. In this step, you execute the deployment of Cortex Data Foundation.

Build process

After configuring the config.json file as described in Step 5: Configure deployment, follow these instructions to build your process.

  1. Run the following command to locate yourself in the cloned repository:

    cd cortex-data-foundation

  2. Run the build command with the target log bucket:

    gcloud builds submit --project EXECUTION_PROJECT\
    --substitutions=_GCS_BUCKET=LOGS_BUCKET

    Replace the following:

    • EXECUTION_PROJECT with the execution project, likely the source project.
    • LOGS_BUCKET with the bucket name for logs storage. Cloud Build Service Account needs access to write them here.

  3. Follow the main build process from looking at the logs in the terminal or in the Cloud Build console, if you have enough permissions. See the following images for more reference.

    Logs progress

    Figure 1. Example of viewing logs progress in the terminal.

    Logs progress

    Figure 2. Example of viewing logs progress in the console.

  4. Track the child build steps triggered from the Cloud Build console or within the logs created from the steps. See the following images for more reference.

    Child build steps tracking

    Figure 3. Example of tracking child build steps in the console.

    Child build steps tracking

    Figure 4. Example of tracking child build steps within the logs.

  5. Identify any issues with individual builds. Correct errors, if any. It's recommended to paste the generated SQL into BigQuery to identify and correct the errors. Most errors are related to fields that are selected, but not present in the replicated source. The BigQuery UI helps to identify and comment those out.

    Identifying issues

    Figure 5. Example of identifying issues through Cloud Build logs.

Move files to the Cloud Composer (Airflow) DAG bucket

If you opted to generate integration or CDC files and have an instance of Cloud Composer (Airflow), you can move them into their final bucket with the following command:

  gcloud storage -m cp -r  gs://OUTPUT_BUCKET/dags/ gs://COMPOSER_DAG_BUCKET/
  gcloud storage -m cp -r  gs://OUTPUT_BUCKET/data/ gs://COMPOSER_DAG_BUCKET/

Replace the following:

  • OUTPUT_BUCKET with the output bucket.
  • COMPOSER_DAG_BUCKET with the Cloud Composer (Airflow) DAG bucket.

Customize and prepare for upgrade

In addition to the CORTEX-CUSTOMER tags, you might need to further customize the following by committing all of these changes with a clear tag in the code to your own forked or cloned repository:

  • Adding business rules.
  • Adding other datasets and joining them with existing views or tables
  • Reusing the provided templates to call additional APIs.
  • Modifying deployment scripts.
  • Applying further data mesh concepts.
  • Adapting some tables or landed APIs to include additional fields not included in the standard.

Use the command grep -R CORTEX-CUSTOMER to check all ## CORTEX-CUSTOMER comments that you should customize. Many SAP and Salesforce customers have specific customizations of their systems, such as additional documents in a flow or specific types of a record. These are specific to each customer and configured by functional analysts as the business needs arise.

Adopt a CI/CD pipeline that works for your organization to keep these enhancements tested and your overall solution in a reliable and robust state. A pipeline can reuse the cloudbuild.yaml scripts to trigger end-to-end deployment periodically, or based on git operations depending on your repository of choice by automating builds.

Use theconfig.json file to define different sets of projects and datasets for development, staging, and production environments. Use automated testing with your own sample data to ensure the models always produce what you expect.

Tagging your own changes visibly in your fork or clone of a repository together with some deployment and testing automation helps perform upgrades.

Support

If you encounter any issues or have feature requests related to these models or deployers, create an issue in the Cortex Data Foundation repository. To assist with gathering the necessary information, execute support.sh from the cloned directory. This script guides you through a series of steps to help troubleshoot.

For any Cortex Framework requests or issues, go to the support section in the overview page.

What's next?

Looker prebuilt blocks, also known as Looker Blocks, are essentially reusable data models for common analytical patterns and data sources. See available Cortex Looker deployment dashboards for different workloads: