Task dependent DAGs

This page outlines the steps to enable task dependent Directed Acyclic Graphs (DAGs) to specify task dependencies between SQL table nodes within a single DAG, rather than relying on cron scheduling across multiple DAGs. Customizable settings are used to generate Cloud Composer DAGs with multiple table refresh nodes that depend on each other. Cortex Framework provides recommended settings for task dependent SAP DAGs (ECC and S/4HANA). However, you can customize them further or define task dependent DAGs for any data sources.

Enable task dependent DAGs

1. Modify the config.json file by setting the enableTaskDependencies field to True. This setting activates the Cortex Framework's search for task-dependent reporting settings files with the suffix _task_dep.yaml.
2. Create a dedicated reporting settings file with the suffix _task_dep.yaml for each data source requiring task dependencies. For more details, see Define task-dependent reporting settings.
3. Customize the task dependencies by adding table_setting.dag_setting as a new section to table type nodes. For more details, see Specify and customize task dependencies.
4. Build Cortex Framework using the standard build process.
5. Examine the generated files that are located in the target bucket under dags/data_source/reporting/task_dep_dags/dag_name. This folder will contain a Python file defining the Cloud Composer DAG and a SQL file with the refresh query for each table node within the DAG.
6. Copy the files to the Cloud Composer DAG bucket to deploy Cortex Framework using the standard build process.

Define task-dependent reporting settings

For each data source requiring task dependencies, Cortex Framework expects task-dependent reporting settings files with the suffix _task_dep.yaml. When creating and updating these files, consider the following:

• For SAP you may customize and use the provided setting files reporting_settings_ecc_task_dep.yaml and reporting_settings_s4_task_dep.yaml.
• For other data sources, create your own task-dependent reporting settings alongside the original, for example: reporting_settings_task_dep.yaml.

For more information about the fields available within reporting settings files, see dag_types.py.

Specify and customize task dependencies

Customize the task dependencies by adding table_setting.dag_setting as a new section to table type nodes:

    - sql_file: dependent_table2.sql
      type: table
      table_setting:
        dag_setting:
          name: "dag1"
          parents: ["dependent_table1.sql"]

• DAG settings include two fields:

  • name: A required string for all nodes in a task dependent DAG that designates the name of the DAG to which the table node belongs. This includes top level nodes which are referenced as a parent by other nodes within the DAG.
  • parents: An optional list of strings containing the sql_file path of other table nodes within the same DAG. These parents must run successfully before the node is triggered.

Additional considerations:

• Nodes without defined parents are considered top level nodes and will run at the start of the DAG.
• At least one top level node must have table_setting.load_frequency defined, which will be used as the DAG schedule.
• If multiple top level nodes have defined load_frequency, they must be the same.
• Child nodes that have parents defined can't define load_frequency.
• Nodes that don't have dag_setting defined will be generated the same as before as a DAG with a single table refresh node and no task dependencies.
• Other node types like views and scripts can't be included in task dependent DAGs, which only generate nodes with DML to refresh tables.