Task dependent DAGs
This page outlines the steps to enable task dependent Directed Acyclic Graphs (DAGs) to specify task dependencies between SQL table nodes within a single DAG, rather than relying on cron scheduling across multiple DAGs. Customizable settings are used to generate Cloud Composer DAGs with multiple table refresh nodes that depend on each other. Cortex Framework provides recommended settings for task dependent SAP DAGs (ECC and S/4HANA). However, you can customize them further or define task dependent DAGs for any data sources.
Enable task dependent DAGs
- Modify the config.json file by setting the
enableTaskDependencies
field toTrue
. This setting activates the Cortex Framework's search for task-dependent reporting settings files with the suffix_task_dep.yaml
. - Create a dedicated reporting settings file with the suffix
_task_dep.yaml
for each data source requiring task dependencies. For more details, see Define task-dependent reporting settings. - Customize the task dependencies by adding
table_setting.dag_setting
as a new section totable
type nodes. For more details, see Specify and customize task dependencies. - Build Cortex Framework using the standard build process.
- Examine the generated files that are located in the target bucket under
dags/data_source/reporting/task_dep_dags/dag_name
. This folder will contain a Python file defining the Cloud Composer DAG and a SQL file with the refresh query for each table node within the DAG. - Copy the files to the Cloud Composer DAG bucket to deploy Cortex Framework using the standard build process.
Define task-dependent reporting settings
For each data source requiring task dependencies, Cortex Framework
expects task-dependent reporting settings files with the suffix _task_dep.yaml
.
When creating and updating these files, consider the following:
- For SAP you may customize and use the provided setting files
reporting_settings_ecc_task_dep.yaml
andreporting_settings_s4_task_dep.yaml
. - For other data sources, create your own task-dependent reporting settings alongside the original, for example:
reporting_settings_task_dep.yaml.
For more information about the fields available within reporting settings files, see dag_types.py.
Specify and customize task dependencies
Customize the task dependencies by adding
table_setting.dag_setting
as a new section to table
type nodes:
- sql_file: dependent_table2.sql
type: table
table_setting:
dag_setting:
name: "dag1"
parents: ["dependent_table1.sql"]
DAG settings include two fields:
name
: A required string for all nodes in a task dependent DAG that designates the name of the DAG to which the table node belongs. This includes top level nodes which are referenced as a parent by other nodes within the DAG.parents
: An optional list of strings containing thesql_file
path of other table nodes within the same DAG. These parents must run successfully before the node is triggered.
Additional considerations:
- Nodes without defined parents are considered top level nodes and will run at the start of the DAG.
- At least one top level node must have
table_setting.load_frequency
defined, which will be used as the DAG schedule. - If multiple top level nodes have defined
load_frequency
, they must be the same. - Child nodes that have
parents
defined can't defineload_frequency
. - Nodes that don't have
dag_setting
defined will be generated the same as before as a DAG with a single table refresh node and no task dependencies. - Other node types like views and scripts can't be included in task dependent DAGs, which only generate nodes with DML to refresh tables.