Enabling and disabling DAG serialization

Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

Follow the instructions on this page to enable or disable DAG serialization.

In a Cloud Composer environment, both the Airflow scheduler and the Airflow web server continuously process DAGs. You can improve the reliability and performance of the Airflow web server by enabling DAG serialization.

Enabling DAG serialization forces the scheduler to process DAG files before they are sent to the web server. The web server does not process DAGs. Instead, it reads the serialized DAGs from the Airflow database. Serializing DAGs in this way reduces the CPU and memory usage of the web server, especially when processing a large number of DAGs.

Before you begin

DAG serialization affects the following Cloud Composer features:

  • DAG serialization cannot be enabled at the same time as asynchronous DAG loading.
  • Enabling DAG serialization disables all Airflow web server plugins for Cloud Composer. This does not impact scheduler or worker plugins, including Airflow operators, sensors, and so on.

Enabling DAG serialization

You can enable DAG serialization for an existing environment or when creating a new environment.

To enable DAG serialization, override the following Airflow configuration options:

Section Key Value
core store_serialized_dags True
core store_dag_code True
core min_serialized_dag_update_interval 30
scheduler dag_dir_list_interval 30

The min_serialized_dag_update_interval Airflow configuration option controls how frequently serialized DAGs are updated in the database. The dag_dir_list_interval option controls how frequently removed DAGs are deleted from the Airflow database. A high update frequency can negatively impact performance. We recommend setting min_serialized_dag_update_interval and dag_dir_list_interval to 30 seconds.

Disabling DAG serialization

You can enable DAG serialization for an existing environment or when creating a new environment.

To disable DAG serialization, override the following Airflow configuration options:

Section Key Value
core store_serialized_dags False
core store_dag_code False