Access Airflow command-line interface

Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

Apache Airflow has a command-line interface (CLI) that you can use to perform tasks such as triggering and managing DAGs, getting information about DAG runs and tasks, adding and deleting connections and users.

Supported Airflow CLI commands

Airflow uses Airflow 2 CLI syntax, which is described in the Airflow documentation.

For a full list of supported Airflow CLI commands, see the reference for the gcloud composer environments run command.

Before you begin

  • You must have permissions to use Google Cloud CLI with Cloud Composer and run Airflow CLI commands.

  • It is not possible to run Airflow CLI commands through kubectl in Cloud Composer 3. This approach was deprecated and is required only for Cloud Composer versions earlier than 2.4.0.

Run Airflow CLI commands

To run Airflow CLI commands in your environments, use gcloud CLI:

gcloud composer environments run ENVIRONMENT_NAME \
    --location LOCATION \
    SUBCOMMAND \
    -- SUBCOMMAND_ARGUMENTS

Replace the following :

  • ENVIRONMENT_NAME: the name of your environment.
  • LOCATION: the region where the environment is located.
  • SUBCOMMAND: one of the supported Airflow CLI commands.
  • SUBCOMMAND_ARGUMENTS with arguments for the Airflow CLI command.

Sub-command arguments separator

Separate the arguments for the specified Airflow CLI command with --:

  • Specify compound CLI commands as a sub-command.
  • Specify any arguments for compound commands as sub-command arguments, after a -- separator.

Example:

gcloud composer environments run example-environment \
    dags list -- --output=json

Default location

Most gcloud composer commands require a location. You can specify the location with the --location flag, or by setting the default location.

For example, to trigger a DAG named sample_quickstart with the ID 5077 in your Cloud Composer environment:

gcloud composer environments run example-environment \
    --location us-central1 dags trigger -- sample_quickstart \
    --run-id=5077

Run Airflow CLI commands through Cloud Composer API

In Cloud Composer 3, you can run Airflow CLI commands through Cloud Composer API.

Execute a command

Construct an environments.executeAirflowCommand API request:

{
  "environment": "projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME",
  "command": "AIRFLOW_COMMAND",
  "subcommand": "AIRFLOW_SUBCOMMAND",
  "parameters": [
    "SUBCOMMAND_PARAMETER"
  ]
}

Replace the following:

  • PROJECT_ID: the Project ID.
  • LOCATION: the region where the environment is located.
  • ENVIRONMENT_NAME: the name of your environment.
  • AIRFLOW_COMMAND: Airflow CLI command that you want to execute, such as dags.
  • AIRFLOW_SUBCOMMAND: Sub-command for the Airflow CLI command that you want to execute, such as list.
  • (optional) SUBCOMMAND_PARAMETER: Parameters for the sub-command. If you want to use more than one parameter, add more items to the list.

Example:

// POST https://composer.googleapis.com/v1/{environment=projects/*/locations/*/environments/*}:executeAirflowCommand
{
  "environment": "projects/example-project/locations/us-central1/environments/example-environment",
  "command": "dags",
  "subcommand": "list",
  "parameters": [
    "-o json",
    "--verbose"
  ]
}

Poll command status

After you execute an Airflow CLI command through Cloud Composer API, check if the command was successfully completed by making a PollAirflowCommand request and inspecting the fields in exitInfo for errors and status codes. The output field contains log lines.

To get the command execution status and fetch logs, provide executionId, pod, and podNamespace values returned by ExecuteAirflowCommandRequest:

Example:

// POST https://composer.googleapis.com/v1/{environment=projects/*/locations/*/environments/*}:pollAirflowCommand
{
  "executionId": "a117da94-355d-4ad4-839e-ac39ccb0bf48",
  "pod": "airflow-webserver-66d96b858f-tn96b",
  "podNamespace": "airflow-2-10-2-build-13-226523e4",
  "nextLineNumber": 1
}

What's next