Create workflows

To request support or provide feedback for this feature, email bigquery-workflows-preview-feedback@google.com.

This document describes how to create workflows in BigQuery. Workflows are powered by Dataform.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Google Cloud project.

  4. Enable the BigQuery, Dataform, and Vertex AI APIs.

    Enable the APIs

  5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  6. Make sure that billing is enabled for your Google Cloud project.

  7. Enable the BigQuery, Dataform, and Vertex AI APIs.

    Enable the APIs

  8. Follow the instructions at Enable code asset management to save, share, and manage versions of code assets such as workflows.
  9. If this is the first time you are creating a code asset, set the default region for storing code assets. You can't change the region for a code asset after it is created.

Required roles

To get the permissions that you need to create workflows, ask your administrator to grant you the following IAM roles on the project:

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

For more information about Dataform IAM, see Access control with IAM.

Create a workflow

To create a workflow, follow these steps:

  1. Go to the BigQuery page.

    Go to BigQuery

  2. In the tab bar of the editor pane, click the arrow drop-down menu next to the + sign and then click Workflow.

  3. Optional: To rename the workflow, click the workflow name, and then type a new name.

  4. Click Get started, and then go to the Settings tab.

  5. In the Service account field, select a Dataform service account.

  6. In the Location section, select a processing region for the workflow.

    1. To select a specific region, select Region, and then select a region in the Region drop-down menu.
    2. To select a multi-region, select Multi-region, and then select a multi-region in the Multi-region drop-down menu.

    The workflow processing region doesn't need to match your default storage region for code assets.

  7. If you plan to add a notebook to your workflow, do the following:

    1. In the Cloud Storage bucket field, click Browse and select or create a Cloud Storage bucket for storing the output of notebooks in your workflow.

    2. Follow Add a principal to a bucket-level policy to add your custom Dataform service account as a principal to the Cloud Storage bucket which you plan to use for storing output of scheduled workflow runs, and grant the Storage Admin (roles/storage.admin) role to this principal.

      The selected custom Dataform service account must be granted the Storage Admin IAM role on the selected bucket.

Add a workflow task

To add a task to a workflow, follow these steps:

  1. In the Google Cloud console, go to the BigQuery page.

    Go to BigQuery

  2. In the Explorer pane, expand your project and the Workflows folder, and then select a workflow.

  3. To add a SQL query or a notebook, do the following:

    SQL query

    1. Click Add task, and then select Query.

    2. Optional: In the Query task details pane, in the Run after drop-down menu, select a task that will precede your query.

      Your query will depend on the preceding task.

    3. To create the query, click Create query.

    4. In the new query, add query syntax, and then click Save query.

    5. Optional: To rename the query, click the query name and type a new name.

    Notebook

    1. Click Add task, and then select Notebook.

    2. Optional: In the Notebook task details pane, in the Run after drop-down menu, select a task that will precede your notebook.

      Your notebook will depend on the preceding task.

    3. To create the notebook, click Create notebook.

    4. In the new notebook, add notebook content, and then click Save.

    5. Optional: To rename the notebook, click the notebook name and type a new name.

Edit a workflow task

To edit a workflow task, follow these steps:

  1. In the Google Cloud console, go to the BigQuery page.

    Go to BigQuery

  2. In the Explorer pane, expand your project and the Workflows folder, and then select a workflow.

  3. Click the selected task.

  4. To change the preceding task, in the Run after drop-down menu, select a task that will precede your notebook.

  5. To edit the contents of the selected task, click Edit.

  6. In the new tab that opens, edit task contents, and then save changes to the task.

Delete a workflow task

To delete a task from a workflow, follow these steps:

  1. In the Google Cloud console, go to the BigQuery page.

    Go to BigQuery

  2. In the Explorer pane, expand your project and the Workflows folder, and then select a workflow.

  3. Click the selected task.

  4. In the Task details pane, click the DeleteDelete icon.

Run a workflow

To manually run the current version of a workflow, follow these steps:

  1. In the Google Cloud console, go to the BigQuery page.

    Go to BigQuery

  2. In the Explorer pane, expand your project and the Workflows folder, and then select a workflow.

  3. Click Run.

  4. Optional: To inspect the run, view past manual runs.

What's next