Manage workflows

To request support or provide feedback for this feature, email bigquery-workflows-preview-feedback@google.com.

This document describes how to manage workflows, including how to schedule and delete workflows.

Workflows are powered by Dataform.

Before you begin

Before you begin, create a workflow.

Required roles

To get the permissions that you need to manage workflows, ask your administrator to grant you the following IAM roles:

• To delete workflows: Dataform Admin (roles/dataform.Admin) on the workflow
• To create, edit, run, and delete workflow schedules: Dataform Admin (roles/dataform.Admin) on the workflow
• To view and run workflows: Dataform Viewer (roles/dataform.Viewer) on the project
• To view workflow schedules: Dataform Editor (roles/dataform.Editor) on the project

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

For more information about Dataform IAM, see Access control with IAM.

Schedule a workflow

You can schedule a workflow to automatically run at a specified time and frequency. For more information, see About scheduling workflows.

Enable workflow scheduling

To schedule workflows, you must grant the following roles to the service account which you plan to use for workflow schedules:

Service Account User (roles/iam.serviceAccountUser)

Follow Grant a single role on a service account to add your service account as a principal to itself. In other words, add the service account as a principal to the same service account. Then, grant the Service Account User role to this principal.

If your workflow contains SQL queries, you must grant the following roles to the service account which you plan to use for workflow schedules:

BigQuery Job User (roles/bigquery.jobUser)

Follow Grant a single role on a project to grant the BigQuery Job User role to your service account on projects from which your workflows read data.

BigQuery Data Viewer (roles/bigquery.dataViewer)

Follow Grant a single role on a project to grant the BigQuery Data Viewer role to your service account on projects from which your workflows read data.

BigQuery Data Editor (roles/bigquery.dataEditor)

Follow Grant a single role on a project to grant the BigQuery Data Editor role to your service account on projects to which your workflows write data.

If your workflow contains notebooks, you must grant the following roles to the service account which you plan to use for workflow schedules:

Notebook Executor User (roles/aiplatform.notebookExecutorUser)

Follow Grant a single role on a project to grant the Notebook Executor User role to your service account on the selected project.

Storage Admin (roles/storage.admin)

Follow Add a principal to a bucket-level policy to add your service account as a principal to the Cloud Storage bucket which you plan to use for storing output of notebooks executed in scheduled workflow runs, and grant the Storage Admin role to this principal.

Additionally, you must grant the following role to the default Dataform service account:

Service Account Token Creator(roles/iam.serviceAccountTokenCreator)

Follow Grant token creation access to a service account to add the default Dataform service account as a principal to your service account, and grant the Service Account Token Creator role to this principal.

To learn more about service accounts in Dataform, see About service accounts in Dataform.

Create a workflow schedule

To schedule a workflow, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Workflows folder, and then select a workflow.

3. Click Schedule.

4. In the Schedule workflow pane, in the Schedule name field, enter a name for the schedule.

5. In the Service account field, select a service account.

6. If your workflow contains a notebook, in the Cloud Storage bucket field, click Browse and select or create a Cloud Storage bucket for storing the output of notebooks in your workflow.

   Your selected service account must be granted the Storage Admin IAM role on the selected bucket. For more information, see Enable workflow scheduling.

7. In the Schedule frequency section, do the following:

   1. In the Repeats drop-down, select the frequency of scheduled workflow runs.
   2. In the At time field, enter the time for scheduled workflow runs.
   3. In the Timezone drop-down, select the timezone for the schedule.

8. Click Create schedule.

When you create the schedule, the current version of the workflow is automatically deployed. To update the schedule with a new version of the workflow, deploy the workflow.

The latest deployed version of the workflow runs at the selected time and frequency.

Deploy a workflow

Deploying a workflow updates its schedule with the current version of the workflow. Schedules run the latest deployed version of the workflow.

To deploy a workflow, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Workflows folder, and then select a workflow.

3. Click Deploy.

The corresponding schedule is updated with the current version of the workflow. The latest deployed version of the workflow runs at the scheduled time.

Disable a schedule

To pause scheduled runs of a selected workflow without deleting the schedule, you can disable the schedule.

To disable a schedule for a selected workflow, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Workflows folder, and then select a workflow.

3. Click View schedule.

4. In the Schedule details table, in the Schedule state row, click the Schedule is enabled toggle.

Enable a schedule

To resume scheduled runs of a disabled workflow schedule, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Workflows folder, and then select a workflow.

3. Click View schedule.

4. In the Schedule details table, in the Schedule state row, click the Schedule is disabled toggle.

Manually run a deployed workflow

When you manually run a workflow deployed in a selected schedule, BigQuery executes the deployed workflow once, independently from the schedule.

To manually run a deployed workflow, follow these steps:

1. In the Google Cloud console, go to the Orchestration page.

   Go to the Orchestration page

2. Click the name of the selected workflow schedule.

3. On the Schedule details page, click Run.

View all workflows schedules

To view all workflows in your Google Cloud project, follow these steps:

1. In the Google Cloud console, go to the Orchestration page.

   Go to the Orchestration page

2. Optional: To display additional columns with workflow schedule details, click view_column Column display options, and then select columns and click OK.

View workflow schedule details

To view details for a selected workflow schedule, follow these steps:

View past scheduled runs

To view past runs of a selected workflow schedule, follow these steps:

Edit a workflow schedule

To edit a workflow schedule, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Workflows folder, and then select a workflow.

3. Click View schedule, and then click Edit.

4. In the Schedule workflow dialog, edit the schedule, and then click Update schedule.

Delete a workflow schedule

To permanently delete a workflow schedule, follow these steps:

1. In the Google Cloud console, go to the Orchestration page.

   Go to the Orchestration page

2. Do either of the following:

   • Click the name of the selected workflow schedule, and then on the Schedule details page, click Delete.

   • In the row that contains the selected workflow schedule, click more_vert View actions in the Actions column, and then click Delete.

3. In the dialog that appears, click Delete.

View past manual runs

To view past manual runs of a selected workflow, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Workflows folder, and then select a workflow.

3. Click Executions.

4. Optional: To refresh the list of past runs, click Refresh.

Configure alerts for failed workflow runs

Each workflow has a corresponding Dataform repository ID. Each BigQuery workflow run is logged in Cloud Logging using the corresponding Dataform repository ID. You can use Cloud Monitoring to observe trends in Cloud Logging logs for BigQuery workflow runs and to notify you when conditions you describe occur.

To receive alerts when a BigQuery workflow run fails, you can create a log-based alerting policy for the corresponding Dataform repository ID. For instructions, see Configure alerts for failed workflow invocations.

To find the Dataform repository ID of your workflow, do the following:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Workflows folder, and then select a workflow.

3. Click Settings.

   The Dataform repository ID of your workflow is displayed at the bottom of the Settings tab.

Delete a workflow

To permanently delete a workflow, follow these steps:

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. In the Explorer pane, expand your project and the Workflows folder. Find the Workflow that you want to delete.

3. Click more_vert View actions next to the workflow, and then click Delete.

4. Click Delete.

What's next

• Learn more about workflows in BigQuery.
• Learn how to create workflows.