Orchestrate data preparations
This document describes how to orchestrate data preparation pipelines, including how to perform manual and scheduled runs.
Data preparations are powered by Dataform.
Data preparations run using custom Dataform service accounts, which you select when you configure schedules or test runs. For more information, see About service accounts in Dataform.
Changes you make to the data preparation steps aren't automatically saved. You must save and deploy the changes before they can be executed with a schedule. Schedules always run the latest deployed version of your data preparation and exclude any undeployed changes you might be developing.
Before you begin
Before you begin, create a data preparation.
Required roles
To run data preparations, you must grant the roles to the service account which you plan to use for executing the data preparation runs. For more information, see the required roles.
Develop a data preparation
As you develop a data preparation, you can manually run the steps and inspect the output before you deploy the changes to production. You can test the current version you're developing on your data, while BigQuery continues to run the latest deployed version, according to a schedule. Before you can perform the run, you must configure the destination, and fix any validation errors.
Manually run a data preparation in development
To test your data preparation steps and validate the results in your destination table, run the data preparation manually from the data preparation editor:
In the Google Cloud console, go to the BigQuery page.
In the Explorer pane, expand your project and the Data preparations folder. Click the name of the data preparation that you want to run.
Configure the permissions on the service account for the run:
- In the data preparation editor toolbar, hold the pointer over the disabled Run option.
- On the dialog that appears with information about configuring the service account, click Configure.
- In the Service account settings dialog, select a service account.
- If the service account needs additional permissions, grant it the required roles by clicking Grant all.
- Click Save.
Optional: To update the service account for future runs, go to the data preparation editor toolbar and click More > Configure run now experience, and then update and save the service account settings.
Fix any validation errors that appear.
From the data preparation editor toolbar, click Run.
In the Run now dialog, click Confirm to acknowledge that this manual run writes data to a destination table, which you might also be using for scheduled runs.
The run then executes your steps and loads the output to the destination.
Optional: After the run is complete, you can view the details about the execution in the Executions pane.
Deploy a data preparation
To schedule runs for a version of your data preparation, you must first deploy it. Schedules run the most recently deployed version.
To deploy a data preparation, follow these steps:
In the Google Cloud console, go to the BigQuery page.
In the Explorer pane, expand your project and the Data preparations folder. Click the name of the chosen data preparation.
The data preparation editor opens.
In the data preparation editor toolbar, click Deploy.
Create a schedule
To create a schedule that executes the deployed data preparation steps and loads the prepared data into the destination table, schedule a data preparation run. To schedule the run, you must configure the destination, and fix any validation errors.
To create a schedule, follow these steps:
In the Google Cloud console, go to the BigQuery page.
In the Explorer pane, expand your project and the Data preparations folder. Click the name of the data preparation that you want to schedule.
From the data preparation editor toolbar, click Schedule.
Enter a schedule name.
Enter the service account name associated with the execution.
Schedule a frequency.
Click Create schedule.
Manually run a scheduled data preparation
When you manually run a data preparation in a selected schedule, BigQuery executes the data preparation once, independently from the schedule.
To manually run a scheduled data preparation, follow these steps:
In the Google Cloud console, go to the Scheduling page.
Click the name of the selected data preparation schedule.
On the Schedule details page, click Run.
View schedules
You can view data preparation schedules from the data preparation editor or the Scheduling page.
Data preparation editor
To view the schedule for a data preparation, follow these steps:
- In the data preparation editor toolbar, click schedule View schedule.
- Optional: To view the schedule history, click View past executions.
Scheduling page
To view all data preparation schedules in your project, follow these steps:
In the Google Cloud console, go to the Scheduling page.
Optional: To view the run history and details of a selected schedule, click the name of the schedule. History of manual runs is not shown.
Edit a schedule
You can edit a schedule from the data preparation editor or the Scheduling page.
Data preparation editor
To edit a schedule, follow these steps:
- In the data preparation editor toolbar, click schedule View schedule.
- In the Schedule data preparation dialog, click Edit and then update the schedule.
- Click Update schedule.
Scheduling page
To edit a schedule, follow these steps:
In the Google Cloud console, go to the Scheduling page.
Click the name of the selected data preparation schedule.
On the Schedule details page, click Edit.
Click View schedule.
In the Schedule data preparation dialog, click Edit and then update the schedule.
Click Update schedule.
Delete a schedule
To permanently delete a schedule for a selected data preparation, follow these steps:
In the Google Cloud console, go to the Scheduling page.
In the row that contains the schedule, click more_vert Actions > Delete.
What's next
- Learn how to create data preparations.
- Learn more about managing data preparations.