Manage data preparations

This document describes how to manage data preparations in BigQuery, including how to deploy and schedule data preparations. Data preparations are BigQuery resources powered by Dataform.

Before you begin

Required roles

To ensure that the Dataform service account has the necessary permissions to prepare data in BigQuery, see the required roles for Dataform service accounts.

To get the permissions that you need to prepare data in BigQuery, ask your administrator to grant you the following IAM roles on the project:

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

View existing data preparations

To view a list of existing data preparations, follow these steps:

  1. On the BigQuery Studio page, go to the Explorer pane.
  2. Expand your project.
  3. Expand the Data preparations list.

Schedule data preparations

You can create schedules in the data preparation editor and manage schedules in the BigQuery Orchestration page.

Create a schedule

To create a schedule that executes the data preparation steps and loads the prepared data into the destination table, schedule a one-time or a recurring data preparation run:

  1. From the data preparation toolbar, click Schedule.
  2. Enter a schedule name.
  3. Enter the service account name associated with the execution.
  4. Schedule a frequency.
  5. Click Create schedule.

View schedules

To view all data preparation schedules in your project, follow these steps:

  1. In the Google Cloud console, go to the Orchestration page.

    Go to Orchestration

  2. Optional: To view details of a selected schedule and its past runs, click the name of the schedule.

Delete a schedule

To permanently delete a schedule for a selected data preparation, follow these steps:

  1. In the Google Cloud console, go to the Orchestration page.

    Go to Orchestration

  2. In the row that contains the schedule, click Actions > Delete.

Optimize data preparation by incrementally processing data

To configure the way your prepared data is written into a destination table, follow these steps.

  1. In the Google Cloud console, go to the BigQuery Studio page.

    Go to BigQuery

  2. In the Activity pane, select your data preparation.

  3. In the toolbar of your data preparation, select More > Write mode.

  4. Select one of the options. For more information, see Write mode.

  5. Click Save.

Data preparation versions

BigQuery data preparations don't support viewing, comparing, or restoring data preparation versions.

For a list of data preparation versions in chronological order, follow these steps:

  1. On the BigQuery Studio page, go to the Explorer pane.
  2. Select your data preparation. Versions are listed on the Activity tab in the Explorer pane.

Download a data preparation

To download a data preparation in a YAML file, follow these steps:

  1. In the Google Cloud console, go to the BigQuery Studio page.

    Go to BigQuery

  2. In the Explorer pane, expand your project and the Data preparations folder. Click the name of the data preparation that you want to download.

  3. Click Download. The data preparation is saved in the YAML file format—for example, NAME data preparation.dp.yaml.

Upload a data preparation

To upload a data preparation from a YAML file, follow these steps:

  1. In the Google Cloud console, go to the BigQuery Studio page.

    Go to BigQuery

  2. In the Explorer pane, expand your project.

  3. Go to the Data preparations folder and click more_vert Menu > Upload to Data preparation.

  4. In the Upload data preparation dialog, select a file to upload, or enter the URL of the data preparation.

  5. Enter a name for the data preparation.

  6. Select a data preparation location where resources are managed and stored.

  7. Click Upload.

What's next