Orchestrate notebooks

To provide feedback for notebook orchestration, email bigquery-notebooks-scheduling-support@google.com.

This document describes how to orchestrate Colab Enterprise notebooks in BigQuery, including how to schedule notebooks and inspect scheduled notebook runs.

Notebooks are code assets powered by Dataform.

You can schedule a notebook to automatically run at a specified time and frequency—for example, to train an ML model, call external APIs, or run BigQuery DataFrames code.

Changes that you make to a notebook are automatically saved, but are available only to you and to users who have access to the notebook. To update the schedule with a new version of the notebook, you need to deploy the notebook. By deploying a notebook, you update its schedule with your current version of the notebook. Schedules run the latest deployed version of the notebook.

Each notebook schedule is run by a custom Dataform service account, which you select during schedule creation. For more information about the types of service accounts in Dataform, see About service accounts in Dataform.

Dataform writes the output of scheduled notebook runs to the Cloud Storage bucket selected during schedule creation.

Notebook schedules use a standard E2 runtime. Colab Enterprise charges for runtimes apply. You are charged for runtime processing based on the E2 machine type. For information about pricing of standard E2 runtimes, see Colab Enterprise pricing.

Before you begin

Before you begin, create a notebook.

Enable notebook scheduling

To schedule notebooks, you must grant the following roles to the custom Dataform service account which you plan to use for notebook schedules:

Notebook Executor User (roles/aiplatform.notebookExecutorUser)
Follow Grant a single role on a project to grant the Notebook Executor User role to your custom Dataform service account on the selected project.
Storage Admin (roles/storage.admin)
Follow Add a principal to a bucket-level policy to add your custom Dataform service account as a principal to the Cloud Storage bucket which you plan to use for storing output of scheduled notebook runs, and grant the Storage Admin role to this principal.
Service Account User (roles/iam.serviceAccountUser)
Follow Grant a single role on a service account to add your custom Dataform service account as a principal to itself. In other words, add the custom Dataform service account as a principal to the same custom Dataform service account. Then, grant the Service Account User role to this principal.

Additionally, you must grant the following role to the default Dataform service account:

Service Account Token Creator(roles/iam.serviceAccountTokenCreator)
Follow Grant token creation access to a custom Dataform service account to add the default Dataform service account as a principal to your custom Dataform service account, and grant the Service Account Token Creator role to this principal.

To learn more about service accounts in Dataform, see About service accounts in Dataform.

Required permissions

To create notebook schedules, you need the following roles:

To edit notebook schedules (Preview), you need the Dataform Editor (roles/dataform.editor) role.

To view notebook schedules (Preview), you need the Dataform Viewer (roles/dataform.viewer) role.

For more information about BigQuery IAM, see Access control with IAM.

Create a schedule

To schedule a notebook, follow these steps:

  1. In the Google Cloud console, go to the BigQuery page.

    Go to BigQuery

  2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the notebook that you want to schedule.

  3. Click Schedule.

  4. In the Schedule Notebook pane, in the Schedule name field, enter a name for the schedule.

  5. In the Runtime template field, select a runtime.

  6. In the Service account field, select a custom Dataform service account.

  7. In the Cloud Storage bucket field, click Browse and select or create a Cloud Storage bucket.

    The selected custom Dataform service account must be granted the Storage Admin IAM role on the selected bucket. For more information, see Enable notebook scheduling.

  8. In the Schedule frequency section, do the following:

    1. In the Repeats drop-down, select the frequency of scheduled notebook runs.
    2. In the At time field, enter the time for scheduled notebook runs.
    3. In the Timezone drop-down, select the timezone for the schedule.

  9. Click Create schedule.

When you create the schedule, the current version of the notebook is automatically deployed. To update the schedule with a new version of the notebook, deploy the notebook.

The latest deployed version of the notebook runs at the selected time and frequency.

Deploy a notebook

Deploying a notebook updates its schedule with the current version of the notebook. Schedules run the latest deployed version of the notebook.

To deploy a notebook, follow these steps:

  1. In the Google Cloud console, go to the BigQuery page.

    Go to BigQuery

  2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the selected notebook.

  3. Click Deploy.

The corresponding schedule is updated with the current version of the notebook. The latest deployed version of the notebook runs at the scheduled time.

View schedule details

To view schedule details for a selected notebook, follow these steps:

  1. In the Google Cloud console, go to the BigQuery page.

    Go to BigQuery

  2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the selected notebook.

  3. Click View schedule.

View past executions

To view past executions of a selected notebook schedule, follow these steps:

  1. In the Google Cloud console, go to the BigQuery page.

    Go to BigQuery

  2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the selected notebook.

  3. Click View schedule, and then click View past executions.

Disable a schedule

To pause scheduled runs of a selected notebook without deleting the schedule, you can disable the schedule.

To disable a schedule for a selected notebook, follow these steps:

  1. In the Google Cloud console, go to the BigQuery page.

    Go to BigQuery

  2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the selected notebook.

  3. Click View schedule.

  4. In the schedule details table, in the Schedule state row, click the Schedule is enabled toggle.

Enable a schedule

To resume scheduled runs of a disabled schedule, follow these steps:

  1. In the Google Cloud console, go to the BigQuery page.

    Go to BigQuery

  2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the selected notebook.

  3. Click View schedule.

  4. In the schedule details table, in the Schedule state row, click the Schedule is disabled toggle.

Edit a schedule

To edit a schedule, follow these steps:

  1. In the Google Cloud console, go to the BigQuery page.

    Go to BigQuery

  2. In the Explorer pane, expand your project and the Notebooks folder, and if necessary, the Shared notebooks folder. Click the name of the selected notebook.

  3. Click View schedule, and then click Edit.

  4. In the Schedule notebook dialog, edit the schedule, and then click Update schedule.

View all schedules

To view all notebook schedules in your project, follow these steps:

  1. In the Google Cloud console, go to the Orchestration page.

    Go to the Orchestration page

  2. Optional: To view details of a selected schedule and its past runs, click the name of the schedule.

Delete a schedule

To permanently delete a schedule for a selected notebook, follow these steps:

  1. In the Google Cloud console, go to the Orchestration page.

    Go to the Orchestration page

  2. In the list of all schedules, select the row that contains the schedule that you want to delete.

  3. Optional: To delete multiple schedules at once, select multiple rows.

  4. Click Delete.

  5. In the dialog that appears, click Delete.

What's next