Run notebook files with the executor

This page describes how to use the executor in a Vertex AI Workbench managed notebooks instance to run notebook files as a one-time execution and on a schedule.

Overview

The executor lets you submit a notebook (ipynb) file to run on Vertex AI custom training. You can set parameter values for each execution of a notebook file. You can also run a notebook file on a recurring schedule. After the execution completes, you can view the execution results and share them with others.

When a notebook file is submitted to Vertex AI custom training, Vertex AI creates a new custom training job that executes your notebook file following the lifecycle of a training job.

Requirements for notebook code run by the executor

When you write notebook code to run in the executor, keep in mind that the code will run in a tenant project separate from your managed notebooks instance's project. This section describes how this affects your code when it is run in the executor.

Ensure package installations are available to the executor

If your notebook depends on package installations that are not already included in the managed notebooks kernel that you are using, make sure your packages are available to your notebook code in the executor in one of the following ways:

• Use a custom container that already has the package installed, and then execute your notebook on that custom container. See the requirements for using a custom container with the executor.

• Install the package within your notebook file's code. The package is installed every time the notebook file is executed, but this ensures that the package is available on the container that you select when you execute your notebook.

Use explicit project selection

When you access resources through code run by the executor, the executor might not connect to the correct Google Cloud project. If you encounter permission errors, connecting to the wrong project might be the problem.

This problem occurs because the executor does not run your code directly in your managed notebooks instance's Google Cloud project. Instead, the executor runs your code in Vertex AI custom training within a tenant project managed by Google. Therefore, don't try to infer a project ID from the environment in your notebook code; specify project IDs explicitly.

If you don't want to hardcode a project ID in your code, you can reference the CLOUD_ML_PROJECT_ID environment variable. Vertex AI sets this environment variable in every custom training container to contain the project number of the project where you initiated custom training. Many Google Cloud tools can accept a project number wherever they take a project ID.

For example, if you want to use the Python Client for Google BigQuery to access a BigQuery table in the same project, then do not infer the project in your notebook code:

Implicit project selection

from google.cloud import bigquery

client = bigquery.Client()

Instead use code that explicitly selects a project:

Explicit project selection

import os

from google.cloud import bigquery

project_number = os.environ["CLOUD_ML_PROJECT_ID"]

client = bigquery.Client(project=project_number)

Authenticate access using service accounts

By default, your managed notebooks instance can have access to resources that exist in the same project. Therefore, when you run your notebook file's code manually, these resources do not need additional authentication. However, because the executor runs in a separate tenant project, it does not have the same default access.

Also, the executor cannot use end-user credentials to authenticate access to resources, for example, the gcloud auth login command.

To resolve these issues, in your notebook file's code, authenticate access to resources through a service account.

Then when you create an execution or schedule, specify the service account.

For example, while you create an execution, complete these steps:

1. In the Submit notebooks to Executor dialog, expand Advanced options.

2. In the Identity and API access section, clear the check mark next to Use Vertex AI Training's default service account and enter the specific service account to use.

See the full list of steps for creating an execution.

Requirements when using a custom container

You can use the executor to run notebook code on a custom container. Your custom container must include the nbexecutor extension, which enables the executor to run notebook code as a Vertex AI custom training job. To ensure that your custom container has the nbexecutor extension, you can modify one of the Deep Learning Containers container images to create a derivative container image. Deep Learning Containers images include the nbexecutor extension.

Before you begin

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Go to project selector

3. Make sure that billing is enabled for your Google Cloud project.

4. Enable the Notebooks and Vertex AI APIs.

   Enable the APIs

5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Go to project selector

6. Make sure that billing is enabled for your Google Cloud project.

7. Enable the Notebooks and Vertex AI APIs.

   Enable the APIs

8. If you haven't already, create a managed notebooks instance.

Required roles

To ensure that your instance's service account has the necessary permissions to interact with the Vertex AI Workbench executor, ask your administrator to grant your instance's service account the following IAM roles on the project:

• Notebooks Viewer (roles/notebooks.viewer)
• Vertex AI User (roles/aiplatform.user)
• Storage Admin (roles/storage.admin)

For more information about granting roles, see Manage access to projects, folders, and organizations.

Your administrator might also be able to give your instance's service account the required permissions through custom roles or other predefined roles.

Open JupyterLab

To open JupyterLab and prepare a notebook file to run, complete the following steps.

1. Open JupyterLab.

2. Upload a notebook (ipynb) file, open an existing file, or open a new notebook file and add the code that you want to run to the notebook.

3. Make sure your notebook file's code meets the requirements for using the executor.

Create an execution

To create an execution that runs your notebook file, complete the following steps. These steps cover both scheduling executions and creating a one-time execution.

1. In your managed notebooks instance's JupyterLab user interface, open the notebook file that you want to run.

2. Click the  Execute button.

3. In the Submit notebooks to Executor dialog, in the Execution name field, enter a name for your execution.

4. Select a Machine type and Accelerator type.

5. Select an Environment.

6. In the Type field, select One-time execution or select Schedule-based recurring executions and complete the dialog for scheduling executions.

7. In Advanced options, select the Region where you want to run your notebook.

8. In the Cloud Storage bucket field, select an available Cloud Storage bucket or enter a name for a new bucket and click Create and select. The executor stores your notebook output in this Cloud Storage bucket.

9. Optional: in the Notebook parameterization section, in the Input parameters text box, add notebook parameters separated by commas, for example optimizer="SGD",learning_rate=0.01.

   Learn more about how to use notebook parameters.

10. Optional: in the Identity and API access section, select Use Vertex AI Training's default service account or clear the check mark and enter a specific service account to use.

11. Optional: in the Networking section, specify a Virtual Private Cloud network. Using a VPC network for your execution requires a private services access connection.

12. Click Submit.

    One-time executions begin immediately. Scheduled executions run automatically on the schedule that you set.

In the Google Cloud console, on the Vertex AI Workbench page, you can view your completed executions on the Executions tab and view your schedules on the Schedules tab.

View, share, and import an executed notebook file

By using your managed notebooks instance's JupyterLab user interface, you can view an executed notebook's output, share the results with others, and import the executed notebook file into JupyterLab.

View a notebook execution's results

You can view a notebook execution's results in the Google Cloud console or in the JupyterLab user interface.

Share a notebook execution's results

You can share execution results by providing access to the Cloud Storage bucket that contains your notebook execution. Providing this access also grants users access to any other resources in the same Cloud Storage bucket. To share execution results, complete the following steps.

Import an executed notebook into JupyterLab

To import an executed notebook into JupyterLab, complete the following steps.

1. In your managed notebooks instance's JupyterLab user interface, in the navigation menu, click the  Notebook Executor button.

2. Click the Executions tab.

3. Select the Region that contains your executed notebook.

4. Next to the execution that you want to import, click the more_vert options menu, and select Import executed notebook.

5. Select the kernel that you want to open the notebook.

   The executor opens the executed notebook file in JupyterLab and stores this notebook file in the JupyterLab File Browser in a folder named imported_notebook_jobs.

View or delete a schedule

You can view and delete schedules by using either the Google Cloud console or your managed notebooks instance's JupyterLab user interface.

View a schedule

View a schedule to see the frequency settings of the schedule or to view results of your notebook executions.

Delete a schedule

Deleting a schedule does not delete the executions that were generated from that schedule.

Jobs on Vertex AI custom training

Because notebook executions are run on Vertex AI custom training, they are exposed as custom training jobs in Vertex AI. You can view these custom training jobs in the Google Cloud console, on the Custom jobs tab of the Vertex AI Training page. Learn more about how to work with Vertex AI custom training jobs.

What's next

• Learn how to run notebook executions with parameters.

• Learn more about Vertex AI custom training.