Extensions by Google

This document shows you how to register and use Google-provided Extensions from the Google Cloud console and the Vertex AI API.

The Code Interpreter extension lets you generate and run Python code to:

  • Analyze, clean, transform, and reshape your datasets
  • Visualize data in charts and graphs
  • Run calculations

The Vertex AI Search extension lets you access and search website corpuses and unstructured data to provide relevant responses to natural language questions, such as:

  • "How did the competitive threats for the company change from Q1 of last year to Q1 of this year?"
  • "What parts of the company are growing the fastest? How fast?"

To learn about Google extensions with end-to-end tutorials, see the following Jupyter notebooks:

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Google Cloud project.

  4. Enable the Vertex AI API.

    Enable the API

    5.

  5. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  6. Make sure that billing is enabled for your Google Cloud project.

  7. Enable the Vertex AI API.

    Enable the API

    8.

Code Interpreter extension

The Code Interpreter extension uses the code_interpreter_tool to generate and run Python code from a natural language description. The code_interpreter_tool is defined in an OpenAPI Specification code_interpreter.yaml file.

openapi: "3.0.0"
info:
  version: 1.0.0
  title: code_interpreter_tool
  description: >
    This tool supports the following operations based on user input:

    1. **Generates and Executes Code:** Accepts an user query in natural language, generates corresponding code, and executes it to produce results for the user query.


    Supported AuthTypes:

    - `GOOGLE_SERVICE_ACCOUNT_AUTH`: (Vertex AI Extension Service Agent is supported).
paths:
  /generate_and_execute:
    post:
      operationId: generate_and_execute
      description: >
        Get the results of a natural language query by generating and executing a code snippet.
        Example queries: "Find the max in [1, 2, 5]" or "Plot average sales by year (from data.csv)".
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              required:
              - query
              properties:
                query:
                  type: string
                  description: >
                    Required. The Natural language query to get the results for.
                    The query string can optionally contain data to use for the code generated.
                    For example: "I have a list of numbers: [1, 2, 3, 4]. Find the largest number in the provided data."
                timeout:
                  type: number
                  description: >
                    Optional. Timeout in miliseconds for the code execution. Default value: 30000.
                files:
                  type: array
                  description: >
                    Optional. Input files to use when executing the generated code.
                    If specified, the file contents are expected be base64-encoded.
                    For example: [{"name": "data.csv", "contents": "aXRlbTEsaXRlbTI="}]

                    Only one of `file_gcs_uris` and `files` field should be provided.
                  items:
                    $ref: "#/components/schemas/File"
                file_gcs_uris:
                  type: array
                  description: >
                    Optional. GCS URIs of input files to use when executing the generated code.
                    For example: ["gs://input-bucket/data.csv"]

                    Only one of `file_gcs_uris` and `files` field should be provided.
                    This option is only applicable when `file_input_gcs_bucket` is specified in `Extension.CodeInterpreterRuntimeConfig`.
                  items:
                    type: string
      responses:
        '200':
          description: >
            The results of generating and executing code based on the natual language query.
            The result contains the generated code, and the STDOUT, STDERR, and output files from code execution.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/GenerationAndExecutionResult"
components:
  schemas:
    File:
      description: >
        File used as inputs and outputs of code execution. The `contents` string should be base64-encoded bytes.
        For example: [{"name": "data.csv", "contents": "aXRlbTEsaXRlbTI="}]
      type: object
      properties:
        name:
          type: string
        contents:
          type: string
          format: byte
    GenerationAndExecutionResult:
      description: >
        The results of generating and executing code based on the natual language query.
      properties:
        generated_code:
          type: string
          description: >
            The generated code in markdown format.
            For example: "```python\nprint(\"Hello World\")\n```"
        execution_result:
          type: string
          description: >
            The code execution result string from STDOUT.
        execution_error:
          type: string
          description: >
            The code execution error string from STDERR.
        output_files:
          type: array
          description: >
            The output files generated from code execution.
            If present, the file contents are required be base64-encoded.
            For example: [{"name": "data.csv", "contents": "aXRlbTEsaXRlbTI="}]
          items:
            $ref: "#/components/schemas/File"
        output_gcs_uris:
          type: array
          description: >
            The output GCS URIs of files generated from code execution.
            For example: ["gs://output-bucket/subfolder/output.csv"]

            This field is only applicable when `file_output_gcs_bucket` is specified in `Extension.CodeInterpreterRuntimeConfig`.
          items:
            type: string

Register, query, and run the Code Interpreter extension

The following sections show you how to register the Code Interpreter extension using the Google Cloud console and the Vertex AI API. After registering the extension, you can query it using the Google Cloud console or run it using the Vertex AI API.

Console

Register the extension

Perform the following steps to register the Code Interpreter extension using the Google Cloud console.

  1. In the Google Cloud console, go to the Vertex AI Extensions page.

    Go to Vertex AI Extensions

  2. Click Create Extension.

  3. In the Create a new extension dialog, do the:

    • Extension name: Enter a name for your extension, such as "code_interpreter_extension".
    • Description: (Optional) Enter an extension description, such as "A code interpreter extension".
    • Extension type: Select Code interpreter.

  4. In the OpenAPI Spec file section that now appears, confirm that the following fields are set correctly:

    • API name: code_interpreter_tool.
    • API description: Tool to generate and run valid Python code from a natural language description, or to run custom Python code...
    • Source: Cloud Storage.
    • OpenAPI Spec: vertex-extension-public/code_interpreter.yaml.
    • Authentication: Google service account.

  5. (Optional) In the Runtime configurations section, provide the input bucket and the output bucket.

  6. Click Create Extension.

(Optional) Query the extension

You can use the Google Cloud console to experiment with your Code Interpreter extension. Perform the following steps to invoke the extension with natural language prompts.

  1. In the Google Cloud console, go to the Vertex AI Extensions page.

    Go to Vertex AI Extensions

  2. Click the Code Interpreter extension name to open the Extensions details page.

    Code Interpreter Name.

  3. In the Enter a message box, enter a query, then view the response. Expand Extension Response sections to view the code that the extension generated and ran to produce the result.

    The following example shows the results of a query that calculated the mean value of a list of numbers entered by the user.

    Mean Value Query.

REST

Register the extension

Submit a Vertex AI API extensions.import request to register the Code Interpreter extension.

Before using any of the request data, make the following replacements:

HTTP method and URL: 

POST https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions:import

Request JSON body: 

{
  "displayName":"DISPLAY_NAME",
  "description":"DESCRIPTION",
  "manifest":{
    "name":"code_interpreter_tool",
    "description":"A Google Code Interpreter tool",
    "apiSpec":{
      "openApiGcsUri":"gs://vertex-extension-public/code_interpreter.yaml"
    },
    "authConfig":{
      "authType":"GOOGLE_SERVICE_ACCOUNT_AUTH",
      "googleServiceAccountConfig":{
        "serviceAccount":"SERVICE_ACCOUNT"
      }
    }
  }
  "runtimeConfig": {
     "codeInterpreterRuntimeConfig": {
        "fileInputGcsBucket": "INPUT_BUCKET",
        "fileOutputGcsBucket": "OUTPUT_BUCKET"
     }
  }
}

To send your request, choose one of these options:

curl

Save the request body in a file named request.json, and execute the following command: 

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions:import"

PowerShell

Save the request body in a file named request.json, and execute the following command: 

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions:import" | Select-Object -Expand Content

Run the extension

You can submit an execute operation to the Vertex AI API to generate and run Python code based on a natural language query.

Query examples:

  • Simple query: Find the max value of a list of numbers.
  • Query inline data: Data to query is provided in the request body.
  • Query with file data: Print file data.
  • Query with Cloud Storage data: Read Cloud Storage data.

Simple query

Before using any of the request data, make the following replacements:

  • PROJECT_ID: The ID of your Google Cloud project.
  • REGION: A Compute Engine region.
  • EXTENSION_ID: The ID of your code interpreter extension listed in the Extension details in the Google Cloud console.

HTTP method and URL: 

POST https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions/EXTENSION_ID:execute

Request JSON body: 

{
  "operation_id":"generate_and_execute",
  "operation_params":{
    "query":"find the max value in the list: [1,2,3,4,-5]"
  }
}

To send your request, choose one of these options:

curl

Save the request body in a file named request.json, and execute the following command: 

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions/EXTENSION_ID:execute"

PowerShell

Save the request body in a file named request.json, and execute the following command: 

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions/EXTENSION_ID:execute" | Select-Object -Expand Content

Inline data

Before using any of the request data, make the following replacements:

  • PROJECT_ID: The ID of your Google Cloud project.
  • REGION: A Compute Engine region.
  • EXTENSION_ID: The ID of your code interpreter extension listed in the Extension details in the Google Cloud console.

HTTP method and URL: 

POST https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions/EXTENSION_ID:execute

Request JSON body: 

{
  "operation_id":"generate_and_execute",
  "operation_params":{
    "query":"Calculate the total values of each column(mobile_subscribers, percent_internet_users, total_internet_users, fixed_broadband_subscribers) from the below dataset.\n\n\ncountry_name        country_code        year        mobile_subscribers        percent_internet_users        total_internet_users        fixed_broadband_subscribers\nUnited States        US        2023        333.4        90.5        303.1        200.3\nChina        CN        2023        1.613        70.2        1131.4        512.2\nIndia        IN        2023        1.165        50.7        688.5        557.2\nJapan        JP        2023        124.3        88.2        109.5        114.8\nGermany        DE        2023        102.1        90.5        92.1        100\nUnited Kingdom        UK        2023        67.1        92.7        62.2        65\nFrance        FR        2023        66.7        89        63        69.7\nBrazil        BR        2023        213.5        68        144.1        69.4\nRussia        RU        2023        203.8        74.9        152.7        51.1"
  }
}

To send your request, choose one of these options:

curl

Save the request body in a file named request.json, and execute the following command: 

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions/EXTENSION_ID:execute"

PowerShell

Save the request body in a file named request.json, and execute the following command: 

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions/EXTENSION_ID:execute" | Select-Object -Expand Content

File print

Before using any of the request data, make the following replacements:

  • PROJECT_ID: The ID of your Google Cloud project.
  • REGION: A Compute Engine region.
  • EXTENSION_ID: The ID of your code interpreter extension listed in the Extension details in the Google Cloud console.
  • FILE_NAME: The CSV file data in the request body is written to this file in the working directory.
  • BASE64_ENCODED_FILE_BYTES: File bytes in the request body must be base64-encoded.

HTTP method and URL: 

POST https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions/EXTENSION_ID:execute

Request JSON body: 

{
  "operation_id":"generate_and_execute",
  "operation_params":{
    "query":"print the csv file",
    "files":[
      {
        "name":"FILE_NAME",
        "contents":"BASE64_ENCODED_FILE_BYTES"
      }
    ]
  }
}

To send your request, choose one of these options:

curl

Save the request body in a file named request.json, and execute the following command: 

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions/EXTENSION_ID:execute"

PowerShell

Save the request body in a file named request.json, and execute the following command: 

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions/EXTENSION_ID:execute" | Select-Object -Expand Content

Cloud Storage read

Before using any of the request data, make the following replacements:

  • PROJECT_ID: The ID of your Google Cloud project.
  • REGION: A Compute Engine region.
  • EXTENSION_ID: The ID of your code interpreter extension listed in the Extension details in the Google Cloud console.
  • BUCKET_NAME: The Cloud Storage bucket that contains the CSV file to print. You must have specified this input bucket when you registered the code interpreter extension.
  • FILE_NAME: The CSV file data in the BUCKET_NAME to print.

HTTP method and URL: 

POST https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions/EXTENSION_ID:execute

Request JSON body: 

{
  "operation_id":"generate_and_execute",
  "operation_params":{
    "query":"print the csv file",
    "file_gcs_uris": ["gs://BUCKET_NAME/FILE_NAME"]
  }
}

To send your request, choose one of these options:

curl

Save the request body in a file named request.json, and execute the following command: 

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions/EXTENSION_ID:execute"

PowerShell

Save the request body in a file named request.json, and execute the following command: 

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions/EXTENSION_ID:execute" | Select-Object -Expand Content

Vertex AI Search extension

The Vertex AI Search extension uses Vertex AI Search to retrieve meaningful results from your datastore. The Vertex AI Search extension is defined in an OpenAPI Specification vertex_ai_search.yaml file.

To use the Vertex AI Search extension, you must Create a datastore in the global region with a specified search scope. For best search results, enable advanced indexing for website data and the Enterprise edition for unstructured data. See About advanced features for more information.

openapi: "3.0.0"
info:
  title: Vertex AI Search
  version: v1alpha
  description: >
    Performs search on user ingested data including website and unstructured data type.

    This extension is used when user wants to search or retrieve meaningful results from their ingested data in the Vertex AI Search service.

    Supported AuthTypes:
    - GOOGLE_SERVICE_ACCOUNT_AUTH: (only supports using Vertex AI Extension Service Agent).
paths:
  /search:
    get:
      operationId: search
      description: Retrieves the results from user's query by searching in the data store.
      parameters:
      - name: query
        in: query
        schema:
          type: string
        description: User natural language instructions for search.
        required: true
      responses:
        default:
          description: Search execution result.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/SearchResult"

components:
  schemas:
    SearchResult:
      description: Top results from search response.
      type: object
      properties:
        results:
          type: array
          items:
            type: object
            properties:
              title:
                type: string
                description: Retrieved document title.
              display_link:
                type: string
                description: Retrieved document link to display.
              link:
                type: string
                description: Retrieved document link.
              extractive_segments:
                type: array
                description: Extractive segments from the retrieved file.
                items:
                  type: string
              extractive_answers:
                type: array
                description: Extractive answers from the retrieved file. These are generated from the extractive segments.
                items:
                  type: string

Register and run the Vertex AI Search extension

The following sections show you how to register the Vertex AI Search extension using the Google Cloud console and the Vertex AI API. After registering the extension, you can run it using the Vertex AI API.

Console

Register the extension

Perform the following steps to register the Vertex AI Search extension using the Google Cloud console.

  1. In the Google Cloud console, go to the Vertex AI Extensions page.

    Go to Vertex AI Extensions

  2. Click Create Extension.

  3. In the Create a new extension dialog, fill in the following fields:

    • Extension name: Enter a name for your extension, such as "vertex_search_extension".
    • Description: (Optional) Enter an extension description, such as "A Vertex AI search extension".
    • Extension type: Select Vertex AI search.

  4. In the OpenAPI Spec file section that now appears, confirm that the following fields are set correctly:

    • API name: vertex_ai_search.
    • API description: Performs search on user ingested data including website and unstructured data type...
    • Source: Cloud Storage.
    • OpenAPI Spec: vertex-extension-public/vertex_ai_search.yaml.
    • Authentication: Google service account.

  5. In the Runtime configurations section, provide a serving configuration name. The serving configuration name is specified in the vertexAiSearchRuntimeConfig. It is formatted and completed as follows: projects/PROJECT_ID/locations/global/collections/COLLECTION_NAME/engines/ENGINE/servingConfigs/SERVING_CONFIG

    • Set COLLECTION_NAME to default_collection.
    • Set ENGINE to the application ID you received when you created your search application. To learn more, see Create a search app for website data.
    • Set SERVING_CONFIG to default_search.

  6. Click Create Extension.

REST

Register the extension

Submit an Vertex AI API extensions.import request to register the Vertex AI Search extension.

Before using any of the request data, make the following replacements:

  • PROJECT_ID: The ID of your Google Cloud project.
  • REGION: A Compute Engine region.
  • DISPLAY_NAME: The name extension that is displayed to users, such as "my_search_extension".
  • DESCRIPTION: (Optional) The extension description that is displayed to users, such as "A search extension".
  • SERVICE_ACCOUNT: (Optional) The Vertex AI Search extension uses GOOGLE_SERVICE_ACCOUNT_AUTH as shown in the sample request body. If you do not specify a service account, the extension uses the default Vertex AI Extension Service Agent service account. If you specify a different service account, grant the iam.serviceAccounts.getAccessToken permission to the Vertex AI Extension Service Agent service account on the specified service account.
  • SERVING_CONFIG_NAME: The serving configuration name is specified in the vertexAiSearchRuntimeConfig. It is formatted and completed as follows: projects/PROJECT_ID/locations/global/collections/COLLECTION_NAME/engines/ENGINE/servingConfigs/SERVING_CONFIG.
    • PROJECT_ID: The ID of your Google Cloud project.
    • Set COLLECTION_NAME to default_collection.
    • ENGINE: The application ID you received when you created your search application. To learn more, see Create a search app for website data.
    • Set SERVING_CONFIG to default_search.
    The Cloud Storage bucket that the extension will use to read input files, including the gs:// prefix, for example, gs://sample-bucket-name. If specified, you must assign the roles/storage.objectViewer role on this bucket to the Vertex Extension Custom Code Service Agent.

HTTP method and URL: 

POST https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions:import

Request JSON body: 

{
  "displayName":"DISPLAY_NAME",
  "description":"DESCRIPTION",
  "manifest":{
    "name":"code_interpreter_tool",
    "description":"A Google Code Interpreter tool",
    "apiSpec":{
      "openApiGcsUri":"gs://vertex-extension-public/code_interpreter.yaml"
    },
    "authConfig":{
      "authType":"GOOGLE_SERVICE_ACCOUNT_AUTH",
      "googleServiceAccountConfig":{
        "serviceAccount":"SERVICE_ACCOUNT"
      }
    }
  }
  "runtimeConfig": {
     "vertexAiSearchRuntimeConfig": {
        "servingConfigName": "SERVING_CONFIG_NAME",
     }
  }
}

To send your request, choose one of these options:

curl

Save the request body in a file named request.json, and execute the following command: 

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions:import"

PowerShell

Save the request body in a file named request.json, and execute the following command: 

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions:import" | Select-Object -Expand Content

Run the extension

You can submit an execute operation to the Vertex AI API to obtain meaningful results from your datastore.

Before using any of the request data, make the following replacements:

  • PROJECT_ID: The ID of your Google Cloud project.
  • REGION: A Compute Engine region.
  • EXTENSION_ID: The ID of your Vertex AI Search extension listed in the Extension details in the Google Cloud console.

HTTP method and URL: 

POST https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions/EXTENSION_ID:execute

Request JSON body: 

{
  "operation_id":"search",
  "operation_params":{
    "query":"Housing affordability since 2010",
  }
}

To send your request, choose one of these options:

curl

Save the request body in a file named request.json, and execute the following command: 

curl -X POST \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     -H "Content-Type: application/json; charset=utf-8" \
     -d @request.json \
     "https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions/EXTENSION_ID:execute"

PowerShell

Save the request body in a file named request.json, and execute the following command: 

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method POST `
    -Headers $headers `
    -ContentType: "application/json; charset=utf-8" `
    -InFile request.json `
    -Uri "https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION/extensions/EXTENSION_ID:execute" | Select-Object -Expand Content