Playbook tools

Using tools, you can connect playbooks to external systems. These systems can augment the knowledge of playbooks and empower them to execute complex tasks efficiently.

You can use built-in tools or build customized tools to suit your requirements.

Tool testing

Once you have created a tool, you can use the tool testing feature to verify that it works. When viewing a tool, click the Test button above the tool pane. This opens the tool for input in the simulator. Provide tool input, then click View Output to verify the correct tool output.

You can also use the tool testing feature when you add a tool to an example.

Built-in tools

Built-in tools are hosted by Google. You can activate these tools in agents without the need for manual configuration.

The supported built-in tools are:

  • Code Interpreter: A Google first-party tool that combines the capability of code generation and code execution and allows the user to perform various tasks, including: data analysis, data visualization, text processing, solving equations or optimization problems.

Your agent is optimized to determine how and when these tools should be invoked, but you can provide additional examples to fit your use cases.

Examples should have a schema like the following:

{
  "toolUse": {
    "tool": "projects/PROJECT_ID/locations/LOCATION_ID/agents/AGENT_ID/tools/df-code-interpreter-tool",
    "action": "generate_and_execute",
    "inputParameters": [
      {
        "name": "generate_and_execute input",
        "value": "4 + 4"
      }
    ],
    "outputParameters": [
      {
        "name": "generate_and_execute output",
        "value": {
          "output_files": [
            {
              "name": "",
              "contents": ""
            }
          ],
          "execution_result": "8",
          "execution_error": "",
          "generated_code": "GENERATED_CODE"
        }
      }
    ]
  }
}

OpenAPI tools

An agent can connect to an external API using an OpenAPI tool by providing the OpenAPI schema. By default, the agent will call the API on your behalf. Alternatively, you can execute OpenAPI tools on the client side.

Example schema:

openapi: 3.0.0
info:
  title: Simple Pets API
  version: 1.0.0
servers:
  - url: 'https://api.pet-service-example.com/v1'
paths:
  /pets/{petId}:
    get:
      summary: Return a pet by ID.
      operationId: getPet
      parameters:
        - in: path
          name: petId
          required: true
          description: Pet id
          schema:
            type: integer
      responses:
        200:
          description: OK
  /pets:
    get:
      summary: List all pets
      operationId: listPets
      parameters:
        - name: petName
          in: query
          required: false
          description: Pet name
          schema:
            type: string
        - name: label
          in: query
          description: Pet label
          style: form
          explode: true
          required: false
          schema:
            type: array
            items:
              type: string
        - name: X-OWNER
          in: header
          description: Optional pet owner provided in the HTTP header
          required: false
          schema:
            type: string
        - name: X-SESSION
          in: header
          description: Dialogflow session id
          required: false
          schema:
            $ref: "@dialogflow/sessionId"
      responses:
        '200':
          description: An array of pets
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: '#/components/schemas/Pet'
    post:
      summary: Create a new pet
      operationId: createPet
      requestBody:
        description: Pet to add to the store
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Pet'
      responses:
        '201':
          description: Pet created
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Pet'
components:
  schemas:
    Pet:
      type: object
      required:
        - id
        - name
      properties:
        id:
          type: integer
          format: int64
        name:
          type: string
        owner:
          type: string
        label:
          type: array
          items:
            type: string

You can optionally use the internal schema reference @dialogflow/sessionId as parameter schema type. With this parameter schema type, the Dialogflow session ID for the current conversation will be supplied as a parameter value. For example:

- name: X-SESSION
   in: header
   description: Dialogflow session id
   required: false
   schema:
     $ref: "@dialogflow/sessionId"

OpenAPI tool limitations

The following limitations apply:

  • Supported parameter types are path, query, header. The cookie parameter type is not supported yet.
  • Parameters defined by OpenAPI schema support the following data types: string, number, integer, boolean, array. The object type is not supported yet.
  • You currently can't specify query parameters in the console example editor.
  • Request and response body must be empty or JSON.

OpenAPI tool API authentication

The following authentication options are supported when calling an external API:

  • Dialogflow Service Agent auth
    • Dialogflow can generate an ID token or access token using Dialogflow Service Agent. The token is added in the authorization HTTP header when Dialogflow calls an external API.
    • An ID token can be used to access Cloud Run functions and Cloud Run services after you grant the roles/cloudfunctions.invoker and roles/run.invoker roles to service-agent-project-number@gcp-sa-dialogflow.iam.gserviceaccount.com. If the Cloud Run functions and Cloud Run services are in the same resource project, you don't need additional IAM permission to call them.
    • An access token can be used to access other Google Cloud APIs after you grant required roles to service-agent-project-number@gcp-sa-dialogflow.iam.gserviceaccount.com.

  • Service Account auth

    • Service accounts can be used to authenticate tool requests to any Google APIs which support it. Because service accounts are principals, they can access resources in your project by granting it a role, just like you would for any other principal. The service account email will be used to generate an access token which will be sent in the Authorization header of the tool request.

    • To use this feature, the following permissions are required:

      • roles/iam.serviceAccountUser role to the user who is creating or updating the tool to use the service account auth.
      • roles/iam.serviceAccountTokenCreator role to the Dialogflow Service Agent.

    • If you are trying to use service accounts across multiple projects, ensure that your organization policy supports it. Both the permissions need to be granted in the project which contains the service account.

  • API key

    • You can configure API key authentication by providing the key name, request location (header or query string) and API key so that Dialogflow passes the API key in the request.
    • We recommend you to supply your API key using Secret Manager.

  • OAuth

    • OAuth Client Credential flow is supported for server-to-server authentication:

      • This flow can be used if the Agent Builder console is the resource owner and no end-user authorization is needed.
      • Client ID, Client Secret and Token endpoint from OAuth provider need to be configured in Dialogflow.
      • Dialogflow exchanges an OAuth access token from OAuth provider and passes it in the auth header of the request.

    • For other OAuth flows that requires end-user authorization like Authorization Code Flow and PKCE flow:

      1. You will need to implement your own sign-in UI and obtain the access token on the client side.

      2. You can then either:

        a. Use the Bearer Token authentication option to pass the token to the OpenAPI Tool. Dialogflow will include this token in the authorization header when invoking the tool.

        b. Use the Function tool to invoke the tool yourself on the client side and pass the tool call result to Dialogflow.

  • Bearer Token

    • You can configure Bearer authentication to dynamically pass the Bearer token from the client. This token is included in the auth header of the request.
    • When setting up tool authentication, you can designate a session parameter to act as the Bearer token. For instance, use $session.params.<parameter-name-for-token> to specify the token.
    • At runtime, assign the Bearer token to the session parameter:
    DetectIntentRequest {
  ...
  query_params {
    parameters {
      <parameter-name-for-token>: <the-auth-token>
    }
  }
  ...
}
    • If you need to configure a static token instead of fetching the token from a session parameter, we recommend you to supply your token using Secret Manager.

  • Mutual TLS authentication

    • See the Mutual TLS authentication documentation.
    • Custom client certificates are supported. You can set up client certificates at the agent level under the security tab in agent settings. The certificate (PEM format) and private key (PEM format) are required fields. Once set, this client certificate will be used during mutual TLS for all tools and webhooks.

  • Custom CA certificate

Secret Manager Authentication

If you use OAuth, API key, or Bearer token, you can store the credentials as secrets using Secret Manager. Here are the necessary steps to authenticate your tool using secrets:

  1. Create your secret if you don't have one already.
  2. Grant Dialogflow Service Agent the Secret Manager Secret Accessor (roles/secretmanager.secretAccessor) role on the new secret.
  3. Copy your credential into the clipboard.
  4. Add a new secret version to your secret. Paste your credential as the secret value.
    • Omit any newline character at the end.
  5. Copy the name of the secret version you just added. The name format is projects/{project_id}/secrets/{secret_id}/versions/{version_id}".
  6. Open the tool edit screen, then:
    • If you use OAuth, select OAuth as the Authentication type, then click Secret version under Client secret and paste the secret version name to the Secret version input box.
    • If you use API key, select API key as the Authentication type, then click Secret version under API key. Paste the secret version name to the Secret version input box.
    • If you use Bearer token, select Bearer token as the Authentication type, then click Secret version under Bearer token. Paste the secret version name to the Secret version input box.
  7. Click Save.

OpenAPI tool private network access

OpenAPI tool integrates with Service Directory private network access, so it can connect to API targets inside your VPC network. This keeps the traffic within the Google Cloud network and enforces IAM and VPC Service Controls.

To set up an OpenAPI tool targeting a private network:

  1. Follow Service Directory private network configuration to configure your VPC network and Service Directory endpoint.

  2. The Dialogflow Service Agent service account with the following address must exist for your agent project: 

    service-agent-project-number@gcp-sa-dialogflow.iam.gserviceaccount.com
     Grant the Dialogflow Service Agent service account the following IAM roles:

    • servicedirectory.viewer of the Service Directory project
    • servicedirectory.pscAuthorizedService of the network project

  3. Provide Service Directory Service along with the OpenAPI schema and optional authentication information when creating the tool.

OpenAPI tool session parameter access

Open API tool inputs are derived from the users conversation with the LLM using the schema as a guide. In some situations, inputs may need to be derived from session parameters collected during a flow or provided as a query parameter input along with the user input.

The session parameter that needs to be passed as an input can be specified as

     parameters:
       - in: query
         name: petId
         required: true
         description: Pet id
         schema:
           type: integer
           x-agent-input-parameter: petId # Reads from the $session.params.petId
       - in: header
         name: X-other
         schema:
           type: string
           x-agent-input-parameter: $request.payload.header # Reads from the header specified in the request payload input
     requestBody:
       required: false
       content:
         application/json:
           schema:
             type: object
             properties:
               name:
                 type: string
                 x-agent-input-parameter: petName # Reads from the $session.params.petName
                 description: Name of the person to greet (optional).
               breed:
                 type: string
                 description: Bread of the pet.

If no such session parameter is available, the LLM generated input will be passed to the tool.

OpenAPI tool default values

The Open API schema can be used to specify default values. The default values are only used if there is no LLM generated input value or session parameter based input value for that parameter or property.

The default values can be specified as part of the schema as follows:

     parameters:
       - in: query
         name: zipcode
         required: true
         description: Zip code to search for
         schema:
           type: integer
           default: 94043
     requestBody:
       content:
         application/json:
           schema:
             type: object
             properties:
               breed:
                 type: string
                 description: Bread of the pet.
               page_size:
                 type: integer
                 description: Number of pets to return.
                 default: 10

If no LLM generated value, session parameter value or default value is present, the input will be unspecified.

Data store tools

For information about using data store tools with a playbook, see the data store tools documentation.

Connector tools

Connector tools can be used by an agent to perform actions using your Connections configured in Integration Connectors. Each connector tool is configured with a single Connection and one or more actions. If needed, multiple tools can be created for a single Connection to group different actions together for your agent to use.

The connector tool supports the following Connector types:

Examples should be used to enhance the agent use of connector tools by demonstrating how the agent should call the tool and use the response.

Create a connection

To create a connection and connect it to your agent, you can navigate to the Tools > Create, select the Connector tool type, your chosen connector type, and use the Create Connection button. This will navigate you to Integration Connectors creation with a number of fields pre-filled.

Alternatively, you can navigate to Integration Connectors and follow the instructions to create a connection.

Connector actions

For each connector tool, there are two types of actions that can be made available to your agent (see Entities, operations, and actions for more information):

  1. Entity CRUD operations

    Each of your Connections have "entities" corresponding to the objects of that data source (for BigQuery, these are tables; for Salesforce, these are objects, like 'Order' or 'Case').

    You can perform CRUD operations on each entity:

    • Create: creates an entity with specified field values
    • List: filter-based search of entity instances
    • Update: filter-based method for altering entity field values
    • Delete: deletes an entity
    • Get retrieves a single entity using the entityId

    Learn more about entity CRUD operations details in the Connectors docs.

  2. Connector-specific actions

    Many Connectors support an 'ExecuteCustomQuery' action, which allows for executing a SQL query against the data source, where each of the data source entities can be referenced as tables. See this list for supported connectors.

    Additional actions differ based on the connector type - for example, see the BigQuery connector actions or the Salesforce connector actions.

Configuring input / output fields for CRUD operations

By selecting specific input or output fields for your connector tool action to use, you can limit the complexity of these actions for the agent.

For example, if you only need to create an entity with a subset of its fields, configuring this set of fields in your action simplifies the action for the agent.

Specifying a set of output fields reduces the tool response size (helpful if token limits are a concern) and simplifies the agent's handling of the output by exposing only the relevant fields.

Authentication

If the connection you're using is configured to allow authentication override, then the tool can be configured to pass through credentials from specified session parameters.

You as the agent builder are responsible for how these credentials are populated to the session parameters and the tool will automatically pass them through to the data source to use for authentication when the tool's actions are called.

Function tools

If you have functionality accessible by your client code, but not accessible by OpenAPI tools, you can use function tools. Function tools are always executed on the client side, not by the agent.

The process is as follows:

  1. Your client code sends a detect intent request.
  2. The agent detects that a function tool is required, and the detect intent response contains the name of the tool along with input arguments. This session is paused until another detect intent request is received with the tool result.
  3. Your client code calls the tool.
  4. Your client code sends another detect intent request that provides the tool result as output arguments.

The following example shows the input and output schema of a function tool:

{
  "type": "object",
  "properties": {
    "location": {
      "type": "string",
      "description": "The city and state, for example, San Francisco, CA"
    }
  },
  "required": [
    "location"
  ]
}

{
  "type": "object",
  "properties": {
    "temperature": {
      "type": "number",
      "description": "The temperature"
    }
  }
}

The following example shows the initial detect intent request and response using REST:

HTTP method and URL:
POST https://REGION_ID-dialogflow.googleapis.com/v3/projects/PROJECT_ID/locations/LOCATION_ID/agents/AGENT_ID/sessions/SESSION_ID:detectIntent
{
  "queryInput": {
    "text": {
      "text": "what is the weather in Mountain View"
    },
    "languageCode": "en"
  }
}

{
  "queryResult": {
    "text": "what is the weather in Mountain View",
    "languageCode": "en",
    "responseMessages": [
      {
        "source": "VIRTUAL_AGENT",
        "toolCall": {
          "tool": "<tool-resource-name>",
          "action": "get-weather-tool",
          "inputParameters": {
            "location": "Mountain View"
          }
        }
      }
    ]
  }
}

The following example shows the second detect intent request, which provides the tool result:

{
  "queryInput": {
    "toolCallResult": {
      "tool": "<tool-resource-name>",
      "action": "get-weather-tool",
      "outputParameters": {
        "temperature": 28.0
      }
    },
    "languageCode": "en"
  }
}

Client side execution

Like function tools, OpenAPI and data store tools can be executed on the client side by applying an API override when interacting with the session.

For example:

DetectIntentRequest {
  ...
  query_params {
    playbook_state_override {
      playbook_execution_mode: ALWAYS_CLIENT_EXECUTION
    }
  }
  ...
}

The process is as follows:

  1. Your client code sends a detect intent request that specifies client execution.
  2. The agent detects that a tool is required, and the detect intent response contains the name of the tool along with input arguments. This session is paused until another detect intent request is received with the tool result.
  3. Your client code calls the tool.
  4. Your client code sends another detect intent request that provides the tool result as output arguments.
Previous
Playbook parameters
Next
Playbook settings