Evaluate Gen AI agents

After you build and evaluate your Gen AI model, you might use the model to build an agent such as a chatbot. The Gen AI evaluation service lets you measure your agent's ability to complete tasks and goals for your use case.

Overview

You have the following options to evaluate your agent:

• Final response evaluation: Evaluate the final output of an agent (whether or not the agent achieved its goal).

• Trajectory evaluation: Evaluate the path (sequence of tool calls) the agent took to reach the final response.

With the Gen AI evaluation service, you can trigger an agent execution and get metrics for both trajectory evaluation and final response evaluation in one Vertex AI SDK query.

Supported agents

The Gen AI evaluation service supports the following categories of agents:

Defining metrics for agent evaluation

Define your metrics for final response or trajectory evaluation:

Final response evaluation

Final response evaluation follows the same process as model response evaluation. For more information, see Define your evaluation metrics.

Trajectory evaluation

The following metrics help you to evaluate the model's ability to follow the expected trajectory:

In addition, the following two agent performance metrics are added to the evaluation results by default. You don't need to specify them in EvalTask.

Prepare your dataset for agent evaluation

Prepare your dataset for final response or trajectory evaluation.

The data schema for final response evaluation is similar to that of model response evaluation.

For computation-based trajectory evaluation, your dataset needs to provide the following information:

Evaluation dataset examples

The following examples show datasets for trajectory evaluation. Note that reference_trajectory is required for all metrics except trajectory_single_tool_use.

reference_trajectory = [
# example 1
[
  {
    "tool_name": "set_device_info",
    "tool_input": {
        "device_id": "device_2",
        "updates": {
            "status": "OFF"
        }
    }
  }
],
# example 2
[
    {
      "tool_name": "get_user_preferences",
      "tool_input": {
          "user_id": "user_y"
      }
  },
  {
      "tool_name": "set_temperature",
      "tool_input": {
          "location": "Living Room",
          "temperature": 23
      }
    },
  ]
]

predicted_trajectory = [
# example 1
[
  {
    "tool_name": "set_device_info",
    "tool_input": {
        "device_id": "device_3",
        "updates": {
            "status": "OFF"
        }
    }
  }
],
# example 2
[
    {
      "tool_name": "get_user_preferences",
      "tool_input": {
          "user_id": "user_z"
      }
    },
    {
      "tool_name": "set_temperature",
      "tool_input": {
          "location": "Living Room",
          "temperature": 23
      }
    },
  ]
]

eval_dataset = pd.DataFrame({
    "predicted_trajectory": predicted_trajectory,
    "reference_trajectory": reference_trajectory,
})

Import your evaluation dataset

You can import your dataset in the following formats:

• JSONL or CSV file stored in Cloud Storage

• BigQuery table

• Pandas DataFrame

The following code demonstrates how to import example datasets from a Cloud Storage bucket:

# dataset name to be imported
dataset = "on-device" # customer-support, content-creation

# copy the tools and dataset file
!gcloud storage cp gs://cloud-ai-demo-datasets/agent-eval-datasets/{dataset}/tools.py .
!gcloud storage cp gs://cloud-ai-demo-datasets/agent-eval-datasets/{dataset}/eval_dataset.json .

# load the dataset examples
import json

eval_dataset = json.loads(open('eval_dataset.json').read())

# run the tools file
%run -i tools.py

Run agent evaluation

Run an evaluation for trajectory or final response evaluation:

For agent evaluation, you can mix response evaluation metrics and trajectory evaluation metrics like in the following code:

single_tool_use_metric = TrajectorySingleToolUse(tool_name='tool_name')

eval_task = EvalTask(
    dataset=EVAL_DATASET,
    metrics=[
        "rouge_l_sum",
        "bleu",
        custom_response_eval_metric,
        "trajectory_exact_match",
        "trajectory_precision",
        single_tool_use_metric
        pointwise_trajectory_eval_metric # llm-based metric
    ],
)
eval_result = eval_task.evaluate(
    runnable=RUNNABLE,
)

Metric customization

You can customize a large language model-based metric for trajectory evaluation using a templated interface or from scratch. You can also define a custom computation-based metric for trajectory evaluation.

View and interpret results

For trajectory evaluation or final response evaluation, the evaluation results are displayed as follows:

The evaluation results contain the following information: