Define your evaluation metrics

The first step to evaluate your generative models or applications is to identify your evaluation goal and define your evaluation metrics. This page provides an overview of concepts related to defining evaluation metrics for your use case.

Overview

Generative AI models can be used to create applications for a wide range of tasks, such as summarizing news articles, responding to customer inquiries, or assisting with code writing. The Gen AI Evaluation Service in Vertex AI lets you evaluate any model with explainable metrics.

For example, you might be developing an application to summarize articles. To evaluate your application's performance on that specific task, consider the criteria you would like to measure and the metrics that you would use to score them:

• Criteria: Single or multiple dimensions you would like to evaluate upon, such as conciseness, relevance, correctness, or appropriate choice of words.

• Metrics: A single score that measures the model output against criteria.

The Gen AI Evaluation Service provides two major types of metrics:

• Model-based metrics: Our model-based metrics use a proprietary Google model as a judge. You can measure model-based metrics pairwise or pointwise:

  • Pointwise metrics: Let the judge model assess the candidate model's output based on the evaluation criteria. For example, the score could be 0~5, where 0 means the response does not fit the criteria, while 5 means the response fits the criteria well.

  • Pairwise metrics: Let the judge model compare the responses of two models and pick the better one. This is often used when comparing a candidate model with the baseline model.

• Computation-based metrics: These metrics are computed using mathematical formulas to compare the model's output against a ground truth or reference. Commonly used computation-based metrics include ROUGE and BLEU.

You can use computation-based metrics standalone, or together with model-based metrics. Use the following table to decide when to use model-based or computation-based metrics:

To get started, see Prepare your dataset and Run evaluation.

Define your model-based metrics

Model-based evaluation involves using a large language model (LLM) as a judge model to evaluate the outputs of the candidate model. Our judge models are calibrated with human raters to ensure their quality. They are managed and available out of the box. The process of model-based evaluation varies based on the evaluation metrics you provide.

Model-based evaluation follows this process:

1. Data preparation: You provide evaluation data in the form of input prompts. The candidate models receive the prompts and generate corresponding responses.

2. Evaluation: The evaluation metrics and generated responses are sent to the judge model. The judge model evaluates each response individually, providing a row-based assessment.

3. Aggregation and explanation: The Gen AI Evaluation Service aggregates these individual assessments into an overall score. The output also includes chain-of-thought explanations for each judgment, outlining the rationale behind the selection.

The Gen AI Evaluation Service offers the following options to set up your model-based metrics with the Vertex AI SDK:

As an example, you might want to develop a generative AI application that returns fluent and entertaining responses. For this application, you can define two criteria for evaluation using the templated interface:

• Fluency: Sentences flow smoothly, avoiding awkward phrasing or run-on sentences. Ideas and sentences connect logically, using transitions effectively where needed.

• Entertainment: Short, amusing text that incorporates emoji, exclamations, and questions to convey quick and spontaneous communication and diversion.

To turn those two criteria into a metric, you want an overall score ranging from -1 ~ 1 called custom_text_quality. You can define a metric like this:

# Define a pointwise metric with two criteria: Fluency and Entertaining.
custom_text_quality = PointwiseMetric(
    metric="custom_text_quality",
    metric_prompt_template=PointwiseMetricPromptTemplate(
        criteria={
            "fluency": (
                "Sentences flow smoothly and are easy to read, avoiding awkward"
                " phrasing or run-on sentences. Ideas and sentences connect"
                " logically, using transitions effectively where needed."
            ),
            "entertaining": (
                "Short, amusing text that incorporates emojis, exclamations and"
                " questions to convey quick and spontaneous communication and"
                " diversion."
            ),
        },
        rating_rubric={
            "1": "The response performs well on both criteria.",
            "0": "The response is somewhat aligned with both criteria",
            "-1": "The response falls short on both criteria",
        },
    ),
)

For a complete list of metric prompt templates, see Metric prompt templates for evaluation.

Choose between pointwise or pairwise evaluation

Use the following table to decide when you want to use pointwise or pairwise evaluation:

Computation-based metrics

Computation-based metrics compare whether the LLM-generated results are consistent with a ground-truth dataset of input and output pairs. The commonly used metrics can be categorized into the following groups:

• Lexicon-based metrics: Use math to calculate the string similarities between LLM-generated results and ground truth, such as Exact Match and ROUGE.
• Count-based metrics: Aggregate the number of rows that hit or miss certain ground-truth labels, such as F1-score, Accuracy, and Tool Name Match.
• Embedding-based metrics: Calculate the distance between the LLM-generated results and ground truth in the embedding space, reflecting their level of similarity.

General text generation

The following metrics help you to evaluate the model's ability to ensure the responses are useful, safe, and effective for your users.

Tool use and function calling

The following metrics help you to evaluate the model's ability to predict a valid tool (function) call.

In the generative AI evaluation service, you can use computation-based metrics through the Vertex AI SDK for Python.

Baseline evaluation quality for generative tasks

When evaluating the output of generative AI models, note that the evaluation process is inherently subjective, and the quality of evaluation can vary depending on the specific task and evaluation criteria. This subjectivity also applies to human evaluators. For more information about the challenges of achieving consistent evaluation for generative AI models, see Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena and Learning to summarize from human feedback.

What's next

• Find a model-based metrics template.

• Prepare your evaluation dataset.

• Run an evaluation.

• Try an evaluation example notebook.