Evaluation methods and metrics

This page provides an overview of our current evaluation metrics and how to use each metric.

Pointwise versus pairwise

You must identify your evaluation goal before determining which metrics to apply. This includes determining whether to perform pointwise or pairwise evaluation, as mentioned in Evaluation paradigms.

Paradigm	When to use
Pointwise	Understanding how your model behaves in production: Explore strengths and weaknesses of a single model. Identifying which behaviors to focus on when tuning. Getting the baseline performance of a model.
Pairwise	Determining which model to put into production: Choose between model types. For example, Gemini-Pro versus Claude 3. Choose between different prompts. Determines if tuning made improvements to a baseline model.

Tasks and metrics

You can evaluate large language models (LLMs) across the following four broad tasks:

• Summarization
• Question answering
• Tool use
• General text generation

For each task, you can evaluate LLMs using a fixed set of granular metrics, such as quality, relevance, and helpfulness. You can evaluate any combination of these metrics on a given evaluation instance. For each metric, you must specify the input parameters.

To help you identify which tasks and metrics you want to evaluate, consider the role of your model and the model behaviors that are most important to you.

Summarization

The following metrics help you to evaluate model summarization.

Question answering

The following metrics help you to evaluate the model's ability to answer questions.

Tool use

The following metrics help you to evaluate the model's ability to predict a valid tool call.

General text generation

The following metrics help you to evaluate the model's ability to ensure the responses are useful, safe, and effective for your users.

Understand metric results

Different metrics produce different output results. Therefore, we explain the meaning of the results and how they are produced so that you can interpret your evaluations.

Score and pairwise choice

Based on the evaluation paradigm you choose, you will see score in a pointwise evaluation result or pairwise_choice in your pairwise evaluation result.

For pointwise evaluation, the score in the evaluation result is the numerical representation of the performance or the quality of the model output being assessed. The score scales are different per metric: They can be binary (0 and 1), Likert scale (1 to 5, or -2 to 2), or float (0.0 to 1.0). See the tasks and metrics section for a detailed description of the score values for each metric.

For pairwise metrics, the pairwise_choice in the evaluation result is an enumeration that indicates whether the candidate or baseline prediction is better with the following possible values:

• BASELINE: baseline prediction is better
• CANDIDATE: candidate prediction is better

When running pairwise evaluations with the evaluation pipeline service, 'A' and 'B' are output choice options instead of baseline and candidate predictions.

Explanation and confidence score

Explanation and confidence score are features of model-based evaluation.

Metric	Definition	Type	How it works
Explanation	The autorater's reason for its choice.	String	We use chain-of-thought reasoning to guide the AutoRater to explain its rationale behind each verdict. Forcing the AutoRater to reason is shown to improve evaluation accuracy.
Confidence score	A score between 0 and 1, which signifies how confident the AutoRater was with its verdict. A score closer to 1 means higher confidence.	Float	Model-based evaluation utilizes the self-consistency decoding strategy to determine evaluation results, which is shown to improve evaluation accuracy. Thus, for a single evaluation input we sample the AutoRater several times and return the consensus result. The variation of these sampled results is a measurement of AutoRater's confidence in its verdict.

Examples

These examples let you practice how to read and understand the results.

Example 1

In the first example, consider evaluating a pointwise evaluation instance for summarization_quality: The score from the pointwise evaluation of summarization_quality metric is 4 (from scale 1 to 5), which means the prediction is a good summary. Furthermore, the explanation in the evaluation result shows why the autorater thinks the prediction deserves the score 4, and not a score that's higher or lower. The confidence score from the evaluation result shows how confident the autorater is about the score, and a confidence score of 0.8 (from scale 0.0 to 1.0) means that the autorater is confident the summary is worth scoring 4.

Dataset

• instruction: "Summarize the text in a way that a five-year-old can understand."
• context: "Social Media Platform Faces Backlash Over Content Moderation Policies\nA prominent social media platform finds itself embroiled in controversy as users and content creators express discontent over its content moderation policies. Allegations of biased censorship, inconsistent enforcement, and suppression of certain viewpoints have sparked outrage among users who claim that the platform is stifling free speech. On the other hand, the platform asserts that its policies are designed to maintain a safe and inclusive online environment. This controversy raises broader questions about the role of social media in shaping public discourse and the responsibilities of platforms in curating online content."
• prediction: "People are upset with a website because they think it's not being fair with what people can post. Some people say the website is stopping them from saying what they want. But the website says it's trying to keep everyone safe. This makes people wonder if websites should control what people can say online."

Result

• score: 4,
• explanation: The summary in response follows the instruction to summarize the context in a way that a five-year old can understand. It is grounded in the context and provides important details in its summarization. However, the language used in the response is a bit verbose.
• confidence: 0.8

Example 2

The second example is a pairwise side-by-side comparison evaluation on pairwiseQuestionAnsweringQuality: The pairwiseChoice result shows the candidate response "France is a country located in Western Europe." is preferred by the autorater as compared to the baseline response "France is a country." to answer the question in the instruction with background information from the context. Similar to pointwise results, an explanation and confidence score are also provided to explain why the candidate response is better than the baseline response (candidate response is more helpful in this case) and how confident the autorater is about this choice (the confidence 1 here means the autorater is as certain as possible about this choice).

Dataset

• prediction: "France is a country located in Western Europe.",
• baseline_prediction: "France is a country.",
• instruction: "Where is France?",
• context: "France is a country located in Western Europe. It's bordered by Belgium, Luxembourg, Germany, Switzerland, Italy, Monaco, Spain, and Andorra. France's coastline stretches along the English Channel, the North Sea, the Atlantic Ocean, and the Mediterranean Sea. Known for its rich history, iconic landmarks like the Eiffel Tower, and delicious cuisine, France is a major cultural and economic power in Europe and throughout the world.",

Result

• pairwiseChoice: CANDIDATE,
• explanation: BASELINE response is grounded but does not fully answer the question. CANDIDATE response, however, is correct and provides helpful details on the location of France.
• confidence: 1

What's next

• Try an evaluation example notebook.
• Learn about online versus pipeline evaluation.
• Learn about computation-based evaluation.
• Learn about pairwise model-based evaluation.
• Learn how to tune a foundation model.