Evaluate models using Vertex AI

This page describes how to evaluate models using Vertex AI. For an overview, see model evaluation in Vertex AI.

Prerequisites

  1. Follow the steps at Set up a project and a development environment. In addition, enable the following services:

  2. Vertex AI can evaluate models that are trained either through AutoML or custom training. For the Google Cloud console guide, you should have a trained model imported to Vertex AI Model Registry.

  3. Upload your test dataset to BigQuery or Cloud Storage. The test dataset should contain the ground truth, which is the actual result expected for an inference. Obtain the link to the file or the dataset ID.

  4. Have a batch inference output in the form of a BigQuery table or Cloud Storage URI.

  5. Make sure your default Compute Engine service account has the following IAM permissions:

    • Vertex AI Administrator (aiplatform.admin)
    • Vertex AI Service Agent (aiplatform.serviceAgent)
    • Storage Object Admin (storage.objectAdmin)
    • Dataflow Worker (dataflow.worker)
    • BigQuery Data Editor (bigquery.dataEditor) (only required if you are providing data in the form of BigQuery tables)

Create an evaluation

Console

  1. In the Google Cloud console, go to the Vertex AI Models page.

    Go to the Models page

  2. Click the name of the model you want to evaluate.

  3. Click the version number for the model.

  4. On the Evaluate tab, click Create Evaluation.

  5. Enter an Evaluation name.

  6. Select an Objective, such as classification or regression.

  7. Enter the Evaluation target column name, which is the column from the training data that the model is trained to predict.

  8. For Select source, select the source for your test dataset.

    1. For BigQuery table, enter the BigQuery path.

    2. For File on Cloud Storage, enter the Cloud Storage path.

  9. For Batch prediction output, select an output format.

    1. Enter the BigQuery path or Cloud Storage URI.

  10. Click Start Evaluation.

Python

To view the Vertex AI API model evaluation workflow in Vertex AI Pipelines, see the example notebooks for the following model types:

Python SDK

The SDK for evaluating models with Vertex AI is in Experimental. To sign up for the Experimental, fill out the onboarding form.

Vertex AI automatically sends an email notification when a model evaluation job is complete.

View evaluation metrics

Console

  1. In the Google Cloud console, go to the Vertex AI Models page.

    Go to the Models page

  2. Navigate to the model version.

  3. View metrics in the Evaluate tab.

Python

To view the Vertex AI API model evaluation workflow in Vertex AI Pipelines, see the example notebooks for the following model types:

Python SDK

The SDK for evaluating models with Vertex AI is in Experimental. To sign up for the Experimental, fill out the onboarding form.

Compare evaluation metrics

You can compare evaluation results across different models, model versions, and evaluation jobs. For more information about model versioning, see Versioning in Model Registry.

You can only compare models of the same type, such as classification, regression, or forecasting. When comparing different models, all the model versions must be the same type.

You can only compare 5 or fewer evaluations at a time.

  1. Go to the Vertex AI Model Registry in the Google Cloud console:

    Go to the Models page

  2. Navigate to your model or model version:

    • To compare across different models on the Models page, select the checkboxes next to the names of the models you want to compare.

    • To compare across different model versions:

      1. Click the name of your model on the Models page to open the list of model versions.

      2. Select the checkboxes next to the versions you want to compare.

    • To compare across evaluation jobs for the same model version:

      1. Click the name of your model on the Models page to open the list of model versions.

      2. Click the version number.

      3. Select the checkboxes next to the evaluation jobs you want to compare.

  3. Click Compare.

What's next