評估 AutoML 分類和迴歸模型

本頁面說明如何評估 AutoML 分類和迴歸模型。

Vertex AI 提供模型評估指標，協助您判斷模型效能，例如精確度和召回率指標。Vertex AI 會使用測試集計算評估指標。

事前準備

您必須先訓練模型，才能評估模型。

如何使用模型評估指標

模型評估指標會透過量化方式，評估模型在測試集的效能。您如何解讀及使用這些指標，取決於您的業務需求和模型接受訓練解決的問題。舉例來說，您可能對偽陽性比偽陰性有較低的容忍度，反之亦然。這類問題會影響您要著重的指標。

取得評估指標

您可以取得模型的評估指標匯總資料，對於某些目標，您還可以取得特定類別或標籤的評估指標。特定類別或標籤的評估指標也稱為評估區塊。以下內容說明如何使用 Google Cloud 主控台或 API，取得匯總評估指標和評估區塊。

Google Cloud 控制台

  1. 在 Google Cloud 控制台的 Vertex AI 專區中，前往「Models」頁面。

    前往「Models」(模型) 頁面

  2. 在「Region」下拉式選單中，選取模型所在的地區。

  3. 在模型清單中點選模型，即可開啟模型的「Evaluate」分頁。

    在「Evaluate」分頁中，您可以查看模型的匯總評估指標，例如「Average precision」和「Recall」。

    如果模型目標含有評估配量，主控台會顯示標籤清單。您可以按一下標籤，查看該標籤的評估指標，如以下範例所示：

    主控台中的標籤選項

API

每種資料類型和目標的評估指標 API 要求都相同，但輸出結果不同。以下範例顯示相同要求，但回應不同。

取得匯總模型評估指標

匯總模型評估指標可提供模型的整體資訊。如要查看特定切片的相關資訊，請列出模型評估切片。

如要查看匯總模型評估指標，請使用 projects.locations.models.evaluations.get 方法。

請選取下方目標的分頁標籤：

分類

Vertex AI 會傳回信心指標陣列。每個元素會顯示不同 confidenceThreshold 值的評估指標 (從 0 開始，最高為 1)。查看不同的門檻值，即可瞭解門檻對精確度和查全率等其他指標的影響。

請選取對應於您語言或環境的分頁：

REST

使用任何要求資料之前，請先替換以下項目：

  • LOCATION：模型儲存的區域。
  • PROJECT：您的專案 ID。
  • MODEL_ID：模型資源的 ID。
  • PROJECT_NUMBER：系統自動產生的專案編號。
  • EVALUATION_ID：模型評估 ID (會顯示在回應中)。

HTTP 方法和網址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations

如要傳送要求，請選擇以下其中一個選項：

curl

執行下列指令：

curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations"

PowerShell

執行下列指令：

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations" | Select-Object -Expand Content

您應該會收到如下的 JSON 回應：

Java

在試用這個範例之前，請先按照 Vertex AI 快速入門：使用用戶端程式庫中的操作說明設定 Java。詳情請參閱 Vertex AI Java API 參考說明文件。

如要向 Vertex AI 進行驗證，請設定應用程式預設憑證。詳情請參閱「為本機開發環境設定驗證機制」。


import com.google.cloud.aiplatform.v1.ModelEvaluation;
import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class GetModelEvaluationTabularClassificationSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    getModelEvaluationTabularClassification(project, modelId, evaluationId);
  }

  static void getModelEvaluationTabularClassification(
      String project, String modelId, String evaluationId) throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);
      ModelEvaluation modelEvaluation = modelServiceClient.getModelEvaluation(modelEvaluationName);

      System.out.println("Get Model Evaluation Tabular Classification Response");
      System.out.format("\tName: %s\n", modelEvaluation.getName());
      System.out.format("\tMetrics Schema Uri: %s\n", modelEvaluation.getMetricsSchemaUri());
      System.out.format("\tMetrics: %s\n", modelEvaluation.getMetrics());
      System.out.format("\tCreate Time: %s\n", modelEvaluation.getCreateTime());
      System.out.format("\tSlice Dimensions: %s\n", modelEvaluation.getSliceDimensionsList());
    }
  }
}

Node.js

在試用這個範例之前，請先按照 Vertex AI 快速入門：使用用戶端程式庫中的操作說明設定 Node.js。詳情請參閱 Vertex AI Node.js API 參考說明文件。

如要向 Vertex AI 進行驗證，請設定應用程式預設憑證。詳情請參閱「為本機開發環境設定驗證機制」。

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function getModelEvaluationTabularClassification() {
  // Configure the parent resources
  const name = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    name,
  };

  // Get model evaluation request
  const [response] = await modelServiceClient.getModelEvaluation(request);

  console.log('Get model evaluation tabular classification response');
  console.log(`\tName : ${response.name}`);
  console.log(`\tMetrics schema uri : ${response.metricsSchemaUri}`);
  console.log(`\tMetrics : ${JSON.stringify(response.metrics)}`);
  console.log(`\tCreate time : ${JSON.stringify(response.createTime)}`);
  console.log(`\tSlice dimensions : ${response.sliceDimensions}`);

  const modelExplanation = response.modelExplanation;
  console.log('\tModel explanation');
  if (!modelExplanation) {
    console.log('\t\t{}');
  } else {
    const meanAttributions = modelExplanation.meanAttributions;
    if (!meanAttributions) {
      console.log('\t\t\t []');
    } else {
      for (const meanAttribution of meanAttributions) {
        console.log('\t\tMean attribution');
        console.log(
          `\t\t\tBaseline output value : \
            ${meanAttribution.baselineOutputValue}`
        );
        console.log(
          `\t\t\tInstance output value : \
            ${meanAttribution.instanceOutputValue}`
        );
        console.log(
          `\t\t\tFeature attributions : \
            ${JSON.stringify(meanAttribution.featureAttributions)}`
        );
        console.log(`\t\t\tOutput index : ${meanAttribution.outputIndex}`);
        console.log(
          `\t\t\tOutput display name : \
            ${meanAttribution.outputDisplayName}`
        );
        console.log(
          `\t\t\tApproximation error : \
            ${meanAttribution.approximationError}`
        );
      }
    }
  }
}
getModelEvaluationTabularClassification();

Python 適用的 Vertex AI SDK

如要瞭解如何安裝或更新 Python 適用的 Vertex AI SDK，請參閱「安裝 Python 適用的 Vertex AI SDK」。 詳情請參閱 Vertex AI SDK for Python API 參考說明文件。

from google.cloud import aiplatform


def get_model_evaluation_tabular_classification_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    name = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.get_model_evaluation(name=name)
    print("response:", response)

迴歸

請選取對應於您語言或環境的分頁：

REST

使用任何要求資料之前，請先替換以下項目：

  • LOCATION：模型儲存的區域。
  • PROJECT：您的專案 ID。
  • MODEL_ID：模型資源的 ID。
  • PROJECT_NUMBER：系統自動產生的專案編號。
  • EVALUATION_ID：模型評估 ID (會顯示在回應中)。

HTTP 方法和網址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations

如要傳送要求，請選擇以下其中一個選項：

curl

執行下列指令：

curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations"

PowerShell

執行下列指令：

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations" | Select-Object -Expand Content

您應該會收到如下的 JSON 回應：

Java

在試用這個範例之前，請先按照 Vertex AI 快速入門：使用用戶端程式庫中的操作說明設定 Java。詳情請參閱 Vertex AI Java API 參考說明文件。

如要向 Vertex AI 進行驗證，請設定應用程式預設憑證。詳情請參閱「為本機開發環境設定驗證機制」。


import com.google.cloud.aiplatform.v1.ModelEvaluation;
import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class GetModelEvaluationTabularRegressionSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    getModelEvaluationTabularRegression(project, modelId, evaluationId);
  }

  static void getModelEvaluationTabularRegression(
      String project, String modelId, String evaluationId) throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);
      ModelEvaluation modelEvaluation = modelServiceClient.getModelEvaluation(modelEvaluationName);

      System.out.println("Get Model Evaluation Tabular Regression Response");
      System.out.format("\tName: %s\n", modelEvaluation.getName());
      System.out.format("\tMetrics Schema Uri: %s\n", modelEvaluation.getMetricsSchemaUri());
      System.out.format("\tMetrics: %s\n", modelEvaluation.getMetrics());
      System.out.format("\tCreate Time: %s\n", modelEvaluation.getCreateTime());
      System.out.format("\tSlice Dimensions: %s\n", modelEvaluation.getSliceDimensionsList());
    }
  }
}

Node.js

在試用這個範例之前，請先按照 Vertex AI 快速入門：使用用戶端程式庫中的操作說明設定 Node.js。詳情請參閱 Vertex AI Node.js API 參考說明文件。

如要向 Vertex AI 進行驗證，請設定應用程式預設憑證。詳情請參閱「為本機開發環境設定驗證機制」。

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function getModelEvaluationTabularRegression() {
  // Configure the parent resources
  const name = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    name,
  };

  // Get model evaluation request
  const [response] = await modelServiceClient.getModelEvaluation(request);

  console.log('Get model evaluation tabular regression response');
  console.log(`\tName : ${response.name}`);
  console.log(`\tMetrics schema uri : ${response.metricsSchemaUri}`);
  console.log(`\tMetrics : ${JSON.stringify(response.metrics)}`);
  console.log(`\tCreate time : ${JSON.stringify(response.createTime)}`);
  console.log(`\tSlice dimensions : ${response.sliceDimensions}`);

  const modelExplanation = response.modelExplanation;
  console.log('\tModel explanation');
  if (!modelExplanation) {
    console.log('\t\t{}');
  } else {
    const meanAttributions = modelExplanation.meanAttributions;
    if (!meanAttributions) {
      console.log('\t\t\t []');
    } else {
      for (const meanAttribution of meanAttributions) {
        console.log('\t\tMean attribution');
        console.log(
          `\t\t\tBaseline output value : \
            ${meanAttribution.baselineOutputValue}`
        );
        console.log(
          `\t\t\tInstance output value : \
            ${meanAttribution.instanceOutputValue}`
        );
        console.log(
          `\t\t\tFeature attributions : \
            ${JSON.stringify(meanAttribution.featureAttributions)}`
        );
        console.log(`\t\t\tOutput index : ${meanAttribution.outputIndex}`);
        console.log(
          `\t\t\tOutput display name : \
            ${meanAttribution.outputDisplayName}`
        );
        console.log(
          `\t\t\tApproximation error : \
            ${meanAttribution.approximationError}`
        );
      }
    }
  }
}
getModelEvaluationTabularRegression();

Python 適用的 Vertex AI SDK

如要瞭解如何安裝或更新 Python 適用的 Vertex AI SDK，請參閱「安裝 Python 適用的 Vertex AI SDK」。 詳情請參閱 Vertex AI SDK for Python API 參考說明文件。

from google.cloud import aiplatform


def get_model_evaluation_tabular_regression_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    name = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.get_model_evaluation(name=name)
    print("response:", response)

列出所有評估配量 (僅限分類模型)

projects.locations.models.evaluations.slices.list 方法會列出模型的所有評估配量。您必須擁有模型的評估 ID，您可以查看匯總的評估指標來取得。

您可以使用模型評估配量，判斷模型在特定標籤上的成效。value 欄位會指出指標的標籤。

Vertex AI 會傳回信心指標陣列。每個元素會顯示不同 confidenceThreshold 值的評估指標 (從 0 開始，最高為 1)。查看不同的門檻值，即可瞭解門檻對精確度和查全率等其他指標的影響。

REST

使用任何要求資料之前，請先替換以下項目：

  • LOCATION：模型所在的區域。例如：us-central1。
  • PROJECT：您的專案 ID。
  • MODEL_ID：模型的 ID。
  • EVALUATION_ID：包含要列出的評估區塊的模型評估 ID。

HTTP 方法和網址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices

如要傳送要求，請選擇以下其中一個選項：

curl

執行下列指令：

curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices"

PowerShell

執行下列指令：

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices" | Select-Object -Expand Content

您應該會收到如下的 JSON 回應：

Java

在試用這個範例之前，請先按照 Vertex AI 快速入門：使用用戶端程式庫中的操作說明設定 Java。詳情請參閱 Vertex AI Java API 參考說明文件。

如要向 Vertex AI 進行驗證，請設定應用程式預設憑證。詳情請參閱「為本機開發環境設定驗證機制」。


import com.google.cloud.aiplatform.v1.ModelEvaluationName;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice.Slice;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class ListModelEvaluationSliceSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    listModelEvaluationSliceSample(project, modelId, evaluationId);
  }

  static void listModelEvaluationSliceSample(String project, String modelId, String evaluationId)
      throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationName modelEvaluationName =
          ModelEvaluationName.of(project, location, modelId, evaluationId);

      for (ModelEvaluationSlice modelEvaluationSlice :
          modelServiceClient.listModelEvaluationSlices(modelEvaluationName).iterateAll()) {
        System.out.format("Model Evaluation Slice Name: %s\n", modelEvaluationSlice.getName());
        System.out.format("Metrics Schema Uri: %s\n", modelEvaluationSlice.getMetricsSchemaUri());
        System.out.format("Metrics: %s\n", modelEvaluationSlice.getMetrics());
        System.out.format("Create Time: %s\n", modelEvaluationSlice.getCreateTime());

        Slice slice = modelEvaluationSlice.getSlice();
        System.out.format("Slice Dimensions: %s\n", slice.getDimension());
        System.out.format("Slice Value: %s\n\n", slice.getValue());
      }
    }
  }
}

Node.js

在試用這個範例之前，請先按照 Vertex AI 快速入門：使用用戶端程式庫中的操作說明設定 Node.js。詳情請參閱 Vertex AI Node.js API 參考說明文件。

如要向 Vertex AI 進行驗證，請設定應用程式預設憑證。詳情請參閱「為本機開發環境設定驗證機制」。

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service Client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');

// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};

// Instantiates a client
const modelServiceClient = new ModelServiceClient(clientOptions);

async function listModelEvaluationSlices() {
  // Configure the parent resources
  const parent = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}`;
  const request = {
    parent,
  };

  // Get and print out a list of all the evaluation slices for this resource
  const [response] =
    await modelServiceClient.listModelEvaluationSlices(request);
  console.log('List model evaluation response', response);
  console.log(response);
}
listModelEvaluationSlices();

Python 適用的 Vertex AI SDK

如要瞭解如何安裝或更新 Python 適用的 Vertex AI SDK，請參閱「安裝 Python 適用的 Vertex AI SDK」。 詳情請參閱 Vertex AI SDK for Python API 參考說明文件。

from google.cloud import aiplatform


def list_model_evaluation_slices_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    parent = client.model_evaluation_path(
        project=project, location=location, model=model_id, evaluation=evaluation_id
    )
    response = client.list_model_evaluation_slices(parent=parent)
    for model_evaluation_slice in response:
        print("model_evaluation_slice:", model_evaluation_slice)

取得單一切片的指標

如要查看單一切片的評估指標，請使用 projects.locations.models.evaluations.slices.get 方法。您必須擁有切片 ID，這會在列出所有切片時提供。以下範例適用於所有資料類型和目標。

REST

使用任何要求資料之前，請先替換以下項目：

  • LOCATION：模型所在的區域。例如 us-central1。
  • PROJECT：您的專案 ID。
  • MODEL_ID：模型的 ID。
  • EVALUATION_ID：包含要擷取的評估區塊的模型評估 ID。
  • SLICE_ID：要取得的評估區塊 ID。
  • PROJECT_NUMBER：系統自動產生的專案編號。
  • EVALUATION_METRIC_SCHEMA_FILE_NAME：結構定義檔案的名稱，用於定義要傳回的評估指標，例如 classification_metrics_1.0.0。

HTTP 方法和網址：

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices/SLICE_ID

如要傳送要求，請選擇以下其中一個選項：

curl

執行下列指令：

curl -X GET \
     -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices/SLICE_ID"

PowerShell

執行下列指令：

$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
    -Method GET `
    -Headers $headers `
    -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/models/MODEL_ID/evaluations/EVALUATION_ID/slices/SLICE_ID" | Select-Object -Expand Content

您應該會收到如下的 JSON 回應：

Java

在試用這個範例之前，請先按照 Vertex AI 快速入門：使用用戶端程式庫中的操作說明設定 Java。詳情請參閱 Vertex AI Java API 參考說明文件。

如要向 Vertex AI 進行驗證，請設定應用程式預設憑證。詳情請參閱「為本機開發環境設定驗證機制」。


import com.google.cloud.aiplatform.v1.ModelEvaluationSlice;
import com.google.cloud.aiplatform.v1.ModelEvaluationSlice.Slice;
import com.google.cloud.aiplatform.v1.ModelEvaluationSliceName;
import com.google.cloud.aiplatform.v1.ModelServiceClient;
import com.google.cloud.aiplatform.v1.ModelServiceSettings;
import java.io.IOException;

public class GetModelEvaluationSliceSample {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // To obtain evaluationId run the code block below after setting modelServiceSettings.
    //
    // try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings))
    // {
    //   String location = "us-central1";
    //   ModelName modelFullId = ModelName.of(project, location, modelId);
    //   ListModelEvaluationsRequest modelEvaluationsrequest =
    //   ListModelEvaluationsRequest.newBuilder().setParent(modelFullId.toString()).build();
    //   for (ModelEvaluation modelEvaluation :
    //     modelServiceClient.listModelEvaluations(modelEvaluationsrequest).iterateAll()) {
    //       System.out.format("Model Evaluation Name: %s%n", modelEvaluation.getName());
    //   }
    // }
    String project = "YOUR_PROJECT_ID";
    String modelId = "YOUR_MODEL_ID";
    String evaluationId = "YOUR_EVALUATION_ID";
    String sliceId = "YOUR_SLICE_ID";
    getModelEvaluationSliceSample(project, modelId, evaluationId, sliceId);
  }

  static void getModelEvaluationSliceSample(
      String project, String modelId, String evaluationId, String sliceId) throws IOException {
    ModelServiceSettings modelServiceSettings =
        ModelServiceSettings.newBuilder()
            .setEndpoint("us-central1-aiplatform.googleapis.com:443")
            .build();

    // Initialize client that will be used to send requests. This client only needs to be created
    // once, and can be reused for multiple requests. After completing all of your requests, call
    // the "close" method on the client to safely clean up any remaining background resources.
    try (ModelServiceClient modelServiceClient = ModelServiceClient.create(modelServiceSettings)) {
      String location = "us-central1";
      ModelEvaluationSliceName modelEvaluationSliceName =
          ModelEvaluationSliceName.of(project, location, modelId, evaluationId, sliceId);

      ModelEvaluationSlice modelEvaluationSlice =
          modelServiceClient.getModelEvaluationSlice(modelEvaluationSliceName);

      System.out.println("Get Model Evaluation Slice Response");
      System.out.format("Model Evaluation Slice Name: %s\n", modelEvaluationSlice.getName());
      System.out.format("Metrics Schema Uri: %s\n", modelEvaluationSlice.getMetricsSchemaUri());
      System.out.format("Metrics: %s\n", modelEvaluationSlice.getMetrics());
      System.out.format("Create Time: %s\n", modelEvaluationSlice.getCreateTime());

      Slice slice = modelEvaluationSlice.getSlice();
      System.out.format("Slice Dimensions: %s\n", slice.getDimension());
      System.out.format("Slice Value: %s\n", slice.getValue());
    }
  }
}

Node.js

在試用這個範例之前，請先按照 Vertex AI 快速入門：使用用戶端程式庫中的操作說明設定 Node.js。詳情請參閱 Vertex AI Node.js API 參考說明文件。

如要向 Vertex AI 進行驗證，請設定應用程式預設憑證。詳情請參閱「為本機開發環境設定驗證機制」。

/**
 * TODO(developer): Uncomment these variables before running the sample
 * (not necessary if passing values as arguments). To obtain evaluationId,
 * instantiate the client and run the following the commands.
 */
// const parentName = `projects/${project}/locations/${location}/models/${modelId}`;
// const evalRequest = {
//   parent: parentName
// };
// const [evalResponse] = await modelServiceClient.listModelEvaluations(evalRequest);
// console.log(evalResponse);

// const modelId = 'YOUR_MODEL_ID';
// const evaluationId = 'YOUR_EVALUATION_ID';
// const sliceId = 'YOUR_SLICE_ID';
// const project = 'YOUR_PROJECT_ID';
// const location = 'YOUR_PROJECT_LOCATION';

// Imports the Google Cloud Model Service client library
const {ModelServiceClient} = require('@google-cloud/aiplatform');
// Specifies the location of the api endpoint
const clientOptions = {
  apiEndpoint: 'us-central1-aiplatform.googleapis.com',
};
// Specifies the location of the api endpoint
const modelServiceClient = new ModelServiceClient(clientOptions);

async function getModelEvaluationSlice() {
  // Configure the parent resource
  const name = `projects/${project}/locations/${location}/models/${modelId}/evaluations/${evaluationId}/slices/${sliceId}`;
  const request = {
    name,
  };

  // Get and print out a list of all the endpoints for this resource
  const [response] =
    await modelServiceClient.getModelEvaluationSlice(request);

  console.log('Get model evaluation slice');
  console.log(`\tName : ${response.name}`);
  console.log(`\tMetrics_Schema_Uri : ${response.metricsSchemaUri}`);
  console.log(`\tMetrics : ${JSON.stringify(response.metrics)}`);
  console.log(`\tCreate time : ${JSON.stringify(response.createTime)}`);

  console.log('Slice');
  const slice = response.slice;
  console.log(`\tDimension :${slice.dimension}`);
  console.log(`\tValue :${slice.value}`);
}
getModelEvaluationSlice();

Python 適用的 Vertex AI SDK

如要瞭解如何安裝或更新 Python 適用的 Vertex AI SDK，請參閱「安裝 Python 適用的 Vertex AI SDK」。 詳情請參閱 Vertex AI SDK for Python API 參考說明文件。

from google.cloud import aiplatform


def get_model_evaluation_slice_sample(
    project: str,
    model_id: str,
    evaluation_id: str,
    slice_id: str,
    location: str = "us-central1",
    api_endpoint: str = "us-central1-aiplatform.googleapis.com",
):
    """
    To obtain evaluation_id run the following commands where LOCATION
    is the region where the model is stored, PROJECT is the project ID,
    and MODEL_ID is the ID of your model.

    model_client = aiplatform.gapic.ModelServiceClient(
        client_options={
            'api_endpoint':'LOCATION-aiplatform.googleapis.com'
            }
        )
    evaluations = model_client.list_model_evaluations(parent='projects/PROJECT/locations/LOCATION/models/MODEL_ID')
    print("evaluations:", evaluations)
    """
    # The AI Platform services require regional API endpoints.
    client_options = {"api_endpoint": api_endpoint}
    # Initialize client that will be used to create and send requests.
    # This client only needs to be created once, and can be reused for multiple requests.
    client = aiplatform.gapic.ModelServiceClient(client_options=client_options)
    name = client.model_evaluation_slice_path(
        project=project,
        location=location,
        model=model_id,
        evaluation=evaluation_id,
        slice=slice_id,
    )
    response = client.get_model_evaluation_slice(name=name)
    print("response:", response)

模型評估指標

Vertex AI 會傳回多項不同的評估指標，例如精確度、召回率和信心閾值。Vertex AI 傳回的指標取決於模型的目標。舉例來說，Vertex AI 會為圖像分類模型提供不同的評估指標，而非圖像物件偵測模型。

模式檔案會決定 Vertex AI 為每個目標提供哪些評估指標。

您可以查看及下載下列 Cloud Storage 位置中的結構定義檔案：
gs://google-cloud-aiplatform/schema/modelevaluation/

評估指標如下：

分類

  • AuPRC：精確度和喚回度 (PR) 曲線下的面積，也稱為平均精確度。這個值的範圍從零到一，值越大代表模型品質越高。
  • AuROC：接收者操作特徵曲線下的面積。範圍從零到一，值越大代表模型品質越高。
  • 對數損失：模型預測與目標值之間的交叉熵。範圍從零到無限大，值越低代表模型品質越高。
  • 可信度門檻：可信度分數，決定要傳回哪些預測結果。模型會傳回這個值或更高的預測值。可信度門檻越高，精確度就會提高，但喚回率會降低。Vertex AI 會以不同的閾值值傳回可信度指標，顯示閾值對精確度和召回率的影響。
  • 喚回率：模型正確預測出含有此類別的預測值的比例。也稱為「真陽率」。
  • Recall at 1：僅考量預測分數最高且不低於每個範例可信度門檻的標籤時的回溯率 (真陽率)。
  • 精確度：模型產生的正確分類預測比例。
  • 精確度為 1：只考量預測分數最高且不低於每個範例可信度門檻的標籤的精確度。
  • F1 分數：精確度與喚回度的調和平均數。如果您要在精確度與喚回度之間找出平衡，且類別分布並不平均，F1 這個指標很有用。
  • F1 分數為 1：喚回率為 1 和精確度為 1 的調和平均數。
  • 真陰性計數：模型正確預測負類的次數。
  • 真陽性計數：模型正確預測正類的次數。
  • 偽陰性計數：模型誤判負類的次數。
  • 偽陽性計數：模型錯誤預測正類的次數。
  • 偽陽率：在所有預測結果中，錯誤預測結果的比例。
  • 偽陽率為 1：僅考量預測分數最高且不低於每個範例可信度門檻的標籤時，偽陽率為 1。
  • 混淆矩陣：混淆矩陣會顯示模型正確預測結果的頻率。如果是預測結果錯誤的情況，矩陣會顯示模型預測的結果。混淆矩陣可協助您瞭解模型在哪些地方「混淆」兩個結果。
  • 模型特徵歸因：Vertex AI 會顯示各項特徵對模型的影響程度。每項特徵的值會以百分比表示：百分比越高，代表該特徵對模型訓練的影響越大。請詳閱這項資訊，確保所有最重要的功能都適用於您的資料和業務問題。如需更多資訊，請參閱「分類和迴歸的功能歸因」。

迴歸

  • MAE：平均絕對誤差 (MAE) 是目標值與預測值之間的平均絕對差異。這個指標的範圍從零到無限大，值越低代表模型品質越好。
  • RMSE：均方根誤差是目標值與預測值之間的均方差平方根。RMSE 對離群值比 MAE 更敏感，所以如果您擔心大型誤差，RMSE 可作為更實用的評估指標。和 MAE 一樣，值越小代表模型品質越高 (0 代表完美預測因子)。
  • RMSLE：均方根對數誤差指標與 RMSE 相似，但會使用預測值和實際值的對數加上 1。相較於均方根誤差，均方根對數誤差對低估的懲罰較重。如果您不希望系統對大預測值的差異處以比小預測值更嚴厲的懲罰，這也是不錯的指標。這個指標的範圍從零到無限大，值越低代表模型品質越高。只有在所有標記和預測值皆為非負值時，系統才會傳回 RMSLE 評估指標。
  • r^2：r 平方 (r^2) 是標籤與預測值之間的皮爾森相關係數平方，這項指標的範圍介於 0 至 1 之間。值越高，表示與回歸線的擬合度越高。
  • MAPE：平均絕對百分比誤差 (MAPE) 是標籤和預測值之間的平均絕對百分比差異。這個指標的範圍介於零到無限大之間，值越小代表模型品質越好。
    如果目標資料欄含有任何 0 值，系統就不會顯示 MAPE。在這種情況下，MAPE 未定義。
  • 模型特徵歸因：Vertex AI 會顯示各項特徵對模型的影響程度。每項特徵的值會以百分比提供：百分比越高，代表該特徵對模型訓練的影響越大。請詳閱這項資訊，確保所有最重要的特徵對您的資料和業務問題具有合理意義。如需更多資訊，請參閱「分類和迴歸的功能歸因」。

後續步驟

當您準備好使用分類或迴歸模型進行預測時，有兩種方法可供選擇：

此外，您還可以：