Model evaluation components

Model evaluation components take ground truth and batch prediction results as input and generate evaluation metrics. Ground truth data is "correctly labeled" data as determined by humans, and is usually provided as the target field in the test dataset you used to train the model. For more information, see Model evaluation in Vertex AI.

Vertex AI provides the following model evaluation components:

• ModelEvaluationClassificationOp
• ModelEvaluationForecastingOp
• ModelEvaluationRegressionOp

Model type support

The following table shows supported model types for each model evaluation component:

Remove the target field

For some model types, the BatchPredictionJob component requires you to exclude the target column (ground truth) from your dataset.

Configure and format slices for your tabular dataset

A slice is a subset of tabular data with specified characteristics. Sliced metrics offers more granular evaluation metrics for AutoML tabular and custom training classification models.

For example, consider a pet adoption dataset where a model is trained to predict whether a pet will be adopted within a week. While it's useful to look at metrics for the whole dataset, you may be interested in metrics related to the pet's species and age. In other words, you're interested in the following subset of the dataset:

This fine-grained analysis can be helpful in identifying where the model might be most error prone, or ensuring that the model performs well on critical subsets of the data.

Configure slices

To configure a slice, create a JSON file with the following configuration:

{
  "FEATURE": {
    "KIND": {
      CONFIG
    }
  }
}

Where:

• FEATURE is a feature for which you want to create a slice. For example, age.

• KIND is one of the following:

  • value

    value creates a single slice on a single value of a feature. Specify CONFIG as a key-value pair where the key is float_value or string_value. For example, "float_value": 1.0.

    The following example configuration creates a single slice containing data with the age of 1:

    {
      "age": {
        "value": {
          "float_value": 1.0
        }
      }
    }

  • range

    range creates a single slice that includes data within a specified range. Specify CONFIG as two key-value pairs where the keys are low and high.

    The following example configuration creates a single slice containing data where the age is any value between 1 and 3:

    {
      "age": {
        "range": {
          "low": 1,
          "high": 3
        }
      }
    }

  • all_values

    all_values creates a separate slice for each possible label for the feature. Specify CONFIG as "value": true.

    The following example configuration creates three slices—one each for "dog", "cat", and "rabbit":

    {
      "species": {
        "all_values": {
          "value": true
        }
      }
    }

You can also use multiple features to create one or more slices.

The following configuration creates a single slice containing data where the age is between 1 and 3 and the species is "dog":

{
  "age": {
    "range": {
      "low": 1,
      "high": 3
    }
  },
  "species": {
    "value": {
      "string_value": "dog"
    }
  }
}

The following configuration creates multiple slices—one for each unique species in the dataset, where the age is 1.

{
  "species": {
    "all_values": {
      "value": true
    }
  },
  "age": {
    "value": {
      "float_value": 1.0
    }
  }
}

The resulting slices contain the following data from the example dataset:

• Slice 1: age:1 and species:"dog"

• Slice 2: age:1 and species:"cat"

• Slice 3: age:1 and species:"rabbit"

Format slices

To format slices for the ModelEvaluationClassificationOp component, do the following:

1. Create a slicing_spec. For example:

   from google.cloud.aiplatform_v1.types.ModelEvaluationSlice.Slice import SliceSpec
   from google.cloud.aiplatform_v1.types.ModelEvaluationSlice.Slice.SliceSpec import SliceConfig
   
   slicing_spec = SliceSpec(configs={ 'feature_a': SliceConfig(SliceSpec.Value(string_value='label_a') ) })

2. Create a list to store the slice configurations. For example:

   slicing_specs = []

3. Format each slicing_spec as JSON or a dictionary. For example:

   slicing_spec_json = json_format.MessageToJson(slicing_spec)
   
   slicing_spec_dict = json_format.MessageToDict(slicing_spec)

4. Combine each slicing_spec into a list. For example:

   slicing_specs.append(slicing_spec_json)

5. Pass slicing_specs as a parameter to the ModelEvaluationClassificationOp component. For example:

   ModelEvaluationClassificationOp(slicing_specs=slicing_specs)

Format fairness metrics configuration slices

To evaluate your model for fairness, use the following components:

• DetectDataBiasOp: returns data bias metrics for the slices you pass into the component.

• DetectModelBiasOp: returns model bias metrics for the slices you pass into the component.

To format your slices for the DetectDataBiasOp and DetectModelBiasOp components, do the following:

1. Create a BiasConfig. For example:

   from google.cloud.aiplatform_v1.types.ModelEvaluation import BiasConfig
   from google.cloud.aiplatform_v1.types.ModelEvaluationSlice.Slice import SliceSpec
   from google.cloud.aiplatform_v1.types.ModelEvaluationSlice.Slice.SliceSpec import SliceConfig
   
   bias_config = BiasConfig(bias_slices=SliceSpec(configs={ 'feature_a': SliceConfig(SliceSpec.Value(string_value='label_a') ) }))

2. Create a list to store the bias configurations. For example:

   bias_configs = []

3. Format each bias_config as JSON or a dictionary. For example:

   bias_config_json = json_format.MessageToJson(bias_config)
   
   bias_config_dict = json_format.MessageToDict(bias_config)

4. Combine each bias_config into a list. For example:

   bias_configs.append(bias_config_json)

5. Pass bias_configs as a parameter to the DetectDataBiasOp or DetectModelBiasOp component. For example:

   DetectDataBiasOp(bias_configs=bias_configs)
   
   DetectModelBiasOp(bias_configs=bias_configs)

Required input parameters

The required input parameters for model evaluation components vary based on the type of model you are evaluating.

AutoML models

When training AutoML models, Vertex AI uses default schemas. Some input parameters for the model evaluation components are only required if you used a non-default schema to train your model.

You can view and download schema files from the following Cloud Storage location:
gs://google-cloud-aiplatform/schema/modelevaluation/

The following tabs show example batch prediction output files and required input parameters for each model type:

{
  "instance": {
    "male": "1",
    "age": "49",
    "heartRate": "67",
    "education": "medium_high",
    "totChol": "282",
    "diaBP": "79",
    "BPMeds": "0",
    "currentSmoker": "current_high",
    "cigsPerDay": "25",
    "prevalentStroke": "stroke_0",
    "prevalentHyp": "hyp_0",
    "TenYearCHD": "FALSE",
    "diabetes": "0",
    "sysBP": "134",
    "BMI": "26.87",
    "glucose": "78"
    },
  "prediction": {
      "scores": [0.3666940927505493, 0.6333059072494507],
      "classes": ["0", "1"]
    }
}
          

{
  "instance": {
    "BPMeds": "0",
    "diabetes": "0",
    "diaBP": "65",
    "cigsPerDay": "45",
    "prevalentHyp": "hyp_0",
    "age": "46",
    "male": "1",
    "sysBP": "112.5",
    "glucose": "78",
    "BMI": "27.48",
    "TenYearCHD": "FALSE",
    "totChol": "209",
    "education": "high",
    "prevalentStroke": "stroke_0",
    "heartRate": "75",
    "currentSmoker": "current_high"
  },
  "prediction": {
    "value": 44.96103286743164,
    "lower_bound": 44.61349868774414,
    "upper_bound": 44.590206146240234
  }
}
          

date,store_name,city,zip_code,county,sale_dollars
2020-03-17,Thriftway,,,,774.08999999999992
2020-03-10,Thriftway,,,,1160.67
2020-03-03,Thriftway,,,,2247.24
2020-06-08,New Star / Fort Dodge,,,,753.98
2020-06-01,New Star / Fort Dodge,,,,967.73
2020-01-10,Casey's General Store #1280 / Fort Dodge,,,,1040.92
2020-10-30,KUM & GO #76 / ADAIR,Adair,50002,ADAIR,1387.02
          

date,store_name,city,zip_code,county
2020-03-17,Thriftway,nan,nan,nan
2020-03-10,Thriftway,nan,nan,nan
2020-03-03,Thriftway,nan,nan,nan
2020-06-08,New Star / Fort Dodge,nan,nan,nan
2020-06-01,New Star / Fort Dodge,nan,nan,nan
2020-01-10,Casey's General Store #1280 / Fort Dodge,nan,nan,nan
2020-10-30,KUM & GO #76 / ADAIR,Adair,50002.0,ADAIR
          

{
  "instance": {
    "content": "gs://kbn-us-central1-test/datasets/text/happy_11556.txt",
    "mimeType":"text/plain"
  },
  "prediction": {
    "ids":    ["7903916851837534208","3292230833410146304","986387824196452352","2139309328803299328","5598073842623840256","6750995347230687232","4559431178561519616"],
    "displayNames": ["affection","bonding","achievement","exercise","nature","enjoy_the_moment","leisure"],
    "confidences": [0.99483216,0.005162797,4.1117933E-6,3.9997E-7,2.4624453E-7,1.9969502E-7,1.16997434E-7]
  }
}
          

{
  "content": "gs://kbn-us-central1-test/datasets/text/happy_0.txt",
  "mimeType": "text/plain",
  "label": "affection"
}
          

{
  "content": "gs://kbn-us-central1-test/datasets/text/happy_0.txt",
  "mimeType": "text/plain"
}
          

{
  "instance": {
    "content": "gs://kbn-us-central1-test/datasets/text/happy_11556.txt",
    "mimeType":"text/plain"
  },
  "prediction": {
    "ids":    ["7903916851837534208","3292230833410146304","986387824196452352","2139309328803299328","5598073842623840256","6750995347230687232","4559431178561519616"],
    "displayNames": ["affection","bonding","achievement","exercise","nature","enjoy_the_moment","leisure"],
    "confidences": [0.99483216,0.005162797,4.1117933E-6,3.9997E-7,2.4624453E-7,1.9969502E-7,1.16997434E-7]
  }
}
          

{
  "content": "gs://automl-video-demo-data/hmdb51/Aerial_Cartwheel_Tutorial_By_Jujimufu_cartwheel_f_nm_np1_ri_med_0.avi",
  "ground_truth": "cartwheel",
  "timeSegmentStart": "0.0",
  "timeSegmentEnd": "inf"
}
          

{
  "content": "gs://automl-video-demo-data/hmdb51/Aerial_Cartwheel_Tutorial_By_Jujimufu_cartwheel_f_nm_np1_ri_med_0.avi",
  "timeSegmentStart": "0.0",
  "timeSegmentEnd": "inf"
}
          

{
  "instance": {
    "content": "gs://automl-video-demo-data/hmdb51/20_Marine_Corps_Pull_Ups___JimmyDShea_pullup_f_cm_np1_ba_bad_3.avi"
    "mimeType": "video/avi",
    "timeSegmentStart": "0.0",
    "timeSegmentEnd": "inf"
  },
  "prediction": [
    {
      "id":"5015420640832782336",
      "displayName":"pullup",
      "type":"segment-classification",
      "timeSegmentStart":"0s",
      "timeSegmentEnd":"2.600s",
      "confidence":0.98152995
    },
    {
      "id":"2709577631619088384",
      "displayName":"cartwheel",
      "type":"segment-classification",
      "timeSegmentStart":"0s",
      "timeSegmentEnd":"2.600s",
      "confidence":0.0047166348
    },
    {
      "id":"403734622405394432",
      "displayName":"golf",
      "type":"segment-classification",
      "timeSegmentStart":"0s",
      "timeSegmentEnd":"2.600s",
      "confidence":0.0046260506
    },
    {
      "id":"1556656127012241408",
      "displayName":"ride_horse",
      "type":"segment-classification",
      "timeSegmentStart":"0s",
      "timeSegmentEnd":"2.600s",
      "confidence":0.004590442
    },
    {
      "id":"7321263650046476288",
      "displayName":"kick_ball",
      "type":"segment-classification",
      "timeSegmentStart":"0s",
      "timeSegmentEnd":"2.600s",
      "confidence":0.004536863
    }
  ]
}
          

Custom-trained models

For custom-trained models, the BatchPredictionJob component requires you to exclude the target column (ground truth) from your dataset. You can use the TargetFieldDataRemoverOp component for a more efficient workflow.

The following example shows a batch prediction output file and required input parameters for a custom-trained tabular classification model:

petal_length,petal_width,sepal_length,sepal_width,species
6.4,2.8,5.6,2.2,2
5.0,2.3,3.3,1.0,1
4.9,2.5,4.5,1.7,2
4.9,3.1,1.5,0.1,0
          

petal_length,petal_width,sepal_length,sepal_width
6.4,2.8,5.6,2.2
5.0,2.3,3.3,1.0
4.9,2.5,4.5,1.7
4.9,3.1,1.5,0.1
          

{
  "instance": [5.6, 2.5, 3.9, 1.1],
  "prediction": [0.0005816521588712931, 0.9989032745361328, 0.0005150380893610418]
}
{
  "instance": [5.0, 3.6, 1.4, 0.2],
  "prediction": [0.999255359172821, 0.000527293945197016, 0.00021737271163146943]
}
{
  "instance": [6.0, 2.9, 4.5, 1.5],
  "prediction": [0.00025063654175028205, 0.9994204044342041, 0.00032893591560423374]
}
          

Model evaluations without Vertex AI-generated batch predictions

You can use a model evaluation pipeline component with a batch prediction that you didn't generate in Vertex AI. However, note that the input for the model evaluation pipeline component must be a batch prediction directory containing files that match the following prefixes:

• prediction.results-

• explanation.results-

• predictions_

API reference

• For component reference, see the Google Cloud Pipeline Components SDK reference page for Model evaluation components.

• For examples of how to use the Model Evaluation components, see the notebooks listed at Create an evaluation.

Version history and release notes

To learn more about the version history and changes to the Google Cloud Pipeline Components SDK, see the Google Cloud Pipeline Components SDK Release Notes.