Method: locations.evaluateDataset

Full name: projects.locations.evaluateDataset

Evaluates a dataset based on a set of given metrics.

Endpoint

post https://aiplatform.googleapis.com/v1/{location}:evaluateDataset

Path parameters

location string

Required. The resource name of the Location to evaluate the dataset. Format: projects/{project}/locations/{location}

Request body

The request body contains data with the following structure:

Fields
dataset object (EvaluationDataset)

Required. The dataset used for evaluation.

metrics[] object (Metric)

Required. The metrics used for evaluation.

outputConfig object (OutputConfig)

Required. Config for evaluation output.

autoraterConfig object (AutoraterConfig)

Optional. Autorater config used for evaluation. Currently only publisher Gemini models are supported. Format: projects/{PROJECT}/locations/{LOCATION}/publishers/google/models/{MODEL}.

Response body

If successful, the response body contains an instance of Operation.

EvaluationDataset

The dataset used for evaluation.

Fields
source Union type
The source of the dataset. source can be only one of the following:
gcsSource object (GcsSource)

Cloud storage source holds the dataset. Currently only one Cloud Storage file path is supported.

bigquerySource object (BigQuerySource)

BigQuery source holds the dataset.

JSON representation 
{

  // source
  "gcsSource": {
    object (GcsSource)
  },
  "bigquerySource": {
    object (BigQuerySource)
  }
  // Union type
}

BigQuerySource

The BigQuery location for the input content.

Fields
inputUri string

Required. BigQuery URI to a table, up to 2000 characters long. Accepted forms:

  • BigQuery path. For example: bq://projectId.bqDatasetId.bqTableId.
JSON representation 
{
  "inputUri": string
}

Metric

The metric used for running evaluations.

Fields
aggregationMetrics[] enum (AggregationMetric)

Optional. The aggregation metrics to use.

metric_spec Union type
The spec for the metric. It would be either a pre-defined metric, or a inline metric spec. metric_spec can be only one of the following:
pointwiseMetricSpec object (PointwiseMetricSpec)

Spec for pointwise metric.

pairwiseMetricSpec object (PairwiseMetricSpec)

Spec for pairwise metric.

exactMatchSpec object (ExactMatchSpec)

Spec for exact match metric.

bleuSpec object (BleuSpec)

Spec for bleu metric.

rougeSpec object (RougeSpec)

Spec for rouge metric.

JSON representation 
{
  "aggregationMetrics": [
    enum (AggregationMetric)
  ],

  // metric_spec
  "pointwiseMetricSpec": {
    object (PointwiseMetricSpec)
  },
  "pairwiseMetricSpec": {
    object (PairwiseMetricSpec)
  },
  "exactMatchSpec": {
    object (ExactMatchSpec)
  },
  "bleuSpec": {
    object (BleuSpec)
  },
  "rougeSpec": {
    object (RougeSpec)
  }
  // Union type
}

AggregationMetric

The aggregation metrics supported by EvaluationService.EvaluateDataset.

Enums
AGGREGATION_METRIC_UNSPECIFIED Unspecified aggregation metric.
AVERAGE Average aggregation metric. Not supported for Pairwise metric.
MODE Mode aggregation metric.
STANDARD_DEVIATION Standard deviation aggregation metric. Not supported for pairwise metric.
VARIANCE Variance aggregation metric. Not supported for pairwise metric.
MINIMUM Minimum aggregation metric. Not supported for pairwise metric.
MAXIMUM Maximum aggregation metric. Not supported for pairwise metric.
MEDIAN Median aggregation metric. Not supported for pairwise metric.
PERCENTILE_P90 90th percentile aggregation metric. Not supported for pairwise metric.
PERCENTILE_P95 95th percentile aggregation metric. Not supported for pairwise metric.
PERCENTILE_P99 99th percentile aggregation metric. Not supported for pairwise metric.

OutputConfig

Config for evaluation output.

Fields
destination Union type
The destination for evaluation output. destination can be only one of the following:
gcsDestination object (GcsDestination)

Cloud storage destination for evaluation output.

JSON representation 
{

  // destination
  "gcsDestination": {
    object (GcsDestination)
  }
  // Union type
}