Method: projects.locations.evaluateDataset

Evaluates a dataset based on a set of given metrics.

Endpoint

post https://{service-endpoint}/v1beta1/{location}:evaluateDataset

Where {service-endpoint} is one of the supported service endpoints.

Path parameters

location string

Required. The resource name of the location to evaluate the dataset. Format: projects/{project}/locations/{location}

Request body

The request body contains data with the following structure:

Fields
dataset object (EvaluationDataset)

Required. The dataset used for evaluation.

metrics[] object (Metric)

Required. The metrics used for evaluation.

outputConfig object (OutputConfig)

Required. Config for evaluation output.

autoraterConfig object (AutoraterConfig)

Optional. Autorater config used for evaluation. Currently only publisher Gemini models are supported. Format: projects/{PROJECT}/locations/{LOCATION}/publishers/google/models/{MODEL}.

Response body

If successful, the response body contains an instance of Operation.

EvaluationDataset

The dataset used for evaluation.

Fields
source Union type
The source of the dataset. source can be only one of the following:
gcsSource object (GcsSource)

Cloud storage source holds the dataset. Currently only one Cloud Storage file path is supported.

bigquerySource object (BigQuerySource)

BigQuery source holds the dataset.

JSON representation 
{

  // source
  "gcsSource": {
    object (GcsSource)
  },
  "bigquerySource": {
    object (BigQuerySource)
  }
  // Union type
}

Metric

The metric used for dataset level evaluation.

Fields
aggregationMetrics[] enum (AggregationMetric)

Optional. The aggregation metrics to use.

metric_spec Union type
The metric spec used for evaluation. metric_spec can be only one of the following:
pointwiseMetricSpec object (PointwiseMetricSpec)

Spec for pointwise metric.

pairwiseMetricSpec object (PairwiseMetricSpec)

Spec for pairwise metric.

exactMatchSpec object (ExactMatchSpec)

Spec for exact match metric.

bleuSpec object (BleuSpec)

Spec for bleu metric.

rougeSpec object (RougeSpec)

Spec for rouge metric.

JSON representation 
{
  "aggregationMetrics": [
    enum (AggregationMetric)
  ],

  // metric_spec
  "pointwiseMetricSpec": {
    object (PointwiseMetricSpec)
  },
  "pairwiseMetricSpec": {
    object (PairwiseMetricSpec)
  },
  "exactMatchSpec": {
    object (ExactMatchSpec)
  },
  "bleuSpec": {
    object (BleuSpec)
  },
  "rougeSpec": {
    object (RougeSpec)
  }
  // Union type
}

AggregationMetric

The aggregation metrics supported by EvaluationService.EvaluateDataset.

Enums
AGGREGATION_METRIC_UNSPECIFIED Unspecified aggregation metric.
AVERAGE Average aggregation metric.
MODE Mode aggregation metric.
STANDARD_DEVIATION Standard deviation aggregation metric.
VARIANCE Variance aggregation metric.
MINIMUM Minimum aggregation metric.
MAXIMUM Maximum aggregation metric.
MEDIAN Median aggregation metric.
PERCENTILE_P90 90th percentile aggregation metric.
PERCENTILE_P95 95th percentile aggregation metric.
PERCENTILE_P99 99th percentile aggregation metric.

OutputConfig

Config for evaluation output.

Fields
destination Union type
The destination for evaluation output. destination can be only one of the following:
gcsDestination object (GcsDestination)

Cloud storage destination for evaluation output.

JSON representation 
{

  // destination
  "gcsDestination": {
    object (GcsDestination)
  }
  // Union type
}