- Resource: QaQuestion
- QaQuestionType
- PredefinedQuestionConfig
- PredefinedQuestionType
- AnswerChoice
- Metrics
- TuningMetadata
- DatasetValidationWarning
- Methods
Resource: QaQuestion
A single question to be scored by the Insights QA feature.
JSON representation |
---|
{ "name": string, "abbreviation": string, "createTime": string, "updateTime": string, "questionType": enum ( |
Fields | |
---|---|
name |
Identifier. The resource name of the question. Format: projects/{project}/locations/{location}/qaScorecards/{qaScorecard}/revisions/{revision}/qaQuestions/{qaQuestion} |
abbreviation |
Short, descriptive string, used in the UI where it's not practical to display the full question body. E.g., "Greeting". |
createTime |
Output only. The time at which this question was created. Uses RFC 3339, where generated output will always be Z-normalized and uses 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: |
updateTime |
Output only. The most recent time at which the question was updated. Uses RFC 3339, where generated output will always be Z-normalized and uses 0, 3, 6 or 9 fractional digits. Offsets other than "Z" are also accepted. Examples: |
questionType |
The type of question. |
predefinedQuestionConfig |
The configuration of the predefined question. This field will only be set if the Question Type is predefined. |
questionBody |
Question text. E.g., "Did the agent greet the customer?" |
answerInstructions |
Instructions describing how to determine the answer. |
answerChoices[] |
A list of valid answers to the question, which the LLM must choose from. |
tags[] |
Questions are tagged for categorization and scoring. Tags can either be: - Default Tags: These are predefined categories. They are identified by their string value (e.g., "BUSINESS", "COMPLIANCE", and "CUSTOMER"). - Custom Tags: These are user-defined categories. They are identified by their full resource name (e.g., projects/{project}/locations/{location}/qaQuestionTags/{qaQuestionTag}). Both default and custom tags are used to group questions and to influence the scoring of each question. |
order |
Defines the order of the question within its parent scorecard revision. |
metrics |
Metrics of the underlying tuned LLM over a holdout/test set while fine tuning the underlying LLM for the given question. This field will only be populated if and only if the question is part of a scorecard revision that has been tuned. |
tuningMetadata |
Metadata about the tuning operation for the question.This field will only be populated if and only if the question is part of a scorecard revision that has been tuned. |
QaQuestionType
Enum representing the type of the question.
Enums | |
---|---|
QA_QUESTION_TYPE_UNSPECIFIED |
The type of the question is unspecified. |
CUSTOMIZABLE |
The default question type. The question is fully customizable by the user. |
PREDEFINED |
The question type is using a predefined model provided by CCAI teams. Users are not allowed to edit the questionBody, answerChoices, upload feedback labels for the question nor fine-tune the question. However, users may edit other fields like question tags, question order, etc. |
PredefinedQuestionConfig
Configuration for a predefined question. This field will only be set if the Question Type is predefined.
JSON representation |
---|
{
"type": enum ( |
Fields | |
---|---|
type |
The type of the predefined question. |
PredefinedQuestionType
Enum representing the type of the predefined question.
Enums | |
---|---|
PREDEFINED_QUESTION_TYPE_UNSPECIFIED |
The type of the predefined question is unspecified. |
CONVERSATION_OUTCOME |
A prebuilt classifier classfying the outcome of the conversation. For example, if the customer issue mentioned in a conversation has been resolved or not. |
CONVERSATION_OUTCOME_ESCALATION_INITIATOR_ROLE |
A prebuilt classifier classfying the initiator of the conversation escalation. For example, if it was initiated by the customer or the agent. |
AnswerChoice
Message representing a possible answer to the question.
JSON representation |
---|
{ "key": string, // Union field |
Fields | |
---|---|
key |
A short string used as an identifier. |
Union field value . The answer value may be one of a few different types. value can be only one of the following: |
|
strValue |
String value. |
numValue |
Numerical value. |
boolValue |
Boolean value. |
naValue |
A value of "Not Applicable (N/A)". If provided, this field may only be set to |
score |
Numerical score of the answer, used for generating the overall score of a QaScorecardResult. If the answer uses naValue, this field is unused. |
Metrics
A wrapper representing metrics calculated against a test-set on a LLM that was fine tuned for this question.
JSON representation |
---|
{ "accuracy": number } |
Fields | |
---|---|
accuracy |
Output only. Accuracy of the model. Measures the percentage of correct answers the model gave on the test set. |
TuningMetadata
Metadata about the tuning operation for the question. Will only be set if a scorecard containing this question has been tuned.
JSON representation |
---|
{
"totalValidLabelCount": string,
"datasetValidationWarnings": [
enum ( |
Fields | |
---|---|
totalValidLabelCount |
Total number of valid labels provided for the question at the time of tuining. |
datasetValidationWarnings[] |
A list of any applicable data validation warnings about the question's feedback labels. |
tuningError |
Error status of the tuning operation for the question. Will only be set if the tuning operation failed. |
DatasetValidationWarning
Enum for the different types of issues a tuning dataset can have. These warnings are currentlyraised when trying to validate a dataset for tuning a scorecard.
Enums | |
---|---|
DATASET_VALIDATION_WARNING_UNSPECIFIED |
Unspecified data validation warning. |
TOO_MANY_INVALID_FEEDBACK_LABELS |
A non-trivial percentage of the feedback labels are invalid. |
INSUFFICIENT_FEEDBACK_LABELS |
The quantity of valid feedback labels provided is less than the recommended minimum. |
INSUFFICIENT_FEEDBACK_LABELS_PER_ANSWER |
One or more of the answers have less than the recommended minimum of feedback labels. |
ALL_FEEDBACK_LABELS_HAVE_THE_SAME_ANSWER |
All the labels in the dataset come from a single answer choice. |
Methods |
|
---|---|
|
Create a QaQuestion. |
|
Deletes a QaQuestion. |
|
Gets a QaQuestion. |
|
Lists QaQuestions. |
|
Updates a QaQuestion. |