The virtual agent platform is a feature within Quality AI that provides insights into the performance of conversational agents created with Dialogflow.
View platform
Follow these steps to view the platform in the Conversational Insights console.
- Click Quality AI > Agents > Virtual agent.
- Select an agent.
Virtual agent performance
The virtual agent platform provides details about virtual agent performance. The platform displays details of both operational and Quality AI metrics.
Operational metrics
The virtual agent platform displays each of the following metrics as a number or percentage.
- Total sessions: Total number of conversations handled by this agent.
- Escalation rate: Percentage of conversations escalated to a human agent, calculated using a signal from Dialogflow.
- Turns per session: Average number of turns per conversation.
- No match rate: Percentage of conversations that did not match any intent, applicable only for flow-based virtual agents.
The virtual agent platform displays a graph of the change over time for each of the following metrics.
- Volume: Total number of conversations handled by this agent.
- Escalation rate: Percentage of conversations escalated to a human agent, calculated using a signal from Dialogflow.
- Escalation type breakdown: Number of conversations per escalation initiator, which could be the user or the agent, analyzed based on the predefined question Who escalated the conversation?
- Tool failure rate: Percentage of tool calls that failed across all uses of the tool in conversations for a specific agent, in the selected time period and conversation medium.
- Tool latency: Average latency of a tool call across all uses of the tool in the conversations for the specific agent, in the selected time period and conversation medium.
- No match rate: Percentage of conversations that did not match any intent, applicable only for flow-based virtual agents.
The virtual agent platform also displays a graph called E2E latency breakdown, which shows the average latency in milliseconds for tools calls, large language model (LLM) calls, and Text-to-Speech (TTS) calls. To compute this average, the tool, LLM, or TTS latency is averaged across interactions in a conversation, then averaged across conversations.
Escalation type breakdown
The escalation type breakdown shows the number of conversations for each escalation initiator: user or agent. Quality AI determines the escalation initiator by answering the predefined question Who escalated the conversation? You can drill down on an escalation initiator to view a list of conversations with it.
Tools
Tool metrics are computed for Conversational agent tools. Aggregated metrics like tool latency and failure rates help bot builders identify performance bottlenecks across conversations.
Quality AI metrics
The virtual agent platform displays the following Quality AI metrics.
- Quality score: Average quality score per scorecard over conversations handled by this agent.
- Overall sentiment: Average sentiment score over conversations handled by this agent.
- Sentiment breakdown: A color-coded bar chart to illustrate the number of conversations of this agent per conversation-level sentiment category: negative, neutral, or positive.
- Conversation outcome: Number of conversations for each possible outcome.
- Sentiment by topic: Breakdown of the number of conversations per conversation-level sentiment category for this metric.
The platform also displays graphs of the change over time for the following Quality AI metrics.
- Quality score: Percentage of quality scores for all conversations handled by this agent.
- Score category breakdown: Quality score numbers for each predefined category: business, compliance, customer, and any custom categories.
Conversation outcome
The conversation outcome graph shows the number of conversations that ended with each of the following possible outcomes:
- Abandoned
- Partially resolved
- Escalated
- Redirected
- Successfully resolved
- Unknown
Compute conversation outcomes using predefined questions in Quality AI.
- To view outcome data in this graph, run a Quality AI analysis with a scorecard containing a predefined question.
- Click an outcome to see a list of conversations with that outcome.
Sentiment breakdown and sentiment by topic
After you run sentiment analysis on all your conversations, the sentiment breakdown chart displays the number of conversations with an overall sentiment in each category: negative, neutral, and positive.
With your deployed topic model, you can also view the sentiment for each topic. Choose a sentiment category in one of the graphs to view a list of conversations with that sentiment.