Live API reference

This document provides a reference for the Live API and shows you how to use it for bidirectional streaming with Gemini. It covers the following topics:

• Capabilities: Learn about the key features of the Live API, including multimodality and low-latency interaction.
• Get started: Find a quick-start example for using the API with text-to-text generation.
• Integration guide: Understand the core concepts for integrating with the API, such as sessions, message types, and function calling.
• Limitations: Review the current limitations of the API, including session duration and authentication.
• Messages and events: See the detailed reference for all client and server messages.

The following diagram summarizes the overall workflow:

The Live API lets you have low-latency, bidirectional voice and video conversations with Gemini. You can create natural, human-like interactions for your users, including the ability to interrupt the model's responses with voice commands. The Live API processes text, audio, and video input, and provides text and audio output.

For more information about the Live API, see Live API.

Capabilities

The Live API includes the following key capabilities:

• Multimodality: The model can see, hear, and speak.
• Low-latency, real-time interaction: Get fast responses from the model for a fluid conversational experience.
• Session memory: The model retains memory of all interactions within a single session, recalling previously heard or seen information.
• Support for function calling, code execution, and Search as a Tool: You can integrate the model with external services and data sources.

The Live API is designed for server-to-server communication. For web and mobile apps, we recommend that you use the integration from our partners at Daily.

Supported models

• Gemini 2.5 Flash with Live API native audio (Preview)
• Gemini 2.0 Flash with Live API (Preview)
• Gemini 2.5 Flash

Get started

To try the Live API, go to the Vertex AI Studio, and then click Start Session.

The Live API is a stateful API that uses WebSockets.

This section provides an example of how to use the Live API for text-to-text generation with Python 3.9 or later.

pip install --upgrade google-genai

# Replace the `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION` values
# with appropriate values for your project.
export GOOGLE_CLOUD_PROJECT=GOOGLE_CLOUD_PROJECT
export GOOGLE_CLOUD_LOCATION=global
export GOOGLE_GENAI_USE_VERTEXAI=True

from google import genai
from google.genai.types import (Content, HttpOptions, LiveConnectConfig,
                                Modality, Part)

client = genai.Client(http_options=HttpOptions(api_version="v1beta1"))
model_id = "gemini-2.0-flash-live-preview-04-09"

async with client.aio.live.connect(
    model=model_id,
    config=LiveConnectConfig(response_modalities=[Modality.TEXT]),
) as session:
    text_input = "Hello? Gemini, are you there?"
    print("> ", text_input, "\n")
    await session.send_client_content(
        turns=Content(role="user", parts=[Part(text=text_input)])
    )

    response = []

    async for message in session.receive():
        if message.text:
            response.append(message.text)

    print("".join(response))
# Example output:
# >  Hello? Gemini, are you there?
# Yes, I'm here. What would you like to talk about?

Integration guide

This section describes how to integrate your application with the Live API.

Sessions

A WebSocket connection establishes a session between the client and the Gemini server. After you initiate a new connection, your client can exchange messages with the server to do the following:

• Send text, audio, or video to the Gemini server.
• Receive audio, text, or function call requests from the Gemini server.

Send the session configuration in the first message after you establish a connection. A session configuration includes the model, generation parameters, system instructions, and tools.

The following is an example configuration:

{
  "model": string,
  "generationConfig": {
    "candidateCount": integer,
    "maxOutputTokens": integer,
    "temperature": number,
    "topP": number,
    "topK": integer,
    "presencePenalty": number,
    "frequencyPenalty": number,
    "responseModalities": [string],
    "speechConfig": object
  },

  "systemInstruction": string,
  "tools": [object]
}

For more information, see BidiGenerateContentSetup.

Send messages

Messages are JSON-formatted objects that are exchanged over the WebSocket connection.

To send a message, your client sends a JSON object over an open WebSocket connection. The JSON object must have exactly one of the fields from the following object set:

{
  "setup": BidiGenerateContentSetup,
  "clientContent": BidiGenerateContentClientContent,
  "realtimeInput": BidiGenerateContentRealtimeInput,
  "toolResponse": BidiGenerateContentToolResponse
}

Supported client messages

The following table describes the messages that you can send to the server.

Receive messages

To receive messages from Gemini, listen for the WebSocket message event. Then, parse the result according to the definitions of the supported server messages.

For example:

ws.addEventListener("message", async (evt) => {
  if (evt.data instanceof Blob) {
    // Process the received data (audio, video, etc.)
  } else {
    // Process JSON response
  }
});

Server messages have exactly one of the fields from the following object set:

{
  "setupComplete": BidiGenerateContentSetupComplete,
  "serverContent": BidiGenerateContentServerContent,
  "toolCall": BidiGenerateContentToolCall,
  "toolCallCancellation": BidiGenerateContentToolCallCancellation
  "usageMetadata": UsageMetadata
  "goAway": GoAway
  "sessionResumptionUpdate": SessionResumptionUpdate
  "inputTranscription": BidiGenerateContentTranscription
  "outputTranscription": BidiGenerateContentTranscription
}

Supported server messages

The following table describes the messages that you can receive from the server.

Incremental content updates

Use incremental updates to send text input, establish session context, or restore session context. For short contexts, you can send turn-by-turn interactions to represent the exact sequence of events. For longer contexts, we recommend that you provide a single message summary to free up the context window for follow-up interactions.

The following is an example of a context message:

{
  "clientContent": {
    "turns": [
      {
          "parts":[
          {
            "text": ""
          }
        ],
        "role":"user"
      },
      {
          "parts":[
          {
            "text": ""
          }
        ],
        "role":"model"
      }
    ],
    "turnComplete": true
  }
}

While content parts can be of a functionResponse type, don't use BidiGenerateContentClientContent to respond to function calls from the model. Instead, use BidiGenerateContentToolResponse. Use BidiGenerateContentClientContent only to establish previous context or provide text input to the conversation.

Streaming audio and video

Code execution

To learn more about code execution, see Code execution.

Function calling

You must declare all functions at the start of the session by sending tool definitions as part of the BidiGenerateContentSetup message.

You define functions by using JSON, specifically with a select subset of the OpenAPI schema format.

A single function declaration can include the following parameters:

• name (string): The unique identifier for the function within the API call.
• description (string): A comprehensive explanation of the function's purpose and capabilities.
• parameters (object): Defines the input data required by the function.
  • type (string): Specifies the overall data type, such as object.
  • properties (object): Lists individual parameters, each with:
    • type (string): The data type of the parameter, such as string, integer, or boolean.
    • description (string): A clear explanation of the parameter's purpose and expected format.
  • required (array): An array of strings listing the parameter names that are mandatory for the function to operate.

For code examples of a function declaration using curl commands, see Function calling with the Gemini API. For examples of how to create function declarations using the Gemini API SDKs, see the Function calling tutorial.

From a single prompt, the model can generate multiple function calls and the code to chain their outputs. This code executes in a sandbox environment and generates subsequent BidiGenerateContentToolCall messages. Execution pauses until the results of each function call are available, which provides sequential processing.

Your client should respond with BidiGenerateContentToolResponse.

To learn more, see Introduction to function calling.

Audio formats

See the list of supported audio formats.

System instructions

You can provide system instructions to better control the model's output and specify the tone and sentiment of audio responses.

System instructions are added to the prompt before the interaction begins and remain in effect for the entire session. You can only set them at the beginning of a session, immediately after the initial connection. To provide more input to the model during the session, use incremental content updates.

Interruptions

Users can interrupt the model's output at any time. When Voice activity detection (VAD) detects an interruption, the ongoing generation is canceled and discarded. Only the information that has already been sent to the client is retained in the session history. The server then sends a BidiGenerateContentServerContent message to report the interruption.

In addition, the Gemini server discards any pending function calls and sends a BidiGenerateContentServerContent message with the IDs of the canceled calls.

Voices

To specify a voice, set the voiceName within the speechConfig object as part of your session configuration.

The following is a JSON representation of a speechConfig object:

{
  "voiceConfig": {
    "prebuiltVoiceConfig": {
      "voiceName": "VOICE_NAME"
    }
  }
}

To see the list of supported voices, see Change voice and language settings.

Limitations

Consider the following limitations of the Live API and Gemini 2.0 when you plan your project.

Client authentication

The Live API provides only server-to-server authentication and isn't recommended for direct client use. Route client input through an intermediate application server for secure authentication with the Live API.

Maximum session duration

The default maximum length of a conversation session is 10 minutes. For more information, see Session length.

Voice activity detection (VAD)

By default, the model automatically performs voice activity detection (VAD) on a continuous audio input stream. You can configure VAD with the RealtimeInputConfig.AutomaticActivityDetection field of the setup message.

The API supports two VAD modes:

• Automatic VAD (default): The server detects user speech.

  • If the audio stream is paused for more than a second (for example, when the user mutes the microphone), an AudioStreamEnd event is sent to flush any cached audio.
  • Your client can resume sending audio data at any time.

• Manual VAD: To use this mode, set RealtimeInputConfig.AutomaticActivityDetection.disabled to true in the setup message.

  • Your client is responsible for detecting user speech and sending ActivityStart and ActivityEnd messages at the appropriate times.
  • An AudioStreamEnd event isn't sent in this configuration. Instead, an ActivityEnd message marks any interruption of the stream.

Additional limitations

• Manual endpointing isn't supported.
• Audio inputs and outputs can negatively impact the model's ability to use function calling.

Token count

Token count isn't supported.

Rate limits

The following rate limits apply:

• 5,000 concurrent sessions per API key
• 4M tokens per minute

Messages and events

What's next

• Learn more about function calling.
• See the Function calling reference for examples.