Multimodal Live API

The Multimodal Live API enables low-latency, two-way interactions that use text, audio, and video input, with audio and text output. This facilitates natural, human-like voice conversations with the ability to interrupt the model at any time. The model's video understanding capability expands communication modalities, enabling you to share camera input or screencasts and ask questions about them.

Capabilities

Multimodal Live API includes the following key capabilities:

• Multimodality: The model can see, hear, and speak.
• Low-latency realtime interaction: The model can provide fast responses.
• Session memory: The model retains memory of all interactions within a single session, recalling previously heard or seen information.
• Support for function calling, code execution, and Search as a Tool: You can integrate the model with external services and data sources.

Multimodal Live API is designed for server-to-server communication.

For web and mobile apps, we recommend using integration from our partners at Daily.

Get started

Multimodal Live API is a stateful API that uses WebSockets.

This section shows an example of how to use Multimodal Live API for text-to-text generation, using Python 3.9+.

Install the Gemini API library

To install the google-genai package, use the following pip command:

!pip3 install google-genai

Import dependencies

To import dependencies:

import asyncio
import base64
import contextlib
import datetime
import os
import json
import wave
import itertools

from IPython.display import display, Audio
from google import genai

Set the model name and the API key

MODEL = "models/gemini-2.0-flash-exp"
os.environ['GOOGLE_API_KEY'] = 'YOUR_API_KEY'

Send and receive text messages

client = genai.Client(
  http_options={
    'api_version': 'v1alpha',
    'url': 'generativelanguage.googleapis.com',
  }
)

config={
  "generation_config": {"response_modalities": ["TEXT"]}
}

async with client.aio.live.connect(model=MODEL, config=config) as session:
  message = "Hello? Gemini are you there?"
  print("> ", message, "\n")
  await session.send(message, end_of_turn=True)

  # For text responses, when the model's turn is complete it breaks out of the loop.
  async for response in session.receive():
    model_turn = response.server_content.model_turn
    for part in model_turn.parts:
      print("- ", part.text)

Integration guide

This section describes how integration works with Multimodal Live API.

Sessions

A session represents a single WebSocket connection between the client and the Gemini server.

After a client initiates a new connection the session can exchange messages with the server to:

• Send text, audio, or video to the Gemini server.
• Receive audio, text, or function call responses from the Gemini server.

The session configuration is sent in the first message after connection. A session configuration includes the model, generation parameters, system instructions, and tools.

See the following example configuration:

{​​
  "model": string,
  "generation_config": {​​
    "candidate_count": integer,
    "max_output_tokens": integer,
    "temperature": number,
    "top_p": number,
    "top_k": integer,
    "presence_penalty": number,
    "frequency_penalty": number,
    "response_modalities": string,
    "speech_config":object
  },

  "system_instruction": "",
  "tools":[]
}

For more information, see BidiGenerateContentSetup.

Send messages

Messages are JSON-formatted strings exchanged over the WebSocket connection.

To send a message the client must send a supported client message in a JSON formatted string with one of over an open WebSocket connection.

Supported client messages

See the supported client messages in the following table:

Receive messages

To receive messages from Gemini, listen for the WebSocket 'message' event, and then parse the result according to the definition of supported server messages.

See the following:

ws.addEventListener("message", async (evt) => {
  if (evt.data instanceof Blob) {
    // Process the received data (audio, video, etc.)
  } else {
    // Process JSON response
  }
});

Supported server messages

See the supported server messages in the following table:

Incremental content updates

Use incremental updates to send text input or establish/restore session context. For short contexts you can send turn-by-turn interactions to represent the exact sequence of events. For longer contexts it's recommended to provide a single message summary to free up the context window for the follow up interactions.

See the following example context message:

{
  "client_content": {
    "turns": [
      {
          "parts":[
          {
            "text": ""
          }
        ],
        "role":"user"
      },
      {
          "parts":[
          {
            "text": ""
          }
        ],
        "role":"model"
      }
    ],
    "turn_complete": true
  }
}

Note that while content parts can be of a functionResponse type, BidiGenerateContentClientContent shouldn't be used to provide a response to the function calls issued by the model. BidiGenerateContentToolResponse should be used instead. BidiGenerateContentClientContent should only be used to establish previous context or provide text input to the conversation.

Function calling

All functions must be declared at the start of the session by sending tool definitions as part of the BidiGenerateContentSetup message.

You define functions by using JSON, specifically with a select subset of the OpenAPI schema format. A single function declaration can include the following parameters:

• name (string): The unique identifier for the function within the API call.
• description (string): A comprehensive explanation of the function's purpose and capabilities.
• parameters (object): Defines the input data required by the function.
  • type (string): Specifies the overall data type, such as object.
  • properties (object): Lists individual parameters, each with:
  • type (string): The data type of the parameter, such as string, integer, boolean.
  • description (string): A clear explanation of the parameter's purpose and expected format.
  • required (array): An array of strings listing the parameter names that are mandatory for the function to operate.

For code examples of a function declaration using cURL commands, see Intro to function calling with the Gemini API. For examples of how to create function declarations using the Gemini API SDKs, see the Function calling tutorial.

From a single prompt, the model can generate multiple function calls and the code necessary to chain their outputs. This code executes in a sandbox environment, generating subsequent BidiGenerateContentToolCall messages. The execution pauses until the results of each function call are available, which ensures sequential processing.

The client should respond with BidiGenerateContentToolResponse.

Audio formats

Multimodal Live API supports the following audio formats:

• Input audio format: Raw 16 bit PCM audio at 16kHz little-endian
• Output audio format: Raw 16 bit PCM audio at 24kHz little-endian

System instructions

You can provide system instructions to better control the model's output and specify the tone and sentiment of audio responses.

System instructions are added to the prompt before the interaction begins and remain in effect for the entire session.

System instructions can only be set at the beginning of a session, immediately following the initial connection. To provide further input to the model during the session, use incremental content updates.

Interruptions

Users can interrupt the model's output at any time. When Voice activity detection (VAD) detects an interruption, the ongoing generation is canceled and discarded. Only the information already sent to the client is retained in the session history. The server then sends a BidiGenerateContentServerContent message to report the interruption.

In addition, the Gemini server discards any pending function calls and sends a BidiGenerateContentServerContent message with the IDs of the canceled calls.

Voices

Multimodal Live API supports the following voices:

• Puck
• Charon
• Kore
• Fenrir
• Aoede

To specify a voice, set the voice_name within the speech_config object, as part of your session configuration.

See the following JSON representation of a speech_config object:

{
  "voice_config": {
    "prebuilt_voice_config ": {
      "voice_name": VOICE_NAME
    }
  }
}

Limitations

Consider the following limitations of Multimodal Live API and Gemini 2.0 when you plan your project.

Client authentication

Multimodal Live API only provides server to server authentication and isn't recommended for direct client use. Client input should be routed through an intermediate application server for secure authentication with the Multimodal Live API.

Conversation history

While the model keeps track of in-session interactions, explicit session history accessible through the API isn't available yet. When a session is terminated the corresponding context is erased.

In order to restore a previous session or provide the model with historic context of user interactions, the application should maintain its own conversation log and use a BidiGenerateContentClientContent message to send this information at the start of a new session.

Maximum session duration

Session duration is limited to up to 15 minutes for audio or up to 2 minutes of audio and video. When the session duration exceeds the limit, the connection is terminated.

The model is also limited by the context size. Sending large chunks of content alongside the video and audio streams may result in earlier session termination.

Voice activity detection (VAD)

The model automatically performs voice activity detection (VAD) on a continuous audio input stream. VAD is always enabled, and its parameters aren't currently configurable.

Additional limitations

Manual endpointing isn't supported.

Audio inputs and audio outputs negatively impact the model's ability to use function calling.

Token count

Token count isn't supported.

Rate limits

The following rate limits apply:

• 3 concurrent sessions per API key
• 4M tokens per minute

Messages and events

See also

• For more information on commonly-used API fields (for example,Content and Tool), see Generate content.
• Learn more about function calling.
• See the Function calling reference for examples.