Batch prediction with Gemini

Get asynchronous, high-throughput, and cost-effective inference for your large-scale data processing needs with Gemini's batch prediction capabilities. This guide will walk you through the value of batch prediction, how it works, its limitations, and best practices for optimal results.

Why use batch prediction?

In many real-world scenarios, you don't need an immediate response from a language model. Instead, you might have a large dataset of prompts that you need to process efficiently and affordably. This is where batch prediction shines.

Key benefits include:

  • Cost-Effectiveness: Batch processing is offered at a 50% discounted rate compared to real-time inference, making it ideal for large-scale, non-urgent tasks.
  • High rate limits: Process hundreds of thousands of requests in a single batch with a higher rate limit compared to the real time Gemini API.
  • Simplified Workflow: Instead of managing a complex pipeline of individual real-time requests, you can submit a single batch job and retrieve the results once the processing is complete. The service will handle format validation, parallelize requests for concurrent processing, and automatically retry to strive for a high completion rate with 24 hours turnaround time.

Batch prediction is optimized for large-scale processing tasks like:

  • Content Generation: Generate product descriptions, social media posts, or other creative text in bulk.
  • Data Annotation and Classification: Classify user reviews, categorize documents, or perform sentiment analysis on a large corpus of text.
  • Offline Analysis: Summarize articles, extract key information from reports, or translate documents at scale.

Gemini models that support batch predictions

The following base and tuned Gemini models support batch predictions:

Quotas and limits

While batch prediction is powerful, it's important to be aware of the following limitations.

  • Quota: There are no predefined quota limits on your usage. Instead, batch service provides access to a large, shared pool of resources, dynamically allocated based on availability of resources and real-time demand across all customers of that model. When more customers are active and saturated our capacity, your batch requests may be queued for capacity.
  • Queue Time: When our service experiences high traffic, your batch job will queue for capacity. The job will be in queue for up to 72 hours before it expires.
  • Request Limits: A single batch job may include up to 200,000 requests. If you are using Cloud Storage as input, there is also a file size limit of 1GB.
  • Processing Time: Batch jobs are processed asynchronously and are not designed for real-time applications. Most jobs complete within 24 hours after it starts running (not counting the queue time). After 24 hours, incomplete jobs will be cancelled, and you will only be charged for completed requests.
  • Unsupported features: Batch prediction does not support Context Caching, RAG, or Global endpoints.

Best practices

To get the most out of batch prediction with Gemini, we recommend the following best practices:

  • Combine jobs: To maximize throughput, combine smaller jobs into one large job, within system limits. For example, submitting one batch job with 200,000 requests will give you better throughput than 1000 jobs with 200 requests each.
  • Monitor Job Status: You can monitor job progress using API, SDK, or UI. For more information, see monitor the job status. If a job fails, check the error messages to diagnose and troubleshoot the issue.
  • Optimize for Cost: Take advantage of the cost savings offered by batch processing for any tasks that don't require an immediate response.

What's next