This document contains a list of notebook tutorials for Generative AI on Vertex AI. They're end-to-end tutorials that show you how to use some of the GenAI LLMs.
Featured tutorials
-
Multimodal use cases with Gemini
Explore various use cases with multimodal with Gemini.
-
Function Calling with the Vertex AI Gemini API & Python SDK
Use the Vertex AI Gemini API with the Vertex AI SDK for Python to make function calls using the Gemini 1.0 Pro (
gemini-1.0-pro) model. -
Get Started with Grounding with Gemini in Vertex AI
Use generative text models to generate content grounded in your documents and data.
List of tutorials
| Gemini Multimodal |
Introduction to Gemini 1.5 Pro Use Gemini 1.5 Pro to analyze audio files, understand video, extract information from a PDF, and process multiple types of media simultaneously. |
|
| Gemini Multimodal |
Use this notebook to learn how to generate code, summarize a codebase, debug, improve code, and assess code with Gemini 1.5 Pro. |
|
| Gemini Multimodal |
Get started with Gemini (cUrl) Use the Gemini API, which gives you access to Google's latest large language models, with REST/curl. |
|
| Gemini Multimodal |
Get started with Gemini (Python SDK) Use the Gemini API, which gives you access to Google's latest large language models, with the Vertex AI SDK for Python. |
|
| Gemini Multimodal |
Multimodal use cases with Gemini The Gemini model is a groundbreaking multimodal language model developed by Google AI, capable of extracting meaningful insights from a diverse array of data formats, including images, and video. This notebook explores various use cases with multimodal prompts. |
|
| Gemini evaluation |
Getting Started with Gen AI Evaluation Service in the Vertex AI SDK for Python Use Gen AI Evaluation Service SDK to evaluate the Gemini model on an evaluation task, with the Vertex AI SDK for Python. |
|
| Gemini evaluation |
Prompt Engineering, Evaluation, and Prompt Templating with Gemini Use Gen AI Evaluation Service SDK for prompt engineering and evaluation with the Gemini model, with the Vertex AI SDK for Python. |
|
| Gemini evaluation |
Compare and Select Generative Models Use Gen AI Evaluation Service SDK to compare and evaluate different generative models on a specific evaluation task, then visualize and compare the evaluation results for the task, with the Vertex AI SDK for Python. |
|
| Gemini evaluation |
Evaluate and Compare Gemini Model Generation Settings Use Gen AI Evaluation Service SDK to evaluate and select temperature and other model generation configurations of Gemini and compare the metric results of different generation settings, with the Vertex AI SDK for Python. |
|
| Gemini evaluation |
Compare and migrate from PaLM to Gemini Model This notebook guides you through using Gen AI Evaluation Service SDK to evaluate PaLM and Gemini foundation models with multiple evaluation metrics, to support decisions around migrating from one model to another. These metrics are visualized to help you gain insights into the strengths and weaknesses of each model, ultimately helping you make an informed decision about which one aligns best with the specific requirements of your use case. |
|
| Gemini evaluation |
Gen AI Evaluation Service SDK Preview-to-GA Migration Guide In this tutorial, you will get detailed guidance on how to migrate from the Preview version to the latest GA version of Vertex AI Python SDK for Gen AI Evaluation Service to evaluate Retrieval-Augmented Generation (RAG) and compare two models side-by-side (SxS). |
|
| Gemini evaluation |
Evaluate and Compare Gemini Model Generation Settings Use Gen AI Evaluation Service SDK to evaluate and select temperature and other model generation configurations of Gemini and compare the metric results of different generation settings, with the Vertex AI SDK for Python. |
|
| Gemini evaluation |
Customize Model-based Metrics to Evaluate a Gen AI Model Use Gen AI Evaluation Service SDK to customize the model-based metrics and evaluate a generative AI model based on your criteria, with the Vertex AI SDK for Python. |
|
| Gemini evaluation |
Bring Your Own Autorater Using Custom Metrics to Evaluate a Gen AI Model Use Gen AI Evaluation Service SDK to evaluate generative AI models with your locally-defined custom metrics, and use your own autorater model to perform model-based metric evaluation, with the Vertex AI SDK for Python. |
|
| Gemini evaluation |
Bring Your Own Computation-based Custom Metrics to Evaluate a Gen AI Model Use Gen AI Evaluation Service SDK to evaluate generative AI models with your locally-defined computation-based custom metrics, with the Vertex AI SDK for Python. |
|
| Gemini evaluation |
Evaluate the Gemini 1.0 Pro Model Function Call Quality Generate function calls with Gemini 1.0 Pro model, and use Gen AI Evaluation Service SDK to evaluate the Gemini 1.0 Pro model function call quality, with the Vertex AI SDK for Python. |
|
| Gemini evaluation |
Use Gen AI Evaluation Service SDK to evaluate Retrieval-Augmented Generation (RAG) generated answers for a Question Answering (QA) task, with the Vertex AI SDK for Python. |
|
| Gemini Multimodal Streamlit |
Deploy a Streamlit app to Cloud Run with Gemini Pro Sample app to deploy a simple chatbot app using Streamlit to Cloud Run with Gemini Pro. |
|
| Gemini Multimodal Retrieval-augmented-generation |
Extending from RAG, which is traditionally performed over text data, this notebook shows how you can perform RAG over multimodal data to do Q&A in a scientific paper containing text and images. |
|
| Gemini Multimodal |
Using the Gemini model in education, with various examples of prompts, and across modalities including images and video. |
|
| Gemini Multimodal |
Using Gemini for Multimodal Retail Recommendations In the world of retail, recommendations play a pivotal role in influencing customer decisions and driving sales. In this notebook, you will learn how to harness the power of multimodality to perform retail recommendations to help a customer choose the best chair among four images of chairs, based on an image of their living room. |
|
| Gemini Multimodal Function-calling |
Intro to Function Calling with Gemini Use the Gemini Pro model to:
|
|
| Tuning |
Tune foundational models with Vertex AI Walk through the entire setup and integration process. From environment setup, to foundational model selection, and then to tuning with Vertex AI. |
|
| Evaluation |
Vertex AI LLM Evaluation Services Use Vertex AI LLM Evaluation Services in conjunction with other Vertex AI services. |
|
| LangChain |
Run a LangChain chain and print out details of what's happening in each step of the chain and with optional debugging breakpoints. |
|
| Text |
Advanced Prompt Engineering Training Use Chain of Thought and ReAct (Reasoning + Acting) to engineer prompts and reduce hallucinations. |
|
| Embeddings |
Use Vertex AI Embeddings for Multimodal and Vector Search Create text-to-image embeddings using the DiffusionDB dataset and the Vertex AI Embeddings for Multimodal model. The embeddings are uploaded to the Vector Search service, which is a high scale, low latency solution to find similar vectors for a large corpus. |
|
| Embeddings |
Semantic Search using Embeddings Create an embedding generated from text and perform a semantic search. The embeddings are generated using Google ScaNN: Efficient Vector Similarity Search. |
|
| Evaluation |
AutoSxS: Evaluate an LLM in Vertex AI Model Registry against a third-party model Use Vertex AI automatic side by side (AutoSxS) to evaluate the performance between a generative AI model in Vertex AI model registry and a third-party language model. |
|
| Evaluation |
AutoSxS: Check autorater alignment against a human-preference dataset Use Vertex AI automatic side by side (AutoSxS) to determine how well the autorater aligns with human raters. |
|
| Tuning |
Vertex AI LLM Reinforcement Learning from Human Feedback Use Vertex AI RLHF to tune a large-language model (LLM). This workflow improves a model's accuracy by fine-tuning a base model with a training dataset. |
|
| Tuning |
Vertex AI Batch Inference with RLHF-tuned Models This tutorial demonstrates how to perform inference on RLHF-tuned OSS large-language models (LLMs) end-to-end with Vertex AI. |
|
| Embeddings |
Try the new text embedding models. |
|
| Tuning |
Tune a PEFT large-language model (LLM) and make a prediction. This workflow improves a model's accuracy by fine-tuning a base model with a training dataset. |
|
| Text |
Use the Vertex AI SDK with Large Language Models Use the Vertex AI SDK to run Large Language Models on Vertex AI. Test, tune, and deploy generative AI language models. Get started by exploring examples of content summarization, sentiment analysis, chat, text embedding, and prompt tuning. |
|
Data store Indexing Search Conversation |
Vertex AI Search and Conversation Data Store Status Checker Vertex AI Search and Conversation Data Store Status Checker is a notebook that uses the Cloud Discovery Engine API to check a Data Store for indexed documents. It allows the user to perform the following tasks:
|
|
Speech Recognition Chirp |
Get started with Chirp on Google Cloud This notebook is an introduction to Chirp, a speech-to-text service that uses Google's state-of-the-art speech recognition technology. It provides a simple and easy-to-use interface for developers to build speech-enabled applications. |
|
Filter Metadata Search |
Vertex AI Search with Filters & Metadata Vertex AI Search is a fully managed service that lets you build and deploy search applications on Google Cloud. This notebook shows how to use filters and metadata in search requests to Vertex AI Search. |
|
Document Question-answering Retrieval-augmented-generation |
Document Q&A With Retrieval Augmented Generation This notebook demonstrates how to use Retrieval Augmented Generation (RAG) to build a question answering system for Google documentation. It shows how to use RAG to generate text that answers a given question, and how to use RAG to improve the performance of a question answering system. |
|
Contract Generation Retrieval Search |
This notebook demonstrates how to use retrieval augmented generation to generate contract text. It uses the Palm2 & LangChain models, which are pre-trained on large corpora of legal and financial text. |
|
Question-answering Retrieval-augmented-generation Search LangChain |
Question Answering Over Documents This notebook shows how to ask and answer questions about your data by combining a Vertex AI Search engine with LLMs. In particular, we focus on querying 'unstructured' data such as PDFs and HTML files. In order to run this notebook you must have created an unstructued search engine and ingested PDF or HTML documents into it. |
|
Bulk-question-answering Vertex AI Search Question-answering Search |
Bulk Question Answering with Vertex AI Search This notebook shows how to answer questions from a CSV using a Vertex AI Search data store. It can be run in Colab or Vertex AI Workbench. |
|
Language Orchestration LangChain PaLM |
Getting Started with LangChain 🦜️🔗 + PaLM API This notebook provides an introduction to LangChain, a language model orchestration framework. It shows how to use LangChain with the PaLM API to create and deploy a text-to-text generation model. |
|
BigQuery Data Loader LangChain |
How to use the LangChain 🦜️🔗 BigQuery Data Loader This notebook demonstrates how to use the LangChain BigQuery Data Loader to load data from BigQuery into a LangChain model. The notebook provides step-by-step instructions on how to set up the data loader, load data into the model, and train the model. |
|
Code Code-generation Retrieval-augmented-generation Codey |
Use Retrieval Augmented Generation (RAG) with Codey APIs This notebook demonstrates how to use Retrieval Augmented Generation (RAG) with Codey APIs. RAG is a technique that combines code retrieval with code generation to produce more accurate and informative code completion suggestions. |
|
Codey Code-generation Language |
Getting Started with the Vertex AI Codey APIs - Code Generation This notebook provides an introduction to the Vertex AI Codey APIs for code generation. It covers the basics of how to use the APIs, including how to create and deploy code generation models, and how to use them to generate code. |
|
Codey Code-completion Code-generation |
Getting Started with the Vertex AI Codey APIs - Code Completion This notebook demonstrates how to use the Vertex AI Codey APIs to get code completion suggestions for your Python code. It also shows how to use the APIs to generate code snippets and run code snippets in a remote environment. |
|
Codey Code-chat Chat Code-generation Text-generation |
Getting Started with the Vertex AI Codey APIs - Code Chat This notebook is an introduction to the Vertex AI Codey APIs. It covers the basics of how to use the APIs, including how to create and deploy models, and how to interact with them using the Codey CLI. |
|
Language PaLM Python SDK |
Getting Started with the PaLM API & Python SDK This notebook provides an introduction to the PaLM API and Python SDK. It covers the basics of how to use the API, including how to create and deploy models, and how to use the API to generate text, translate languages, and write different kinds of creative content. |
|
Language prompts |
Prompt Design - Best Practices This notebook provides an introduction to prompt design for text-based language models. It covers the basics of prompts, including how they work and how to write them. The notebook also provides tips on how to improve your prompts and avoid common pitfalls. |
|
Text-extraction |
Text Extraction with Generative Models on Vertex AI This notebook demonstrates how to use generative models to extract text from images. It uses the text-to-image model from the Vertex AI generative-ai library and the text-extraction model from the Vertex AI text-extraction library. |
|
Text-classification |
Text Classification with Generative Models on Vertex AI This notebook demonstrates how to use generative models to perform text classification on Vertex AI. It covers the following topics: * Preparing data * Training a model * Deploying a model * Using a model to classify text |
|
Chain of thought React |
This notebook introduces Chain of Thought and ReAct, two tools that can be used to improve the performance of reinforcement learning algorithms. Chain of Thought is a technique that can be used to improve the efficiency of value iteration, while ReAct is a technique that can be used to improve the stability of actor-critic algorithms. |
|
Language prompts Ideation |
Ideation with Generative Models on Vertex AI This notebook demonstrates how to use generative models to generate text, images, and code. It also shows how to use Vertex AI to deploy and manage generative models. |
|
Summarization |
Text Summarization with Generative Models on Vertex AI This notebook demonstrates how to use Vertex AI to train and deploy a text summarization model. It uses the BART model, which is a large language model that has been pre-trained on a massive dataset of text. The model is then fine-tuned on a dataset of text summaries, and can then be used to generate summaries of new text. |
|
Question-answering |
Question Answering with Generative Models on Vertex AI This notebook demonstrates how to use generative models to answer open-domain questions. It uses the Vertex AI Transformer model to generate text based on a given question. |
|
Text-generation Foundation-model Tuning Deploy |
Tuning and deploy a foundation model This notebook shows how to tune a foundation model using Vertex AI. It also shows how to deploy the tuned model to a Vertex AI endpoint. |
|
Document-summarization Summarization |
Text Summarization of Large Documents This notebook demonstrates how to use the t5 large model to summarize large documents. The model is trained on a massive dataset of text and code, and it can generate summaries that are both accurate and concise. |
|
Document-summarization LangChain Summarization |
Text Summarization of Large Documents using LangChain 🦜🔗 This notebook demonstrates how to use the LangChain model to summarize large documents. LangChain is a large language model that can generate text, translate languages, write different kinds of creative content, and answer your questions in an informative way. |
|
Document-summarization Document AI Language-model Summarization Text-summarization |
Summarization with Large Documents using Document AI and PaLM APIs This notebook demonstrates how to use Document AI and PaLM APIs to summarize large documents. It also shows how to use the Document AI API to extract entities and key phrases from a document. |
|
Chatbot Text-generation |
GroceryBot, a sample grocery and recipe assistant - RAG + ReAct This notebook is about a sample grocery and recipe assistant that uses RAG and ReAct. It can help you find recipes, create shopping lists, and answer questions about food. |
|
Question-answering Document-QA LangChain |
Question Answering with Large Documents using LangChain 🦜🔗 This notebook demonstrates how to use the LangChain model to build a question answering system that can answer questions about long documents. The model is trained on a large corpus of text and can be used to answer questions about any topic. |
|
Question-answering Document-QA LangChain Vector Search |
Question Answering with Documents using LangChain 🦜️🔗 and Vertex AI Vector Search This notebook demonstrates how to use LangChain and Vertex AI Vector Search (previously Matching Engine) to build a question answering system for documents. The system can answer questions about entities, dates, and numbers in documents. |
|
Document AI Question-answering PaLM |
Question answering with Documents using Document AI, Pandas, and PaLM This notebook demonstrates how to use Document AI, Pandas, and PaLM to build a question answering system. It first uses Document AI to extract structured data from a document, then uses Pandas to create a dataframe from the extracted data, and finally uses PaLM to generate answers to questions about the data. |
|
Question-answering Document-QA |
Question Answering with Large Documents This notebook demonstrates how to use the Vertex AI Question Answering service to build a question answering model that can answer questions from large documents. The model is trained on a dataset of Wikipedia articles and can answer questions about a variety of topics. |
|
Image Generation |
Product Description Generator From Image This notebook demonstrates how to generate product descriptions from images using a text-to-image model. The model is trained on a dataset of product images and their corresponding descriptions. |
|
Generation Retail LangChain |
DescriptionGen: SEO-optimized product description generation for retail using LangChain 🦜🔗 This notebook demonstrates how to use the LangChain model to generate SEO-optimized product descriptions for retail. The model takes as input a list of product attributes and outputs a short description that highlights the key features of the product. |
|
BigQuery DataFrames Text-generation |
BigQuery DataFrames ML: Drug Name Generation This notebook demonstrates how to use BigQuery DataFrames ML to generate drug names. It uses a pre-trained language model to generate text, and then filters the results to remove drug names that are already in use. |
|
BigQuery DataFrames Code-generation |
Use BigQuery DataFrames with Generative AI for code generation This notebook demonstrates how to use BigQuery DataFrames with Generative AI for code generation. It shows how to use a pre-trained language model to generate code that transforms a BigQuery table into a Pandas DataFrame. |
|
BigQuery Language-model |
Using Vertex AI LLMs with data in BigQuery This notebook demonstrates how to use Vertex AI LLMs with data in BigQuery. It shows how to load data from BigQuery, create an LLM model, and then use the model to generate text based on the data. |
|
Embeddings Similarity Visualization |
Visualizing embedding similarity from text documents using t-SNE plots This notebook demonstrates how to visualize embedding similarity from text documents using t-SNE plots. It uses a dataset of movie reviews from the [IMDB dataset](https://datasets.imdbws.com/). |
|
Text-embeddings Vector Search |
Getting Started with Text Embeddings + Vertex AI Vector Search This notebook provides an introduction to text embeddings and how to use them with Vertex AI Vector Search. It covers the basics of text embeddings, how to train them, and how to use them to perform vector search. |
|
Embeddings Vector Search |
Vertex AI Vector Search Quickstart This notebook is a quickstart for using Vertex AI Vector Search. It covers the basics of vector search, including how to create a vector index, how to upload data to the index, and how to perform vector search queries. |
|
Imagen 3 image generation |
Image Generation with Imagen on Vertex AI In this notebook, you explore the image generation features of Imagen using the Vertex AI SDK for Python and Imagen 3 standard and low-latency models. Learn more about Imagen's image generation feature. |
|
Imagen 2 image generation |
Image Generation with Imagen on Vertex AI In this notebook, you explore the image generation features of Imagen using the Vertex AI SDK for Python. Learn more about Imagen's image generation feature. |
|
Imagen 3 image generation |
Create high quality visual assets with Imagen 3 and Gemini 1.5 Pro In this notebook, you create high quality visual assets for a restaurant menu using Imagen 3 and Gemini 1.5 Pro. Learn more about image generation and multimodal models. |
|
Imagen 2 image editing |
Create high quality visual assets with Imagen 2 edit using automatically generated mask areas In this notebook, you will be exploring the image editing features of Imagen using the Vertex AI SDK for Python. |
|
Imagen image Visual Question Answering (VQA) |
Visual Question Answering (VQA) with Imagen on Vertex AI This notebook demonstrates how to use Imagen to generate images that answer given questions. It also shows how to deploy a model to Vertex AI and use it to generate images in response to user-provided questions. |
|
Imagen image captioning |
Visual captioning with Imagen on Vertex AI This notebook demonstrates how to use Imagen, a large language model for image generation, to generate captions for images. It also shows how to deploy the model on Vertex AI. |
What's next
- Learn about LLMs, Vertex AI, and PaLM models with notebook tutorials.
- Explore more resources in the Generative AI GitHub repo.
- See other Vertex AI notebook tutorials in the Tutorials overview.