Grounding with Google Maps in Vertex AI

This page describes how Grounding with Google Maps in Vertex AI can help to enhance your generative AI applications by providing geospatial context.

Overview

Grounding with Google Maps in Vertex AI is a Preview service that connects Gemini models with geospatial data from Google Maps. Google Maps has access to information on millions of locations, including businesses, landmarks, and points of interest. This data gives you access to information on over 250 million places that can be used to ground your model's responses, enabling your AI applications and agents to provide local data and geospatial context.

Uses of Grounding with Google Maps

You can use Grounding with Google Maps for various applications, such as:

  • Chat-based agents.
  • Summarization of place information.
  • Content translation.
  • Conversational assistants that can answer questions about nearby places, such as "Are there any parks nearby?".
  • Personalized descriptions of places, such as "Can you tell me more about the parks and any family-friendly restaurants that are within a walkable distance?"

This can be useful for industries like real estate, travel, mobility, and social applications.

For technical support with Grounding with Google Maps, email maps-grounding-feedback-external@google.com.

Supported models

This section lists the models that support grounding with Google Maps.

For more information about the Gemini models, see Gemini models.

Prerequisites

Grounding with Google Maps in Vertex AI checks your project's billing account country during each request and won't accept requests from personal or service accounts with billing addresses in the European Economic Area (EEA). To perform the check, Grounding with Google Maps in Vertex AI must have IAM permissions to call the Vertex AI API.

To allow Google Cloud to automatically check your account's billing address, make sure your account has the following predefined IAM roles:

  • resourcemanager.projects.get: Gets the project.
  • billing.accounts.get: Gets the linked billing account. For instructions on how to find the billing account ID, see Find a Cloud Billing account ID.

Add permissions

Choose Google Cloud CLI or Google Cloud console to add your permissions:

Google Cloud CLI

The following Google Cloud CLI commands add the necessary permissions for each predefined role:

  • resourcemanager.projects.get: Gets the project.
gcloud projects add-iam-policy-binding ${PROJECT_ID?} --member="serviceAccount:${SERVICE_ACCOUNT?}" --role=roles/viewer
  • billing.accounts.get: Gets the linked billing account.
gcloud billing accounts add-iam-policy-binding ${BILLING_ACCOUNT_ID?} --member="serviceAccount:${SERVICE_ACCOUNT?}" --role=roles/billing.viewer

Google Cloud console

For information on how to add permissions using the Google Cloud console, see Grant a single IAM role.

Test permissions using the Google Cloud CLI

The following Google Cloud CLI commands test whether a service account has the necessary permissions for each predefined role.

  • resourcemanager.projects.get

    gcloud policy-troubleshoot iam "//cloudresourcemanager.googleapis.com/projects/${PROJECT_ID?}" --principal-email="${SERVICE_ACCOUNT?}" --permission="resourcemanager.projects.get"
    
  • billing.accounts.get

    gcloud policy-troubleshoot iam "//cloudbilling.googleapis.com/billingAccounts/${BILLING_ACCOUNT_ID?}" --principal-email="${SERVICE_ACCOUNT?}" --permission="billing.accounts.get"
    

The commands print access: GRANTED on success or a relevant AccessState in other cases.

For more information on testing permissions, see the Policy Troubleshooter.

Use Grounding with Google Maps to ground your model's responses

This code sample demonstrates how to use Grounding with Google Maps to ground your model's responses.

Console

To use Grounding with Google Maps with Vertex AI Studio, follow these steps:

  1. In the Google Cloud console, go to the Vertex AI Studio page.

    Go to Vertex AI Studio

  2. Click the Freeform tab.
  3. In the side panel, click the Ground model responses toggle.
  4. Click Customize, and set Google Maps as the source.
  5. Enter your prompt in the text box, and click Submit.

Your prompt responses ground to Google Maps.

REST

Before using any of the request data, make the following replacements:

  • LOCATION: The region to process the request.
  • PROJECT_ID: Your project ID.
  • MODEL_ID: The model ID of the multimodal model.
  • TEXT: The text instructions to include in the prompt.
  • LATITUDE: The latitude of the location.
  • LONGITUDE: The longitude of the location.

HTTP method and URL:

POST https://LOCATION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/LOCATION/publishers/google/models/MODEL_ID:generateContent

Request JSON body:

{
  "contents": [{
    "role": "user",
    "parts": [{
      "text": "TEXT"
    }]
  }],
  "tools": [{
    "googleMaps": {}
  }],
  "toolConfig": {
    "retrievalConfig": {
      "latLng": {
        "latitude": LATITUDE,
        "longitude": LONGITUDE
      }
    }
  },
  "model": "projects/PROJECT_ID/locations/LOCATION/publishers/google/models/MODEL_ID"
}

To send your request, expand one of these options:

You should receive a JSON response similar to the following:

{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": "\"The Italian Place\" in Alexandria, VA, is good for children and offers takeout. It has a rating of 4.2 stars based on 411 reviews."
          }
        ]
      },
      "finishReason": "STOP",
      "groundingMetadata": {
        "groundingChunks": [
          {
            "maps": {
              "uri": "https://maps.google.com/?cid=9001322937822692826",
              "title": "The Italian Place",
              "text": "**About:**\n\n* **Type:** Italian Restaurant\n* **Address:** 621 Wythe St, Alexandria, VA 22314, USA\n* **Open Now:** Yes\n* **Rating:** 4.2 (411 reviews)\n* **Price Level:** Moderate\n* **Phone:** (571) 777-8981\n* **Summary:** Down-to-earth, counter-serve stop offering Italian sandwiches, coffee & market goods.\n* **Additional Summary:** Relaxed Italian eatery known for sandwiches and pizza along with gourmet food items and gelato.\n* **Offers Takeout:** Yes\n* **Offers Delivery:** Yes\n* **Offers Dine-in:** Yes\n* **Good for Children:** Yes\n* **Has Restroom:** Yes\n* **Outdoor Seating:** Yes\n* **Live Music:** No\n* **Menu for Children:** No\n* **Serves Dessert:** Yes\n* **Serves Coffee:** Yes\n* **Good for Watching Sports:** No\n* **Serves Lunch:** Yes\n* **Serves Dinner:** Yes\n* **Serves Beer:** Yes\n* **Serves Vegetarian Food:** Yes\n\n**Opening Hours (local time):**\n\n* Monday: 11:00 AM – 7:00 PM\n* Tuesday: 10:00 AM – 7:00 PM\n* Wednesday: 11:00 AM – 7:00 PM\n* Thursday: 11:00 AM – 7:00 PM\n* Friday: 11:00 AM – 7:00 PM\n* Saturday: 11:00 AM – 7:00 PM\n* Sunday: 12:00 – 7:00 PM\n\n**Parking options:**\n\n* **Free parking lot:** Yes\n* **Free street parking:** Yes\n* **Valet parking:** No\n\n**Accessibility:**\n\n* **Wheelchair accessible parking:** Yes\n* **Wheelchair accessible restroom:** Yes\n\n**Payment options:**\n\n* **Credit Card:** Yes\n* **Debit Card:** Yes\n* **Cash Only:** No\n* **NFC:** Yes\n\n**Distance & Travel Time:**\n\n* 384.6 kilometers\n* 4.2 hours",
              "placeId": "places/ChIJOTRDf_qwt4kR2kV_WYUf63w"
            }
          },
          ...
        ],
        "groundingSupports": [
          {
            "segment": {
              "endIndex": 79,
              "text": "\"The Italian Place\" in Alexandria, VA, is good for children and offers takeout."
            },
            "groundingChunkIndices": [
              0
            ]
          },
          {
            "segment": {
              "startIndex": 80,
              "endIndex": 130,
              "text": "It has a rating of 4.2 stars based on 411 reviews."
            },
            "groundingChunkIndices": [
              0
            ]
          }
        ],
        "googleMapsWidgetContextToken": "widgetcontent/..."
      }
    }
  ],
  ...
}

Optional: Google Maps contextual widget

The contextual widget is a Google Maps Pre-GA Offering that's a visual container used to support or supplement other Google Maps content. The Google Maps contextual widget lets you integrate Grounding with Google Maps into your applications to create a conversational LLM-powered chat experience. The contextual widget is rendered using the context token, googleMapsWidgetContextToken, which is returned in the Vertex AI API response and can be used to render visual content.

The contextual widget serves different functions depending on your scenario:

  • It displays subjective user-generated content (UGC) in the scenario where Google Maps prompting is used for answer generation.

  • It helps to enrich results with map visualizations and data when Vertex AI generates just a text response.

For more information on the contextual widget, see Maps grounding widget.

Render the Google Maps contextual widget

To render and use the Google Maps contextual widget, use the alpha version of the Google Maps JavaScript API on the page that displays the widget. For more information, see Load the Maps JavaScript API.

The following code samples demonstrate how to use a contextual widget:

HTML

  1. Create a contextual widget.

      <body>
        <gmp-place-contextual id="widget"></gmp-place-contextual>
      </body>
    
  2. In any response that is grounded with Google Maps, there's a corresponding googleMapsWidgetContextToken that's used to render the contextual widget and placed in close proximity to the generated response.

    To update the context token, set the widget.contextToken property.

    "googleMapsWidgetContextToken": "widgetcontent/AcBXPQdpWQWbap9H-OH8sEKmOXxmEKAYvff0tvthhneMQC3VrqWCjpnPBl4-Id98FGiA_S_t8aeAeJj0T6JkWFX56Bil8oBSR0W8JH3C_RSYLbTjxKdpxc9yNn6JcZTtolIRZon9xi6WpNGuSyjcIxWu2S0hwpasNOpUlWrG1RxVCB4WD1fsz_pwR236mG36lMxevXTQ_JnfdYNuQwQ4Lc3vn...<snip>...
    Ts5VJE_b3IC5eE_6wez0nh61r7foTUZpP7BXMwxR-7Wyfcj6x1v6mIWsFGr1o0p_HSAMNqWPg-aFVnkPLhAkOR6MaNZOfezTva-gxHlu7z_haFvYxcUE1qfNVQ",
    
    function updateWidget(contextToken) {
      let widget = document.querySelector('#widget');
      widget.contextToken = contextToken;
    }
    
  3. Optional: Specify the list layout. Valid values include the following:

    • Compact layout: <gmp-place-contextual-list-config layout="compact">
    • Vertical layout: <gmp-place-contextual-list-config layout="vertical">

    This code sample demonstrates changing the list layout to a compact layout.

        <gmp-place-contextual id="widget">
          <gmp-place-contextual-list-config layout="compact">
          </gmp-place-contextual-list-config>
        </gmp-place-contextual>
    
  4. Optional: Change the map mode. Valid values include the following:

    • 2D roadmap map: map-mode="roadmap"
    • 3D hybrid map: map-mode="hybrid"
    • No map: map-mode="none"

    This code sample demonstrates changing the map mode to a 2D map.

        <gmp-place-contextual id="widget">
          <gmp-place-contextual-list-config map-mode="roadmap">
          </gmp-place-contextual-list-config>
        </gmp-place-contextual>
    

JavaScript

  1. Create a contextual widget.

    async function createWidget(contextToken) {
      await google.maps.importLibrary('places');
      let widgetContainer = document.querySelector('#wc');  // a div that contains the widget
      const placeContextualElement = new
          google.maps.places.PlaceContextualElement({ contextToken });
      widgetContainer.appendChild(placeContextualElement);
    }
    
  2. In any response that is grounded with Google Maps, there's a corresponding googleMapsWidgetContextToken that's used to render the contextual widget and placed in close proximity to the generated response.

    To update the context token, set the widget.contextToken property.

      "googleMapsWidgetContextToken": "widgetcontent/AcBXPQdpWQWbap9H-OH8sEKmOXxmEKAYvff0tvthhneMQC3VrqWCjpnPBl4-Id98FGiA_S_t8aeAeJj0T6JkWFX56Bil8oBSR0W8JH3C_RSYLbTjxKdpxc9yNn6JcZTtolIRZon9xi6WpNGuSyjcIxWu2S0hwpasNOpUlWrG1RxVCB4WD1fsz_pwR236mG36lMxevXTQ_JnfdYNuQwQ4Lc3vn...<snip>...
      Ts5VJE_b3IC5eE_6wez0nh61r7foTUZpP7BXMwxR-7Wyfcj6x1v6mIWsFGr1o0p_HSAMNqWPg-aFVnkPLhAkOR6MaNZOfezTva-gxHlu7z_haFvYxcUE1qfNVQ",
    
      function updateWidget(contextToken) {
        widget.contextToken = contextToken;
      }
    
  3. Optional: Specify the list layout. Valid values include the following:

    • Compact layout: layout: google.maps.places.PlaceContextualListLayout.COMPACT
    • Vertical layout: layout: google.maps.places.PlaceContextualListLayout.VERTICAL

    This code sample demonstrates changing the list layout to a compact layout.

      const widgetConfig = new google.maps.places.PlaceContextualListConfigElement({
        layout: google.maps.places.PlaceContextualListLayout.COMPACT
      });
      widget.appendChild(widgetConfig);
    
  4. Optional: Change the map mode. Valid values include the following:

    • 2D roadmap map: mapMode: google.maps.places.PlaceContextualListMapMode.ROADMAP
    • 3D hybrid map: mapMode: google.maps.places.PlaceContextualListMapMode.HYBRID
    • No map: mapMode: google.maps.places.PlaceContextualListMapMode.NONE

    This code sample demonstrates changing the map mode to a 2D map.

      const widgetConfig = new google.maps.places.PlaceContextualListConfigElement({
        mapMode: google.maps.places.PlaceContextualListMapMode.ROADMAP
      });
      widget.appendChild(widgetConfig);
    

Source attribution requirements

When presenting Vertex AI-generated content that directly references information made available using Grounding with Google Maps, you must specify the Google Maps sources that were used to support the response.

This image shows Google Maps sources that were used to support the model's response.

Prompt with response showing sources

Inform the end-user about the use of grounding sources

You must inform your users about the following:

  • The grounding sources that were used to support the LLM-generated content within close proximity to the content.
  • The grounding sources must be viewable within one user interaction.

Display Google Maps source URLs

Google Maps sources are returned within the groundingMetadata within groundingChunks and groundingSupports. Google Maps sources are returned for both Places and for supporting Place answer content like user reviews, which have been used to help generate the response.

This code sample demonstrates a Place source and a Place answer source in the response:

  "groundingChunks": [
            {
              "maps": {
                "uri": "{Link to Maps Content}",
                "title": "{Name of Maps Place}",
                "text": "{Maps content that was sent to the model for this place}"
                "placeId": "{Place ID}",
                "placeAnswerSources":
                                  {
                    "review": "",
                    "authorAttribution": {
                      "displayName": "",
                      "photoUri": ""
                    },
                    "flagContentUri": "",
                    "googleMapsUri": ""
                  },
              },
               "flagContentUri": ""
              }
            }
          }
        ],

For each source referenced by the LLM, a link preview must be generated following these requirements:

  • Attribute each source to Google Maps following the Text attribution guidelines.
  • Display the Open Graph page title (og:title) or the title with the format:

    [Place Name] - Google Maps

  • Navigate to the source material using the source URL.

If there are Place answer sources in groundingChunks, you must do the following:

  • Render review links inside grounding_chunks.maps.placeAnswerSources.reviewSnippets.googleMapsUri.
  • Display the Open Graph Title or a title with the format:

    Google Review of [Place Name] by [Author Name].

    Place Name can be found in grounding_chunks.maps.title and Author Name can be found in grounding_chunks.maps.placeAnswerSources.reviewSnippets.authorAttribution.displayName.
  • Optional: Enhance the link preview with additional content, such as:

    • Google Maps favicon (<link rel="icon" href="https://www.google.com/favicon.ico">) inserted before the Google Maps text attribution.
    • Description (og:description)
    • Photo (og:image)

These images show the minimum requirements, which is to show the Place links.

Prompt with response showing sources

You can collapse the view of the Sources.

Prompt with response and sources collapsed

This image shows the photo of the place, which is an optional link to preview attributes of the sources.

Prompt with response and sources

Prohibited territories

Google Maps restricts certain content and activities to maintain a safe and reliable platform. For a list of prohibited territories, see Google Maps Platform Prohibited Territories.

Place properties

This section lists place properties that are used to describe locations and used by Grounding with Google Maps to generate responses. These properties are used to determine the types of questions that Grounding with Google Maps can answer.

Sample place properties

This list provides an alphabetized sampling of properties about places that can be used by your model to generate responses.

  • Address
  • Curbside pickup
  • Debit card
  • Distance
  • Free parking lot
  • Live music
  • Menu for children
  • Opening hours
  • Payment options (such as cash or credit card)
  • Place answer
  • Pet friendly
  • Serves beer
  • Serves vegetarian food
  • Wheelchair accessible
  • Wifi

Place answers are a response from Grounding with Google Maps based on information derived from user reviews. If there is an issue with the content of the Place answer provided in the metadata, you can report it to Google using a link in the flagContentUri field inside the PlaceAnswerSources object in the API response.

Examples of using place properties

The following examples use place properties in questions about different types of places. Grounding with Google Maps uses the properties to understand your intent and then provides relevant answers based on the data associated with places in Google Maps.

  • Plan a family dinner: You might ask, Is "The Italian Place" good for children, and do they offer takeout? What is their rating?

    Answers to these questions help you to determine if a restaurant is suitable for a family and if the restaurant offers a convenient service.

  • Check accessibility for a friend: You might ask, I need a restaurant that has a wheelchair accessible entrance.

    A response to this prompt might help you to determine if the location meets specific accessibility needs.

  • Find a location for a late-night snack: You might ask, Is "Burger Joint" open now? Do they serve dinner? What are their opening hours for Friday?

    Answers to these questions help you to find an open establishment serving a specific meal during a particular time.

  • Meet a client for coffee: You might ask, Does "Cafe Central" have Wifi? Do they serve coffee? What is their price level, and do they accept credit cards?

    Answers to these questions help you to assess the suitability of a cafe for a business meeting based on amenities, offerings, and payment options.

What's next

  • To learn more about how to ground Gemini models to your data, see Grounding with your data.
  • To learn more about responsible AI best practices and Vertex AI's safety filters, see Responsible AI.