Configure access controls for custom data sources

This page describes how to enforce data source access control (also referred to as ACLs) for search apps you create using Cloud Storage or BigQuery.

Overview

Access control for your data sources in Agentspace limits the data that users can view in your search app's results. Google uses your identity provider to identify the end user performing a search and determine if they have access to the documents that are returned as results.

For example, say that employees at your company search across BigQuery documents using your search app. However, you need to make sure they can't view content through the app that they aren't allowed to access. If you have set up a workforce pool in Google Cloud for your organization's identity provider, then you can also specify that workforce pool in Agentspace. Now, if an employee uses your app, they get search results only for documents that their account already has access to in BigQuery.

Turning on access control is a one-time procedure. To apply access control to a BigQuery or Cloud Storage data source, use the following steps depending on your data type, such as structured or unstructured data.

Unstructured data from Cloud Storage
Structured data from Cloud Storage
Unstructured data from BigQuery
Structured data from BigQuery

Unstructured data from Cloud Storage

When setting up a data store for unstructured data from Cloud Storage, you need to also upload ACL metadata and set the data store as access controlled:

When preparing your data, include ACL information in your metadata using the acl_info field. For example:

{
   "id": "<your-id>",
   "jsonData": "<JSON string>",
   "content": {
     "mimeType": "<application/pdf or text/html>",
     "uri": "gs://<your-gcs-bucket>/directory/filename.pdf"
   },
   "acl_info": {
     "readers": [
       {
         "principals": [
           { "group_id": "group_1" },
           { "user_id": "user_1" }
         ]
       }
     ]
   }
 }

For more information about unstructured data with metadata, see the Unstructured data section of Prepare data for ingesting.

When following the steps for data store creation in Create a first-party data store, you can enable access control by doing the following in either the console or using the API:
- Console: When creating a data store, select This data store contains access control information during data store creation.
- API: When creating data store, include the field "aclEnabled": "true" in your JSON payload.
When following the steps for data import in Create a first-party data store, make sure to do the following:
- Upload your metadata with ACL information from the same bucket as your unstructured data
- If using the API, set GcsSource.dataSchema to document

Structured data from Cloud Storage

When setting up a data store for structured data from Cloud Storage, you need to also upload ACL metadata and set the data store as access controlled:

When preparing your data, include ACL information in your metadata using the acl_info field. For example:

{
   "id": "<your-id>",
   "jsonData": "<JSON string>",
   "acl_info": {
     "readers": [
       {
         "principals": [
           { "group_id": "group_1" },
           { "user_id": "user_1" }
         ]
       }
     ]
   }
 }

When following the steps for data store creation in Create a first-party data store, you can enable access control by doing the following in either the console or using the API:
- Console: When creating a data store, select This data store contains access control information during data store creation.
- API: When creating data store, include the field "aclEnabled": "true" in your JSON payload.
When following the steps for data import in Create a first-party data store, make sure to do the following:
- Upload your metadata with ACL information from the same bucket as your structured data
- If using the API, set GcsSource.dataSchema to document

Unstructured data from BigQuery

When setting up a data store for unstructured data from BigQuery, you need to set the data store as access controlled and provide ACL metadata using a predefined schema for Agentspace:

When preparing your data, specify the following schema. Don't use a custom schema.

[
  {
    "name": "id",
    "mode": "REQUIRED",
    "type": "STRING",
    "fields": []
  },
  {
    "name": "jsonData",
    "mode": "NULLABLE",
    "type": "STRING",
    "fields": []
  },
  {
    "name": "content",
    "type": "RECORD",
    "mode": "NULLABLE",
    "fields": [
      {
        "name": "mimeType",
        "type": "STRING",
        "mode": "NULLABLE"
      },
      {
        "name": "uri",
        "type": "STRING",
        "mode": "NULLABLE"
      }
    ]
  },
  {
    "name": "acl_info",
    "type": "RECORD",
    "mode": "NULLABLE",
    "fields": [
      {
        "name": "readers",
        "type": "RECORD",
        "mode": "REPEATED",
        "fields": [
          {
            "name": "principals",
            "type": "RECORD",
            "mode": "REPEATED",
            "fields": [
              {
                "name": "user_id",
                "type": "STRING",
                "mode": "NULLABLE"
              },
              {
                "name": "group_id",
                "type": "STRING",
                "mode": "NULLABLE"
              }
            ]
          }
        ]
      }
    ]
  }
]

Include your ACL metadata as a column in your BigQuery table.
When following the steps in Create a first-party data store, enable access control in either the console or using the API:
- Console: When creating a data store, select This data store contains access control information during data store creation.
- API: When creating data store, include the field "aclEnabled": "true" in your JSON payload.
When following the steps for data import in Create a first-party data store, if using the API, set BigQuerySource.dataSchema to document.

Structured data from BigQuery

When setting up a data store for structured data from BigQuery, you need to set the data store as access controlled and provide ACL metadata using a predefined schema for Agentspace: