Customer-managed encryption keys

By default, Google Agentspace Enterprise encrypts customer content at rest. Agentspace Enterprise handles encryption for you without any additional actions on your part. This option is called Google default encryption.

If you want to control your encryption keys, then you can use customer-managed encryption keys (CMEKs) in Cloud KMS with CMEK-integrated services including Agentspace Enterprise. Using Cloud KMS keys gives you control over their protection level, location, rotation schedule, usage and access permissions, and cryptographic boundaries. Using Cloud KMS also lets you track key usage, view audit logs, and control key life cycles. Instead of Google owning and managing the symmetric key encryption keys (KEKs) that protect your data, you control and manage these keys in Cloud KMS.

After you set up your resources with CMEKs, the experience of accessing your Agentspace Enterprise resources is similar to using Google default encryption. For more information about your encryption options, see Customer-managed encryption keys (CMEK).

Limitations of Cloud KMS in Agentspace Enterprise

The following limitations apply to CMEK (Cloud KMS) keys in Agentspace Enterprise:

  • Keys that are already applied to a data store can't be changed.

  • After a key has been registered, it can't be deregistered or removed from a data store.

  • You must use US or EU multi-region data stores and apps (not global ones). For more information about multi-regions and data residency, including limits associated with using non-global locations, see Agentspace Enterprise locations.

  • If you need to register more than one key for a project, contact your Google account team to request a quota increase for CMEK configurations, providing a justification for why you need more than one key.

  • Using external key manager (EKM) or hardware security module (HSM) with CMEK is in GA with allowlist. To use EKM or HSM with CMEK, contact your Google account team.

    The following limitations apply to EKM or HSM with CMEK:

    • Your EKM and HSM quota for encrypt and decrypt calls should have at least 1,000 QPM of headroom. For how to check your quotas, see Check your Cloud KMS quotas.

    • If using EKM, the key must be reachable for more than 90% of any time window of longer than 30 seconds. If the key isn't reachable for this amount of time, it can negatively impact indexing and search freshness.

    • If there are billing issues, persistent out-of-quota issues, or persistent unreachability issues for more than 12 hours, the service automatically turns down the CmekConfig associated with the EKM or HSM key.

  • Data stores created before a key is registered to the project can't be protected by the key.

  • You can't tune search models for data stores that are protected by CMEK keys.

  • Except for the "import-once" data stores for BigQuery and Cloud Storage, first-party connector are not CMEK-compliant.

  • Key rotation is not supported for the single-region keys for the third-party connectors. If you disable or destroy a key version that protects a data store associated with a third-party connector, the connector stops working.

  • Key rotation is not compatible with analytics. If you rotate keys for a data store, apps that use that data store no longer display analytics.

About single-region keys for third-party connectors

If you use third-party connectors and want to use your own keys to protect the connected data, then you need to create three supplemental single-region keys in addition to the multi-region key. The commands for creating keys are given in the following procedure, Register your Cloud KMS key.

The single keys need to be created for the following regions:

Multi-region Single-regions
eu europe-west1 europe-west4 europe-north1
us us-east1 us-central1 us-west1

Before you begin

Make sure you satisfy the following prerequisites:

  • A symmetric Cloud KMS key with the rotation period set to Never (Manual rotation). See Create a key ring and Create a key in the Cloud KMS documentation.

  • The CryptoKey Encrypter/Decrypter IAM role (roles/cloudkms.cryptoKeyEncrypterDecrypter) on the key has been granted to the Discovery Engine service agent. For general instructions on how to add a role to a service agent, see Grant or revoke a single role.

  • The CryptoKey Encrypter/Decrypter IAM role (roles/cloudkms.cryptoKeyEncrypterDecrypter) on the key has been granted to the Cloud Storage service agent. If this role is not granted, data import for CMEK-protected data stores will fail because Discovery Engine is not able to make the CMEK-protected, temporary bucket and directory that is required for importing.

  • Don't create any data stores or apps that you want managed by your key until after you have completed the key registration instructions on this page.

Register your Cloud KMS key

To register your own key for Agentspace Enterprise, follow these steps:

  1. Call the UpdateCmekConfig method with the Cloud KMS key that you want to register.

    curl -X PATCH \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-d '{"kms_key":"projects/KMS_PROJECT_ID/locations/KMS_LOCATION/keyRings/KEY_RING/cryptoKeys/KEY_NAME"}' \
"https://LOCATION-discoveryengine.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/cmekConfigs/CMEK_CONFIG_ID?set_default=SET_DEFAULT"

    Replace the following:

    • KMS_PROJECT_ID: the ID of your project that contains the key. The project number won't work.
    • KMS_LOCATION: the multi-region of your KMS key: us or europe.
    • KEY_RING: the name of the key ring that holds the key.
    • KEY_NAME: the name of the key.
    • PROJECT_ID: the ID of your project that contains the data store.
    • LOCATION: the multi-region of your data store: us or eu.
    • CMEK_CONFIG_ID: the ID of the CmekConfig resource.
    • SET_DEFAULT: set to true to use the key as the default key for subsequent data stores created in the multi-region.

  2. Optional: Record the name value returned by the method and follow the instructions in Get details about a long-running operation to see when the operation is complete.

    It typically takes a few minutes to register a key. After the operation completes, new data stores in that multi-region are protected by the key. For general information about creating data stores, see About apps and data stores.

  3. If you use third-party connectors and want to protect your third-party data with your own key, then create three additional single-region keys as follows:

    curl -X PATCH \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-d '{"single_region_keys": [ \
     {"kms_key": "projects/KMS_PROJECT_ID/locations/REGION_1/keyRings/KEY_RING/cryptoKeys/KEY_NAME"}, \
     {"kms_key": "projects/KMS_PROJECT_ID/locations/REGION_2/keyRings/KEY_RING/cryptoKeys/KEY_NAME"}, \
     {"kms_key": "projects/KMS_PROJECT_ID/locations/REGION_3/keyRings/KEY_RING/cryptoKeys/KEY_NAME"}] \
}' \
"https://LOCATION-discoveryengine.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/cmekConfigs/CMEK_CONFIG_ID?set_default=SET_DEFAULT"

Replace the following:

  • REGION_1: for the eu multi-region, set this to europe-west1, and for the us multi-region, set this to us-east1.

  • REGION_2: for the eu multi-region, set this to europe-west4, and for the us multi-region, set this to us-central1.

  • REGION_3: for the eu multi-region, set this to europe-north1, and for the us multi-region, set this to us-west1.

    After the operation completes, new data connectors in that multi-region are protected by those keys.

View Cloud KMS keys

To view a registered key for Agentspace Enterprise, do one of the following:

  • If you have the CmekConfig resource name, call the GetCmekConfig method:

    curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://LOCATION-discoveryengine.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/cmekConfigs/CMEK_CONFIG_ID"

    Replace the following:

    • LOCATION: the multi-region of your data store: us or eu.
    • PROJECT_ID: the ID of your project that contains the data
    • CMEK_CONFIG_ID: the ID of the CmekConfig resource.

    An example curl call and response looks like this: 

    $ curl -X GET
-H "Authorization: Bearer $(gcloud auth print-access-token)"
"https://us-discoveryengine.googleapis.com/v1/projects/my-ai-app-project-123/locations/us/cmekConfigs/cmek-config-1"

    
{
  "name": "projects/my-ai-app-project-123/locations/us/cmekConfigs/cmek-config-1",
  "kms_key": "projects/key-project-456/locations/us/keyRings/my-key-ring/cryptoKeys/my-key"
  "state": "ACTIVE"
  "is_default": true
}

  • If you don't have the CmekConfig resource name, call the ListCmekConfigs method:

    curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://LOCATION-discoveryengine.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/cmekConfigs"

    Replace the following:

    • LOCATION: the multi-region of your data store: us or eu.
    • PROJECT_ID: the ID of your project that contains the data

    An example curl call and response looks like this: 

    $ curl -X GET
-H "Authorization: Bearer $(gcloud auth print-access-token)"
"https://us-discoveryengine.googleapis.com/v1/projects/my-ai-app-project-123/locations/us/cmekConfigs"

    
{
  "cmek_configs": [
    {
      "name": "projects/my-ai-app-project-123/locations/us/cmekConfigs/cmek-config-1",
      "kms_key": "projects/key-project-456/locations/us/keyRings/my-key-ring/cryptoKeys/my-key"
      "state": "ACTIVE"
      "is_default": true
    }
    {
      "name": "projects/my-ai-app-project-123/locations/us/cmekConfigs/cmek-config-2",
      "kms_key": "projects/key-project-456/locations/us/keyRings/my-key-ring/cryptoKeys/my-key-2"
      "state": "ACTIVE"
    }
  ]
}

Verify that a data store is protected by a key

Data stores that were created before your key was registered are not protected by the key. If you want to confirm that a particular data store is protected by your key, follow these steps:

  1. Run the following curl command on the data store:

    curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-H "x-goog-user-project: PROJECT_ID" \
"https://LOCATION-discoveryengine.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/collections/default_collection/dataStores/DATA_STORE_ID"

    Replace the following:

    • LOCATION: the multi-region of your data store: us or eu.
    • PROJECT_ID: the ID of your project that contains the data store.
    • DATA_STORE_ID: the ID of the data store.

    An example curl call looks like this: 

    curl -X GET
-H "Authorization: Bearer $(gcloud auth print-access-token)"
-H "Content-Type: application/json"
-H "x-goog-user-project: my-ai-app-project-123"
"https://us-discoveryengine.googleapis.com/v1/projects/my-ai-app-project-123/locations/us/collections/default_collection/dataStores/my-data-store-1"

  2. Review the output from the command: If the cmekConfig field is in the output and the kmsKey field shows the key that you registered, then the data store is protected by the key.

    An example response looks like this:

    {
 "name": "projects/969795412903/locations/us/collections/default_collection/dataStores/my-data-store-1",
 "displayName": "my-data-store-1",
 "industryVertical": "GENERIC",
 "createTime": "2023-09-05T21:20:21.520552Z",
 "solutionTypes": [
   "SOLUTION_TYPE_SEARCH"
 ],
 "defaultSchemaId": "default_schema",
 "cmekConfig": {
   "name": "projects/969795412903/locations/us/collections/default_collection/dataStores/my-data-store-1/cmekConfigs/cmek-config-1",
   "kmsKey": "projects/my-ai-app-project-123/locations/us/keyRings/my-key-ring/cryptoKeys/my-key"
 }
}

Rotate keys

When you rotate keys, you are creating a new version of the key and setting the new version as the primary version. Leave the original version of the key enabled for a while before disabling it. This gives any long-running operations that might be using the older key time to complete.

The following procedure outlines the steps to rotate keys for a Agentspace Enterprise data store. For general information about rotating keys, see Key rotation in the Cloud KMS guide.

Important: Do not rotate keys on data stores associated with recommendations apps or with any apps that need analytics, and don't rotate supplemental keys used for third-party connectors. See Limitations of Cloud KMS in Agentspace Enterprise.

  1. Reregister your key. Do this by repeating step 1 of Register your Cloud KMS key.

  2. See the instructions in the Manage keys section of the Cloud KMS guide to do the following:

    1. Create a new key version, enable it, and make it primary.

    2. Leave the older key version enabled.

    3. After a week or so, disable the older key version and make sure that everything is working as before.

    4. At some later date, when you are certain that no problems were caused by disabling the older key version, you can destroy the older key version.

If a key is disabled or revoked

If a key is disabled or permissions for the key are revoked, the data store stops ingesting data and stops serving data within 15 minutes. However, re-enabling a key or restoring permissions takes a long time. It can take up to 24 hours before the data store can resume serving data.

Therefore, don't disable a key unless necessary. Disabling and enabling a key on a data store is a time-consuming operation. For example, repeatedly switching a key between disabled and enabled means it will take a long time for the data store to reach a protected state. Disabling a key and re-enabling it immediately afterward could result in days of downtime because the key is first disabled from the data store and subsequently re-enabled.