Enable customer-managed encryption keys

This document describes how to encrypt Dataplex data with customer-managed encryption keys (CMEK).

Overview

By default, Dataplex encrypts customer content at rest. Dataplex handles encryption for you without any additional actions on your part. This option is called Google default encryption.

If you want to control your encryption keys, then you can use customer-managed encryption keys (CMEKs) in Cloud KMS with CMEK-integrated services including Dataplex. Using Cloud KMS keys gives you control over their protection level, location, rotation schedule, usage and access permissions, and cryptographic boundaries. Using Cloud KMS also lets you track key usage, view audit logs, and control key life cycles. Instead of Google owning and managing the symmetric key encryption keys (KEKs) that protect your data, you control and manage these keys in Cloud KMS.

After you set up your resources with CMEKs, the experience of accessing your Dataplex resources is similar to using Google default encryption. For more information about your encryption options, see Customer-managed encryption keys (CMEK).

Dataplex uses one CMEK per location for all Dataplex resources.

You can configure a CMEK key at the organization level in Dataplex.

For more information about CMEK in general, including when and why to enable it, see Customer-managed encryption keys (CMEKs).

Benefits of CMEK

CMEK lets you do the following:

  • Manage key lifecycle operations and access permissions.
  • Track key usage with the Key Inventory API and Key Usage dashboards in Cloud KMS, which lets you see things like which keys protect which resources. Cloud Logging tells you when the keys were accessed and by whom.
  • Meet specific regulatory requirements by managing your encryption keys.

How CMEK works with Dataplex

Dataplex encryption administrators in your Google Cloud project can configure CMEK for Dataplex by providing the Cloud KMS key. Then, Dataplex uses the specified Cloud KMS key to encrypt all data, including existing data and any new resources created within Dataplex.

Supported features

  • Dataplex supports CMEK encryption for the following features:
    • Data quality and Profile Scan
    • Data discovery
    • Metadata insights
  • Assured Workloads customers can't use other Dataplex features as CMEK encryption is not yet supported for them.
  • Customers who don't use Assured Workloads can use all the features but their data would be encrypted using Google default encryption.

Considerations

  • By default, each organization is provisioned using Google default encryption.
  • The Organizational Administrator can switch to CMEK in Dataplex for any location.
  • Dataplex supports Cloud KMS keys, Cloud HSM keys, and Cloud External Key Manager keys.
  • Key rotation is supported, and after it is available, the new key version is automatically used for data encryption. Existing data is also encrypted with this new version.
  • Dataplex retains data backups for a maximum of 15 days. Any backups created after you enable CMEK are encrypted using the specified KMS key. Data backed up before enabling CMEK remains encrypted with Google's default encryption for a maximum of 15 days.

Limitations

  • Switching to CMEK is an irreversible process. After you opt for CMEK, you cannot revert to Google default encryption.
  • After a Cloud KMS key is configured for Dataplex, it cannot be updated or changed.
  • Dataplex supports organization-level encryption only. As a result, the encryption configuration is set at the organization level for a given location and is used to encrypt Dataplex data for all projects within that organization and location. CMEK encryption is not supported for specific projects under an organization or folder. Setting CMEK-related organization policies requires careful consideration.
  • Dataplex doesn't support CMEK in the global region.
  • After you have enabled CMEK, don't move projects from one organization to another, as this action results in data loss.

Manage your encryption keys

  • Ensure that your CMEK keys remain enabled and accessible. If a key is disabled or destroyed, Dataplex data becomes inaccessible. If the key is unavailable for more than 30 days, data encrypted with that key is automatically deleted and cannot be recovered.
  • If the Cloud KMS key is destroyed and unrecoverable, all associated Dataplex data will be permanently lost.
  • In cases where Cloud KMS is temporarily unavailable, Dataplex continues to support full operations on a best-effort basis for up to one hour. After this period, data will become temporarily inaccessible as a protective measure.
  • When using Cloud EKM, be aware that Google doesn't control the availability of your externally managed keys. Short-term key unavailability results in temporary data inaccessibility. Key unavailability that continues for 30 days results in permanent data loss.

Dataplex availability

The following sections outline the process and expected operational impact when you enable CMEK for your Dataplex organization.

Initial infrastructure provisioning

After you save the encryption configuration, Dataplex sets up the necessary infrastructure. This process typically takes 6 to 8 hours. During this provisioning phase, you retain full access to all Dataplex features and functionalities, and the data remains encrypted through Google-managed encryption. If the organization policy constraints/gcp.restrictNonCmekServices is set, then resource creation requests fail until the provisioning phase is complete.

Data encryption and API availability

After infrastructure provisioning, Dataplex begins encrypting existing data stored within the organization. To ensure data integrity and prevent potential inconsistencies during this encryption process, Dataplex APIs are temporarily unavailable. This restriction prevents data update operations. When you initially activate CMEK for Dataplex, all existing data is encrypted. This one-time operation is estimated to take up to two hours.

Post-encryption operations

After successful completion of the existing data encryption, Dataplex APIs are fully available. Creating or modifying data within Dataplex is automatically encrypted using the configured CMEK, without any operational interruptions or API restrictions.

Create a key and grant permissions

The following instructions explain how to create a key. You can use a key created directly in Cloud KMS or an externally managed key that you make available with Cloud EKM.

  1. In the Google Cloud project where you want to manage your keys, do the following:

    1. Enable the Cloud Key Management Service API.

    2. Create a Cloud KMS key ring in the location where you want to use it.

    3. Create a key using one of the following options:

  2. Create and display the Google-managed service account:

    gcloud beta services identity create \
    --service=dataplex.googleapis.com \
    --organization=ORG_ID

    Replace ORG_ID with the ID of the organization that contains the key.

    If you're prompted to install the Google Cloud CLI beta commands component, enter Y.

    The gcloud CLI services identity command creates or gets the specific Google-managed service account that Dataplex can use to access the Cloud KMS key.

    The service account ID is formatted as service-org-ORG_ID@gcp-sa-dataplex.iam.gserviceaccount.com.

  3. Grant the CryptoKey Encrypter/Decrypter IAM role (roles/cloudkms.cryptoKeyEncrypterDecrypter) to the Dataplex service account. Grant this permission on the key you created.

    Console

    1. Go to the Key management page.

      Go to Key management

    2. Click the key ring.

    3. In the list of available keys, click the key you created.

    4. Click the Permissions tab.

    5. Click Grant access.

    6. In the Grant access pane that opens, follow these steps to grant access to the Dataplex service account:

      1. In Add principals, enter the service account service-org-ORG_ID@gcp-sa-dataplex.iam.gserviceaccount.com.
      2. In Assign roles, select the Cloud KMS CryptoKey Encrypter/Decrypter role.
      3. Click Save.

    gcloud

    Grant the service account the cloudkms.cryptoKeyEncrypterDecrypter role:

    gcloud kms keys add-iam-policy-binding KEY_NAME \
    --location=LOCATION \
    --keyring KEY_RING \
    --project=KEY_PROJECT_ID \
    --member serviceAccount:service-org-ORG_ID@gcp-sa-dataplex.iam.gserviceaccount.com \
    --role roles/cloudkms.cryptoKeyEncrypterDecrypter

    Replace the following:

    • KEY_NAME: the key name
    • LOCATION: the location
    • KEY_RING: the key ring
    • KEY_PROJECT_ID: the key project ID

  4. Assign the Dataplex Encryption Admin role to yourself.

    Console

    Follow the instructions to grant an IAM role.

    gcloud 

    gcloud organizations add-iam-policy-binding ORG_ID \
    --member='user:USER_EMAIL' \
    --role='roles/dataplex.encryptionAdmin'

    Replace the following:

    • ORG_ID: the ID of the organization that contains the key.
    • USER_EMAIL: the email address of the user.

  5. Configure Dataplex to use your CMEK key.

    Console

    1. In the Google Cloud console, go to the Dataplex page.

      Go to Dataplex

    2. Click Settings.

    3. In Select region for CMEK, select a region. The region that you select must match the location of the Cloud KMS key.

    4. In Select encryption key, select the key you created.

    5. Click Save.

      The data encryption process takes a while to complete. When the process completes, the following message appears: Data Encryption is complete. Your selected CMEK key is now protecting your data.

    gcloud

    1. Set the encryption config in Dataplex:

      gcloud alpha dataplex encryption-config create default \
    --location=LOCATION \
    --organization=ORG_ID \
    --key=KEY_RESOURCE_ID

      Replace the following:

      • ORG_ID: the ID of the organization that contains the key.
      • KEY_RESOURCE_ID: the key resource ID - for example, projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY_NAME. Replace PROJECT_ID with the key project ID.

    2. Check that the encryption process is complete:

      gcloud alpha dataplex encryption-config describe default \
    --location=LOCATION \
    --organization=ORG_ID

    The data encryption process takes a while to complete. When the process completes, the following message appears: encryptionState: COMPLETED.

Logging and monitoring

Audit Dataplex requests to Cloud KMS by enabling audit logging for the Cloud KMS API.

CMEK organization policies

Google Cloud provides organization policy constraints to enforce CMEK usage and control the permissible Cloud KMS keys within your organization. These constraints help to ensure that data within Dataplex is consistently protected by CMEK.

  • constraints/gcp.restrictNonCmekServices enforces mandatory CMEK usage for Dataplex resources.

    • Adding dataplex.googleapis.com to the list of Google Cloud service names and setting the constraint to Deny prohibits the creation of Dataplex resources that don't have CMEK protection.

    • If a Cloud KMS key is not specified for the requested location in the CMEK encryption settings, requests to create resources within Dataplex will fail.

    • This policy is validated at the individual resource project level.

  • constraints/gcp.restrictCmekCryptoKeyProjects restricts the selection of Cloud KMS keys for CMEK to designated resource hierarchies.

    • By configuring a list of resource hierarchy indicators (projects, folders, or organizations) and setting the constraint to Allow, Dataplex is restricted to using CMEK keys only from the specified locations.

    • If a Cloud KMS key from a disallowed project is provided, requests to create CMEK-protected resources in Dataplex will fail.

    • This policy is validated at the resource project level during resource creation.

    • This policy is validated at the organization level when configuring CMEK encryption settings.

    • To prevent inconsistencies, make sure that project-level configurations align with organization-wide policies.

For more information about configuring organization policies, see CMEK organization policies.

What's next

  • Learn more about CMEK.