This document describes how to configure and manage customer-managed encryption keys (CMEK) for Cloud Logging to meet your organization's compliance needs. You can configure CMEK as a default resource setting for an organization, a folder, or for both. When configured, Cloud Logging ensures that all new log buckets in the organization or folder are encrypted with a customer-managed key.
Overview
By default, Cloud Logging encrypts customer content stored at rest. Data stored in log buckets by Logging is encrypted using key-encryption keys, a process known as envelope encryption. Access to your logging data requires access to those key-encryption keys, which Google manages for you without any actions on your part.
Your organization might have regulatory, compliance-related, or advanced encryption requirements that our default encryption at rest doesn't provide. To meet your organization's requirements, instead of Google managing the encryption keys that protect your data, you can configure CMEK to control and manage your own encryption.
For specific information about CMEK, including its advantages, and limitations, see Customer-managed encryption keys.
When you configure CMEK as a default resource setting for Logging, new log buckets in the organization or folder are automatically encrypted with the configured key. However, you can change that key or create log buckets and specify a different key. For more information, see Configure CMEK for log buckets.
Prerequisites
To get started, complete the following steps:
There are some limitations when using CMEK. Before you create a log bucket with CMEK enabled, review the Limitations.
Install the Google Cloud CLI, then initialize it by running the following command:
gcloud init
This guide provides instructions using the Google Cloud CLI.
Find the identifiers for the organization or folder for which you want to enable CMEK:
- ORGANIZATION_ID is the unique numeric identifier of the Google Cloud organization for which you are enabling CMEK. You don't need this value if you only plan to configure CMEK for a folder. For information about getting this identifier, see Getting your organization ID.
- FOLDER_ID is the unique numeric identifier of the Google Cloud folder for which you are enabling CMEK. You don't need this value if you only plan to configure CMEK for an organization. For information about using folders, see Creating and managing folders.
-
To get the permissions that you need to create keys, ask your administrator to grant you the Cloud KMS Admin (
roles/cloudkms.admin
) IAM role on the project or a parent resource. For more information about granting roles, see Manage access to projects, folders, and organizations.You might also be able to get the required permissions through custom roles or other predefined roles.
Ensure that you have the following Cloud Logging permissions for the organization:
logging.settings.get
logging.settings.update
Enable the Cloud KMS API for the Google Cloud project that runs Cloud KMS.
Create a key ring and keys for the Google Cloud project that runs Cloud KMS.
Cloud Logging lets you use a key from any region. However, when you create a log bucket, the location of the log bucket must match the location of the key. For information about supported regions, see the following:
If you configure CMEK as a default resource setting for Logging by using the steps in this document, then new log buckets created in the organization or folder are automatically configured for CMEK. Also, because the location of a log bucket must match the location of the key, after you configure CMEK as a default resource setting, you can't create log buckets in the
global
region.Identify values for the following Cloud KMS parameters:
- KMS_PROJECT_ID is the unique alphanumeric identifier, composed of your Google Cloud project name and a randomly assigned number, of the Google Cloud project running Cloud KMS. For information about getting this identifier, see Identifying projects.
- KMS_KEY_LOCATION is the Cloud KMS key's region.
- KMS_KEY_RING is the Cloud KMS key ring's name.
- KMS_KEY_NAME with the Cloud KMS key's name.
It is formatted like this:
projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEYRING/cryptoKeys/KEY
Enable CMEK for an organization or folder
After you've completed the prerequisite steps, follow these instructions to enable CMEK for your Google Cloud organization.
Determine the service account ID
To determine the service-account ID associated with the organization or folder
for which CMEK will apply, run the following
gcloud logging settings describe
command:
FOLDER
gcloud logging settings describe --folder=FOLDER_ID
ORGANIZATION
gcloud logging settings describe --organization=ORGANIZATION_ID
Running the previous command generates a service account for the organization
or folder, when one doesn't exist already,
and returns the ID in the kmsServiceAccountId
field.
The following illustrates a sample response to the previous command when an
organization is specified:
kmsServiceAccountId: cmek-p12345@gcp-sa-logging.iam.gserviceaccount.com name: organizations/ORGANIZATION_ID/settings
In this example, the value of SERVICE_ACCT_NAME is cmek-p12345
.
Run the provisioning process only once per resource. Running the describe
command multiple times returns the same value for the kmsServiceAccountId
field.
If you can't use the Google Cloud CLI, then run the
Cloud Logging API method getSettings
.
Assign the Encrypter/Decrypter role
To use CMEK, give the service account permission to use your Cloud KMS by assigning the Cloud KMS CryptoKey Encrypter/Decrypter role to the service account:
gcloud
gcloud kms keys add-iam-policy-binding \ --project=KMS_PROJECT_ID \ --member serviceAccount:SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com \ --role roles/cloudkms.cryptoKeyEncrypterDecrypter \ --location=KMS_KEY_LOCATION \ --keyring=KMS_KEY_RING \ KMS_KEY_NAME
Set the variables in the command as follows:
Replace SERVICE_ACCT_NAME with the
serviceAccountId
value that you determined in the previous step.Replace the other variables with the values you determined in the prerequisite steps.
Console
- Open the Cloud Key Management Service Keys browser in the Google Cloud console.
Open the Cloud KMS Keys browser Click on the name of the key ring that contains the desired key.
Select the checkbox for the desired key.
The Permissions tab becomes available.
In the Add members dialog, specify the email address of the Logging service account you are granting access.
In the Select a role drop down, select Cloud KMS CryptoKey Encrypter/Decrypter.
Click Add.
Configure the organization policies
Logging supports organization policies that can require CMEK protection and can limit which Cloud KMS CryptoKeys can be used for CMEK protection:
When
logging.googleapis.com
is in theDeny
policy list of services for the constraintconstraints/gcp.restrictNonCmekServices
, Logging refuses to create new user-defined buckets that aren't CMEK-protected. However, this constraint doesn't prevent Cloud Logging from creating the_Required
and_Default
log buckets which are created when a Google Cloud project is created.When
constraints/gcp.restrictCmekCryptoKeyProjects
is enforced, Logging creates CMEK-protected resources that are protected by a CryptoKey from an allowed project, folder, or organization.
For more information about CMEK and organization policies, see CMEK organization policies.
When an organization policy that specifies a CMEK constraint exists, ensure that those constraints are consistent with Logging's default resource settings for an organization or a folder. Further, if you plan to modify your default resource settings, before you update the default resource settings, review and, if necessary, update the organization policies.
To view or configure organization policies, do the following:
-
In the Google Cloud console, go to the Organization Policies page:
If you use the search bar to find this page, then select the result whose subheading is IAM & Admin.
- Select your organization.
Verify, and if necessary, update the constraints specific to CMEK.
For information about how to modify an organization policy, see Creating and editing policies.
Configure Cloud Logging with the Cloud KMS key
To configure CMEK as a default resource setting for Logging,
run the following
gcloud logging settings update
command:
FOLDER
gcloud logging settings update \ --folder=FOLDER_ID \ --kms-location=KMS_KEY_LOCATION \ --kms-key-name=KMS_KEY_NAME \ --kms-keyring=KMS_KEY_RING \ --kms-project=KMS_PROJECT_ID
The previous command updates the default resource settings to store information about the Cloud KMS key. You must ensure that the default storage location for the folder is set to the value of the KMS_KEY_LOCATION. If you haven't set the default storage location, or if the value of that location doesn't match the value of KMS_KEY_LOCATION, then append the following to the previous command:
--storage-location = KMS_KEY_LOCATION
The --storage-location
flag lets you set or update the
default storage location for the folder.
ORGANIZATION
gcloud logging settings update \ --organization=ORGANIZATION_ID \ --kms-location=KMS_KEY_LOCATION \ --kms-key-name=KMS_KEY_NAME \ --kms-keyring=KMS_KEY_RING \ --kms-project=KMS_PROJECT_ID
The previous command updates the default resource settings to store information about the Cloud KMS key. You must ensure that the default storage location for the organization is set to the value of the KMS_KEY_LOCATION. If you haven't set the default storage location, or if the value of that location doesn't match the value of KMS_KEY_LOCATION, then append the following to the previous command:
--storage-location = KMS_KEY_LOCATION
The --storage-location
flag lets you set or update the
default storage location for the organization.
After the key is applied, new log buckets in the organization or folder
are configured to encrypt their data at rest using this key. You can also
change keys
for individual log buckets. You can't create log buckets in
the global
region because you must use a key whose
region matches the regional scope of your data.
If you can't use the Google Cloud CLI, then run the
Cloud Logging API method updateSettings
.
Verify key enablement
To verify that you've successfully enabled CMEK for an organization or
a folder, run the following
gcloud logging settings describe
command:
FOLDER
gcloud logging settings describe --folder=FOLDER_ID
ORGANIZATION
gcloud logging settings describe --organization=ORGANIZATION_ID
The previous command returns the Cloud KMS key name:
kmsKeyName: KMS_KEY_NAME kmsServiceAccountId: SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com
If the kmsKeyName
field is populated, then CMEK is enabled for the
organization or folder.
Route logs to supported destinations
Cloud Logging log buckets can be configured to encrypt data with CMEK. When you configure CMEK as a default resource setting for an organization or folder, new log buckets in the organization or folder automatically use CMEK. You can change the key of these log buckets and you can create log buckets that use a different KMS key than the one specified by the default resource settings.
For information about CMEK as applied to log buckets, including how to change keys and limitations when you enable CMEK on a log bucket, see Configure CMEK for log buckets.
Cloud Storage supports CMEK for routing logs. For instructions about how to configure CMEK for Cloud Storage, see Using customer-managed encryption keys.
If data is lost due to key unavailability when routing logs data to Cloud Storage, then you can retroactively copy logs in bulk to Cloud Storage when those logs are also stored in a log bucket. For details, see Copy log entries.
BigQuery, by default, encrypts customer content stored at rest. For details, see Protecting data with Cloud Key Management Service keys.
Pub/Sub, by default, encrypts customer content stored at rest. For details, see Configuring message encryption.
Manage your Cloud KMS key
The following sections explain how to change, revoke access for, or disable your Cloud KMS key.
Change your Cloud KMS key
To change the Cloud KMS key associated with an organization or a folder, create a key and update the CMEK settings with the new Cloud KMS key name:
FOLDER
gcloud logging settings update \ --folder=FOLDER_ID --kms-key-name=NEW_KMS_KEY_NAME --kms-location=NEW_KMS_KEY_LOCATION \ --kms-keyring=NEW_KMS_KEY_RING \ --kms-project=NEW_KMS_PROJECT_ID
You must ensure that the default storage location for the folder is set to the value of the KMS_KEY_LOCATION. If you haven't set the default storage location, or if the value of that location doesn't match the value of KMS_KEY_LOCATION, then append the following to the previous command:
--storage-location = NEW_KMS_KEY_LOCATION
ORGANIZATION
gcloud logging settings update \ --organization=ORGANIZATION_ID --kms-key-name=NEW_KMS_KEY_NAME --kms-location=NEW_KMS_KEY_LOCATION \ --kms-keyring=NEW_KMS_KEY_RING \ --kms-project=NEW_KMS_PROJECT_ID
You must ensure that the default storage location for the organization is set to the value of the KMS_KEY_LOCATION. If you haven't set the default storage location, or if the value of that location doesn't match the value of KMS_KEY_LOCATION, then append the following to the previous command:
--storage-location = NEW_KMS_KEY_LOCATION
Revoke access to the Cloud KMS key
You can revoke Logging's access to the Cloud KMS key by removing the configured service account's IAM permission for that key.
If you remove Logging's access to a key, then it can take up to one hour for the change to take effect.
To revoke Logging's access to the Cloud KMS key, run the following Google Cloud CLI command:
gcloud kms keys remove-iam-policy-binding \ --project=KMS_PROJECT_ID \ --member serviceAccount:SERVICE_ACCT_NAME@gcp-sa-logging.iam.gserviceaccount.com \ --role roles/cloudkms.cryptoKeyEncrypterDecrypter \ --location=KMS_KEY_LOCATION \ --keyring=KMS_KEY_RING \ KMS_KEY_NAME
Disable CMEK
Disabling CMEK for an organization or a folder removes CMEK policy enforcement for future operations only; any previously applied configurations remain intact.
To disable CMEK on a resource that has CMEK configured as a default resource setting, run the following Google Cloud CLI command:
FOLDER
gcloud logging settings update --folder=FOLDER_ID --clear-kms-key
ORGANIZATION
gcloud logging settings update --organization=ORGANIZATION_ID --clear-kms-key
If you want to destroy your key, see Destroying and restoring key versions.
Cloud KMS key rotation considerations
Cloud Logging doesn't automatically rotate the encryption key for temporary disaster recovery files when the Cloud KMS key associated with the Google Cloud organization or folder rotates. Existing recovery files continue to use the key version with which they were created. New recovery files use the current primary key version.
Limitations
The following are known limitations when you configure CMEK as a default resource setting for Logging.
Disaster recovery file unavailability
A Cloud KMS key is considered available and accessible by Logging when both of the following are true:
- The key is enabled.
- The Logging service account has encrypt and decrypt permissions on the key.
If Logging loses access to the Cloud KMS key, then Logging is unable to write temporary disaster-recovery files and, for users, queries stop functioning. Query performance may remain degraded even after key access is restored.
Routing logs to Cloud Storage might also be affected because the Logging is unable to write temporary files required to facilitate the routing. If an error is encountered while encrypting or decrypting data, then a notification is sent to the Google Cloud project that contains the Cloud KMS key.
Client library availability
Logging client libraries don't provide methods for configuring CMEK.
Degradation due to Cloud EKM key unavailability
When you use a Cloud EKM key, Google has no control over the availability of your externally managed key in the external key-management partner system.
When CMEK is configured as default resource setting for an organization or a folder, if an externally managed key is unavailable, then Cloud Logging continually retries accessing the key. Cloud Logging also buffers the incoming log data for up to one hour. After an hour, if Cloud Logging is still unable to access the externally managed key, then Cloud Logging begins dropping the data.
If CMEK is applied to a log bucket and if an externally managed key is unavailable, then Cloud Logging continues storing logs in log buckets, but users won't be able to access that data.
See the Cloud External Key Manager documentation for more considerations, and potential alternatives, when using external keys.
Limitations on log buckets
For limitations when you use CMEK with log buckets, see Limitations.
Quotas
For details on Logging usage limits, see Quotas and limits.
Troubleshoot configuration errors
For information about troubleshooting CMEK configuration errors, see Troubleshoot CMEK and default setting errors.