Aggregate and store your organization's logs

This document describes how to manage the log entries generated by the resources contained in your Google Cloud organization by using a non-intercepting aggregated sink.

You can configure an aggregated sink to be intercepting or non-intercepting, depending on whether you want control over which log entries can be queried in, or routed through the sinks in child resources. In this tutorial, you create an aggregated sink that routes your organization's audit logs to a Google Cloud project, which then routes the aggregated audit logs to a log bucket.

For more information about aggregated sinks, see Collate and route organization- and folder-level logs to supported destinations.

In this tutorial, you perform the following steps:

  1. You start by creating a log bucket and a log sink in the Google Cloud project where you want to store your aggregated log entries.

  2. Next, you create a non-intercepting aggregated sink at the organization level to route log entries to the Google Cloud project that contains the log bucket.

  3. You then configure read access to log views on the new log bucket.

  4. Lastly, you query and view your log entries from the Logs Explorer page.

Before you begin

Ensure the following:

  • To get the permissions that you need to configure an aggregated sink, ask your administrator to grant you the following IAM roles on your organization:

    • To create log buckets and sinks in a project: Logs Configuration Writer (roles/logging.configWriter) - your project
    • To create an aggregated sink: Logs Configuration Writer (roles/logging.configWriter) - your organization
    • To grant roles to principals: Owner (roles/owner) - your project

    For more information about granting roles, see Manage access to projects, folders, and organizations.

    You might also be able to get the required permissions through custom roles or other predefined roles.

  • If you use VPC Service Controls, then you must add an ingress rule to the service perimeter. For more information about VPC Service Controls limitations, see Aggregated sinks and VPC Service Controls limitations.

  • In the Google Cloud console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

Create a log bucket

Log buckets store the log entries that are routed from other Google Cloud projects, folders, or organizations. For more information, see Configure log buckets.

To create the log bucket in the Google Cloud project that you want to aggregate log entries into, complete the following steps:

  1. Open a shell.

    For example, to use the Cloud Shell, do the following:

    1. Go to the Google Cloud console:

      Go to Google Cloud console

    2. In the toolbar, click  Activate Cloud Shell.

  2. In the shell, run the gcloud logging buckets create command.

    Before running the following command, make the following replacements:

    • BUCKET_NAME: The name of the log bucket.
    • LOCATION: The location of the log bucket. After you create your log bucket, you can't change its location.
    • PROJECT_ID: The identifier of the project in which to create the log bucket.

    Execute the gcloud logging buckets create command:

     gcloud logging buckets create BUCKET_NAME \
   --location=LOCATION --project=PROJECT_ID

  3. Verify that the log bucket was created:

    gcloud logging buckets list --project=PROJECT_ID

    The response of the command is a list of the log buckets in your project.

  4. Log buckets have configurable retention periods. If you want to set the retention period of the log entries in your log bucket, then use the gcloud logging buckets update command. For example, the following command extends the retention of log entries stored in the log bucket to 365 days:

    gcloud logging buckets update BUCKET_NAME \
   --location=LOCATION --project=PROJECT_ID \
   --retention-days=365

    For more information about options, see gcloud logging buckets update.

Create the project-level log sink

You route log entries to a log bucket by creating a sink. A sink includes an inclusion filter, exclusion filters, and a destination. In this tutorial, you configure an inclusion filter and the destination to your new log bucket. Your sink doesn't contain any exclusion filters. For more information about sinks, see Route logs to supported destinations.

To create a sink that routes log entries to the log bucket you just created, run the gcloud logging sinks create command.

Before running the following command, make the following replacements:

  • PROJECT_LEVEL_SINK_NAME: The name of the project-level log sink.

  • SINK_DESTINATION: The log bucket where your log entries are routed. The destination path format for a log bucket is the following:

    logging.googleapis.com/projects/PROJECT_ID/locations/LOCATION/buckets/BUCKET_NAME

  • PROJECT_ID: The identifier of the project in which to create the log sink. Set this option to the same project where you created the log bucket.

  • Include the following options:

    • --log-filter : Use this option to set a filter that matches the log entries you want to include in your sink. In this tutorial, the filter is set to select all audit log entries. If you don't set a filter, then all log entries from your Google Cloud project are routed to the destination.
    • --description: Use this option to describe the purpose or use case for the sink.

Execute the gcloud logging sinks create command:

gcloud logging sinks create PROJECT_LEVEL_SINK_NAME SINK_DESTINATION
--project=PROJECT_ID
--log-filter='logName:cloudaudit.googleapis.com' \
--description="Audit logs from my organization" \

Create the aggregated sink

Aggregated sinks combine and route log entries from the resources contained by an organization or folder to a destination.

In this tutorial, you create an aggregated sink that is non-intercepting. This means every log entry that is routed by the aggregated sink is also routed by the sinks in the resource in which the log entry originates. For example, an audit log that originates in a project is routed by the aggregated sink and the sinks in that project. Therefore, it is possible for you to store multiple copies of a log entry.

You can create intercepting sinks. For more information, see Collate and route organization- and folder- level logs to supported destinations.

Set up the sink at the organization level

To create an aggregated sink that is non-intercepting and that routes log entries to a project, complete the following steps:

  1. Run the gcloud logging sinks create command.

    Before running the following command, make the following replacements:

    • SINK_NAME: The name of the log sink. You can't change the name of a sink after you create it.
    • PROJECT_ID: The identifier of the project which stores the log bucket.
    • ORGANIZATION_ID: The identifier of the organization.

    Execute the gcloud logging sinks create command:

    gcloud logging sinks create SINK_NAME \
logging.googleapis.com/projects/PROJECT_ID  \
  --log-filter='logName:cloudaudit.googleapis.com' \
  --description="Audit logs from my organization" \
  --organization=ORGANIZATION_ID \
  --include-children

    The --include-children option is important. This option ensures that log entries from all the Google Cloud projects and folders within your organization are routed. For more information, see Collate and route organization-level logs to supported destinations.

  2. Verify that the sink was created:

    gcloud logging sinks list --organization=ORGANIZATION_ID

  3. Get the name of the service account:

    gcloud logging sinks describe SINK_NAME --organization=ORGANIZATION_ID

    The output looks similar to the following:

    writerIdentity: serviceAccount:o1234567890-ORGANIZATION_ID@gcp-sa-logging.iam.gserviceaccount.com

  4. Copy the value of the serviceAccount field into your clipboard.

Grant access to the sink

After creating the aggregated sink, you must grant permission for the sink to write log entries to the project that you set as the destination. You can grant permission by using the Google Cloud console or by editing the Identity and Access Management (IAM) policy, as described in Set destination permissions.

To grant your sink permission to write log entries, do the following:

  1. In the Google Cloud console, go to the IAM page:

    Go to IAM

    If you use the search bar to find this page, then select the result whose subheading is IAM & Admin.

  2. Select the Google Cloud project that contains your log bucket.

  3. Click Grant access and add the service account as a new principal. Don't include the serviceAccount: prefix.

  4. In the Select a role menu, select Logs Writer.

  5. Click Save.

Generate log entries to assist in sink verification

To verify that your aggregated sink is properly configured, try the following:

  1. Generate audit log entries that should be routed to your log bucket.

    • If you have many Google Cloud projects in your organization, then you might have enough audit-log traffic that you don't need to create any for validation purposes. Go to the next step.

    • Otherwise, go to a different project, create a Compute Engine VM instance, and then delete the instance you created. Audit logs are written when a VM is created, started, and deleted.

  2. Follow the procedure in the section titled View logs in the Logs Explorer page to view your audit logs. Be sure that you select the _AllLogs view.

Configure read access to a log view on a log bucket

When you create a log bucket, Cloud Logging automatically creates a log view named _AllLogs. This view includes every log entry stored in the log bucket.

To restrict a principal to having access only to specific log entries, create a log view, and then do one of the following:

  • Grant them the role of roles/logging.viewAccessor along with an IAM condition that restricts the grant to the log view.

  • On the IAM policy associated with the log view, grant a principal access. We recommend this approach when you create a large number of log views.

For more information about these two approaches, see Control access to a log view.

In the following steps, you grant a principal the role of roles/logging.viewAccessor along with an IAM condition that restricts the grant to the view named _AllLogs:

  1. In the Google Cloud console, go to the IAM page:

    Go to IAM

    If you use the search bar to find this page, then select the result whose subheading is IAM & Admin.

  2. Select the Google Cloud project that contains the log bucket.

  3. Click Add.

  4. In the New principal field, add a principal.

  5. In the Select a role menu, select Logs Views Accessor.

    If you don't add a condition to this role, then the principal has access to all log views on all user-defined log buckets in the Google Cloud project.

  6. Add an IAM condition to the binding:

    1. Click Add condition, enter a title and description.
    2. In the Condition type menu, scroll to Resource, and then select Name.
    3. In the Operator menu, select Ends with.

    4. In the Value field, enter the full name of the log view:

      locations/LOCATION/buckets/BUCKET_NAME/views/_AllLogs

    5. Click Save to save the condition.

  7. Click Save to save the binding.

View log entries in the Logs Explorer page

To view the log entries in your log bucket, do the following:

  1. In the Google Cloud console, go to the Logs Explorer page:

    Go to Logs Explorer

    If you use the search bar to find this page, then select the result whose subheading is Logging.

  2. Select Refine Scope.

  3. On the Refine scope panel, select Log view.

  4. Select the log view, or log views, whose log entries you want to see. For example, to view all log entries, select the view named _AllLogs.

  5. Click Apply.

    The Logs Explorer refreshes to show log entries from your log bucket. For information about using the Logs Explorer, see Using the Logs Explorer.