Collect AWS CloudTrail logs

This document describes how to collect AWS CloudTrail logs by setting up a Google Security Operations feed and how log fields map to Google SecOps Unified Data Model (UDM) fields.

For more information, see Data ingestion to Google SecOps.

A typical deployment consists of AWS CloudTrail and the Google SecOps feed configured to send logs to Google SecOps. Your deployment might be different from the typical deployment described in this document. The deployment contains the following components:

• AWS CloudTrail: The platform that collects logs.

• AWS S3: The platform that stores logs.

• Google SecOps feed: The Google SecOps feed that fetches logs from AWS S3 and writes logs to Google SecOps.

• Google SecOps: The platform that retains and analyzes the logs from AWS CloudTrail.

An ingestion label identifies the parser that normalizes raw log data to structured UDM format. The information into this document applies to the parser with the AWS_CLOUDTRAIL ingestion label.

Before you begin

Make sure you have the following prerequisites:

• AWS Account
• prerequisites for using AWS CloudTrail are met. For more information, see AWS CloudTrail setup.
• All systems in the deployment architecture use the UTC time zone

Basic steps to ingest logs from S3 with SQS

This section describes the basic steps for ingesting AWS CloudTrail logs into your Google SecOps instance. The steps describe how to do this using Amazon S3 with Amazon SQS as the feed source type.

Configure AWS CloudTrail and S3

In this procedure, you configure AWS CloudTrail logs to be written to an S3 bucket.

1. In the AWS console, search for CloudTrail.
2. Click Create trail.
3. Provide a Trail name.
4. Select Create new S3 bucket. You may also choose to use an existing S3 bucket.
5. Provide a name for AWS KMS alias, or choose an existing AWS KMS Key.
6. You can leave the other settings as default, and click Next.
7. Choose Event type, add Data events as required, and click Next.
8. Review the settings in Review and create, and click Create trail.
9. In the AWS console, search for Amazon S3 Buckets.
10. Click the newly created log bucket, and select the folder AWSLogs. Then click Copy S3 URI and save it for use in the following steps.

Set up a Standard SQS Queue and SNS

If you use an SQS queue, it must be a Standard queue, not a FIFO queue.

1. Enable AWS CloudTrail and configure it to deliver logs to an S3 bucket using a new or existing trail.
2. Open the AWS SNS console and create a new Standard topic. Name it, e.g., CloudTrail-Notification-Topic.

3. Create a SQS queue using the AWS SQS console, e.g., CloudTrail-Notification-Queue, and update its access policy to allow the SNS topic ARN to send messages. For details about creating SQS queues, see Getting started with Amazon SQS.

   Example SQS policy snippet:

   {
      "Version": "2012-10-17",
      "Id": `PolicyForSNS`,
      "Statement": [
         {
            "Sid": "AllowSNS",
            "Effect": "Allow",
            "Principal": { "Service": "sns.amazonaws.com" },
            "Action": "SQS:SendMessage",
            "Resource": "arn:aws:sqs:REGION:ACCOUNT_ID:CloudTrail-Notification-Queue",
            "Condition": {
            "ArnEquals": { "aws:SourceArn": "arn:aws:sns:REGION:ACCOUNT_ID:CloudTrail-Notification-Topic"}
            }
         }
      ]
   }
   

4. Go to the SNS topic → Subscriptions → Create subscription, set Protocol to SQS, and Endpoint to the ARN of the SQS queue.

5. CloudTrail doesn't natively push new logs to SNS. To enable notifications, you can use a CloudTrail Event Selector for Management Events, or use CloudTrail integration with CloudWatch Logs and then create a CloudWatch Event Rule that triggers notifications by setting an SNS topic as the target. For more details, see set up notifications on your S3 bucket.

   Example event pattern:

   {
      "source": ["aws.s3"],
      "detail-type": ["AWS API Call via CloudTrail"],
      "detail": {
         "eventName": ["PutObject"],
         "requestParameters": {
            "bucketName": [`CloudTrail-Notification-Topic`]
         }
      }
   }
   

6. Make sure IAM roles or policies allow CloudWatch Events to publish to SNS, and make sure SNS is allowed to send messages to SQS.

Configure AWS IAM user

Configure an AWS IAM user which Google SecOps will be used to access both the SQS queue (if used) and the S3 bucket.

1. In the AWS console, search for IAM.
2. Click Users, and then in the following screen, click Create Users.
3. Provide a name for the user, for example, chronicle-feed-user, and select Provide user access to the AWS Management Console.
4. Select Attach existing policies directly and select AmazonS3ReadOnlyAccess or AmazonS3FullAccess, as required. AmazonS3FullAccess would be used if Google SecOps should clear the S3 buckets after reading logs, to optimize AWS S3 storage costs.
5. As a recommended alternative to the previous step, you can further restrict access to only the specified S3 bucket by creating a custom policy. Click Create policy and follow the AWS documentation to create a custom policy.
6. When you apply a policy, make sure that you have included sqs:DeleteMessage. Google SecOps isn't able to delete messages if the sqs:DeleteMessage permission isn't attached to the SQS queue. All the messages are accumulated on the AWS side, which causes a delay as Google SecOps repeatedly attempts to transfer the same files.
7. Click Next:Tags.
8. Add any tags if required, and click Next:Review.
9. Review the configuration and click Create user.
10. Once the user is created, go to the Security Credentials tab, and click Create Access Key.
11. Choose CLI and click Next:Tags.
12. Add any tags if required, then click Create Access Key: Review.
13. Copy the Access Key ID and Secret Access Key of the created user, for use in the next step.

Configuring KMS key permissions

A KMS key is required to decrypt CloudTrail logs, which are encrypted server-side. AWS KMS provides enhanced encryption and security for sensitive data stored in Amazon S3.

1. In the AWS console, search for Key Management Service (KMS).
2. Click Create Key:Next.
3. Add an Alias for the key. Optionally, add a Description and Tags if required. Click Next: Review.
4. After reviewing the configuration, click Next.
5. Select the Key Users who should have access to this key, then click Finish.

Set up feeds

There are two different entry points to set up feeds in the Google SecOps platform:

• SIEM Settings > Feeds
• Content Hub > Content Packs

Set up feeds from SIEM Settings > Feeds

To configure an ingestion feed, do the following:

1. Go to SIEM Settings > Feeds.
2. Click Add New Feed.
3. On the next page, click Configure a single feed.
4. Select Amazon SQS as the Source Type.
5. Select AWS CloudTrail as the Log type to create a feed for AWS CloudTrail.
6. Click Next
7. Configure the following input parameters:
   • Region: the S3 region associated with the URI.
   • Queue name: the SQS queue name.
   • Account number: the SQS account number.
   • Queue access key ID: Account access key that is a 20-character alphanumeric string.
   • Queue secret access key: Account access key that is a 40-character alphanumeric string.
   • Select Source Deletion Option.
   • S3 bucket access key ID: An account access key that is a 20-character alphanumeric string.
   • S3 bucket secret access key: An account access key that is a 20-character alphanumeric string.
8. Click Next and then click Submit.

If you encounter issues when you ingest AWS CloudTrail logs, contact Google SecOps support.

Set up feeds from the Content Hub

Specify values for the following fields:

• Region: the S3 region associated with the URI.
• Queue name: the SQS queue name.
• Account number: the SQS account number.
• Queue access key ID: Account access key that is a 20-character alphanumeric string.
• Queue secret access key: Account access key that is a 40-character alphanumeric string.
• Select Source Deletion Option.
• S3 bucket access key ID: An account access key that is a 20-character alphanumeric string.
• S3 bucket secret access key: An account access key that is a 20-character alphanumeric string.

Advanced options:

• Feed Name: A prepopulated value that identifies the feed.
• Source Type: Method used to collect logs into Google SecOps.
• Asset Namespace: Namespace associated with the feed.
• Ingestion Labels: Labels applied to all events from this feed.

Supported AWS CloudTrail log types

The AWS CloudTrail parser supports the following services:

Field mapping reference

Field mapping reference: Event Identifier to Event Type

The following table lists the AWS Cloudtrail log type eventName and their corresponding UDM event types.

Field mapping reference: AWS CloudTrail - Common Fields

The following table lists common fields for the AWS Cloudtrail log and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - Resource Explorer

The following table lists the log fields for the Resource Explorer log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - Resource Groups

The following table lists the log fields for the Resource Groups log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - Elastic Kubernetes Service

The following table lists the log fields for the Elastic Kubernetes Service log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - ECR-Public

The following table lists the log fields for the ECR-Public log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - Elastic Container Registry

The following table lists the log fields for the Elastic Container Registry log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - Elasticache

The following table lists the log fields for the Elasticache log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - Elastic Container Service

The following table lists the log fields for the Elastic Container Service log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - Route53

The following table lists the log fields for the Route53 log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - Organizations

The following table lists the log fields for the Organizations log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - Config

The following table lists the log fields for the Config log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - Miscellaneous Services

The following table lists the log fields for the Miscellaneous Services log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - S3

The following table lists the log fields for the S3 log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - Elastic Load Balancer

The following table lists the log fields for the Elastic Load Balancer log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - APIGateway And SES

The following table lists the log fields for the APIGateway And SES log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - AutoScale and Appconfig

The following table lists the log fields for the AutoScale and Appconfig log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - Security Identity Compliance

The following table lists the log fields for the Security Identity Compliance log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - Cloudtrail

The following table lists the log fields for the Cloudtrail log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - Cloudshell

The following table lists the log fields for the Cloudshell log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - Lambda

The following table lists the log fields for the Lambda log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - RDS

The following table lists the log fields for the RDS log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - EC2

The following table lists the log fields for the EC2 log type and their corresponding UDM fields.

Field mapping reference: AWS CloudTrail - Dynamo

The following table lists the log fields for the Dynamo log type and their corresponding UDM fields.

What's next

• Data ingestion to Google SecOps

Need more help? Get answers from Community members and Google SecOps professionals.