Collect AWS S3 server access logs

Supported in:

This document explains how to collect AWS S3 server access logs by setting up a Google Security Operations feed. The parser extracts fields using grok patterns, handles potential JSON input, and maps the extracted fields to the UDM. It performs data transformations, type conversions, and conditional logic based on the presence and values of specific fields to ensure accurate UDM representation.

Before you begin

Ensure you have the following prerequisites:

  • Google SecOps instance
  • Privileged access to AWS

How to configure AWS S3 server access logging

Google SecOps supports log collection using Amazon S3 through Amazon SQS.

  1. Sign in to the AWS Management console.
  2. Access the Amazon S3 console.
  3. Go to Amazon S3 > Buckets.
  4. Select an existing bucket or create a new one.
  5. Click Properties.
  6. In the Server access logging section, click Edit.
  7. Select Enable.
  8. In the Target bucket field, enter a name for the new bucket to send the log record objects to or select an existing bucket as the target.
  1. Click Save changes.
  2. To create the SQS queue for the S3 bucket, configure an Amazon SQS instance with the S3 storage. For more information, see Configuring a bucket for notifications (SNS topic or SQS queue).

Based on the service and region, identify the endpoints for connectivity by referring to the following AWS documentation:

Set up feeds

There are two different entry points to set up feeds in the Google SecOps platform:

  • SIEM Settings > Feeds
  • Content Hub > Content Packs

Set up feeds from SIEM Settings > Feeds

To configure multiple feeds for different log types within this product family, see Configure feeds by product.

To configure a single feed, follow these steps:

  1. Go to SIEM Settings > Feeds.
  2. Click Add New Feed.
  3. On the next page, click Configure a single feed.
  4. In the Source type list, select Amazon S3 or Amazon SQS.
  5. In the Log type list, select AWS S3 server access.
  6. Click Next.
  7. Google SecOps supports log collection using an access key ID and secret method. To create the access key ID and secret, see Configure tool authentication with AWS.
  8. Based on the AWS S3 server access configuration that you created, specify values for the input parameters:
    • If using Amazon S3, specify values for the following fields:
      • Region
      • S3 URI
      • URI is a
      • Source deletion option
      • Access key ID
      • Secret access key
    • If using Amazon SQS, specify values for the following fields:
      • Region
      • Queue name
      • Account number
      • Queue access key ID
      • Queue secret access key
      • Source deletion option
  9. Click Next, and then click Submit.

Set up feeds from the Content Hub

You can configure the ingestion feed in Google SecOps using either Amazon SQS (preferred) or Amazon S3.

Specify values for the following fields:

  • Region: Region where the S3 bucket or SQS queue is hosted.
  • Queue Name: Name of the SQS queue from which to read log data.
  • Account Number: Account number that owns the SQS queue.
  • Queue Access Key ID: 20-character account access key ID. For example, AKIAOSFOODNN7EXAMPLE.
  • Queue Secret Access Key: 40-character secret access key. For example, wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY.
  • Source deletion option: Option to delete files and directories after transferring the data.

Advanced options

  • Feed Name: A prepopulated value that identifies the feed.
  • Source Type: Method used to collect logs into Google SecOps.
  • Asset Namespace: Namespace associated with the feed.
  • Ingestion Labels: Labels applied to all events from this feed.

UDM Mapping Table

Log Field UDM Mapping Logic
aclRequired target.resource.attribute.labels.key: "aclRequired"
target.resource.attribute.labels.value: Value of aclRequired		 Directly mapped from the raw log field aclRequired.
authenticationtype extensions.auth.auth_details Directly mapped from the raw log field authenticationtype.
bucket target.resource.name Directly mapped from the raw log field bucket.
bucket target.resource.resource_type: "STORAGE_BUCKET" The parser sets the resource_type to "STORAGE_BUCKET" if the bucket field is present.
bucketowner target.resource.product_object_id Directly mapped from the raw log field bucketowner.
bytes_sent network.sent_bytes Directly mapped from the raw log field bytes_sent after converting it to an unsigned integer and replacing "-" with "0".
ciphersuite network.application_protocol: "HTTPS" The parser sets the application_protocol to "HTTPS" if the ciphersuite field is present.
ciphersuite network.tls.cipher Directly mapped from the raw log field ciphersuite.
errorcode security_result.action_details Directly mapped from the raw log field errorcode.
errorcode security_result.action: "BLOCK" The parser sets the action to "BLOCK" if the errorcode field contains "AccessDenied" (case-insensitive).
hostheader target.hostname Extracted from the raw log field hostheader, potentially removing the port number.
hostheader target.port Extracted from the raw log field hostheader if a port number is present.
hostid target.resource.attribute.labels.key: "S3 Extended Request ID"
target.resource.attribute.labels.value: Value of hostid		 Directly mapped from the raw log field hostid.
http_capture network.http.method The HTTP method is extracted from the http_capture field.
http_capture network.http.version The HTTP version is extracted from the http_capture field.
http_capture target.url The target URL is constructed using hostheader and http_request_uri (extracted from http_capture), prefixed with "http://" or "https://" based on the presence of ciphersuite.
httpstatus network.http.response_code Directly mapped from the raw log field httpstatus after converting it to an integer.
object_version_id target.resource.product_object_id Directly mapped from the raw log field object_version_id.
objectsize target.file.size Directly mapped from the raw log field objectsize after converting it to an unsigned integer and replacing "-" with "0".
operation metadata.product_event_type Directly mapped from the raw log field operation.
referrer network.http.referral_url Directly mapped from the raw log field referrer after removing quotes.
remoteip metadata.event_type: "USER_RESOURCE_ACCESS" The parser sets the event_type to "USER_RESOURCE_ACCESS" if the remoteip field is empty.
remoteip principal.ip Directly mapped from the raw log field remoteip.
requester target.resource.attribute.labels.key: "Access Point ARN"
target.resource.attribute.labels.value: Value of requester		 Directly mapped from the raw log field requester.
requester_user principal.user.userid Directly mapped from the raw log field requester_user.
requestid network.session_id Directly mapped from the raw log field requestid.
request_time_ms network.session_duration.nanos Directly mapped from the raw log field request_time_ms after converting it to an integer, replacing "-" with "0", and padding with zeros to represent nanoseconds.
signatureversion target.resource.attribute.labels.key: "Signature Version"
target.resource.attribute.labels.value: Value of signatureversion		 Directly mapped from the raw log field signatureversion.
time metadata.event_timestamp Parsed from the raw log field time and converted to a timestamp.
tlsVersion network.tls.version Directly mapped from the raw log field tlsVersion.
useragent network.http.user_agent Directly mapped from the raw log field useragent after removing quotes.
(Parser Logic) metadata.event_type: "NETWORK_HTTP" The parser sets the default event_type to "NETWORK_HTTP".
(Parser Logic) metadata.log_type: "AWS_S3_SERVER_ACCESS" The parser sets the log_type to "AWS_S3_SERVER_ACCESS".
(Parser Logic) metadata.product_name: "AWS S3 Server Access" The parser sets the product_name to "AWS S3 Server Access".
(Parser Logic) metadata.product_version: "HTTP/http_version" The parser sets the product_version using the extracted http_version.
(Parser Logic) metadata.vendor_name: "AMAZON" The parser sets the vendor_name to "AMAZON".
(Parser Logic) network.application_protocol: "HTTP" The parser sets the application_protocol to "HTTP" if the ciphersuite field is not present.
(Parser Logic) timestamp The parser sets the event timestamp to the current time when the event is processed.

Need more help? Get answers from Community members and Google SecOps professionals.