Collect AWS VPC Flow Logs

This document explains how to ingest AWS VPC Flow Logs to Google Security Operations using three different methods: Amazon S3 (Text format), Amazon CloudWatch Logs with Kinesis Data Firehose, and CSV format in Amazon S3. AWS VPC Flow Logs is a feature that lets you capture information about the IP traffic going to and from network interfaces in your VPC. This integration lets you send these logs to Google SecOps for analysis and monitoring.

Supported AWS VPC Flow Log formats

Google SecOps supports the ingestion of AWS VPC Flow Logs in two primary text formats:

• JSON Format: The AWS_VPC_FLOW log type parses logs in JSON format. In this format, each log entry includes both a key and its corresponding value, making the data self-describing.
• CSV Format: Google SecOps also provides a parser for AWS VPC Flow Logs in CSV format. This format lists field keys only once in the header row, with subsequent rows containing only comma-separated values.

Because the CSV format doesn't include field keys in each log entry, the AWS_VPC_FLOW_CSV parser relies on a strict, predefined order of values. Your CSV files must adhere to the following field order for correct parsing:

   Version,Account_id,Interface_id,Srcaddr,Dstaddr,Srcport,Dstport,Protocol,Packets,Bytes,Start,End,Action,Log_status,Vpc_id,Subnet_id,Instance_id,Tcp_flags,Type,Pkt_srcaddr,Pkt_dstaddr,Region,Az_id,Sublocation_type,Sublocation_id,Pkt_src_aws_service,Pkt_dst_aws_service,Flow_direction,Traffic_path,Ecs_cluster_arn,Ecs_cluster_name,Ecs_container_instance_arn,Ecs_container_instance_id,Ecs_container_id,Ecs_second_container_id,Ecs_service_name,Ecs_task_definition_arn,Ecs_task_arn,Ecs_task_id

The following is an example of a CSV log line:

   7,369096419186,eni-0520bb5efed19d33a,10.119.32.34,10.119.223.3,51256,16020,6,14,3881,1723542839,1723542871,ACCEPT,OK,vpc-0769a6844ce873a6a,subnet-0cf9b2cb32f49f258,i-088d6080f45f5744f,0,IPv4,10.119.32.34,10.119.223.3,ap-northeast-1,apne1-az4,-,-,-,-,ingress,,-,-,-,-,-,-,-,-,-,-

For fields where no value is available, an empty value (for example, , ,) should be passed to maintain the correct positional order within the CSV row.

Before you begin

Make sure you have the following prerequisites:

• Google SecOps instance.
• Privileged access to AWS.

Option 1: Configure AWS VPC Flow Logs export using AWS S3 (Text format)

The following section outlines how to configure Amazon S3 and Identity and Access Management permissions to enable the export of VPC Flow Logs for analysis by Google SecOps.

Configure AWS S3 bucket and IAM for Google SecOps

1. Create Amazon S3 bucket following this user guide: Creating a bucket.
2. Save bucket Name and Region for future reference (for example, aws-vpc-flowlogs).
3. Create a User following this user guide: Creating an IAM user.
4. Select the created User.
5. Select the Security credentials tab.
6. Click Create Access Key in section Access Keys.
7. Select Third-party service as Use case.
8. Click Next.
9. Optional: Add a description tag.
10. Click Create access key.
11. Click Download CSV file to save the Access Key and Secret Access Key for future reference.
12. Click Done.
13. Select the Permissions tab.
14. Click Add permissions in section Permissions policies.
15. Select Add permissions.
16. Select Attach policies directly.
17. Search for AmazonS3FullAccess policy.
18. Select the policy.
19. Click Next.
20. Click Add permissions.

Create VPC Flow Logs (destination: Amazon S3, Text format)

1. Open AWS Console > VPC > Your VPCs/Subnets/Network interfaces and select the scope you want to log.
2. Click Actions > Create flow log.
3. Provide the following configuration details:
   • Filter: Choose All (or Accept / Reject) per your policy.
   • Maximum aggregation interval: Select 1 minute (recommended) or 10 minutes.
   • Destination: Send to an Amazon S3 bucket.
   • S3 bucket ARN: Enter the bucket name created in the previous section in the following format: arn:aws:s3:::<your-bucket>.
   • Log record format: Select AWS default format.
   • Log file format: Select Text (Plain).
   • Optional: Disable Hive-compatible prefixes and Hourly partitions unless you need them.
4. Click Create flow log.

Configure a feed in Google SecOps to ingest AWS VPC Flow Logs (S3 Text)

1. Go to SIEM Settings > Feeds.
2. Click + Add New Feed.
3. In the Feed name field, enter a name for the feed (for example, AWS VPC Flow Logs - S3 (Text)).
4. Select Amazon S3 V2 as the Source type.
5. Select AWS VPC Flow as the Log type.
6. Click Next.
7. Specify values for the following input parameters:
   • S3 URI: Enter the S3 bucket address (for example, s3://<your-bucket>/AWSLogs/<account-id>/vpcflowlogs/<region>/).
   • Source deletion options: Select deletion option according to your preference.
   • Maximum File Age: Default 180 Days.
   • Access Key ID: User access key with access to the S3 bucket.
   • Secret Access Key: User secret key with access to the S3 bucket.
   • Asset namespace: The asset namespace.
   • Ingestion labels: The label applied to the events from this feed.
8. Click Next.
9. Review your new feed configuration in the Finalize screen, and then click Submit.

Option 2: Configure AWS VPC Flow Logs export using Amazon CloudWatch Logs and Kinesis Data Firehose

After setting up the flow logs to go to CloudWatch, this option provides an additional layer of data export by streaming that log data to a destination of your choice using Kinesis Data Firehose.

Create VPC Flow Logs (destination: Amazon CloudWatch Logs)

1. Open AWS Console > VPC > Your VPCs/Subnets/Network interfaces.
2. Click Actions > Create flow log.
3. Provide the following configuration details:
   • Filter: Choose All (or Accept/Reject) per your policy.
   • Maximum aggregation interval: Select 1 minute (recommended) or 10 minutes.
   • Destination: Select Send to CloudWatch Logs.
   • Destination log group: Select or create a log group (for example, /aws/vpc/flowlogs).
   • IAM role: Select a role that can write to CloudWatch Logs.
   • Log record format: Select AWS default (version 2) or Custom (includes additional fields).
4. Click Create flow log.

Create a feed in Google SecOps to get Endpoint URL and Secret Key

1. Go to SIEM Settings > Feeds.
2. Click + Add New Feed.
3. In the Feed name field, enter a name for the feed (for example, AWS VPC Flow Logs - CloudWatch via Firehose).
4. Select Amazon Data Firehose as the Source type.
5. Select AWS VPC Flow as the Log type.
6. Click Next.
7. Specify values for the following input parameters:
   • Split delimiter: Optional n.
   • Asset namespace: The asset namespace (for example, aws.vpc.flowlogs.cwl).
   • Ingestion labels: The label to be applied to the events from this feed (for example, source=vpc_flow_firehose).
8. Click Next.
9. Review the feed configuration and click Submit.
10. Click Generate Secret Key to generate a secret key to authenticate this feed.
11. Copy and save the secret key as you cannot view this secret again.
12. Go to the Details tab.
13. Copy the feed endpoint URL from the Endpoint Information field.
14. Click Done.

Create an API key for the Amazon Data Firehose feed

1. Go to the Google Cloud console Credentials page.
2. Click Create credentials, and then select API key.
3. Copy and save the key in a secure location.
4. Restrict the API key access to the Google SecOps API.

Configure IAM permissions for CloudWatch Logs to Firehose

1. In the AWS Console, go to IAM > Policies > Create policy > JSON.

2. Paste the following policy JSON, replacing <region> and <account-id> with your AWS Region and account ID:

   {
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": [
         "firehose:PutRecord",
         "firehose:PutRecordBatch"
         ],
         "Resource": "arn:aws:firehose:<region>:<account-id>:deliverystream/cwlogs-to-secops"
      }
   ]
   }
   

3. Name the policy CWLtoFirehoseWrite and click Create policy.

4. Go to IAM > Roles > Create role.

5. Select Custom trust policy and paste:

   {
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": {
         "Service": "logs.<region>.amazonaws.com"
         },
         "Action": "sts:AssumeRole"
      }
   ]
   }
   

6. Attach the policy CWLtoFirehoseWrite to the role.

7. Name the role CWLtoFirehoseRole and click Create role.

Configure Amazon Kinesis Data Firehose to Google SecOps

1. In the AWS Console, go to Kinesis > Data Firehose > Create delivery stream.
2. Provide the following configuration details:
   • Source: Select Direct PUT or other sources.
   • Destination: Choose HTTP endpoint.
   • Name: cwlogs-to-secops
   • HTTP endpoint URL: Enter the Feed HTTPS endpoint URL from Google SecOps with the API Key appended: <ENDPOINT_URL>?key=<API_KEY>
   • HTTP method: Select POST.
3. Under Access key:
   • Enter the Secret key generated in Google SecOps feed (this becomes the X-Amz-Firehose-Access-Key header).
   • Buffering hints: set Buffer size = 1 MiB, Buffer interval = 60 seconds.
   • Compression: select Disabled.
   • S3 backup: select Disabled.
   • Leave retry and logging settings as default.
4. Click Create delivery stream.

Subscribe the CloudWatch Logs group to the Firehose stream

1. Go to CloudWatch > Logs > Log groups.
2. Select the target log group (for example, /aws/vpc/flowlogs).
3. Open the Subscription filters tab and click Create.
4. Choose Create Amazon Kinesis Data Firehose subscription filter.
5. Provide the following configuration details:
   • Destination: Select delivery stream cwlogs-to-secops.
   • Grant permission: Choose role CWLtoFirehoseRole.
   • Filter name: Enter all-events.
   • Filter pattern: Leave empty to send all events.
6. Click Start streaming.

Option 3: Configure AWS VPC Flow Logs in CSV format using Amazon S3

Transform logs to CSV format (optional)

1. Ensure your CSV rows follow a strict, consistent column order that matches the fields you selected in your VPC Flow Log custom format (for example, the canonical v2 field set, or your v5/v7 set). Do not include a header row in production files unless your parser option expects one.
2. Write CSV files to a stable prefix, for example: s3://<your-bucket>/vpcflowlogs-csv/<region>/year=<year>/month=<month>/day=<day>/.

Configure a feed in Google SecOps to ingest AWS VPC Flow Logs (CSV)

1. Go to SIEM Settings > Feeds.
2. Click + Add New Feed.
3. In the Feed name field, enter a name for the feed (for example, AWS VPC Flow Logs - S3 (CSV)).
4. Select Amazon S3 V2 as the Source type.
5. Select AWS VPC Flow (CSV) as the Log type.
6. Click Next.
7. Specify values for the following input parameters:
   • S3 URI: Enter the S3 bucket address (for example, s3://<your-bucket>/vpcflowlogs-csv/<region>/).
   • Source deletion options: Select deletion option according to your preference.
   • Maximum File Age: Default 180 Days.
   • Access Key ID: User access key with access to the S3 bucket.
   • Secret Access Key: User secret key with access to the S3 bucket.
   • Asset namespace: The asset namespace.
   • Ingestion labels: The label applied to the events from this feed.
8. Click Next.
9. Review your new feed configuration in the Finalize screen, and then click Submit.

UDM mapping table

Need more help? Get answers from Community members and Google SecOps professionals.