Collect AWS EC2 Hosts logs

This document explains how to integrate AWS EC2 Hosts logs into Google Security Operations for monitoring and analysis. The integration involves parsing and mapping EC2 host logs to the Unified Data Model (UDM), performing data transformation, and creating relationships between EC2 hosts and instances. The logs provide valuable information about the instances, host properties, instance types, and performance metrics that can be used for security monitoring, audit, and compliance.

Before you begin

• Ensure that you have a Google SecOps instance.
• Ensure that you have privileged access to AWS.

Configure AWS IAM and S3

1. Create an Amazon S3 bucket following this user guide: Creating a bucket.
2. Save the bucket Name and Region for later use.
3. Create a user following this user guide: Creating an IAM user.
4. Select the created User.
5. Select the Security credentials tab.
6. Click Create Access Key in the Access Keys section.
7. Select Third-party service as the Use case.
8. Click Next.
9. Optional: add a description tag.
10. Click Create access key.
11. Click Download CSV file to save the Access Key and Secret Access Key for later use.
12. Click Done.
13. Select the Permissions tab.
14. Click Add permissions in the Permissions policies section.
15. Select Add permissions.
16. Select Attach policies directly
17. Search for and select the AmazonS3FullAccess policy.
18. Click Next.
19. Click Add permissions.

Configure CloudTrail for AWS KMS

1. Sign in to the AWS Management Console.
2. In the search bar, type and select CloudTrail from the services list.
3. Click Create trail.
4. Provide a Trail name; for example, EC2-Activity-Trail.
5. Select the Enable for all accounts in my organization checkbox.
6. Type the S3 bucket URI created earlier (the format should be: s3://your-log-bucket-name/), or create a new S3 bucket.
7. If SSE-KMS is enabled, provide a name for AWS KMS alias, or choose an existing AWS KMS Key.
8. You can leave the other settings as default.
9. Click Next.
10. Select Management events and Data events under Event Types to capture EC2 host activity.
11. Click Next.
12. Review the settings in Review and create.
13. Click Create trail.
14. Optional: if you created a new bucket, continue with the following process:
    • Go to S3.
    • Identify and select the newly created log bucket.
    • Select the folder AWSLogs.
    • Click Copy S3 URI and save it.

Configure a feed in Google SecOps to ingest AWS EC2 Hosts

1. Go to SIEM Settings > Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed; for example, AWS EC2 Hosts Logs.
4. Select Amazon S3 as the Source type.
5. Select AWS EC2 Hosts as the Log type.
6. Click Next.

7. Specify values for the following input parameters:

   • Region: the region where the Amazon S3 bucket is located.
   • S3 URI: the bucket URI.
     • s3://your-log-bucket-name/
       • Replace your-log-bucket-name with the actual name of the bucket.
   • URI is a: select Directory or Directory which includes subdirectories.

   • Source deletion options: select the deletion option according to your preference.

   • Access Key ID: the User access key with access to the s3 bucket.

   • Secret Access Key: the User secret key with access to the s3 bucket.

   • Asset namespace: the asset namespace.

   • Ingestion labels: the label to be applied to the events from this feed.

8. Click Next.

9. Review your new feed configuration in the Finalize screen, and then click Submit.

UDM Mapping Table

Changes

2024-01-31

• Added support for new schema.

2023-12-14

• Newly created parser.

Need more help? Get answers from Community members and Google SecOps professionals.