Collect AWS EC2 Instance logs

This document explains how to configure AWS EC2 Instance logs into Google Security Operations for monitoring and analysis. The parser extracts data from instance reservation JSON logs, restructures and renames fields to conform to the UDM, handling various data types and nested structures, including network interfaces, groups, and tags, while also generating asset relationships and metadata. It also performs error handling and dropping malformed JSON messages.

Before you begin

• Ensure that you have a Google SecOps instance.
• Ensure that you have privileged access to AWS.

Configure AWS IAM and S3

1. Create an Amazon S3 bucket following this user guide: Creating a bucket.
2. Save the bucket Name and Region for later use.
3. Create a user following this user guide: Creating an IAM user.
4. Select the created User.
5. Select the Security credentials tab.
6. Click Create Access Key in the Access Keys section.
7. Select Third-party service as the Use case.
8. Click Next.
9. Optional: add a description tag.
10. Click Create access key.
11. Click Download CSV file to save the Access Key and Secret Access Key for later use.
12. Click Done.
13. Select the Permissions tab.
14. Click Add permissions in the Permissions policies section.
15. Select Add permissions.
16. Select Attach policies directly
17. Search for and select the AmazonS3FullAccess policy.
18. Click Next.
19. Click Add permissions.

Configure EC2 to send logs to CloudWatch Logs

1. Use SSH to connect to your EC2 instance, providing your key pair for authentication.

   ssh -i your-key.pem ec2-user@your-ec2-public-ip
   

2. Install the CloudWatch Logs agent:

   • To install the CloudWatch Logs agent on Amazon Linux, use the following command:

   sudo yum install -y awslogs
   

   • To install the CloudWatch Logs agent on Ubuntu, use the following command:

   sudo apt-get install -y awslogs
   

3. Open the CloudWatch Logs configuration file:

   sudo vi /etc/awslogs/awslogs.conf
   

4. Create a script that fetches this Log Instance Metadata and writes it to a file:

   #!/bin/bash
   echo "Architecture: $(curl -s http://169.254.169.254/latest/meta-data/architecture)" >> /var/log/instance_metadata.log
   echo "AmiLaunchIndex: $(curl -s http://169.254.169.254/latest/meta-data/ami-launch-index)" >> /var/log/instance_metadata.log
   echo "BootMode: $(curl -s http://169.254.169.254/latest/meta-data/boot-mode)" >> /var/log/instance_metadata.log
   

5. Save the script as /etc/init.d/metadata_script.sh and run it at instance startup using crontab or rc.local.

6. Open the configuration file for the CloudWatch Logs agent:

   sudo vi /etc/awslogs/awslogs.conf
   

7. Add the following to the configuration file:

   [/var/log/messages]
   file = /var/log/messages
   log_group_name = /ec2/system/logs
   log_stream_name = {instance_id}
   
   [/var/log/secure]
   file = /var/log/secure
   log_group_name = /ec2/security/logs
   log_stream_name = {instance_id}
   
   [/var/log/auth.log]
   file = /var/log/auth.log
   log_group_name = /ec2/auth/logs
   log_stream_name = {instance_id}
   
   [/var/log/httpd/access_log]
   file = /var/log/httpd/access_log
   log_group_name = /ec2/application/apache/access_logs
   log_stream_name = {instance_id}
   
   [/var/log/httpd/error_log]
   file = /var/log/httpd/error_log
   log_group_name = /ec2/application/apache/error_logs
   log_stream_name = {instance_id}
   

8. Save the configuration and exit the editor.

9. Start the CloudWatch Logs agent:

   • On Amazon Linux:

   sudo service awslogs start
   

   • On Ubuntu:

   sudo service awslogs start
   

10. Verify that the agent is running:

    sudo service awslogs status
    

Configure IAM Permissions for Lambda and S3

1. In the AWS IAM console, create a new IAM role with the following permissions:

   • logs:PutSubscriptionFilter
   • logs:DescribeLogGroups
   • logs:GetLogEvents
   • s3:PutObject

2. Attach this role to your Lambda function that will export the logs to S3.

Configure Lambda to Export Logs to S3

1. Go to the Lambda console and create a new function.

   import boto3
   import gzip
   from io import BytesIO
   
   s3 = boto3.client('s3')
   logs = boto3.client('logs')
   
   def lambda_handler(event, context):
       log_group = event['logGroup']
       log_stream = event['logStream']
   
       log_events = logs.get_log_events(
           logGroupName=log_group,
           logStreamName=log_stream,
           startFromHead=True
       )
   
       log_data = "\n".join([event['message'] for event in log_events['events']])
   
       # Compress and upload to S3
       compressed_data = gzip.compress(log_data.encode('utf-8'))
       s3.put_object(
           Bucket='your-s3-bucket-name',
           Key='logs/ec2-log.gz',
           Body=compressed_data
       )
     ```
   

   • Replace your-s3-bucket-name with the actual name of your S3 bucket.

2. Attach the IAM role to the Lambda function created earlier.

3. In the CloudWatch console, go to the Logs section.

4. Select the log group; for example, /ec2/system/logs.

5. Click Actions > Create Subscription Filter.

6. Set the destination to the Lambda function created previously.

Configure a feed in Google SecOps to ingest AWS EC2 Instance logs

1. Go to SIEM Settings > Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed; for example, AWS EC2 Instance Logs.
4. Select Amazon S3 as the Source type.
5. Select AWS EC2 Instance as the Log type.
6. Click Next.

7. Specify values for the following input parameters:

   • Region: the region where the Amazon S3 bucket is located.
   • S3 URI: the bucket URI.
     • s3://your-log-bucket-name/
       • Replace your-log-bucket-name with the actual name of the bucket.
   • URI is a: select Directory or Directory which includes subdirectories.

   • Source deletion options: select the deletion option according to your preference.

   • Access Key ID: the User access key with access to the s3 bucket.

   • Secret Access Key: the User secret key with access to the s3 bucket.

   • Asset namespace: the asset namespace.

   • Ingestion labels: the label to be applied to the events from this feed.

8. Click Next.

9. Review your new feed configuration in the Finalize screen, and then click Submit.

UDM Mapping Table

Changes

2024-01-31

• Added support for new schema.

2023-12-14

• Newly created parser.

Need more help? Get answers from Community members and Google SecOps professionals.