Collect AWS IAM logs

This document explains how to ingest AWS IAM logs to Google Security Operations. The parser transforms raw JSON formatted logs into a structured Unified Data Model (UDM). It extracts relevant fields like user details, role information, permissions, and timestamps, mapping them to corresponding UDM fields for consistent security analysis.

Before you begin

• Ensure that you have a Google SecOps instance.
• Ensure that you have privileged access to AWS.

Configure AWS IAM and S3

1. Create an Amazon S3 bucket following this user guide: Creating a bucket.
2. Save the bucket Name and Region for later use.
3. Create a user following this user guide: Creating an IAM user.
4. Select the created User.
5. Select the Security credentials tab.
6. Click Create Access Key in the Access Keys section.
7. Select Third-party service as the Use case.
8. Click Next.
9. Optional: add a description tag.
10. Click Create access key.
11. Click Download CSV file to save the Access Key and Secret Access Key for later use.
12. Click Done.
13. Select the Permissions tab.
14. Click Add permissions in the Permissions policies section.
15. Select Add permissions.
16. Select Attach policies directly
17. Search for and select the AmazonS3FullAccess policy.
18. Click Next.
19. Click Add permissions.

Configure CloudTrail to capture IAM logs

1. Sign in to the AWS Management Console.
2. In the search bar, type and select CloudTrail from the services list.
3. Click Create trail.
4. Provide a Trail name; for example, IAMActivityTrail.
   • Apply trail to all regions: select Yes to capture activities across all regions.
   • Storage location: select the S3 bucket created earlier or create a new one.
   • S3 Bucket: enter a name for the S3 bucket; for example, iam-logs-bucket.
   • Select Create a new IAM role (if not created earlier).
   • Management events: select Read and Write to capture both read and write events on IAM resources.
   • Data events: enable S3 and Lambda data events.
5. Click Create to create the trail.

Configure CloudTrail to Export Logs to S3

1. Go to Services > S3.
2. Select the S3 bucket where CloudTrail logs are stored; for example, iam-logs-bucket.
3. Ensure that CloudTrail has the right permissions to write logs to the bucket.

4. Add the following policy if it's not already present:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "CloudTrailS3Access",
         "Effect": "Allow",
         "Principal": {
           "Service": "cloudtrail.amazonaws.com"
         },
         "Action": "s3:PutObject",
         "Resource": "arn:aws:s3:::your-bucket-name/AWSLogs/*"
       }
     ]
   }
   

5. Enable Versioning on the S3 bucket to ensure that logs are stored with multiple versions.

6. Go to Properties > Bucket Versioning > Enable.

Optional: Configure Lambda for real-time export

1. Go to the AWS Lambda Console.
2. Click Create function.
3. Select Author from Scratch.
4. Set the function name as ExportIAMLogsToS3.
5. Select a Python 3.x runtime.

6. Assign an IAM Role to the function that has permissions to:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "logs:GetLogEvents",
           "logs:FilterLogEvents",
           "logs:DescribeLogGroups",
           "logs:DescribeLogStreams"
         ],
         "Resource": "*"
       },
       {
         "Effect": "Allow",
         "Action": [
           "s3:PutObject"
         ],
         "Resource": "arn:aws:s3:::your-bucket-name/*"
       }
     ]
   }
   

7. Use the following Python code to fetch IAM logs and upload them to S3:

   import boto3
   import gzip
   from io import BytesIO
   
   s3 = boto3.client('s3')
   logs = boto3.client('logs')
   
   def lambda_handler(event, context):
       log_group = event['logGroup']
       log_stream = event['logStream']
   
       log_events = logs.get_log_events(
           logGroupName=log_group,
           logStreamName=log_stream,
           startFromHead=True
       )
   
       log_data = "\n".join([event['message'] for event in log_events['events']])
   
       # Compress and upload to S3
       compressed_data = gzip.compress(log_data.encode('utf-8'))
       s3.put_object(
           Bucket='your-s3-bucket-name',
           Key='iam-logs/{log_stream}.gz',
           Body=compressed_data
       )
   

• Replace your-s3-bucket-name with your actual bucket name.

Configure Lambda Trigger for CloudWatch Logs

1. In the Lambda Console, go to Designer.
2. Choose Add Trigger > CloudWatch Logs.
3. Select the CloudWatch Logs log group associated with your IAM logs; for example, /aws/cloudtrail/.
4. Click Add.

Configure a feed in Google SecOps to ingest AWS IAM logs

1. Go to SIEM Settings > Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed; for example, AWS IAM Logs.
4. Select Amazon S3 as the Source type.
5. Select AWS IAM as the Log type.
6. Click Next.

7. Specify values for the following input parameters:

   • Region: the region where the Amazon S3 bucket is located.
   • S3 URI: the bucket URI.
     • s3://your-log-bucket-name/
       • Replace your-log-bucket-name with the actual name of the bucket.
   • URI is a: select Directory or Directory which includes subdirectories.

   • Source deletion options: select the deletion option according to your preference.

   • Access Key ID: the User access key with access to the s3 bucket.

   • Secret Access Key: the User secret key with access to the s3 bucket.

   • Asset namespace: the asset namespace.

   • Ingestion labels: the label to be applied to the events from this feed.

8. Click Next.

9. Review your new feed configuration in the Finalize screen, and then click Submit.

UDM Mapping Table

Changes

2023-12-14

• Newly created parser.

Need more help? Get answers from Community members and Google SecOps professionals.