Collect Elastic Auditbeat logs

This document explains how to ingest Elastic Auditbeat logs to Google Security Operations using Amazon S3. The parser extracts fields from the JSON logs, normalizes them into the Unified Data Model (UDM), and enriches the data with additional context like host information, network details, and security result classifications. It handles various event types by mapping event1.action and other fields to specific UDM metadata event types, defaulting to GENERIC_EVENT or more specific categories when possible.

Before you begin

Make sure you have the following prerequisites:

• A Google SecOps instance.
• Privileged access to Elastic Auditbeat server.
• Privileged access to Logstash server.
• Privileged access to AWS (S3, Identity and Access Management (IAM)).

Get Elastic Auditbeat prerequisites

1. Make sure Elastic Auditbeat is installed and configured on your servers.
2. Install Logstash on a dedicated server or alongside Auditbeat.
3. Note the Auditbeat configuration file location (typically /etc/auditbeat/auditbeat.yml).

Configure AWS S3 bucket and IAM for Google SecOps

1. Create Amazon S3 bucket following this user guide: Creating a bucket
2. Save bucket Name and Region for future reference (for example, elastic-auditbeat-logs).
3. Create a User following this user guide: Creating an IAM user.
4. Select the created User.
5. Select Security credentials tab.
6. Click Create Access Key in section Access Keys.
7. Select Third-party service as Use case.
8. Click Next.
9. Optional: Add description tag.
10. Click Create access key.
11. Click Download .CSV file to save the Access Key and Secret Access Key for future reference.
12. Click Done.
13. Select Permissions tab.
14. Click Add permissions in section Permissions policies.
15. Select Add permissions.
16. Select Attach policies directly.
17. Search for AmazonS3FullAccess policy.
18. Select the policy.
19. Click Next.
20. Click Add permissions.

Configure Auditbeat to send to Logstash

1. Edit the Auditbeat configuration file /etc/auditbeat/auditbeat.yml.
2. Comment out any existing output configuration (Elasticsearch, etc.).

3. Add the Logstash output configuration:

   # ==================== Outputs ====================
   output.logstash:
     hosts: ["localhost:5044"]
     # If Logstash is on a different server, use its IP/hostname
     # hosts: ["logstash-server:5044"]
   
     # Optional: Enable load balancing if using multiple Logstash instances
     loadbalance: true
   
     # Optional: Configure bulk settings (default is 2048)
     bulk_max_size: 2048
   
     # Optional: Configure SSL if needed
     # ssl.enabled: true
     # ssl.certificate_authorities: ["/path/to/ca.crt"]
   

4. Restart Auditbeat to apply changes:

   sudo systemctl restart auditbeat
   

Configure Logstash pipeline

1. Create a new Logstash pipeline configuration file /etc/logstash/conf.d/auditbeat-to-s3.conf:

   input {
     beats {
       port => 5044
       # Optional: Configure SSL
       # ssl => true
       # ssl_certificate => "/path/to/server.crt"
       # ssl_key => "/path/to/server.key"
     }
   }
   
   filter {
     # Add any necessary transformations here
     # The data should remain in raw JSON format for Chronicle parsing
   
     # Optional: Add metadata for debugging
     mutate {
       add_field => { "[@metadata][pipeline]" => "auditbeat-to-s3" }
     }
   }
   
   output {
     s3 {
       # AWS credentials
       access_key_id => "YOUR_AWS_ACCESS_KEY_ID"
       secret_access_key => "YOUR_AWS_SECRET_ACCESS_KEY"
   
       # S3 bucket configuration
       region => "us-east-1"  # Replace with your bucket region
       bucket => "elastic-auditbeat-logs"  # Replace with your bucket name
   
       # Organize logs by date using Logstash timestamp interpolation
       prefix => "auditbeat/%{+YYYY}/%{+MM}/%{+dd}/"
   
       # File rotation settings
       size_file => 10485760  # 10MB files
       time_file => 5  # Rotate every 5 minutes
   
       # Compression for cost optimization
       encoding => "gzip"
   
       # Output format - keep as JSON for Chronicle
       codec => "json_lines"
   
       # Optional: Server-side encryption
       # server_side_encryption => true
       # server_side_encryption_algorithm => "AES256"
     }
   
     # Optional: Keep a local copy for debugging
     # stdout { 
     #   codec => rubydebug 
     # }
   }
   

   • Replace YOUR_AWS_ACCESS_KEY_ID and YOUR_AWS_SECRET_ACCESS_KEY with your actual AWS credentials.
   • Update the region and bucket values to match your S3 configuration.
   • Start or restart Logstash:

   sudo systemctl restart logstash
   

(Optional) Create read-only IAM user & keys for Google SecOps

1. Go to AWS Console > IAM > Users.
2. Click Add users.
3. Provide the following configuration details:
   • User: Enter secops-reader.
   • Access type: Select Access key – Programmatic access.
4. Click Create user.
5. Attach minimal read policy (custom): Users > secops-reader > Permissions > Add permissions > Attach policies directly > Create policy.

6. JSON:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": ["s3:GetObject"],
         "Resource": "arn:aws:s3:::elastic-auditbeat-logs/*"
       },
       {
         "Effect": "Allow",
         "Action": ["s3:ListBucket"],
         "Resource": "arn:aws:s3:::elastic-auditbeat-logs"
       }
     ]
   }
   

7. Name = secops-reader-policy.

8. Click Create policy > search/select > Next > Add permissions.

9. Create access key for secops-reader: Security credentials > Access keys.

10. Click Create access key.

11. Download the .CSV. (You'll paste these values into the feed).

Configure a feed in Google SecOps to ingest Elastic Auditbeat logs

1. Go to SIEM Settings > Feeds.
2. Click + Add New Feed.
3. In the Feed name field, enter a name for the feed (for example, Elastic Auditbeat Logs).
4. Select Amazon S3 V2 as the Source type.
5. Select Elastic Audit Beats as the Log type.
6. Click Next.
7. Specify values for the following input parameters:
   • S3 URI: s3://elastic-auditbeat-logs/auditbeat/
   • Source deletion options: Select deletion option according to your preference.
   • Maximum File Age: Include files modified in the last number of days. Default is 180 days.
   • Access Key ID: User access key with access to the S3 bucket.
   • Secret Access Key: User secret key with access to the S3 bucket.
   • Asset namespace: The asset namespace.
   • Ingestion labels: The label applied to the events from this feed.
8. Click Next.
9. Review your new feed configuration in the Finalize screen, and then click Submit.

UDM mapping table

Need more help? Get answers from Community members and Google SecOps professionals.