Collect Elastic Packet Beats logs

This document explains how to ingest Elastic Packet Beats logs to Google Security Operations using Bindplane. The parser first initializes default values for various fields found in Elastic Packet Beats logs. Then, it extracts data from the log messages using a combination of grok patterns and json filters, performs data type conversions, and maps the extracted fields to the corresponding fields in the Unified Data Model (UDM) based on the event dataset type (for example, flow, dns, http, tls, dhcpv4).

Before you begin

Make sure you have the following prerequisites:

• A Google SecOps instance.
• A Windows 2016 or later, or Linux host with systemd.
• If running behind a proxy, make sure firewall ports are open per the Bindplane agent requirements.
• Privileged access to the Elastic Packet Beats management console or appliance.
• Logstash installed and configured.

Get Google SecOps ingestion authentication file

1. Sign in to the Google SecOps console.
2. Go to SIEM Settings > Collection Agents.
3. Download the Ingestion Authentication File.
   • Save the file securely on the system where Bindplane will be installed.

Get Google SecOps customer ID

1. Sign in to the Google SecOps console.
2. Go to SIEM Settings > Profile.
3. Copy and save the Customer ID from the Organization Details section.

Install the Bindplane agent

Install the Bindplane agent on your Windows or Linux operating system according to the following instructions.

Windows installation

1. Open the Command Prompt or PowerShell as an administrator.

2. Run the following command:

   msiexec /i "https://github.com/observIQ/bindplane-agent/releases/latest/download/observiq-otel-collector.msi" /quiet
   

Linux installation

1. Open a terminal with root or sudo privileges.

2. Run the following command:

   sudo sh -c "$(curl -fsSlL https://github.com/observiq/bindplane-agent/releases/latest/download/install_unix.sh)" install_unix.sh
   

Additional installation resources

For additional installation options, consult the installation guide.

Configure the Bindplane agent to ingest Syslog and send to Google SecOps

1. Access the configuration file:

   • Locate the config.yaml file. Typically, it's in the /etc/bindplane-agent/ directory on Linux or in the installation directory on Windows.
   • Open the file using a text editor (for example, nano, vi, or Notepad).

2. Edit the config.yaml file as follows:

   receivers:
     udplog:
       # Replace the port and IP address as required
       listen_address: "0.0.0.0:514"
   
   exporters:
     chronicle/chronicle_w_labels:
       compression: gzip
       # Adjust the path to the credentials file you downloaded in Step 1
       creds_file_path: '/path/to/ingestion-authentication-file.json'
       # Replace with your actual customer ID from Step 2
       customer_id: YOUR_CUSTOMER_ID
       endpoint: malachiteingestion-pa.googleapis.com
       # Add optional ingestion labels for better organization
       log_type: 'ELASTIC_PACKETBEATS'
       raw_log_field: body
       ingestion_labels:
   
   service:
     pipelines:
       logs/source0__chronicle_w_labels-0:
         receivers:
           - udplog
         exporters:
           - chronicle/chronicle_w_labels
   

   • Replace the port and IP address as required in your infrastructure.
   • Replace YOUR_CUSTOMER_ID with the actual customer ID.
   • Update /path/to/ingestion-authentication-file.json to the path where the authentication file was saved in the Get Google SecOps ingestion authentication file section.

Restart the Bindplane agent to apply the changes

• To restart the Bindplane agent in Linux, run the following command:

  sudo systemctl restart observiq-otel-collector
  

• To restart the Bindplane agent in Windows, you can either use the Services console, or enter the following command:

  net stop observiq-otel-collector && net start observiq-otel-collector
  

Configure Syslog forwarding on Elastic Packet Beats

Since Packetbeat does not support direct syslog output, you must use Logstash as an intermediary.

Configure Packetbeat to send logs to Logstash

1. Sign in to the Elastic Packet Beats Management Console.
2. Go to Settings > Log Forwarding.
3. Click the + Add or Enable button.
4. Provide the following configuration details:
   • Name: Enter a descriptive name (for example, Logstash Output).
   • Host: Enter the Logstash server IP address.
   • Port: Enter the Logstash beats input port (typically 5044).
   • Protocol: Select Beats protocol.
   • Format: Select JSON.
   • Timezone: Select UTC time zone for universal consistency across systems.
   • Go to the Events section and select the relevant log types or all.

5. Save the configuration.

   Alternative: Edit packetbeat.yml directly:

   # /etc/packetbeat/packetbeat.yml
   packetbeat.protocols:
     - type: dns
       ports: [53]
     - type: http
       ports: [80, 8080, 8000, 5000, 8002]
       send_headers: true
       send_all_headers: true
     - type: tls
       ports: [443, 993, 995, 5223, 8443, 8883, 9243]
     - type: dhcpv4
       ports: [67, 68]
   
   # Enable processors for additional fields
   processors:
     - add_network_direction:
         source: private
         destination: private
         internal_networks:
           - private
     - community_id:
   
   # Send to Logstash using beats protocol
   output.logstash:
     hosts: ["LOGSTASH_IP:5044"]
   

   Replace LOGSTASH_IP with your Logstash server's IP address.

Configure Logstash to forward to BindPlane using Syslog

1. Create a Logstash pipeline configuration file:

   sudo nano /etc/logstash/conf.d/packetbeat-to-bindplane.conf
   

2. Add the following configuration:

   # Receive from Packetbeat
   input {
     beats {
       port => 5044
     }
   }
   
   # Optional: Add filters for data enrichment
   filter {
     # Preserve original message structure
     mutate {
       copy => { "@metadata" => "[@metadata_backup]" }
     }
   }
   
   # Send to BindPlane via syslog
   output {
     syslog {
       host => "BINDPLANE_IP"
       port => 514
       protocol => "udp"
       rfc => "rfc5424"
       facility => "local0"
       severity => "informational"
       sourcehost => "%{[agent][hostname]}"
       appname => "packetbeat"
       procid => "%{[agent][id]}"
       msgid => "ELASTIC_PACKETBEATS"
       structured_data => "packetbeat@32473"
       message => "%{message}"
     }
   }
   

   Replace BINDPLANE_IP with your BindPlane agent's IP address.

• Restart Logstash to apply the configuration:

  sudo systemctl restart logstash
  

UDM mapping table

Need more help? Get answers from Community members and Google SecOps professionals.