Collect Linux auditd and AIX systems logs

Supported in:

This parser handles Linux audit logs in SYSLOG format, transforming them into UDM. It processes both JSON-formatted and plain text log messages, extracting fields using grok, XML parsing, and JSON parsing techniques, and mapping them to appropriate UDM fields based on the event type. The parser also handles specific audit log formats from AIX systems and enriches the UDM with additional fields like security_result and intermediary details.

Before you begin

  • Ensure that you have a Google Security Operations instance.
  • Ensure that you have root access to the Auditd host.
  • Ensure that you installed rsyslog on the Auditd host.
  • Ensure that you have a Windows 2012 SP2 or later or Linux host with systemd.
  • If running behind a proxy, ensure that the firewall ports are open.

Get Google SecOps ingestion authentication file

  1. Sign in to the Google SecOps console.
  2. Go to SIEM Settings > Collection Agents.
  3. Download the Ingestion Authentication File.

Get Google SecOps customer ID

  1. Sign in to the Google SecOps console.
  2. Go to SIEM Settings > Profile.
  3. Copy and save the Customer ID from the Organization Details section.

Install BindPlane Agent

  1. For Windows installation, run the following script:
    msiexec /i "https://github.com/observIQ/bindplane-agent/releases/latest/download/observiq-otel-collector.msi" /quiet
  2. For Linux installation, run the following script:
    sudo sh -c "$(curl -fsSlL https://github.com/observiq/bindplane-agent/releases/latest/download/install_unix.sh)" install_unix.sh
  3. Additional installation options can be found in this installation guide.

Configure BindPlane Agent to ingest Syslog and send to Google SecOps

  1. Access the machine where BindPlane is installed.

  2. Edit the config.yaml file as follows:

    receivers:
  tcplog:
    # Replace the below port <54525> and IP <0.0.0.0> with your specific values
    listen_address: "0.0.0.0:54525" 

exporters:
    chronicle/chronicle_w_labels:
        compression: gzip
        # Adjust the creds location below according the placement of the credentials file you downloaded
        creds: '{ json file for creds }'
        # Replace <customer_id> below with your actual ID that you copied
        customer_id: <customer_id>
        endpoint: malachiteingestion-pa.googleapis.com
        # You can apply ingestion labels below as preferred
        ingestion_labels:
        log_type: SYSLOG
        namespace: auditd
        raw_log_field: body
service:
    pipelines:
        logs/source0__chronicle_w_labels-0:
            receivers:
                - tcplog
            exporters:
                - chronicle/chronicle_w_labels

  3. Restart BindPlane Agent to apply the changes using the following command: sudo systemctl bindplane restart

Exporting Syslog from Auditd

  1. Access the machine from which you want to export audit logs.

  2. Open the Auditd configuration file (typically located at /etc/audit/auditd.conf).

    sudo vi /etc/audit/auditd.conf

  3. Find or add the following line(s) to configure auditd:

    active = yes
output = syslog
log_format = ENRICHED
dispatcher = /sbin/audispd

Optional: Specify Syslog Facility: Add or modify the following line in auditd.conf:

```none
syslog_facility = LOG_AUTHPRIV
```

  1. Open audispd configuration file (typically located at /etc/audisp/plugins.d/syslog.conf):

    sudo vi /etc/audisp/plugins.d/syslog.conf

  2. Find or add the following line(s) to configure audispd:

    active = yes
direction = out
path = builtin_syslog
type = builtin
args = LOG_INFO
format = string

  3. Restart Auditd service to apply the changes:

    sudo systemctl restart auditd

  4. Use a tool like tail to monitor the syslog and verify that Auditd logs are being sent:

    tail -f /var/log/syslog | grep auditd # Follow syslog and filter for auditd messages (path may vary depending on your system)

  5. Edit the rsyslog.conf or create a custom configuration:

    sudo vi /etc/rsyslog.d/50-audit-forwarding.conf

  6. Add a rule to forward logs:

    • if $programname == 'auditd' then @@<BindPlane_Agent>:<BindPlane_Port>
    • Use @for UDP or @@ for TCP
    • Replace <BindPlane_Agent> with your server's IP/hostname.
    • Replace <BindPlane_Port> with your server's Port.

  7. Restart rsyslog service to apply the changes:

    sudo systemctl restart rsyslog

UDM Mapping Table

Log Field UDM Mapping Logic
acct target.user.user_display_name The value of acct from the raw log is mapped to the target.user.user_display_name field in the UDM. This represents the account associated with the event.
addr principal.ip The value of addr from the raw log is mapped to the principal.ip field in the UDM. This represents the IP address of the principal involved in the event.
additional.fields additional.fields Additional fields from parsed key-value pairs or labels are added to the additional.fields array in the UDM.
agent.googleapis.com/log_file_path (Not Mapped) This label is present in some raw logs but is not mapped to the IDM object in the UDM.
algo (Not used in this example) Although present in the parser and some raw logs, this field isn't used in the provided example and doesn't appear in the final UDM.
application principal.application Derived from the terminal field in the raw log or other fields like exe depending on the log type. Represents the application involved.
arch security_result.about.platform_version The architecture from the raw log's arch field is mapped to security_result.about.platform_version.
auid about.user.userid, security_result.detection_fields.auid The audit user ID (auid) is mapped to about.user.userid and added as a detection field in security_result.
cmd target.process.command_line The command from the raw log's cmd field is mapped to target.process.command_line.
collection_time (Not Mapped) This field is the log collection time and is not mapped to the IDM object in the UDM.
comm principal.application The command name (comm) is mapped to principal.application.
compute.googleapis.com/resource_name principal.hostname The resource name from this label is mapped to principal.hostname.
create_time (Not Mapped) This field is not mapped to the IDM object in the UDM.
cwd security_result.detection_fields.cwd The current working directory (cwd) is added as a detection field in security_result.
data (Processed) The data field contains the main log message and is processed by the parser to extract various fields. It is not directly mapped to a single UDM field.
exe principal.process.file.full_path, target.process.file.full_path The executable path (exe) is mapped to either principal.process.file.full_path or target.process.file.full_path depending on the context.
extensions.auth.type extensions.auth.type The authentication type is set by the parser logic based on the event type. Often set to MACHINE or AUTHTYPE_UNSPECIFIED.
fp network.tls.client.certificate.sha256 The fingerprint (fp) is parsed to extract the SHA256 hash and mapped to network.tls.client.certificate.sha256.
insertId (Not Mapped) This field is not mapped to the IDM object in the UDM.
jsonPayload.message (Processed) This field contains the main log message in JSON format and is processed by the parser.
key security_result.about.registry.registry_key The key field is mapped to security_result.about.registry.registry_key.
labels (Processed) Labels from the raw log are processed and mapped to various UDM fields or added to additional.fields.
logName (Not Mapped) This field is not mapped to the IDM object in the UDM.
msg security_result.summary The message (msg) is often used to populate the security_result.summary field.
network.application_protocol network.application_protocol Set by the parser logic based on the event type (e.g., SSH, HTTP).
network.direction network.direction Set by the parser logic based on the event type (e.g., INBOUND, OUTBOUND).
network.ip_protocol network.ip_protocol Set by the parser logic, usually to TCP for SSH events.
network.session_id network.session_id Mapped from the ses field or derived from other fields.
network.tls.cipher network.tls.cipher The cipher information is extracted from the raw log and mapped to this field.
network.tls.curve network.tls.curve The key exchange curve is extracted from the raw log and mapped to this field.
pid principal.process.pid, target.process.pid The process ID (pid) is mapped to either principal.process.pid or target.process.pid depending on the context.
ppid principal.process.parent_process.pid, target.process.parent_process.pid The parent process ID (ppid) is mapped to either principal.process.parent_process.pid or target.process.parent_process.pid depending on the context.
principal.asset.hostname principal.asset.hostname Copied from principal.hostname.
principal.asset.ip principal.asset.ip Copied from principal.ip.
principal.platform principal.platform Set by the parser logic based on the operating system (e.g., LINUX).
principal.port principal.port The port number associated with the principal.
principal.user.group_identifiers principal.user.group_identifiers Group IDs associated with the principal user.
receiveTimestamp (Not Mapped) This field is the log receive timestamp and is not mapped to the IDM object in the UDM.
res security_result.action_details The result (res) is mapped to security_result.action_details.
resource.labels (Not Mapped) These labels are present in some raw logs but are not mapped to the IDM object in the UDM.
resource.type (Not Mapped) This field is present in some raw logs but is not mapped to the IDM object in the UDM.
security_result.action security_result.action Set by the parser logic based on the res field (e.g., ALLOW, BLOCK).
security_result.detection_fields security_result.detection_fields Various fields from the raw log are added as key-value pairs to this array for context.
security_result.rule_id security_result.rule_id Set by the parser logic, often to the type_name for syscall events.
security_result.severity security_result.severity Set by the parser logic based on the severity level in the raw log.
security_result.summary security_result.summary A summary of the event, often derived from the msg field or other relevant fields.
ses network.session_id The session ID (ses) is mapped to network.session_id.
source (Not Mapped) This field contains metadata about the log source and is not mapped to the IDM object in the UDM.
subj (Processed) The subject field (subj) is processed to extract user and security context information.
syscall security_result.about.labels.Syscall The syscall number is added as a label within security_result.about.
target.administrative_domain target.administrative_domain The domain of the target user.
target.group.group_display_name target.group.group_display_name The name of the target group.
target.ip target.ip The IP address of the target.
target.port target.port The port number associated with the target.
target.process.command_line target.process.command_line The command line of the target process.
target.resource.type target.resource.type The type of the target resource, set by the parser logic (e.g., CREDENTIAL, SETTING).
target.user.attribute.permissions target.user.attribute.permissions Permissions related to the target user.
target.user.group_identifiers target.user.group_identifiers Group IDs associated with the target user.
target.user.userid target.user.userid The user ID of the target.
textPayload (Processed) The text payload of the log, processed by the parser to extract various fields.
timestamp metadata.event_timestamp The timestamp of the event.
tty security_result.about.labels.tty The tty is added as a label within security_result.about.
type metadata.product_event_type The event type (type) is mapped to metadata.product_event_type.
uid principal.user.userid, target.user.userid The user ID (uid) is mapped to either principal.user.userid or target.user.userid depending on the context.

Changes

2024-05-08

  • When the value is not "?", then mapped "field" to "field33" to "security_result.detection_fields".
  • When "type_name" is "CRYPTO_KEY_USER", then mapped "exe" to "principal.process.file.full_path".
  • When "type_name" is "CRYPTO_KEY_USER", then mapped "fp" to "network.tls.client.certificate.sha256".
  • When "type_name" is "CRYPTO_KEY_USER", then mapped "pid" to "principal.process.pid".
  • Added Grok patterns to parse new pattern of logs.
  • Mapped "syslog-tag" to "security_result.detection_fields".
  • Mapped "inter_ip" to "intermediary.ip".
  • Mapped "inter_hostname" to "intermediary.hostname".

2024-05-02

  • When "type_name" is "USER_MGMT", then mapped "grp" to "target.group.group_display_name".
  • When "type_name" is "USER_MGMT", then changed mapping of "uid" from "principal.user.userid" to "target.user.userid".
  • When "type_name" is "USER_MGMT" and "op" is equal to "deleting-user-from-group", then set "metadata.event_type" to "GROUP_MODIFICATION".
  • When "type_name" is "USER_MGMT", then changed mapping of "exe" from "target.process.file.full_path" to "principal.process.file.full_path".
  • When "type_name" is "USER_MGMT", then mapped "id" to "about.user.userid".

2024-04-08

  • When "type_name" is "ADD_USER", principal_user_present is "true", target_user_present is "true", and has_principal is "true", then set "metadata.event_type" to "USER_CREATION".
  • When "type_name" is "USER_AUTH", then mapped "acct" to "target.user.user_display_name".
  • When "type_name" is "USER_AUTH", then mapped "uid" to "principal.user.userid".
  • When "type_name" is not in "ADD_USER","USER_AUTH","CRED_ACQ", and "USER_MGMT", then mapped "auid" to "about.user.userid".
  • When "type_name" is "ADD_USER", then mapped "auid" to "target.user.userid".
  • When "type_name" is "ADD_USER" or "USER_AUTH" then mapped "exe" to "principal.process.file.full_path".
  • When "type_name" is "ADD_USER", then mapped "op" and "id" to "security_result.summary".
  • When "type_name" is "USER_AUTH", then mapped "op" and "acct" to "security_result.summary".

2024-03-22

  • Added support for new pattern of JSON logs.
  • Mapped "labels.compute.googleapis.com/resource_name","jsonPayload._HOSTNAME" , "CollectorHostName", "HOSTNAME", and "Computer" to "principal.hostname".
  • Mapped "HostIP" to "principal.ip".
  • Mapped "ProcessID" and "jsonPayload._PID" to "principal.process.pid".
  • Mapped "SyslogMessage" to "metadata.description".
  • Mapped "TenantId", "_ItemId", "_Internal_WorkspaceResourceId", "_ResourceId", and "Facility" to "additional.fields".
  • Mapped "SeverityLevel" to "security_result.severity".
  • Mapped "SourceSystem" to "principal.platform".
  • Mapped "jsonPayload._COMM" to "principal.application".
  • Mapped "jsonPayload._EXE" to "target.process.file.full_path".
  • Mapped "jsonPayload._AUDIT_FIELD_FILE" to "target.file.full_path".
  • Mapped "jsonPayload._AUDIT_FIELD_HASH" to "target.file.hash".
  • Mapped "jsonPayload._AUDIT_SESSION" to "network.session_id".
  • Mapped "jsonPayload._PPID" to "principal.process.parent_process.pid".
  • Mapped "jsonPayload._AUDIT_FIELD_A0", "jsonPayload._AUDIT_FIELD_A1", "jsonPayload._AUDIT_FIELD_A2", "jsonPayload._AUDIT_FIELD_A3", "jsonPayload._BOOT_ID", and "jsonPayload._AUDIT_FIELD_EXIT" to "security_result.detection_fields".

2023-11-27

  • Added validation check to ensure either "principal_user_present", "target_user_present" or "has_principal" is true before setting "metadata.event_type" to "USER_LOGIN".
  • Added validation check to ensure either "principal_user_present", "target_user_present" or "has_principal" is true before setting "metadata.event_type" to "USER_LOGOUT".
  • Added validation check to ensure either "principal_user_present", "target_user_present" or "has_principal" is true before setting "metadata.event_type" to "USER_CREATION".
  • Added validation check to ensure either "principal_user_present", "target_user_present" or "has_principal" is true before setting "metadata.event_type" to "USER_DELETION".
  • Added validation check to ensure either "principal_user_present", "target_user_present" or "has_principal" is true before setting "metadata.event_type" to "USER_UNCATEGORIZED".
  • Added validation check to ensure either "principal_user_present", "target_user_present" or "has_principal" is true before setting "metadata.event_type" to "USER_RESOURCE_ACCESS".
  • Added validation check to ensure either "principal_user_present", "target_user_present" or "has_principal" is true before setting "metadata.event_type" to "USER_CHANGE_PERMISSIONS".
  • When user details are present and principal machine details are not present then changed mapping of "metadata.event_type" from "USER_CREATION" to "USER_UNCATEGORIZED".
  • When user details are present and principal machine details are not present then changed mapping of "metadata.event_type" from "USER_DELETION" to "USER_UNCATEGORIZED".

2023-09-06

  • Added mapping of "CMD" to "target.process.command_line" for "cron daemon(CROND)".

2023-06-20

  • Enhancement - Added or modified the following mappings when type="ADD_USER" and "DEL_USER"-
  • Modified the mapping of "uid" from "target.user.userid" to "principal.user.userid".
  • Mapped "id" to "target.user.userid".
  • Mapped "ID" to "target.user.user_display_name".
  • Modified the mapping of "UID" from "principal.user.userid" to "principal.user.user_display_name".
  • Modified the mapping of "acct" from "principal.user.user_display_name" to "target.user.user_display_name" and "target.user.userid".

2023-06-09

  • Enhancement - Modified "event_type" from "USER_LOGIN" to "USER_CREATION" when "type=ADD_USER".

2023-04-17

  • Enhancement
  • Added gsub function to replace "GS - Group separator" character which is breaking the JSON construction.

2023-04-10

  • Enhancement
  • Added 'gid','euid','egid','suid','fsuid','sgid','fsgid','tty','items' fields to security_result.detection_fields.
  • Additionally mapped 'gid' to 'principal.user.group_identifiers'.
  • Mapped 'euid' to 'target.user.userid'.
  • Mapped 'egid' to 'target.user.group_identifiers'.

2023-03-27

  • Enhancement - Added support for "jsonPayload" containing logs.

2023-02-28

  • Bug-fix - Enhanced parser to convert hex encoded string to ASCII.

2023-02-09

  • Enhancement - Modified grok for logs containing "type=PATH" to fetch the correct hostname from logs.

2023-01-24

  • Enhancement -
  • Parsed log with eventType as "tac_plus".
  • Added conditions for mapping different event_types "NETWORK_CONNECTION", "NETWORK_HTTP", "USER_LOGIN".

2022-12-02

  • Enhancement -
  • Mapped "user_name" to "principal.user.userid".
  • added conditional check for "dst_ip", "dst_port".

2022-10-31

  • Enhancement -
  • Enhanced the parser to parse the log with type=ADD_USER, USER_MGMT, DEL_USER.
  • Added null checks for "principal_hostname".
  • Added on_error checks for "principal.process.file.full_path", "type_syscall_props.key", "type_syscall_props.arch", "msg2".
  • Added conditional checks for mapping to event_type="FILE_OPEN", "USER_UNCATEGORIZED", "STATUS_UPDATE", "USER_DELETION".
  • Mapped "principal_user_userid" to "principal.user.userid".

2022-11-16

  • Enhancement -
  • Improved "GENERIC_EVENT" to "STATUS_UPDATE" for log types containing "Access Logs".

2022-10-14

  • Enhancement -
  • Migrated customer parser to default parser.

2022-10-13

  • Enhancement - Mapped "vendor_name" to "Linux".
  • Mapped "product_name" to "AuditD".
  • Parsed the logs containing "ProxySG" and mapped "ip" to "target.ip", "port" to "target.port" wherever possible.
  • Modified "event_type" from "GENERIC_EVENT" to "STATUS_UPDATE".
  • Modified mapping for "intermediary.hostname" to "principal.hostname".

2022-07-28

  • Enhancement -
  • Mapped the field 'auid' to about.user.userid'.
  • Mapped the field 'AUID' to 'about.user.user_display_name'.
  • Mapped the field 'proctitle' to 'target.process.file.full_path'.
  • Enhanced the parser to parse the log with type=DAEMON_END, CRYPTO_SESSION, CONFIG_CHANGE, PROCTITLE, USER_ERR, CRYPTO_KEY_USER.
  • Added conditional check for laddr, addr, cipher, pfs, direction, acct, pid, ppid, cmd, exe, ses.

2022-06-17

  • Enhancement - Mapped/Modified the following fields :
  • Changed mapping of "auid" from "security_result.about.user.userid" to "about.user.userid".
  • Changed "event_type" for type=SYSCALL from "SYSTEM_AUDIT_LOG_UNCATEGORIZED" to "USER_UNCATEGORIZED".
  • Mapped "success" to "security_result.summary".
  • Mapped "syscall", "exit", "tty", "a0", "a1", "a2", "a3" to "security_result.about.labels".
  • Dropped the logs in ASCII format.

2022-06-14

  • enhancement
  • Enhanced the parser to parse the USER_CMD type of logs.
  • Mapped the field 'cmd' to 'principal.process.command_line'.
  • Mapped the field 'ses' to 'network.session_id'.
  • Mapped the field 'res' to 'security_result.action' and 'security_result.action_details'.
  • Mapped the fields 'auid' and 'cwd' to 'security_result.detection_fields'.

2022-04-26

  • enhancement
  • Increased the parsing percentage by parsing all the unparsed logs.