Collect McAfee Web Gateway logs

Supported in:

This document explains how to ingest the McAfee Web Gateway logs to Google Security Operations using a Bindplane agent. The parser extracts fields from the logs in SYSLOG + KV (CEF), JSON, and raw formats. It uses grok and csv filters to parse different log structures, and normalizes field names. It then maps the extracted fields to the Unified Data Model (UDM) schema, handling various edge cases and data inconsistencies to create a unified output.

Before you begin

Ensure that you have the following prerequisites:

  • Google SecOps instance
  • Windows 2016 or later or Linux host with systemd
  • If running behind a proxy, firewall ports are open
  • Privileged access to McAfee Web Gateway

Get Google SecOps ingestion authentication file

  1. Sign in to the Google SecOps console.
  2. Go to SIEM Settings > Collection Agents.
  3. Download the Ingestion Authentication File. Save the file securely on the system where Bindplane will be installed.

Get Google SecOps customer ID

  1. Sign in to the Google SecOps console.
  2. Go to SIEM Settings > Profile.
  3. Copy and save the Customer ID from the Organization Details section.

Install the Bindplane agent

Windows installation

  1. Open the Command Prompt or PowerShell as an administrator.

  2. Run the following command:

    msiexec /i `https://github.com/observIQ/bindplane-agent/releases/latest/download/observiq-otel-collector.msi` /quiet

Linux installation

  1. Open a terminal with root or sudo privileges.

  2. Run the following command:

    sudo sh -c `$(curl -fsSlL https://github.com/observiq/bindplane-agent/releases/latest/download/install_unix.sh)` install_unix.sh

Additional installation resources

For additional installation options, consult the installation guide.

Configure the Bindplane agent to ingest Syslog and send to Google SecOps

  1. Access the configuration file:
    • Locate the config.yaml file. Typically, it's in the /etc/bindplane-agent/ directory on Linux or in the installation directory on Windows.
    • Open the file using a text editor (for example, nano, vi, or Notepad).

  2. Edit the config.yaml file as follows:

    receivers:
    udplog:
        # Replace the port and IP address as required
        listen_address: `0.0.0.0:514`

exporters:
    chronicle/chronicle_w_labels:
        compression: gzip
        # Adjust the path to the credentials file you downloaded in Step 1
        creds: '/path/to/ingestion-authentication-file.json'
        # Replace with your actual customer ID from Step 2
        customer_id: <customer_id>
        endpoint: malachiteingestion-pa.googleapis.com
        # Add optional ingestion labels for better organization
        ingestion_labels:
            log_type: MCAFEE_WEBPROXY
            raw_log_field: body

service:
    pipelines:
        logs/source0__chronicle_w_labels-0:
            receivers:
                - udplog
            exporters:
                - chronicle/chronicle_w_labels

  3. Replace the port and IP address as required in your infrastructure.

  4. Replace <customer_id> with the actual Customer ID.

  5. Update /path/to/ingestion-authentication-file.json to the path where the authentication file was saved in the Get Google SecOps ingestion authentication file section.

Restart the Bindplane agent to apply the changes

  • To restart the Bindplane agent in Linux, run the following command:

    sudo systemctl restart bindplane-agent

  • To restart the Bindplane agent in Windows, you can either use the Services console or enter the following command: 

    net stop BindPlaneAgent && net start BindPlaneAgent

Configure Syslog in McAfee Web Gateway

  1. Sign in to your McAfee Web Gateway web UI.
  2. Go to Policy > Rule Sets.
  3. Click Log Handler, then expand the Default rule set, and select the nested CEF Syslog rule set.
  4. Enable Send to Syslog rule.
  5. Click Save Changes.
  6. Go to Configuration > Appliances > Log File Manager > Settings.
  7. Select Write audit log to syslog.
  8. Go to Configuration > File Editor.
  9. Select rsyslog.conf on the file tree.
  10. Edit the file as follows:
    • Locate the line (or similar): *.info;mail.none;authpriv.none;cron.none /var/log/messages.
    • Add a daemon in this line and insert a - (dash) before the path information: *.info;daemon.!=info;mail.none;authpriv.none;cron.none -/var/log/messages

  11. Add a new line at the bottom of the file to send the info messages to the Bindplane agent IP address.

    • For syslog over UDP:

      daemon.info;auth.=info @<bindplane-server-ip>:<bindplane-port>

    • For syslog over TCP:

      daemon.info;auth.=info @@<bindplane-server-ip>:<bindplane-port>

UDM mapping table

Log Field UDM Mapping Logic
application_name principal.application Directly mapped from the application_name field in KV format or user_agent_product in JSON format.
auth_user principal.user.userid Directly mapped from the auth_user field in KV format.
block_reason security_result.summary Directly mapped from the block_reason field in JSON and CSV JSON formats, or _block_reason in raw format, or block_reason in KV format.
block_res security_result.action Mapped from block_res field in KV format. If block_res is DENIED or contains Block, the action is BLOCK. If block_res is 0 or contains Allow, the action is ALLOW. Special values like 50, 51, 52, 53, 58, 59, 81, 80, 82, 83, 84, 110, 111 are used to determine security_result.category.
bytes_from_client network.sent_bytes Directly mapped from the bytes_from_client field in KV format, or sr_bytes in raw format, or client_to_server_bytes in JSON and CSV JSON formats.
bytes_to_client network.received_bytes Directly mapped from the bytes_to_client field in KV format, or rs_bytes in raw format, or server_to_client_bytes in JSON and CSV JSON formats.
categories security_result.category_details Directly mapped from the categories field in KV format, or _category in raw format, or category in JSON and CSV JSON formats.
client_ip principal.ip, intermediary.ip Directly mapped from the client_ip field in JSON format.
clientIP principal.ip Directly mapped from the clientIP field in CEF format.
csmethod network.http.method Directly mapped from the csmethod field in raw format.
day metadata.event_timestamp Part of the timestamp, extracted from the time_stamp field in KV format.
destination_ip target.ip Directly mapped from the destination_ip field in JSON format.
destination_port target.port Directly mapped from the destination_port field in JSON format.
domain target.hostname, target.url Directly mapped from the domain field in raw format. Used to construct the target.url if uri is present.
header intermediary.hostname Extracted from the beginning of the log message. Used to extract intermediary.hostname.
host target.hostname Directly mapped from the host field in KV format.
hostname principal.hostname Directly mapped from the hostname field in JSON format.
hour metadata.event_timestamp Part of the timestamp, extracted from the time_stamp field in KV format.
http_action network.http.method Directly mapped from the http_action field in JSON format.
http_status_code network.http.response_code Directly mapped from the http_status_code field in JSON and CSV JSON formats, or status_code in raw and KV formats.
kv_entry.application_name principal.application Directly mapped from the application_name field within the KV entry.
kv_entry.auth_user principal.user.userid Directly mapped from the auth_user field within the KV entry.
kv_entry.block_reason security_result.summary Directly mapped from the block_reason field within the KV entry.
kv_entry.block_res security_result.action, security_result.category Mapped from block_res field within the KV entry. Logic for determining action and category is the same as for the top-level block_res field.
kv_entry.bytes_from_client network.sent_bytes Directly mapped from the bytes_from_client field within the KV entry.
kv_entry.bytes_to_client network.received_bytes Directly mapped from the bytes_to_client field within the KV entry.
kv_entry.categories security_result.category_details Directly mapped from the categories field within the KV entry.
kv_entry.host target.hostname Directly mapped from the host field within the KV entry.
kv_entry.method network.http.method Directly mapped from the method field within the KV entry.
kv_entry.rep_level security_result.severity_details Directly mapped from the rep_level field within the KV entry.
kv_entry.server_ip target.ip Directly mapped from the server_ip field within the KV entry.
kv_entry.status_code network.http.response_code Directly mapped from the status_code field within the KV entry.
kv_entry.time_stamp metadata.event_timestamp Directly mapped from the time_stamp field within the KV entry.
kv_entry.url target.url Directly mapped from the url field within the KV entry.
kv_entry.url_port target.port Directly mapped from the url_port field within the KV entry.
kv_entry.user_agent network.http.parsed_user_agent Directly mapped from the user_agent field within the KV entry, then parsed into a structured object.
last_rule security_result.rule_name Directly mapped from the last_rule field in JSON format.
loc principal.location.country_or_region Directly mapped from the loc field extracted from tgt_ip_or_location.
location principal.location.country_or_region Directly mapped from the location field in JSON format.
log.file.path principal.process.file.full_path Directly mapped from the log.file.path field in JSON format.
message Various The raw log message. Parsed differently depending on its format (raw, JSON, KV, CEF).
method network.http.method Directly mapped from the method field in KV and raw formats, or http_action in JSON format, or derived from CEF data. If the value is one of GET, POST, HEAD, OPTIONS, PUT, CONNECT, the metadata.event_type is set to NETWORK_HTTP. If the value is - or CERTVERIFY, the metadata.event_type is set to NETWORK_CONNECTION.
mins metadata.event_timestamp Part of the timestamp, extracted from the time_stamp field in KV format.
month metadata.event_timestamp Part of the timestamp, extracted from the time_stamp field in KV format, or rt field in CEF format.
monthday metadata.event_timestamp Part of the timestamp, extracted from the beginning of the log message.
protocol network.application_protocol Directly mapped from the protocol field in raw format, or uri_scheme in JSON format, or derived from the url field in KV format.
query target.url Directly mapped from the query field in raw format. Appended to the url field.
rep_level security_result.severity_details Directly mapped from the rep_level field in KV format, or reputation in JSON format, or _risk in raw format. Used to determine security_result.severity.
request target.url Directly mapped from the request field in CEF format.
requestClientApplication network.http.user_agent Directly mapped from the requestClientApplication field in CEF format.
requestContext network.http.referral_url Directly mapped from the requestContext field in CEF format.
requestMethod network.http.method Directly mapped from the requestMethod field in CEF format.
requested_host target.url Directly mapped from the requested_host field in JSON format. Used to construct the target.url if requested_path is also present.
requested_path target.url Directly mapped from the requested_path field in JSON format. Appended to requested_host to form the target.url.
request_timestamp metadata.event_timestamp Directly mapped from the request_timestamp field in JSON format.
result security_result.action, security_result.category Directly mapped from the result field in JSON and CSV JSON formats, or block_res in KV format. Used to determine security_result.action and security_result.category.
rt metadata.event_timestamp Directly mapped from the rt field in CEF format.
secs metadata.event_timestamp Part of the timestamp, extracted from the time_stamp field in KV format.
server_ip target.ip Directly mapped from the server_ip field in KV format.
source_ip principal.ip Directly mapped from the source_ip field in JSON, CSV JSON, raw, and KV formats, or src in CEF format, or src_ip in raw format.
src principal.ip Directly mapped from the src field in CEF format.
status_code network.http.response_code Directly mapped from the status_code field in raw format.
summary security_result.summary Directly mapped from the summary field in CSV format, or block_reason in JSON format.
system principal.platform Directly mapped from the system field in JSON format. Converted to uppercase.
target_ip target.ip Directly mapped from the target_ip field in raw format, or dst in CEF format.
tgtport target.port Directly mapped from the tgtport field in raw format.
time metadata.event_timestamp Part of the timestamp, extracted from the beginning of the log message, or the rt field in CEF format, or the time_stamp field in KV format.
timestamp metadata.event_timestamp Directly mapped from the @timestamp field in JSON format.
timezone metadata.event_timestamp Part of the timestamp, extracted from the time_stamp field in KV format.
uri target.url Directly mapped from the uri field in raw format. Used to construct the target.url.
uri_scheme network.application_protocol Directly mapped from the uri_scheme field in JSON format. Converted to uppercase.
url target.url Directly mapped from the url field in raw, KV, and JSON formats, or constructed from domain, uri, and query in raw format, or requested_host and requested_path in JSON format, or request in CEF format.
url_port target.port Directly mapped from the url_port field in KV format.
user principal.user.userid Directly mapped from the user field in JSON format, or username in JSON format, or auth_user in KV format, or suser in raw format.
user_agent network.http.parsed_user_agent Directly mapped from the user_agent field in raw and KV formats, or user_agent_comment in JSON format, or requestClientApplication in CEF format, or constructed from agent.type and agent.version in JSON format. Parsed into a structured object.
user_agent_comment network.http.parsed_user_agent Directly mapped from the user_agent_comment field in JSON format.
user_agent_product principal.application Directly mapped from the user_agent_product field in JSON format.
username principal.user.userid Directly mapped from the username field in JSON format.
year metadata.event_timestamp Part of the timestamp, extracted from the time_stamp field in KV format, or rt field in CEF format.
N/A metadata.event_type Determined by the parser based on the method field. Can be NETWORK_HTTP, NETWORK_CONNECTION, GENERIC_EVENT, or STATUS_UPDATE.
N/A metadata.log_type Hardcoded to MCAFEE_WEBPROXY.
N/A metadata.product_name Hardcoded to MCAFEE_WEBPROXY.
N/A metadata.vendor_name Hardcoded to MCAFEE.
N/A network.direction Hardcoded to OUTBOUND.
N/A security_result.action Determined by the parser based on the block_reason or result fields. Can be ALLOW or BLOCK.
N/A security_result.category Determined by the parser based on the result field. Can be NETWORK_CATEGORIZED_CONTENT, NETWORK_DENIAL_OF_SERVICE, MAIL_SPAM, AUTH_VIOLATION, SOFTWARE_MALICIOUS, NETWORK_SUSPICIOUS, or NETWORK_MALICIOUS.
N/A security_result.severity Determined by the parser based on the risk field. Can be LOW, MEDIUM, or HIGH.

Changes

2025-01-30

Enhancement:

  • Added support for parsing previously unparsed syslog logs.

2024-12-19

Bug fix:

  • Added support to parse unparsed syslog logs.
  • Changed the mapping of metadata.event_type from STATUS_UPDATE to NETWORK_HTTP.

2024-10-24

Bug fix:

  • Added support to parse unparsed logs.

2023-06-17

-Enhancement:

  • Mapped sr_bytes to network.send_bytes.
  • Mapped user to principal.user.userid.
  • Mapped client_ip to principal.ip.

2023-05-31

Bug fix:

  • Added Grok pattern to parse logs failing due to extra '-' in CSV format logs.

2023-01-27

Enhancement:

  • Mapped requested_host and requested_path to target.url.
  • Mapped username to principal.user.userid.
  • Mapped destination_ip to target.ip.
  • Mapped destination_port to target.port.
  • Mapped client_ip to intermediary.ip.
  • Mapped user_agent to network.http.parsed_user_agent.

2023-01-16

Enhancement:

  • Added grok pattern for unparsed log.
  • Mapped target_ip to target.ip.
  • Mapped response_code to network.http.response_code.
  • Mapped category_details to security_result.category_details.
  • Mapped risk to security_result.category_details.

2022-09-21

Enhancement:

  • Merged customer specific parser to default.
  • Added Mapping for unparsed log.
  • Added on error check for kv_entry.server_ip,kv_entry.method,kv_entry.src_ip,kv_entry.server_ip,kv_entry.url_port,kv_entry.url,kv_entry.status_code,kv_entry.auth_user,kv_entry.host,kv_entry.user_agent,kv_entry.bytes_from_client,kv_entry.bytes_to_client,kv_entry.rep_level,kv_entry.block_reason, kv_entry.categories,kv_entry.application_name,kv_entry.block_res

Need more help? Get answers from Community members and Google SecOps professionals.