Collect Apache logs

Supported in:

This document explains how to ingest Apache logs to Google Security Operations using Bindplane. The parser code first attempts to parse the raw log message as JSON. If that fails, it uses regular expressions (grok patterns) to extract fields from the message based on common Apache log formats.

Before you begin

  • Ensure that you have a Google SecOps instance.
  • Ensure that you are using Windows 2016 or later, or a Linux host with systemd.
  • If running behind a proxy, ensure firewall ports are open.
  • Ensure that you have privileged access to an Apache instance.

Get Google SecOps ingestion authentication file

  1. Sign in to the Google SecOps console.
  2. Go to SIEM Settings > Collection Agents.
  3. Download the Ingestion Authentication File. Save the file securely on the system where Bindplane will be installed.

Get Google SecOps customer ID

  1. Sign in to the Google SecOps console.
  2. Go to SIEM Settings > Profile.
  3. Copy and save the Customer ID from the Organization Details section.

Install the Bindplane agent

Windows installation

  1. Open the Command Prompt or PowerShell as an administrator.

  2. Run the following command:

    msiexec /i "https://github.com/observIQ/bindplane-agent/releases/latest/download/observiq-otel-collector.msi" /quiet

Linux installation

  1. Open a terminal with root or sudo privileges.

  2. Run the following command:

    sudo sh -c "$(curl -fsSlL https://github.com/observiq/bindplane-agent/releases/latest/download/install_unix.sh)" install_unix.sh

Additional installation resources

Configure the Bindplane agent to ingest Syslog and send to Google SecOps

  1. Access the configuration file:

    1. Locate the config.yaml file. Typically, it's in the /etc/bindplane-agent/ directory on Linux or in the installation directory on Windows.
    2. Open the file using a text editor (for example, nano, vi, or Notepad).

  2. Edit the config.yaml file as follows:

    receivers:
    udplog:
        # Replace the port and IP address as required
        listen_address: "0.0.0.0:514"

exporters:
    chronicle/chronicle_w_labels:
        compression: gzip
        # Adjust the path to the credentials file you downloaded in Step 1
        creds: '/path/to/ingestion-authentication-file.json'
        # Replace with your actual customer ID from Step 2
        customer_id: <customer_id>
        endpoint: malachiteingestion-pa.googleapis.com
        # Add optional ingestion labels for better organization
        ingestion_labels:
            log_type: 'APACHE'
            raw_log_field: body

service:
    pipelines:
        logs/source0__chronicle_w_labels-0:
            receivers:
                - udplog
            exporters:
                - chronicle/chronicle_w_labels

  3. Replace the port and IP address as required in your infrastructure.

  4. Replace <customer_id> with the actual customer ID.

  5. Update /path/to/ingestion-authentication-file.json to the path where the authentication file was saved in the Get Google SecOps ingestion authentication file section.

Restart the Bindplane agent to apply the changes

  • To restart the Bindplane agent in Linux, run the following command:

    sudo systemctl restart bindplane-agent

  • To restart the Bindplane agent in Windows, you can either use the Services console or enter the following command:

    net stop BindPlaneAgent && net start BindPlaneAgent

Configure Syslog in Apache

  1. Sign in to the Ubuntu server using SSH.

  2. Create a file under /etc/rsyslog.d/ named 02-apache2.conf:

    vim /etc/rsyslog.d/02-apache2.conf

  3. Add the following code to the file:

    module(load="imfile" PollingInterval="10" statefile.directory="/var/spool/rsyslog")
input(type="imfile"
        File="/var/log/apache2/access.log"
        Tag="http_access"
        Severity="info"
        Facility="local6")
Local6.info        @<bindplane-agnet-ip>:<vindplane-agent-port>
module(load="imfile" PollingInterval="10" statefile.directory="/var/spool/rsyslog")
input(type="imfile"
        File="/var/log/apache2/error.log"
        Tag="http_error"
    • Replace the bindplane-agent-ip> and bindplane-agent-port with the IP address and port configured for the Bindplane Agent
    • If you are using the TCP prototol, append an additional @ to the host line so it will look like this: @@<bindplane-agnet-ip>:<vindplane-agent-port>.

  4. Restart RSyslog services:

    sudo service rsyslog restart

UDM Mapping Table

Log field UDM mapping Logic
bytes network.received_bytes Bytes received from the client.
bytes network.sent_bytes Bytes sent to the client.
bytes_out network.sent_bytes Bytes sent to the client.
bytes_received network.received_bytes Bytes received from the client.
Content network.http.method HTTP method extracted from the "Content" field.
Content target.url Target URL extracted from the "Content" field.
cookie additional.fields.value.string_value Value of the "cookie" field.
dest_ip target.ip IP address of the target.
dest_name target.hostname Hostname of the target.
dest_port target.port Port of the target.
description metadata.description Description of the event.
duration_microseconds additional.fields.value.string_value Value of the "duration_microseconds" field.
file_full_path target.file.full_path Full path of the target file.
hostname target.hostname Hostname of the target.
http_content_type additional.fields.value.string_value Value of the "http_content_type" field.
http_host principal.hostname Hostname of the principal.
http_method network.http.method HTTP method.
http_referrer network.http.referral_url HTTP referrer URL.
http_user_agent network.http.user_agent HTTP user agent.
ID metadata.id ID of the event.
insertId metadata.product_log_id Product log ID.
ip principal.ip IP address of the principal.
jsonPayload.cIP target.ip IP address of the target.
jsonPayload.cPort target.port Port of the target.
jsonPayload.csBytes network.sent_bytes Bytes sent to the client.
jsonPayload.csMethod network.http.method HTTP method.
jsonPayload.csMimeType target.file.mime_type MIME type of the target file.
jsonPayload.csReferer network.http.referral_url HTTP referrer URL.
jsonPayload.csURL target.url Target URL.
jsonPayload.csUserAgent network.http.user_agent HTTP user agent.
jsonPayload.sHierarchy additional.fields.value.string_value Value of the "sHierarchy" field.
jsonPayload.sHostname principal.hostname Hostname of the principal.
jsonPayload.sIP principal.ip IP address of the principal.
jsonPayload.scBytes network.received_bytes Bytes received from the client.
jsonPayload.scHTTPStatus network.http.response_code HTTP response code.
jsonPayload.scResultCode additional.fields.value.string_value Value of the "scResultCode" field.
LastStatus network.http.response_code HTTP response code.
log_level security_result.severity Severity of the security result.
logName security_result.category_details Category details of the security result.
method network.http.method HTTP method.
pid principal.process.pid Process ID of the principal.
Port target.port Port of the target.
proto network.application_protocol Application protocol.
referer network.http.referral_url HTTP referrer URL.
RemoteHost principal.ip IP address of the principal.
RemoteUser principal.user.userid User ID of the principal.
resource.labels.instance_id target.resource.product_object_id Product object ID of the target resource.
resource.labels.project_id target.resource.attribute.labels.value Value of the "project_id" label.
resource.labels.zone target.resource.attribute.cloud.availability_zone Availability zone of the target resource.
resource.type target.resource.resource_type Resource type of the target.
response network.http.response_code HTTP response code.
SizeBytes network.received_bytes Bytes received from the client.
src_ip principal.ip IP address of the principal.
src_port principal.port Port of the principal.
ssl_cipher network.tls.cipher TLS cipher.
ssl_version network.tls.version_protocol TLS version protocol.
status network.http.response_code HTTP response code.
target target.url Target URL.
target_ip target.ip IP address of the target.
target_port target.port Port of the target.
time metadata.event_timestamp Event timestamp.
uri_path target.process.file.full_path Full path of the target file.
user principal.user.userid User ID of the principal.
useragent network.http.user_agent HTTP user agent.
version_protocol network.tls.version_protocol TLS version protocol.
Workername principal.hostname Hostname of the principal.
x_forwarded_for Value of the "X-Forwarded-For" header.
metadata.log_type The value is set to "APACHE" in the parser code.
metadata.product_name The value is set to "Apache Web Server" in the parser code.
metadata.vendor_name The value is set to "Apache" in the parser code.
metadata.event_type The value is determined based on the presence of principal and target information. If both principal and target are present, the event type is set to "NETWORK_HTTP". If only principal is present, the event type is set to "STATUS_UPDATE". Otherwise, it is set to "GENERIC_EVENT".
additional.fields.key The key is set to "keep_alive", "duration_microseconds", "cookie", "http_content_type", "sHierarchy", "scResultCode" in the parser code based on the field.
target.port If the "proto" field is "HTTP", the port is set to 80. If the "proto" field is "HTTPS", the port is set to 443. If the "proto" field is "FTP", the port is set to 21.
target.resource.attribute.labels.key The key is set to "project_id" in the parser code.

Changes

2025-01-09

Enhancement:

  • Added a new Grok pattern to parse the new log format.
  • Mapped "user_location" to "principal.location.country_or_region".
  • Mapped "proto", "proto_version", "uri_path", and "uri_query" to "additional.fields".

2024-12-19

Enhancement:

  • Added a Grok pattern to parse the unparsed logs.

2024-09-10

Enhancement:

  • Added support to parse unparsed logs.

2024-08-05

Enhancement:

  • Added a Grok pattern to parse the "jsonPayload.message" field into "additional.fields".
  • Mapped "ip_msg" to "principal.ip" and "principal.asset.ip".
  • Mapped "msg_method" to "network.http.method".
  • Mapped "response_code" to "network.http.response_code".
  • Mapped "useragentvalue" to "network.http.user_agent".

2024-06-11

Enhancement:

  • Added a Grok pattern to parse the new pattern of SYSLOG format logs.

2024-01-25

Enhancement:

  • Added a new Grok pattern to parse syslog logs that contain symbol "+".

2024-01-25

Enhancement:

  • Added a new Grok pattern to parse syslog logs that contain symbol "+".

2023-12-21

Enhancement:

  • Handled unparsing JSON logs.
  • Mapped "src_port" to "principal.port".
  • Mapped "x_forwarded_for" to "principal.ip".
  • Mapped "keep_alive", "duration_microseconds", "cookie", "http_content_type" to "additional.fields".
  • Mapped "user" to "principal.user.userid".
  • Mapped "http_host" to "principal.hostname.
  • Mapped "file_full_path" to "target.file.full_path".
  • Mapped "ssl_version" to "network.tls.version_protocol".
  • Mapped "ssl_cipher" to "network.tls.cipher".
  • Mapped "uri_path" to "target.process.file.full_path".
  • Mapped "http_referrer" to "network.http.referral_url".
  • Mapped "http_user_agent" to "network.http.user_agent".
  • Mapped "http_method" to "network.http.method".
  • Mapped "protocol" to "network.application_protocol".
  • Mapped "dest_port" to "target.port".
  • Mapped "dest_name" to "target.hostname".
  • Mapped "bytes_out" to "network.sent.bytes".

2023-07-31

Enhancement:

  • Modified Grok pattern to handle hyphen("-") when bytes are not available.

2023-06-05

Enhancement:

  • Mapped "Content" to "target.url", "network.http.method" and "network.tls.version_protocol".
  • Mapped "LastStatus" to "network.http.response_code".
  • Mapped "SizeBytes" to "network.received_bytes".
  • Mapped "Workername" to "principal.hostname".
  • Mapped "Port" to "target.port".
  • Mapped "ID" to "metadata.id".
  • Mapped "XForwardedForIP" and "RemoteHost" to "principal.ip".
  • Mapped "Remoteuser" to "principal.user.userid".
  • Modified Grok pattern to support incorrect log parsing.

2023-02-20

Enhancement:

  • Modified Grok pattern to support incorrect log parsing.
  • Converted "user_agent" to "network.http.parsed_user_agent".

2022-09-21

Enhancement:

  • Migrated to default parser.

2022-09-07

Enhancement:

  • Added grok patterns to parse logs with json + syslog format.
  • Mapped the field 'host.name' and 'hostname' to 'target.hostname'.
  • Mapped the field 'log.file.path' to 'principal.process.file.full_path'.
  • Mapped the field 'mac' to 'principal.mac'.
  • Mapped the field 'ip' to 'principal.asset.ip'.
  • Mapped the field 'os.version' to 'principal.platform_version'.
  • Mapped the field 'os.kernel' to 'principal.platform_patch_level'.
  • Mapped the field 'os.platform' to 'principal.platform'.
  • Mapped the field 'architecture' to 'principal.asset.hardware'.
  • Mapped the field 'id' to 'principal.asset.asset_id'.

2022-05-12

Enhancement:

  • Added grok patterns for unique unparsed logs.
  • Added conditional checks for 'network.http.user_agent' and 'network.http.referral_url'.
  • Added support for Apache Traffic Server (ATS) proxy logs.

Need more help? Get answers from Community members and Google SecOps professionals.