Use the BindPlane agent

The BindPlane agent (also referred as collection agent) is an open-source agent, based on the OpenTelemetry Collector, which collects logs from a variety of sources, including Microsoft Windows event logs, and sends them to Google Security Operations.

The observIQ BindPlane OP Management console provides a comprehensive and unified platform for managing your OpenTelemetry (OTel) collector deployments in Google SecOps and Google Cloud. observIQ provides a BindPlane for Google edition of the management console. For more information, see observIQ solutions. The management console is optional. You can use the agent with or without the console. For more information about the console, see BindPlane OP Management console.

This is the same solution used by Cloud Logging for on-premises deployments.

Before you begin

To install the agent, you need the following:

Google SecOps ingestion authentication file

To download the authentication file, follow these steps:
1. Open the Google SecOps console.
2. Go to SIEM Settings > Collection Agent.
3. Download the Google SecOps ingestion authentication file.
Google SecOps customer ID

To find the customer ID, follow these steps:
1. Open the Google SecOps console.
2. Go to SIEM Settings > Profile.
3. Copy the customer ID from the Organization Details section.
Windows 2012 SP2 or later or Linux host with systemd
Internet connectivity
GitHub access

Verify the firewall configuration

Any firewalls or authenticated proxies between the agent and the internet require rules to open access to the following hosts:

Connection Type	Destination	Port
TCP	malachiteingestion-pa.googleapis.com	443
TCP	asia-northeast1-malachiteingestion-pa.googleapis.com	443
TCP	asia-south1-malachiteingestion-pa.googleapis.com	443
TCP	asia-southeast1-malachiteingestion-pa.googleapis.com	443
TCP	australia-southeast1-malachiteingestion-pa.googleapis.com	443
TCP	europe-malachiteingestion-pa.googleapis.com	443
TCP	europe-west2-malachiteingestion-pa.googleapis.com	443
TCP	europe-west3-malachiteingestion-pa.googleapis.com	443
TCP	europe-west6-malachiteingestion-pa.googleapis.com	443
TCP	europe-west12-malachiteingestion-pa.googleapis.com	443
TCP	me-central1-malachiteingestion-pa.googleapis.com	443
TCP	me-central2-malachiteingestion-pa.googleapis.com	443
TCP	me-west1-malachiteingestion-pa.googleapis.com	443
TCP	northamerica-northeast2-malachiteingestion-pa.googleapis.com	443
TCP	accounts.google.com	443
TCP	oauth2.googleapis.com	443

BindPlane OP Management console

The BindPlane OP Management console offers the following key features:

Centralized management: The console lets you manage all of your OTel collector deployments across Google Cloud. You can view the status of each deployment, as well as perform common management tasks such as starting, stopping, and restarting collectors.
Real-time monitoring: The console provides real-time monitoring of your OTel collector deployments. You can track metrics such as CPU usage, memory usage, and throughput, as well as view logs and traces to troubleshoot issues.
Alerting and notifications: The console lets you set up alerts and notifications for important events, such as when a collector goes down or when a metric threshold is exceeded.
Configuration management: The console lets you centrally manage the configuration of your OTel collectors. You can edit configuration files, set environment variables, and apply security policies to all your deployments.
Integration with Google Cloud: You can create and manage OTel collector deployments in Google Cloud and use the console to access your Google Cloud resources.

There are two ways to deploy the BindPlane OP Management console:

Download and install on a Linux host: Available as a DEB package, RPM package, or Docker image.
Install and provision from the Google Cloud Marketplace.

Install the BindPlane agent

This section describes how to install the agent on different host operating systems.

Windows

To install the BindPlane agent on Windows, run the following PowerShell command.

msiexec /i "https://github.com/observIQ/bindplane-agent/releases/latest/download/observiq-otel-collector.msi" /quiet

Alternatively, to install using an installation wizard, download the latest installer for Windows.

After you download the installer, open the installation wizard and follow the instructions to configure and install the BindPlane agent. For more installation information, see installing on Windows.

Linux

You can install the agent on Linux using a script that automatically determines which package to install. You can also use this script to update an existing installation.

To install using the installation script, run the following script:

sudo sh -c "$(curl -fsSlL https://github.com/observiq/bindplane-agent/releases/latest/download/install_unix.sh)" install_unix.sh

Installation from local package

To install the agent from a local package, use -f with the path to the package.

sudo sh -c "$(curl -fsSlL https://github.com/observiq/bindplane-agent/releases/latest/download/install_unix.sh)" install_unix.sh -f path_to_package

RPM installation

Download the RPM package for your architecture from the releases page and install the package using rpm. Refer to the following example for installing the amd64 package:

sudo rpm -U ./observiq-otel-collector_v${VERSION}_linux_amd64.rpm
sudo systemctl enable --now observiq-otel-collector

Replace VERSION with the version of the package you have downloaded.

DEB installation

Download the DEB package for your architecture from the releases page and install the package using dpkg. Refer to the following example for installing the amd64 package:

sudo dpkg -i --force-overwrite ./observiq-otel-collector_v${VERSION}_linux_amd64.deb
sudo systemctl enable --now observiq-otel-collector

Replace VERSION with the version of the package you have downloaded.

For more information, see BindPlane Agent Installation.

Configure the agent

You can configure the agent either manually or using the BindPlane OP Management console. If you are configuring the agent manually, you need to update the exporter parameters to ensure that the agent authenticates with Google SecOps.

After installing the agent, the observiq-otel-collector service runs and is ready for configuration. The agent logs to C:\Program Files\observIQ OpenTelemetry Collector\log\collector.log by default.

The standard error log for the agent process can be found at C:\Program Files\observIQ OpenTelemetry Collector\log\observiq_collector.err.

By default, the agent configuration file is located at C:\Program Files\observIQ OpenTelemetry Collector\config.yaml. When changing the configuration, you must restart the agent service for the configuration changes to take effect.

You can download a sample configuration file and authentication token used by the agent from the Google SecOps console > SIEM Settings > Collection Agent.

Customize these two sections in the configuration file:

Receiver: Specifies which logs the agent should collect and send to Google SecOps.
Exporter: Specifies the destination where the agent sends the logs. The following exporters are supported:
- Google SecOps exporter: Sends logs directly to Google SecOps ingestion API
- Google SecOps forwarder exporter: Sends logs to Google SecOps forwarder
- Cloud Logging exporter: Sends logs to (Cloud Logging)

In the exporter, customize the following:

customer_id: Google SecOps customer ID
endpoint: Google SecOps regional endpoint
creds: Authentication token

Alternatively, you can use creds_file_path to reference the credentials file directly. For the Windows configuration, escape the path with backslashes.
log_type: Log type
ingestion_labels: Optional ingestion labels
namespace: Optional namespace

Each log type requires you to configure an exporter.

Architecture

The following options are available for the agent architecture.

Option 1: Collection agent sends logs to Google SecOps forwarder

Collection agent sends logs to Google SecOps forwarder

The Google SecOps forwarder receives multiple syslog streams. Each syslog data source is distinguished by the listening port configured on the Google SecOps forwarder. The forwarder makes an encrypted GRPC connection to your Google SecOps instance to deliver collected logs.

Note that the forwarder option allows for aggregation of logs before sending them to Google SecOps.

Option 2: Collection agent sends logs directly to Google SecOps ingestion API

Collection agent sends logs directly to Google SecOps ingestion API

Option 3: Collection agent sends logs directly to Cloud Logging

Collection agent sends logs directly to Cloud Logging

Option 4: Collection agent sends logs to multiple destinations

Collection agent sends logs to multiple destinations

Scalability

Agent collectors typically use minimal resources, but when handling large volumes of telemetry (logs or traces) on a system, be mindful of resource consumption to avoid impacting other services. For more information, see Agent Sizing and Scaling

Support

For any issues related to the collector agent, contact Google Cloud support.

For any issues related to the BindPlane OP Management, contact ObservIQ support.

Additional log collection configuration samples

The following sections list the additional log collection configuration samples.

Send Windows events and sysmon directly to Google SecOps

Configure these parameters in the sample:

chronicleexporter
- namespace
- ingestion_labels
- log_type
- customer_id
- creds

Sample configuration:

receivers:
  windowseventlog/sysmon:
    channel: Microsoft-Windows-Sysmon/Operational
    raw: true
  windowseventlog/security:
    channel: security
    raw: true
  windowseventlog/application:
    channel: application
    raw: true
  windowseventlog/system:
    channel: system
    raw: true

processors:
  batch:

exporters:
  chronicle/sysmon:
    endpoint: malachiteingestion-pa.googleapis.com
    creds: '{
  "type": "service_account",
  "project_id": "malachite-projectname",
  "private_key_id": "abcdefghijklmnopqrstuvwxyz123456789",
  "private_key": "-----BEGIN PRIVATE KEY-----abcdefg-----END PRIVATE KEY-----\n",
  "client_email": "account@malachite-projectname.iam.gserviceaccount.com",
  "client_id": "123456789123456789",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/account%40malachite-projectname.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}' 
    log_type: 'WINDOWS_SYSMON'
    override_log_type: false
    raw_log_field: body
    customer_id: 'dddddddd-dddd-dddd-dddd-dddddddddddd'
  chronicle/winevtlog:
    endpoint: malachiteingestion-pa.googleapis.com
    creds: '{
  "type": "service_account",
  "project_id": "malachite-projectname",
  "private_key_id": "abcdefghijklmnopqrstuvwxyz123456789",
  "private_key": "-----BEGIN PRIVATE KEY-----abcdefg-----END PRIVATE KEY-----\n",
  "client_email": "account@malachite-projectname.iam.gserviceaccount.com",
  "client_id": "123456789123456789",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/account%40malachite-projectname.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}'
    log_type: 'WINEVTLOG'
    override_log_type: false
    raw_log_field: body
    customer_id: 'dddddddd-dddd-dddd-dddd-dddddddddddd'

service:
  pipelines:
    logs/sysmon:
      receivers: [windowseventlog/sysmon]
      processors: [batch]
      exporters: [chronicle/sysmon]
    logs/winevtlog:
      receivers: 
        - windowseventlog/security
        - windowseventlog/application
        - windowseventlog/system
      processors: [batch]
      exporters: [chronicle/winevtlog]

Send Windows events and syslog directly to Google SecOps

Configure these parameters in the sample:

windowseventlogreceiver
tcplogreceiver
- listen_address
chronicleexporter
- namespace
- ingestion_labels
- log_type
- customer_id
- creds

Sample configuration:

receivers:
    tcplog:
      listen_address: "0.0.0.0:54525"
    windowseventlog/source0__application:
        attributes:
            log_type: windows_event.application
        channel: application
        max_reads: 100
        poll_interval: 1s
        raw: true
        start_at: end
    windowseventlog/source0__security:
        attributes:
            log_type: windows_event.security
        channel: security
        max_reads: 100
        poll_interval: 1s
        raw: true
        start_at: end
    windowseventlog/source0__system:
        attributes:
            log_type: windows_event.system
        channel: system
        max_reads: 100
        poll_interval: 1s
        raw: true
        start_at: end
exporters:
    chronicle/chronicle_w_labels:
        compression: gzip
        creds: '{ json blob for creds }'
        customer_id: <customer_id>
        endpoint: malachiteingestion-pa.googleapis.com
        ingestion_labels:
            env: dev
        log_type: <applicable_log_type>
        namespace: testNamespace
        raw_log_field: body
service:
    pipelines:
        logs/source0__chronicle_w_labels-0:
            receivers:
                - windowseventlog/source0__system
                - windowseventlog/source0__application
                - windowseventlog/source0__security
            exporters:
                - chronicle/chronicle_w_labels
        logs/source1__chronicle_w_labels-0:
            receivers:
                - tcplog
            exporters:
                - chronicle/chronicle_w_labels

Send Windows events and syslog to Google SecOps forwarder

Configure these parameters in the sample:

Sample configuration:

receivers:
tcplog:
    listen_address: "0.0.0.0:54525"
    windowseventlog/source0__application:
        attributes:
            log_type: windows_event.application
        channel: application
        max_reads: 100
        poll_interval: 1s
        raw: true
        start_at: end
    windowseventlog/source0__security:
        attributes:
            log_type: windows_event.security
        channel: security
        max_reads: 100
        poll_interval: 1s
        raw: true
        start_at: end
    windowseventlog/source0__system:
        attributes:
            log_type: windows_event.system
        channel: system
        max_reads: 100
        poll_interval: 1s
        raw: true
        start_at: end
exporters:
    chronicleforwarder/forwarder:
        export_type: syslog
        raw_log_field: body
        syslog:
            endpoint: 127.0.0.1:10514
            transport: udp
service:
    pipelines:
        logs/source0__forwarder-0:
            receivers:
                - windowseventlog/source0__system
                - windowseventlog/source0__application
                - windowseventlog/source0__security
            exporters:
                - chronicleforwarder/forwarder
        logs/source1__forwarder-0:
            receivers:
                - tcplog
            exporters:
                - chronicleforwarder/forwarder

Send syslog directly to Google SecOps

Configure these parameters in the sample:

tcplogreceiver
- listen_address
chronicleexporter
- namespace
- ingestion_labels
- log_type
- customer_id
- Creds

Sample configuration:

receivers:
  tcplog:
    listen_address: "0.0.0.0:54525"

exporters:
    chronicle/chronicle_w_labels:
        compression: gzip
        creds: '{ json blob for creds }'
        customer_id: <customer_id>
        endpoint: malachiteingestion-pa.googleapis.com
        ingestion_labels:
            env: dev
        log_type: <applicable_log_type>
        namespace: testNamespace
        raw_log_field: body
service:
    pipelines:
        logs/source0__chronicle_w_labels-0:
            receivers:
                - tcplog
            exporters:
                - chronicle/chronicle_w_labels

Collect Windows events remotely and send them directly to Google SecOps

Configure these parameters in the sample:

windowseventlogreceiver
- username
- password
- server
chronicleexporter
- namespace
- ingestion_labels
- log_type
- customer_id
- creds

Sample configuration:

receivers:
    windowseventlog/system:
        channel: system
        max_reads: 100
        start_at: end
        poll_interval: 10s
        raw: true
        remote:
            username: "username"
            password: "password"
            server: "remote-server"
    windowseventlog/application:
        channel: application
        max_reads: 100
        start_at: end
        poll_interval: 10s
        raw: true
        remote:
            username: "username"
            password: "password"
            server: "server-ip"
    windowseventlog/security:
        channel: security
        max_reads: 100
        start_at: end
        poll_interval: 10s
        raw: true
        remote:
            username: "username"
            password: "password"
            server: "server-ip"
exporters:
    chronicle/chronicle_w_labels:
        compression: gzip
        creds: '{ json blob for creds }'
        customer_id: <customer_id>
        endpoint: malachiteingestion-pa.googleapis.com
        ingestion_labels:
            env: dev
        log_type: WINEVTLOG
        namespace: testNamespace
        raw_log_field: body
service:
    pipelines:
        logs/source0__chronicle_w_labels-0:
            receivers:
                - windowseventlog/system
                - windowseventlog/application
                - windowseventlog/security
            exporters:
                - chronicle/chronicle_w_labels

Send data to Cloud Logging

Configure the credentials_file parameter in the sample.

Sample configuration:

exporters:
  googlecloud:
    credentials_file: /opt/observiq-otel-collector/credentials.json

Query a SQL database and send the results to Google SecOps

Configure these parameters in the sample:

sqlqueryreceiver
chronicleexporter
- namespace
- ingestion_labels
- log_type
- customer_id
- creds

Sample configuration:

receivers:
  sqlquery/source0:
    datasource: host=localhost port=5432 user=postgres password=s3cr3t sslmode=disable
    driver: postgres
    queries:
      - logs:
          - body_column: log_body
        sql: select * from my_logs where log_id > $$1
        tracking_column: log_id
        tracking_start_value: "10000"
processors:
  transform/source0_processor0__logs:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - set(attributes["chronicle_log_type"], "POSTGRESQL") where true
exporters:
  chronicle/chronicle_sql:
    compression: gzip
    creds: '{
  "type": "service_account",
  "project_id": "malachite-projectname",
  "private_key_id": "abcdefghijklmnopqrstuvwxyz123456789",
  "private_key": "-----BEGIN PRIVATE KEY-----abcdefg-----END PRIVATE KEY-----\n",
  "client_email": "account@malachite-projectname.iam.gserviceaccount.com",
  "client_id": "123456789123456789",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/account%40malachite-projectname.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}' 
    customer_id: customer_id
    endpoint: malachiteingestion-pa.googleapis.com
    log_type: POSTGRESQL
    namespace: null
    raw_log_field: body
    retry_on_failure:
      enabled: false
    sending_queue:
      enabled: false
service:
  pipelines:
    logs/source0_chronicle_sql-0:
      receivers:
        - sqlquery/source0
      processors:
        - transform/source0_processor0__logs
      exporters:
        - chronicle/chronicle_sql

Drop logs that match a regular expression

You can configure the collector to drop logs that match a regular expression. This is useful for filtering out unwanted logs, such as known errors or debugging messages.

To drop logs that match a regular expression, add a processor of type filter/drop-matching-logs-to-Chronicle to your configuration. This processor uses the IsMatch function to evaluate the log body against the regular expression. If the function returns true, the log is dropped.

The following example configuration drops logs that contain the strings <EventID>10</EventID> or <EventID>4799</EventID> in the log body.

You can customize the regular expression to match any pattern you need. The IsMatch function uses the RE2 regular expression syntax.

Sample configuration:

processors:
    filter/drop-matching-logs-to-Chronicle:
        error_mode: ignore
        logs:
            log_record:
                - (IsMatch(body, "<EventID>10</EventID>")) or (IsMatch(body, "<EventID>4799</EventID>"))

The following example adds the processor to the pipeline in the same configuration:

service:
  pipelines:
    logs/winevtlog:
      receivers: 
        - windowseventlog/security
        - windowseventlog/application
        - windowseventlog/system
      processors: 
      - filter/drop-matching-logs-to-Chronicle # Add this line
      - batch
      exporters: [chronicle/winevtlog]

Reference documentation

For more information about observIQ, see: