Google Security Operations data ingestion

Supported in:

Google Security Operations ingests logs from customers, normalizes the data, and detects security alerts. Google Security Operations SIEM provides self-service features around data ingestion, threat detection, alerts, and case management. Google Security Operations can also ingest alerts from other SIEM systems. These alerts are ingested into your Google Security Operations SIEM account, where they can be analyzed.

Google Security Operations SIEM log ingestion

The Google Security Operations SIEM ingestion service acts as a gateway for all data. Google Security Operations SIEM ingests data using the following systems:

  • Forwarders: Google Security Operations SIEM forwarders are remote agents installed at customer endpoints. The forwarders send data to the Google Security Operations SIEM ingestion service. For more information, see installing the Linux or Windows forwarders.

  • BindPlane agent: The BindPlane agent collects logs from various sources, and sends them to Google Security Operations. This agent can be managed using the optional BindPlane OP Management console. For more information, see Use the BindPlane agent.

  • Ingestion APIs: Google Security Operations SIEM has public ingestion APIs, and customers can send data directly to these APIs. For more information, see the Ingestion API.

  • Google Cloud: Google Security Operations SIEM can pull data directly from your Google Cloud account. For more information, see Ingest Google Cloud data to Google SecOps.

  • Data feeds: Google Security Operations SIEM supports a set of data feeds which can pull data from static external locations (for example, Amazon S3) and third party APIs (for example, Okta). These data feeds send logs directly to the Google Security Operations SIEM ingestion service. For more information, see the feed management documentation.

Ingested data is further processed by the Google Security Operations SIEM parsers, which convert the raw logs from customer systems into a Unified Data Model (UDM) that downstream systems within Google Security Operations SIEM can use to provide additional capabilities, including Rules and UDM Search. Google Security Operations SIEM can ingest both logs and alerts. For alerts, Google Security Operations SIEM can only ingest single-event alerts. Google Security Operations SIEM does not support the ingestion of multi-event alerts. UDM Search can be used to search for both ingested alerts and Google Security Operations SOAR alerts.

Google Security Operations ingestion process

The Google Security Operations ingestion mode includes the following types of data ingestion:

  • Ingestion of raw logs into Google Security Operations: Raw logs are ingested using the Google Security Operations SIEM forwarders, ingestion API, directly from Google Cloud, or using a data feed.

  • Ingestion of alerts generated by other SIEMs: Alerts generated in other SIEMs are ingested as follows:

    1. Google Security Operations ingests alerts from the other SIEM systems, EDRs, or ticketing systems using Google Security Operations SOAR connectors or Google Security Operations SOAR webhooks.
    2. Google Security Operations SOAR ingests the events associated with the alerts and creates a corresponding detection.
    3. Google Security Operations SOAR processes the alerts and the ingested events.

    Customers can create detection engine rules to identify patterns in ingested events and generate additional detections.

Limitations

Data feeds have a maximum log line size of 4 MB.