Google SecOps data ingestion

Supported in:

Google Security Operations ingests logs from customers, normalizes the data, and detects security alerts. Google SecOps provides self-service features around data ingestion, threat detection, alerts, and case management. Google SecOps can also ingest alerts from other SIEM systems. These alerts are ingested into your Google SecOps account, where they can be analyzed.

Google SecOps log ingestion

The Google SecOps ingestion service acts as a gateway for all data. Google SecOps ingests data using the following systems:

  • Forwarders: Google SecOps forwarders are remote agents installed at customer endpoints. The forwarders send data to the Google SecOps ingestion service. For more information, see installing the Linux or Windows forwarders.

  • BindPlane agent: The Bindplane agent collects logs from various sources and sends them to Google SecOps. This agent can be managed using the optional Bindplane OP Management console. For more information, see Use the Bindplane agent.

  • Ingestion APIs: Google SecOps has public ingestion APIs, and customers can send data directly to these APIs. For more information, see the Ingestion API.

  • Google Cloud: Google SecOps can pull data directly from your Google Cloud account. For more information, see Ingest Google Cloud data to Google SecOps.

  • Data feeds: Google SecOps supports a set of data feeds which can pull data from static external locations (for example, Amazon S3) and third party APIs (for example, Okta). These data feeds send logs directly to the Google SecOps ingestion service. For more information, see the feed management documentation.

Ingested data is further processed by the Google SecOps parsers, which convert the raw logs from customer systems into a Unified Data Model (UDM) that downstream systems within Google SecOps can use to provide additional capabilities, including Rules and UDM Search. Google SecOps can ingest both logs and alerts. For alerts, Google SecOps can only ingest single-event alerts. Google SecOps does not support the ingestion of multi-event alerts. UDM Search can be used to search for both ingested alerts and Google SecOps alerts.

Google SecOps ingestion process

The Google SecOps ingestion mode includes the following types of data ingestion:

  • Ingestion of raw logs into Google SecOps: Raw logs are ingested using the Google SecOps forwarders, ingestion API, directly from Google Cloud, or using a data feed.

  • Ingestion of alerts generated by other SIEMs: Alerts generated in other SIEMs are ingested as follows:

    1. Google SecOps ingests alerts from the other SIEM systems, EDRs, or ticketing systems using Google SecOps connectors or Google SecOps webhooks.
    2. Google SecOps ingests the events associated with the alerts and creates a corresponding detection.
    3. Google SecOps processes the alerts and the ingested events.

    Customers can create detection engine rules to identify patterns in ingested events and generate additional detections.

Limitations

Data feeds have a maximum log line size of 4 MB.

Need more help? Get answers from Community members and Google SecOps professionals.