Google SecOps data in BigQuery

Supported in:

Google SecOps provides a managed data lake of normalized and threat intelligence enriched telemetry by exporting data to BigQuery. This enables you to do the following:

  • Run ad-hoc queries directly in BigQuery.
  • Use your own business intelligence tools, such as Looker or Microsoft Power BI, to create dashboards, reports, and analytics.
  • Join Google SecOps data with third-party datasets.
  • Run analytics using data science or machine learning tools.
  • Run reports using predefined default dashboards and custom dashboards.

Google SecOps exports the following categories of data to BigQuery:

  • UDM event records: UDM records created from log data ingested by customers. These records are enriched with aliasing information.
  • Rules matches (detections): instances where a rule matches one or more events.
  • IoC matches: artifacts (for example domains, IP addresses) from events that match Indicator of Compromise (IoC) feeds. This includes matches to from global feeds and customer-specific feeds.
  • Ingestion metrics: include statistics, such as number of log lines ingested, number of events produced from logs, number of log errors indicating that logs could not be parsed, and the state of Google SecOps forwarders. For more information, see Ingestion metrics BigQuery schema.
  • Entity graph and entity relationships: stores the description of entities and their relationships with other entities.

Data export flow

The data export flow is as follows:

  1. A set of Google SecOps data, specific to a use case, is exported to a BigQuery instance that exists in a customer-specific Google Cloud project and is managed by Google. Data for each use case is exported to a separate table. This is exported from Google SecOps to BigQuery in a customer-specific project.
  2. As part of the export, Google SecOps creates a pre-defined Looker data model for each use case.
  3. Google SecOps default dashboards are built using the predefined Looker data models. You can create custom dashboards in Google SecOps using the predefined Looker data models.
  4. Customers can write ad-hoc queries against Google SecOps data stored in BigQuery tables.

  5. Customers can also create more advanced analytics using other third-party tools that integrate with BigQuery.

    Data export to process to BigQuery

The BigQuery instance is created in the same region as the Google SecOps tenant. One BigQuery instance is created for each customer ID. Raw logs are not exported to the Google SecOps data lake in BigQuery. Data is exported on a fill forward basis. As data is ingested and normalized in Google SecOps it is exported to BigQuery. You cannot backfill previously ingested data.

For Looker connections, contact your Google SecOps representative for service account credentials that enable you to connect your Looker instance to Google SecOps data in BigQuery. The service account will have read-only permission.

Overview of the tables

Google SecOps creates the datalake dataset in BigQuery and the following tables:

  • entity_enum_value_to_name_mapping: for enumerated types in the entity_graph table, maps the numerical values to the string values.
  • entity_graph: stores data about UDM entities.
  • events: stores data about UDM events.
  • ingestion_metrics: stores statistics related to ingestion and normalization of data from specific ingestion sources, such as Google SecOps forwarders, feeds, and Ingestion API.
  • ioc_matches: stores IOC matches found against UDM events.
  • job_metadata: an internal table used to track the export of data to BigQuery.
  • rule_detections: stores detections returned by rules run in Google SecOps.
  • rulesets: stores information about Google SecOps curated detections, including the category each rule set belongs to, whether it is enabled, and the current alerting status.
  • udm_enum_value_to_name_mapping: For enumerated types in the events table, maps the numerical values to the string values.
  • udm_events_aggregates: stores aggregated data summarized by hour of normalized events.

Access data in BigQuery

You can run queries directly in BigQuery or connect your own business intelligence tool, such as Looker or Microsoft Power BI, to BigQuery.

To enable access to the BigQuery instance, use either the Google SecOps CLI or the Google SecOps BigQuery Access API. You can provide an email address for either a user or a group that you own. If you configure access to a group, use the group to manage which team members can access the BigQuery instance.

To connect Looker or another business intelligence tool to BigQuery, contact your Google SecOps representative for service account credentials that enable you to connect an application to the Google SecOps BigQuery dataset. The service account will have IAM BigQuery Data Viewer role (roles/bigquery.dataViewer) and BigQuery Job Viewer role (roles/bigquery.jobUser).

What's next

Need more help? Get answers from Community members and Google SecOps professionals.