Google SecOps provides a managed data lake of normalized and threat intelligence enriched
telemetry by exporting data to BigQuery. This lets you do the
following:
Run ad-hoc queries directly in BigQuery.
Use your own business intelligence tools, such as Looker or Microsoft
Power BI, to create dashboards, reports, and analytics.
Join Google SecOps data with third-party datasets.
Run analytics using data science or machine learning tools.
Run reports using predefined default dashboards and custom dashboards.
Google SecOps exports the following categories of data to BigQuery:
UDM event records: UDM records created from log data ingested by customers.
These records are enriched with aliasing information.
Rules matches (detections): instances where a rule matches one or
more events.
IoC matches: artifacts (for example domains, IP addresses) from events that
match Indicator of Compromise (IoC) feeds. This includes matches to from global
feeds and customer-specific feeds.
Ingestion metrics: include statistics, such as number of log lines
ingested, number of events produced from logs, number of log errors indicating
that logs couldn't be parsed, and the state of Google SecOps forwarders.
For more information, see Ingestion metrics BigQuery schema.
Entity graph and entity relationships: stores the description of
entities and their relationships with other entities.
Overview of the tables
Google SecOps creates the datalake dataset in BigQuery and the following tables:
entity_enum_value_to_name_mapping: for enumerated types in the
entity_graph table, maps the numerical values to the string values.
ingestion_metrics:
stores statistics related to ingestion and normalization of data from specific
ingestion sources, such as Google SecOps forwarders, feeds, and Ingestion API.
ioc_matches: stores IOC matches found against UDM events.
job_metadata: an internal table used to track the export of data to
BigQuery.
rule_detections: stores detections returned by rules run in Google SecOps.
rulesets: stores information about Google SecOps curated detections,
including the category each rule set belongs to, whether it is enabled, and
the current alerting status.
udm_enum_value_to_name_mapping: For enumerated types in the events
table, maps the numerical values to the string values.
udm_events_aggregates: stores aggregated data summarized by hour of
normalized events.
Access data in BigQuery
You can run queries directly in BigQuery or connect your own business
intelligence tool, such as Looker or Microsoft Power BI, to BigQuery.
To enable access to the BigQuery instance, use the
Google SecOps BigQuery Access API.
You can provide an email address for either a user or a group that you own. If you
configure access to a group, use the group to manage which team members can
access the BigQuery instance.
To connect Looker or another business intelligence tool to BigQuery, contact
your Google SecOps representative for service account credentials that enable you to
connect an application to the Google SecOps BigQuery dataset. The service
account will have IAM BigQuery Data Viewer role (roles/bigquery.dataViewer) and BigQuery Job Viewer role (roles/bigquery.jobUser).
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-29 UTC."],[[["\u003cp\u003eGoogle Security Operations exports normalized and enriched telemetry data to BigQuery, enabling users to perform ad-hoc queries, use business intelligence tools, join with third-party datasets, and run advanced analytics.\u003c/p\u003e\n"],["\u003cp\u003eFrom December 31, 2024, only Enterprise Plus Tier customers will have access to the managed BigQuery data lake, with the managed resources and API keys fully deprecated by March 31, 2025, for others.\u003c/p\u003e\n"],["\u003cp\u003eThe exported data includes UDM event records, rule matches, IoC matches, ingestion metrics, and entity graph data, all stored in a customer-specific Google Cloud project managed by Google.\u003c/p\u003e\n"],["\u003cp\u003eData is exported on a fill-forward basis with a 365-day retention period, and raw logs are not exported to the Google Security Operations data lake in BigQuery.\u003c/p\u003e\n"],["\u003cp\u003eAccess to the BigQuery instance can be managed through the Google Security Operations CLI or API, and service account credentials for Looker and other BI tools can be obtained from a Google Security Operations representative.\u003c/p\u003e\n"]]],[],null,["# Google SecOps data in BigQuery\n==============================\n\nSupported in: \nGoogle secops [SIEM](/chronicle/docs/secops/google-secops-siem-toc)\n| **Note:** This option is available for Google SecOps Enterprise Plus customers only. For all other customers, see [Configure data export to BigQuery in a self-managed Google Cloud project](/chronicle/docs/preview/cloud-integration/export-to-customer-managed-project).\n\nGoogle SecOps provides a managed data lake of normalized and threat intelligence enriched\ntelemetry by exporting data to BigQuery. This lets you do the\nfollowing:\n\n- Run ad-hoc queries directly in BigQuery.\n- Use your own business intelligence tools, such as Looker or Microsoft Power BI, to create dashboards, reports, and analytics.\n- Join Google SecOps data with third-party datasets.\n- Run analytics using data science or machine learning tools.\n- Run reports using predefined default dashboards and custom dashboards.\n\nGoogle SecOps exports the following categories of data to BigQuery:\n\n- **UDM event records:** UDM records created from log data ingested by customers. These records are enriched with aliasing information.\n- **Rules matches (detections)**: instances where a rule matches one or more events.\n- **IoC matches**: artifacts (for example domains, IP addresses) from events that match Indicator of Compromise (IoC) feeds. This includes matches to from global feeds and customer-specific feeds.\n- **Ingestion metrics:** include statistics, such as number of log lines ingested, number of events produced from logs, number of log errors indicating that logs couldn't be parsed, and the state of Google SecOps forwarders. For more information, see [Ingestion metrics BigQuery schema](/chronicle/docs/reference/ingestion-metrics-schema).\n- **Entity graph and entity relationships**: stores the description of entities and their relationships with other entities.\n\nOverview of the tables\n----------------------\n\nGoogle SecOps creates the `datalake` dataset in BigQuery and the following tables:\n\n- `entity_enum_value_to_name_mapping`: for enumerated types in the `entity_graph` table, maps the numerical values to the string values.\n- `entity_graph`: stores data about UDM entities.\n- [`events`](/chronicle/docs/reports/events-schema-overview): stores data about UDM events.\n- [`ingestion_metrics`](/chronicle/docs/reference/ingestion-metrics-schema): stores statistics related to ingestion and normalization of data from specific ingestion sources, such as Google SecOps forwarders, feeds, and Ingestion API.\n- `ioc_matches`: stores IOC matches found against UDM events.\n- `job_metadata`: an internal table used to track the export of data to BigQuery.\n- `rule_detections`: stores detections returned by rules run in Google SecOps.\n- `rulesets`: stores information about Google SecOps curated detections, including the category each rule set belongs to, whether it is enabled, and the current alerting status.\n- `udm_enum_value_to_name_mapping`: For enumerated types in the events table, maps the numerical values to the string values.\n- `udm_events_aggregates`: stores aggregated data summarized by hour of normalized events.\n\nAccess data in BigQuery\n-----------------------\n\nYou can run queries directly in BigQuery or connect your own business\nintelligence tool, such as Looker or Microsoft Power BI, to BigQuery.\n\nTo enable access to the BigQuery instance, use the\n[Google SecOps BigQuery Access API](/chronicle/docs/reference/bigquery-access-api#access_api_reference).\nYou can provide an email address for either a user or a group that you own. If you\nconfigure access to a group, use the group to manage which team members can\naccess the BigQuery instance.\n\nTo connect Looker or another business intelligence tool to BigQuery, contact\nyour Google SecOps representative for service account credentials that enable you to\nconnect an application to the Google SecOps BigQuery dataset. The service\naccount will have IAM BigQuery Data Viewer role (`roles/bigquery.dataViewer`) and BigQuery Job Viewer role (`roles/bigquery.jobUser`).\n\nWhat's next\n-----------\n\n- Learn more about the following schemas:\n - [`events`](/chronicle/docs/reports/events-schema-overview)\n - [`ingestion_metrics`](/chronicle/docs/reference/ingestion-metrics-schema)\n- For information about accessing and running queries in BigQuery, see [Run interactive and batch query jobs](/bigquery/docs/running-queries).\n- For information about how to query partitioned tables, see [Query partitioned tables](/bigquery/docs/querying-partitioned-tables).\n- For information about how to connect Looker to BigQuery, see Looker documentation about [connecting to BigQuery](/looker/docs/db-config-google-bigquery).\n\n**Need more help?** [Get answers from Community members and Google SecOps professionals.](https://security.googlecloudcommunity.com/google-security-operations-2)"]]
