Service limits

This page provides details on the limits that apply to Google Security Operations. You can request API limit increases by contacting Cloud Customer Care.

API limits

The following APIs enforce limits on the volume of requests that can be made by you against the Google Security Operations platform. The limits are measured in queries per second (QPS) or queries per hour (QPH).

Data table limits

• Maximum number of data tables for a Google SecOps account: 1,000.

• Only the CSV file type is supported for uploads.

• The limits on the number of in statements when referencing a reference list in a query also apply to in statements in a data table.

• Maximum number of in statements in a query: 10.

• Maximum number of in statements in a query for String and Number data type columns: 7.

• Maximum number of in statements with regular expression operators: 4.

• Maximum number of in statements with CIDR operators: 2.

• Maximum columns per data table: 1,000.

• Maximum rows per data table: 10 million.

• Maximum aggregate limit of data volume across data tables in a account: 1 TB.

• Maximum display limit in web page for data table rows in text and table editor view: 10,000 rows.

• Maximum row limit when uploading a file to a new data table in the web page: 10,000 rows.

• Maximum file upload size limit for data table creation from API: 1 GB.

• Placeholders aren't allowed in the setup section.

• Unmapped columns of a data table with data type set to string can only be joined with string fields of UDM event or UDM entity.

• Use only unmapped columns in a data table with a data type set to cidr or regex for CIDR or regular expression.

• Data table lookups: Regular expression wildcards aren't supported and search terms are limited to 100 characters.

Joins

• Fetching all event samples for detections isn't supported when using data table joins with events.

• Unlike entities and UDM, data tables don't support placeholders. This means you can't:

  • Apply one set of filters to a data table and join it with a UDM entity.

  • Apply a different set of filters to the same data table while joining it with another UDM placeholder.

  For example, a data table named dt with 3 columns: my_hostname, org, and my_email and with the following rule:

  events:
  $e1.principal.hostname =  %dt.my_hostname
  %dt.org ="hr"
  
  $e2.principal.email =  %dt.my_email
  %dt.org !="hr"
  

All filters on a data table are applied first, and then the filtered rows from the data table are joined with UDM. In this case, the contradictory filters (%dt.org ="hr" and %dt.org !="hr") on the dt table result in an empty data table, which is then joined with both e1 and e2.

Use data tables with rules

The following limitations apply to data tables when used with rules.

Run frequency

Real-time run frequency isn't supported for rules with data tables.

Output to data tables

• any and all modifiers aren't supported for repeated field columns in data tables.

• Array indexing isn't supported for repeated fields columns in data tables.

• You can only export outcome variables to a data table. You can't export event path or data table columns directly.

• Column lists must include the primary key columns for data tables.

• You can have a maximum of 20 outcomes.

• If a data table doesn't exist, a new table is created with the default string data type for all columns, following the order specified.

• Only one rule can write to a data table at a time. If a rule tries to write to a data table that another rule is already writing to, the rule compilation fails.

• There's no guarantee that a producer rule can add rows to a data table before a consumer rule for that data table starts.

• A single rule has a limit on the number of outcomes rows. A maximum 10,000-row limit applies over the result and persisted data and to data tables.

• If a row with the same primary key already exists in the data table, it's non-primary key columns are replaced with the new values.

Entity enrichment from data tables

• You can apply only one enrichment operation (either override, append, or exclude) to a single entity graph variable.

• Each enrichment operation can use only one data table.

• You can define a maximum of two enrichment operations of any type in the setup section of a YARA-L rule.

In the following example, an override operation is applied to the entity graph variable $g1 and an append operation is applied to the entity graph variable $g2.

    setup:
    graph_override($g1.graph.entity.user.userid = %table1.myids)
    graph_append [$g2, %table1]

In the preceding example, the same data table (table1) is used to enhance different entity graphs. You can also use different data tables to enhance the different entity graphs, as follows:

    setup:
    graph_override($g1.graph.entity.user.userid = %table1.myids)
    graph_append [$g2, %table2]

Use data tables with Search

The following limitations apply to data tables when used with Search.

• You can't run search queries on data tables using the Chronicle API. Queries are only supported through the web interface.

• A single query execution can output a maximum of 1 million rows to a data table or 1 GB, whichever limit comes first.

• Search output to a data table skips event rows if they exceed 5 MB.

• Entity enrichment is not supported with Search.

• Data tables are not supported for customer-managed encryption keys (CMEK) users.

• Writes are limited to 6 per minute per customer.

• API support is not available Search-related data table operations.

• Statistics queries aren't supported with data table joins.

• Data table and data table joins are only supported with UDM events, and not with entities.

  Supported: %datatable1.column1 = %datatable2.column1 Not supported: graph.entity.hostname = %sample.test

• You can't include a match variable in statistics query in the export section of a statistics query.

  For example, the following is not supported:

  match:
      principal.hostname
  export:
      %sample.write_row(
      row: principal.hostname
    )

Reference list limits

A reference list is a generic list of values which can be used to analyze your data. For more information, see Reference lists.

String lists

String lists have the following limits:

• Maximum list size: 6MB
• Maximum length of any single list content line: 5000 characters

Regular expression lists

Regular expression lists have the following size limits:

• Maximum list size: 0.1MB
• Maximum number of lines: 100
• Maximum length of each content line: 5000 characters

CIDR lists

CIDR lists have the following size limits:

• Maximum list size: 0.1MB
• Maximum number of lines: 150
• Maximum length of each content line: 5000 characters

Ingestion rate

When the data ingestion rate for a tenant reaches a certain threshold, Google Security Operations dynamically adjusts the ingestion rate to ensure availability for new data feeds. The ingestion volume and tenant's usage history determines the threshold. For information on volume of data which can be ingested into Google SecOps by a single customer, see Burst limits.

Dashboard search limit

In search, the quota is per user per hour but for dashboards it is per Google SecOps instance. For more information about dashboards, see Dashboards.