Apply a rule to live data

When you create a rule, it does not initially search for detections based on events received in your Google Security Operations account in real time. However, you set the rule to search for detections in real time by setting the Live Rule toggle to enabled.

When a rule is configured to search for detections in real time, it prioritizes live data for immediate threat detection.

To set a rule to live, complete the following steps:

1. Click Detection > Rules & Detections.

2. Click the Rules Dashboard tab.

3. Click the more_vert Rules option icon for a rule and toggle the Live Rule to enabled.

   

   Live Rule

4. Select View Rule Detections to view detections from a live rule.

Display Rules quota

At the top right of the Rules dashboard, click Rules capacity to display the limits to the number of rules that can be enabled as live.

Google SecOps imposes the following rule limits:

• Multiple Events Rules Quota: Displays the current count of live-enabled Multiple Event rules and the maximum allowed. Learn more about the difference between Single Event and Multiple Event rules.
• Total Rules Quota: Displays the current total count of rules enabled as live across all rule types and the maximum number of rules that can be enabled as live.

Rules executions

Live rules executions for a given event time bucket are triggered with decreasing frequency. A final cleanup run occurs, after which no further executions start.

Each execution runs over the latest versions of reference lists used in the rules, and against the latest event and entity data enrichment.

Some detections can be retrospectively generated if they're detected only by later executions. For example, the last execution might use the latest version of the reference list, which now detects more events, and events and entity data can be reprocessed due to new enrichments.

Deduplication

The Rule Engine automatically identifies and removes duplicate detections from rules. This process applies only to rules with match variables, as they rely on time-based windows. Detections with identical match variable values, within overlapping time windows, are suppressed as duplicates.

Detection latencies

Various factors determine how long it takes for a detection to be generated from a live rule. The following list includes the different factors that contribute to detection delays:

• Rule types
• Run frequency
• Ingestion delay
• Contextual joins
• Enriched UDM data
• Timezone issues
• Reference lists

Rule types

• Single-event rules are executed near-real time in a streaming approach. Use these rules, when possible, to minimize latency.
• Multi-event rules execute on a scheduled basis, which results in higher latency due to the time between scheduled runs.

Run frequency

To achieve faster detections, use a shorter run frequency and smaller match window. Using shorter match windows (under one hour) enables more frequent runs.

Ingestion delay

Ensure that data is sent to Google Security Operations as soon as the event occurs. When reviewing a detection, carefully check the UDM event and ingestion timestamps.

Contextual joins

Multi-event rules that use contextual data, such as UEBA or Entity Graph, might experience higher delays. The contextual data must first be generated by Google SecOps.

Enriched UDM data

Google SecOps enriches events with data from other events. To identify if a rule is evaluating an enriched field, review the Event Viewer. If the rule is evaluating an enriched field, the detection might be delayed.

Timezone issues

Rules execute more frequently for real-time data. Data might arrive in real-time, but Google SecOps might still treat it as late-arriving if the event time is incorrect due to timezone discrepancies.

The Google SecOps SIEM default timezone is UTC. If the original data has an event timestamp set to another timezone besides UTC, update the data timezone. If updating the timezone at the log source is not possible, then contact Support, to override the timezone.

Non-existence rules

Rules that check for non-existence (for example, rules that contain !$e or #e=0) are executed with a minimum delay of one hour to ensure that data has time to arrive.

Reference lists

Rule executions always use the most up-to-date version of a reference list. If the reference list was recently updated, a new detection might appear late. This happens because the detection might only be included in new contents of the updated list during later executions of the scheduled rule.

To achieve lower detection latencies, we recommend doing the following:

• Send log data to Google SecOps as soon as the event occurs.
• Review the audit rules to determine if it is necessary to use non-existence or context-enriched data.
• Configure a smaller run frequency.

Rule status

Live rules can have one of the following statuses:

• Enabled: Rule is active and working normally as a live rule.

• Disabled: Rule is disabled.

• Limited: Live rules can be set to this status when they show unusually high resource usage. Limited rules are isolated from the other live rules in the system to maintain the stability of Google SecOps.

  For Limited live rules, successful rule executions aren't guaranteed. However, if the rule execution succeeds, detections are retained and available for you to review. Limited live rules always generate an error message, which includes recommendations about how to improve the performance of the rule.

  If the performance of a Limited rule doesn't improve within 3 days, its status is changed to Paused.

  Note: If there've been no recent changes to this rule, the errors might be intermittent and could resolve automatically.

• Paused: Live rules enter this status when they've been in Limited status for 3 days and haven't shown any performance improvement. Executions for this rule have been paused and error messages with suggestions on how to improve the rule's performance are returned.

To return any live rule to the Enabled status, follow YARA-L best practices to optimize its performance of your rule and save the changes. After the rule has been saved, the rule is reset to the Enabled state, and it will take at least an hour before it reaches the Limited status again.

You can potentially resolve performance issues with a rule by configuring it to run less frequently. For example, you could reconfigure a rule from running every 10 minutes to running once an hour or once every 24 hours. However, changing the execution frequency of a rule won't change its status back to Enabled. If you make a small modification to the rule and save it, you can automatically reset its status to Enabled.

Rule statuses are displayed in the Rules Dashboard and are also accessible through the Detection Engine API. Errors generated by rules in the Limited or Paused status are available using the ListErrors API method. The error indicates that the rule is either in the Limited or Paused status, and provides a link to documentation on how to resolve the issue.

Need more help? Get answers from Community members and Google SecOps professionals.