Understand rule detection delays

This document explains rule detection delays in Google Security Operations, identifies contributing factors, outlines troubleshooting approaches, and suggests techniques to reduce delays where possible.

Detection rules

Detection rules examine both regular and entity Universal Data Model (UDM) events, which are normalized raw logs, to generate detections according to the rule's specifications. Entity UDM events typically contain context information like user or asset details. Rules also generate detections based on previously generated detections.

Expected and unpredicted delays

Rule detections always have some delay, ranging from near real-time to a few minutes or many hours. Factors like rule type, run frequency, and detection generation method affect these delays. This document explores these and other delay factors.

We categorize delays as either expected or unpredicted.

• Expected delays: These delays are a result of the ingestion process and the configuration choices you make when setting up the detection rule.
  For example, creating a detection takes time. These delays depend on known structural factors, such as the rule type, run frequency, the detection generation method, known limitations, and other predictable factors.
  You can minimize these delays by changing or tuning detection rule configurations, as described in this document.

• Unpredicted delays: These are rule-specific or event-specific delays caused by many factors, including delays in event data arriving at Google SecOps, transient slowness in processing pipelines within Google SecOps services, and re-enrichment processing.

Analyze rule detection delays

To analyze rule detection delays, find information about the rule and its surrounding factors:

1. In the Google SecOps console, go to Detection > Rules and detections.

   The Rules dashboard shows rule metadata like Rule name, Rule type, and Run frequency.

   For more details, see Viewing rules in Rules dashboard.

2. In the Rules dashboard, use the search to identify rules experiencing lengthy detection delays.

3. For any particular rule execution, there are a number of factors that can impact detection latency. Dimensions like Rule type, Run frequency, Event type, Event time, and Ingested time are good heuristics for understanding why a particular detection may have been delayed.

4. Familiarize yourself with the following topics to understand how these factors influence rule detection delays:

   • Detection generation methods
   • Known limitations
   • Factors contributing to rule detection delays

Detection generation methods

Learn how the system creates rule detections to understand how the detection generation method affects detection delays.

The system generates rule detections in the following ways:

1. Streaming engine

   The Streaming engine is a fast pipeline that typically operates with less than a five-minute delay. It processes single-event rules with no match section and no external data sets like reference lists or data tables.

2. Query engine

   The Query engine processes rules as follows:

   • Complex single-event rules:

     • Single-event rules that reference ref-lists or data tables.
     • Windowed single-event rules that use a match window with a simple condition. For example, "an event where our count of events > 0". These rules run on a high-frequency (real-time) query rate as new data is ingested and made available for rule processing.

   • Multi-event rules: These rules query data over blocks of event time (10-minute, hourly, daily), according to a schedule that you set.
     For example, for a 10-minute schedule, the rule re-queries event time blocks after 5 to 8 hours and 24 to 48 hours, depending on upstream processing timelines.

3. Rules run against historical data

   For details, see Rules created from retro hunts.

4. Re-enrichment of UDM events

   For details, see Re-enrichment of UDM events and Entity context graph processing

Known limitations

Here are some standard limitations that contribute to rule detection delays:

• Enrichment delays can sometimes take longer than expected. Enrichment re-processing causes later rule runs to re-evaluate data. The system performs multiple enrichment runs, with the last run occurring three days after the event time.
• Multiple-event rules often join contextual data that arrives many hours after the primary event's event time. High latency in this contextual data can cause re-enrichment processing and rule replays to interact, resulting in a delayed detection. While Google SecOps uses this feature to handle extremely late-arriving data, it appears as a large span of time between the detection window (event time block) and the alert's created time.
• Context-aware rules are rules that rely on enrichment sources such as asset and identity aliasing, or the Entity context graph. Because these rules rely on multiple event sources, they are more susceptible to latency.
• The system re-executes rules between 5 and 8 hours, and again between 24 and 48 hours after the initial rule execution. These two separate rule replays occur depending on the reprocessing pipeline execution times.

Troubleshoot rule detection delays

Troubleshoot rule detection delays through a process of elimination.

Follow this suggested approach to investigate and troubleshoot rule detection delays:

1. Check for obvious delays:

   Determine if any ingestion delay exists:

   1. In the Google SecOps console, go to Detection > Rules and detections.

   2. Search for the rule to analyze in the Rules Dashboard.

   3. Compare the Event time to the Ingested time.

      For example, for a particular rule detection, if a large gap exists between the Event time and the Ingested time, you can likely attribute the detection delay to an expected delay.

2. Review the context source collection time:

   Check the context source's collection time.

   Context-aware rules can include the following context sources. Check their collection times:

   • Fields derived from UDM enrichment.
   • Events that include a principal field.

   • Rules that reference a graph.entity field.

     Rules referencing the entity context graph (ECG) with graph.entity syntax can cause particularly high latency. For example, the ECG pipeline generates context data, a process that can take 30 hours or, in some cases, up to 8 days, depending on the data type.

   For more details, see Data processing delays.

3. Examine the rule's run frequency and match window configuration:

   • Frequency: Check the rule's run frequency. A rule configured to run less frequently naturally has longer detection delays.
   • Match window: If a rule has a match window, the minimum delay is the duration of that window.
   • Frequency and match window relationship: Ensure the run frequency is compatible with the match window. For example, if the match window is 5 minutes, a 10-minute run frequency is acceptable. However, if the match window is more than 10 minutes, use the next available run frequency, 1 hour.

4. Check for recent incidents:

   Look for any recent incidents that could have caused delays or issues with data feeds.

Tips to shorten delays

To update detection rule configurations, see Manage rules using the Rules editor.

Use the following techniques to reduce delays where possible:

• Use real-time (10 minutes) for latency-sensitive rules:

  • Increase the rule frequency:

    For rules with match windows over 60 minutes, increase the rule frequency as follows:

    • For multiple-event rules: 10 minutes
    • For single-event rules: Near Real Time

    For more details, see Set the run frequency.

  • Reduce match window duration:

    Smaller match windows are more efficient. Change a 60-minute match window to 59 minutes to enable 10-minute processing.

• Avoid late-arriving data:

  Late-arriving data misses the initial query, and the system processes it only when it re-queries its event time block 5 to 8 hours later, causing significant delays. On-time data typically has about a 20-minute delay.

Factors contributing to rule detection delays

The rule type, run frequency, and the speed of Google SecOps's ingestion are key factors in rule detection delays.

The following factors contribute to rule detection delays.

Rule types

• Single-event rules:

  Because Single-event rules are executed in near real-time using a streaming approach, use them to minimize delays, where possible.

  • Simple single-event rules

    These rules detect single events and don't use reference lists, data tables, match windows, or User and Entity Behavior Analytics (UEBA). These rules execute in near-real time, in a streaming fashion, and have the shortest detection delays.

  • Complex single-event rules

    • Window single-event rules

      These are single-event rules that include a match window and typically have a slightly longer delay than other single-event rules. A match window is typically a period of time in which a rule examines one or more events.

    • Reference single-event rules

      These are single-event rules that include reference lists or data tables.

• Multiple-event rules:

  Multiple-event rules execute on a scheduled basis, which results in longer delays due to the time between scheduled runs.

  • Multiple-event rules

    These rules examine two or more UDM event conditions. They typically have a match window and multiple conditions.

  • Context-aware rules

    Context-aware rules help you understand both the behavioral patterns in telemetry and the context of the impacted entities from those patterns.

    • These rules consist of two or more conditions, but at least one condition is a UDM Entity event (where the UDM event is of context type, such as user_context).
    • These types of rules have longer delays, and it's not uncommon for detections to take a few days.
    • Context-aware rules generally have the longest delays because the system must first generate the necessary context data.

Learn more about the difference between Single-Event and Multiple-Event rules.

Rule run frequency

Rule run frequency directly impacts detection delay.

• Near real-time: Rules execute more frequently for real-time data, constantly running. This applies only to single-event rules.
• Other frequencies: For other rule types, you can set the following frequencies:
  • 10-minute frequency is valid for match windows < 60 minutes.
  • 1-hour and 24-hour frequency is valid for all match windows < 48 hours.
  • 24-hour frequency is valid for all match windows >= 48 hours.

Possible workaround: To achieve faster detections, use a shorter run frequency and smaller match window. Using shorter match windows (less than one hour) enables more frequent runs.

Ingestion delay

Ingestion delay refers to the time Google SecOps takes to ingest data after the event occurs.

If data arrives late, it misses the initial query window. A subsequent historical processing query picks it up, but this can introduce delays of 5 to 8 hours.

For example: Event A (9:03 AM event time) and Event B (9:05 AM event time) are part of a rule looking for two events within 30 minutes. If Event A arrives at 10:05 AM (1 hour late), it misses the initial queries of the 9:00-9:30 AM block. A follow-up query for that block between 2-5 PM then generates the detection, resulting in delays of 5 to 8 hours.

Troubleshooting: Verify that you send data to Google SecOps as soon as the event occurs. When reviewing a detection, carefully check the UDM event and ingestion timestamps.

Time zone issues

The Google SecOps SIEM default time zone is UTC. If logs don't include an explicit time zone definition, the system interprets them as UTC. Incorrect interpretation can cause the logs to be treated as late-arriving, which results in detection delays, even if the system receives them in real-time.

For example, a log with an event time of 10:00 AM Eastern Time (15:00 UTC) arrives at 15:05 UTC, but lacking a time zone. If the log lacks a time zone, the system interprets the event time as 10:00 UTC. The system then calculates a 5-hour delay between the interpreted event time (10:00 UTC) and the actual ingestion time (15:05 UTC). This calculated delay triggers detection delays because rules prioritize processing based on real-time ingestion.

Workarounds: If the original data's event timestamp is in a time zone other than UTC, try one of the following:

• Update the original data's event time zone.
• If you cannot update the time zone at the log source, contact Support to override the time zone.
• Alternatively, use a BindPlane processor to correct the timestamp and format it as UTC, or add the appropriate time zone indicator. For details, see Modifying log body timestamps using BindPlane.

Match window

If a rule has a match window, the window's duration determines the minimum detection delay, as the system must wait for the entire window to occur.

Contextual joins

Multiple-event rules that use contextual data, such as UEBA or Entity Graph, might experience longer delays. Google SecOps must first generate the contextual data.

Enrichment delays

Google SecOps enriches events with data from other events. Delays in adding additional data (enrichment) to UDM events can prolong detection times.

To identify if a rule is evaluating an enriched field, review the Event Viewer. If the rule is evaluating an enriched field, the system might delay the detection.

Reference lists

Rule executions always use the latest version of a reference list. When scheduled rules run again, the system can create new detections based on updated reference list contents. These detections might appear late because they're based on data ingested prior to the reference list update.

To shorten detection delays, do the following:

• Send log data to Google SecOps as soon as the event occurs.
• Review the audit rules to determine whether to use non-existence or context-enriched data.
• Configure a smaller run frequency.

Non-existence rules

The system executes rules that check for non-existence (for example, rules that contain !$e or #e=0) with a minimum delay of one hour to verify that data has time to arrive.

Rules created from retro hunts

When you run a rule against historical data using a retro hunt, the system only creates the detection after the retro hunt process finishes. This process can take a significant amount of time, which causes a detection delay.

Note: You cannot distinguish latency monitoring metrics for live detection rules from retrohunt detection rules. To avoid skewing your detection latency monitoring metrics, don't use a live rule to simulate a retrohunt rule. As a best practice, create a dedicated detection rule and run it as a retrohunt rule.

Data processing delays

In the following cases, the system might continue processing data even after creating an initial detection, potentially leading to new or updated detections.

Enrichment system

The Enrichment system processes data and applies enhancements. The process generally updates within 30 minutes. However, if the underlying data changes upon event ingestion, the system re-processes historical data and updates events for up to 24 hours. For example, if the enrichment system updates entity or process metadata, IP geolocation, or VirusTotal indicators, the rules engine re-evaluates these blocks 24 to 48 hours later to capture those updates.

For example: An event at 9:03 AM has entity.asset.hostname = hostnameA but no IP. A DHCP log from 8:55 AM shows hostnameA = IP 1.2.3.4. The rules engine runs at 9:10 AM, and the rule doesn't match. The enrichment processing pipeline correlates hostnameA to 1.2.3.4 for that time window, updating the UDM event. Now the rule matches, and the system creates a detection.

Re-enrichment of UDM events

After the system ingests an event, it adds enrichment data.

• If you send context data, such as a hostname, for example, three days after the initial log, the system re-enriches the UDM event. Rules that look for this re-enriched data then run again and create a detection. This feature ensures that the system creates detections even when context is delayed.

• Changes in enrichment data can cause a rule to match later, even if it didn't initially. For example: An event at 9:03 AM has entity.ip_geo_artifact.country_or_region = USA. The rules engine runs at 9:10 AM, querying 9:00-10:00 AM, and the rule doesn't match. Later, enrichment reprocessing updates the geolocation to Canada. When the rule runs again, it now matches, and the system creates a detection.

Entity context graph processing

The system generates and adds entity context graph (ECG) information to log data to provide context, for example, indicators of compromise (IOCs) or asset context data. Because the ECG pipeline relies mostly on batch processing, entity context information often updates only after a rule execution creates a detection.

Need more help? Get answers from Community members and Google SecOps professionals.