Metrics in UDM search using YARA-L 2.0

Metrics functions in Search let you analyze aggregated historical behavioral data in your search results. You can include these functions in the outcome section of the UDM search query.

Function parameters

Metrics functions can perform entity behavioral analytics within your searches. For example, you can construct a search query to determine the maximum number of daily bytes a specific IP address has sent within the past month. To represent the specific IP address, you can directly input the IP value in your search query instead of using placeholder variables.

Due to the large number of arguments used in these functions, they use named parameters, which can be specified in any order. The parameters are as follows:

Period

The length of time over which individual log events are combined into a single observation. The only values allowed are 1h and 1d.

Window

The length of time over which individual observations are aggregated into a single value, such as the average and the maximum. The allowed values for the window are based on the metric period. The valid mappings are as follows:

period:1h : window:today

period:1d : window:30d

Example: The following query returns the first and last time a user logged in successfully in a given day, over the last 30 days.

  $u1.principal.asset.ip = $ip
    $ip = "10.138.0.4"
    $user = $u1.target.user.userid

  match:
    $ip, $user

  outcome:

    // First and last time a user logged in successfully

     $first_seen = max(metrics.auth_attempts_success(
        period:1d, window:30d,
        metric:first_seen,
        agg:min,
        target.user.userid:$user))


    $last_seen = max(metrics.auth_attempts_success(
        period:1d, window:30d,
        metric:last_seen,
        agg:max,
        target.user.userid:$user))

Metrics

Within each period, each observation has a number of metrics associated with it. One of these must be selected for aggregation over the entire window. The supported metric types are:

• event_count_sum—Number of unique log events within each period.

• first_seen—First seen timestamp of a matching log event within each period.

• last_seen—Last seen timestamp of a matching log event within each period.

• value_sum—Sum of the number of bytes in all the log events combined within the period. You can only use this value for metrics functions with bytes in the name.

• num_unique_filter_values—Metric that isn't pre-computed by Google SecOps but can be computed during search query execution. See Count unique metrics for more details and requirements.

Aggregations

Aggregations are applied over the entire window (for example, the highest daily value over the past 30 days). The allowed values are:

• avg—Average value per period. This is a statistical mean that doesn't include zero values.

• max—Greatest value per period.

• min—Smallest value per period.

• num_metric_periods—Number of periods within the time window that had a non-zero metric value.

• stddev—Standard deviation of the value per period. This is a statistical standard deviation which doesn't include zero values.

• sum—Sum of each value per period, over the entire window.

• earliest—Timestamp of the first (oldest) event with microsecond resolution.

• latest—Timestamp of the last (most recent) event with microsecond resolution.

For examples, see Statistics and aggregations in UDM search using YARA-L 2.0.

Filter

Filters allow filtering metrics before aggregation by a value in the pre-computed metric (see values in Metric). Filters can be any valid event expression (a single line in the event section) which doesn't contain any event fields or placeholders. The only variables that can be included in this condition are metric types.

Example: The following outcome section of a query returns the maximum number of daily bytes that a specific IP address has sent in the past month.

$ip = principal.asset.ip
match:
    $ip
outcome:
$max_bytes_per_day = max(metrics.network_bytes_outbound(
    period: 1d, window: 30d,
    metric: value_sum,
    agg:max,
    principal.asset.ip: $ip,
    filter:value_sum > 10 AND event_count_sum > 2))

UDM fields

Metrics are filtered by one, two, or three UDM fields, depending on the function. For information on functions, see Functions.

The following types of UDM fields are used for metrics functions:

• Dimension (Required): Different combinations are listed on this page. You can't join a metric with a default value ("" for string and 0 for int).
• Namespaces (Optional): You can only use namespaces for entities that you specify in dimensions. For example, if you use principal.asset.hostname filter, you can use a principal.namespace filter as well. If you don't include a namespace filter, the data across all namespaces is aggregated together. You can use a default value as a namespace filter.

Window calculations

Google SecOps calculates metrics using either a daily or hourly metric window.

Daily windows

All daily windows, such as 30d, are determined the same way. Google SecOps uses the latest available metrics data that has been generated that doesn't overlap with the search query time range. Calculation of the daily metrics can take up to 6 hours to complete and doesn't start until the end of the day in UTC. Metric data for the day before is available at or before 6:00 UTC each day.

For example, for a search query that is running over event data from 2023-10-31 4:00 UTC to 2023-10-31 7:00 UTC, the daily metrics for 2023-10-30 is likely generated, so the metric calculation uses the data from 2023-10-01 to 2023-10-30 (inclusive). Whereas for a search query that is running over event data from 2023-10-31 1:00 UTC to 2023-10-31 3:00 UTC, the daily metrics for 2023-10-30 is likely not generated, so the metric calculation uses the data from 2023-09-30 to 2023-10-29 (inclusive).

Hourly today window

The hourly metric window is calculated differently from the window for daily metrics. The hourly metric window of today is not a static size like the 30d window for daily metrics. The hourly metric window today fills in as much data as possible between the end of the daily window and the start of the search query time window.

For example, for a search query that is running over event data from 2023-10-31 4:00:00 UTC to 2023-10-31 7:00:00 UTC, the daily metric calculation uses the data from 2023-10-01 to 2023-10-30 (inclusive) and the hourly metric window uses data from 2023-10-31 00:00:00 UTC to 2023-10-31 4:00:00 UTC.

Count unique metrics

The metric num_unique_filter_values is not pre-computed by Google SecOps. It is computed when the search query is run. This is done by aggregating over an existing dimension in a pre-computed metric. For example, the metric "daily total count of distinct countries that a user attempted to authenticate in" can be derived from the pre-computed auth_attempts_total metric on the dimensions target.user.userid and principal.ip_geo_artifact.location.country_or_region by performing a count unique aggregation over the latter dimension.

Example:

$userid = target.user.userid
match:
    $userid
outcome:
$outcome_variable = max(metrics.auth_attempts_total(
    period: 1d,
    window: 30d,
    // This metric type indicates any filter with a wildcard value should be
    // aggregated over each day to produce a new metric dynamically
    metric: num_unique_filter_values,
    agg: max,
    target.user.userid: $userid,
    // Filter whose value should be counted over each day to produce the
    // num_unique_filter_values metric.
    principal.ip_geo_artifact.location.country_or_region: *
))

Limitation: Computing unique metric count can only be aggregated over a single filter dimension. This is indicated by using the wildcard token * as the filter value.

Functions

This section includes documentation on the specific metrics functions supported by Google SecOps.

Alert Events

metrics.alert_event_name_count precomputes historical values for UDM events that have had alerts generated by Carbon Black, CrowdStrike Falcon, Microsoft Graph API Alerts, or Microsoft Sentinel.

Authentication Attempts

metrics.auth_attempts_total precomputes historical values for UDM events with a USER_LOGIN event type.

metrics.auth_attempts_success further requires that the event had at least one SecurityResult.Action of ALLOW.

metrics.auth_attempts_fail requires instead that none of the SecurityResult.Actions were ALLOW.

DNS Bytes Outbound

metrics.dns_bytes_outbound precomputes historical values for UDM events where network.sent_bytes is greater than 0, and the target port is 53/udp, 53/tcp, or 3000/tcp. network.sent_bytes is available as value_sum.

DNS Queries

metrics.dns_queries_total precomputes historical values for UDM events that have a value in network.dns.id.

metrics.dns_queries_success further requires that the network.dns.response_code was 0 (NoError).

metrics.dns_queries_fail only considers events with a network.dns.response_code greater than 0.

File Executions

metrics.file_executions_total precomputes historical values for UDM events with a PROCESS_LAUNCH event type.

metrics.file_executions_success further requires that the event had at least one SecurityResult.Action of ALLOW.

metrics.file_executions_fail instead requires that none of the SecurityResult.Actions were ALLOW.

HTTP Queries

metrics.http_queries_total precomputes historical values for UDM events that have a value in network.http.method.

metrics.http_queries_success further requires that network.http.response_code is less than 400.

metrics.http_queries_fail only considers events with a network.http.response_code is greater than or equal to 400.

Network Bytes

metrics.network_bytes_inbound precomputes historical values for UDM events that have a non-zero value for network.received_bytes, and makes that field available as value_sum.

metrics.network_bytes_outbound requires a non-zero value for network.sent_bytes, and makes that field available as value_sum.

metrics.network_bytes_total considers events that have a non-zero value for either network.received_bytes or network.sent_bytes (or both), and make the sum of those two fields available as value_sum.

Resource Creation

metrics.resource_creation_total precomputes historical values for UDM events with a RESOURCE_CREATION event type or a USER_RESOURCE_CREATION event type.

For a list of equivalent event types, see Metadata Event Types

metrics.resource_creation_success further requires that the event have at least one SecurityResult.Action of ALLOW.

Resource Deletion

metrics.resource_deletion_success precomputes historical values for UDM events with a RESOURCE_DELETION event type and further requires that the event have at least one SecurityResult.Actions of ALLOW.

Resource Read

metrics.resource_read_success precomputes historical values for UDM events with a RESOURCE_READ event type and further requires that the event have at least one SecurityResult.Action of ALLOW.

metrics.resource_read_fail instead requires that none of the SecurityResult.Actions are ALLOW.

Limitations

When creating YARA-L search queries with metrics, the following limitations apply:

• You can't join a metric with a default value ("" for string and 0 for int).
  • Default values:
    • If there is no metric data that corresponds to an event, the returned value from the metrics function is 0.
    • If there is an event which has no metric data, using min to aggregate over the function might return 0.
• Metrics functions can only be used in the outcome section. They must be aggregated in search queries with a match section.

Need more help? Get answers from Community members and Google SecOps professionals.