[[["易于理解","easyToUnderstand","thumb-up"],["解决了我的问题","solvedMyProblem","thumb-up"],["其他","otherUp","thumb-up"]],[["很难理解","hardToUnderstand","thumb-down"],["信息或示例代码不正确","incorrectInformationOrSampleCode","thumb-down"],["没有我需要的信息/示例","missingTheInformationSamplesINeed","thumb-down"],["翻译问题","translationIssue","thumb-down"],["其他","otherDown","thumb-down"]],["最后更新时间 (UTC)：2025-09-02。"],[],[],null,["# Use deduplication in search and dashboards\n==========================================\n\nSupported in: \nGoogle secops [SIEM](/chronicle/docs/secops/google-secops-siem-toc)\n\nThis document explains what happens when you search data in\nGoogle Security Operations. Sometimes, results can include duplicates. This often\noccurs because enterprise infrastructure generates logs for the same event from\nmultiple systems. For example, both your authentication and security systems\nmight log a single login event.\n\nTo reduce duplicate results, use UDM fields in the `dedup` section in your\nYARA-L syntax. Add UDM fields to this section to return a single result for each\ndistinct combination of values. .\n\nDeduplication in queries\n------------------------\n\nDeduplication applies to the following types of search and dashboard queries:\n\n- **Aggregated search queries** : Includes `match`, `match` and `outcome`, or\n `aggregated outcome` sections. Deduplication occurs after outcomes are\n determined.\n\n For aggregated search queries, include these fields to the `dedup` section:\n - Fields from the `match` section\n - Fields from the `outcome` section\n- **UDM search queries** : Exclude the `match`, `outcome`, or `aggregated\n outcome` sections. Note that UDM search queries can include an `outcome`\n section as long as there aren't any aggregates and there isn't a `match`\n section.\n\n For UDM queries, add these fields to the `dedup` section:\n - Any non-repeated, non-array, and non-grouped event fields\n - Placeholder fields from the `events` section\n - Outcome variables from the `outcome` section\n\nDeduplication examples in Search\n--------------------------------\n\nThis section shows the YARA-L syntax and can be run in\n[Search](/chronicle/docs/investigation/udm-search).\n\n### Example: Simple search for unique IP addresses\n\nThe following example search displays network connections between events where a\nunique IP address within your enterprise (`principal.ip`) connects to a unique,\nexternal IP address outside of your enterprise (`target.ip`). The events are\ndeduplicated based on the `principal.ip`. \n\n events:\n metadata.event_type = \"NETWORK_CONNECTION\"\n target.ip != \"\"\n principal.ip != \"\"\n match:\n target.ip, principal.ip\n dedup:\n principal.ip\n\n### Example: Unique IP addresses\n\nSimilar to the previous example, the following example search displays network\nconnection events with unique IP addresses. Applying `dedup` to `principal.ip`\nnarrows results to events associated with unique IPs. The `outcome` section\ndisplays the total bytes sent between `principal.ip` and `target.ip`, ordering\nresults from highest to lowest traffic volume. \n\n events:\n metadata.event_type = \"NETWORK_CONNECTION\"\n target.ip != \"\"\n principal.ip != \"\"\n match:\n target.ip, principal.ip\n outcome:\n $total_bytes = sum(network.sent_bytes)\n dedup:\n principal.ip\n order:\n $total_bytes desc\n\n### Example: Simple search for unique hostnames\n\nThe following example searches for each unique hostname accessed from your\nenterprise. Applying `dedup` to `target.hostname` narrows results to events\nassociated with unique external hostnames. \n\n metadata.log_type != \"\"\n dedup:\n target.hostname\n\nThe following is an equivalent example without the `dedup` option. It typically\nreturns substantially more events. \n\n metadata.log_type != \"\" AND target.hostname != \"\"\n\n### Example: Unique hostnames\n\nSimilar to the previous example, this search displays network connection events\nwith unique hostnames. Applying the `dedup` option to `principal.hostname`\nnarrows results to events associated with unique hosts: \n\n events:\n metadata.event_type = \"NETWORK_CONNECTION\"\n target.hostname != \"\"\n principal.hostname != \"\"\n match:\n target.hostname, principal.hostname\n outcome:\n $total_bytes = sum(network.sent_bytes)\n dedup:\n principal.hostname\n order:\n $total_bytes desc\n\n**Need more help?** [Get answers from Community members and Google SecOps professionals.](https://security.googlecloudcommunity.com/google-security-operations-2)"]]
