Change log for DARKTRACE

Date Changes
2025-05-20 Enhancement:
- Added grok pattern for new log format.
- Updated grok pattern for `dvc` raw field to parse IP correctly.
- `event.idm.read_only_udm.security_result.rule_id`: Newly Mapped `issue_code` raw field with `event.idm.read_only_udm.security_result.rule_id` UDM field.
- `event.idm.read_only_udm.security_result.detection_fields` : Newly Mapped `tags` raw field with `event.idm.read_only_udm.security_result.detection_fields` UDM field.
- `event.idm.read_only_udm.security_result.severity`: Newly Mapped `severity` raw field with `event.idm.read_only_udm.security_result.severity` UDM field.
- 'event.idm.read_only_udm.security_result.attack_details.techniques': Newly Mapped `mitrld` raw field with `event.idm.read_only_udm.security_result.attack_details.techniques` UDM field.
- `event.idm.read_only_udm.principal.mac`: Newly Mapped `DevicemacAddress` raw field with `event.idm.read_only_udm.principal.mac` UDM field.
- `event.idm.read_only_udm.additional.fields`: Removed mapping of `externalId` raw field from `event.idm.read_only_udm.additional.fields` UDM field.
- `event.idm.read_only_udm.security_result.threat_id`: Newly Mapped `externalId` raw field with `event.idm.read_only_udm.security_result.threat_id` UDM field.
2025-05-19 Enhancement:
- event.idm.read_only_udm.intermediary.ip: Newly mapped `host_ip` raw log field with `event.idm.read_only_udm.intermediary.ip` UDM field.
- Replaced "event.idm.read_only_udm.intermediary" with "intermediary".
2025-05-19 Enhancement:
- event.idm.read_only_udm.intermediary.ip: Newly mapped `host_ip` raw log field with `event.idm.read_only_udm.intermediary.ip` UDM field.
- Replaced "event.idm.read_only_udm.intermediary" with "intermediary".
2025-05-09 Enhancement:
- Added a gsub to replace "\r\n" with " " in "message" field.
- event.idm.read_only_udm.principal.ip, event.idm.read_only_udm.principal.asset.ip: Removed mapping of `dvc` from `event.idm.read_only_udm.principal.ip, event.idm.read_only_udm.principal.asset.ip` UDM field and mapped `dvc_ip` instead.
- Added a `on_error` check before mapping `dvc_ip` to `principal_ip` raw log field.
- Added a Grok pattern to extract `device_ip2_` from `device_ip2` raw log field.
- event.idm.read_only_udm.principal.ip, event.idm.read_only_udm.principal.asset.ip: Removed mapping of `ip.ip` from `event.idm.read_only_udm.principal.ip, event.idm.read_only_udm.principal.asset.ip` UDM field and mapped `device_ip2_` instead.
- Added a `on_error` check before mapping `device_ip2_` to `principal_ip` raw log field.
- event.idm.read_only_udm.security_result.attack_details.techniques: Newly mapped `technique_data` raw log field with `event.idm.read_only_udm.security_result.attack_details.techniques` UDM field.
- event.idm.read_only_udm.security_result.attack_details.tactics: Newly mapped `tactics_data` raw log field with `event.idm.read_only_udm.security_result.attack_details.tactics` UDM field.
2025-04-28 Enhancement:
- Added the Grok patterns to parse the new pattern of logs with SYSLOG format.
- event.idm.read_only_udm.metadata.event_timestamp: Newly mapped `event_time` and `timestamp` fields with `event.idm.read_only_udm.metadata.event_timestamp` UDM field.
- Set `event.idm.read_only_udm.security_result.severity` UDM field to `INFORMATIONAL` if the value of `severity` field is 0, 1 or 2.
- Set `event.idm.read_only_udm.security_result.severity` UDM field to `LOW` if the value of `severity` field is 3 or 4.
- Set `event.idm.read_only_udm.security_result.severity` UDM field to `MEDIUM` if the value of `severity` field is 5 or 6.
- Set `event.idm.read_only_udm.security_result.severity` UDM field to `HIGH` if the value of `severity` field is 7 or 8 and set `event.idm.is_alert` and `event.idm.is_significant` UDM fields to `true`.
- Set `event.idm.read_only_udm.security_result.severity` UDM field to `CRITICAL` if the value of `severity` field is 9 or 10 and set `event.idm.is_alert` and `event.idm.is_significant` UDM fields to `true`.
2025-04-22 Enhancement:
- event.idm.read_only_udm.security_result.severity: Newly mapped severity raw log field with event.idm.read_only_udm.security_result.severity UDM field.
2025-04-08 Enhancement:
- `event.idm.read_only_udm.principal.user.userid`: Removed mapping of `model_uuid` raw field from `event.idm.read_only_udm.principal.user.userid` UDM field.
- `event.idm.read_only_udm.metadata.product_log_id`: Mapped `model_uuid` raw field to `event.idm.read_only_udm.metadata.product_log_id` UDM field.
- `event.idm.read_only_udm.metadata.product_version`: Newly Mapped `version` raw field to `event.idm.read_only_udm.metadata.product_version` UDM field.
- `event.idm.read_only_udm.metadata.vendor_name`: Newly Mapped `vendor` raw field to `event.idm.read_only_udm.metadata.vendor_name` UDM field.
- `event.idm.read_only_udm.metadata.product_name`: Newly Mapped `product` raw field to `event.idm.read_only_udm.metadata.product_name` UDM field.
- `event.idm.read_only_udm.metadata.product_log_id`: Newly Mapped `deviceExternalId` raw field to `event.idm.read_only_udm.metadata.product_log_id` UDM field.
- `event.idm.read_only_udm.principal.labels` : Newly Mapped `cs3` raw field to `event.idm.read_only_udm.principal.labels` UDM field.
- Added a Grok pattern to support new `syslog` format.
2025-03-12 Enhancement:
- When "model.created.userid" is present, then mapped "model.created.userid" to "principal.user.userid" and "model.uuid" to "principal.user.product_object_id".
- Mapped "model.edited.userid" to "additional.fields" with key as "model.edited.by".
2025-01-16 Enhancement:
- Added support to parse unparsed logs.
2024-11-21 Enhancement:
- Corrected spelling of "email_direciton_error" to "email_direction_error".
2024-11-21 Enhancement:
- Corrected spelling of "email_direciton_error" to "email_direction_error".
2024-10-30 Enhancement:
- Added a Grok pattern to parse the SYSLOG+JSON logs.
2024-10-25 Enhancement:
- Mapped "direction" to "network.direction" for "INBOUND," "OUTBOUND," or "BROADCAST." Otherwise, mapped it to "additional.fields".
2024-10-24 Enhancement:
- Added a new Grok pattern to parse new JSON logs.
2024-10-08 Enhancement:
- changed the "event_type" from "USER_UNCATEGORIZED" to "EMAIL_UNCATEGORIZED" when "from" field is present.
- "from" mapped to "newtwork.email.from" & "principal.user.email_addresses".
- "recipients" mapped to "newtwork.email.to" & "target.user.email_addresses".
- changed "subject" mapping from "metadata.description" to "newtwork.email.subject".
- changed "message_id" mapping from "additional.fields" to "newtwork.email.mail_id".
- changed "uuid" mapping from "principal.user.userid" to "metadata.product_log_id".
2024-10-07 Enhancement:
- Mapped "filterType" under "triggeredFilters" to "additional.fields".
- When "trigger.value" is having non IP value, then mapped "trigger.value" under "triggeredFilters" to "additional.fields".
2024-09-25 Enhancement:
- Mapped "description" to "metadata.description".
- Mapped "score" field to "security_result.priority_details".
2024-09-19 Enhancement:
- Mapped all fields under "triggeredFilters" to "additional.fields".
2024-09-09 Enhancement:
- Mapped "uuid" to "principal.user.userid".
- Mapped "from" to "principal.user.email_addresses".
- Mapped "subject" to "metadata.description".
- Mapped "anomaly_score", "tags", "link_hosts", and "message_id" to "additional.fields".
- Mapped "recipients" to "observer.user.email_addresses".
- Mapped "attachment_sha1s" and "attachment_sha256s" to "security_result.detection_fields".
2024-08-29 Enhancement:
- Mapped "hostname" field to "principal.hostname" and "principal.asset.hostname".
- Mapped "label" field to "security_result.attribute.label".
- Mapped "ip_address" field to "principal.ip" and "principal.asset.ip".
- Mapped "priority" field to "security_result.priority_details".
- Mapped "priority_level" field to "security_result.priority".
- Mapped "alert_name" field to "security_result.rule_name".
- Mapped "message" field to "security_result.description".
- Mapped "url" field to "security_result.url_back_to_product".
2024-08-06 Enhancement:
- When "filterType" is "Destination IP", then mapped "triggeredFilter.trigger.value" to "target.ip".
- When principal and target machine data is absent but user data is available then mapped "metadata.event_type" to "USER_UNCATEGORIZED".
2024-04-05 Bug-Fix:
- Changed mapping for "model.name" and "model.now.name" from "principal.user.user_display_name" to "metadata.product_event_type".
- When principal machine data and target machine data are present, then changed mapping for "metadata.event_type" from "GENERIC_EVENT" or "USER_UNCATEGORIZED" to "NETWORK_CONNECTION", else mapping it to "USER_RESOURCE_ACCESS".
2023-12-20 Bug-Fix: Fixed the flaky results for the mapping "sec_result.about.resource.attribute.labels" where "key" is "details".
2023-11-20 Enhancement, Bug-Fix:
- Parsed subfields in the "message" field of the raw log.
- Mapped "uuid" to "principal.user.userid" and set "metadata.event_type" to "USER_UNCATEGORIZED" when "uuid" is present.
- Mapped "direction" to "network.direction".
- Mapped "from" to "network.email.from".
- Mapped "subject" to "network.email.subject".
- Mapped "attachment_sha1s", "attachment_sha256s", "recipients", "link_hosts", "tags", "actions", "anomaly_score", "message_id" to "security_result.detection_fields".
- Mapped "url" to "security_result.url_back_to_product".
- Mapped "severity" to "security_result.severity".
- Mapped "hostname" to "principal.hostname".
- Added "on_error" to a JSON block to parse unparsed set of JSON logs.
- Mapped "model.pid" to "principal.process.pid".
- Mapped "model.uuid" to "principal.user.userid".
- Mapped "model.name" to "principal.user.user_display_name".
- Mapped "breachUrl" to "security_result.url_back_to_product".
- Mapped "device.typelabel", "device.sid", "device.typename" to "principal.resource.attribute.labels".
- Mapped "device.ip" to "principal.ip".
- Mapped "device.ips.0.subnet" to "additional_fields".
- Mapped "device.did" to "principal.asset.asset_id".
- Mapped "device.customFields.DT-AUTO.macaddress" to "principal.mac".
- Mapped "device.firstSeen" to "principal.asset.first_seen_time".
- Mapped "device.device.lastSeen" to "principal.asset.last_seen_time".
- Mapped "mitreTechniques" to "security_result.attack_details.techniques".
2023-09-26 Enhancement:
- Adjusted the parser to support nested JSON.
- Fixed the parser to handle special characters in the log.
- Mapped the fields of new log type.
2023-08-29 Enhancement:
- Mapped "details" to "sec_result.about.resource.attribute.labels".
- Mapped "principal_port_no" to "principal.port".
- Mapped "ip_protocol" to "network.ip_protocol".
- Mapped "location" to "principal.location.country_or_region".
- Mapped "target_host" to "target.hostname".
- Mapped "target_ip" to "target.ip".
- Mapped "source_ip" to "principal.ip".
- Mapped "source_port" to "principal.port".
- Mapped "dest_ip" to "target.ip".
- Mapped "dest_port" to "target.port".
- Mapped "@host" to "principal.hostname".
- Mapped "uid" to "principal.user.userid".
- Mapped "note" to "principal.application".
- Mapped "@type" to "sec_result.about.resource.attribute.labels".
- Mapped "opcode" to "sec_result.about.resource.attribute.labels".
- Mapped "trans_id" to "sec_result.about.resource.attribute.labels".
- Mapped "query_class" to "sec_result.about.resource.attribute.labels".
2023-07-14 Enhancement:
- Mapped "dvchost" to "principal.hostname".
- Mapped "deviceMacAddress" to "principal.mac".
- Modified mapping of "dvc" to map to "principal.ip" only if it's a valid IP address.
2023-03-24 Enhancement:
- Mapped 'model.now.category' to 'security_result.severity'.
- Mapped 'model.now.message' to 'security_result.description'.
- Mapped 'model.now.description' to 'metadata.description'.
- Mapped 'model.now.uuid' to 'principal.user.userid'.
- Mapped 'model.now.pid' to 'principal.process.pid'.
- Mapped 'model.now.name' to 'principal.user.user_display_name'.
- Mapped 'score' to 'security_result.priority'.
- Mapped 'triggeredComponents.port' to 'intermediary.port'.
- Mapped 'triggeredComponents.ip' to 'intermediary.ip'.
- Mapped 'device.ip' to 'principal.ip'.
- Mapped 'device.macaddress' to 'principal.mac'.
- Mapped 'device.hostname' to 'principal.hostname'.
- Mapped 'model.then.logic.data.cid', 'model.now.logic.data.cid', 'model.now.tags' to 'additional.fields'.
- Mapped 'Mapped 'model.then.description', 'model.then.uuid', 'model.then.name', 'model.then.pid' to 'principal.resource.attribute.labels'.
- Modified 'metadata.event_type' from 'GENERIC_EVENT' to 'STATUS_UPDATE' wherver 'principal.ip' or 'principal.hostname' is present.
2022-10-31 Enhancement:
- Mapped the field 'time' to 'metadata.event_timestamp'.
- Mapped the field 'model.description' to 'metadata.description'.
- Mapped the field 'model.name' to 'principal.user.user_display_name'.
- Mapped the field 'model.pid' to 'principal.process.pid'.
- Mapped the field 'device.did' to 'principal.asset.asset_id'.
- Mapped the field 'device.objecttype' to 'principal.asset.type'.
- Mapped the field 'device.ips' to 'principal.ip'.
- Mapped the field 'device.firstSeen' to 'principal.asset.first_seen_time'.
- Mapped the field 'device.lastSeen' to 'principal.asset.last_discover_time'.
- Mapped the fields 'device.sid', 'device.typename' and 'device.typelabel' to 'principal.resource.attribute.labels'.
- Mapped the field 'model.tags' and 'model.logic.data' to 'additional.fields'.
- Mapped the field 'breachUrl' to 'security_result.url_back_to_product'.
- Mapped the field 'mitreTechniques' to 'security_result.detection_fields'.
- Added conditional checks for 'details.0.0.contents.2.values.0' mapped to 'principal.port'.
- Dropped the logs having incorrect json format.
2022-10-13 Added grok to parse new json type logs.
Mapped 'category' to 'security_result.severity'.
Mapped 'title' to 'security_result.summary'.
Mapped 'details.0.0.contents.1.values.0.hostname' to 'principal.hostname'.
Mapped 'details.0.0.contents.1.values.0.ip' to 'principal.ip'.
Mapped 'details.0.0.contents.2.values.0' to 'principal.port'.
Mapped 'details.0.0.contents.4.values.0' to 'principal.location.country_or_region'.
Mapped 'details.0.1.contents.0.values.0.hostname' to 'target.hostname'.
Mapped 'details.0.1.contents.0.values.0.ip' to 'target.ip'.
Mapped 'incidentEventUrl' to 'principal.url'.
Mapped 'summary' to 'metadata.description'.
Mapped 'model.uuid' to 'principal.user.userid'.
Mapped 'relatedBreaches.0.modelName' to 'security_result.description'.
2022-04-22 Added support for issue code being non-numeric in CEF message