Collect Proofpoint On-Demand logs
This document explains how to ingest Proofpoint On-Demand logs to Google Security Operations using built-in API integration. The parser extracts fields from JSON logs, transforming them into the Chronicle UDM format. It handles two primary log formats: one containing email metadata and the other containing SMTP transaction details, using conditional logic to parse fields appropriately and populate the Unified Data Model (UDM) fields based on the available data. The parser also performs data cleaning, such as removing extraneous characters and converting timestamps.
Before you begin
Make sure you have the following prerequisites:
- Google SecOps instance
- Proofpoint On-Demand Remote Syslog license
- Privileged access to Proofpoint
Configure Proofpoint On-Demand API access
- Sign in to the Proofpoint Admin portal.
- Copy your Cluster ID (displayed on the upper-right corner of your management interface, next to the release number).
- Go to Settings > API Key Management.
- Click Create New to open the Create New API Key dialog.
- Enter a unique Name (for example,
Google SecOps Key
). - Generate the API Key.
- Select View Details from the ellipsis menu on the new API Key.
- Copy the API Key.
Set up feeds
There are two different entry points to set up feeds in the Google SecOps platform:
- SIEM Settings > Feeds
- Content Hub > Content Packs
Set up feeds from SIEM Settings > Feeds
To configure a feed, follow these steps:
- Go to SIEM Settings > Feeds.
- Click Add New Feed.
- On the next page, click Configure a single feed.
- In the Feed name field, enter a name for the feed (for example,
PoD Logs
). - Select Third Party API as the Source type.
- Select the Proofpoint On-Demand log type.
- Click Next.
Specify values for the following input parameters:
- Authentication HTTP headers: Enter the Proofpoint API Key in a
Authorization: Bearer {API_KEY}
format, and add in a new line with the following WebSocket-Key data:Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==
. - Cluster ID: Enter the Proofpoint Cluster ID that you copied earlier.
- Authentication HTTP headers: Enter the Proofpoint API Key in a
Click Next.
Review the feed configuration in the Finalize screen, and then click Submit.
Set up feeds from the Content Hub
Specify values for the following fields:
- Authentication HTTP headers: Enter the Proofpoint API Key in a
Authorization: Bearer {API_KEY}
format, and add in a new line with the following WebSocket-Key data:Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==
. - Cluster ID: Enter the Proofpoint Cluster ID that you copied earlier.
Advanced options
- Feed Name: A prepopulated value that identifies the feed.
- Source Type: Method used to collect logs into Google SecOps.
- Asset Namespace: Namespace associated with the feed.
- Ingestion Labels: Labels applied to all events from this feed.
UDM mapping table
Log Field | UDM Mapping | Logic |
---|---|---|
classification |
security_result.detection_fields.classification |
The value comes directly from the classification field in the raw log. |
cluster |
security_result.detection_fields.cluster |
The value comes directly from the cluster field in the raw log. |
completelyRewritten |
security_result.detection_fields.completelyRewritten |
The value comes directly from the completelyRewritten field in the raw log. |
connection.country |
principal.location.country_or_region |
The value comes directly from the connection.country field in the raw log, unless it is "**". |
connection.host |
principal.hostname |
The value comes directly from the connection.host field in the raw log. |
connection.ip |
principal.ip |
The value comes directly from the connection.ip field in the raw log, if it is a valid IPv4 address. It is also merged with senderIP if present. |
connection.protocol |
network.application_protocol |
The protocol part before the colon in connection.protocol is extracted using gsub and mapped. For example, "smtp:smtp" becomes "SMTP". |
connection.tls.inbound.cipher |
network.tls.cipher |
The value comes directly from the connection.tls.inbound.cipher field in the raw log, unless it is "NONE". |
connection.tls.inbound.version |
network.tls.version |
The value comes directly from the connection.tls.inbound.version field in the raw log, unless the cipher is "NONE". |
envelope.from |
network.email.from |
The value comes directly from the envelope.from field in the raw log. It is also replaced by sm.from or fromAddress if present. |
envelope.rcpts |
network.email.to |
The email addresses in envelope.rcpts are merged into the network.email.to field if they are valid email addresses. It is also merged with sm.to or toAddresses if present. |
envelope.rcptsHashed |
read_only_udm.additional.fields |
The hashed email addresses in envelope.rcptsHashed are added as additional fields with keys like "toHashed_0", "toHashed_1", etc. |
eventTime |
@timestamp |
The value is parsed from the eventTime field using the ISO8601 or RFC 3339 format. |
eventType |
security_result.summary |
The value comes directly from the eventType field in the raw log. |
filter.disposition |
security_result.action_details |
The value comes directly from the filter.disposition field in the raw log, unless tls.verify is present. |
filter.modules.av.virusNames.0 |
security_result.threat_name |
The value comes directly from the filter.modules.av.virusNames.0 field in the raw log. |
filter.modules.dmarc.authResults |
read_only_udm.additional.fields |
The method and result from each entry in filter.modules.dmarc.authResults are added as additional fields with keys like "authResultsMethod_0", "authResults_result_0", "authResultsMethod_1", etc. |
filter.modules.spam.langs |
read_only_udm.additional.fields |
Each language in filter.modules.spam.langs is added as an additional field with keys like "lang_0", "lang_1", etc. |
filter.modules.spam.safeBlockedListMatches.0.listType |
security_result.detection_fields.safeBlockedListMatches_listType |
The value comes directly from the filter.modules.spam.safeBlockedListMatches.0.listType field in the raw log. |
filter.modules.spam.safeBlockedListMatches.0.rule |
security_result.detection_fields.safeBlockedListMatches_rule |
The value comes directly from the filter.modules.spam.safeBlockedListMatches.0.rule field in the raw log. |
filter.modules.spam.scores.classifiers.adult |
security_result.detection_fields.adult |
The value comes directly from the filter.modules.spam.scores.classifiers.adult field in the raw log, if it is not empty or 0. |
filter.modules.spam.scores.classifiers.bulk |
security_result.detection_fields.bulk |
The value comes directly from the filter.modules.spam.scores.classifiers.bulk field in the raw log, if it is not empty or 0. |
filter.modules.spam.scores.classifiers.impostor |
security_result.detection_fields.impostor |
The value comes directly from the filter.modules.spam.scores.classifiers.impostor field in the raw log, if it is not empty or 0. |
filter.modules.spam.scores.classifiers.lowpriority |
security_result.detection_fields.lowpriority |
The value comes directly from the filter.modules.spam.scores.classifiers.lowpriority field in the raw log, if it is not empty or 0. |
filter.modules.spam.scores.classifiers.malware |
security_result.detection_fields.malware |
The value comes directly from the filter.modules.spam.scores.classifiers.malware field in the raw log, if it is not empty or 0. |
filter.modules.spam.scores.classifiers.mlx |
security_result.detection_fields.mlx |
The value comes directly from the filter.modules.spam.scores.classifiers.mlx field in the raw log, if it is not empty or 0. |
filter.modules.spam.scores.classifiers.mlxlog |
security_result.detection_fields.mlxlog |
The value comes directly from the filter.modules.spam.scores.classifiers.mlxlog field in the raw log, if it is not empty or 0. |
filter.modules.spam.scores.classifiers.phish |
security_result.detection_fields.phish |
The value comes directly from the filter.modules.spam.scores.classifiers.phish field in the raw log, if it is not empty or 0. |
filter.modules.spam.scores.classifiers.spam |
security_result.detection_fields.spam |
The value comes directly from the filter.modules.spam.scores.classifiers.spam field in the raw log, if it is not empty or 0. |
filter.modules.spam.scores.classifiers.suspect |
security_result.detection_fields.suspect |
The value comes directly from the filter.modules.spam.scores.classifiers.suspect field in the raw log, if it is not empty or 0. |
filter.modules.spam.scores.engine |
security_result.detection_fields.engine |
The value comes directly from the filter.modules.spam.scores.engine field in the raw log, if it is not empty or 0. |
filter.modules.spam.scores.overall |
security_result.detection_fields.overall |
The value comes directly from the filter.modules.spam.scores.overall field in the raw log, if it is not empty or 0. |
filter.modules.spam.version.definitions |
security_result.summary |
The value comes directly from the filter.modules.spam.version.definitions field in the raw log. |
filter.modules.spam.version.engine |
metadata.product_version |
The value comes directly from the filter.modules.spam.version.engine field in the raw log. |
filter.modules.urldefense.counts.rewritten |
read_only_udm.additional.fields.urldefenseCountsRewritten |
The value comes directly from the filter.modules.urldefense.counts.rewritten field in the raw log. |
filter.modules.urldefense.counts.total |
security_result.detection_fields.urldefense_total |
The value comes directly from the filter.modules.urldefense.counts.total field in the raw log. |
filter.modules.zerohour.score |
read_only_udm.additional.fields.zeroHourScore |
The value comes directly from the filter.modules.zerohour.score field in the raw log. |
filter.origGuid |
read_only_udm.additional.fields.origGuid |
The value comes directly from the filter.origGuid field in the raw log. |
filter.qid |
read_only_udm.additional.fields.filterQid |
The value comes directly from the filter.qid field in the raw log. |
filter.quarantine.folder |
security_result.detection_fields.filter_quarantine_folder |
The value comes directly from the filter.quarantine.folder field in the raw log. |
filter.quarantine.folderId |
security_result.detection_fields.filter_quarantine_folderId |
The value comes directly from the filter.quarantine.quarantine.folderId field in the raw log. |
filter.quarantine.module |
security_result.detection_fields.filter_quarantine_module |
The value comes directly from the filter.quarantine.module field in the raw log. |
filter.quarantine.rule |
security_result.detection_fields.filter_quarantine_rule |
The value comes directly from the filter.quarantine.rule field in the raw log. |
filter.quarantine.type |
security_result.detection_fields.filter_quarantine_type |
The value comes directly from the filter.quarantine.type field in the raw log. |
filter.routeDirection |
network.direction |
If filter.routeDirection is "inbound", network.direction is set to "INBOUND". If filter.routeDirection is "outbound", network.direction is set to "OUTBOUND". |
filter.routes |
read_only_udm.additional.fields |
Each route in filter.routes is added as an additional field with keys like "filterRoutes_0", "filterRoutes_1", etc. |
fromAddress |
network.email.from |
The email addresses in fromAddress are replaced into the network.email.from field if they are valid email addresses. |
guid |
metadata.product_log_id |
The value comes directly from the guid field in the raw log. |
GUID |
metadata.product_log_id |
The value comes directly from the GUID field in the raw log. |
headerFrom |
network.email.from |
The value comes directly from the headerFrom field in the raw log. |
impostorScore |
security_result.detection_fields.impostorScore |
The value comes directly from the impostorScore field in the raw log. |
malwareScore |
security_result.detection_fields.malwareScore |
The value comes directly from the malwareScore field in the raw log. |
messageID |
network.email.mail_id |
The value comes directly from the messageID field in the raw log. |
messageSize |
security_result.detection_fields.messageSize |
The value comes directly from the messageSize field in the raw log. |
messageTime |
@timestamp |
The value is parsed from the messageTime field using the ISO8601 or RFC 3339 format. |
metadata.customerId |
principal.labels.customerId |
The value comes directly from the metadata.customerId field in the raw log. |
metadata.origin.data.agent |
network.http.user_agent |
The value comes directly from the metadata.origin.data.agent field in the raw log. |
metadata.origin.data.cid |
principal.user.userid |
The value comes directly from the metadata.origin.data.cid field in the raw log. |
metadata.origin.data.version |
metadata.product_version |
The value comes directly from the metadata.origin.data.version field in the raw log. |
msg.header.from |
read_only_udm.additional.fields.msgHeaderFrom |
The value comes directly from the msg.header.from.0 field in the raw log. |
msg.header.reply-to |
network.email.reply_to |
The email address enclosed in <> in msg.header.reply-to.0 is extracted and mapped. |
msg.header.subject |
network.email.subject |
The value comes directly from the msg.header.subject field in the raw log. |
msg.header.to |
read_only_udm.additional.fields.msgHeaderTo |
The value comes directly from the msg.header.to field in the raw log. |
msg.normalizedHeader.subject |
network.email.subject |
The value comes directly from the msg.normalizedHeader.subject field in the raw log. |
msg.parsedAddresses.cc |
network.email.cc |
The email addresses in msg.parsedAddresses.cc are merged into the network.email.cc field if they are valid email addresses. |
msg.parsedAddresses.ccHashed |
read_only_udm.additional.fields |
The hashed email addresses in msg.parsedAddresses.ccHashed are added as additional fields with keys like "ccHashed_0", "ccHashed_1", etc. |
msg.parsedAddresses.from |
read_only_udm.additional.fields.msgParsedAddressesFrom |
The value comes directly from the msg.parsedAddresses.from.0 field in the raw log. |
msg.parsedAddresses.from.0 |
principal.user.email_addresses |
The value comes directly from the msg.parsedAddresses.from.0 field in the raw log. |
msg.parsedAddresses.fromHashed |
read_only_udm.additional.fields.fromHashed |
The value comes directly from the msg.parsedAddresses.fromHashed.0 field in the raw log. |
msg.parsedAddresses.to |
target.user.email_addresses |
The email addresses in msg.parsedAddresses.to are merged into the target.user.email_addresses field if they are valid email addresses. |
msgParts |
read_only_udm.about |
Multiple about objects are created, one for each entry in msgParts . File hashes, MIME type, size, and other metadata are extracted. |
QID |
security_result.detection_fields.QID |
The value comes directly from the QID field in the raw log. |
recipient |
target.user.email_addresses |
The email addresses in recipient are merged into the target.user.email_addresses field if they are valid email addresses. |
replyToAddress |
network.email.reply_to |
The email addresses in replyToAddress are replaced into the network.email.reply_to field if they are valid email addresses. |
sender |
principal.user.email_addresses |
The value comes directly from the sender field in the raw log, if it is a valid email address. |
senderIP |
principal.ip |
The value comes directly from the senderIP field in the raw log. |
sm.from |
network.email.from |
The value comes directly from the sm.from field in the raw log. |
sm.msgid |
network.email.mail_id |
The value comes directly from the sm.msgid field in the raw log, after removing "<" and ">". |
sm.proto |
network.application_protocol |
The value comes directly from the sm.proto field in the raw log. |
sm.qid |
security_result.detection_fields.QUID |
The value comes directly from the sm.qid field in the raw log. |
sm.relay |
intermediary.hostname , intermediary.ip |
The hostname and IP address are extracted from sm.relay using grok. |
sm.stat |
security_result.detection_fields.Stat |
The first word of sm.stat is extracted using grok and mapped. |
sm.to |
network.email.to |
The email addresses in sm.to are merged into the network.email.to field if they are valid email addresses. |
spamScore |
security_result.detection_fields.spamScore |
The value comes directly from the spamScore field in the raw log. |
subject |
network.email.subject |
The value comes directly from the subject field in the raw log. |
threat |
security_result.detection_fields.threat |
The value comes directly from the threat field in the raw log. |
threatsInfoMap |
security_result.detection_fields |
Key-value pairs from each entry in threatsInfoMap are added as detection fields. |
threatType |
security_result.detection_fields.threatType |
The value comes directly from the threatType field in the raw log. |
tls.cipher |
network.tls.cipher |
The value comes directly from the tls.cipher field in the raw log, unless it is "NONE". |
tls.verify |
security_result.action_details |
The value comes directly from the tls.verify field in the raw log. |
tls.version |
network.tls.version |
The value comes directly from the tls.version field in the raw log, unless the cipher is "NONE". |
toAddresses |
network.email.to |
The email addresses in toAddresses are merged into the network.email.to field if they are valid email addresses. |
ts |
@timestamp |
The value is parsed from the ts field using the ISO8601 or RFC 3339 format, after some preprocessing to handle extra fractional seconds. |
Need more help? Get answers from Community members and Google SecOps professionals.