Collect Datadog logs

Overview

This parser extracts fields from Datadog logs, performs several mutations and Grok matching to structure the data, and maps the extracted fields to the UDM. It handles different log formats within the message field, including key-value pairs and JSON objects, and converts specific fields into UDM-compliant labels and additional fields.

Before you begin

• Ensure that you have a Google SecOps instance.
• Ensure that you have privileged access to Google Cloud IAM.
• Ensure that you have privileged access to Google Cloud Storage.
• Ensure that you have logs_write_archive user access to Datadog.

Option 1: Datadog log sharing through Cloud Storage configuration

Configure Datadog integration with Google Cloud Platform

• Set up an integration for Google Cloud Platform in Datadog. For more information, see the Datadog Google Cloud integration setup.

Create a Google Cloud Storage Bucket

1. Sign in to the Google Cloud console.

2. Go to the Cloud Storage Buckets page.

   Go to Buckets

3. Click Create.

4. On the Create a bucket page, enter your bucket information. After each of the following steps, click Continue to proceed to the next step:

   1. In the Get started section, do the following:

      1. Enter a unique name that meets the bucket name requirements (for example, datadog-data).

      2. To enable hierarchical namespace, click the expander arrow to expand the Optimize for file oriented and data-intensive workloads section, and then select Enable Hierarchical namespace on this bucket.

      3. To add a bucket label, click the expander arrow to expand the Labels section.

      4. Click Add label, and specify a key and a value for your label.

   2. In the Choose where to store your data section, do the following:

      1. Select a Location type.
      2. Use the location type drop-down to select a Location where object data within your bucket will be permanently stored.
         • If you select the dual-region location type, you can also choose to enable turbo replication by using the relevant checkbox.
      3. To set up cross-bucket replication, expand the Set up cross-bucket replication section.

   3. In the Choose a storage class for your data section, either select a default storage class for the bucket, or select Autoclass for automatic storage class management of your bucket's data.

   4. In the Choose how to control access to objects section, select not to enforce public access prevention, and select an access control model for your bucket's objects.

   5. In the Choose how to protect object data section, do the following:

      1. Select any of the options under Data protection that you want to set for your bucket.
      2. To choose how your object data will be encrypted, click the expander arrow labeled Data encryption, and select a Data encryption method.

5. Click Create.

Create a Google Cloud Service Account

1. Go to IAM & Admin > Service Accounts.
2. Create a new service account.
3. Give it a descriptive name (For example, datadog-user).
4. Grant the service account with Storage Object Admin role on the Cloud Storage bucket you created in the previous step.
5. Create an SSH key for the service account.
6. Download a JSON key file for the service account. Keep this file secure.

Configure Datadog to send logs to Cloud Storage

1. Sign in to Datadog using a privileged account.
2. Go to Logs > Log Forwarding.
3. Click + Create New Archive.
4. Select Google Cloud Storage.
5. Input the required parameters and click Save.

Option 2: Datadog log sharing through Webhook configuration

Configure a feed in Google SecOps to ingest the Datadog logs

1. Go to SIEM Settings > Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed (for example, Datadog Logs).
4. Select Webhook as the Source type.
5. Select Datadog as the Log type.
6. Click Next.
7. Optional: Specify values for the following input parameters:
   • Split delimiter: the delimiter that is used to separate log lines, such as \n.
   • Asset namespace: the asset namespace.
   • Ingestion labels: the label applied to the events from this feed.
8. Click Next.
9. Review the feed configuration in the Finalize screen, and then click Submit.
10. Click Generate Secret Key to generate a secret key to authenticate this feed.
11. Copy and store the secret key. You cannot view this secret key again. If needed, you can regenerate a new secret key, but this action makes the previous secret key obsolete.
12. From the Details tab, copy the feed endpoint URL from the Endpoint Information field. You need to specify this endpoint URL in your client application.
13. Click Done.

Create an API key for the webhook feed

1. Go to Google Cloud console > Credentials.

   Go to Credentials

2. Click Create credentials, and then select API key.

3. Restrict the API key access to the Chronicle API.

Specify the endpoint URL

1. In your client application, specify the HTTPS endpoint URL provided in the webhook feed.

2. Enable authentication by specifying the API key and secret key as part of the custom header in the following format:

   X-goog-api-key = API_KEY
   X-Webhook-Access-Key = SECRET
   

   Recommendation: Specify the API key as a header instead of specifying it in the URL.

3. If your webhook client doesn't support custom headers, you can specify the API key and secret key using query parameters in the following format:

   ENDPOINT_URL?key=API_KEY&secret=SECRET
   

   Replace the following:

   • ENDPOINT_URL: the feed endpoint URL.
   • API_KEY: the API key to authenticate to Google SecOps.
   • SECRET: the secret key that you generated to authenticate the feed.

Configure Datadog to send logs to webhook

1. Sign in to Datadog using a privileged account.
2. Go to Logs > Log Forwarding.
3. Select Custom Destinations.
4. Click + Create a New Destination.
5. Specify values for the following input parameters:
   1. Choose a destination type: Select HTTP.
   2. Name the destination: Provide a descriptive name for the webhook (for example, Google SecOps Webhook).
   3. Configure the destination: Enter the ENDPOINT_URL, followed by the API_KEY and SECRET.
   4. Configure authentication settings: Add a general header like the following, this won't malform the HTTP request and allow Datadog to complete webhook creation.
      • Header name: Accept.
      • Header value: application/json.
   5. Click Save.

Reference Links

• Sharing Logs to Cloud Storage
• Forwarding Logs to Webhook

UDM Mapping Table

Changes

2023-07-21

• Parser created.