Create and manage feeds using the feed management UI

This document provides information about how to create, manage, and troubleshoot feeds using the feed management UI. Managing the feeds includes modifying, enabling, and deleting the feeds.

Before you begin

Each data feed has its own set of prerequisites that must be completed prior to setting up the feed in Google Security Operations. For information about prerequisites specific to a feed type, see Configuration by source type. Search for the data feed type you need to set up and follow the instructions provided.

Supported compression formats

Supported compression formats for feed ingestion include: .gz, .tar.gz, .tar, .targz, and solr.gz. The following table outlines the different file sizes that Central Data Acquisition (CDA) feeds transformation supports:

Add a feed

To add a feed to your Google SecOps account, complete the following steps:

1. In the Google SecOps menu, select Settings, and then click Feeds. The data feeds listed on the Feeds page include all the feeds that Google has configured for your account in addition to the feeds that you have configured.

2. Click Add New. The Add feed window displays.

3. Add a feed name.

4. In the Source type list, select the source type for importing data into Google SecOps. You can select from the following feed source types:

   • Amazon Data Firehose
   • Amazon S3
   • Amazon SQS
   • Google Cloud Pub/Sub
   • Cloud Storage
   • HTTP(S) Files (non-API)
   • Microsoft Azure Blob Storage
   • Third party API
   • Webhook

5. In the Log type list, select the log type that corresponds to the logs you want to ingest. The available logs vary depending on the source type you selected earlier.

   If you select Cloud Storage as the source type, use the Get service account option to get a unique service account. See Google Cloud Storage feed setup example.

6. Click Next.

7. Specify the parameters needed from the Input Parameters tab. The options presented here vary depending on the source and log type selected on the Set Properties tab. Hold the pointer over the question icon for each field to get additional information on what you need to provide.

8. Optional: You can specify a namespace in the Set Properties tab. For more information about namespaces, see Work with asset namespaces.

9. Click Next.

10. Review your new feed configuration on the Finalize tab.

11. Click Submit. Google SecOps completes a validation check of the new feed. If the feed passes the check, a name is generated for the feed, it's submitted to Google SecOps, and Google SecOps begins to attempt to fetch data.

    

Google Cloud Storage feed setup example

1. From the Google SecOps menu, select Settings, and then click Feeds.
2. Click Add New.
3. Select Cloud Storage for Source Type.
4. Select the Log type. For example, to create a feed for Google Kubernetes Engine audit logs, select Google Kubernetes Engine audit logs as the Log Type.
5. Click Get service account. Google SecOps provides a unique service account that Google SecOps uses to ingest data.
6. Optional: Configure the service account.For more information, see Grant access to the Google SecOps service account.
7. Click Next.

8. Based on the Cloud Storage configuration that you created, specify values for the following fields:

   • Storage bucket URI
   • URI is a
   • Source deletion option

   To learn more about how to set up Cloud Storage buckets, see Create Buckets.

9. Click Next and then click Submit.

Delete source files

The source deletion option lets you delete feed source objects (files and folders) from storage, after a successful transfer. This option is only available for selected feed source types, including Cloud Storage. These feed source types include the SOURCE DELETION OPTION field in their Add new and Edit feed workflows.

Source deletion options

• For supported feed source types, including Cloud Storage, the SOURCE DELETION OPTION field offers these options:

  • Never delete files
  • Delete transferred files and empty directories
  • Delete transferred files

• Microsoft Azure Blob Storage (AZURE_BLOBSTORE) doesn't support deletion of source files. For the SOURCE DELETION OPTION field, only select the Never delete files option.

• For the following feed sources ("feedSourceType"): GOOGLE_CLOUD_STORAGE_V2, AMAZON_S3_V2, AMAZON_SQS_V2, and AZURE_BLOBSTORE_V2, the SOURCE DELETION OPTION field offers two options:

  • NEVER: Never deletes any files after transfers.
  • ON_SUCCESS: Deletes all files and empty directories after transfer.

Set up a Pub/Sub push feed

To set up a Pub/Sub push feed, do the following:

1. Create a Pub/Sub push feed.
2. Specify the endpoint URL in a Pub/Sub subscription.

Create a Pub/Sub push feed

1. In the Google SecOps menu, select Settings, and then click Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed.
4. In the Source type list, select Google Cloud Pub/Sub Push.
5. Select the Log type. For example, to create a feed for Open Cybersecurity Schema Framework, select Open Cybersecurity Schema Framework (OCSF) as the Log type.
6. Click Next.
7. Optional: Specify values for the following input parameters:
   • Split delimiter: the delimiter that is used to separate log lines, such as \n.
   • Asset namespace: the asset namespace.
   • Ingestion labels: the label to be applied to the events from this feed.
8. Click Next.
9. Review your new feed configuration in the Finalize screen, and then click Submit.
10. From the Details tab, copy the feed endpoint URL from the Endpoint Information field. You need this endpoint URL to create a push subscription in Pub/Sub.
11. Optional: Click the Feed Enabled toggle to disable the feed. The feed is enabled by default.
12. Click Done.

Specify the endpoint URL

After you create a Pub/Sub push feed, specify the endpoint URL as follows: In Pub/Sub, create a push subscription, specify the HTTPS endpoint. Select Enable authentication and a service account.

1. Create a push subscription in Pub/Sub. For more information about how to create a push subscription, see Create push subscriptions.
2. Specify the endpoint URL, which is available in the Google Cloud Pub/Sub push feed.
3. Select Enable authentication, and select a service account.

Set up an Amazon Data Firehose feed

To set up an Amazon Data Firehose feed, do the following:

1. Create an Amazon Data Firehose feed and copy the endpoint URL and secret key.
2. Create an API key to authenticate to Google SecOps. You can also reuse your existing API key to authenticate to Google SecOps.
3. Specify the endpoint URL in Amazon Data Firehose.

Create an Amazon Data Firehose feed

1. In the Google SecOps menu, select Settings, and then click Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed.
4. In the Source type list, select Amazon Data Firehose.
5. Select the Log type. For example, to create a feed for Open Cybersecurity Schema Framework, select Open Cybersecurity Schema Framework (OCSF) as the Log type.
6. Click Next.
7. Optional: Specify values for the following input parameters:
   • Split delimiter: the delimiter that is used to separate log lines, such as \n.
   • Asset namespace: the asset namespace.
   • Ingestion labels: the label to be applied to the events from this feed.
8. Click Next.
9. Review your new feed configuration in the Finalize screen, and then click Submit.
10. Click Generate Secret Key to generate a secret key to authenticate this feed.
11. Copy and store the secret key as you cannot view this secret again. You can generate a new secret key again, but regeneration of the secret key makes the previous secret key obsolete.
12. On the Details tab, copy the feed endpoint URL from the Endpoint Information field. You need this endpoint URL when you specify the destination settings for your delivery stream in Amazon Data Firehose.
13. Optional: Click the Feed Enabled toggle to disable the feed. The feed is enabled by default.
14. Click Done.

Create an API key for the Amazon Data Firehose feed

To create an API key for the Amazon Data Firehose feed, do the following: 1. Go to the Google Cloud console Credentials page. 1. Click Create credentials, and then select API key. 1. Restrict the API key access to the Chronicle API.

Specify the endpoint URL

In Amazon Data Firehose, specify the HTTPS endpoint and access key, as follows:

1. Append the API key to the feed endpoint URL and specify this URL as the HTTP endpoint URL in the following format:

     ENDPOINT_URL?key=API_KEY
   

   Replace the following:

   • ENDPOINT_URL: the feed endpoint URL.
   • API_KEY: the API key to authenticate to Google SecOps.

2. For the access key, specify the secret key that you obtained when you created the Amazon Data Firehose feed.

Set up an HTTPS webhook feed

To set up an HTTPS webhook feed, do the following:

1. Create an HTTPS webhook feed and copy the endpoint URL and secret key.
2. Create an API key that is specified with the endpoint URL. You can also reuse your existing API key to authenticate to Google SecOps.
3. Specify the endpoint URL in your application.

Prerequisites

• Ensure that a Google Cloud project for Google SecOps is configured and the Chronicle API is enabled for the project.

• Link a Google SecOps instance to Google Cloud services.

Create an HTTPS webhook feed

1. In the Google SecOps menu, select Settings, and then click Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed.
4. In the Source type list, select Webhook.
5. Select the Log type. For example, to create a feed for Open Cybersecurity Schema Framework, select Open Cybersecurity Schema Framework (OCSF) as the Log type.
6. Click Next.
7. Optional: Specify values for the following input parameters:
   • Split delimiter: the delimiter that is used to separate log lines, such as \n.
   • Asset namespace: the asset namespace.
   • Ingestion labels: the label to be applied to the events from this feed.
8. Click Next.
9. Review your new feed configuration in the Finalize screen, and then click Submit.
10. Click Generate Secret Key to generate a secret key to authenticate this feed.
11. Copy and store the secret key as you cannot view this secret again. You can generate a new secret key again, but regeneration of the secret key makes the previous secret key obsolete.
12. From the Details tab, copy the feed endpoint URL from the Endpoint Information field. You need to specify this endpoint URL in your client application.
13. Optional: Click the Feed Enabled toggle to disable the feed. The feed is enabled by default.
14. Click Done.

Create an API key for the webhook feed

1. Go to the Google Cloud console Credentials page.
2. Click Create credentials, and then select API key.
3. Restrict the API key access to the Chronicle API.

Specify the endpoint URL

1. In your client application, specify the HTTPS endpoint, which is available in the webhook feed.

2. Enable authentication by specifying the API key and secret key as part of the custom header in the following format:

   X-goog-api-key = API_KEY

   X-Webhook-Access-Key = SECRET

   We recommend that you specify the API key as a header instead of specifying it in the URL. If your webhook client doesn't support custom headers, you can specify the API key and secret key by using query parameters in the following format:

     ENDPOINT_URL?key=API_KEY&secret=SECRET
   

   Replace the following:

   • ENDPOINT_URL: the feed endpoint URL.
   • API_KEY: the API key to authenticate to Google SecOps.
   • SECRET: the secret key that you generated to authenticate the feed.

Grant access to the Google SecOps service account

1. In the Google Cloud console, go to the Cloud Storage Buckets page.
   

   Go to Buckets

2. Grant access to the service account to the relevant Cloud Storage objects.

   • To grant read permission to a specific file, complete the following steps:

     1. Select the file and click Edit access.
     2. Click Add principal.
     3. In the New principals field, enter the name of the Google SecOps service account.
     4. Assign a role that contains the read permission to the Google SecOps service account. For example, Storage Object Viewer (roles/storage.objectViewer). This can only be done if you've not enabled uniform bucket-level access.
     5. Click Save.

   • To grant read permission to multiple files, grant access at the bucket level as follows:

     • For "feedSourceType": "GOOGLE_CLOUD_STORAGE":
       1. Add the Google SecOps service account as a principal to your storage bucket and grant it the IAM Storage Object Viewer (roles/storage.objectViewer) role.
       2. If you configure the feed to delete source files, you must add the Google SecOps service account as a principal on your bucket and grant it the IAM Storage Object Admin (roles/storage.objectAdmin) role.
     • For "feedSourceType": "GOOGLE_CLOUD_STORAGE_V2":
       1. Add the Google SecOps service account as a principal to your storage bucket and grant it the IAM Storage Legacy Bucket Viewer (roles/storage.legacyBucketViewer) role.
       2. If you configure the feed to delete source files, you must add the Google SecOps service account as a principal on your bucket and grant it the IAM Storage Legacy Bucket Writer (roles/storage.legacyBucketWriter) role.

Configure VPC Service Controls

If VPC Service Controls is enabled, an ingress rule is required to provide access to the Cloud Storage bucket.

The following Cloud Storage methods must be allowed in the ingress rule:

• google.storage.objects.list. Required for a single file feed.
• google.storage.objects.get. Required for feeds that require directory or subdirectory access.
• google.storage.objects.delete. Required for feeds that require deletion of the source file.

Sample ingress rule

- ingressFrom:
  identities:
    - serviceAccount:8911409095528497-0-account@partnercontent.gserviceaccount.com
  sources:
  - accessLevel: "*"
  ingressTo:
  operations:
  - serviceName: storage.googleapis.com
    methodSelectors:
    - method: google.storage.objects.list
    - method: google.storage.objects.get
    - method: google.storage.objects.delete
  resources:
  - projects/PROJECT_ID

Feed status

You can monitor the status of the feed on the initial Feeds page, where feeds can have the following statuses:

• Active: Feed is configured and ready to ingest data into your Google SecOps account.
• InProgress: Google SecOps attempts to pull data from the configured third party.
• Completed: Data successfully retrieved by this feed.
• Archived: Disabled feed.

• Failed: Feed is failing to successfully fetch data. This is likely due to a configuration issue. Click the question to display the configuration error. Once you've corrected the error and resubmitted the feed, return to the Feeds page to determine whether or not the feed is now working.

Edit feeds

On the Feeds page, you can edit an existing feed, as follows:

1. Hold the pointer over an existing feed and click more_vert in the right column.

2. Click Edit Feed. You can now modify the input parameters for the feed and resubmit it to Google SecOps, which will attempt to use the updated feed.

Enable and disable feeds

In the Status column, enabled feeds are labeled as Active, InProgress, Completed, or Failed. Disabled fields are labeled as Archived. For a description, see the feed status.

From the Feeds page, you can enable or disable any of the existing feeds:

1. Hold the pointer over an existing feed and click more_vert in the right column.

2. Optional: Click the Feed Enabled toggle to disable the feed.

3. Optional: Click the Disable Feed toggle to disable the feed. The feed is now labeled as Archived.

Delete feeds

On the Feeds page, you can also delete an existing feed:

1. Hold the pointer over an existing feed and click more_vert in the right column.

2. Click Delete Feed. The DELETE FEED window opens. To permanently delete the feed, click Yes, delete it.

Control the rate of ingestion

When the data ingestion rate for a tenant reaches a certain threshold, Google Security Operations restricts the rate of ingestion for new data feeds to prevent a source with a high ingestion rate from affecting the ingestion rate of another data source. In this case, there is a delay but no data is lost. The ingestion volume and tenant's usage history determine the threshold.

You can request a rate limit increase by contacting Cloud Customer Care.

Troubleshooting

On the Feeds page, you can view details such as source type, log type, feed ID, and status of the existing feeds, as follows:

1. Hold the pointer over an existing feed and click more_vert in the right column.

2. Click View Feed. A dialog appears showing the feed details. For a failed feed, you can find error details under Details > Status.

For a failed feed, the details include the cause of the error and steps to fix it. The following table describes the error messages that you might encounter when working with data feeds:

Need more help? Get answers from Community members and Google SecOps professionals.