Create and manage feeds using the feed management UI

This page provides information about how to create, manage, and troubleshoot feeds using the feed management UI. Managing the feeds includes modifying, enabling, and deleting the feeds.

Before you begin

Each data feed has its own set of prerequisites that must be completed prior to setting up the feed in Google Security Operations. For information about prerequisites specific to a feed type, see Configuration by source type. Search for the data feed type you need to setup and follow the instructions provided.

Supported compression formats

Supported compression formats for feed ingestion include: .gz, .tar.gz, .tar, .targz, and solr.gz.

Add a feed

To add a feed to your Google Security Operations account, complete the following steps.

1. From the Google Security Operations menu, select Settings, and then click Feeds. The data feeds listed on this page include all the feeds that Google has configured for your account in addition to the feeds that you have configured.

2. Click Add New. The Add feed window is displayed.

3. Add a feed name.

1. In the Source type list, select the source type through which you intend to bring data into Google Security Operations. You can select from the following feed source types:

   • Amazon Data Firehose
   • Amazon S3
   • Amazon SQS
   • Google Cloud Pub/Sub
   • Google Cloud Storage
   • HTTP(S) Files (non-API)
   • Microsoft Azure Blob Storage
   • Third party API
   • Webhook

2. In the Log type list, select the log type corresponding to the logs that you want to ingest. The logs available vary depending on which source type you selected previously. Click Next.

   If you select Google Cloud Storage as the source type, use the Get service account option to get a unique service account. In this document, see Google Cloud Storage feed setup example.

3. Specify the parameters needed from the Input Parameters tab. The options presented here vary depending on the source and log type selected on the Set Properties tab. Hold the pointer over the question icon for each field to get additional information on what you need to provide.

4. (Optional) You can specify a namespace here. For more information about namespaces, see the asset namespace documentation.

5. Click Next.

6. Review your new feed configuration from the Finalize tab. Click Submit when you are ready. Google Security Operations completes a validation check of the new feed. If the feed passes the check, a name is generated for the feed, it is submitted to Google Security Operations, and Google Security Operations begins to attempt to fetch data.

   

Delete Source Files

For several feed types, including Cloud Storage, there is a field in the Add new or Edit feed workflow labeled SOURCE DELETION OPTION. This menu has three options:

1. Never delete files
2. Delete transferred files and empty directories
3. Delete transferred files

Options 2 and 3 involve deletions: one for files and one for files and any empty directories. If you select either of those options, you need to add the permissions specific to your feed type. For information about permissions specific to a feed type, see Configuration by source type.

This option lets you delete an object out of the storage system after you have transferred it. Feeds always remember which objects (or files) they have transferred and never transfer the same file twice (unless it has been updated), but you have to set this option if you want the system to delete the source object after it has been (successfully) transferred.

Microsoft Azure Blob Storage doesn't support deletion of source files. The following source deletion options mustn't be used with Microsoft Azure Blob Storage source type:

• Delete transferred files and empty directories
• Delete transferred files

When you create a feed with Microsoft Azure Blob Storage source, select only the Never delete files option.

Set up a Pub/Sub push feed

To set up a Pub/Sub push feed, do the following:

1. Create a Pub/Sub push feed.
2. Specify the endpoint URL in a Pub/Sub subscription.

Create a Pub/Sub push feed

1. From the Google Security Operations menu, select Settings, and then click Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed.
4. In the Source type list, select Google Cloud Pub/Sub Push.
5. Select the Log type. For example, to create a feed for Open Cybersecurity Schema Framework, select Open Cybersecurity Schema Framework (OCSF) as the Log type.
6. Click Next.
7. Optional: Specify values for the following input parameters:
   • Split delimiter: the delimiter that is used to separate log lines, such as \n.
   • Asset namespace: the asset namespace.
   • Ingestion labels: the label to be applied to the events from this feed.
8. Click Next.
9. Review your new feed configuration in the Finalize screen, and then click Submit.
10. From the Details tab, copy the feed endpoint URL from the Endpoint Information field. You need this endpoint URL to create a push subscription in Pub/Sub.
11. To disable the feed, click the Feed Enabled toggle. The feed is enabled by default.
12. Click Done.

Specify the endpoint URL

After you create a Pub/Sub push feed, in Pub/Sub, create a push subscription, specify the HTTPS endpoint, and enable authentication.

1. Create a push subscription in Pub/Sub. For more information about how to create a push subscription, see Create push subscriptions.
2. Specify the endpoint URL, which is available in the Google Cloud Pub/Sub push feed.
3. Select Enable authentication, and select a service account.

Set up an Amazon Data Firehose feed

To set up an Amazon Data Firehose feed, do the following:

1. Create an Amazon Data Firehose feed and copy the endpoint URL and secret key.
2. Create an API key to authenticate to Google Security Operations. You can also reuse your existing API key to authenticate to Google Security Operations.
3. Specify the endpoint URL in Amazon Data Firehose.

Create an Amazon Data Firehose feed

1. From the Google Security Operations menu, select Settings, and then click Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed.
4. In the Source type list, select Amazon Data Firehose.
5. Select the Log type. For example, to create a feed for Open Cybersecurity Schema Framework, select Open Cybersecurity Schema Framework (OCSF) as the Log type.
6. Click Next.
7. Optional: Specify values for the following input parameters:
   • Split delimiter: the delimiter that is used to separate log lines, such as \n.
   • Asset namespace: the asset namespace.
   • Ingestion labels: the label to be applied to the events from this feed.
8. Click Next.
9. Review your new feed configuration in the Finalize screen, and then click Submit.
10. Click Generate Secret Key to generate a secret key to authenticate this feed.
11. Copy and store the secret key as you cannot view this secret again. You can generate a new secret key again, but regeneration of the secret key makes the previous secret key obsolete.
12. From the Details tab, copy the feed endpoint URL from the Endpoint Information field. You need this endpoint URL when you specify the destination settings for your delivery stream in Amazon Data Firehose.
13. To disable the feed, click the Feed Enabled toggle. The feed is enabled by default.
14. Click Done.

Create an API key for the Amazon Data Firehose feed

1. Go to the Google Cloud console Credentials page.
2. Click Create credentials, and then select API key.
3. Restrict the API key access to the Chronicle API.

Specify the endpoint URL

In Amazon Data Firehose, specify the HTTPS endpoint and access key.

1. Append the API key to the feed endpoint URL and specify this URL as the HTTP endpoint URL in the following format:

     ENDPOINT_URL?key=API_KEY
   

   Replace the following:

   • ENDPOINT_URL: the feed endpoint URL.
   • API_KEY: the API key to authenticate to Google Security Operations.

2. For the access key, specify the secret key that you obtained when you created the Amazon Data Firehose feed.

Set up an HTTPS webhook feed

To set up an HTTPS webhook feed, do the following:

1. Create an HTTPS webhook feed and copy the endpoint URL and secret key.
2. Create an API key that is specified with the endpoint URL. You can also reuse your existing API key to authenticate to Google Security Operations.
3. Specify the endpoint URL in your application.

Prerequisites

• Ensure that a Google Cloud project for Google Security Operations is configured and the Chronicle API is enabled for the project.

• Link a Google Security Operations instance to Google Cloud services.

Create an HTTPS webhook feed

1. From the Google Security Operations menu, select Settings, and then click Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed.
4. In the Source type list, select Webhook.
5. Select the Log type. For example, to create a feed for Open Cybersecurity Schema Framework, select Open Cybersecurity Schema Framework (OCSF) as the Log type.
6. Click Next.
7. Optional: Specify values for the following input parameters:
   • Split delimiter: the delimiter that is used to separate log lines, such as \n.
   • Asset namespace: the asset namespace.
   • Ingestion labels: the label to be applied to the events from this feed.
8. Click Next.
9. Review your new feed configuration in the Finalize screen, and then click Submit.
10. Click Generate Secret Key to generate a secret key to authenticate this feed.
11. Copy and store the secret key as you cannot view this secret again. You can generate a new secret key again, but regeneration of the secret key makes the previous secret key obsolete.
12. From the Details tab, copy the feed endpoint URL from the Endpoint Information field. You need to specify this endpoint URL in your client application.
13. To disable the feed, click the Feed Enabled toggle. The feed is enabled by default.
14. Click Done.

Create an API key for the webhook feed

1. Go to the Google Cloud console Credentials page.
2. Click Create credentials, and then select API key.
3. Restrict the API key access to the Chronicle API.

Specify the endpoint URL

1. In your client application, specify the HTTPS endpoint, which is available in the webhook feed.

2. Enable authentication by specifying the API key and secret key as part of the custom header in the following format:

   X-goog-api-key = API_KEY

   X-Webhook-Access-Key = SECRET

   We recommend that you specify the API key as a header instead of specifying it in the URL. If your webhook client doesn't support custom headers, you can specify the API key and secret key by using query parameters in the following format:

     ENDPOINT_URL?key=API_KEY&secret=SECRET
   

   Replace the following:

   • ENDPOINT_URL: the feed endpoint URL.
   • API_KEY: the API key to authenticate to Google Security Operations.
   • SECRET: the secret key that you generated to authenticate the feed.

Google Cloud Storage feed setup example

1. From the Google Security Operations menu, select Settings, and then click Feeds.
2. Click Add New.
3. Select Google Cloud Storage for Source Type.
4. Select the Log type. For example, to create a feed for Google Kubernetes Engine audit logs, select Google Kubernetes Engine audit logs as the Log Type.
5. Click Get service account. Google Security Operations provides a unique service account that Google Security Operations uses to ingest data.
6. Configure access for the service account to access the Cloud Storage objects. In this document, see Grant access to the Google Security Operations service account.
7. Click Next.
8. Based on the Cloud Storage configuration that you created, specify values for the following fields:
   • Storage bucket URI
   • URI is a
   • Source deletion option
9. Click Next and then click Submit.

Grant access to the Google Security Operations service account

1. In the Google Cloud console, go to the Cloud Storage Buckets page.
   

   Go to Buckets

2. Grant access to the service account to the relevant Cloud Storage objects.

   • To grant read permission to a specific file, complete the following steps:

     1. Select the file and click Edit access.
     2. Click Add principal.
     3. In the New principals field, enter the name of the Google Security Operations service account.
     4. Assign a role that contains the read permission to the Google Security Operations service account. For example, Storage Object Viewer (roles/storage.objectViewer). This can only be done if you have not enabled uniform bucket-level access.
     5. Click Save.

   • To grant read permission to multiple files, you must grant access at the bucket level. You must add the Google Security Operations service account as a principal to your storage bucket and grant it the IAM Storage Object Viewer (roles/storage.objectViewer) role.

     If you configure the feed to delete source files, you must add the Google Security Operations service account as a principal on your bucket and grant it the IAM Storage Object Admin (roles/storage.objectAdmin) role.

Configure VPC Service Controls

If VPC Service Controls is enabled, an ingress rule is required to provide access to the Cloud Storage bucket.

The following Cloud Storage methods must be allowed in the ingress rule:

• google.storage.objects.list. Required for a single file feed.
• google.storage.objects.get. Required for feeds that require directory or subdirectory access.
• google.storage.objects.delete. Required for feeds that require deletion of the source file.

Sample ingress rule

- ingressFrom:
  identities:
    - serviceAccount:8911409095528497-0-account@partnercontent.gserviceaccount.com
  sources:
  - accessLevel: "*"
  ingressTo:
  operations:
  - serviceName: storage.googleapis.com
    methodSelectors:
    - method: google.storage.objects.list
    - method: google.storage.objects.get
    - method: google.storage.objects.delete
  resources:
  - projects/PROJECT_ID

Feed status

You can monitor the status of the feed from the initial Feeds page. Feeds can have the following statuses:

• Active—Feed is configured and ready to ingest data into your Google Security Operations account.
• InProgress—Google Security Operations is now attempting to pull data from the configured third party.
• Completed—Data successfully retrieved by this feed.
• Archived—Disabled feed.

• Failed—Feed is failing to successfully fetch data. This is likely due to a configuration issue. Click the question to display the configuration error. Once you have corrected the error and resubmitted the feed, return to the Feeds page to determine whether or not the feed is now working.

Edit feeds

From the Feeds page, you can edit an existing feed:

1. Hold the pointer over an existing feed and click more_vert in the right column.

2. Click Edit Feed. You can now alter the input parameters for the feed and resubmit it to Google Security Operations. Google Security Operations will attempt to use the edited feed.

Enable and disable feeds

In the Status column, enabled feeds are labeled as Active, InProgress, Completed, or Failed. Disabled fields are labeled as Archived. For a description, see the feed status.

From the Feeds page, you can enable or disable any of the existing feeds:

1. Hold the pointer over an existing feed and click more_vert in the right column.

2. To enable a feed, click the Enable Feed toggle.

3. To disable a feed, click the Disable Feed toggle. The feed is now labeled as Archived.

Delete feeds

From the Feeds page, you can also delete an existing feed:

1. Hold the pointer over an existing feed and click more_vert in the right column.

2. Click Delete Feed. The DELETE FEED window opens. To permanently delete the feed, click Yes, delete it.

Control the rate of ingestion

When the data ingestion rate for a tenant reaches a certain threshold, Google Security Operations restricts the rate of ingestion for new data feeds to prevent a source with a high ingestion rate from affecting the ingestion rate of another data source. In this case, there is a delay but no data is lost. The ingestion volume and tenant's usage history determine the threshold.

You can request a rate limit increase by contacting Cloud Customer Care.

Troubleshooting

From the Feeds page, you can view details such as source type, log type, feed ID, and status of the existing feeds:

1. Hold the pointer over an existing feed and click more_vert in the right column.

2. Click View Feed. A dialog appears showing the feed details. For a failed feed, you can find error details under Details > Status.

For a failed feed, the details include the cause of the error and steps to fix it. The following table describes the error messages that you might encounter when working with data feeds.