Set up and manage data processing pipelines

Supported in:

The Data Processing Pipeline feature provides robust control over Google Security Operations data ingestion. Data processing pipelines let you manipulate incoming data before it's parsed by Google Security Operations. For example, filter and transform events, or redact sensitive values. This process can help optimize data for Google SecOps, reduce costs, protect sensitive information, and improve compatibility.

This document shows how to use the Bindplane console to configure a connection to a Google SecOps destination instance, create a new stream, set up the Data Processing Pipeline (sources and processors), roll it out to initiate data processing, and view pipeline sources and processors in the Google SecOps console. Example use cases include:

  • Remove empty key-value pairs from raw logs.
  • Redact sensitive data.
  • Add ingestion labels from raw log content.
  • In multi-instance environments, apply ingestion labels to direct-ingestion log data, to indicate which source instance the data came from, (for example, Google Cloud, Workspace).
  • Filter Palo Alto Cortex data by field values.
  • Reduce SentinelOne data by category.
  • Parse hosts from feeds and direct-ingestion logs into the ingestion_source field for Cloud Monitoring.

You can configure data processing pipelines for both on-premises and cloud data sources, using either the Bindplane management console or directly using the public Google SecOps Data Pipeline APIs.

A data processing pipeline consists of the following elements:

  • Sources: One or more data sources feed data into the data processing pipeline, each configured for different data source types.
  • Processor node: A data processing pipeline has one Processor node that contains one or more processors. Each processor specifies an action to perform on the data (for example, filter, transform, and redact) as it flows through the pipeline.
  • Destination: The Google SecOps destination instance is where the processed data is sent.

Prerequisites

If you intend to use the Bindplane console to manage your Google SecOps data processing pipeline, perform the following steps:

  1. In the Google Security Operations console, grant the installer the required predefined administrator roles. For details, see Assign the Project IAM Admin role in a dedicated project. Under Assign Roles, select the following predefined Identity and Access Management roles:
    • Chronicle API Admin (roles/chronicle.admin)
    • Chronicle Service Admin (roles/chroniclesm.admin)
    • Chronicle SOAR Admin Beta (roles/chronicle.soarAdmin)
    • Project IAM Admin (roles/resourcemanager.projectIamAdmin)
  2. Install the Bindplane Server console. For SaaS or on-premises, see Install the Bindplane Server console.
  3. In the Bindplane console, connect a Google SecOps destination instance to your Bindplane organization. For details, see Connect to a Google SecOps instance.

Potential increased acknowledgment time for low volume streams

Ingestion API users who configure their own agent, may experience a potential increase in acknowledgment time for low volume streams in the Data processing pipeline. The average expected acknowledgment time may increase from 700ms up to 2 seconds. In such a case, you may need to increase timeout periods and memory accordingly. Acknowledgment time should drop as data throughput increases to more than 4MBps.

Connect to a Google SecOps instance

Connect to a Google SecOps instance, which will serve as the destination for the output from your data processing pipelines.

To connect to a Google SecOps instance using the Bindplane console:

  1. In the Bindplane console, go to the Manage your organization page.
  2. Go to the Integrations card and click Connect to Google SecOps.

  3. In the Edit Integration window that opens, enter the details of the Google SecOps destination instance, that will ingest the output from your data processing pipelines, as follows:

    Field Description
    Region The region of your Google SecOps instance. To find the instance, go to the Google Cloud console, navigate to the Google Security Operationspage, and click Instance details.
    Customer ID The customer ID of your Google SecOps instance. In the Google SecOps console, go to Settings > Profile > Organization Details.
    Google Cloud project number The Google Cloud Project Number of your Google SecOps instance.
    To find the project number in the Google SecOps console, go to Settings > Profile > Organization Details.
    Credentials Credentials for the service account to access the Google SecOps Data Pipeline APIs.
    This is a JSON value available in the Google Service Account credential file. The service account must be in the same project as your Google SecOps instance. For information about how to create a service account and download the JSON file, see Create and delete service account keys.

  4. Click Connect. If your connection details are correct and you successfully connect to Google SecOps, you can expect the following:

    • Google SecOps instance details (encrypted) are saved in your Organization object.
    • A connection to the Google SecOps instance is opened.
    • When you connect for the first time, you can see the Streams tab on the Bindplane console.
    • The Bindplane console now displays any data processing pipelines you previously set up for this instance using the API. The system converts some processors you configured using the API into Bindplane processors, and displays others in their raw OpenTelemetry Transformation Language (OTTL) format. You can use the Bindplane console to edit pipelines and processors previously set up using the API.

  5. After you successfully create a connection to a Google SecOps instance, you can create a stream and set up the data processing pipeline. For details, see Set up a data processing pipeline using the Bindplane console.

Set up a data processing pipeline using the Bindplane console

Using the Bindplane console, you can manage your Google SecOps data processing pipelines, including pipelines set up using the API.

Follow these steps to create a new stream and set up the data processing pipeline, configure data processing pipeline sources and processors, and roll out a data processing pipeline to initiate data processing:

  1. Create a new stream
  2. Configure a data processing pipeline
    1. Configure sources
    2. Configure processors
  3. Roll out a data processing pipeline

Create a new stream

A stream is a container for you to configure one data processing pipeline.
To create a new stream, do the following:

  1. In the Bindplane console, click the Streams tab to open the Streams page.
  2. Click Create stream.
  3. In the Create new stream window, set the Stream type to Google SecOps (default).
  4. Enter a Stream name and Description.
  5. Click Create.

Configure a data processing pipeline

A data processing pipeline specifies data Sources to ingest and Processors (for example, filter, transform, or redact) to manipulate the data as it flows to the Google SecOps Destination instance.

A Pipeline configuration card is a visualization of the data processing pipeline where you can configure the data Sources and the Processor node. The Processor node consists of processors that manipulate the data as it flows to the Google SecOps Destination instance.

To configure a data processing pipeline, first Create a new stream, and then do the following:

  1. In the Bindplane console, click the Streams tab to open the Streams page.
  2. Select the stream where you want to configure the new data processing pipeline. The Pipeline configuration card opens.

  3. Configure the following:

    1. A Source. See Configure sources for details.

    2. The Processor node:

      • To add a processor using the Bindplane console, see Configure processors for details.
      • Some custom processors let you edit their raw OTTL code directly.

  4. Once these configurations are complete, see Roll out a data processing pipeline to begin processing the data.

Configure sources

A Source ingests data according to its configured specifications, and feeds it into the pipeline. A data processing pipeline can have one or more Sources, each configured for a different data source.

To add a Source, do the following:

  1. In the Pipeline configuration card, click add Add Source to open the Create SecOps Data Source window.

  2. In the Create SecOps Data Source window, enter details for these field:

    Field Description
    Log type Log type of the data to ingest.
    Select the log type to ingest. For example, "CrowdStrike Falcon (CS_EDR)".

    Note: You can't select a log type with a warning warning icon.
    A warning icon indicates that the log type is already configured in another source (in this pipeline or another pipeline in your Google SecOps instance).
    If you want to use such a log type, you must first delete it from the other source configuration.
    To find the other source configuration where the log type is configured, see Filter stream (pipeline) configurations.
    Ingestion method Ingestion method to use to ingest the data for the selected Log type.
    These ingestion methods where previously defined for your Google SecOps instance.
    Select one of the following:
    • All Ingestion Methods

      Note, selecting this narrows your options when adding your next Sources:
      Selecting All Ingestion Methods will prevent you from adding other Sources for specific Ingestion Methods, for this Log type.
    • Select a specific ingestion method.
      For example, one of the following: "Bindplane Agent", "Cloud Native Ingestion", "Feed", "Ingestion API", or "Workspace".
      • Note, selecting this narrows your options when you want to add your next Sources:
        Selecting a specific ingestion method will prevent you from adding another Source using "All Ingestion Methods", for this Log type.
        You will still be able to select other unconfigured specific Ingestion methods for this Log type.
      • If you selected Feed, this displays a list of Feeds in the next field, for you to select from, as the ingestion source. (See the next field.)
    Feed The feed to use to ingest source data.
    If you select Feed in the Ingestion method field, the Feed field displays a list of feed names (previously defined for your Google SecOps instance) for the selected Log type.
    Select a specific Feed from the list.

    Note: To see a list of your feeds in your Google SecOps console, go to Settings > Feeds table.

  3. Click Add Source to save the new data source.

    • The new data Source is now displayed on the data processing pipeline in the Pipeline configuration card.
    • It is automatically connected to the Processor node and the Google SecOps Destination.
Filter Stream (Pipeline) Configurations

The search bar on the Streams page lets you filter your Streams (data processing pipelines) based on multiple configuration elements, for example, log type, ingestion method, and feed name. You can use the following syntax to filter: logtype:value, ingestionmethod:value and feed:value.

For example, to use the search bar to identify Source configurations containing a specific log type, enter logtype: in the search bar, and select the log type from the list.

Configure processors

A data processing pipeline has one Processor node, containing one or more processors. Each processor manipulates the source data as it flows through the pipeline, in the sequence that the processors appear in the Processors pane. The first processor processes the source data, then the resulting output is processed by the next processor, and then by subsequent processors.

Configure the Processor node by adding, removing, or changing the sequence of one or more processors.

To add a processor, follow these steps:

  1. In the Pipeline configuration card, click the Processor node to open the Edit Processors window.
    The Edit Processors window consists of three panes:

    • Left pane: Recent incoming source log data (before processing)
    • Middle pane: Processors and their configurations
    • Right pane: Recent outgoing result log data (after processing)

    If the pipeline has been rolled out before, then the system shows the recent incoming log data (before processing) and the recent outgoing log data (after processing) in the panes.

  2. To add a processor, click Add Processor to display the processor list. For your convenience, the processor list is grouped by processor type.
    (To organize the processor list, you can add your own bundles by selecting one or more processors and clicking Add new Processor bundles.)

  3. Select a Processor to add from the list.

  4. Configure the processor as necessary.

  5. Click Save to save the processor configuration in the Processor node.

The system tests the new processor configuration by processing a fresh sample of the incoming source log data (from the left pane) and displays the outgoing result data (in the right pane).

Roll out a data processing pipeline

Once the source and processor configurations are complete, roll out the pipeline to begin processing data.

To roll out a data processing pipeline, click Start rollout. This activates the data processing pipeline and allows Google's secure infrastructure to begin processing data according to the data processing pipeline configuration.

If the rollout is successful, the data processing pipeline configuration version number is incremented and displayed next to the data processing pipeline name.

To view the configuration history, click the history link next to the data processing pipeline name. The configuration changes between each data processing pipeline version are displayed.

What's next?

You can view active data streams in a view-only mode from within Google SecOps. For details, see View data processing pipeline information from the Google SecOps console.

View data processing pipeline information from the Google SecOps console

The following sections describe how to view data processing pipeline information from the Google SecOps console:

View configured feeds

The Feeds page shows all the feeds that you configured.

  1. In the Google SecOps console, go to Settings > Feeds. The main page displays all your configured feeds.
  2. Hold the pointer over each row to display the ⋮ More menu. From the menu, you can view feed details, edit, disable, or delete the feed.
  3. Click View Details to view the details window.
  4. Click Open in Bindplane to open the source configuration for that feed in the Bindplane console.

View data processing pipeline information from the Logtypes page

The Logtypes page shows all available log types. To view data processing pipeline details:

  1. In the Google SecOps console, go to Settings > Logtypes. The main page displays all your log types.
  2. Hold the pointer over each row to display the ⋮ More menu. From the menu, you can view logtype details.
  3. Click View Data Processing to view the details window.
  4. Click Open in Bindplane to open the processor configuration for that processor in the Bindplane console.

Use Google SecOps Data Pipeline APIs

The Google SecOps Data Pipeline APIs allow you to manage your data processing pipelines. The APIs cover all the Data Pipeline functionality, such as creating, updating, deleting, and listing pipelines and associated feeds and log types within them.

Need more help? Get answers from Community members and Google SecOps professionals.