Forwarder Management API

You can use the Google Security Operations Forwarder Management API to do the following programmatically:

• Create and manage forwarders.
• Create and manage collectors.
• Get the file contents for a Google Security Operations forwarder's configuration.(.conf) and authentication (_auth.conf) files.

Forwarders are composed of one or more collectors. Each collector's configuration specifies its ingestion mechanism (for example, File, Kafka, PCAP, Splunk, or Syslog) and log type.

Assuming hardware requirements are met, you can use many collectors on the same forwarder to ingest data from a variety of mechanisms and log types. For example, you can install a forwarder with two syslog collectors listening for PAN_FIREWALL and CISCO_ASA_FIREWALL data on separate ports, respectively.

The API lets you create forwarders and their collectors in your Google Security Operations instance. Once a forwarder is created, you can use the Generate Forwarder Files endpoint to get the file contents (as JSON payload) for a forwarder's configuration (.conf) and authentication (_auth.conf) files. These contents can then be written to their respective .conf files for deployment with the Google Security Operations Forwarder service on a Windows or Linux system.

For Python samples that use the Forwarder Management API, see the GitHub repository.

Create a forwarder and its collector(s)

A forwarder must be created before any of its collectors can be created.

To create a forwarder and its collector(s):

1. Create a forwarder.
2. Create a collector for the forwarder.
3. (Optional) Repeat Step 2 to add more collectors.

How to authenticate with the Google Security Operations API

This Google Security Operations API uses the OAuth 2.0 protocol for authentication and authorization. Your application can complete these tasks using either of the following implementations:

• Using the Google API Client Library for your computer language.

• Directly interfacing with the OAuth 2.0 system using HTTP.

See the reference documentation for the Google Authentication library in Python.

Google Authentication libraries are a subset of the Google API client libraries. See other language implementations.

Getting API authentication credentials

Your Google Security Operations representative will provide you with a Google Developer Service Account Credential to enable the API client to communicate with the API.

You also must provide the Auth Scope when initializing your API client. OAuth 2.0 uses a scope to limit an application's access to an account. When an application requests a scope, the access token issued to the application is limited to the scope granted.

Use the following scope to initialize your Google API client:

https://www.googleapis.com/auth/chronicle-backstory

Python example

The following Python example demonstrates how to use the OAuth2 credentials and HTTP client using google.oauth2 and googleapiclient.

# Imports required for the sample - Google Auth and API Client Library Imports.
# Get these packages from https://pypi.org/project/google-api-python-client/ or run $ pip
# install google-api-python-client from your terminal
from google.auth.transport import requests
from google.oauth2 import service_account

SCOPES = ['https://www.googleapis.com/auth/chronicle-backstory']

# The apikeys-demo.json file contains the customer's OAuth 2 credentials.
# SERVICE_ACCOUNT_FILE is the full path to the apikeys-demo.json file
# ToDo: Replace this with the full path to your OAuth2 credentials
SERVICE_ACCOUNT_FILE = '/customer-keys/apikeys-demo.json'

# Create a credential using Google Developer Service Account Credential and Google Security Operations API
# Scope.
credentials = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_FILE, scopes=SCOPES)

# Build a requests Session Object to make authorized OAuth requests.
http_session = requests.AuthorizedSession(credentials)

# Your endpoint GET|POST|PATCH|etc. code will vary below

# Reference List example (for US region)
url = 'https://backstory.googleapis.com/v2/lists/COLDRIVER_SHA256'

# You might need another regional endpoint for your API call; see
# https://cloud.google.com/chronicle/docs/reference/ingestion-api#regional_endpoints

# requests GET example
response = http_session.request("GET", url)

# POST example uses json
body = {
  "foo": "bar"
}
response = http_session.request("POST", url, json=body)

# PATCH example uses params and json
params = {
  "foo": "bar"
}
response = http_session.request("PATCH", url, params=params, json=body)

# For more complete examples, see:
# https://github.com/chronicle/api-samples-python/

Chronicle API query limits

The Chronicle API enforces limits on the volume of requests that can be made by any one customer against the Google Security Operations platform. If you reach or exceed the query limit, the Chronicle API server returns HTTP 429 (RESOURCE_EXHAUSTED) to the caller. When developing applications for the Chronicle API, Google recommends that you enforce rate limits within your system to avoid resource exhaustion. These limits apply to all of the Chronicle APIs, including the Search, Forwarder Management, and Tooling APIs.

See the detailed list of the Chronicle API query limits.

The following limit for the Chronicle Forwarder Management API is being enforced and is measured in queries per second (QPS):

Forwarder API reference

This section describes the endpoints for creating and managing forwarders. For endpoints for creating and managing collectors, see the Collector API reference.

Create Forwarder

Creates a new forwarder in the Google SecOps instance. The new forwarder will include any forwarder configuration values provided in the request body. Configuration values for collectors must be specified using Create Collector after using Create Forwarder.

For certain settings, configuration values that are missing or zero-valued in the request body are set to default values. For details about forwarder fields and values, see Forwarder configuration fields.

Request

POST https://backstory.googleapis.com/v2/forwarders

Request body

{
  "display_name": string,
  "config": {
    object (ForwarderConfig)
  }
}

Body parameters

Request example

This example shows the required key:value pairs in a Create Forwarder request. If a field isn't specified in the request and has a default value, the default value is applied during forwarder creation. For details about default values, see Forwarder configuration fields.

POST https://backstory.googleapis.com/v2/forwarders
{
  "display_name": "chronicle_forwarder"
}

Response

If the request is successful, the response returns an HTTP status code of 200 (OK).

The response shows the configuration values applied during creation of the forwarder. Default configuration values are applied for certain settings during resource creation if those fields are missing or zero-valued in the request body. For details, see Forwarder configuration fields.

Response fields

In addition to the fields specified in the request and the fields for which default values are applied, the response includes the following generated and output-only fields.

Response example

This is an example of the response returned for the request example above.

{
  "name": "forwarders/12ab3cd4-56ef-7abc-d89e-1f23a4bcde56",
  "displayName": "chronicle_forwarder",
  "config": {
    "uploadCompression": "false",
    "serverSettings": {
      "gracefulTimeout": 15,
      "drainTimeout": 10,
      "httpSettings": {
        "port": "8080",
        "host": "0.0.0.0",
        "readTimeout": "3",
        "readHeaderTimeout": "3",
        "writeTimeout": "3",
        "idleTimeout": "3"
        "routeSettings": {
          "availableStatusCode": "204",
          "readyStatusCode": "204",
          "unreadyStatusCode": "503"
        },
      },
    },
  },
  "state": "ACTIVE"
}

Get Forwarder

Returns a forwarder.

Request

GET https://backstory.googleapis.com/v2/forwarders/{forwarderID}

Request body

Don't include a request body.

Request example

GET https://backstory.googleapis.com/v2/forwarders/12ab3cd4-56ef-7abc-d89e-1f23a4bcde56

Response example

{
  "name": "forwarders/12ab3cd4-56ef-7abc-d89e-1f23a4bcde56",
  "displayName": "chronicle_forwarder",
  "config": {
    "uploadCompression": "false",
    "serverSettings": {
      "gracefulTimeout": 15,
      "drainTimeout": 10,
      "httpSettings": {
        "port": "8080",
        "host": "0.0.0.0",
        "readTimeout": "3",
        "readHeaderTimeout": "3",
        "writeTimeout": "3",
        "idleTimeout": "3"
        "routeSettings": {
          "availableStatusCode": "204",
          "readyStatusCode": "204",
          "unreadyStatusCode": "503"
        },
      },
    },
  },
  "state": "ACTIVE"
}

List Forwarders

Lists all of the forwarders for a Google SecOps instance.

Request

GET https://backstory.googleapis.com/v2/forwarders

Request example

GET https://backstory.googleapis.com/v2/forwarders

Response

Returns a list of forwarders.

Response example

{
  "forwarders": [
    {
      "name": "forwarders/12ab3cd4-56ef-7abc-d89e-1f23a4bcde56",
      "displayName": "chronicle_forwarder_1",
      "config": {
        "uploadCompression": "false",
        "serverSettings": {
          "gracefulTimeout": 15,
          ...
         },
      },
      "state": "ACTIVE"
    },
    {
      "name": "forwarders/12ab3cd4-56ef-7abc-d89e-1f23a4bcde57",
      "displayName": "chronicle_forwarder_2",
      "config": {
        "uploadCompression": "false",
        "serverSettings": {
          "gracefulTimeout": 15,
       ...
       },
      },
      "state": "ACTIVE"
    }
  ]
}

Update Forwarder

You can update a forwarder by using the updateMask URL query parameter to specify the fields to update.

For example, to update the display name, you would use the updateMask query parameter as follows in the patch request:

?updateMask=displayName

The request body should only contain the fields that you wish to update (in their exact locations).

Request

PATCH https://backstory.googleapis.com/v2/forwarders/{forwarderID}?updateMask=<field_1,field_2>

Request body

{
  "display_name": string,
  "config": {
    object (ForwarderConfig)
  },
}

Body parameters

Request example

This is an example of an Update Forwarder request where the request specifies new values for displayName and adds a Metadata label.

PATCH https://backstory.googleapis.com/v2/forwarders/12ab3cd4-56ef-7abc-d89e-1f23a4bcde56?updateMask=displayName,config.metadata.labels
{
  "display_name": "UpdatedForwarder",
  "config": {
    "metadata": {
      "labels": [
        {
          "key": "office",
          "value": "corporate",
        }
      ]
    }
  }
}

Response example

This is an example of the response returned for the request example above.

{
  "name": "forwarders/{forwarderUUID}",
  "displayName": "UpdatedForwarder",
  "config": {
    "uploadCompression": "false",
    "metadata": {
      "labels": [
        {
          "key": "office",
          "value": "corporate"
        }
      ]
    }
  },
  "state": "ACTIVE"
}

Delete Forwarder

Deletes a forwarder.

Request

DELETE https://backstory.googleapis.com/v2/forwarders/{forwarderID}

Request body

Don't include a request body.

Request example

DELETE https://backstory.googleapis.com/v2/forwarders/12ab3cd4-56ef-7abc-d89e-1f23a4bcde56

Response example

If the operation is successful, Delete Forwarder returns an empty response with an HTTP status code 200 (OK).

{}

Generate Forwarder Files

Generates and returns the contents of the forwarder's configuration (.conf) and authentication (_auth.conf) files.

Request

GET https://backstory.googleapis.com/v2/forwarders/{forwarderID}:generateForwarderFiles

Request body

Don't include a request body.

Request example

GET https://backstory.googleapis.com/v2/forwarders/12ab3cd4-56ef-7abc-d89e-1f23a4bcde56:generateForwarderFiles

Response example

If the operation is successful, it returns an HTTP status code 200 (OK). It also returns the contents of a forwarder configuration file, including the configuration data for the forwarder's collectors, as well as the contents of the authentication (_auth.conf) file that is used by the forwarder to authenticate with the Google SecOps instance.

Forwarder configuration fields

The following table lists the forwarder configuration settings that you can specify using Create Forwarder and Update Forwarder. If you don't specify a value for a setting when you use Create Forwarder, the setting's default value (shown below) is applied.

The following fields can be provided in the config object of the request body.

Collector API reference

This section describes the endpoints for working with collectors.

When creating and updating collectors, note that each collector configuration can specify ingestion settings for one, but not more than one, of the following:

• Log file data
• Kafka topics
• Packet data (pcap)
• Splunk data
• Syslog data

For endpoints for working with forwarders, see the Forwarder API reference.

Create Collector

Creates a new collector in the Google SecOps account. Configuration values for collectors must be specified using Create Collector after using Create Forwarder.

For certain settings, configuration values that are missing or zero-valued in the request body are set to default values. For details about collector configuration fields and values, see Collector configuration fields.

Request

POST https://backstory.googleapis.com/v2/forwarders/{forwarderID}/collectors

Request body

{
  "display_name": string,
  "config": {
    object (CollectorConfig)
  }
  "state": enum
}

Body parameters

Request example

This example shows the required key:value pairs in a Create Collector request. For any fields that aren't provided, default values are applied during collector creation.

In this example, the collector type is file, so the collector configuration includes file_settings to indicate the collector type and its settings. If the collector type is syslog, then the collector configuration includes syslog_settings. For more information, see Collector configuration fields.

POST https://backstory.googleapis.com/v2/forwarders/12ab3cd4-56ef-7abc-d89e-1f23a4bcde56/collectors
{
  "display_name": "abc_collector",
  "config" {
    "log_type": "CS_EDR"
    "file_settings": {
      "file_path": "/opt/chronicle/edr/output/sample.txt",
    }
  }
}

Response

If the request is successful, the response returns an HTTP status code of 200 (OK).

The response shows the configuration values applied during creation of the collector. Default configuration values are applied for certain settings during resource creation if those fields are missing or zero-valued in the request body. For details, see Collector configuration fields.

Response fields

In addition to the fields specified in the request and the fields for which default values are applied, the response includes the following fields:

Response example

This is an example of the response returned for the request example above.

{
  "name": "forwarders/12ab3cd4-56ef-7ghi-j89k-1l23m4nopq56/collectors/
     98ab7cd6-54ef-3abc-d21e-1f23a4bcde56",
  "displayName": "abc_collector",
  "config": {
    "logType": "tomcat",
    "maxSecondsPerBatch": "10",
    "maxBytesPerBatch": "1048576"
  }
}

Get Collector

Returns a collector.

Request

GET https://backstory.googleapis.com/v2/forwarders/{forwarderID}/collectors/{collectorID}

Request body

Don't include a request body.

Request example

GET
https://backstory.googleapis.com/v2/forwarders/12ab3cd4-56ef-7abc-d89e-1f23a4bcde56/collectors/98ab7cd6-54ef-3abc-d21e-1f23a4bcde56

Response example

{
  "name": "?",
  "displayName": "abc_collector",
  "config": {
    "logType": "tomcat",
    "maxSecondsPerBatch": "10",
    "maxBytesPerBatch": "1048576"
  }
}

List Collectors

Lists the existing collectors for the specified forwarder.

Request

GET https://backstory.googleapis.com/v2/forwarders/{forwarderID}/collectors

Request example

GET https://backstory.googleapis.com/v2/forwarders/12ab3cd4-56ef-7abc-d89e-1f23a4bcde56/collectors

Response

Returns multiple collectors.

Response example

{
  "collectors": [
    {
      "name": "?",
      "displayName": "abc_collector_1",
      "config": {
        "logType": "tomcat",
        "maxSecondsPerBatch": "10",
        "maxBytesPerBatch": "1048576"
      }
    },
    {
      "name": "?",
      "displayName": "abc_collector_2",
      "config": {
        "logType": "tomcat",
        "maxSecondsPerBatch": "10",
        "maxBytesPerBatch": "1048576"
      }
    }
  ]
}

Update Collector

When updating a collector with the API, you can choose to overwrite the entire collector configuration or to overwrite only specific fields of the collector configuration. It is usually best to overwrite specific fields, so you avoid accidentally overwriting all of your data. To overwrite specific fields, supply a FieldMask to your update request.

To supply a FieldMask to update the display name for a collector, supply the updateMask URL query parameter in the patch request. For example:

?updateMask=displayName

The request body should only contain the fields that you wish to update (in their exact locations).

Request

PATCH https://backstory.googleapis.com/v2/forwarders/{forwarderID}/collectors/{collectorID}?updateMask=<field_1,field_2>

Request body

{
  "display_name": string,
  "config": {
    object (CollectorConfig)
  },
}

Body parameters

Request example

This is an example of an Update Collector request where the request specifies new values for displayName, logType, assetNamespace, and protocol.

PATCH https://backstory.googleapis.com/v2/forwarders/12ab3cd4-56ef-7abc-d89e-1f23a4bcde56/collectors/98ab7cd6-54ef-3abc-d21e-1f23a4bcde56?updateMask=displayName,config.logType,config.metadata.assetNamespace,config.syslogSettings.protocol
{
  "display_name": "UpdatedCollector"
  "config": {
    "metadata": {
      "asset_namespace": "COLLECTOR",
      },
      "log_type": "CISCO_ASA_FIREWALL",
      "syslog_settings": {
        "protocol": "TCP",
      }
    }
  }

Response example

This is an example of the response returned for the request example above.

{
  "name": "forwarders/12ab3cd4-56ef-7abc-d89e-1f23a4bcde56/collectors/98ab7cd6-54ef-3abc-d21e-1f23a4bcde56",
  "displayName": "UpdatedCollector",
  "config": {
    "logType": "CISCO_ASA_FIREWALL",
    "metadata": {
      "assetNamespace": "COLLECTOR"
    },
    "maxSecondsPerBatch": 10,
    "maxBytesPerBatch": "1048576",
    "syslogSettings": {
      "protocol": "TCP",
      "address": "0.0.0.0",
      "port": 10514,
    }
  },
  "state": "ACTIVE"
}

Delete Collector

Deletes a collector.

Request

DELETE https://backstory.googleapis.com/v2/forwarders/{forwarderID}/collectors/{collectorID}

Request body

Don't include a request body.

Request example

DELETE https://backstory.googleapis.com/v2/forwarders/12ab3cd4-56ef-7abc-d89e-1f23a4bcde56/collectors/98ab7cd6-54ef-3abc-d21e-1f23a4bcde56

Response example

If the operation is successful, Delete Collector returns an empty response with an HTTP status code 200 (OK).

{}

Collector configuration fields

The following fields can be provided in the config object of the request body.