Elasticsearch

Integration version: 39.0

Accessing Elasticsearch

Google Security Operations SOAR accesses Elasticsearch via RESTful API on TCP port 9200 by default. The Google Security Operations SOAR server will need access to the relevant Elasticsearch nodes on TCP 9200 (default) or an alternate port if the default port was not used during Elasticsearch deployment.

Configure Elasticsearch integration in Google Security Operations SOAR

For detailed instructions on how to configure an integration in Google Security Operations SOAR, see Configure integrations.

Configure Elasticsearch integration with a CA certificate

You can verify your connection with a CA certificate file if needed.

Before you start, ensure you have the following:

• The CA certificate file
• The latest Elasticsearch integration version

To configure the integration with a CA certificate, complete the following steps:

1. Parse your CA certificate file into a Base64 String.
2. Open the integration configuration parameters page.
3. Insert the string in the CA Certificate File field.
4. To test that the integration is successfully configured, select the Verify SSL checkbox and click Test.

Integration parameters

Use the following parameters to configure the integration:

Parameter Display Name	Type	Default Value	Is mandatory	Description
Instance Name	String	N/A	No	Name of the Instance you intend to configure integration for.
Description	String	N/A	No	Description of the Instance.
Server Address	Sring	x.x.x.x	Yes	IP address of the Elasticsearch server.
Username	String	N/A	Yes	The email address of the user which should be used to connect to Elasticsearch.
Password	Password	N/A	Yes	The password of the according user.
Authenticate	Checkbox	Unchecked	No	N/A
Verify SSL	Checkbox	Unchecked	No	Use this checkbox, if your Elasticsearch connection requires an SSL verification.
Run Remotely	Checkbox	Unchecked	No	Check the field in order to run the configured integration remotely. Once checked, the option appears to select the remote user (agent).

Actions

Advanced ES Search

Description

An Elasticsearch test that is pre-made, and returns a word dictionary.

Parameters

Parameters	Type	Default Value	Is Mandatory	Description
Index	String	*	No	Search pattern for a Elasticsearch index. In Elasticsearch, index is like a DatabaseName, and data is stored across various indexes.This parameter defines in what index(es) to search. It can be an exact name ie: "smp_playbooks-2019.06.13" or you can use a wildcard to search by a pattern. e: "smp_playbooks-2019.06 "or "smp". To learn more about Elasticsearch indexes visit https://www.elastic.co/blog/what-is-an-elasticsearch-index
Query	String	*	No	The search query to perform. It is in Lucene syntax. IE1: "*" (this is a wildcard that will return all record) IE2: "level:error" IE3: "level:information" IE4: "level:error OR level:warning" To learn more about Lucene syntax, visit https://www.elastic.co/guide/en/kibana/current/lucene-query.html#lucene-query\r\nhttps://www.elastic.co/guide/en/elasticsearch/reference/7.1/query-dsl-query-string-query.html#query-string-syntax
Limit	String	100	No	Limits the document return count, ie: 10. 0 = No limit.
Display Field	String	*	No	Limits the returned fields. Default "*" = Return all fields. You can state a single field. ie: "level"
Search Field	String	_all	No	Search field for free text queries (When query doesn't specify a field name). Default is "_all", which means all fields are searched. It is best to use proper Lucene syntax on "_all" fields, or textual search on a specific field. Ie1: Search Field = "_all". Query = "level:error" Query will return all records where "level" field, equals "error". Ie2: Search Field = "Message", query = "Login Alarm". Query will return all records, which their "Message" field, contains the text "Login Alarm"
Timestamp Field	String	@timestamp	No	The name of the field to run time-based filtering against. Default is @timestamp. If both Earliest Date and Oldest Date are empty, no time-based filtering will occur.
Oldest Date	String	now-1d	No	Start date of the search. Search will return only records equal or after this point in time. Input may be in exact UTC: Format: YYYY-MM-DDTHH:MM:SSZ ie: 2019-06-04T10:00:00Z Input may also be in relative form (using date-math): tie: "now", "now-1d", "now-1d/d", "now-2h/h" To learn more about date-math visit https://www.elastic.co/guide/en/elasticsearch/reference/7.1/common-options.html#date-math
Earliest Date	String	now	No	End date of the search. Search will return only records equal or before this point in time. Input may be in exact UTC: Format: YYYY-MM-DDTHH:MM:SSZ ie: 2019-06-04T10:00:00Z Input may also be in relative form (using date-math): ie: "now", "now-1d", "now-1d/d", "now-2h/h" To learn more about date-math visit https://www.elastic.co/guide/en/elasticsearch/reference/7.1/common-options.html#date-math

Run On

This action runs on all entities.

Action Results

Script Result

Script Result Name	Value Options	Example
results	N/A	N/A

DSL Search

Description

Searches through everything in Elasticsearch and returns back results in a dictionary format. This action supports only queries without time range, if you want to use time range in your query use Advanced ES Search action.

Parameters

Parameters	Type	Default Value	Is Mandatory	Description
Index	String	*	No	Search pattern for a Elasticsearch index. In Elasticsearch, index is like a DatabaseName, and data is stored across various indexes. This param defines in what index(es) to search. It can be an exact name ie: \"smp_playbooks-2019.06.13\"\r\nor you can use a () wildcard to search by a pattern. e: \"smp_playbooks-2019.06\" or \"smp*\". To learn more about Elasticsearch indexes visit https://www.elastic.co/blog/what-is-an-elasticsearch-index
Query	String	*	No	The search query to perform. It is in Lucene syntax. IE1: \"*\" (this is a wildcard that will return all record) IE2: \"level:error\" IE3: \"level:information\" IE4: \"level:error OR level:warning\" To learn more about Lucene syntax, visit https://www.elastic.co/guide/en/kibana/current/lucene-query.html#lucene-query\r\nhttps://www.elastic.co/guide/en/elasticsearch/reference/7.1/query-dsl-query-string-query.html#query-string-syntax
Limit	String	100	No	NoLimits the document return count, ie: 10. 0 = No limit

Run On

This action runs on all entities.

Action Results

Script Result

Script Result Name	Value Options	Example
results	N/A	N/A

JSON Result

[
    {
        "_score": 0.2876821,
        "_type": "person",
        "_id": "2",
        "_source": {
            "lastname": "Smith",
            "name": "John",
            "job_description": "Systems administrator"
        },
        "_index": "accounts"
    }, {
        "_score": 0.28582606,
        "_type": "person",
        "_id": "1",
        "_source":
        {
            "lastname": "Doe",
            "name": "John",
            "job_description": "Systems administrator and Linux specialist"
        },
        "_index": "accounts"
    }
]

Ping

Description.

Test Verifies connectivity to the Elasticsearch server.

Parameters

N/A

Run On

This action runs on all entities.

Action Results

Script Result

Script Result Name	Value Options	Example
is_success	True/False	is_success:False

Simple ES Search

Description

Action searches through everything in Elasticsearch and returns back results in a dictionary format.

Parameters

Parameters	Type	Default Value	Is Mandatory	Description
Index	String	*	No	Search pattern for a Elasticsearch index. In Elasticsearch, index is like a DatabaseName, and data is stored across various indexes. This param defines in what index(es) to search. It can be an exact name ie: \"smp_playbooks-2019.06.13\" or you can use a () wildcard to search by a pattern. e: \"smp_playbooks-2019.06\" or \"smp*\". To learn more about Elasticsearch indexes visit https://www.elastic.co/blog/what-is-an-elasticsearch-index
Query	String	*	No	The search query to perform. It is in Lucene syntax. IE1: \"*\" (this is a wildcard that will return all record) IE2: \"level:error\" IE3: \"level:information\" IE4: \"level:error OR level:warning\" To learn more about Lucene syntax, visit https://www.elastic.co/guide/en/kibana/current/lucene-query.html#lucene-query\r\nhttps://www.elastic.co/guide/en/elasticsearch/reference/7.1/query-dsl-query-string-query.html#query-string-syntax
Limit	String	100	No	Limits the document return count, ie: 10. 0 = No limit.

Run On

This action runs on all entities.

Action Results

Script Result

Script Result Name	Value Options	Example
results	N/A	N/A

JSON Result

[{
    "_score": 0.2876821,
    "_type": "person",
    "_id": "2",
    "_source":
        {
          "lastname": "Smith",
          "name": "John",
          "job_description": "Systems administrator"
         },
     "_index": "accounts"
 },
 {
     "_score": 0.28582606,
     "_type": "person",
     "_id": "1",
     "_source":
       {
         "lastname": "Doe",
         "name": "John",
         "job_description": "Systems administrator and Linux specialist"
       },
    "_index": "accounts"
  }
 ]

Connectors

Configure Elasticsearch connectors in Google Security Operations SOAR

For detailed instructions on how to configure a connector in Google Security Operations SOAR, see Configuring the connector.

To configure the selected connector use the connector-specific parameters listed in the following tables:

• Elasticsearch Connector configuration parameters
• Elasticsearch DSL Connector configuration parameters

Elasticsearch Connector

Description

This topic shows how Google Security Operations SOAR integrates Elasticsearch with the mechanism and configuration for ingesting and processing.

Elasticsearch alert forwarding to Google Security Operations SOAR

Google Security Operations SOAR will search specified Elasticsearch indexes with a provided query (using Lucene query syntax) and return Elasticsearch documents that will be translated and contextualized as "alerts" for cases.

Connector parameters

Use the following parameters to configure the connector:

Parameter Display Name	Type	Default Value	Is Mandatory	Description
Default Environment	String	N/A	No	Select the required environment. For example, "Customer One".
Run Every	Integer	0:0:0:10	No	Select the amount of time to run the connection. For example, "every day".
Product Field Name	String	device_product	Yes	The field name used to determine the device product. Example: _type.
Event Field Name	String	name	Yes	The field name used to determine the event name (sub-type). Example: _source_match_event_id.
Script Timeout (Seconds)	String	60	Yes	The timeout limit (in seconds) for the python process running current script.
Server Address	String	N/A	Yes	The Elasticsearch server address, for example: http://{ip_address}:{port}
Username	String	N/A	Yes	Elasticsearch username.
Password	Password	N/A	Yes	Elasticsearch password.
Authenticate	Checkbox	Unchecked	Yes	Whether to authenticate on connection or not.
Verify SSL	Checkbox	Unchecked	No	Whether to use ssl on connection or not.
Alert Name Field	String	N/A	Yes	The name of the field where the alert name is located (flat field path). Example: _source_alert_info_alert
Timestamp Field	String	N/A	Yes	The name of the field where the timestamp is located (flat field path). Example: source@timestamp
Environment Field	String	N/A	No	The name of the field where the environment is located (flat field path). Example: _source_environment
Indexes	String	N/A	No	Index pattern to search by. Example: '*'
Query	String	N/A	No	Search pattern query (Lucene query syntax). Example: '*'
Alerts Count Limit	Integer	20	Yes	Max count of alerts to pull in one cycle. Example: 20
Max Days Backwards	Integer	1	Yes	Max number of days to fetch alerts since. Example: 3.
Severity Field Name	String	N/A	No	If you want to map severity based on the string value then you would need to create a mapping file. Please refer to documentation portal for more details.
Proxy Server Address	String	N/A	No	The address of the proxy server to use.
Proxy Username	String	N/A	No	The proxy username to authenticate with.
Proxy Password	Password	N/A	No	The proxy password to authenticate with.
Environment Regex Pattern	String	.*	No	A regex pattern to run on the value found in the "Environment Field Name" field. Default is .* to catch all and return value unchanged. Used to allow the user to manipulate the environment field via regex logic If the regex pattern is null or empty, or the environment value is null, the final environment result is "".
CA Certificate File	String	N/A	No	CA Certificate File

How to map severity in the connector

In order to map severity you need to specify what field should be used to get value for severity in the "Severity Field Name" parameter. In the response you can get 3 types of values: integers, floats and strings. For integers and floats, you don't need to do any additional configuration. The connector will read those values and map them according to the Google Security Operations SOAR standards. A quick reminder of how integer values are mapped:

• 100 - Critical
• 100 > x >= 80 High
• 80 > x >=60 Medium
• 60 > x >=40 Low
• 40 > x Informational

If in the response, we are working with strings then additional configuration is required. In the folder, where connector scripts are located you will have a config file name severity_map_config.json. This file defines mapping rules for the severity.

Initially, the file will look like this:

{
    "Default": 50
}

Imagine a situation, where the needed values are located in the event.severity. event.severity can contain the following values: "Malicious", "Benign", "Unknown".

First, we have to specify in the "Severity Field Name" parameter that we will use event.severity.

Secondly, we have to update the config file.

After changes, this is how severity_map_config.json file should look like:

{
    "event.severity": {
        "Malicious": 100,
        "Unknown": 60,
        "Benign": -1
    },
    "Default": 50
}

Now, when the connector will get an event with event.severity = "Malicious" it will give it Critical severity.

Connector rules

Whitelist/Blacklist

The connector doesn't support Whitelist/Blacklist.

Proxy support

The connector supports proxy.

Elasticsearch DSL Connector

Description

The connector works by making a REST API call with a DSL query.

Use cases and examples

Ability to use DSL queries as a search parameter in Elasticsearch.

Connector parameters

Use the following parameters to configure the connector:

Parameter Display Name	Type	Default Value	Is Mandatory	Description
Product Field Name	String	device_product	Yes	Describes the name of the field where the product name is stored.
Environment Field Name	String	""	No	Describes the name of the field where the environment name is stored. If the environment field isn't found, the environment is "".
Environment Regex Pattern	String	.*	No	A regex pattern to run on the value found in the "Environment Field Name" field. Default is .* to catch all and return value unchanged. Used to allow the user to manipulate the environment field via regex logic If the regex pattern is null or empty, or the environment value is null, the final environment result is "".
Script Timeout (Seconds)	Integer	60	Yes	Timeout limit for the python process running the current script.
Server Address	String	N/A	Yes	IP address of the Elasticsearch API server.
Port	String	N/A	Yes	Port of Elasticsearch API server.
Query	String	N/A	Yes	DSL Query that is used for the search. Valid JSON format needed. To make the connector more stable it is recommended to add a sorting timestamp key in the ascending order.
Index	String	N/A	Yes	Index that is used for a search. For example: _all
Timestamp Field	String	N/A	Yes	The name of the field where the timestamp is located. Example: source@timestamp
Alert Field Name	String	N/A	Yes	The name of the field where the alert name is located. Example: _source_info_alertname
Description Field	String	N/A	No	The name of the field where the description is located. Example: _source_alert_info_description
Severity	String	Medium	Yes	Severity of the alerts. Possible value: Info Low Medium High Critical
Alerts Count Limit	Integer	100	No	Limit the number of alerts returned by the connector per 1 iteration.
Authenticate	Checkbox	Unchecked	No	Whether to authenticate on a connection or not.
Username	String	N/A	No	Elasticsearch account username.
Password	Password	N/A	No	Elasticsearch account password.
Use SSL	Checkbox	Unchecked	No	Option to enable SSL/TLS connection.
Severity Field Name	String	N/A	No	If you want to map severity based on the string value then you would need to create a mapping file. Please refer to documentation portal for more details.
Alert Severity	String	N/A	No	The severity of the alerts. Possible value: Info, Low, Medium, High, Critical. Note: this parameter has priority over "Severity Field Name". If you want to work with "Severity Field Name", this field should be left empty.
Proxy Server Address	String	N/A	No	The address of the proxy server to use.
Proxy Username	String	N/A	No	The proxy username to authenticate with.
Proxy Password	Password	N/A	No	The proxy password to authenticate with.

Supported Notations

The connector supports three notations. For example, if you want use event.type in the "Event Field Name" parameter. In that case, you can either provide _source_event_type, event_type or event.type. All of these values will behave the same way.

For parameters:

• Product Field Name
• Event Field Name
• Severity Field Name
• Environment Field
• Timestamp Field
• Alert Name Field
• Alert Description Field - this one is only for DSL connectors

How to map severity in the connector

In order to map severity you need to specify what field should be used to get value for severity in the "Severity Field Name" parameter. In the response you can get 3 types of values: integers, floats and strings. For integers and floats, you don't need to do any additional configuration. The connector will read those values and map them according to the Google Security Operations SOAR standards. A quick reminder of how integer values are mapped:

• 100 - Critical
• 100 > x >= 80 High
• 80 > x >=60 Medium
• 60 > x >=40 Low
• 40 > x Informational

If in the response, we are working with strings then additional configuration is required. In the folder, where connector scripts are located you will have a config file name severity_map_config.json. This file defines mapping rules for the severity.

Initially, the file will look like this:

{
    "Default": 50
}

Imagine a situation, where the needed values are located in the event.severity. event.severity can contain the following values: "Malicious", "Benign", "Unknown".

First, we have to specify in the "Severity Field Name" parameter that we will use event.severity.

Secondly, we have to update the config file.

After changes, this is how severity_map_config.json file should look like:

{
    "event.severity": {
        "Malicious": 100,
        "Unknown": 60,
        "Benign": -1
    },
    "Default": 50
}

Now, when the connector will get an event with event.severity = "Malicious" it will give it Critical severity.

Connector rules

Whitelist/Blacklist

The connector doesn't support Whitelist/Blacklist.

Proxy support

The connector supports proxy.