Ingestion metrics schema
This document describes Google Security Operations ingestion_metric
table. The fields
in the ingestion_metric
table represent the dimensions that appear in the
Ingestion metrics Explore interface. You use the Ingestion metrics Explore
interface to create a new dashboard.
The following Ingestion metrics schema helps you understand the data types and format of the dimension fields available in the Ingestion metrics Explore interface:
Fields | Type | Description |
---|---|---|
component
|
STRING | The service or pipeline to which the metric applies, which includes the Forwarder ,
Ingestion API , Out-of-Band Processor (Chronicle API feed), and Normalizer .
|
collector_id
|
STRING | The unique identifier of the collection mechanism. For push sources, the forwarder ID or generated ID is used. For Chronicle API or Chronicle API feed, the ID has the following format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. |
feed_id
|
STRING | The ID for the specific feed which the log belongs to. |
log_type
|
STRING | The source of the log which identifies the log entries in the batch. For example, WINDOWS_DNS .
|
start_time
|
TIMESTAMP | The start time associated with the metric in microseconds. |
end_time
|
TIMESTAMP | The end time associated with the metric in microseconds. |
input_type
|
STRING | This field is populated if the ingestion source is the Google Security Operations
forwarder. Based on the data that the forwarder sends, this field contains pcap ,
syslog , or splunk .
|
namespace
|
STRING | Namespace that the log belongs to. |
event_type
|
STRING | The event type determines which fields are included with the event. The event
type includes values such as PROCESS_OPEN , FILE_CREATION , USER_CREATION , and NETWORK_DNS .
|
drop_reason_code
|
STRING | This field is populated if the ingestion source is the Google SecOps forwarder and indicates the reason why a log was dropped during normalization. Possible values are the following:
|
last_heartbeat_time
|
TIMESTAMP | This field is populated if the ingestion source is the Google Security Operations forwarder or Chronicle API feed. The last timestamp when the forwarder or API feed was active in microseconds. When the feed is active, it sends the data through the |
log_volume
|
FLOAT64 | The volume of logs during the interval in bytes. The
|
drop_count
|
FLOAT64 | This field is populated if the ingestion source is the Google Security Operations forwarder. and indicates the number of logs dropped during the interval. |
log_count
|
FLOAT64 | The number of logs ingested during the interval. The
|
event_count
|
FLOAT64 | The number of events generated during the interval. |
state
|
STRING | The final status of the event or log. The status is one of the following:
|
memory_used |
FLOAT64 | The percentage of memory used by the forwarder container. |
disk_used |
FLOAT64 | The percentage of disk storage used by the forwarder container along with disk identifier. |
cpu_used |
FLOAT64 | The percentage of CPU used by the forwarder container. |
buffer_used |
FLOAT64 | The percentage of buffer used by the forwarder buffer type per forwarder input type. |
buffer_type |
STRING | The type of buffer used by the forwarder. |
ingestion_source |
STRING | The ingestion source present in ingestion label when the logs are ingested through ingestion private API. |
You can filter ingestion metrics based on the field values. For example, Out-of-Band Processor
feeds have collector_id
as aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
. Here is an example
query to filter Out-of-Band feeds:
SELECT component, collector_id, count(component) FROM chronicle-tla.datalake.ingestion-metrics WHERE DATE(start_time) = DATE_SUB(CURRENT_DATE(), INTERVAL 60 DAY) AND component IN ("Out-of-Band Processor","Ingestion API", "Forwarder") AND (collector_id != "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa" OR collector_id is null) group by 1,2
The following fields that appear in the Ingestion metrics Explore interface are reserved for future use:
- Bucketer growth factor
- Bucketer num finite buckets
- Bucketer scale factor
- Bucketer width
- Error code
- Latency count
- Latency overflow
- Latency time
- Latency underflow
- Regex filter
Ingestion metrics examples
The following table lists the metrics and example values that help you understand the fields in the ingestion_metric
schema:
Metrics | component | collector_id | feed_id | log_type | start_time | end_time | input_type | last_heartbeat_time | log_volume | drop_count | log_count | memory_used | cpu_used | disk_used | buffer_used | ingestion_source | drop_reason_code |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Heartbeat | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | syslog | 2022-04-21T13:18:55.000+00:00 | ||||||||||
Log Bytes Count | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | pcap | 149.0 | ||||||||||
Log Record Count | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | pcap | 154.0 | ||||||||||
Drop Count (Backlog) | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | pcap | 4.0 | backlog | |||||||||
Drop Count (Invalid Config) | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | pcap | 4.0 | invalid_config | |||||||||
Drop Count (Regex) | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | pcap | 4.0 | regex | |||||||||
Log Record Count | Ingestion API | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DHCP | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | 3578.0 | |||||||||||
Log Bytes Count | Ingestion API | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DHCP | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | 2802.0 | |||||||||||
Log Record Count | Out-of-Band Processor | feeds/aaaaaaaaaaaaaa | ARUBA_IPS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | 3578.0 | |||||||||||
Log Bytes Count | Out-of-Band Processor | feeds/aaaaaaaaaaaaaa | ARUBA_IPS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | 319563.0 | |||||||||||
Last Ingested Timestamp | Out-of-Band Processor | feeds/aaaaaaaaaaaaaa | ARUBA_IPS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | 2022-04-21T13:18:55.000+00:00 | |||||||||||
Log Count | Normalizer | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | ||||||||||||
Log Size | Normalizer | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | ||||||||||||
Event Count | Normalizer | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | ||||||||||||
Container Memory Used | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | 0.32 | |||||||||||
Container Disk Used | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | 0.5 | |||||||||||
Container CPU Used | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | 0.545 | |||||||||||
Buffer Used | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | 0.562 | |||||||||||
Ingestion Source | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | windows-spain-dc-1 |
What's next
- Read about Ingestion metrics field reference for dashboards.