Dokumen ini memberikan ringkasan tentang cara data diekstrak secara otomatis untuk meningkatkan kemampuan penyerapan, pemrosesan, dan analisis data.
Google Security Operations menggunakan parser bawaan
untuk mengekstrak dan menyusun data log menggunakan skema Model Data Terpadu (UDM). Mengelola dan memelihara parser ini bisa jadi sulit karena beberapa batasan: ekstraksi data yang tidak lengkap, jumlah parser yang terus bertambah untuk dikelola, dan persyaratan untuk pembaruan yang sering karena format log terus berkembang.
Untuk mengatasi tantangan ini, Anda dapat menggunakan fitur ekstraksi otomatis. Fitur ini secara otomatis mengekstrak key-value pair dari log berformat JSON yang di-ingest ke Google SecOps. Log yang diformat Syslog yang menyertakan pesan JSON juga didukung. Data yang diekstrak ini disimpan dalam UDM, kolom
jenis peta bernama extracted. Kemudian, Anda dapat menggunakan data ini dalam kueri penelusuran UDM, Dasbor Native, dan aturan YARA-L.
Sebagai praktik terbaik, penelusuran UDM yang menggunakan kolom yang diekstrak harus menyertakan metadata.log_type
dalam kueri untuk meningkatkan performa kueri penelusuran.
Manfaat ekstraksi otomatis adalah berkurangnya ketergantungan pada parser, sehingga memastikan data tetap tersedia, meskipun parser tidak ada atau gagal mem-parsing log.
Mengurai dan mengekstrak data dari log mentah
Parsing: Google SecOps mencoba mem-parsing log menggunakan parser khusus untuk jenis log, jika tersedia. Jika tidak ada parser tertentu, atau jika parsing gagal, Google SecOps menggunakan parser umum untuk mengekstrak informasi dasar seperti stempel waktu yang diproses, jenis log, dan label metadata.
Ekstraksi Data: Semua titik data diekstrak secara otomatis dari log.
Pengayaan Peristiwa: Google SecOps menggabungkan data yang diuraikan dan kolom berformat kustom untuk membuat peristiwa yang diperkaya, sehingga memberikan lebih banyak konteks dan detail.
Transfer Data Hilir: Peristiwa yang telah di-enrich ini kemudian dikirim ke sistem lain untuk analisis dan pemrosesan lebih lanjut.
Bekerja dengan ekstraktor
Dengan ekstraktor, Anda dapat mengekstrak kolom dari sumber log bervolume tinggi, dan dirancang untuk mengoptimalkan pengelolaan log. Dengan menggunakan ekstraktor, Anda dapat mengurangi ukuran peristiwa,
meningkatkan efisiensi parsing, dan mendapatkan kontrol yang lebih baik atas ekstraksi data.
Hal ini sangat berguna untuk mengelola jenis log baru atau meminimalkan waktu pemrosesan.
Anda dapat membuat ekstraktor menggunakan menu Setelan SIEM atau dengan melakukan penelusuran log mentah.
Membuat pengekstrak
Buka panel Ekstrak Kolom Tambahan menggunakan salah satu
metode berikut:
Klik Setelan SIEM > Parser, lalu lakukan hal berikut:
Di tabel PARSER yang muncul, identifikasi parser (sumber log)
dan klik more_vertMenu >
Extend Parser > Extract Additional Fields.
Di tab PERISTIWA di hasil penelusuran UDM, pilih sumber log untuk melihat panel Event Viewer.
Di tab Raw Log, klik Manage Parser > Extend Parser >
Extract Additional Fields.
Di tab Pilih Ekstraktor di panel Ekstrak Kolom Tambahan,
pilih kolom log mentah yang diperlukan. Secara default, Anda dapat memilih hingga 100 kolom.
Jika tidak ada kolom tambahan yang tersedia untuk diekstrak, pemberitahuan peringatan akan ditampilkan.
Klik tab Reference Raw Log untuk melihat data log mentah dan melihat pratinjau output UDM.
Klik Simpan.
Ekstraktor yang baru dibuat diberi label EXTRACTOR.
Kolom yang diekstrak ditampilkan dalam output UDM sebagaiextracted.field{"fieldName"}.
Melihat detail ekstraktor
Buka baris ekstraktor di tabel PARSER, lalu klik more_vertMenu > Perluas Parser > Lihat Ekstensi.
Di halaman LIHAT PARSER KUSTOM, klik tab Ekstensi dan Kolom yang Diekstrak.
Tab ini menampilkan informasi tentang ekstensi parser dan kolom ekstraktor.
Anda dapat mengubah atau menghapus kolom dan melihat pratinjau output parser dari halaman LIHAT PARSER KUSTOM.
[[["Mudah dipahami","easyToUnderstand","thumb-up"],["Memecahkan masalah saya","solvedMyProblem","thumb-up"],["Lainnya","otherUp","thumb-up"]],[["Sulit dipahami","hardToUnderstand","thumb-down"],["Informasi atau kode contoh salah","incorrectInformationOrSampleCode","thumb-down"],["Informasi/contoh yang saya butuhkan tidak ada","missingTheInformationSamplesINeed","thumb-down"],["Masalah terjemahan","translationIssue","thumb-down"],["Lainnya","otherDown","thumb-down"]],["Terakhir diperbarui pada 2025-08-21 UTC."],[[["\u003cp\u003eAuto extraction is a feature in Google SecOps that automatically extracts key-value pairs from JSON-formatted logs, storing them in a UDM map-type field called \u003ccode\u003eextracted\u003c/code\u003e.\u003c/p\u003e\n"],["\u003cp\u003eThis feature reduces reliance on prebuilt parsers and ensures data availability even when a specific parser is absent or fails.\u003c/p\u003e\n"],["\u003cp\u003eAuto extraction enhances the ability to ingest, process, and analyze data, which is then searchable through UDM queries, Preview Dashboards, and YARA-L rules.\u003c/p\u003e\n"],["\u003cp\u003eUDM searches using extracted fields should include \u003ccode\u003emetadata.log_type\u003c/code\u003e to optimize search query performance.\u003c/p\u003e\n"],["\u003cp\u003eThe process includes parsing logs, extracting data, enriching events with parsed and custom fields, and then transferring this data downstream for analysis.\u003c/p\u003e\n"]]],[],null,["# Auto Extraction overview\n========================\n\nSupported in: \nGoogle secops [SIEM](/chronicle/docs/secops/google-secops-siem-toc)\n| **Note:** This feature is covered by [Pre-GA Offerings Terms](https://chronicle.security/legal/service-terms/) of the Google Security Operations Service Specific Terms. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the [Google SecOps Technical Support Service guidelines](https://chronicle.security/legal/technical-support-services-guidelines/) and the [Google SecOps Service Specific Terms](https://chronicle.security/legal/service-terms/).\n\nThis document provides an overview of how data is automatically extracted to\nenhance the ability to ingest, process, and analyze data.\n\nGoogle Security Operations uses [prebuilt parsers](/chronicle/docs/ingestion/parser-list/supported-default-parsers)\nto extract and structure log data using the Unified Data Model (UDM) schema. Managing and\nmaintaining these parsers can be challenging due to several limitations: incomplete\ndata extraction, the growing number of parsers to manage, and the requirement for\nfrequent updates as log formats evolve.\n\nTo address these challenges, you can use the auto extraction\nfeature. This feature automatically extracts key-value pairs from JSON-formatted\nand XML-formatted logs ingested into Google SecOps. It also supports Syslog-formatted\nlogs that include a JSON message. This extracted data is stored in a UDM, map-type\nfield called `extracted`. You can then use this data within UDM search queries,\n[Native Dashboards](/chronicle/docs/reports/native-dashboards), and YARA-L\nrules.\n\nAs a best practice, the UDM searches using extracted fields must include `metadata.log_type`\nin their query to improve search query performance.\n\nThe benefit of auto extraction is reduced reliance on parsers, ensuring that data\nremains available, even when a parser is not present or fails to parse a log.\n| **Note:** This feature is being rolled out in phases, so some eligible log sources may not yet show extracted fields.\n\nParse and extract data from the raw log\n---------------------------------------\n\n1. **Parsing**: Google SecOps attempts to parse logs using a parser\n specific to the log type, if available. If no specific parser exists, or if parsing\n fails, Google SecOps uses a general parser to extract basic information like\n ingested timestamp, log type, and metadata labels.\n\n2. **Data Extraction**: All data points are automatically extracted from the logs.\n\n3. **Event Enrichment**: Google SecOps combines the parsed data and any\n custom-formatted fields to create enriched events, providing more context and detail.\n\n4. **Downstream Data Transfer**: These enriched events are then sent to other\n systems for further analysis and processing.\n\nWork with extractors\n--------------------\n\nExtractors let you extract fields from high-volume log sources, and are designed\nto optimize log management. By using extractors, you can reduce event size,\nenhance parsing efficiency, and gain better control over data extraction.\nThis is especially useful for managing new log types or minimizing processing\ntime.\n\nYou can create extractors using the **SIEM Settings** menu or by performing a\nraw log search.\n\n### Create extractors\n\n1. Go to the **Extract Additional Fields** pane using either of the following\n methods:\n\n - Click **SIEM Settings** \\\u003e **Parsers** , and do the following:\n 1. In the **PARSERS** table that appears, identify a parser (log source) and click more_vert **Menu** \\\u003e **Extend Parser** \\\u003e **Extract Additional Fields**.\n - Use [Raw Log Scan](/chronicle/docs/investigation/search-raw-logs) and do the following:\n 1. Select the required log sources (parsers) from the **Log Sources** menu.\n 2. From the raw log results, select a log source to open the **EVENT DATA** pane.\n 3. In the **EVENT DATA** pane, click **Manage Parser** \\\u003e **Extend Parser** \\\u003e **Extract Additional Fields**.\n - Use [UDM search](/chronicle/docs/investigation/udm-search#access_search) and do the following:\n 1. On the **EVENTS** tab in the UDM search results, select a log source to view the **Event Viewer** pane.\n 2. On the **Raw Log** tab, click **Manage Parser** \\\u003e **Extend Parser** \\\u003e **Extract Additional Fields**.\n2. On the **Select Extractors** tab in the **Extract Additional fields** pane,\n select the required raw log fields. By default, you can select up to 100 fields.\n If no additional fields are available for extraction, a warning notice displays.\n\n Click the **Reference Raw Log** tab to view the raw log data and preview\n the UDM output.\n3. Click **Save**.\n\nThe newly created extractor is labeled as `EXTRACTOR`.\nExtracted fields are displayed in the UDM output as`extracted.field{\"fieldName\"}`.\n| **Note:** For certain low-volume log types, auto extraction is enabled by default, and all fields are extracted automatically. These log types aren't labeled with the `EXTRACTOR` tag. If you try to extract additional fields, the system displays a message indicating that all fields are already being extracted and no further selection is needed.\n\n### View extractor details\n\n1. Go to the extractor row in the **PARSERS** table and click more_vert **Menu** \\\u003e **Extend Parser** \\\u003e **View Extension**.\n2. On the **VIEW CUSTOM PARSERS** page, click the **Extensions and Extracted Fields** tab.\n\nThis tab displays information on parser extensions and extractor fields.\nYou can modify or remove fields and preview the parser output from the **VIEW CUSTOM PARSERS**\npage.\n\n**Need more help?** [Get answers from Community members and Google SecOps professionals.](https://security.googlecloudcommunity.com/google-security-operations-2)"]]
