Collect MISP IOC logs

This guide outlines the steps to ingest MISP (Malware Information Sharing Platform) IOC (Indicators of Compromise) logs to Google Security Operations using Bindplane. The parser extracts IOCs from MISP data formatted as JSON or CSV. It parses the input, maps fields to the Unified Data Model (UDM), handles various IOC types (such as IP, domain, and file hashes), and enriches the data with threat intelligence context like confidence and severity. The parser also performs specific logic for different data formats and handles cases with missing or unsupported fields.

Before you begin

Make sure you have the following prerequisites:

• Google SecOps instance
• Linux host with systemd
• If running behind a proxy, firewall ports are open
• Privileged access to your MISP server

Get MISP API Key

1. Sign in to your MISP web UI as an Administrator.
2. Go to Administration > List Auth Keys.
3. Click Add authentication key.
4. Provide the following Key configuration:
   • User: Select the user account associated with the key.
   • Allowed IPs: You can optionally specify allowed IP addresses for the key.
   • Copy and save the key in a secure location.
   • Click I have noted down my key.

Configure MISP logs export

1. Sign in to your MISP instance using SSH.

2. Install PyMISP using the following command:

   pip3 install pymisp 
   

3. Modify the get_csv.py export script using the following:

   #!/usr/bin/env python3
   # -*- coding: utf-8 -*-
   
   import argparse
   
   from pymisp import ExpandedPyMISP
   from keys import misp_url, misp_key, misp_verifycert
   
   if __name__ == '__main__':
       parser = argparse.ArgumentParser(description='Get MISP data in a CSV format.')
       parser.add_argument("--controller", default='attributes', help="Attribute to use for the search (events, objects, attributes)")
       parser.add_argument("-e", "--event_id", help="Event ID to fetch. Without it, it will fetch the whole database.")
       parser.add_argument("-a", "--attribute", nargs='+', help="Attribute column names")
       parser.add_argument("-o", "--object_attribute", nargs='+', help="Object attribute column names")
       parser.add_argument("-t", "--misp_types", nargs='+', help="MISP types to fetch (ip-src, hostname, ...)")
       parser.add_argument("-c", "--context", action='store_true', help="Add event level context (tags...)")
       parser.add_argument("-f", "--outfile", help="Output file to write the CSV.")
       parser.add_argument("-l", "--last", required=True, help="can be defined in days, hours, minutes (for example 5d or 12h or 30m).")
   
       args = parser.parse_args()
       pymisp = ExpandedPyMISP(misp_url, misp_key, misp_verifycert, debug=True)
       attr = []
       if args.attribute:
           attr += args.attribute
       if args.object_attribute:
           attr += args.object_attribute
       if not attr:
           attr = None
       print(args.context)
       response = pymisp.search(return_format='csv', controller=args.controller, eventid=args.event_id, requested_attributes=attr, publish_timestamp=args.last,
                               type_attribute=args.misp_types, include_context=args.context)
   
       if args.outfile:
           with open(args.outfile, 'w') as f:
               f.write(response)
       else:
           print(response)
   

   1. Edit keys.py file to include your MISP API credentials and URL:

   misp_url = 'https://<MISP_URL>'
   misp_key = '<MISP_API_KEY>'
   misp_verifycert = False
   misp_client_cert = ''
   

   • Replace <MISP_URL> with your MISP IP or hostname.
   • Replace <MISP_API_KEY with the actual API key generated previously.

4. Open crontab using the crontab -e command and enter the following:

   0 20 * * * python3 /opt/misp/<YOUR_EXPORT_SCRIPT_PATH> -f /opt/misp/ioc_export/url.log -t "url" -l 1d -c
   0 20 * * * python3 /opt/misp/<YOUR_EXPORT_SCRIPT_PATH> -f /opt/misp/ioc_export/ip-dst.log -t "ip-dst" -l 1d -c
   0 20 * * * python3 /opt/misp/<YOUR_EXPORT_SCRIPT_PATH> -f /opt/misp/ioc_export/ip-src.log -t "ip-src" -l 1d -c
   0 20 * * * python3 /opt/misp/<YOUR_EXPORT_SCRIPT_PATH> -f /opt/misp/ioc_export/domain.log -t "domain" -l 1d -c
   0 20 * * * python3 /opt/misp/<YOUR_EXPORT_SCRIPT_PATH> -f /opt/misp/ioc_export/sha256.log -t "sha256" -l 1d -c
   0 20 * * * python3 /opt/misp/<YOUR_EXPORT_SCRIPT_PATH> -f /opt/misp/ioc_export/file.log -t "filename" -l 1d -c
   0 20 * * * python3 /opt/misp/<YOUR_EXPORT_SCRIPT_PATH> -f /opt/misp/ioc_export/registry.log -t "registry" -l 1d -c
   0 20 * * * python3 /opt/misp/<YOUR_EXPORT_SCRIPT_PATH> -f /opt/misp/ioc_export/mutex.log -t "mutex" -l 1d -c
   0 20 * * * python3 /opt/misp/<YOUR_EXPORT_SCRIPT_PATH> -f /opt/misp/ioc_export/threat-actor.log -t "threat-actor" -l 1d -c
   

   • Update <YOUR_EXPORT_SCRIPT_PATH> according to the actual export script location.

Get Google SecOps ingestion authentication file

1. Sign in to the Google SecOps console.
2. Go to SIEM Settings > Collection Agents.
3. Download the Ingestion Authentication File. Save the file securely on the system where Bindplane will be installed.

Get Google SecOps customer ID

1. Sign in to the Google SecOps console.
2. Go to SIEM Settings > Profile.
3. Copy and save the Customer ID from the Organization Details section.

Install the Bindplane agent on MISP Server

Linux installation

1. Open a terminal with root or sudo privileges.

2. Run the following command:

   sudo sh -c "$(curl -fsSlL https://github.com/observiq/bindplane-agent/releases/latest/download/install_unix.sh)" install_unix.sh
   

Additional installation resources

For additional installation options, consult the installation guide.

Configure the Bindplane agent to ingest MISP log files and send to Google SecOps

1. Access the configuration file:
   • Locate the config.yaml file. Typically, it's in the /etc/bindplane-agent/ directory on Linux.
   • Open the file using a text editor (for example, nano, vi, or Notepad).

2. Edit the config.yaml file as follows:

   receivers:
       filelog:
           file_path: /opt/misp/ioc_export/*.log
           log_type: 'file'
   
   exporters:
       chronicle/chronicle_w_labels:
           compression: gzip
           # Adjust the path to the credentials file you downloaded in Step 1
           creds_file_path: '/path/to/ingestion-authentication-file.json'
           # Replace with your actual customer ID from Step 2
           customer_id: <customer_id>
           endpoint: malachiteingestion-pa.googleapis.com
           # Add optional ingestion labels for better organization
           ingestion_labels:
               log_type: 'MISP_IOC'
               raw_log_field: body
   
   service:
       pipelines:
           logs/source0__chronicle_w_labels-0:
               receivers:
                 - filelog
               exporters:
                   - chronicle/chronicle_w_labels
   

   • Replace the port and IP address as required in your infrastructure.
   • Replace <customer_id> with the actual Customer ID.
   • Update /path/to/ingestion-authentication-file.json to the path where the authentication file was saved in the Get Google SecOps ingestion authentication file section.

Restart the Bindplane agent to apply the changes

To restart the Bindplane agent in Linux, run the following command:

  sudo systemctl restart bindplane-agent

UDM mapping table

Need more help? Get answers from Community members and Google SecOps professionals.