Collect Qualys Continuous Monitoring logs

This Logstash parser code first extracts fields such as source IP, user, method, and application protocol from raw log messages using grok patterns. It thenmaps specific fields from the raw log data to their corresponding fields in the Unified Data Model (UDM), performs data type conversions, and enriches the data with additional labels and metadata before finally structuring the output in the desired UDM format.

Before you begin

• Ensure that you have a Google Security Operations instance.
• Ensure that you have privileged access to Google Cloud.
• Ensure that you have privileged access to Qualys.

Enable Required APIs:

1. Sign in to the Google Cloud console.
2. Go to APIs & Services > Library.
3. Search for the following APIs and enable them:
   • Cloud Functions API
   • Cloud Scheduler API
   • Cloud Pub/Sub (required for Cloud Scheduler to invoke functions)

Create a Google Cloud Storage Bucket

1. Sign in to the Google Cloud console.

2. Go to the Cloud Storage Buckets page.

   Go to Buckets

3. Click Create.

4. Configure the bucket:

   • Name: enter a unique name that meets the bucket name requirements (for example, qualys-asset-bucket).
   • Choose where to store your data: select a location.
   • Choose a storage class for your data: either select a default storage class for the bucket, or select Autoclass for automatic storage class management.
   • Choose how to control access to objects: select not to enforce public access prevention, and select an access control model for your bucket's objects.
   • Storage class: choose based on your needs (for example, Standard).

5. Click Create.

Create a Google Cloud Service Account

1. Sign in to the Google Cloud console.
2. Go to to IAM & Admin > Service Accounts.
3. Create a new service account.
4. Give it a descriptive name (for example, qualys-user).
5. Grant the service account with Storage Object Admin role on the GCS bucket you created in the previous step.
6. Grant the service account with Cloud Functions Invoker role.
7. Create an SSH key for the service account.
8. Download a JSON key file for the service account. Keep this file secure.

Optional: Create a dedicated API User in Qualys

1. Sign in to the Qualys console.
2. Go to Users.
3. Click New > User.
4. Enter the General Information required for the user.
5. Select the User Role tab.
6. Make sure the role has the API Access checkbox selected.
7. Click Save.

Identify your specific Qualys API URL

Option 1

Identify your URLs as mentioned in the platform identification.

Option 2

1. Sign in to the Qualys console.
2. Go to Help > About.
3. Scroll to see this information under Security Operations Center (SOC).
4. Copy the Qualys API URL.

Configure the Cloud Function

1. Go to Cloud Functions in the Google Cloud console.
2. Click Create Function.

3. Configure the Function:

   • Name: enter a name for your function (for example, fetch-qualys-cm-alerts).
   • Region: select a region close to your Bucket.
   • Runtime: Python 3.10 (or your preferred runtime).
   • Trigger: choose HTTP trigger if needed or Cloud Pub/Sub for scheduled execution.
   • Authentication: secure with authentication.
   • Write the Code with an inline editor:

   ```python
   from google.cloud import storage
   import requests
   import base64
   import json
   
   # Google Cloud Storage Configuration
   BUCKET_NAME = "<bucket-name>"
   FILE_NAME = "qualys_cm_alerts.json"
   
   # Qualys API Credentials
   QUALYS_USERNAME = "<qualys-username>"
   QUALYS_PASSWORD = "<qualys-password>"
   QUALYS_BASE_URL = "https://<qualys_base_url>"
   
   def fetch_cm_alerts():
       """Fetch alerts from Qualys Continuous Monitoring."""
       auth = base64.b64encode(f"{QUALYS_USERNAME}:{QUALYS_PASSWORD}".encode()).decode()
       headers = {
           "Authorization": f"Basic {auth}",
           "Content-Type": "application/xml"
       }
       payload = """
       <ServiceRequest>
           <filters>
               <Criteria field="alert.date" operator="GREATER">2024-01-01</Criteria>
           </filters>
       </ServiceRequest>
       """
       response = requests.post(f"{QUALYS_BASE_URL}/qps/rest/2.0/search/cm/alert", headers=headers, data=payload)
       response.raise_for_status()
       return response.json()
   
   def upload_to_gcs(data):
       """Upload data to Google Cloud Storage."""
       client = storage.Client()
       bucket = client.get_bucket(BUCKET_NAME)
       blob = bucket.blob(FILE_NAME)
       blob.upload_from_string(json.dumps(data, indent=2), content_type="application/json")
   
   def main(request):
       """Cloud Function entry point."""
       try:
           alerts = fetch_cm_alerts()
           upload_to_gcs(alerts)
           return "Qualys CM alerts uploaded to Cloud Storage successfully!"
       except Exception as e:
           return f"An error occurred: {e}", 500
   ```
   

4. Click Deploy after completing the configuration.

Configure Cloud Scheduler

1. Go to Cloud Scheduler in the Google Cloud console.
2. Click Create Job.

3. Configure the Job:

   • Name: enter a name for your job (for example, trigger-fetch-qualys-cm-alerts).
   • Frequency: use cron syntax to specify the schedule (for example, 0 * * * * to run every hour).
   • Time Zone: set your preferred time zone.
   • Trigger Type: choose HTTP.
   • Trigger URL: Enter the Cloud Function's URL (found in the function details after deployment).
   • Method: Choose POST.

4. Create the job.

Configure a feed in Google SecOps to ingest Qualys Continuous Monitoring logs

1. Go to SIEM Settings > Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed (for example, Qualys Continuous Monitoring Logs).
4. Select Google Cloud Storage as the Source type.
5. Select Qualys Continuous Monitoring as the Log type.
6. Click Next.

7. Specify values for the following input parameters:

   • Storage Bucket URI: the Google Cloud storage bucket source URI..
   • URI is a: select Single file.
   • Source deletion options: select the deletion option according to your preference.
   • Asset namespace: the asset namespace.
   • Ingestion labels: the label to be applied to the events from this feed.

8. Click Next.

9. Review your new feed configuration in the Finalize screen, and then click Submit.

UDM Mapping Table

Changes

2022-08-30

• Newly created parser.