Execution: Malicious Python executed

This document describes a threat finding type in Security Command Center. Threat findings are generated by threat detectors when they detect a potential threat in your cloud resources. For a full list of available threat findings, see Threat findings index.

Finding description

A machine learning model identified executed Python code as malicious. Attackers can use Python to transfer tools and execute commands without binaries. Ensuring that your containers are immutable is an important best practice. Using scripts to transfer tools can mimic the attacker technique of ingress tool transfer and result in unwanted detections.

To respond to this finding, do the following:

Step 1: Review finding details

  1. Open an Execution: Malicious Python executed finding as directed in Reviewing findings. The details panel for the finding opens to the Summary tab.

  2. On the Summary tab, review the information in the following sections:

    • What was detected, especially the following fields:
      • Program binary: details about the interpreter that invoked the script.
      • Script: absolute path of the name of the script on disk; this attribute only appears for scripts written to disk, not for literal script execution—for example, python3 -c.
      • Arguments: the arguments provided when invoking the script.
    • Affected resource, especially the following fields:
      • Resource full name: the full resource name of the cluster, including the project number, location, and cluster name.
    • Related links, especially the following fields:
      • VirusTotal indicator: link to the VirusTotal analysis page.

  3. In the detail view of the finding, click the JSON tab.

  4. In the JSON, note the following fields.

    • finding:
      • processes:
      • script:
        • contents: contents of the executed script, which might be truncated for performance reasons; this can aid in your investigation
        • sha256: the SHA-256 hash of script.contents
    • resource:
      • project_display_name: the name of the project that contains the asset.
    • sourceProperties:
      • Pod_Namespace: the name of the Pod's Kubernetes namespace.
      • Pod_Name: the name of the GKE Pod.
      • Container_Name: the name of the affected container.
      • Container_Image_Uri: the name of the container image being executed.
      • VM_Instance_Name: the name of the GKE node where the Pod executed.

  5. Identify other findings that occurred at a similar time for this container. For instance, if the script drops a binary, check for findings related to the binary.

Step 2: Review cluster and node

  1. In the Google Cloud console, go to the Kubernetes clusters page.

    Go to Kubernetes clusters

  2. On the Google Cloud console toolbar, select the project listed in resource.project_display_name, if necessary.

  3. Select the cluster listed on the Resource full name row in the Summary tab of the finding details. Note any metadata about the cluster and its owner.

  4. Click the Nodes tab. Select the node listed in VM_Instance_Name.

  5. Click the Details tab and note the container.googleapis.com/instance_id annotation.

Step 3: Review Pod

  1. In the Google Cloud console, go to the Kubernetes Workloads page.

    Go to Kubernetes Workloads

  2. On the Google Cloud console toolbar, select the project listed in resource.project_display_name, if necessary.

  3. Filter on the cluster listed in resource.name and the Pod namespace listed in Pod_Namespace, if necessary.

  4. Select the Pod listed in Pod_Name. Note any metadata about the Pod and its owner.

Step 4: Check logs

  1. In the Google Cloud console, go to Logs Explorer.

    Go to Logs Explorer

  2. On the Google Cloud console toolbar, select the project listed in resource.project_display_name, if necessary.

  3. Set Select time range to the period of interest.

  4. On the page that loads, do the following:

    1. Find Pod logs for Pod_Name by using the following filter:
      • resource.type="k8s_container"
      • resource.labels.project_id="resource.project_display_name"
      • resource.labels.location="location"
      • resource.labels.cluster_name="cluster_name"
      • resource.labels.namespace_name="Pod_Namespace"
      • resource.labels.pod_name="Pod_Name"
    2. Find cluster audit logs by using the following filter:
      • logName="projects/resource.project_display_name/logs/cloudaudit.googleapis.com%2Factivity"
      • resource.type="k8s_cluster"
      • resource.labels.project_id="resource.project_display_name"
      • resource.labels.location="location"
      • resource.labels.cluster_name="cluster_name"
      • Pod_Name
    3. Find GKE node console logs by using the following filter:
      • resource.type="gce_instance"
      • resource.labels.instance_id="instance_id"

Step 5: Investigate running container

If the container is still running, it might be possible to investigate the container environment directly.

  1. In the Google Cloud console, go to the Kubernetes clusters page.

    Go to Kubernetes clusters

  2. Click the name of the cluster shown in resource.labels.cluster_name.

  3. On the Clusters page, click Connect, and then click Run in Cloud Shell.

    Cloud Shell launches and adds commands for the cluster in the terminal.

  4. Press Enter and, if the Authorize Cloud Shell dialog appears, click Authorize.

  5. Connect to the container environment by running the following command:

      kubectl exec --namespace=Pod_Namespace -ti Pod_Name -c Container_Name -- /bin/sh

    This command requires the container to have a shell installed at /bin/sh.

Step 6: Research attack and response methods

  1. Review MITRE ATT&CK framework entries for this finding type: Command and Scripting Interpreter, Ingress Tool Transfer.
  2. Check the SHA-256 hash value for the binary flagged as malicious on VirusTotal by clicking the link in VirusTotal indicator. VirusTotal is an Alphabet-owned service that provides context on potentially malicious files, URLs, domains, and IP addresses.
  3. To develop a response plan, combine your investigation results with the MITRE research and VirusTotal analysis.

Step 7: Implement your response

The following response plan might be appropriate for this finding, but might also impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • If Python was making intended changes to the container, rebuild the container image such that no changes are needed. This way, the container can be immutable.
  • Otherwise, contact the owner of the project with the compromised container.
  • Stop or delete the compromised container and replace it with a new container.

What's next