specter-kali 1.1.0
Specter KALI Linux scanning tool
License: MIT
Requires Python: >=3.5
Keywords: Linux,Security,Python

# Specter

[![pypi](https://pypip.in/v/specter-kali/badge.svg)](https://pypi.org/project/specter-kali/)
![Specter PR Workflow](https://github.com/doziermt/specter/workflows/Specter%20PR%20Workflow/badge.svg)
![Specter Deploy Workflow](https://github.com/doziermt/specter/workflows/Specter%20Deploy%20Workflow/badge.svg)

**NOTE:**

> Currenlty Tested on KALI 2020.4

Specter is a [KALI Linux](https://www.kali.org/) CLI tool for identifying network vulnerabilities on a given host using a variety of open-source scanning tools.

#### Contents

- [Prerequisites](#prerequisites)
- [Getting Started](#getting-started)
- [Configuration](#configuration)
- [Files](#files)
- [Supported Commands](#supported-commands)
- [Specter Work Directory](#specter-work-directory)
- [Developer Guide](#developer-guide)
- [Release Guide](#release-guide)

## Prerequisites

Specter should ideally be run on [KALI Linux](https://www.kali.org/). Specter uses the following **required**
third-party scanning tools (that come pre-installed on KALI):

- [Nmap](https://nmap.org/) for generating a "clean target list"
- [Masscan](https://github.com/robertdavidgraham/masscan) for performing port scan operations efficiently
- [Eyewitness](https://github.com/FortyNorthSecurity/EyeWitness) for performing a web security scan

If applicable, please follow the links above for information on installing each tool.

## Getting Started

Ensure that `pip3` is installed.

### Installing Specter

Execute:

> pip3 install specter-kali

### Upgrading Specter

Execute:

> pip3 install specter-kali --upgrade

### Running Specter

The following commands must be executed in order.

* Initialize the [Specter Work Directory](#specter-work-directory):

  > specter init

* Invoke `clean_list` which uses `nmap` to generate a list of IPs to scan:

  > specter clean_list

* Invole `xml_scan` which uses `masscan` to scan all relevant IP addresses:

  ```
  # Without banners.
  specter xml_scan
  # With banners.
  specter xml_scan --banners
  ```

* Invoke `web_scan` which uses `eyewitness` to perform further scanning:

  >  specter web_scan

For a list of additional commands to run, please execute `specter --help`.

## Configuration

After running `specter init`, all configuration options are documented inside the `specter_workdir/settings.toml`.

Ensure that this configuration file uses the correct `interface`, `ports`, etc., in order to generate meaningful results.

### Sample Config File

Specter uses the [Dynaconf](https://github.com/rochacbruno/dynaconf) Python library for its configuration
parsing and validation library. This project's configuration file uses the [TOML](https://github.com/toml-lang/toml) markup language.

Within the source code, a sample configuration file lives at path `samples/settings.sample.toml`. This sample configuration
file is copied to the [Specter Work Directory](#specter-work-directory) after running `specter init`.

## Files

After running all Specter commands, the [Specter Work Directory](#specter-work-directory) will contain many different files,
including configuration and output files. The output files include intermediary files generated between operations (e.g.
`output/xml_clean_target_list.txt` and `output/web_clean_target_list.txt`), while others are generated by `masscan` or
`eyewitness` applications.

These files can be visualized by running `specter tree`:

```
specter_workdir/
├── nmap_exclude_list.txt
├── nmap_target_list.txt
├── output/
│   └── test_2020-08-08_103235/
│       ├── hosts/
│       │   ├── 10.0.0.1.txt
│       │   ├── 10.0.0.13.txt
│       │   ├── 10.0.0.2.txt
│       │   ├── 10.0.0.3.txt
│       │   └── 10.0.0.4.txt
│       ├── ports/
│       │   ├── 10001.txt
│       │   ├── 135.txt
│       │   ├── 139.txt
│       │   ├── 20005.txt
│       │   ├── 21.txt
│       │   ├── 445.txt
│       │   ├── 5000.txt
│       │   ├── 53.txt
│       │   ├── 5357.txt
│       │   ├── 548.txt
│       │   ├── 631.txt
│       │   ├── 80.txt
│       │   ├── 8008.txt
│       │   ├── 8009.txt
│       │   ├── 8200.txt
│       │   ├── 8443.txt
│       │   ├── 9000.txt
│       │   └── 9080.txt
│       ├── web_clean_target_list.txt
│       ├── web_reports/
│       │   └── eyewitness/
│       ├── xml/
│       │   └── masscan.xml
│       └── xml_clean_target_list.txt
└── settings.toml
```

**NOTE**:

> Your Specter work directory may contain different host and IP files depending on your network configuration and scan results.

### Configuration Files

* `settings.toml`: Specter configuration file containing all operational settings.
* `nmap_target_list.txt`: The name of the file containing target list of IPs and IP ranges to scan.
* `nmap_exclude_list.txt`: The name of the file containing the list of excluded IPs that will **NOT** be scanned.

See the [Configuration Section](#configuration) for more information.

### Output Files

* `init`
  * `settings.toml`: Specter configuration file containing all operational settings
  * `nmap_target_list.txt`: The name of the file containing target list of IPs and IP ranges to scan.
  * `nmap_exclude_list.txt`: The name of the file containing the list of excluded IPs that will **NOT** be scanned.

* `clean_list`
  * `output/xml_clean_target_list.txt`: creates a list of IPs or IP ranges for `xml_scan` to scan with excluded IPs removed

* `xml_scan`
  * `output/web_clean_target_list.txt`: creates a list of IPs with ports `80`, `443`, `8000`, `8080`, `8443` open from `masscan.xml`
  * `output/xml/masscan.xml`: `masscan` output file
  * `output/ports/`: `<PORT>`.txt files with lists of IPs with port found open from masscan.xml
  * `output/hosts/`: `<IP>`.txt files with lists of port data found open from masscan.xml

* `web_scan`
  * `ouput/web_reports/Eyewitness`: `eyewitness` output file

## Supported Commands

* `init`: Initializes the [Specter Work Directory](#specter-work-directory). This command should always be exected first.
* `clean_list`: Generates a "clean target list" file, which enumerates the IP addresses to scan using `masscan`.
* `xml_scan`:
  * Scans the IPs from the `xml_clean_target_list.txt` file with `masscan`.
  * Creates `masscan.xml` in `ouput/xml`, `web_clean_target.list.txt` for scanning with `eyewitness`, and `output/hosts` and `output/ports` directory banner data.
* `web_scan`: Scans the "web clean target list" with `eyewitness` and writes the default output to the [Specter Work Directory](#specter-work-directory).
* `tree`: Utility command for visualizing the [Specter Work Directory](#specter-work-directory).

## Specter Work Directory

The Specter Work Directory contains all configuration required by Specter to correctly invoke `nmap`, `masscan`, and
`eyewitness` CLI commands via the Python [subprocess](https://docs.python.org/3/library/subprocess.html) library.
It also contains all intermediary and 3rd-party application output files. "Intermediary" files are those generated by
Specter for feeding "clean target list files" from one application to the subsequent one. The "3rd-party application"
files include those generated by `nmap`, `masscan` and `eyewitness`.

### Snapshotting

Snapshotting of output files allows Specter to retain a history of all output files from previously executed commands.

The Specter Work Directory uses the `[general].sitename` configuration setting along with an auto-generated timestamp to
"snapshot" the output directory. Each time `specter clean_list` is run, a new "snapshot" within the output directory is
created.

**NOTE**:

> **This snapshot is only created for the `clean_list` command.** Running `xml_scan` or `web_scan` will reference the output files
> located within the "snapshot" created by `clean_list`.

For example, running `specter init` followed by `specter clean_list` will generate the following Work Directory structure:

```
specter_workdir/
├── nmap_exclude_list.txt
├── nmap_target_list.txt
├── output/
│   └── test_2020-10-28_111653/
│       ├── hosts/
│       ├── ports/
│       ├── web_reports/
│       │   └── eyewitness/
│       ├── xml/
│       │   └── masscan.xml
│       └── xml_clean_target_list.txt
└── settings.toml
```

In the example above, `test_2020-10-28_111653/` is the "snapshot" generated by the `clean_list` command. When the
`xml_scan` or `web_scan` commands are run, the **same directory snapshot** is used.

To create a new snapsphot, simply re-run `clean_list` which will produce an output structure similar to the following:

```
specter_workdir/
├── nmap_exclude_list.txt
├── nmap_target_list.txt
├── output/
│   ├── test_2020-10-28_111653/
│   │   ├── hosts/
│   │   ├── ports/
│   │   ├── web_reports/
│   │   │   └── eyewitness/
│   │   ├── xml/
│   │   │   └── masscan.xml
│   │   └── xml_clean_target_list.txt
│   └── test_2020-10-28_112148/
│       ├── hosts/
│       ├── ports/
│       ├── web_reports/
│       │   └── eyewitness/
│       ├── xml/
│       └── xml_clean_target_list.txt
└── settings.toml
```

In the example above, 2 "snapshots" can be seen:

* `test_2020-10-28_111653/`
* `test_2020-10-28_112148/`

## Developer Guide

### Dev Environment - Environment Setup

1. Install Python 3: https://www.python.org/downloads/
2. Install `pip3`: https://pip.pypa.io/en/stable/installing/
3. Install `tox` using `pip3` by running `pip3 install tox` in your terminal

### Dev Environment - Installing Specter CLI

To run any of the [Supported Commands](#supported-commands), first install the specter CLI using `pip3`:

```
cd specter/
pip3 install -r requirements.txt
pip3 install -e .
```

Afterward, see the [Running Specter](#running-specter) for a list of available commands to run.

### Dev Environment - Dev Tools

This project makes use of [tox](https://tox.readthedocs.io/en/latest/) to facilitate testing for developers.
The following tox commands can be used:

* `tox -e fmt`: Formats the Python project code using the Google linter tool, [yapf](https://github.com/google/yapf)
* `tox -e lint`: Checks that the Python project code passes `yapf` linter checks

## Release Guide

When ready to create a new release of `specter`, change to the `master` branch:

> git checkout master

And then create the new release using either a `patch`, `minor`, or `major` version update:

> tox -e package-create -- [patch|minor|major]

**NOTE**:

> The task will default to `patch` if no argument is provided.

After running the command above, `git-release-tag` will automatically perform the following steps for you:

1. Update the `__version__` to `X.Y.Z` in `specter/__init__.py`
2. Create a `vX.Y.Z` tagged commit for the release

Where `X.Y.Z` denotes the [semantic release](https://github.com/semantic-release/semantic-release) corresponding to the new release.

Afterward, push the tagged commit to the repository:

> git push origin `vX.Y.Z`

Ultimately, this will trigger [GitHub Actions](#github-actions) causing it to perform its `deploy` workflow.

### Example

Given: `1.0.0` as the current `specter` version

Example: Create a new minor release, `1.1.0`

Steps:

1. `git checkout master`
2. `tox -e package-create -- minor`
3. `git push origin v1.1.0`

### GitHub Actions

GitHub Actions will automatically execute the following `tox` commands to verify and publish the new release:

* `tox -e package-check`: Checks whether your distribution’s long description will render correctly on PyPI.
* `tox -e package-test-upload`: Uploads the new release to https://test.pypi.org/
* `tox -e package-upload`: Uploads the new release to https://pypi.org/

**WARNING**:

> Do **not** run these tasks locally. They are reserved for CI/CD only.