OSF-EIMTC 0.1.61
A Framework for Encrypted Internet and Malicious Traffic Classification.
License: MIT
Keywords: nfstream, pcap, network, deep-learning, extraction

<div align="center">
<img src="/docs/OSF-EIMTC-Logo.png" alt="OSF-EIMTC-Logo" width="500px" >
</div>
<br/><br/>

![PyPi Badge](https://img.shields.io/pypi/v/osf-eimtc.svg)
![PyPI - Downloads](https://img.shields.io/pypi/dm/osf-eimtc)
[<img alt="Ariel University Home" src="https://img.shields.io/badge/Ariel%20University-blue?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAFQAAABUAQMAAAAmpYKCAAAABGdBTUEAALGPC/xhBQAAAAFzUkdCAK7OHOkAAAAJcEhZcwAAEnQAABJ0Ad5mH3gAAAAGUExURQsUK/7//yyPHFQAAADqSURBVCjPbdCxkYRADETRT2FgshmQwmVASpfBTmiEQggyMSj6jEFCKs573u9qYJIkA4Ax3Bgk6QDYQJJOgL36K+kCMFglCeCARZIacFbPkrQB1x3eAaWwWnGMUBqhFJZV+wilEUrh4jRCaYRSuPoZoTRCKVz9jFAKK4WrnxGKsKEIv3yPMBRhQxF++R5hKMKGIvx2H9Hdw909/HYf0d3D5pdlz+4+oruHbzeGf731jxXh21a9ug9Y3CfM7qt6cgvGcGMIb8WEd/i6DVb3Ub24T5jdF0xuVY/hxhDeILxX/7oNftwHfNwnfP4ALPV3tucoteYAAAAASUVORK5CYII=" />](https://www.ariel.ac.il/)
[<img alt="ACIC Home" src="https://img.shields.io/badge/Ariel%20Cyber%20Innovation%20Center%20-blue?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAG8AAABaBAMAAABEXBVNAAAAGFBMVEXj9/kKFTis2uCBv8ZXsLN73eA+Rl4Rpq9f4ewTAAAAB3RSTlMB/UmY4eXFFy+TWQAABOpJREFUWMOtmEFv2zoMxw3lofe0QM56sl97zSQlvg6xDN8XG7kuKbAPUMvQ1x8pybbsyE42PJ261T+R/JOiqCZJdKVSwTqKozwehRBH8Vl8ws/w4z5ZXdft0npd5V62y2sV/HcF/LnCEfD0V3xtt28r4PvyvrDlfs3gji7vufQ7kl7XtMMo2NKeywbdr7d0SdHbigCgTzzMb9vtJ30g+S0exBtdTTKQr/H/dvsRIUWwzuIgBPU+xXJJ+ghSc7f03quwi4CbXrP8HjQnL2xM9RcPgsFmuoQwmoafzPNktyOV6WYgZ5UpaBjNLI0YQKqMbuouWLrTp9RoJJfAVxdg11RzbcCk2VvwFss/guhofacNTyvUZ8jYNP+7G0aomzuDYJKm5mvYPFocpjtUkXQUrGpXwY3pYmkEZyttdZiWTootUXH46R9TXBZAQweQ+WwSVauyrBX8MzNlHPz6QFnfbcqkzQ2srGiErJu6SJIP08TBNhtAKBHDrWElCzgFsgFnc9NUS+B3X3OZFlXLUyBKRZ3DRQrZiHKms+DGgSzXKEqJPlqPparMAqhtIi34UbDclbNyBUgkTxv9GMxAB2tRnfqsCAkFFwcNM60HN6DCiTJYY0KZ6eoFUCBI7IGUqqD3FReCKvj5PIJEBLbofcVpzrIoGFryfWwCnib952x0BISdC9dvyvHTva3dVZDkugLHGPvPdHI8g9aFAaxioIGasZkJDkdny+kBWHFSNc350IQxTi2WVRsFmbEWq3YJbDDGzR2opRYsFYfQ4g+nWg9WrnKSmaq2pYCqAVi4ozdYtODs8tgoxXzlyKC1BZ468OXu1vEFSyZFvg/SqC34PoJE8GmwAUgD0BzwWI1djsj+7vMgfBAFtcQSDMDK1iRNiJUZ7ojzAmhbx9iQM839aXYLz11UHBDtO44JA9iyrBOC9xd+czldoiAUBg0vHQQNtMfCdi2lLuaUB8XKxqRiJw/uR+yRGqYMmIsldATwtf0YQM7S6d2xGcGhK9seQMCEHj+G0g2anMv/UKpZNWk8sBGLdscMxQ/ynxA2nQgvhueLd058QMI9WArqZLE5B0VNojc5jKv2QJqvNNaO/fn/6Tt3GB5TeBtAd0/v76vWXgD9fESUCYQhsmzqWsqL+XEfZJG5SnWiwuk12NrcEqpRWDsSt59rClkMtPlQB2xttS0Zpbr65I6kHYfmnrZ0GI9IfmKXsq7rpqztbef8Jrnhcq6pPVND3SAYzHuK9/1n7msHZUnHEJMcTntZYIFLWP0UjF1gZrKQVsVrn/7MaO3EIYTQUd8c+/pUGqznzTABQjTF/NJKKTbhqUmYAJ2nQzNmjM7n+eur2zAwCWOcvfGu0YF88raCsZaPPYtLa/Bl9TXjXnNgsmW9s+fURojPmZVXCXGvRDiVBbt4R52kZP39iPviCA2C8xRHB7Ss+frLsn9DvPk+jWTJ/SVP1qQJH6YwBHWMQbeqsEqtwfVXOX7x5p8tmhHg7CiAD7n1B5v9ZO8HEbgvXIBPGLQFsvPXkek5sn1o0H1060nHJcf1p+worD8+acEfPcjnUb499cSNPCTDD8m3B0UzfbjzyV8O9s+B6NuOB7vckifX5joKhBHTZ0H3Rwjuud3znM0ckOy4DZx+LkyL4B8ptn/EefIvuJ7c/TEH5Pvfcdg+P1nyP67fYb4noeL/R5wAAAAASUVORK5CYII=" />](https://www.ariel-cyber.co.il/)

**O**pen-**S**ource **F**ramework for **E**ncrypted **I**nternet and **M**alicious **T**raffic **C**lassification.

## Framework publish to PyPi
The framework is been uploaded to the pypi for easy use of the framework.

## Ariel University
This framework was developed by Ariel Universtiy, you may read more about the framework in our paper:
[Open-Source Framework for Encrypted Internet and Malicious Traffic Classification](https://arxiv.org/abs/2206.10144)

## Examples
For usage examples please see the `./examples/` directory for various applications of extracting features from a set of PCAP files or evaluating state-of-the-art deep-learning models.

## Installation
Using pip:
```
python3 -m pip install OSF-EIMTC
```

Uninstall with:
```
pythom3 -m pip uninstall OSF-EIMTC
```

Install tensorflow for deep-learning models:
see the official website: https://www.tensorflow.org/install.
Generally, you can install tensorflow with pip:
```
python3 -m pip install tensorflow
```
But check the offical instructions for more customized options such as for GPU utilization.

For TLS features, TShark is required. While it is possible to install and configure it for Windows usage, the framework is yet to officialy support the system.
```
sudo apt install tshark
```
### Dependencies
- Platform
    - OS: Linux (Tested with WSL Ubuntu 18.04). 
    - Version: Python 3.6.6+ (3.7+ is recommended)

- Python modules
    - NFStream>=6.5.1
    - pandas>=1.1.5
    - numpy>=1.19.5
    - scapy>=2.4.5 (for parsing higher level protocols such as DNS)
    - unittest (for tests)
    - sklearn>=1.1.0

There are some plugins that might require some additional dependancies.
- Optional Python modules
    - pyasn (for ASN info, requires two additional DBs)
    - functools (for caching functions results)
    - click (for CLI)

- For Deep-Learning Models
    - tensorflow>=2.4.0.

- 3rd party tools (that need to be installed)
    - TShark 3.4.8+ (for TLS features). Try install with `sudo apt update` and then `sudo apt install tshark`, if you cannot get the 3.0+ version try to install wireshark instead via `sudo apt install wireshark`.

- 3rd party data files (provided in `./tools` directory)
    - pyasn.db (ASN mapping database) - updated one can be downloaded via https://github.com/hadiasghari/pyasn.
    - ip2asn.tsv (ASN context info database) - for updated data download from https://iptoasn.com/.



## Run Tests
```
python3 -m unittest discover EIMTC
```

## Project Structure
Under `./src`
```
└── EIMTC/             # Project's root directory.
    ├── datasets/      # Links and information for PCAP datasets.
    ├── metrics/       # Evaluation metrics for models.
    ├── models/        # State-of-the-art DL models.
    ├── preprocessing/ 
    ├── selection/ 
    ├── plugins/       # NFStream plugins.
    ├── stats/         
    ├── temp/          # Temporary files generated by the system.
    ├── tests/         # System tests.
    │   └── pcaps/     # PCAP files for tests.
    ├── thrid_party/   # 3rd-party tools used by the system.
    └── tools/         # 3rd-party tools used by the system.
```

## Citation
```
article{BADER2024271,
title = {OSF-EIMTC: An open-source framework for standardized encrypted internet traffic classification},
author = {Ofek Bader and Adi Lichy and Amit Dvir and Ran Dubin and Chen Hajaj},
journal = {Computer Communications},
volume = {213},
pages = {271-284},
year = {2024}
}
```