neopitool 0.5
Maintained NeoPI

Maintained fork of https://github.com/ByteInternet/NeoPI

# What is NeoPI?

NeoPI is a Python script that uses a variety of statistical methods to detect obfuscated and encrypted content within text/script files. The intended purpose of NeoPI is to aid in the detection of hidden web shell code. The development focus of NeoPI was creating a tool that could be used in conjunction with other established detection methods such as Linux Malware Detect or traditional signature/keyword based searches.

NeoPI recursively scans through the file system from a base directory and will rank files based on the results of  a number of tests. It also presents a “general” score derived from file rankings within the individual tests.

# Requirements
NeoPI is platform independent and can be run on any system with Python 2.6 or greater installed installed. The user running the script should have read access to all of the files that will be scanned.

# How to use it
NeoPI is platform independent and will run on both Linux and Windows.  To start using NeoPI first checkout the code from our github repository

Install with

	pip install neopitool

Then just run with

	neopi


The small NeoPI script is now in your local directory.  We are going to go though a few examples on Linux and then switch over to Windows.  

Let’s run `neopi` with the -h flag to see the options.  

	$ neopi -h
	Usage: neopi.py [options] <start directory> <OPTIONAL: filename regex>

	Options:
	  --version             show program's version number and exit
	  -h, --help            show this help message and exit
	  -C FILECSV, --csv=FILECSV
							generate CSV outfile
	  -a, --all             Run all tests [Entropy, Longest Word, Compression
	  -e, --entropy         Run entropy Test
	  -l, --longestword     Run longest word test
	  -c, --ic              Run IC test
	  -A, --auto            Run auto file extension tests

Let’s break down the options into greater detail.

	-C FILECSV, --csv=FILECSV
This generates a CSV output file containing the results of the scan.  

	-a, --all
This runs all tests including entropy, longest word, and index of coincidence.  In general, we suggest running all tests to build the most comprehensive list of possible web shells.

	-e, --entropy
This flag can be set to run only the entropy test.  

	-l, --longestword
This flag can be set to run only the longest word test.  

	-c, --ic
This flag can be set to run only the Index of Coincidence test.  

	-A, --auto 
This flag runs an auto generated regular expression that contains many common web application file extensions.    This list is by no means comprehensive but does include a good ‘best effort’ scan if you are unsure of what web application languages your server is running.  The current list of  extensions are included below:

	valid_regex = re.compile('\.php|\.asp|\.aspx|\.sh|\.bash|\.zsh|\.csh|\.tsch|\.pl|\.py|\.txt|\.cgi|\.cfm')

Now that we are familiar with the flags and we have downloaded a copy of the script from GIT, let’s go head and run it on a web server we think may be infected with obfuscated web shells.    

	$ sudo neopi -C scan1.csv -a -A /var/www/
	
The resulst of the scan we be displayed to console as well as written to 'scan1.csv'.  Here is an example of the scan results:

	$ neopi -a -A /var/www/html/

	[[ Average IC for Search ]]
	0.0372337579606

	[[ Top 10 IC files ]]
	  0.0156    /var/www/html/webmedia/shell3.php
	  0.0178    /var/www/html/phpadmin/phpMyAdmin-3.3.8-all-languages/lang/chinese_simplified-utf-8.inc.php
	  0.0184    /var/www/html/wordpress/wordpress/wp-admin/weevely.php
	  0.0217    /var/www/html/joomla/templates/system/index.php
	  0.0217    /var/www/html/joomla/administrator/templates/system/index.php
	  0.0225    /var/www/html/wordpress/wordpress/wp-admin/js/revisions-js.php
	  0.0229    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Base/Mail/language/phpmailer.lang-ch.php
	  0.0239    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Base/Mail/language/phpmailer.lang-zh.php
	  0.0240    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Base/Mail/language/phpmailer.lang-zh_cn.php
	  0.0248    /var/www/html/phpadmin/shell2.php

	[[ Top 10 entropic files ]]
	  6.3978    /var/www/html/phpadmin/phpMyAdmin-3.3.8-all-languages/lang/chinese_simplified-utf-8.inc.php
	  6.0651    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Base/Mail/language/phpmailer.lang-ch.php
	  6.0061    /var/www/html/webmedia/shell3.php
	  5.9870    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Base/Mail/language/phpmailer.lang-zh.php
	  5.9797    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Base/Mail/language/phpmailer.lang-zh_cn.php
	  5.9245    /var/www/html/phpadmin/shell2.php
	  5.8895    /var/www/html/wordpress/wordpress/wp-admin/js/revisions-js.php
	  5.8580    /var/www/html/phpadmin/phpMyAdmin-3.3.8-all-languages/lang/japanese-utf-8.inc.php
	  5.8400    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Base/Mail/language/phpmailer.lang-ja.php
	  5.7602    /var/www/html/wordpress/wordpress/wp-admin/weevely.php

	[[ Top 10 longest word files ]]
	  111571    /var/www/html/webmedia/shell3.php
		2510    /var/www/html/webmedia/htdocs/templates/main.tpl.php
		1312    /var/www/html/joomla/shell.php
		 728    /var/www/html/wordpress/wordpress/wp-admin/js/revisions-js.php
		 536    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Libs/QuickForm/3.2.11/HTML/QuickForm/Rule/Email.php
		 522    /var/www/html/wordpress/wordpress/wp-includes/functions.php
		 516    /var/www/html/phpadmin/phpMyAdmin-3.3.8-all-languages/libraries/tcpdf/tcpdf.php
		 516    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Libs/PHPExcel/lib/PHPExcel/Shared/PDF/tcpdf.php
		 516    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Libs/TCPDF/tcpdf4/tcpdf.php
		 516    /var/www/html/joomla/libraries/tcpdf/tcpdf.php

	[[ Highest Rank Files Based on test results ]]
		 83%    /var/www/html/webmedia/shell3.php
		 56%    /var/www/html/phpadmin/phpMyAdmin-3.3.8-all-languages/lang/chinese_simplified-utf-8.inc.php
		 43%    /var/www/html/wordpress/wordpress/wp-admin/js/revisions-js.php
		 36%    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Base/Mail/language/phpmailer.lang-ch.php
		 26%    /var/www/html/webmedia/htdocs/templates/main.tpl.php
		 26%    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Base/Mail/language/phpmailer.lang-zh.php
		 23%    /var/www/html/wordpress/wordpress/wp-admin/weevely.php
		 23%    /var/www/html/joomla/shell.php
		 20%    /var/www/html/joomla/templates/system/index.php
		 20%    /var/www/html/epesiBIM/epesi-1.1.3-rev7318/modules/Base/Mail/language/phpmailer.lang-zh_cn.php

We highly recommend that as a baseline, any file that is displayed in the Highest Rank Files list be investigated at a minimum.  We also recommend investigating any files that show up in any of the tests listed above, as some methods are more effective at detecting certain shells than others.  

## Animal Shell
animal_shell_encoder.php and animal_shell_poc.php are two Proof-of-Concept-type examples scripts to implement an encoding that "should" evade many of the statistical tests NeoPI performs. They are poorly commented and the decoder large such that they are impractical.