RePIDS: a Multi Tier Real-Time Payload-Based Intrusion Detection System Aruna Jamdagnia,b, Zhiyuan Tana,b, Xiangjian Hea,*, Priyadarsi Nandaa and Ren Ping Liub aCentre for Innovation in IT Services and Applications (iNEXT), University of Technology, Sydney, Australia bCSIRO, ICT Centre, Australia Arunaj@it.uts.edu.au {Zhiyuan.Tan, xiangjian.he, Priyadarsi.Nanda}@uts.edu.au ren.liu@csiro.au ABSTRACT Intrusion Detection System (IDS) deals with huge amount of network traffic and uses large feature set to discriminate normal pattern and intrusive pattern. However, most of existing systems lack the ability to process data for real-time anomaly detection. In this paper, we propose a 3-Tier Iterative Feature Selection Engine (IFSEng) for feature subspace selection. Principal Component Analysis (PCA) technique is used for the pre-processing of data. Mahalanobis Distance Map (MDM) is used to discover hidden correlations between the features and between the packets. We also propose a novel Real-time Payload-based Intrusion Detection System (RePIDS) that integrates a IFSEng and the MDM approach. Mahalanobis Distance (MD) dissimilarity criterion is used to classify each packet as either a normal or an attack packet. The effectiveness of the proposed RePIDS is evaluated using DARPA 99 dataset and Georgia Institute of Technology attack dataset. The traffic for Web-based application is considered for validating our model. F-Value, a criterion, is used to evaluate the detection performance of RePIDS. Experimental results show that RePIDS achieves better performance (a high F-Value, 0.9958 for DARPA dataset and 0.976 for Georgia Institute of Technology attack dataset respectively, with only 0.85% false alarm rate) and lower computational complexity when compared against two state-of-the-art payload-based intrusion detection systems. Additionally, it has 1.3 time higher throughput in comparison with real scenario of medium size enterprise network. Keywords: Intrusion detection, Data pre-processing, Principal Component Analysis, Mahalanobis Distance Map, Principal Components, Iterative feature selection Tier
First, cumulative energy criterion is applied for the selection, in which we consider 93 percent of cumulative energy level for (6). An equal to 7 is obtained, which means that the first 7 principal components is selected as the best subspace to represent the data by cumulative energy criterion. Then, we use scree test to draw scree plot as a variance captured by a given principal component and to select another set of principal components. Fig. 2(a) shows full scree plot, where we use k (k = 256 in our case) principal components (X-axis) of a particular dataset and the corresponding variances, namely eigenvalues (Y-axis) to draw a scree plot, and the PCs are sorted in descending order with respect to the values of the corresponding variances. In Fig. 2(a), we look for an ‘elbow’, a flattening out of the curve. To provide better vision, we magnify the scree plot and show the first 25 principal components in Fig. 2(b). It can be observed from Fig. 2(b) that there is a sharp decrease of variance in the front part of the plot and then it starts flattening out after the principal component. In Fig. 2(b), we can observe ‘elbow’ somewhere in the range from 6 to 9 principal components and the first = 6 principal components are able to capture about 92 percent of the variance. After the -th point, the remaining (k - k2) principal components capture only around 8 percent of the total variance and are ignored. Full scree plot (b) Enlarged scree plot with first 25 eigenvectors Fig. 2. Scree test plot We use= 6 as a dominant principal components in our case. However, from Fig. 2(b), we have observed a range of principal components from 6 to 9, and it is not very clear that what is the most appropriate value of To overcome this ambiguity, we use parallel analysis criterion as described in the following and to verify the selection of .