Benchmarking datasets for Anomaly-based Network Intrusion Detection: KDD CUP 99 alternatives

• Published in: 2018 IEEE 3rd International Conference on Computing, Communication and Security (ICCCS) pp. 1 - 8
• Main Authors: Divekar, Abhishek, Parekh, Meet, Savla, Vaibhav, Mishra, Rudra, Shirole, Mahesh
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01-10-2018
• Subjects: Benchmark testing; Benchmarking; Conferences; Feature extraction; KDD-99; Machine learning; Network Intrusion Detection; NSL-KDD; Probes; Security; SMOTE; Training; UNSW-NB15
• DOI: 10.1109/CCCS.2018.8586840

Machine Learning has been steadily gaining traction for its use in Anomaly-based Network Intrusion Detection Systems (A-NIDS). Research into this domain is frequently performed using the KDD CUP 99 dataset as a benchmark. Several studies question its usability while constructing a contemporary NIDS, due to the skewed response distribution, non-stationarity, and failure to incorporate modern attacks. In this paper, we compare the performance for KDD-99 alternatives when trained using classification models commonly found in literature: Neural Network, Support Vector Machine, Decision Tree, Random Forest, Naive Bayes and K-Means. Applying the SMOTE oversampling technique and random undersampling, we create a balanced version of NSL-KDD and prove that skewed target classes in KDD-99 and NSL-KDD hamper the efficacy of classifiers on minority classes (U2R and R2L), leading to possible security risks. We explore UNSW-NB15, a modern substitute to KDD-99 with greater uniformity of pattern distribution. We benchmark this dataset before and after SMOTE oversampling to observe the effect on minority performance. Our results indicate that classifiers trained on UNSW-NB15 match or better the Weighted F1-Score of those trained on NSL-KDD and KDD-99 in the binary case, thus advocating UNSW-NB15 as a modern substitute to these datasets.