CECILIA : Enhancing CSIRT Effectiveness with Transformer-Based Cyber Incident Classification (2024)

Abstract: This paper introduces an approach to improv ing incident response times by applying various Artificial Intelligence (AI) classification algorithms based on transformers to analyze the efficacy of these models in categorizing cyber incidents. As a first contribution, we developed a cyber incident dataset, CECILIA-10C-900, collecting cyber incident reports from six qualified web sources. The contribution of creating a dataset on cyber incident detection is remarkable due to the scarcity of such datasets. Each incident has been tagged by hand according to the cyber incident taxonomy defined by the CERT (Computer Emergency Response Team) of the National Institute of Cybersecurity (INCIBE). This dataset is highly unbalanced, so we decided to unify the four least represented classes under the label “others”, leaving a dataset with six categories (CECILIA-6C-900). With these reliable datasets, we performed a comparison of the best algorithms specifically for the cyber incident classification problem, evaluating eight different metrics on two conventional classifiers and six other transformer-based classifiers. Our study highlights the importance of having a rapid classification mechanism for CSIRTs (Computer Security Incident Response Teams) and showcases the potential of machine learning algorithms to improve cyber defense mechanisms. The findings from our analysis provide valuable insights into the strengths and limitations of different classification techniques. It can be used in future work on cyber incident response strategies

## introduction there is a steady increase in cyber attacks worldwide, showing a clear need for better incident response methods. for example, in 2023, x-force recorded the highest number of incidents in europe in the last years, with an increase of 31% compared to 2022 @xcite . csirts need to enhance their capacities to manage a growing number of cyber incidents, especially in the first step of the process: classifying the reported incidents. a good and fast classification makes it possible to follow each incident to the appropriate expert group and directly impacts improving the csirt response times. the traditional automatic classification approach is based on incident reporting standardization. still, it is difficult to achieve cyber incident reporting harmonization, that is, aligning different standards to work together more effectively without losing their individual characteristics @xcite . therefore, multiple standards represent reporting information in diverse formats, making the task of classification difficult. to solve this problem we will work on classifying cyber incidents reported from various sources and without any prior standardization criteria using nlp-based classification techniques in general and transformer classification models in particular, having not found any study that applies transformers to the classification of cyber incidents. the obtained results may be helpful for future work in ai-assisted cyber incident classification processes. this paper introduces cecilia (cyberincidents classified incibe taxonomy), a cyber incident dataset created using different cyber incident reports collected from six selected web sources and manual tagging according to incibe taxonomy foot_0 . we present two versions, cecilia-6c-900 and cecilia-10c-900, where cyber incidents are classified into six and ten categories, respectively. after that, we compute the baseline results for two traditional and six transformer-based approaches using cecilia in the task of cyber incident classification. the rest of the paper is divided as follows: sec-tion 2 analyzes the literature about incident classification using ai, cyber incident datasets, and multilabel classification with transformers. section 3 describes our cecilia-10c-900 dataset, and in section 4, we apply different transformer-based algorithms to this dataset and discuss the results achieved. we also introduce cecilia-6c-900 to avoid unbalancing problems and discuss again the new results obtained with this new dataset. in section 5, we present our conclusions and future work. ## related work depending on the nature of the source, there are different approaches for ai-assisted cyber incident classification. @xcite established a cognitive security model called notas-mh, considering several sources of information, such as those generated by humans, signals from a computer or network equipment, open-source information, sensing instruments, and geospatial systems. @xcite presented discover, an algorithm to predict cyber threats in online discussions using nlp. to test it, they used their own manually curated dataset of security warnings from experts' tweets, security blogs, and dark web forums, obtaining a precision of 84% on tweets, 59% on blogs, and 81% on average. another possible source is osint (open source intelligence), which was used by @xcite , classifying incidents according to their risk with a parameter called "relevancy score". they made this process in four phases: source identification, feature selection, score definition, and model training. in model training, they used five regression algorithms: an support vector machine (svm) regressor, a random forest regressor, a gradient boosting tree regression, an extra trees regressor, and a multi-layer perceptron, and applied them in a dataset with tweets and twitter profiles chosen in a survey with security experts. other approach is the use of standardization for incident reporting. in this way, posea et al. (2022) proposed a common european taxonomy for incident handling and reporting and colome et al. (2019) proposed to work with incident information in incident object description exchange format (r. danyliw (cert), j. meijer (uninet), 2007) format to provide some resolution guidelines using case-based reasoning methods in their dataset with 259 different incidents collected from the se-curity division of a commercial data center. abbiati et al. (2020) merged three different datasets from 2005 to 2018 derived from three websites: prc (privacy rights clearinghouse), which maintains a collection of data-breach records foot_1 , itrc (the identity theft resource center) provides a collection of data breaches on a yearly basis foot_2 and bli (the gemalto data breach level index) containing datasets of publicly disclosed data breaches foot_3 . d'ambrosio et al. (2023) proposed the use of this dataset as future work in risk management using bayesian decision methods, and rafaiani et al. (2023) proposed the cyber risk assessment method that combined probabilistic methods and svm and tested it with this and other two datasets ((upguard, 2023), (ransomfeed, 2023) ). transformers are an excellent option for nlp classification problems, specifically in cases with multiple output classes (multiclass classification), however, to date, no studies have been found on the classification of cyber incidents using transformer models. therefore, we will approach this problem using a generic multiclass classification perspective. in this field, @xcite reviewed the entire process of state-of-the-art text classification models, collecting the benefits and limitations of each model. in the case of transformers, they highlighted the advantage of attention in long sentences, but on the other hand, they are computer-intensive. @xcite presented a survey on text classification with different datasets, types of classification (sentiment analysis (sa), news classification (nc), topic labeling (tl), question answering, natural language inference (nli), multi-label (ml) and others) and metrics for evaluation, finding that the best results for all the datasets were obtained for pre-trained-transformer-based models like bert, roberta, and xlnet. furthermore, @xcite made a survey of text classification for different tasks (sa, tl, nc, qa, nli, named entity recognition and syntactic parsing, discussing the preprocessing, representation, and testing of seven algorithms (naive bayes, linear svm, fasttext classifier, bilstm, xml-cnn, bert and xlm-r) with enwiki-100 and rcv1-57 datasets and found that best results there are several works on ia applied to the classification of cyber incidents but none of them deals specifically with the problem of csirts. there are two ways of working: the standardization of reports, which has the disadvantage that the report must be carried out by specialized personnel, and on the other hand the use of nlp techniques. in this case, traditional classifiers are applied and the scarcity of datasets with cyber incident reports is shown. the novelty of the present study lies in the use of transformers for the classification of cyber incidents, because, to the best of our knowledge, no similar approach exists. to enable a comprehensive comparison with different types of transformers, it was also necessary to create a reliable dataset. this dataset has been labelled according to the in-cibe taxonomy, which is based on the taxonomy of enisa, the european union agency for cybersecurity (security and information, 2018). ## cecilia datasets cecilia datasets comprise 923 cyber incident reports collected from six selected sources and then manually curated and classified using incibe cyber incident taxonomy provided for incident reporting (instituto nacional de ciberseguridad, 2020). after conducting a search for potential websites containing cyber incident reports, we prioritized sources that provided comprehensive compilations of cyber incidents in pdf or csv formats, each incorporating unique classification systems. we selected a set of six urls based on the highest quality of their reports and the prestige of their institution. urls selected are shown in a cybersecurity expert and a labelling assistant with mutual supervision and consensus in difficultto-label samples have done the labelling process. explanations and examples provided by incibe foot_4 were used as criteria to perform the labeling. in-cibe taxonomy divides cyber incidents into 10 categories and 38 subcategories the main ten categories are the ones reflected in cecilia-10c-900 version (10c stands for ten categories): abusive content (ac), malicious code (mc), information gathering (ig), intrusion attempts (ia), intrusions (i), availability (a), information content security (ics), fraud (f), vulnerable (v) and others (o). in fig. 1 , we can observe the imbalanced distribution of the cecilia-10c-900 dataset, where most samples belong to the ics class. the emergence of this distribution may suggest that specific cyber incidents are less frequent in real-world environments. however, a deep study of the real-life distribution should be performed to avoid biased behavior. in section 4.4, an alternative dataset, cecilia-6c-900, is proposed to mitigate the issue of significant imbalance. the dataset has three fields: description, category, and subcategory. incident descriptions are written in non-technical english and span between 103 and 4299 characters. some samples of the cecilia-10c-900 dataset are shown in ## experimentation this section describes the experimental setup, including the transformer-based models and evaluation metrics used to assess the performance of these models in classifying incidents according to incibe's taxonomy. ## conclusions and future works in this work, we have evaluated two traditional classifiers and six models based on transformers using the cecilia-10c-900 dataset. the results show that transformer-based models outperform traditional classifiers. the outstanding performance demonstrated by transformer models strongly suggests that adopting this technology constitutes a promising strategy for the development of applications and services aimed at cyber incident classification. the ability of these models to capture complex contextual dependencies in extensive text sequences allows them to achieve high levels of accuracy in identifying and categorizing texts related to cybersecurity incidents. the implementation of transformers in cybersecurity expanded the ability to anticipate, detect, and respond more effectively to security threats, thereby contributing to the fortification of digital infrastructures against cyber attacks. given the evidence on the superior performance of transformer-based models, developing applications and services focused on cyber incident classification, grounded in this technology, represents an appropriate approach for applying artificial intelligence to cybersecurity. this approach is justified not only by the demonstrated efficacy in precise text classification but also by the adaptability and scalability of transformer models, which can be trained and fine-tuned to meet specific requirements in the field of cybersecurity. future research can be based on conducting further experiments by expanding and balancing the dataset used for training and evaluation. augmenting the dataset can provide a more comprehensive representation of the linguistic and contextual diversity inherent to cybersecurity texts. this expansion is expected to enhance the model's ability to generalize from training to unseen data, thereby improving its robustness and reliability in real-world applications. additionally, addressing the issue of dataset imbalance can avoid bias toward the over-represented classes. by providing a richer and more balanced training foundation, the models are expected to achieve higher levels of performance in terms of accuracy and their capacity to handle a broader spectrum of cyber incident types. from the taxonomy of incibe currently not represented in cecilia dataset. the current dataset, while extensive, does not fully cover all the groups of this taxonomy, resulting in certain types of cyber incidents being underrepresented or absent. by integrating these missing classes into the dataset, the model can be trained to recognize and classify a more complete spectrum of cyber incidents. finally, another way to improve future work involves enhancing the granularity of our classification approach, extending into subcategory precision. this refinement aims to yield a more detailed classification of cyber incidents. moreover, incorporating multi-label classification models or hierarchical classification structures can significantly improve the accuracy and performance of the classification model developed.