WAVE -27 K : Bringing together CTI sources to enhance threat intelligence models (2024)

Abstract: Considering the growing flow of information on the internet, and the increased incident-related data from diverse sources, unstructured text processing gains importance. We have presented an automated approach to link several CTI sources through the mapping of external references. Our method facilitates the automatic construction of datasets, allowing for updates and the inclusion of new samples and labels. Following this method we built a new dataset of unstructured CTI descriptions called Weakness, Attack, Vulnerabilities, and Events 27k (WAVE-27k). Our dataset includes information about 27 different MITRE techniques, containing 22539 samples related one technique and 5262 related to two or more techniques simultaneously. We evaluated five BERT-based models into the WAVE-27K dataset concluding that SecRoBERTa reaches the highest performance with a 77.52% F1 score. Additionally, we compare the performance of the SecRoBERTa on the WAVE-27K dataset and other public datasets. The results show that the model using the WAVE-27K dataset outperforms the others. These results demonstrate that the data within WAVE-27K contains relevant information and that the proposed method effectively built a dataset with a level of quality sufficient to train a machine-learning model.

## introduction the growing flow of information has led to increased incident-related data from diverse sources, such as open-source intelligence (osint) platforms, cybersecurity analyst forums, and several other sources on the internet. therefore, it is crucial to automatically process unstructured texts @xcite to extract information such as tactics, techniques, and procedures (ttps) from different free-text sources to help understand and detect relevant incidents inside the local network. in addition, algorithms designed to process unstructured texts for ttps offer an advantage in their capacity to extract valuable insights from unconventional sources, such as dark web forums and other suspicious platforms where malicious activities are documented and discussed. this capacity not only facilitates the detection and characterization of cyber attacks but also enables the identification of underground networks where new attacks are disseminated. however, regardless of the advantages offered by algorithms designed to process unstructured cti texts and their significant impact on the security of local networks, there is a need for a more extensive dataset of unstructured cyber threat intelligence (cti) texts. we hypothesize that enhancing the quality and quantity of accessible data will substantially improve the efficacy of state-of-the-art models. to supply this lack, we acknowledge it is key to propose a methodology that takes advantage of the increasing flow of information mentioned above, providing automatic methods to create datasets and train algorithms focused on extracting and detecting ttps from unstructured texts. our method uses the information from cyber threat intelligence (cti) sources to automate the data collection process of information related to ttps in unstructured text, reducing costs and ad hoc studies with limited data. our goal is to develop cti tools that contribute to the community and help standardize the datasets in the state of the art allowing state of the art models comparison. consequently, we address an approach that involves three steps: constructing a method for automatically creating cti datasets, collecting a dataset following the proposed method, and evaluating machine learning models to validate the built dataset. as a result, we present a dataset named wave-27k, including unstructured texts with tactical techniques and procedures (ttp) information. wave-27k contains approximately 28,000 cti descriptions associated with seven tactics and 27 different mitre techniques. to the best of our knowledge, wave-27k is the largest dataset available in the cti state of the art. this paper is organized as follows: section 2 provides a related work review, offering context for the research. section 3 details our methodology, including the dataset-building process. section 4 describes the experiments, defining details regarding the models and the metrics used for model evaluation. finally, section 5 presents the results, and section 6 contains our findings and future research. ## background there is two groups from the ttp pattern extraction literature differentiated by their goals. the first group extracts information from unstructured sources and transforms it into structured data. this process implies detecting different entities in a free text sample, and then identifying their relationships to generate knowledge graphs. the second group focuses on classification techniques, addressing cti unstructured data as a classification problem. the primary goal of this group is to detect patterns within the unstructured text and categorize them according to known cyberattack techniques, enabling the identification and classification of relevant information. in this section, we detail significant results presented in the state of the art related to both groups. ## methodology in our data collection process, we employed four primary sources that have been widely used in pre-7 cve2att github repository vious research to construct cti datasets; we selected these sources due to the facility to crossreference their samples as has been proposed previously in the state of the art @xcite @xcite . the first source is the mitre att&ck framework, used as the foundation for standardizing datasets within the classification group @xcite @xcite @xcite . this framework provides information on the tactics and techniques employed by attackers and information about campaigns, the associated threat actor groups, the tools and software used in the attacks, and potential mitigation strategies as well. another source employed in our data collection process is capec, which assists in understanding how adversaries exploit software vulnerabilities. this list of attack patterns includes several columns providing information such as the attack pattern name, description, likelihood, related weaknesses, execution flow, severity, and additional relevant data. taking advantage of the information provided by capec regarding software weaknesses, our third data source is the common weakness enumeration (cwe), containing a list developed by the community of software and hardware vulnerabilities. this list traces each weakness with background details, affected technologies, consequences, impacted architectures, and observable examples. finally, the fourth data source is the common vulnerabilities and exposures (cve) repository, which lists information on known vulnerabilities. each cve entry includes a description of the vulnerability, its complexity, and its impact on confidentiality, integrity, and software availability. we performed a complete review of the fields to identify potential references to external sources for each source. some sources, such as mitre and capec, contain fields that directly present external references. in these cases, the external reference field within an entry was analyzed to verify if it included data from the selected sources. in the case of the cwe source, we analyzed the "observed example" field, which contains information about reported vulnerabilities. using the vulnerability id, we linked the information to the cve source. subsequently, all references were evaluated to determine if the target id in the origin source was included in the data of the target source. the next phase involved matching the extracted ids to establish new relationships and creating those relationships in an stix format. this approach aims to enhance data completeness by adding information from diverse views offered by different sources. as previously mentioned, data collection involves establishing new connections by mapping external references. specifically, we used the following fields: mitre att&ck external references to associate with capec ids, capec external references to correlate with cwe ids, and cve weaknesses to align with cwe ids. cwe plays a central role in this process, as depicted in figure 1 . the strength of this methodology lies in its automatic construction which enables updates and the addition of new samples for entry as well as the creation of new labels if new mitre techniques are reported, besides the possibility of standardizing the dataset construction and normalizing the labels in the state of the art datasets. however, a potential limitation is the coverage of the dataset since there is limited control over the class balance within the dataset. moreover, this approach suggests that the samples collected represent the prevailing trends and patterns observed in cyber attacks, providing valuable insights into real-world threat scenarios. as a result of this methodology, we have created a comprehensive superset called weakness, attack, vulnerabilities, and events (wave). this superset contains all the information downloaded from the sources, as well as all the relationships established through the external reference matching process. with this superset, we can link information from vulnerabilities (cve) to mitre att&ck techniques and even mitre att&ck mitigations. ## results for the first experiment, after training the five proposed bert-based models in section 4 to establish a baseline for comparison on wave-27k, the results indicate that secroberta achieved the highest performance with a 77.52% f1 score and a 83.51% accuracy, followed by bert with 77.31% f1 score and 83.29% accuracy, as shown in the second experiment, we used secroberta as it had the highest performance in the previous experiment. after the training phase, the se-croberta model from the wave-27k dataset regarding the comparison between overlapped classes of wave-27k and the public datasets, the results demonstrate that the model trained with wave-27k outperforms those trained with the cti, tram, and enisa datasets, achieving micro f1-scores of 96.46%, 95.50%, and 92.15%, respectively, as shown in ## conclusions and future work this paper presents an automated approach to link several cti sources through the mapping of external references, resulting in a more complete dataset. the previous is due to the inclusion of insights from different four sources. our method facilitates the automatic construction of datasets, allowing for updates and the inclusion of new samples and labels. to assess the data collection method, we used a subset of features extracted from the consolidation of the four sources, namely weakness, attack, vulnerabilities, and events 27k dataset (wave-27k). the wave-27k includes the cve description as a free-text sample and the corresponding mitre techniques related to the description. while one potential limitation derives from the coverage of the dataset, with limited control over class balance, this approach suggests that the collected samples reflect prevalent trends and patterns in cyberattacks, providing valuable insights into real-world threat scenarios. wave27k contains 27801 samples, where 22539 of them are associated with a single technique and the remaining 5262 samples are linked to two or more techniques. wave-27k includes 27 distinct labels of mitre techniques. we trained five bert-based models in the evaluation process, finding that secroberta reaches the highest performance with a 77.52% f1 score. subsequently, the model trained with the wave-27k dataset achieved a 91.39% accuracy in the single output test. finally, in the comparison of overlapping classes, our model using the wave-27k dataset outperforms others, achieving an f1 score of up to 96.46%. these findings demonstrate that the data within wave-27k contains relevant information for incident classification. the results show that the proposed method effectively built a dataset with a level of quality sufficient to train a machine-learning model. in future research, we aim to explore additional machine learning models that were not considered in this study. additionally, we plan to study the possibility of training specialized models for each class to assess the effectiveness of classification in such a scenario. furthermore, we will explore a cascading classification approach, initially classifying tactics followed by a technique classification using a stacking method. this approach will allow us to determine if hierarchical classification enhances overall performance. we aim to face the challenge of processing longer unstructured text and associating it with relevant tags as well. incorporating state of the art models into this context will improve the capabilities of attack classification systems into a more realistic scenario. this advancement will facilitate the automatic generation of alerts from free and unstructured text, enhancing the efficiency of threat detection and response mechanisms.