Deciphering Cyber Threats: A Unifying Framework with GPT -3.5, BERT opic and Feature Importance (2024)

Abstract: This paper presents a methodology for the categorisation and attribute quantification of cyber threats. The data was sourced from Common Weakness Enumeration (CWE) entries, encompassing 503 hardware and software vulnerabilities. For each entry, GPT-3.5 generated detailed descriptions for 12 key threat attributes. Employing BERTopic for topic modelling, our research focuses on clustering cyber threats and evaluates the efficacy of various dimensionality reduction and clustering algorithms, notably finding that UMAP combined with HDBSCAN, optimised through parameterisation, outperforms other configurations. The study further explores feature importance analysis by converting topic modelling results into a classification paradigm, achieving classification accuracies between 60% and 80% with algorithms such as Random Forest, XGBoost, and Linear SVM. This feature importance analysis quantifies the significance of each threat attribute, with SHAP identified as the most effective method for this calculation.

## introduction in response to the evolving threat landscape, a range of techniques have been employed to enhance the pace and quality of vulnerability discovery and threat analysis. a core activity in this endeavor is the use of cyber threat modelling techniques. cyber threat modelling typically approaches the problem from the perspective of software vulnerabilities @xcite , attacker profiles (mitre), or system assets @xcite . asset-based modelling, in particular, offers several advantages, including the capability to conduct automated reasoning over a threat knowledge base. there are two specific research gaps which this research seeks to address. firstly, there is a lack of concise sources of threat information with sufficient coverage for asset-based threat modelling. for a cyber threat modelling process to be valid, it needs a broad and up-to-date threat information database. however, for structured asset-based models, such as those using the web ontology language (owl), the database must also be concise. an ideal threat database should be generated using a repeatable and automated process to ensure it stays up-to-date as the threat landscape changes. existing open-source threat databases, like cve (mitre, 2023b), cwe @xcite , and capec (mitre, 2023a), are typically too large to be converted into a structured representation for meaningful analysis. this makes it difficult to ensure the validity of the threat model. researchers tend to select a subset of threat entries of these databases, thereby reducing their coverage. even if a complete threat knowledge base is modeled, it quickly becomes outdated as new entries are added. either way, there is a need to develop a technique for repeatably generating a consolidated and up-todate threat knowledge base without compromising coverage. secondly, there is no robust quantitative methodology for characterising cyber threats from a given threat knowledge base. existing techniques, including ontology engineering methodologies @xcite , do not offer quantitative meth-ods for identifying key threat attributes which are pertinent to threat modelling. hence, threat models are typically, at least to some degree, based on the subjective experience and intuition of the designer @xcite , weakening the academic justification for selecting specific threat attributes. therefore, a new robust technique is needed to automatically identify threat attributes from a knowledge base for characterising cyber threats. this research addresses these gaps by demonstrating a viable method to generate a concise threat database using a highly repeatable and largely automated process. it also identifies the key attributes which constitute a cyber threat based on this database. the technique developed involves two main steps. first, it uses topic modelling to cluster primary cyber threat information into groups of normative threat classes. second, it performs feature importance analysis to determine the relative importance of each threat attribute. this allows us to identify the most important concepts for creating a generic threat model for asset-based cyber threat modelling. ## method -topic modelling for this task, we employed "bertopic", an advanced approach built on the foundation of bert (bidirectional encoder representations from transformers) @xcite . bert is renowned for its ability to understand the context in which words are used, making it especially valuable for datasets such as ours which focused on a specialised field like cyber security. our decision to opt for bertopic can be attributed to two reasons. first is contextual understanding: traditional models like lda view texts as simple bags of words, often missing the varied meanings a word can have in different contexts. bert, on the other hand, can discern these distinctions. for instance, it recognises that the word "bank" in "i sat on the bank of the river" and "i went to the bank to withdraw money" conveys different meanings. second is flexibility in handling texts: the bert-based model excels in dealing with shorter texts, whereas many traditional models fail. its ability to understand context ensures that even concise sentences are interpreted correctly, making it invaluable for datasets with varied text lengths. especially, one of the following proposed approaches required iterations of processing on one short sentence. ## method -feature importance analysis in the evolving landscape of feature importance analysis, many recently proposed methods for clustering are model-specific. these tailored techniques impose constraints when applied across different clustering and topic modelling methodologies. given this challenge, our approach leveraged the robust feature importance techniques from the classification paradigm, which exhibit (clustering) model-agnostic properties: our approach encompassed four key facets. firstly, we transformed the clustering outcome as a conversion to classification task. this entailed using classification models, or classifiers, to predict the clustering labels based on bert embeddings. the attribute-specific bert embeddings and clustering labels obtained post-topic modelling served as our input data and target variables for classifier training, respectively. secondly, during classifier training, we utilised three established classifiers: random forest, xgboost, and linear svm. each classifier has its unique mechanism for evaluating feature importance. in the third aspect, external method integra-tion, we broadened our analytical scope by adding external methodologies, specifically shap or permutation importance, to each classifier. these methods provided an independent basis for contrasting with the classifiers' built-in feature importance techniques. lastly, the aggregation and normalisation step was crucial. given that our importance analysis hinged on bert embeddings rather than directly on the 12 threat attributes, an aggregation step was essential. this step summarised the importance values attributed to each of the 12 threat attributes. to ensure a consistent interpretation across different methods, we normalised these importance values into relative percentages. figure 2 highlights the high-level process of feature importance analysis. in contrast to the previous topic modelling task, our feature importance analysis did not employ any dimension reduction techniques and hence the results retained their interpretability. figure 2 : high-level process of feature importance analysis. ## conclusion this paper presents a robust framework for the unsupervised classification of cyber threats and the quantitative analysis of their attributes, harnessing cutting-edge data science methods on textual data generated by gpt-3.5. bertopic successfully addressed the principal challenge of clustering cyber threat texts with specialised and narrow themes. our findings revealed that an optimally parameterised combination of umap and hdbscan-bertopic's default settings-outperformed other configurations in both quantitative metrics and expert qualitative evaluations. of the two text pre-handling strategies we employed, the one that maintained the original attribute structure proved less effective for improved topic modelling. we argue, however, that its high dimensionality may have influenced these results negatively. in the next phase of our study, we ventured into feature importance analysis to characterise cyber threat attributes quantitatively. we sidestepped the limitations of immature feature importance methods for clustering by adopting a classificationbased approach. utilising classifiers such as random forest, xgboost, and linear svm, we achieved classification accuracies ranging from 60% to 80%. among the feature importance techniques evaluated, shap stood out for its strong theoretical foundation and reliable performance. the cyber threat attributes identified using our feature importance technique can serve as the basis for constructing a cyber threat model to automate the analysis of cyber threats using assetbased threat modelling techniques. the methodology could also be applied to more bespoke knowl-edge domains for identifying threat attributes and developing threat databases and models in niche security domains. the concise threat database and corresponding threat model would be beneficial for security experts, researchers and policy makers in tasks such as cyber audits and risk assessments. furthermore, the methodologies and insights from this study hold potential for application in other sectors that rely on text-rich data for analytical interpretation, such as healthcare, law, and social sciences, aiding in extracting meaningful information and facilitating better decision-making. our methodological framework is not only robust but also modular and adaptable, offering promising avenues for future research in the fast-evolving landscape of machine learning and large language models.