U - BERT opic: An Urgency-Aware BERT -Topic Modeling Approach for Detecting C yber S ecurity Issues via Social Media (2024)

Abstract: For computer systems to remain secure, timely information about system vulnerabilities and security threats are vital. Such information can be garnered from various sources, most notably from social media platforms. However, such information may often lack context and structure and, more importantly, are often unlabelled. For such media to act as alert systems, it is important to be able to first distinguish among the topics being discussed. Subsequently, identifying the nature of the threat or vulnerability is of importance as this will influence the remedial actions to be taken, e.g., is the threat imminent? In this paper, we propose U-BERTopic, an urgency-aware BERTtopic modelling approach for detecting cybersecurity issues through social media, by integrating sentiment analysis with contextualized topic modelling like BERTopic. We compare UBERTopic against three other topic modelling techniques using four different evaluation metrics for topic modelling and cybersecurity classification by running on a 2018 cyber security-related Twitter dataset. Our results show that (i) for topic modelling and under certain settings (e.g., number of topics), U-BERTopic often outperforms all other topic modelling techniques and (ii) for attack classification, U-BERTopic performs better for some attacks such as vulnerability identification in some settings.

## introduction there has been a noticeable increase in the number of cyberattacks as well as in the severity of their consequences. the uk department for science, innovation and technology's survey shows that one-tenth of companies and nonprofit organizations fell victim to cybercrime in one year (2023), marking a 29% increase from the previous year @xcite . the financial impact of cyberattacks has also increased dramatically according to ponemon institute and ibm security's report @xcite , with the average cost of a data breach in 2023reaching usd 4.45 million, the highest level ever, representing a 2.3% increase compared to the previous year. to protect it infrastructure from cyberattacks, it is important for security engineers to obtain timely information about system vulnerabilities and threats. social media is proving to be an important outlet where these issues are discussed. however, such information are often unstructured, may lack context and, very often, unlabelled. ## u-bertopic model in the following paragraph, we introduce in detail u-bertopic, which extends traditional topic modelling to focus tones of urgency and necessity characterising cybersecurity issues. first, proposing uc-tf-idf which is cybersecurity focused of bertopic @xcite 's c-tf-idf to include the sentiment, urgent scores of the text. furthermore, we apply cybert @xcite which is a cybersecurity llm model. ## instance examination to further examine u-bertopic's ability to capture the urgency level of cybersecurity discussion and news, a significant series of cyberattacks with a high impact was selected for a case study, in this case the microsoft exchange server attacks in 2021 (cisa, 2021) and 2022 (cisa, 2022). our study evaluates whether the generated topics contain terms uniquely associated with these cyberattacks that will suggest better model performance in detecting urgency. the timeline of these attacks is as shown in figure 1 (they occurred between 2021 and 2022). the 2022 dataset comprises data collected from january 2021 to december 2022 from cyberthreat intelligence x (formerly twitter) accounts to cover the case. after data cleansing, all tweets about the microsoft exchange server were aggregated using keywords such as "microsoft exchange," "outlook," "proxyshell," "proxynot-shell," and "ms exchange." after that, all four topic modelling algorithms are applied and the results are shown in table a6 in the appendix. in this table , u-bertopic can extract more ms exchange server zdas terms (proxyshell, proxynotshell, dearcry, and proxylogon). these urgency-related keywords were unseen before the event and they are either vulnerability or malware names associated with the attack. ## limitations and discussion limitations the proposed u-bertopic model combines contextualized topic modeling with sentiment analysis to improve the system's ability to learn the urgency level of cybersecurity issues. u-bertopic employs lexicon-based sentiment analysis but, to accurately capture urgency within security-related content demands more sophisticated approaches, such as machine learning-based sentiment analysis. existing datasets also lack urgency labels for better model training. additionally, while the cybersecurity keywords and scores used to update the sentiment analysis lexicon are currently collected and estimated manually, cybersecurity events have a variety of terms and trends that need to be taken into account. ## conclusion and future work in this paper, we have introduced u-bertopic, an urgency-aware topic modelling designed to detect cyberattacks and enhance the cti discovery process. u-bertopic leverages probabilistic and neural nlp models, such as transformer-based word architectures and topic models for fine-grained detection of cybersecurity topics and sentiments. by integrating sentiment analysis with contextualized topic modelling like bertopic, we spotlight the topics most representative of ongoing cyberattacks and urgent events. our newly developed method, uc-tf-idf, is tailored to extract requirements that are particularly relevant to urgent cybersecurity events. comprehensive evaluations of topic modelling have been conducted, showing the improved ability of u-bertopic in detecting sentiment-critical cybersecurity topics. future work will expand upon this foundation by further integrating urgency in sentiment analysis into the topic modeling approach and comparing the performance with different large language models (llms). figure a3: coherence (cv) scores figure a4: topic quality scores figure a5: overview of the intrinsic evaluation metrics for topic modeling algorithms applied on the cybersecurity dataset 2018, showcasing measures of diversity, coherence, and quality across varying numbers of topics (@xmath0. model ctm bertopic lda u-bertopic topic1 reading , dark , infosec , security , credentials exchange , microsoft, infosec, the, servers microsoft, infosec, exchange, technology, software techcrunch, technology, software, infosec, raises topic2 hacker , infosec , technology , news , ransomware vulnerability, is, id, cve, unique microsoft, exchange, infosec, vulnerability, software proxyshell, proxylogon, exchange, servers proxynotshell topic3 cybersecurity , read , malware , details , proxyshell techcrunch, technology, software, infosec, toward infosec, microsoft, software, technology, exchange owasp, knowage, xss, parameter, crosssite topic4 deal , bundle , outlook , mac , serghei outlook, serghei, microsoft, emails exchange, infosec, microsoft, software ransomware, servers, deploy, exchange, dearcry topic5 id , cve , unique , vulnerability , remote thx, pogowasright, continued, pcrisk, advintel pogowasright, thx, exchange, microsoft, infosec outlook, serghei, issues, emails, search topic6 server , vulnerability , user , files , attacker office, deal, get, license, bundle microsoft, infosec, software, technology, exchange owasp, cyber, security, new, resources topic7 owasp , suite , knowage , parameter , xss yanluowang, ransomware, gang, decryptor, stolen microsoft, exchange, vulnerability, server, code thx, pogowasright, continued, pcrisk, advintel topic8 exchange , server , vulnerabilities , onpremises , exploited yahoo, gmail, iranian, hackers, tool microsoft, infosec, exchange, software, technology deal, office, bundle, mac, training topic9 toward , techcrunch , raises , technology , infosec broward, breach, health, data, people microsoft, exchange, infosec, technology, software execution, remote, id, cve, unique topic10 surveillance, agents, data, breach, health microsoft, exchange, ransomware, proxyshell, servers spoofing, vulnerability, office, microsoft, feature table a6: comparison between topic modelling results on microsoft exchange server case study dataset