Interpretable Identification of Cybersecurity Vulnerabilities from News Articles (2021)

Abstract: With the increasing adoption of technology, more and more systems become target to information security breaches. In terms of readily identifying zero-day vulnerabilities, a substantial number of news outlets and social media accounts reveal emerging vulnerabilities and threats. However, analysts often spend a lot of time looking through these decentralized sources of information in order to ensure up-to-date countermeasures and patches applicable to their organisation’s information systems. Various automated processing pipelines grounded in Natural Language Processing techniques for text classification were introduced for the early identification of vulnerabilities starting from Open-Source Intelligence (OSINT) data, including news websites, blogs, and social media. In this study, we consider a corpus of more than 1600 labeled news articles, and introduce an interpretable approach to the subject of cyberthreat early detection. In particular, an interpretable classification is performed using the Longformer architecture alongside prototypes from the ProSeNet structure, after performing a preliminary analysis on the Transformer’s encoding capabilities. The best interpretable architecture achieves an 88% F2-Score, arguing for the system’s applicability in real-life monitoring conditions of OSINT data.

## introduction with the increasing number of cybersecurity attacks, institutions need to join forces for the prevention and detection of cyberthreats. malicious entities target companies, but also individuals and public institutions (including health organisations), with an overall expected cost of 6000 bn $ per year @xcite . for example, the number of requests on the french malware assistance plat-form has tripled between 2018 and 2019 1 , and this trend has worsened during the pandemic @xcite due to the increased use of technology, the advent of remote working, coupled with an exponential growth of iot. in their endless race against hackers, security experts need to detect zero-day vulnerabilities and understand new methods employed by hackers to exploit them @xcite . in addition, both institutions and individuals need localized expertise applicable for their environment, given its specificity in terms of employed technologies. large companies often have their dedicated entity, the security operations center (soc), whose experts survey open-source intelligence (osint) data to identify new emerging vulnerabilities @xcite ; however, lower-sized entities and individuals should have this information also readily available. in addition, each data source (and there are a lot of possible venuese.g., more than 40 major blogs and newspapers are reported by feedspot ( 2021 ) ) has its own targeted fields and technologies of main interest, as well as its subjectivity in expressing the breath and impact of the vulnerability. manual searches in multiple sources is an overall tedious process, and the wide scattering of news feeds makes the dayby-day surveillance a daunting task; as such, an automated detection of zero-day vulnerabilities from osint data becomes a necessity @xcite . our goal is to provide an automated filtering of daily news feeds to identifying emergent cybersecurity threats as an initial screening for security experts. we emphasize from the beginning the importance of recall, namely it is critical not to disregard potential threats. moreover, model interpretability is an important dimension of the analysis in order to provide a preliminary grounding for the model's decisions. hence, our research question is: to what extent can automated systems detect cybersecurity threats in news articles while ensuring interpretable results? the paper is structured as follows. the second section introduces related work on real-time identification of threats and interpretable natural language processing (nlp) approaches, while the third section introduces our approach, composed of a deep analysis of the context and the design of 1 https://www.cybermalveillance.gouv.fr/ tous-nos-contenus/actualites/rapportactivite-2020 an interpretable model. the fourth section presents results alongside performance metrics, followed by discussions and conclusions. ## related work in this section, we review relevant related research on real-time threat identification and interpretable models in nlp. the abundance of osint data brought by twitter has enabled socs to develop cyber-threat intelligence with increasing performance, while exploring tweets in different manners (e.g., by cve @xcite or by account @xcite ). nonetheless, to our knowledge, existing studies focus on performance and do not consider interpretability, which seems crucial for such a critical task. ## results in terms of the standard longformer, the best performance was reached with pretraining, highlighting that additional cybersecurity context provides a significant boost in recall. transformer-based models completely outperform mnb, particularly in terms of recall (98% for transformer, only 62 % for mnb), but at the heavy cost of interpretability. prosenet was designed to tackle this problem, but as we can see in ## discussions an encountered issue is that only 11 prototypes were actually selected because of duplicates (the links for the selected papers are presented in while taking a closer look at the articles, our first intuition (after having studied and tediously labelled the entries), is that the selected article seem to cover quite well the different cases. the balance between relevant and irrelevant articles is fulfilled. articles 1, 2, 3, and 7 present general studies or miscellaneous events which are linked to cybersecurity, but not at all to cyberthreat detection. articles 4 and 8 study a cyberattack (without the disclosure of any vulnerability) and a legal case involving a cybercrime group; these are irrelevant. in contrast, relevant articles directly show vulnerability disclosure (articles 5, 9, and 10) or through cybercrime (articles 6 and 11). the security experts responsible for the initial datasets also confirmed our intuition. nonetheless, it is important to acknowledge that the evaluation of interpretability has not been achieved by a representative assembly of cybersecurity experts or members of socs, even though this is the minimum expected if we want organizations to trust our model. still, the analysis provided by the experts grounded an encouraging overview in terms of sparsity, diversity, and simplicity (see 3.5). in order to further support the degree to which the selected prototypes cover the latent space, figure 4 adds the prototypes to the latent space. at first sight, their distribution does not reflect the previously mentioned diversity, as the articles from the region of indecision from the encoder are mostly not covered by prototypes. this further consolidates our previous argument, i.e. the lack of data, which causes prototypes to be obviously labeled articles, and thus hinders performance. moreover, affinity propagation was also been implemented, with the aim of finding a finer-grained split: 54 clusters are identified, which are way too many for only 600 articles. on closer inspection, the grouping was done according to the company affected by the vulnerability. an approach based on keyword mining for each article would be interesting to further explore. ## conclusions and future work in this study we introduce a state-of-the-art architecture based on longformer and prosenet to create an interpretable pipeline to automatically label emerging cybersecurity threats. the similarity with the human decision process, coupled with a balanced performance on rather small dataset (i.e., recall -the most important metric -reaches 91%, while precision is at 78%), argue for the model's adequacy. thus, in response to the initial research question, our architecture provides an efficient filter, while also ensuring interpretability. the architecture is in place, but at a crossroads. first, further data collection is required in order to extend the training dataset. once a substantial number of articles are collected, the quality of our pipeline will be re-assessed, while also including security experts to scrutinize the explanations of our model. this is a long-term endeavour in which