Ignore Me But Don’t Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domain (2024)

Abstract: Cybersecurity information is often technically complex and relayed through unstructured text, making automation of cyber threat intelligence highly challenging. For such text domains that involve high levels of expertise, pretraining on in-domain corpora has been a popular method for language models to obtain domain expertise. However, cybersecurity texts often contain non-linguistic elements (such as URLs and hash values) that could be unsuitable with the established pretraining methodologies. Previous work in other domains have removed or filtered such text as noise, but the effectiveness of these methods have not been investigated, especially in the cybersecurity domain. We experiment with different pretraining methodologies to account for non-linguistic elements (NLEs) and evaluate their effectiveness through downstream tasks and probing tasks. Our proposed strategy, a combination of selective MLM and jointly training NLE token classification, outperforms the commonly taken approach of replacing NLEs. We use our domain-customized methodology to train CyBERTuned, a cybersecurity domain language model that outperforms other cybersecurity PLMs on most tasks.

## introduction cybersecurity is a critical concern as the world continues to grow reliant on technology. modern cybersecurity practice emphasizes the need for preemptive defense utilizing cyber threat intelligence (cti) -actionable information on possible cyber-threats @xcite . however, due to the unstructured and complex nature of such information, leveraging cti requires extensive manual inspection by human experts @xcite . although automating cyber threat intelligence has been regarded as important @xcite , it has been considered highly challenging @xcite . meanwhile, pretrained language models (plms) have shown great potential for text comprehension @xcite . however, plms are unlikely to have developed the necessary expertise for domains that require significant domain knowledge, such as the cybersecurity domain. this could be somewhat addressed by extremely large models @xcite , but this option is costly to train and run. a more common approach to teach domain expertise to plms has been to pretrain on a domain-specific corpus. the effectiveness of such domain-pretrained plms has been demonstrated in the biomedical @xcite , scientific @xcite , and legal @xcite domains to name a few. several cybersecurity domain plms @xcite @xcite were also trained in a similar manner. however, cybersecurity texts often incorporate non-linguistic elements (nles) that could be inappropriate for self-supervised pretraining. selfsupervised objectives like masked language mod-eling (mlm) @xcite or de-noising objectives @xcite learn by recovering original texts from a masked or modified state. while this is mostly beneficial for natural language, such tasks might not be effective when trying to recover tokens in non-linguistic parts of text. figure 1 shows an excerpt from a malware threat report containing a sha hash and a url. the sha hash tokens are linguistically random, and therefore training a model to correctly recover these tokens may not be beneficial. similarly, the url tokens are less predictable compared to the natural language text surrounding it, and therefore potentially unsuitable for pretraining. outside of the cybersecurity domain, previous works addressed such nles through replacement (e.g. replace all urls with "[url]") @xcite @xcite or filtering @xcite @xcite . however, no attempt has been made to verify whether these approaches actually benefit pretraining. it is also unclear whether such practices would have similar benefits in the cybersecurity domain, where it is more common for nles to be used alongside natural language. conversely, pretraining with nles could be beneficial to utilize the informational value of nles. for instance, a model may learn to identify suspicious domains in urls or recognize familiar hash values in the way human cybersecurity experts can. we investigate different strategies of pretraining on the cybersecurity domain. we first identify commonly occurring nle types that can be extracted using regular expressions. we then pretrain models using different mlm strategies, testing the effectiveness of selective masking and nle token classification and comparing to the vanilla mlm and replacement strategy. our experiments suggest that replacement benefits on downstream tasks but harms performance on probing tasks, especially near nles. instead, we find that a strategy of selective masking while jointly training with nle token classification generally outperforms the replacement strategy. using this strategy, we train cybertuned (cybersecurity bert-like utilizing non-linguistic elements of the domain), a cybersecurity domain plm trained with the domaincustomized pretraining methodology. we show cybertuned outperforms comparable cybersecurity domain plms in most tasks. cybertuned model weights, training resources, and code are publicly available at https://github.com/ genesith/cybertuned . our contributions are as follows: ## related work cybersecurity nlp automating cyber threat intelligence has been often discussed in literature @xcite @xcite . classical off-the-shelf nlp methods, such as regex processing and dependency parsing, have been used to extract attack patterns @xcite or malware behaviors (zhu and dumitraş, 2016) from cybersecurity texts. recent works explore the potential of using bert @xcite for more complex tasks such as exploitability prediction @xcite , malware detection @xcite , and dark web analysis @xcite . domain plms plms train with large text corpora using self-supervision tasks @xcite @xcite . many domain plms @xcite @xcite were able to outperform general plms on domain-specific tasks by simply replicating existing pretraining procedures on domain corpora. plms for the cybersecurity domain using this approach have been suggested by several works @xcite @xcite . our work differs in that we use a domain-customized methodology after investigating the effectiveness of various strategies. pretraining strategies self-supervised tasks for pretraining have been investigated by many works @xcite @xcite . some works find improvements by modifying the masking procedure of the mlm task. changing mask- ## method in this section, we first discuss the types of nles in cybersecurity texts and how to extract nle instances. we then propose some methods to utilize the extracted nle spans in pretraining with cybersecurity texts. ## pretraining the models to evaluate our pretraining methodology on the cybersecurity domain, we pretrain models on a cy-bersecurity text corpus using a number of strategies. we first describe the tested pretraining strategies, including the vanilla mlm and ablation settings. we also describe our cybersecurity corpus and show statistics that suggest nles are more frequent in cybersecurity texts. ## experiments we evaluate the models trained by each pretrained strategy with both downstream tasks and probing tasks. for comparison, we also experiment with the base roberta model. ## discussion roberta's performance. the base roberta model achieved good performance on certain tasks after fine-tuning, often outperforming domainpretrained models. a possible interpretation is that previously challenging cybersecurity tasks, such as binary sequence classification of threat tweets, do not require extensive domain knowledge to achieve high performance. it should be noted that the cy-dec dataset reports a human f1-score of 0.59 and the twitterthreats dataset reports a cohen's κ of 0.66, suggesting these models have possibly already exceeded human and annotator performance on these tasks. this underscores the need for more challenging datasets for benchmarking models in the cybersecurity domain. ## conclusion we investigate methods to modify pretraining to suit the cybersecurity domain. we find that a strategy of selective mlm that allows for masking of semi-linguistic elements but not fully-linguistic elements with an auxiliary nle classification task showed best performance. with these findings, we present cybertuned, a cybersecurity plm with our modified pretraining methodology. the final cybertuned model shows strong performance across all cybersecurity downstream tasks. our findings support the importance of adapting pretraining methodologies to suit target domains.