A Pretrained Language Model for Cyber Threat Intelligence (2023)

Abstract: We present a new BERT model for the cybersecurity domain, CTI-BERT, which can improve the accuracy of cyber threat intelligence (CTI) extraction, enabling organizations to better defend against potential cyber threats. We provide detailed information about the domain corpus collection, the training methodology and its effectiveness for a variety of NLP tasks for the cybersecurity domain. The experiments show that CTI-BERT significantly outperforms several general-domain and security-domain models for these cybersecurity applications indicating that the training data and methodology have a significant impact on the model performance.

## introduction in response to rapidly growing cyber-attacks, cybersecurity experts publish many cti reports, detailing on new security vulnerabilities and malware. while these reports help security analysts to better understand the cyber-threats, it is very difficult to digest all the information in a timely manner. thus, automatic extraction of cti from text has gained a lot of attention from the cybersecurity community. however, general-domain language models (lms) are not effective for cybersecurity text due to differences in terminology and styles. earlier studies have demonstrated that domain-specific lms are crucial for domain-specific applications @xcite @xcite @xcite @xcite @xcite . two different approaches have been used to produce domain-specific language models: continual pretraining and pretraining from scratch. the continual pretraining method takes an existing generaldomain model and continues training the model using a domain-specific corpus. while this approach is useful, especially when the size of the domain-specific corpus is small, the vocabulary of the new model remains largely same as that of the original model. most domain-specific terms are thus out of vocabulary. the pretraining from scratch approach trains a new tokenizer to construct a domain-specific vocabulary and trains the language model using only its own corpus. @xcite , @xcite , and @xcite have trained bert models from scratch for the biomedicine, computer science, and political science areas. these studies show that pretraining from scratch outperforms the continual pretraining. recently, a few transformers-based lms have been built for the cybersecurity domain. @xcite ) trains a bert model, and securebert @xcite trains a roberta model using the continual pretraining method. jackaduma (2022) introduces secbert and secroberta models trained from scratch. however, these models either do not provide training details or are not evaluated on many cybersecurity tasks. we present cti-bert, a bert model pretrained from scratch with a high quality cybersecurity corpus containing cti reports and publications. in cti-bert, both the vocabulary and the model weights are learned from our corpus. further, we introduce a variety of sentence-level and token-level classification tasks and benchmark datasets for the security domain. the experimental results demonstrate that cti-bert outperforms other generaldomain and security domain models, confirming that training a domain model from scratch with a high quality domain-specific corpus is critical. to the best of our knowledge, this work provides the most comprehensive evaluations for classification task within the security domain. accomplishing these tasks is a crucial part of the broader process of automatically extracting cti, suggesting appropriate mitigation strategies, and implementing counter-measurements to thwart attacks. thus, we see our work as an essential milestone towards more intelligent tools for cybersecurity systems. the main contributions of our work are the following: ## training datasets we curated a cybersecurity corpus from various reputable data sources. the documents are professionally written and cover key security topics including cyber-campaigns, malware, and security vulnerabilities. most of the documents are in html and pdf formats. we processed the files using the apache tika parsers foot_0 to extract the file content. then, we detected sentence boundaries and discarded sentences if the percentage of word tokens is less than 10% in the sentences. ## training methodology we first train the wordpiece tokenizer after lowercasing the security text and produce a vocabulary with 50,000 tokens. training a tokenizer from scratch is beneficial, as it can recognize domainspecific terms better. ## cybersecurity applications we evaluate cti-bert using several security nlp applications and compare its results with both general-domain models and other cybersecurity domain models. the baseline models are bert-base-uncased, secbert (bert models) and roberta-base, secroberta and securebert (roberta models). all the baseline models are downloaded from huggingface. the downstream applications can be categorized as sentence-level classification tasks and tokenlevel classification tasks. the goal of the experiments is to compare different pretrained models rather than optimizing the classification models for individual tasks. thus, we use the same model architecture and hyper-parameters to fine-tune models for all sub-tasks in each application category. ## related work motivated by the large-scale foundational models's successes in many general domain nlp tasks, several domain-specific language models have been developed @xcite @xcite . in scientific and bio-medical domains, there are scibert @xcite , blue-bert @xcite , clinicalbert @xcite , biobert @xcite and pub-medbert @xcite . in political and legal domains, there are conflictbert @xcite and legalbert @xcite . these domain models have shown to improve the perfor-mance of downstream applications for the domain. there have been several attempts to construct language models for the cybersecurity domain. @xcite propose techniques to efficiently learn domain-specific language models with a small-size in-domain corpus by incorporating external domain knowledge. they train word2vec models using malware descriptions. similarly, @xcite train a word2vec model using security vulnerability-related bulletins and wikipedia pages. recently, transformers-based models have been built for the cybersecurity domain: cy-bert (priyanka ranade and finin, 2021), secbert (jackaduma, 2022) and secure-bert @xcite . cybert is trained with a relatively small corpus consisting of 500 security blogs, 16,000 cve records, and the aptnotes collection. further, cybert applies the continual pretraining and uses the bert model's vocabulary after adding 1,000 most frequent words in their corpus which do not exist in the base vocabulary. secbert provides both bert and roberta models trained on a security corpus consisting of aptnotes, the semeval2018 task8 dataset and stucco-data 13 which contains security blogs and reports. however, the details about the data and any experimental results are not available. securebert trains a roberta model using security reports, white papers, academic books, etc., which are similar to our dataset both in terms of the size and document type. however, the model is built using the continual pretraining method while cti-bert is trained from scratch. we believe that the main difference comes from cti-bert being trained from scratch and having the vocabulary specialized to the domain, compared to the extended vocabulary used in cybert and securebert. ## conclusion we presented a new pretrained bert model tailored for the cybersecurity domain. specifically, we designed the model to improve the accuracy of cyber-threat intelligence extraction and understanding, such as security entity (iocs) extraction and attack technique (ttps) classification. as demonstrated by the experiments in section 4, our model outperforms existing general domain and other cybersecurity domain models with the same base architecture. for future work, we plan to collect more documents to improve the model and also to train other language models to support different security applications.