D ark BERT : A Language Model for the Dark Side of the I nternet (2023)

Abstract: Recent research has suggested that there are clear differences in the language used in the Dark Web compared to that of the Surface Web. As studies on the Dark Web commonly require textual analysis of the domain, language models specific to the Dark Web may provide valuable insights to researchers. In this work, we introduce DarkBERT, a language model pretrained on Dark Web data. We describe the steps taken to filter and compile the text data used to train DarkBERT to combat the extreme lexical and structural diversity of the Dark Web that may be detrimental to building a proper representation of the domain. We evaluate DarkBERT and its vanilla counterpart along with other widely used language models to validate the benefits that a Dark Web domain specific model offers in various use cases. Our evaluations show that DarkBERT outperforms current language models and may serve as a valuable resource for future research on the Dark Web.

## introduction the dark web is a subset of the internet that is not indexed by web search engines such as google and is inaccessible through a standard web browser. to access the dark web, specialized overlay network applications such as tor (the onion router) @xcite are required. tor also hosts hidden services (onion services) -web services in which the client and the server ip addresses are hidden from each other @xcite . this sense of identity obscurity provided to the dark web users comes with a catch; many of the underground activities prevalent in the dark web are immoral/illegal in nature, ranging from content hosting such as data leaks to drug sales @xcite . as such, the popularity of the dark web as a platform of choice for malicious activities has garnered interest from researchers and security experts alike. to handle the ever-changing landscape of modern cyber threats, cybersecurity experts and researchers have started to employ natural language processing (nlp) methods. gaining evidencebased knowledge such as indicators of compromise (ioc) to mitigate emerging threats is an integral part of modern cybersecurity known as cyber threat intelligence (cti) @xcite , and modern nlp tools have become an indispensable part of cti research. as such, the use of nlp techniques has also been extended to the dark web @xcite @xcite @xcite . the continued exploitation of the dark web as a platform of cybercrime makes it a valuable and necessary domain for cti research. recently, @xcite observed that using a bert-based classification model achieves state-ofthe-art performance among available nlp methods in the dark web. however, bert is trained on surface web foot_0 content (i.e., wikipedia and book-corpus) @xcite , which has different linguistic characteristics from that of the dark web @xcite . in the context of cti, this implies that popular pretrained language models such as bert are not ideal for dark web research in terms of extracting useful information due to the differences in the language used in the two domains. consequently, an nlp tool that is suitable for application in dark web domain tasks would prove to be valuable in the ongoing efforts of dark web cybersecurity. in this paper, we propose darkbert, a new language model pretrained on a dark web corpus. to measure the usefulness of darkbert in handling cyber threats in the dark web, we evaluate dark-bert in tasks related to detecting underground activities. we compare darkbert to other widely used pretrained language models bert @xcite and roberta @xcite that are trained on data found in the surface web to verify the efficacy of darkbert in dark web domain texts. our evaluation results show that darkbert-based classification model outperforms that of known pretrained language models. furthermore, we present potential use cases to illustrate the benefits of utilizing darkbert in cybersecurityrelated tasks such as dark web forum thread detection and ransomware leak site detection. our contributions are summarized as follows: ## related work the recent availability of dark web resources @xcite @xcite has made it possible to explore the differences between the languages used in the dark web and the surface web. @xcite explored the differences in the illegal and legal pages in the dark web and found a number of distin-guishing features between the two domains such as named entity, vocabulary, and syntactic structure. their analyses using standard nlp tools have also suggested that processing text in the dark web domain would require considerable domain adaptation. the linguistic differences between the surface web and the dark web were further examined by @xcite through linguistic features such as part-of-speech (pos) distribution and vocabulary usage between the texts in the two domains. recently, @xcite explored the use of pretrained language models over dark web texts to examine the effectiveness of such models, and suggested that lexical and syntactic models such as glove @xcite outperform pretrained models in some specific dark web tasks. meanwhile, @xcite demonstrated that pretrained language models in some dark web tasks such as dark web activity classification perform better than simple lexical models, suggesting that language models like bert show promising results in the dark web. either way, a domain-specific pretrained language model would be beneficial in that it would be able to represent the language used in the dark web, which may effectively reduce the performance issues faced in previous experiments. ## darkbert construction in this section, we describe the process for building our dark web domain-specific pretrained language model, darkbert. we begin by collecting pages to build the text corpus used for pretraining dark-bert (section 3.1). then, we filter the raw text corpus and employ text preprocessing methods for pretraining purposes (section 3.2). finally, we pretrain darkbert using the text corpus (section 3.3). an overview of the darkbert construction process is illustrated in figure 1 . ## evaluation: dark web activity classification in this section, we describe the methods of evaluation and the datasets used to evaluate dark-bert and other language models. since page classification has often been performed in past works @xcite @xcite , we also choose dark web activity classification as the main dark web domain benchmark experiment for evaluation. we additionally conduct experiments on multiple use case scenarios, which is described in detail in section 5. ## use cases in the cybersecurity domain in this section, we introduce three dark web domain use cases for darkbert and demonstrate its effectiveness over existing language models in cybersecurity / cti applications. we list details on the experimental setup for each use case in sections e.2 and e.3 of the appendix. ## conclusion in this study, we propose darkbert, a dark web domain-specific language model based on the roberta architecture. to allow darkbert to adapt well to the language used in the dark web, we pretrain the model on a large-scale dark web corpus collected by crawling the tor network. we also polish the pretraining corpus through data filtering and deduplication, along with data preprocessing to address the potential ethical concerns in dark web texts related to sensitive information. we show that darkbert outperforms existing language models with evaluations on dark web domain tasks, as well as introduce new datasets that can be used for such tasks. darkbert shows promise in its applicability on future research in the dark web domain and in the cyber threat industry. in the future, we also plan to improve the performance of dark web domain specific pretrained language models using more recent architectures and crawl additional data to allow the construction of a multilingual language model.