The Dark Side of the Language: Pre-trained Transformers in the D ark N et (2023)

Abstract: Pre-trained Transformers are challenging human performances in many Natural Language Processing tasks. The massive datasets used for pre-training seem to be the key to their success on existing tasks. In this paper, we explore how a range of pre-trained natural language understanding models performs on definitely unseen sentences provided by classification tasks over a DarkNet corpus. Surprisingly, results show that syntactic and lexical neural networks perform on par with pre-trained Transformers even after fine-tuning. Only after what we call extreme domain adaptation, that is, retraining with the masked language model task on all the novel corpus, pre-trained Transformers reach their standard high results. This suggests that huge pre-training corpora may give Transformers unexpected help since they are exposed to many of the possible sentences.

## introduction transformers @xcite have been rocking the field of nlp. these transformers are outperforming all previous methods and, sometimes, even humans in many nlp tasks @xcite @xcite . pre-training on large corpora seems to be the key that boosts performances, as these transformers may induce clear models of target languages. indeed, bert is pre-trained on an english corpus of 3,300m words consisting of books @xcite and wikipedia. the english version of the last ernie @xcite ) is trained on an even more extensive corpus. megatron-lm @xcite utilizes an incredible corpus of 174 gb, and the chinese version of ernie breaks the records by exploiting a 4tb corpus @xcite . therefore, the challenge is training over always more massive corpora. huge pre-training corpora may give unexpected help to transformers: their successes in downstream tasks can be because transformers have seen large parts of possible sentences. this could be a sort of overfitting. this possible shortcoming is sometimes considered when novel transformers are introduced @xcite . for this reason, @xcite have excluded wikipedia pages for pre-training as it is a common data source for other downstream datasets. yet, when using off-the-shelf pre-trained models, this caution is generally disregarded. for example, the discovering ongoing conversation (doc) task, introduced by zanzotto and ferrone (2017) was found challenging for humans, but the bert baseline model achieved the astonishing 88.4 f1 score @xcite . doc consists of determining if two utterances are contiguous in classical theatrical plays. these plays may be included in the book dataset @xcite used for pretraining bert. this may explain the superhuman performance in such a challenging task. corpora and related tasks derived from the deep-web and darkweb @xcite @xcite offer a tremendous opportunity to study transformers and other natural language models on definitely unseen sentences. performances on these tasks cannot depend on overfitting over-seen sentences. indeed, it is extremely unlikely that texts extracted from these sources are included in pretraining corpora. moreover, language on the dark-net may have very different characteristics with respect to the one accessible from the surface web @xcite . in this paper, we aim to explore how pre-trained natural language understanding models behave on definitely unseen sentences. these definitely unseen sentences are provided by the darknet corpus ## material and methods this section describes the corpus used to challenge transformers that is a source of definitely unseen sentences (section 2.1) and the investigated transformers along with other neural network methods (section 2.2). ## results and discussion this section investigates how pre-trained transformers and other pre-trained language models behave on definitely unseen sentences (sec. 3.3). yet, to exclude that the nature of the task biases our study, we have performed additional analyses: 1) to understand whether the onion language is really different from the surface web language (sec. 3.1), and 2) to determine the nature of the proposed classification task (sec. 3.2). ## conclusions transformers are successful in many downstream tasks, and this success also stems from the huge corpora that they are trained on. since they are so successful, the investigation of their strengths and potential weaknesses is important. our paper and our experiments show that transformers largely outperform other models only when they are pre-trained on texts which are extremely similar to texts in the target application. indeed, only when transformers are trained with masked language model (mlm) on the definitely unseen sentences of the darkweb corpus, do these transformers start to behave extremely better than other techniques. the reason why it is happening is still unclear as, in adapting to the new domain with mlm, transformers are not learning anything about the specific task but they are gaining some general model of novel texts. our results suggest that pre-trained transformers should clearly release the pre-training datasets to allow practitioners to explore if sentences in their dataset are included or partially covered. in our opinion, future work should go in two directions: (1) exploring what transformers are really learning during the masked language model and next sentence prediction; (2) providing measures for understanding how much a pre-trained model knows about given texts and given datasets.