T rel BERT : A pre-trained encoder for P olish T witter (2023)

Abstract: Pre-trained Transformer-based models have become immensely popular amongst NLP practitioners. We present TrelBERT – the first Polish language model suited for application in the social media domain. TrelBERT is based on an existing general-domain model and adapted to the language of social media by pre-training it further on a large collection of Twitter data. We demonstrate its usefulness by evaluating it in the downstream task of cyberbullying detection, in which it achieves state-of-the-art results, outperforming larger monolingual models trained on general-domain corpora, as well as multilingual in-domain models, by a large margin. We make the model publicly available. We also release a new dataset for the problem of harmful speech detection.

## introduction pre-trained language models based on the transformer architecture @xcite have dominated the field of nlp. the models vary in size, with the largest ones reaching hundreds of billions of parameters, and are trained with different objectives, such as causal language modeling @xcite or masked language modeling @xcite . by processing large amounts of text, they learn to capture general knowledge about the language and can then be fine-tuned to perform domain-specific tasks. regardless of the neural network architecture design choices, an important factor is the domain of the training data. for years, research in the field of nlp was mostly focused on the english language, but models and resources for many other languages have also been published recently. multilingual models have also been developed @xcite which are capable of understanding more than 100 languages at once. the corpora used for pre-training are typically mixtures of general-domain data sources, such as crawled websites, books or wikipedia articles. the language can vary significantly across domains, not only in terms of the vocabulary, but also syntax, semantics and pragmatics. while the language of the aforementioned general-domain sources conforms to the linguistic norms, there is a large and important domain where the language is distinctly different and rapidly changing, namely social media. apart from the obvious differences, such as the occurrence of hashtags or emojis, people have figured out how to shout using capital letters, or that ending a message with a period might be perceived as sarcastic. in order to properly represent such characteristics in the language models, it is necessary for them to be exposed to domainspecific texts not only during the supervised finetuning, but also in the pre-training phase. in this work, we introduce trelbert, an encoder-only language model initialized with existing general-domain weights and adapted to the social media domain by pre-training it on over 40 million polish tweets with the masked language modeling objective. trelbert proves to be well-suited for application of nlp in social media, achieving state-of-the-art results for downstream tasks operating on polish twitter data. we make the model publicly available foot_0 . the main contribution of our research is the introduction of the first polish language representation model pre-trained on twitter data. our model achieves state-of-the-art results in the cyberbullying detection task (part of the polish nlp benchmark), outperforming all existing solutions, including larger general-domain polish models, as well as multilingual in-domain models. we also release a dataset of 1000 tweet ids labeled for the problem of harmful speech detection which is a less biased (randomly sampled using the streaming api) and more up-to-date alternative to the existing one. ## related work in this section, we review the current state of polish nlp and provide an overview of language models trained in the social media domain. ## trelbert we introduce a language model trained on polish tweets which we call trelbert foot_4 . it is a transformer encoder model trained with the masked language modeling objective. rather than training trelbert from scratch, we take advantage of existing weights and adapt them to the social media domain. as our starting point, we use herbert-base @xcite , the best performing one among polish base models foot_5 . herbert was initialized with weights from xlm-r @xcite and further pre-trained on a mixture of general-domain polish corpora with 8.6b tokens in total. its tokenizer is a variant of byte-pair encod-ing (bpe-dropout; @xcite and has a vocabulary of 50k tokens. ## evaluation to compare the performance of trelbert with other polish language models and twitter-specific multilingual models, we used the klej benchmark @xcite and the political advertising detection task @xcite . ## conclusion in this paper, we have introduced trelbert, the first polish language representation model pretrained on twitter data. it achieves state-of-theart results in a cyberbullying detection task, outperforming all existing solutions, including larger general-domain polish models, as well as multilingual in-domain models. additionally, we contribute by releasing a harmful speech dataset with labeled tweet ids which could be used as an alternative test set for cyberbullying detection.