V i S o BERT : A Pre-Trained Language Model for V ietnamese Social Media Text Processing (2023)

Abstract: English and Chinese, known as resource-rich languages, have witnessed the strong development of transformer-based language models for natural language processing tasks. Although Vietnam has approximately 100M people speaking Vietnamese, several pre-trained models, e.g., PhoBERT, ViBERT, and vELECTRA, performed well on general Vietnamese NLP tasks, including POS tagging and named entity recognition. These pre-trained language models are still limited to Vietnamese social media tasks. In this paper, we present the first monolingual pre-trained language model for Vietnamese social media texts, ViSoBERT, which is pre-trained on a large-scale corpus of high-quality and diverse Vietnamese social media texts using XLM-R architecture. Moreover, we explored our pre-trained model on five important natural language downstream tasks on Vietnamese social media texts: emotion recognition, hate speech detection, sentiment analysis, spam reviews detection, and hate speech spans detection. Our experiments demonstrate that ViSoBERT, with far fewer parameters, surpasses the previous state-of-the-art models on multiple Vietnamese social media tasks. Our ViSoBERT model is available only for research purposes. Disclaimer: This paper contains actual comments on social networks that might be construed as abusive, offensive, or obscene.

## introduction language models based on transformer architecture @xcite pre-trained on largescale datasets have brought about a paradigm shift in natural language processing (nlp), reshaping how we analyze, understand, and generate text. in particular, bert @xcite and its variants @xcite have achieved state-of-the-art performance on a wide range of downstream nlp tasks, including but not limited to text classification, sentiment analysis, question answering, and machine translation. english is moving for the rapid development of language models across specific domains such as medical @xcite , scientific @xcite , legal @xcite , political conflict and violence @xcite , and especially social media @xcite @xcite . vietnamese is the eighth largest language used over the internet, with around 85 million users across the world 5 . despite a large amount of vietnamese data available over the internet, the advancement of nlp research in vietnamese is still slow-moving. this can be attributed to several factors, to name a few: the scattered nature of available datasets, limited documentation, and minimal community engagement. moreover, most existing pre-trained models for vietnamese were primarily trained on large-scale corpora sourced from general texts @xcite @xcite . while these sources provide broad language coverage, they may not fully represent the sociolinguistic phenomena in vietnamese social media texts. social media texts often exhibit different linguistic patterns, informal language usage, non-standard vocabulary, lacking diacritics and emoticons that are not prevalent in formal written texts. the limitations of using language models pre-trained on general corpora become apparent when processing vietnamese social media texts. the models can struggle to accurately un-derstand and interpret the informal language, using emoji, teencode, and diacritics used in social media discussions. this can lead to suboptimal performance in vietnamese social media tasks, including emotion recognition, hate speech detection, sentiment analysis, spam reviews detection, and hate speech spans detection. we present visobert, a pre-trained language model designed explicitly for vietnamese social media texts to address these challenges. visobert is based on the transformer architecture and trained on a large-scale dataset of vietnamese posts and comments extracted from well-known social media networks, including facebook, tiktok, and youtube. our model outperforms existing pretrained models on various downstream tasks, including emotion recognition, hate speech detection, sentiment analysis, spam reviews detection, and hate speech spans detection, demonstrating its effectiveness in capturing the unique characteristics of vietnamese social media texts. our contributions are summarized as follows. ## fundamental of pre-trained language models for social media texts pre-trained language models (plms) based on transformers @xcite have become a crucial element in cutting-edge nlp tasks, including text classification and natural language generation. since then, language models based on transformers related to our study have been reviewed, including plms for vietnamese social media texts. ## visobert this section presents the architecture, pre-training data, and our custom tokenizer on vietnamese social media texts for visobert. ## result analysis and discussion in this section, we consider the improvement of our plm more compared to powerful others, including phobert and twhin-bert, in terms of different aspects. firstly, we investigate the effects of masking rate on our pre-trained model performance (see section 5.1). additionally, we examine the influence of social media characteristics on the model's ability to process and understand the language used in these social contexts (see section 5.2). lastly, we employed feature-based extraction techniques on task-specific models to verify the potential of leveraging social media textual data to enhance word representations (see section 5.3). ## conclusion and future work we presented visobert, a novel large-scale monolingual pre-trained language model on vietnamese social media texts. we illustrated that vi-sobert with fewer parameters outperforms recent strong pre-trained language models such as vibert, velectra, phobert, xlm-r, xlm-t, twhin-bert, and bernice and achieves stateof-the-art performances for multiple downstream vietnamese social media tasks, including emotion recognition, hate speech detection, spam reviews detection, and hate speech spans detection. we conducted extensive analyses to demonstrate the efficiency of visobert on various vietnamese social media characteristics, including emojis, teencodes, and diacritics. furthermore, our pre-trained language model visobert also shows the potential of leveraging vietnamese social media text to enhance word representations compared to other plms. we hope the widespread use of our opensource visobert pre-trained language model will advance current nlp social media tasks and applications for vietnamese. other low-resource languages can adopt how to create plms for enhancing their current nlp social media tasks and relevant applications.