V i D e BERT a: A powerful pre-trained language model for V ietnamese (2023)

Abstract: This paper presents ViDeBERTa, a new pre-trained monolingual language model for Vietnamese, with three versions - ViDeBERTa_xsmall, ViDeBERTa_base, and ViDeBERTa_large, which are pre-trained on a large-scale corpus of high-quality and diverse Vietnamese texts using DeBERTa architecture. Although many successful pre-trained language models based on Transformer have been widely proposed for the English language, there are still few pre-trained models for Vietnamese, a low-resource language, that perform good results on downstream tasks, especially Question answering. We fine-tune and evaluate our model on three important natural language downstream tasks, Part-of-speech tagging, Named-entity recognition, and Question answering. The empirical results demonstrate that ViDeBERTa with far fewer parameters surpasses the previous state-of-the-art models on multiple Vietnamese-specific natural language understanding tasks. Notably, ViDeBERTa_base with 86M parameters, which is only about 23% of PhoBERT_large with 370M parameters, still performs the same or better results than the previous state-of-the-art model. Our ViDeBERTa models are available at: https://github.com/HySonLab/ViDeBERTa .

## introduction in recent years, pre-trained language models (plms) and transformer-based architecture models have been essential in the advancement of natural language processing (nlp). large-scale transformer-based pre-trained models with the capacity to derive a contextual representation of the languages in the training data include gpt @xcite , bert @xcite , roberta @xcite , xl-net @xcite , electra @xcite , t5 @xcite , and deberta @xcite . following pre-training, these models performed at the cutting edge on various downstream nlp tasks @xcite . the development of pre-trained models in other languages, including vietnamese (phobert @xcite ; @xcite ; vit5 @xcite ), and arabic @xcite , has been spurred on by the success of pretrained models in english. in order to enhance performance across several languages by learning both general and language-specific representations, multilingual pre-trained models ( xlm-r @xcite , mt5 @xcite , and mbart @xcite are also being developed. most recently, phobert @xcite , the first large pre-trained model for vietnamese that inherits the roberta @xcite architecture, has demonstrated the effectiveness of the trained language model compared with current methods modernized in four vietnamesespecific tasks, including part of speech tagging (pos), dependency parsing, named entity recognition (ner), and natural language inference (nli). nevertheless, there are still rooms to build an improved pre-trained language model for vietnamese. firstly, phobert was pre-trained on a relatively small vietnamese dataset of 20gb of uncompressed texts, while pre-trained language models can be significantly improved by using more pre-training data @xcite . secondly, question answering (qa) is one of the most impactful tasks that has mainly focused on the computational linguistics and artificial intelligence research community within information retrieval and information extraction in recent years. however, there are a few pre-trained models for vietnamese that produce efficient results in the qa tasks, especially phobert @xcite and @xcite . last but not least, some previous works point to deberta architecture @xcite using several novel techniques that can significantly outperform roberta and improve the efficiency of model pre-training and the performance of downstream tasks in some respects. inspired by that, we introduce an improved largescale pre-trained language model, videberta, trained on cc100 vietnamese monolingual, following the architecture and pre-training methods of debertav3 @xcite . we comprehensively evaluate and compare our model with competitive baselines, i.e., the previous sota models phobert, vit5, and the multilingual model xlm-r on three vietnamese downstream tasks, including pos tagging, ner, and qa. in this work, we focus on two main categories of qa: machine reading comprehension (mrc) and open-domain question answering (odqa). the experiment results show the performance of our model surpasses all baselines on all tasks. our main contributions are summarized as follows: pre-trained language models for vietnamese. phobert @xcite is the first large-scale plm for vietnamese, which has the same architecture as bert @xcite and the same pre-training approach as roberta @xcite for more robust performance. this fine-tuning tasks. this work utilizes three vietnamese natural language understanding (nlu) tasks, including pos tagging, ner, and qa, for fine-tuning and evaluating our model's performance. for pos tagging and ner, phobert still produces better results than vielectra, phonlp, and vit5 @xcite @xcite . while early qa @xcite @xcite systems were commonly complex and had many parts, mrc models have evolved and now suggest a simpler two-stage retriever-reader framework @xcite . a context retriever first selects a small subset of passages where some of them contain the answer to the question then a machine reader can carefully review the retrieved contexts and determine the correct answer. the tasks based on qa have gained much attention in recent years in the vietnamese natural language processing and computational linguistics community. however, to the best of our knowledge, there is only the work @xcite that proposes the first vietnamese retriever-reader qa system employing a transformer-based model (xlm-r) evaluated on the viquad corpus @xcite . and sentence segmentation using a vietnamese toolkit pyvi foot_1 on the pre-training dataset. after that, we use a pre-trained sentencepiece tokenizer from debertav3 @xcite to segment these sentences with sub-word units, which have a vocabulary of 128k sub-word types. ## conclusion in this paper, we have introduced videberta, a new pre-trained large-scale monolingual language model for vietnamese. we demonstrate the effectiveness of our videberta by showing that vide-berta with fewer parameters performs better than the recent strong pre-trained language models as xlm-r, phobert, and vit5, and achieves sota performances for three downstream vietnamese language understanding tasks, including pos tagging, ner, and especially qa. we hope that our public videberta model will boost ongoing nlp research and applications for vietnamese and other low-resource languages.