VLUE : A New Benchmark and Multi-task Knowledge Transfer Learning for V ietnamese Natural Language Understanding (2024)

Abstract: The success of Natural Language Understanding (NLU) benchmarks in various languages, such as GLUE for English, CLUE for Chinese, KLUE for Korean, and IndoNLU for Indonesian, has facilitated the evaluation of new NLU models across a wide range of tasks. To establish a standardized set of benchmarks for Vietnamese NLU, we introduce the first Vietnamese Language Understanding Evaluation (VLUE) benchmark. The VLUE benchmark encompasses five datasets covering different NLU tasks, including text classification, span extraction, and natural language understanding. To provide an insightful overview of the current state of Vietnamese NLU, we then evaluate seven state-of-the-art pre-trained models, including both multilingual and Vietnamese monolingual models, on our proposed VLUE benchmark. Furthermore, we present CafeBERT, a new state-of-the-art pre-trained model that achieves superior results across all tasks in the VLUE benchmark. Our model combines the proficiency of a multilingual pre-trained model with Vietnamese linguistic knowledge. CafeBERT is developed based on the XLM-RoBERTa model, with an additional pretraining step utilizing a significant amount of Vietnamese textual data to enhance its adaptation to the Vietnamese language. For the purpose of future research, CafeBERT is made publicly available for research purposes.

## introduction recently, the vietnamese natural language processing (nlp) research community has achieved remarkable advancements in the development of pre-trained language models for the vietnamese language @xcite @xcite . the integration of these stateof-the-art models, coupled with the progress made in establishing high-quality benchmarks, has paved the way for a diverse array of applications within vietnam. notably, these advancements have greatly enhanced capabilities in areas of machine reading comprehension @xcite . unfortunately, despite the recent progress in developing large language models for vietnamese, the research community of vietnamese nlp lacks a common ground for evaluating the performance of these models. this lack of standard evaluation metrics and benchmarks makes it difficult to identify the strengths and weaknesses of different approaches in pre-training new models in vietnamese and the overall progress of vietnamese natural language understanding (nlu). as a result, it is crucial for the community to establish a shared set of evaluation metrics and benchmarks that can be used to assess newly proposed language models. inspired by benchmarks evaluating natural language understanding in other languages @xcite @xcite @xcite , in this paper, we propose vlue (vietnamese language understanding evaluation) as a shared set of evaluation metrics and benchmarks for pre-trained models in vietnamese. to the best of our knowledge, our proposed benchmark is the first benchmark for evaluating vietnamese nlu models. we believe that this benchmark will serve as a valuable resource for researchers and practitioners working in the field of vietnamese nlu, and will help drive further advancements in this area. to facilitate the development of new large language models in vietnamese, we, in this work, introduce vietnamese language understanding eval-uation (vlue), a comprehensive language understanding framework that includes five diverse tasks. the tasks include a wide range of applications (question answering, hate speech detection, part-of-speech, emotion recognition, and natural language inference), types of input (single sentences, pair of sentences, sequence of sentences) and objectives of tasks (extracted span, sentence classification, sequence labeling). with its diverse set of benchmarks, vlue establishes a standardized evaluation framework, enabling comprehensive comparisons and evaluations of different models in the context of vietnamese. within this paper, we commence by introducing our novel vlue benchmark, designed to evaluate the language prowess of various models. we conduct a comprehensive analysis of seven models, encompassing four multilingual models as well as three monolingual models. additionally, we present the introduction of a newly developed pretrained model, referred to as cafebert. this model is constructed by leveraging the large-scale xlm-roberta model and further fine-tuning it on an extensive vietnamese corpus, thereby enhancing its proficiency in the vietnamese language and elevating its overall performance. through indepth evaluation, we demonstrate that cafebert achieves state-of-the-art performance across all four tasks presented in our vlue benchmark. in this paper, we make the following contributions: the rest of this paper is structured as follows. section 2 reviews existing nlu benchmarks and pre-trained language models. section 3 introduces the nlu benchmark for vietnamese. in particular, we present experiments and benchmark result in section 4. then section 5 presents a new pretrained language model called cafebert. finally, section 6 presents conclusions and future work. ## related work in this paper, we review data benchmark and pretrained language models related to our work. ## cafebert the results from our analysis on current progress of vietnamese nlu show that the xlm-roberta large achieves the best performance on most tasks of vlue. however, phobert also show a comparable performance on tasks with corpus from social networks, such as vsmec and vihos. this observation drives us to a hypothesis that further adapting multilingual model xlm-roberta large into vietnamese can help improve its performance on vlue. we then propose a new model that is expected to combine the existing knowledge from xlm-roberta and the newly trained knowledge from vietnamese corpus. we continue pre-training xlm-roberta with a vietnamese dataset similar to the data used to train the phobert model. we refer to our proposed model as cafebert. ## conclusion and future works we proposed vlue -the first vietnamese language understanding evaluation benchmark. vlue is used to evaluate pre-trained models in vietnamese with various tasks such as reading comprehension, text classification, natural language inference, hate speech detection, and part-of-speech tagging. we also publicize a pre-trained model, cafebert, which is trained based on the xlm-roberta model with a vast vietnamese text dataset. we show that cafebert achieves sota performance on all vlue benchmark tasks and all vlue domains, such as social networks, wikipedia, and news. we expect vlue to be widely used to evaluate vietnamese-supported pre-trained models. the pre-trained models will be evaluated comprehensively on multiple tasks with different domains. the cafebert model will be applied to many tasks for vietnamese to improve performance and get many applications in the field of natural language processing in vietnamese. in addition, resourcepoor languages can monitor and work our way up to creating great pre-training models that can enhance performance and have many real-world applications.