V i GLUE : A V ietnamese General Language Understanding Benchmark and Analysis of V ietnamese Language Models (2024)

Abstract: As the number of language models has increased, various benchmarks have been suggested to assess the proficiency of the models in natural language understanding. However, there is a lack of such a benchmark in Vietnamese due to the difficulty in accessing natural language processing datasets or the scarcity of task-specific datasets. **ViGLUE**, the proposed dataset collection, is a **Vi**etnamese **G**eneral **L**anguage **U**nderstanding **E**valuation benchmark developed using three methods: translating an existing benchmark, generating new corpora, and collecting available datasets. ViGLUE contains twelve tasks and encompasses over ten areas and subjects, enabling it to evaluate models comprehensively over a broad spectrum of aspects. Baseline models utilizing multilingual language models are also provided for all tasks in the proposed benchmarks. In addition, the study of the available Vietnamese large language models is conducted to explore the language models’ ability in the few-shot learning framework, leading to the exploration of the relationship between specific tasks and the number of shots.

## introduction since the introduction of the transformer architecture @xcite and its variations, there has been significant progress in many natural language processing tasks. the expansion of pre-trained language models leveraging that design, such as bert @xcite , gpt @xcite , and t5 @xcite , is primarily responsible for this progress. besides that, the need to evaluate such models for natural language understanding has been raised. glue @xcite and superglue @xcite are introduced as well-designed benchmarks to evaluate english models in nlu, lacking the ability to analyze models in other languages. consequently, particular benchmarks for other languages have been proposed, such as the clue benchmark @xcite for chinese, the flue benchmark @xcite for french, the klue benchmark @xcite for korean, the russiansuper-glue @xcite for russian, or the in-dicglue @xcite for indian. they are developed for assessing the performance of language-specific pre-trained language models. while numerous benchmarks exist for examining the nlu capabilities of language models, there is a lack of comparable benchmarks in vietnamese, particularly those that are openly accessible for instant usage. therefore, this study aims to address this gap by developing and introducing a comprehensive benchmark designed to evaluate the nlu capabilities of language models in vietnamese. creating such benchmarks is crucial for assessing the performance of language models in vietnamese and fostering advancements in natural language processing specific to this language. to establish such a benchmark, it is necessary to carefully choose and organize a wide range of tasks that encompass different linguistic features, contextual complexities, topic diversity, and domain variety. this study introduces viglue as an evaluation framework over twelve nlu tasks, detailed in in conclusion, the contributions are: ## related work in natural language processing, evaluating the ability of the language models on natural language understanding is necessary. with the birth of pretrained transformer-architecture modes such as bert @xcite and gpt @xcite , glue @xcite and superglue @xcite appear as standard benchmarks to test the methods on different aspects. the glue @xcite dataset focuses on text genres and degrees of difficulty. at the same time, super-glue @xcite is an extended version of glue @xcite with improvement in difficulty and novel tasks. however, while glue and superglue focus on english, using them to evaluate large language models that are created for other languages would be a noticeable problem. in response to this need, many research groups spent effort on creating comparable datasets in different languages, utilizing the building pipeline of glue and superglue, while these evaluation frameworks just assessed models in english. consequently, plenty of benchmarks for languagespecific natural language understanding were developed. the clue benchmark @xcite was designed with nine various chinese nlu tasks, constructed from multiple resources (chinese news, app description, etc.). the flue benchmark @xcite was developed for the french language. the specialty of this dataset is that it includes three over six tasks from cross-lingual datasets. moreover, the klue benchmark @xcite is created with two goals: covering diverse aspects of nlu in korean and minimizing redundancy among tasks. to construct those benchmarks, techniques such as data gathering (klue @xcite , clue @xcite ), re-using existing datasets (flue @xcite ) are employed. an alternative approach is to create new multilingual variances of the original dataset with human and machine translation. xnli @xcite , xcopa @xcite , xtreme @xcite , and the russiansuperglue @xcite are constructed in this way. however, this method depends on the quality of the translation tools @xcite and may not maintain the context of specific tasks. despite this problem, the translation method offers the advantage of utilizing the reliable pool of workers employed in the original datasets @xcite . furthermore, it has been observed that the hypotheses and semantic aspect of the samples remain consistent when applied to different languages @xcite . having such datasets like glue @xcite and superglue @xcite without any restrictions, free to access, and convenient to use for vietnamese would be crucial to evaluate large language models such as phogpt @xcite , vietcuna. even though there are some vietnamese datasets, such as the vietnamese students' feedback corpus (vsfc) @xcite , the vietnamese social media emotion corpus (vsmec) @xcite , and the covid-19 named entity recognition dataset for vietnamese @xcite . however, the availability of additional resources is restricted from direct open access due to the necessity of obtaining user approval, which is a significant inconvenience for researchers working with vietnamese nlp datasets. hence, a new vietnamese nlu benchmark is introduced. this dataset is constructed by utilizing the translating method, building from news sources, and collecting the existing datasets. nevertheless, the back-translation process is proposed to assure the translation quality of the translated texts in terms of both lexical and semantic elements. ## viglue overview viglue contains twelve tasks obtained by three methods: collecting, constructing, and translating. the task statistics are shown in ## experiments this section describes the baseline models used in the experiments conducted on the viglue benchmark and the analysis of evaluation results in the model's language understanding capability. ## few-shot learning with vietnamese large language models besides finetuning the pre-trained encoder-only transformer models, several vietnamese large language models, also known as autoregressive language models, are evaluated on viglue to compare their language understanding ability. this experiment applies the few-shot learning framework @xcite , also referred to as in-context learning, to the bloom model family @xcite and phogpt model @xcite . the advantage of this approach is eliminating the need for model finetuning, thereby reducing the hardware requirements associated with llms. additionally, the notation k denotes the number of samples in the context. from figure 2 in appendix c.1, phogpt outperforms bloom-7b1 on a small number of tasks. when using zero-shot learning (@xmath0, phogpt demonstrates superior performance compared to bloom-7b1 on six out of twelve tasks. similarly, bloom-7b1 gets better performance compared to phogpt on five out of twelve tasks for @xmath1, seven out of twelve tasks for @xmath2, and five out of twelve tasks for @xmath3. furthermore, it is evident that increasing the number of examples in the prompts adversely affects the model's efficacy, namely on cola, mnli, mrpc, and wnli. it is noticeable that zero-shot learning achieves the highest scores in cola, mnli, mrpc, qqp, and wnli, to other values of k even though it does not offer any task-specific instruction. nevertheless, the concept of increasing the number of examples resulting in a performance improvement remains valid for qnli and vnrte. the statistics for qqp, sst2, vsfc, and vsmec exhibit uneven fluctuations when the number of shots changes. figure 3 , in appendix c.2, visualizes the fewshot learning performance of the bloom model family. it is obvious that increasing k benefits the models on qnli for any model size. in contrast, it also shows that cola, mnli, mrpc, and qqp do not benefit from few-shot learning since the performance at @xmath4. regarding the model size, the largest model, bloom-176b, achieves roughly identical performance to other configurations on most workloads (cola, mnli, mrpc, and vtoc). it outperforms models with lower capacities only on qnli, rte, and sst2, while for other tasks, it cannot beat them. when the model size increases from 560m to larger, the performance also increases on several tasks, such as qnli, qqp, rte, sst2, vsfc, and vsmec. nevertheless, the bloom-560m model, which is the smallest in size, outperforms the other models in the bloom family in both vnrte and wnli tasks. on inference tasks, raising the number of parameters in multilingual language models sometimes fails to increase the performance. ## conclusion by providing viglue, the issue of missing a vietnamese natural language understanding benchmark is tackled. viglue is built using three methods, including translating available benchmarks, constructing new corpora, and gathering available vietnamese datasets. the baseline models finetuned on multilingual language models are provided for all viglue tasks and xlm-roberta large achieve the best performance on seven over twelve tasks compared to mbert and mdebertav3. besides exploring the encoder architecture, large language models are also examined using few-shot learning. we observe that on cola, mnli, mrpc, qqp, and wnli, the models perform better without any task instruction. in contrast, models achieve greater performance on qnli and vnrte when increasing the number of samples in the context. the number of shots employed in the experiments is restricted to only four values (0, 1, 2, and 4). for further investigation, experimenting with different and large values of the number of examples in the context is considered. in addition, tasks in viglue focus on short sentences or sentence pairs. future work should focus on long sentences or text at a higher level. finally, viglue covers news, wikipedia, textbooks, and publication domains. therefore, expanding the scope of viglue to medical, law, and education is a good direction.