Revealing Weaknesses of V ietnamese Language Models Through Unanswerable Questions in Machine Reading Comprehension (2023)

Abstract: Although the curse of multilinguality significantly restricts the language abilities of multilingual models in monolingual settings, researchers now still have to rely on multilingual models to develop state-of-the-art systems in Vietnamese Machine Reading Comprehension. This difficulty in researching is because of the limited number of high-quality works in developing Vietnamese language models. In order to encourage more work in this research field, we present a comprehensive analysis of language weaknesses and strengths of current Vietnamese monolingual models using the downstream task of Machine Reading Comprehension. From the analysis results, we suggest new directions for developing Vietnamese language models. Besides this main contribution, we also successfully reveal the existence of artifacts in Vietnamese Machine Reading Comprehension benchmarks and suggest an urgent need for new high-quality benchmarks to track the progress of Vietnamese Machine Reading Comprehension. Moreover, we also introduced a minor but valuable modification to the process of annotating unanswerable questions for Machine Reading Comprehension from previous work. Our proposed modification helps improve the quality of unanswerable questions to a higher level of difficulty for Machine Reading Comprehension systems to solve.

## introduction machine reading comprehension (mrc) is a challenging research field in natural language processing, in which systems learn to predict answers for the questions inputted by users given a relevant context. mrc has many real-world applications such as open domain question answering @xcite and conversational question answering @xcite . thanks to the rapid development of pre-trained large language models, performances of mrc systems show substantial progress. pre-trained large language models are typically deep learning models designed based on the architecture of the transformers model @xcite . these models are pre-trained on very large text corpora using unsupervised tasks such as masked language model and next sentence prediction @xcite . after the pre-training phase, researchers can leverage the language understanding of these models by fine-tuning them on downstream tasks such as mrc. after being fine-tuned, these language models can achieve state-of-the-art performances on many benchmarks. researchers also pre-train multilingual models which are transformers-based models pre-trained with text corpora in over 100 languages @xcite ; @xcite . although multilingual models do not rely on direct crosslingual supervision while being pre-trained, they can achieve surprisingly high performances on different tasks in multilingual settings. besides, these multilingual models also excel in monolingual settings, especially in low-resource languages, where the number of high-quality works in developing monolingual language models is still limited. however, the abilities of multilingual language models are restricted by the curse of multilinguality @xcite : pre-training a multilingual model with a fixed capacity on an increasing number of languages only improves its performances up to a certain point. therefore, pre-trained multilingual models often show many language weaknesses compared to monolingual counterparts in monolingual settings. following the success of pre-trained models in english @xcite , researchers all over the world carry out many highquality works in pre-training monolingual language models such as camembert @xcite in french, gelectra @xcite in german, and phobert (nguyen and tuan nguyen, 1 2020) in vietnamese. these monolingual models also achieve state-of-the-art performance on numerous benchmarks, directly empowering the field of natural language processing to develop in their respective languages. facilitated by the development of pre-trained language models, mrc has recently also shown great progress in many languages. for example, roberta @xcite , camembert @xcite and gelectra @xcite achieve near human performances on squad @xcite @xcite and germanquad @xcite , respectively. however, for other low-resource languages, such as vietnamese, the performances of pre-trained language models are significant far lower than that of humans @xcite . we can explain these difficulties in research by the underdevelopment of vietnamese monolingual language models. as a result, most researchers @xcite @xcite in vietnamese mrc have to use multilingual models, which have many limitations in monolingual settings, as the cores of their mrc systems. the difficulties that vietnamese mrc researchers encounter, together with the limited number of works on vietnamese monolingual models, suggest that more high-quality research into vietnamese monolingual models is urgently needed. therefore, in order to suggest new directions for these future works, we attempt to reveal the language weaknesses of monolingual models by analyzing the performances of monolingual models in comparison with those of multilingual ones. in this work, we choose to investigate the performances of models on mrc because it is a suitable task for exploring the weaknesses of language models from multiple linguistic aspects. mrc allows us to examine the performance of models on lexical aspects, single-sentence level aspects, and multi-sentence level aspects of natural language. for instance, in order to answer "who" questions, mrc models must be competent in recognizing the person's name in a sentence, demonstrating their proficiency in named entity recognition. besides, to fully understand the given context, mrc models are expected to acquire extraordinary reading comprehension skills such as coreference resolution and bridging, which are part of the multisentence level aspects of language understanding. we focus our analysis on unanswerable questions because unanswerable questions proposed by @xcite are much more challenging than answerable questions in the same dataset, which directly creates more materials for us to reveal the language weaknesses of models. additionally, since @xcite proposed a novel method for annotating unanswerable questions, which involves instructing annotators to use various techniques to transform answerable questions into unanswerable ones instead of generating unanswerable questions from scratch, uit-viquad 2.0 has successfully introduced many new types of unanswerable questions. therefore, we have a more diverse range of language aspects to analyze the performances of models on. our contributions are summed as follows: multilingual versus monolingual models. @xcite probe an empirical analysis on monolingual berts and mbert across six languages and five different lexical tasks. they show that monolingual bert encodes significantly more lexical information than mbert. besides, @xcite compare pre-trained multilingual language models with monolingual counterparts regarding their monolingual task performances in nine languages and five tasks to reveal the reason for the gap between the performances of monolingual models and multilingual models. this comprehensive analysis later reveals that while pre-training data size played a vital role in the performances of language models on downstream tasks, the monolingual tokenizers designed by native speakers are also an important reason for the high performances of monolingual models in single-language settings. results from this analysis show that @xcite significantly contributed to the development of vietnamese language models with a high-quality tokenizer that is suitable for the unique linguistic features of vietnamese. ## analysis on uit-viquad 2.0 4.1 overall performance ## conclusion in this paper, we present the first comprehensive analysis to reveal the weaknesses of stateof-the-art vietnamese language models. our experiments show that while vietnamese language models demonstrate good lexical and grammatical abilities in vietnamese, they show inferior performances when questions require high-level semantic knowledge to successfully identify the unanswerability. this general result from our analysis shows that the inferior performances of vietnamese language models on machine reading comprehension task are mainly due to its inferior ability in grasping the big "picture" of the given context. besides, our analysis also show that vietnamese mrc benchmarks overestimate the comprehension skills of models in some language aspects, so stateof-the-art performances on mrc benchmarks does not accurately reflect the progress of vietnamese machine reading comprehension.