A V ietnamese Dataset for Evaluating Machine Reading Comprehension (2020)

Abstract: Over 97 million inhabitants speak Vietnamese as the native language in the world. However, there are few research studies on machine reading comprehension (MRC) in Vietnamese, the task of understanding a document or text, and answering questions related to it. Due to the lack of benchmark datasets for Vietnamese, we present the Vietnamese Question Answering Dataset (UIT-ViQuAD), a new dataset for the low-resource language as Vietnamese to evaluate MRC models. This dataset comprises over 23,000 human-generated question-answer pairs based on 5,109 passages of 174 Vietnamese articles from Wikipedia. In particular, we propose a new process of dataset creation for Vietnamese MRC. Our in-depth analyses illustrate that our dataset requires abilities beyond simple reasoning like word matching and demands complicate reasoning such as single-sentence and multiple-sentence inferences. Besides, we conduct experiments on state-of-the-art MRC methods in English and Chinese as the first experimental models on UIT-ViQuAD, which will be compared to further models. We also estimate human performances on the dataset and compare it to the experimental results of several powerful machine models. As a result, the substantial differences between humans and the best model performances on the dataset indicate that improvements can be explored on UIT-ViQuAD through future research. Our dataset is freely available to encourage the research community to overcome challenges in Vietnamese MRC.

## introduction machine reading comprehension (mrc) is an understanding natural language task that requires computers to understand a text and then answer questions related to it. mrc is an essential core for a range of natural language processing applications such as search engines and intelligent agents (alexa, google assistant, siri, and cortana) in order to evaluate mrc models, gold standard resources with questionanswer pairs based on documents have to be collected or created by human. building a benchmark dataset plays a vital role in evaluating natural language processing models, especially for a low-resource language like vietnamese. typical gold standard mrc resources for english are span-extraction mrc datasets @xcite @xcite , cloze-style mrc datasets @xcite @xcite , multiple-choice mrc datasets @xcite and conversation-based mrc datasets @xcite . for other languages, there are the chinese dataset of the span-extraction mrc @xcite , the traditional chinese dataset of mrc @xcite , the chinese user-query-log-based dataset of dureader @xcite , and the korean mrc dataset @xcite . due to the rapid development of the reading comprehension datasets, various neural network-based models have been proposed and made a significant advancement in this research field such as match-lstm @xcite , bidaf @xcite , r-net @xcite , drqa @xcite , fusionnet @xcite , fastqa @xcite , and qanet @xcite . pre-trained language models, bert @xcite and xlm-r @xcite have recently become extremely popular and achieved state-of-the-art performances for mrc tasks. vietnamese is a language with few resources for natural language processing. the dataset for mrc introduced by @xcite consists of 2,783 multiple-choice questions and answers based on a set of 417 vietnamese texts which are used for evaluating the reading comprehension skill for 1 st to 5 th graders. however, this dataset is relatively small in size to evaluate deep learning models for the vietnamese mrc. thus, we aim to build a new large dataset for evaluating vietnamese mrc. though the deep learning approach has surpassed the human performance on the squad @xcite and newsqa @xcite datasets, we wonder if these state-of-the-art models could also achieve similar performances on datasets of different languages. to further enhance the development of the mrc, we build a span-extraction mrc dataset where answers to questions are always spans from a given text for vietnamese. figure 1 shows several examples for vietnamese span-extraction reading comprehension. in this study, we have four main contributions described as follows. passage: nước biển có độ mặn không đồng đều trên toàn thế giới mặc dù phần lớn có độ mặn nằm trong khoảng từ 3,1% tới 3,8%. khi sự pha trộn với nước ngọt đổ ra từ các con sông hay gần các sông băng đang tan chảy thì nước biển nhạt hơn một cách đáng kể. nước biển nhạt nhất có tại vịnh phần lan, một phần của biển baltic. (english: seawater has uneven salinity throughout the world although most salinity ranges from 3.1% to 3.8%. when the mix with freshwater pouring from rivers or near glaciers is melting, the seawater is significantly lighter. the lightest seawater is found in the gulf of finland, a part of the baltic sea.) question: độ mặn thấp nhất của nước biển là bao nhiêu? (english: what is the lowest salinity of seawater?) answer: 3.1% (english: 3.1%) question: nước biển ở đâu có hàm lượng muối thấp nhất? (english: where is the lowest salt content?) answer: vịnh phần lan. (english: gulf of finland.) the rest of this paper is structured as follows. section 2 reviews existing datasets. section 3 introduces the creation process of our dataset. in-depth analyses of our dataset are presented in section 4. then section 5 presents our experiments and analysis results. finally, section 6 presents conclusions and directions for future work. squad is one of the most popular english datasets of the span-based mrc. @xcite . proposed squad v1.1 created by crowd-workers on 536 wikipedia articles with 107,785 questionanswer pairs. squad v2.0 @xcite was released with adding over 50,000 unanswerable questions created adversarially by crowd-workers according to the original ones. newsqa is another english dataset proposed by @xcite , consisting of 119,633 question-answer pairs generated by crowd-workers on 12,744 news articles from cnn news. this dataset is similar to squad because the answer to each question is a text segment of arbitrary length in the corresponding news article. cmrc @xcite ) is a span-extraction dataset for chinese mrc introduced in the second evaluation workshop on chinese machine reading comprehension 2018, comprising approximately 20,000 human-annotated questions on wikipedia articles. korquad @xcite is a korean dataset for span-based mrc, consisting of over 70,000 human-generated question-answer pairs on korean wikipedia articles. these datasets are studied in the development and evaluation of various deep neural network models in nlp, such as match-lstm @xcite , bidaf @xcite , r-net @xcite , drqa @xcite , fusionnet @xcite , fastqa @xcite and @xcite . most recently, bert @xcite and xlm-r @xcite , which are powerful models trained on multiple languages, have obtained state-of-the-art performances on mrc datasets. until now, there has not been any datasets of vietnamese wikipedia texts for span-based mrc research. as mentioned above, the datasets are benchmarks for the mrc task and may be used for organizing a challenge which encourages researchers to explore the best processing models. therefore, this is our primary motivation to create the new dataset for vietnamese mrc. ## dataset creation in this section, we introduce our proposed process of mrc dataset creation for the vietnamese language. in particular, we build our uit-viquad dataset through five phases consisting of worker recruitment, passage collection, question-answer sourcing, validation and additional answers collection. these phases are described in detail as follows. question-answer creating created dataset self-checking selfchecked dataset cross-checking crosschecked dataset data for test set data for dev set training set worker recruitment additional answers test set dev set phase 2 phase 3 phase 1 phase 4 phase 5 guidelines phase 1 -worker recruitment: the quality of a dataset depends on high-quality workers and the process of data creation. in this section, we present worker recruitment for creating our dataset according to a rigorous process, consisting of four different stages. (1) people apply to become workers for creating answer-question pairs of the dataset; (2) selected people are excellent at general knowledge and passed our reading comprehension test; (3) official workers are carefully trained over 500 question-answer pairs and cross-checked their created data to detect common mistakes that can be avoided when creating data. phase 2 -passage collection: similar to squad, we also use project nayuki's wikipedia's internal pageranks 3 to obtain a set of the top 5,000 vietnamese articles, from which we choose randomly 151 articles for dataset creation. each passage corresponds to a paragraph in an article. images, figures, and tables are excluded. we also delete passages shorter than 300 characters or containing many special characters and symbols. phase 3 -question-answer sourcing: workers comprehend each passage and then create questions and corresponding answers. during the question and answer creation, workers follow rules which are: phase 4 -question and answer validation: in this phase, workers perform two different sub-phases to check mistakes in question-answer pairs including self-checking and cross-checking. the mistakes are classified into five different categories: unclear questions, misspellings, incorrect answers, lack or excess of information in answers, and incorrect-boundary answers. the two sub-phases are described as follows. ## empirical evaluation in this section, we conduct experiments with the state-of-the-art mrc models to evaluate our dataset. to measure the difficulty of our dataset, we also estimate human performance on the task of vietnamese mrc. similar to evaluations on english and chinese datasets @xcite , we used two evaluation metrics, exact match (em) and f1-score, to evaluate performances of mrc models on our dataset. ## conclusion and future work in this paper, we introduce a new span-extraction dataset for evaluating vietnamese mrc. uit-viquad contains over 23,000 questions generated by humans. our experimental results show that the machines could obtain up to 87 percent scores on both the development and test set. however, they are lower than the estimated human performances in f1-score. we hope the release of our dataset contributes to the language diversity in mrc task, and accelerates further investigation on solving difficult questions that need comprehensive reasoning over multiple clues. according to the analysis results, we may extend this work by exploring models to solve challenging questions involving specific question types (where, why, and how), answer types (location, and noun phrases) and reasoning types (single-sentence inference, multiple-sentence inference, ambiguous or insufficient). in future, we plan to enhance the quantity and the quality of our dataset to achieve better performance on deep learning and transformer models. in addition, we would like to open the vietnamese mrc challenging task for researchers in the field.