V log QA : Task, Dataset, and Baseline Models for V ietnamese Spoken-Based Machine Reading Comprehension (2024)

Abstract: This paper presents the development process of a Vietnamese spoken language corpus for machine reading comprehension (MRC) tasks and provides insights into the challenges and opportunities associated with using real-world data for machine reading comprehension tasks. The existing MRC corpora in Vietnamese mainly focus on formal written documents such as Wikipedia articles, online newspapers, or textbooks. In contrast, the VlogQA consists of 10,076 question-answer pairs based on 1,230 transcript documents sourced from YouTube – an extensive source of user-uploaded content, covering the topics of food and travel. By capturing the spoken language of native Vietnamese speakers in natural settings, an obscure corner overlooked in Vietnamese research, the corpus provides a valuable resource for future research in reading comprehension tasks for the Vietnamese language. Regarding performance evaluation, our deep-learning models achieved the highest F1 score of 75.34% on the test set, indicating significant progress in machine reading comprehension for Vietnamese spoken language data. In terms of EM, the highest score we accomplished is 53.97%, which reflects the challenge in processing spoken-based content and highlights the need for further improvement.

## introduction machine reading comprehension (mrc) is a natural language processing (nlp) task that requires machines to comprehend a given context to answer a question @xcite . although there are numerous datasets available for mrc tasks in english @xcite , existing datasets for reading comprehension tasks in vietnamese are relatively limited and they have primarily focused on written documents, such as wikipedia articles, textbooks, and online news articles. spoken language represents an important and distinct domain that has not been fully explored. spoken language exhibits unique characteristics such as slang, regional variations, and informal grammar structures that can present significant challenges for machine learning models. as a result of that, reading comprehension tasks that involve spoken language, which is closer to everyday language, require a different type of dataset. to address this need, we introduce vlogqa -a new vietnamese spoken language corpus for reading comprehension tasks originating from transcripts of youtube vlogs. as a global online videosharing and social media platform, youtube provides a vast amount of spoken language data in natural settings. it is now the second-most visited website foot_0 in the world (and vietnam) and the second-biggest social network with over 2.5 billion monthly users foot_1 . starting with youtube, we aim to establish a solid foundation and gain insights into language patterns. this initial experiment serves as a stepping stone, and if successful, additional platforms catering to diverse audiences can be subsequently incorporated into further research. the dataset contains 10,076 manually annotated question-answer pairs based on 1,230 transcript documents extracted from youtube videos. besides, we provide several baseline models and evaluate them on our new dataset to test the ability of computers to understand the spoken text in vietnamese. overall, this paper makes the following contributions: the paper is structured as follows. section 2 discusses existing studies. section 3 is about corpus creation and its statistics. while section 4 presents information about the language models to be used; the experimental results of human and language models, plus error analysis on the corpus are presented in section 5. finally, section 6 provides conclusions and directions for future work. ## related works uit-viquad @xcite ) is a spandetection dataset for the machine reading comprehension (mrc) task in vietnamese, containing 23,074 questions on 5,109 passages acquired from vietnamese wikipedia articles. this dataset is widely used as a benchmark in vietnamese mrc research and has facilitated innovations in the field. its later version, uit-viquad 2.0 @xcite , includes 9,217 additional unanswerable questions, which addresses a limitation of extractive mrc models that struggle to identify answers that are not explicitly mentioned in the text. building upon the foundation of uit-viquad, @xcite is a sentencedetection dataset converted from uit-viquad and is designed for tasks that focus on sentence-level comprehension. in the health domain, vinewsqa @xcite is a dataset comprising 22,057 questions on 4,416 online health articles from a popular newspaper in vietnam. apart from span-detection datasets, there are other types of question-answering datasets available. vimmrc @xcite is the first vietnamese multiple-choice qa dataset, containing 2,783 four-choice questions based on 417 reading passages from vietnamese literature textbooks. the second version of vimmrc @xcite introduces 699 reading passages and 5,273 questions with variable numbers of choices. uit-vicov19qa @xcite utilizes online faq documents from trusted healthcare organizations to address covid-19-related questions, and is introduced as the first community-based qa dataset in vietnamese with a total of 4500 questions. vimqa @xcite is a wikipedia-based multi-hop dataset that provides over 10,000 questions designed to challenge models to perform complex multi-hop reasoning tasks, requiring them to refer to multiple evidence passages and perform explainable reasoning. the availability and diversity of quality questionanswering datasets are essential for the development of effective machine-learning models for natural language processing tasks. spoken squad @xcite ) is an english dataset that targets spoken content comprehension in the context of wikipedia articles. it is derived from squad @xcite and employs text-to-speech tools to generate the spoken context. similarly, the odsqa @xcite dataset focuses on spoken data and is based on the delta reading comprehension dataset (drcd) @xcite , a chinese contains 30.000+ questions from 2,108 wikipedia articles. however, unlike spoken squad, odsqa's audio is generated by humans. in summary, current vietnamese mrc datasets have mainly concentrated on formal types of content, such as wikipedia articles, textbooks, and online news articles. while there are spoken-based question-answering datasets available in other languages, such as spoken squad and odsqa, they are still limited to wikipedia content. ## models for reading comprehension transformer @xcite is a type of neural network architecture designed to process sequential data. in this paper, we carry out the mrc task and evaluate performance on the following group of transformer-based pre-trained language models: the information about the size of pre-trained language models, the hyperparameter settings, and the environment for experiments are shown in appendix a.3. ## conclusion and future works this paper presents vlogqa -a new vietnamese reading comprehension corpus for spoken context. the corpus consists of 10,076 question-answer pairs generated by humans, sourced from 1,230 transcripts of youtube vlogs. each transcript has an average length of 2,752 words. in terms of question types, the dataset is predominantly composed of what-questions, accounting for 47.52% of the corpus. this is followed by how-questions, which make up 32.57% of the dataset. other question types represented in the corpus include when, who, and why, among others. our experimental results indicate that the annotation of the dataset is acceptably consistent, with an average inter-rater agreement of nearly 44%. the performance of the state-of-the-art multilingual model is comparable with humans in both f1-score and em metrics; however, we believe that there is still room for improvement. in future work, we plan to enhance the corpus both in quality and quantity. we will explore techniques for improving the consistency of annotations and seek to expand the dataset with additional transcripts, spanning more topics. we also plan to augment the corpus with unanswerable questions, which will enable further exploration of machine capabilities. overall, this new vietnamese reading comprehension corpus for spoken context provides a valuable resource for researchers and practitioners in the field of natural language processing. moreover, we anticipate this dataset will facilitate advancements in vietnamese language understanding and provide a benchmark for the evaluation of intelligent question-answering systems on human-spoken language. furthermore, this corpus will enable the development of smart systems capable of retrieving valuable information from spoken language, thus contributing to the advancement of the field.