B angla RQA : A Benchmark Dataset for Under-resourced B angla Language Reading Comprehension-based Question Answering with Diverse Question-Answer Types (2022)

Abstract: High-resource languages, such as English, have access to a plethora of datasets with various question-answer types resembling real-world reading comprehension. However, there is a severe lack of diverse and comprehensive question-answering datasets in under-resourced languages like Bangla. The ones available are either translated versions of English datasets with a niche answer format or created by human annotations focusing on a specific domain, question type, or answer type. To address these limitations, this paper introduces BanglaRQA, a reading comprehension-based Bangla question-answering dataset with various question-answer types. BanglaRQA consists of 3,000 context passages and 14,889 question-answer pairs created from those passages. The dataset comprises answerable and unanswerable questions covering four unique categories of questions and three types of answers. In addition, this paper also implemented four different Transformer models for question-answering on the proposed dataset. The best-performing model achieved an overall 62.42% EM and 78.11% F1 score. However, detailed analyses showed that the performance varies across question-answer types, leaving room for substantial improvement of the model performance. Furthermore, we demonstrated the effectiveness of BanglaRQA as a training resource by showing strong results on the bn_squad dataset. Therefore, BanglaRQA has the potential to contribute to the advancement of future research by enhancing the capability of language models. The dataset and codes are available at https://github.com/sartajekram419/BanglaRQA

## introduction reading comprehension is the ability to process the information gained from reading a context passage, comprehend the meaning of both the context passage and the question asked, and then respond based on the reader's understanding and knowledge of the topic @xcite . this process also involves determining which questions cannot be answered based on the context. therefore, reading comprehension is widely recognized as a crucial test for evaluating humans' and machines' natural language comprehension. reading comprehension-based questionanswering generally comprises three parts: context, question, and answer. each of them has a wide range of types and formats. in the case of the english language, a large number of distinct datasets based on reading comprehension have been developed in order to capture the diversity of these components. for example, datasets in english are accessible with a single document as context @xcite , multiple documents as context @xcite , fill in the blank type questions @xcite , questions in natural language @xcite , single span-based replies @xcite , answers in natural language @xcite , and so forth. bangla is the world's seventh most frequently spoken language, as over 230 million people speak it in bangladesh and india @xcite . although bangla is a rich and diverse language, it is severely under-resourced for natural language processing. this is mostly attributed to the scarcity of necessary resources, such as labeled datasets, language models, and effective machine learning (ml) techniques for a variety of different nlp applications. a few datasets are available for reading comprehension-based question-answering in bangla. these datasets, however, are either translated versions of the english datasets @xcite , or very small datasets that focus on a specific topic area, such as general knowledge @xcite or only for answer type @xcite . this illustrates the need for a diverse and high-quality dataset for nlp research in bangla for question-answer based on reading comprehension. to overcome these challenges, this paper presents banglarqa, a benchmark dataset for bangla question-answering based on reading comprehension that contains a wide variety of question-answer types. this dataset comprises 3000 context passages covering a wide range of domains with 14,889 question-answer pairs. out of the 14,889 questions, 3,631 questions were unanswerable from their respective passages. the unanswerable questions are constructed in such a way that they seem pertinent to the passages to which they belong. this mix of answerable and unanswerable questions trains language models when to answer and when not to respond, resulting in improved linguistic ability. furthermore, the dataset includes a wide variety of question types, and these question types are separated into four categories, as illustrated in figure 1 : factoid, causal, confirmation, and list. consequently, it guarantees that the dataset contains different challenges to answering different types of questions. for the answerable questions, the answers can be classified into one of three groups: single span, multiple spans, or yes/no, covering the extractive question-answering domain. multiple-span answers enable information to be accumulated from different parts of the context passage as the answer. yes/no answers require inference skills based on the passage's context making the dataset more robust. to estimate the difficulty of banglarqa, this study also fine-tuned four different pre-trained transformers models, namely, banglat5 (bhattacharjee et al., 2022b), mt5 @xcite , banglabert @xcite , mbert @xcite . banglat5, the bestperforming model, achieved an average of 62.42% em and 78.11% f1 score on the test set. however, the em and f1 scores were lower for some specific question-answer types (detailed analysis is provided in section: 5), indicating some of our dataset's challenges. after training banglat5 on our dataset, we tested it on the previously available bn_squad dataset (a translated version of squad 2.0) (bhattacharjee et al., 2022a), yielding 70.20% em and 75.79% f1 score. previously @xcite , when banglat5 was trained and tested on the bn_squad dataset, it yielded 68.49% em and 74.77% f1 score. this proves that banglarqa is a valuable resource for training language models by demonstrating the model's capacity to generalize to bn_squad. contributions. our contributions are the following: ## related work this section presents existing works for reading comprehension-based question-answering datasets in english and bangla. ## banglarqa dataset this section explains the whole data collection process of banglarqa, e.g., the source of the data, the criteria for data inclusion-exclusion, instructions given to the annotators, and other details. in addition, a comprehensive analysis of the dataset is included in this section. ## experimental setup the task is reading comprehension-based question-answering, where the model is given the question and its associated context passage as input. the model outputs answers in text format. if the question is unanswerable, then the output is an empty string. four different models were implemented: banglat5, mt5, banglabert, and mbert. this section explains the whole pipeline of the experiments, from preprocessing the data to model training and evaluation.