A r T rivia: Harvesting A rabic W ikipedia to Build A New A rabic Question Answering Dataset (2023)

Abstract: We present ArTrivia, a new Arabic question-answering dataset consisting of more than 10,000 question-answer pairs along with relevant passages, covering a wide range of 18 diverse topics in Arabic. We created our dataset using a newly proposed pipeline that leverages diverse structured data sources from Arabic Wikipedia. Moreover, we conducted a comprehensive statistical analysis of ArTrivia and assessed the performance of each component in our pipeline. Additionally, we compared the performance of ArTrivia against the existing TyDi QA dataset using various experimental setups. Our analysis highlights the significance of often overlooked aspects in dataset creation, such as answer normalization, in enhancing the quality of QA datasets. Our evaluation also shows that ArTrivia presents more challenging and out-of-distribution questions to TyDi, raising questions about the feasibility of using ArTrivia as a complementary dataset to TyDi.

## introduction in recent years, the field of question-answering (qa) in arabic nlp has witnessed more attention with the introduction of several arabic qa datasets, such as tydi qa @xcite , the arabic reading comprehension dataset (arcd) @xcite , and the arabic question-answer dataset (aqad) @xcite . however, existing arabic qa datasets have several issues, such as having limited topic diversity, picking common question patterns, and the limited size of the dataset. first, although having a variety of topics is one of the objectives of tydi qa creators @xcite 1 , many subjects such as classical arabic poetry are less represented in tydi. this issue also 1 tydi qa states that "the prompts are provided merely as inspiration to generate questions on a wide variety of topics" exists in both arcd and aqad datasets since they are generated from a limited number of articles. second, when crowd workers are given passages and asked to formulate questions with less defined guidelines, they tend to pick common patterns, which can compromise the quality of the dataset. our analysis reveals that approximately 33% of the questions in arabic tydi are about explaining entities such as "who is alfred nobel?" or "what is graphic design?" foot_0 . this is in contrast to both arcd and english tydi qa, where such questions consist of only about 4-5% of the dataset. third, when compared to the squad dataset @xcite , which consists of 120,000 examples, arabic qa datasets still have a limited dataset size, mainly due to the constraints imposed by the cost of crowd-sourcing. however, few studies in arabic nlp explore alternative approaches to the crowd-sourcing method. most of these approaches rely on machine translation, a method that has been criticized for it is poor performance in arabic question answering @xcite . one suggested solution to address the existing challenges in arabic question answering is to utilize large language models (llms). these llm models often use a zero-shot learning technique to address qa tasks, eliminating the need to finetune the language model on a specific question answering dataset. the zero-shot approach with llms in english qa tasks has shown promising results, often matching the supervised methods that require a finetuning dataset @xcite . however, recent studies in arabic nlp show that the performance of the zero-shot approach with llms lags behind the supervised approach @xcite . the variation in performance between english and arabic is derived from the fact that the english corpora represent a large portion of llm's pre-training data, resulting in an inherent bias to- @xcite ward english nlp tasks @xcite . the existing issues in the arabic qa dataset, which we previously discussed, motivate us to introduce our artrivia dataset. the artrivia dataset adopts two distinguished approaches. first, we rely on structured datasets from wikipedia and a new proposed pipeline to generate our dataset, thereby mitigating the cost of crowd-sourcing and the issue of picking common patterns in question formulation. second, we prioritize having a variety of topics in our dataset, including underrepresented topics, such as classical arabic poetry. thus, the contributions of our paper are summarized in the following: ## related work in this section, we will provide an overview of existing arabic datasets and arabic language models, all of which are part of our evaluation setup. ## building artrivia dataset our approach to build our artrivia dataset consists of of two components: (1) generate questionanswer pairs from various structured data from wikipedia, and (2) build a new pipeline that consists of multiple functions to generate our questionanswer-passage triplets. first, we will explain in section 3.1, our method to generate questionanswer pairs. then, in section 3.2, we will explain our proposed pipeline to generate our questionanswer-passage triplets. ## dataset analysis in this section, we will focus on the quantitative analysis of our dataset. then, in section 5, we will focus on evaluating the performance of artrivia as a training and evaluation dataset. ## pipeline and dataset evaluation in this section, we will first discuss the evaluation performance of our proposed pipeline to highlight the impact of each stage in our pipeline. then, we will discuss and compare the performance of artrivia and tydi datasets using different setups for training and evaluation sets. this will help us to study the out-of-distribution and study how artrivia can serve as a complementary dataset to tydi qa. ## conclusion in this paper, we introduce artrivia, a novel dataset consisting of over +10,000 question triplets, cover-ing a wide range of 18 diverse topics. we present a detailed description of our proposed pipeline and conduct a comprehensive analysis of the contribution of each component to overall performance. while most of the existing research on question answering datasets primarily focuses on question formulation and passage selection, our work emphasizes the overlooked, yet crucial, role of answer normalization in the quality of qa datasets. our results also highlight the out-of-distribution issue within tydi when presented with more challenging questions. in future work, we plan to adapt our proposed pipeline to different domains and languages such as creating a new multi-language qa dataset.