Qur’an QA 2023 Shared Task: Overview of Passage Retrieval and Reading Comprehension Tasks over the Holy Qur’an (2023)

Abstract: Motivated by the need for intelligent question answering (QA) systems on the Holy Qur’an and the success of the first Qur’an Question Answering shared task (Qur’an QA 2022 at OSACT 2022), we have organized the second version at ArabicNLP 2023. The Qur’an QA 2023 is composed of two sub-tasks: the passage retrieval (PR) task and the machine reading comprehension (MRC) task. The main aim of the shared task is to encourage state-of-the-art research on Arabic PR and MRC on the Holy Qur’an. Our shared task has attracted 9 teams to submit 22 runs for the PR task, and 6 teams to submit 17 runs for the MRC task. In this paper, we present an overview of the task and provide an outline of the approaches employed by the participating teams in both sub-tasks.

## introduction the timeless and sacred qur'an will never cease to attract the interest and inquisition of millions of muslims and non-muslims for its profound teachings, legislation, and fertile knowledge. such inquisitions may be driven by learning, curiosity, or skepticism. the qur'an is composed of 114 surahs and 6,236 verses (ayas) of different lengths, with a total of about 80k words. the words, revealed more than 1,400 years ago, are in classical arabic. extractive question answering (qa) approaches are being formulated in the literature as machine reading comprehension (mrc) tasks @xcite . given a passage of text, a system is evaluated based on its ability to correctly answer a set of questions over the given text. we believe that the resurgence of the mrc field should be harnessed to address the timeless interest in the holy qur'an and the information needs of its inquisitors and knowledge seekers @xcite . this has motivated the inception of the first qur'an question answer-part of the work on this paper was done while being at qatar university. although mrc systems are relieved from the task of passage retrieval (i.e., the task of retrieving candidate passages that potentially contain answers to a given question) to purely focus on inference and reasoning for answer extraction, the retriever component remains an integral contributor to the performance of end-to-end extractive qa systems that adopt a retriever-reader architecture @xcite . prevalent search/retrieval systems on the holy qur'an are either keyword-based, semanticbased, or a hybrid of both paradigms. semanticbased approaches are predominantly ontologybased with almost no use of state-of-the-art approaches such as dense retrieval @xcite , generative retrieval @xcite and beyond @xcite to our knowledge. to this end, and to build on the success of the first qur'an qa 2022 shared task @xcite , we have organized the second qur'an qa shared task (qur'an qa 2023) at arabicnlp 2023. qur'an qa 2023 comprises a qur'anic passage retrieval (pr) task and a machine reading comprehension (mrc) task. the pr task aims at finding all qur'anic passages that have potential answers for a given question that is posed in modern standard arabic (msa). whereas the mrc task targets the extraction of all answers to a given question from a given qur'anic passage. each answer must be a span of text extracted from the given passage. to make both tasks more challenging, we include questions that have no answers in the qur'an. further details about the two tasks are provided in sections 3 and 4, respectively. to encourage quality participation in the task, we allotted five awards. the awards for the best and second-best teams in each task are @xmath0200, respectively, provided that their papers are accepted at the conference. the fifth award is $150 allotted for the best paper. mq241 some gold passages: "2:87-88\t241", 2:97-101\t241", 2:102-103\t241, "2:253-254\t24 ‫السؤال:‬ ‫آن‬ ‫القر‬ ‫في‬ ‫املذكورون‬ ‫املالئكة‬ ‫هم‬ ‫من‬ ‫؟‬ question: who are the angels mentioned in qur'an? gold qur'anic passages ‫الذهبية‬ ‫الفقر‬ ‫ات‬ ‫القرآنية‬ qur'an qa 2023 1 has attracted 38 and 29 teams to sign up for the pr task and the mrc task, respectively. in the final phase, 9 teams participated in the pr task with 22 run submissions, and 6 teams participated in the mrc task with 17 run submissions. the rest of the paper is organized as follows. section 2 outlines the first version of qur'an qa. sections 3 and 4 discuss the pr and mrc tasks, respectively, in detail including the task descriptions, datasets, evaluation setups, results, and analysis of approaches employed by the participating teams. we conclude with final thoughts in section 5. the qur'an qa shared task in its first version in 2022 2 @xcite only comprised an mrc task that is similar to the mrc task proposed this year, but it was relatively simplified. it was defined as follows: given a qur'anic passage that consists of consecutive verses in a specific surah of the holy qur'an and a question posed in msa over that passage, a system is required to extract any correct answer span to that question (regardless if the question had more than one answer in that passage or only one answer). as such, the main measure 1 https://sites.google.com/view/quran-qa-2023 2 https://sites.google.com/view/quran-qa-2022 used in the performance evaluation of participating systems was partial reciprocal rank (prr) (malhas and elsayed, 2020). qur'an qa 2022 has attracted 30 teams to sign up for the task. in the final phase, 13 teams participated, with a total of 30 submitted runs on the test set. ten out of the thirteen teams submitted system description papers, which were peer-reviewed and published in osact 2022 @xcite . ## task a: passage retrieval (pr) in this section, we define the pr task, introduce the dataset, and elaborate on the evaluation setup and teams' results. we conclude this section with an overview of the main methods employed by the participating teams. ## task b: machine reading comprehension (mrc) in this section, we define the mrc task, present the dataset, and detail the evaluation methodology and results. we conclude with an overview of the main methods employed by the participating teams. ## results in this section, we present the results of our two models for task a and task b along with comparing them to the base model in each task. map score an model in this t pends on the b text (amati, 2 mrr scores fo 0.313333 resp bm25 model h an mrr score achieved a ma mrr score of 4.2 task b: r the evaluation tial average p rank-based me tial matching a system in scen may not neces only partially m thermore, pap tions that may within the acco makes pap a m ing task b com employed maximum marginal likelihood (mml) fine-tuning to address this challenge in the mrc task. mml is a form of bayesian fine-tuning that incorporates uncertainty to preclude the trained systems from being overly confident in a single answer span; thus, distributing the probability among more than one answer span in the accompanying passage of a given question. mml fine-tuning seems to be among the main contributors to the leading performance achieved by tce ( an important finding by al-jawaab team, is that despite their careful prompt engineering scenarios to instruct their generative gpt-4-based models @xcite , not to provide out-of-passage answers to a given question, the models sporadically succeeded in providing answer spans strictly from the accompanying passages. among the main problems was "prompt injection", where parts of the textual prompt instruction/question given to the model are injected back into the generated answer. as such, they applied some post-processing heuristics to the answers obtained by their top performing model. ## conclusion with prevalent semantic search approaches on the holy qur'an being predominantly ontology-based, we believe that recent neural dense and generative retrieval approaches coupled with the resurgence of the mrc field would pave the way for more intelligent state-of-the-art qa systems on the holy qur'an. to this end, we organized qur'an qa 2023 shared task, which witnessed the participation of 27 team members from 17 different institutes representing 10 teams. our shared task in its second version comprised two subtasks; a passage retrieval (pr) task and a machine reading comprehension (mrc) task. it attracted 9 teams to submit 22 runs for the pr task, and 6 teams to submit 17 runs for the mrc task. as anticipated, recent transformer-based neural retrieval and reading comprehension approaches were heavily employed by all the participating systems. the majority of the systems deployed encoder-based bert-like models, whereas generative (decoder-based) gpt-like models were used more sparingly in both tasks. the performance of the systems on the test sets in both tasks indicates that encoder-based transformer models are still taking the lead over generative transformer models. interestingly, araelectra and arabert finetuned using large external task-related resources pioneered the arabic transformers scene. these two models were employed by the best-performing team in each task with self-ensemble. the second best-performing teams in both tasks leveraged generative transformer models (llms) using zeroshot learning setups. though in the pr task, the second ranked team utilized an arabic-to-english translation module with their retrieval module. the majority of the semantic search/retrieval systems used bi-encoder and cross-encoder architectures independently or jointly. also, ensemble and selfensemble approaches were employed by many systems to stabilize prediction fluctuations across runs and/or to enhance prediction accuracy through the wisdom of the crowd. for zero-answer questions, the best system adopted a thresholding mechanism to identify questions with a low predicted likelihood of having answers in the holy qur'an (for task a), or in the accompanying passage (for task b). the majority of the teams did not address this challenge explicitly in both tasks, other than the second ranked team adopting a naive handcrafted prompt, engineered to instruct their generative gpt-4-based models to return a "no answer" for the mrc task. as for multi-answer questions in the mrc task, the best performing system employed mml bayesian fine-tuning to address this challenge. no other team addressed this challenge explicitly, other than the second ranked team which used prompt engineering with its generative-based models (as mentioned above). we note that multi-answer (or multi-span) extraction in the literature is an active area of research in the extractive mrc/qa scene that would benefit qur'anic qa research. our prospects towards the third version of the shared task are to aim at including an end-to-end qa task on the holy qur'an.