TCE at Qur’an QA 2023 Shared Task: Low Resource Enhanced Transformer-based Ensemble Approach for Qur’anic QA (2023)

Abstract: In this paper, we present our approach to tackle Qur’an QA 2023 shared tasks A and B. To address the challenge of low-resourced training data, we rely on transfer learning together with a voting ensemble to improve prediction stability across multiple runs. Additionally, we employ different architectures and learning mechanisms for a range of Arabic pre-trained transformer-based models for both tasks. To identify unanswerable questions, we propose using a thresholding mechanism. Our top-performing systems greatly surpass the baseline performance on the hidden split, achieving a MAP score of 25.05% for task A and a partial Average Precision (pAP) of 57.11% for task B.

## introduction ad hoc search is a fundamental task in information retrieval (ir) and serves as the foundation for numerous question answering (qa) systems and search engines. machine reading comprehension (mrc) is a long-standing endeavor in natural language processing (nlp) and plays a significant role in the framework of text-based qa systems. the emergence of bidirectional encoder representations from transformers (bert) and its family of transformer-based pre-trained language models (lm) have revolutionized the landscape of transfer learning systems for nlp and ir as a whole @xcite . arabic is widely spoken in the middle east and north africa, and among muslims worldwide. arabic is known for its extensive inflectional and derivational features. it has three main variants: classical arabic (ca), modern standard arabic (msa), and dialectal arabic (da). qur'an qa 2023 shared task a is a passage retrieval task organized to engage the community in conducting ad hoc search over the holy qur'an @xcite . while qur'an qa 2023 shared task b is a ranking-based mrc over the holy qur'an, which is the second version of qur'an qa 2022 shared task @xcite . this paper presents our approaches to solve the two tasks a and b. for task a, we explore both dual-encoders and cross-encoders for ad hoc search @xcite . for task b, we investigate lms for extractive qa using two learning methods @xcite . for both tasks, we utilize various pre-trained arabic lm variants. moreover, we adopt external arabic resources in our fine-tuning setups @xcite . finally, we employ an ensemble-based approach to account for inconsistencies among multiple runs. we contribute to the nlp community by releasing our experiment codes and trained lms to github foot_0 . in this work, we address the following research questions foot_1 : rq1: what is the impact of using external resources to perform pipelined fine-tuning? rq2: how does ensemble learning improve the performance obtained? rq3: what is the effect of thresholding on zeroanswer questions? rq4 a : what is the impact of hard negatives on the dual-encoders approach? rq5 b : what is the impact of multi answer loss method on multi-answer cases? rq6 b : how is post-processing essential for ranking-based extractive question answering? the structure of our paper is as follows: sections 2 and 3 provide an overview of the datasets used in our study. in section 4, we present the system design and implementation details for both tasks. the main results for both tasks are presented in section 5. section 6 focuses on the analysis and discussion of our research questions rqs. finally, section 7 concludes our work. 2 task a dataset details qur'an qa 2023 shared task a serves as a test collection for the ad hoc retrieval task. the divine text is divided into segments known as the thematic qur'an passage collection (qpc), where logical segments are formed based on common themes found among consecutive qur'anic verses @xcite . in this task, systems are required to provide responses to user questions in msa by retrieving relevant passages from the qpc when possible. this suggests there is a language gap between the questions and the passages, as the passages are in ca. although zero-answer questions account for 15% of the questions in task a test collection, they only contribute to 5% of the question-passage pairs in task b. furthermore, task b has a limited number of unique questions in comparison to their corresponding question-passage pairs as seen from tables 1 and 2 , respectively. as a consequence, task b can have repeated questions and passages among different samples and can be even leaked among training and development splits @xcite . @xcite analyzed this phenomenon and identified sources of leakage in qur'an reading comprehension dataset v1.1 (qr-cdv1.1). in qrcdv1.1, leakage is defined as the presence of passages, questions, or answers that are shared among multiple samples @xcite . this can lead to lms memorizing or overfitting leaked samples @xcite . @xcite categorized qrcdv1.1 into four distinct and mutually exclusive categories based on the type of leakage: pairs of passagequestion, passage-answer, or just questions. (for more information about leakage in task b, please refer to appendix a.4) we extend the analysis made by @xcite for qrcdv1.2. our main observation is that 90% of the samples with no answer belong to the trivial leakage group called d (1) . this group refers to samples with duplicate passageanswer or question-answer pairs. this indicates that zero-answer questions are not just less prevalent in task b but also present a greater challenge in terms of generalization. given the four groups defined by @xcite , they proposed a data re-splitting mechanism for qrcdv1.1 called faithful splits. in this work, we extend their resplitting approach and create faithful splits for qr-cdv1.2. ## system design in this work, we fine-tune a variety of pre-trained arabic lms, namely arabertv0.2-base (antoun et al., 2020), camelbert-ca @xcite , and araelectra @xcite . we utilize transfer learning and ensemble learning for both tasks. to determine zero-answer cases, we apply a thresholding mechanism. (additional information on transfer learning and ensemble learning can be found in appendices b and c, respectively) ## results the results tables for both tasks use the following notational format: we use short forms to refer to combinations of lms and their fine-tuning approaches using superscripts and subscripts. the subscripts ∼ and ≈ denote direct fine-tuning and pipelined fine-tuning, respectively. additionally, the arrows in model names subscripts indicate the stages of pipelined fine-tuning, with the learning resources names listed. superscripts are used to denote the architectures employed for task a and the learning methods for task b. tables 3 and 4 present our detailed results on the development split for both tasks for single and self-ensemble models. both tables present comprehensive results for different question types, as well as the outcomes for a manually set threshold ζ and ζ ⋆ , i.e., the threshold that yields the best performance. (see appendix f for more details about ζ and optimal ζ selection) considering the question types , experiments of (arb ⊗ ∼ ) and (arb ⊗ ≈ ) obtains the best map performance for zero-answer and multi-answer questions for task a. with regard to the hidden split, tables 5 and 6 provide a summary of our official submissions. in task a, as shown in on the other hand, in task b, we experimented with our two best performing models in ## analysis and discussion regarding rq1, external resources always bring significant improvements to the same lm for both tasks. for task a, we have three stages of finetuning as indicated by arrows in to answer our rq2, ensemble learning consistently outperforms single models for both tasks. for instance, (cam ⊗ ≈ ) ensemble surpasses its single model by 3.5% for the map metric for task a. similarly, (elc m ≈ ) ensemble outperforms its corresponding single model by almost a pap of 2% for task b. with regard to rq3, the hyperparameter ζ affects the zero answer type evaluation scores for both tasks. we make best use of the available data by employing a quantile method to determine the threshold ζ for both tasks. however, (arb ⊗ ≈ ) model map performance improves by 3% when the optimal ζ ⋆ is employed for task a. this suggests that there is a room for improvement for the in with regard to rq6, the post-processing approach proposed by @xcite always surpasses the raw prediction score for both single and ensemble models. this is represented by post subscript in ## conclusion in this paper, we have presented our solution for both task a and task b of qur'an qa 2023 shared tasks. we explored various arabic lms using different training approaches and architectures. our best performing systems are ensemble-based, enhanced with transfer learning using external learning resources. lastly, we addressed a set of rqs that highlight the main strengths of our work.