LKAU 23 at Qur’an QA 2023: Using Transformer Models for Retrieving Passages and Finding Answers to Questions from the Qur’an (2023)

Abstract: The Qur’an QA 2023 shared task has two sub tasks: Passage Retrieval (PR) task and Machine Reading Comprehension (MRC) task. Our participation in the PR task was to further train several Arabic pre-trained models using a Sentence-Transformers architecture and to ensemble the best performing models. The results of the test set did not reflect the results of the development set. CL-AraBERT achieved the best results, with a 0.124 MAP. We also participate in the MRC task by further fine-tuning the base and large variants of AraBERT using Classical Arabic and Modern Standard Arabic datasets. Base AraBERT achieved the best result with the development set with a partial average precision (pAP) of 0.49, while it achieved 0.5 with the test set. In addition, we applied the ensemble approach of best performing models and post-processing steps to the final results. Our experiments with the development set showed that our proposed model achieved a 0.537 pAP. On the test set, our system obtained a pAP score of 0.49.

## introduction the arabic language poses many challenges in naturual language processing (nlp), including in the areas of machine reading comprehension (mrc) and passage retrieval (pr). one of the most prominent recent nlp techniques applied to mrc and pr tasks in the arabic language is pre-trained transformer-based models, which can achieve state-of-the-art performance @xcite . there are pr studies that use a dense approach based on pre-trained models @xcite . this approach has outperformed traditional information retrieval, such as tf-idf @xcite with modern standard arabic (msa; @xcite . to our knowledge, the dense approach has not been researched with classical arabic (ca). therefore, our proposed system for task a of the qur'an qa 2023 shared tasks uses the dense approach by fine-tuning the arabic pretrained models and then ensemble the best scores. the idea of task a is to build a system to return a list of qur'anic passages that contain answers to a posed question/query @xcite . however, the challenging aspect of this task is that there are some questions that do not have an answer in the qur'an. the first research question rq1: does using the arabic pre-trained models in pr for ca outperform the traditional approach such as bm25? most recent studies on the qur'an mrc task have tended to use transformers-based models along with qur'anic reading comprehension dataset (qrcd) @xcite . we noticed that they improved the performance of the systems using three approaches: (1) using an additional msa and/or ca datasets in fine-tuning @xcite , (2) constructing an ensemble of different bert models (3) applying appropriate post-processing steps on the result of the final ranked list @xcite . to the best of our knowledge, no studies have combined these three approaches. therefore, we applied the combination of those approaches for task b of the qur'an qa 2023 shared task. the goal of task b was to build a model that took a qur'anic passage and msa question as input and extracted a ranked list of up to 10 answer spans to that question from the passage as output @xcite . the new challenge in the second version of this task was that there were no answers to some questions. the second research questions rq2 in this paper is: does the combination of fine-tuning the models with a large ca dataset and/or msa dataset, ensembling these models and then applying post-processing steps improve the results? the paper's structure is as follows: in section 2, related work is presented. section 3 describes the datasets. this is followed by section 4, which explains the proposed models. in section 5, the results are discussed. finally, the paper provides a conclusion. ## datasets 3.1 task a: passage retrieval the data were comprised of the qur'anic passage collection (qpc) and questions from ayatec @xcite . the qpc was developed by segmenting the qur'an passages into topics, which resulted in 1,266 passages. there were 199 questions that were derived from the ay-atec dataset. the query relevance judgements (qrels) dataset contained 1,132 gold (answerbearing) qur'anic passages that answered the questions; these data were used in training and development sets. finally, the distribution of the dataset was 70%, 10% and 20% for training, development and testing sets respectively. ## conclusion this paper presented our contributions to task a: pr and task b: mrc of the qur'an qa 2023 shared task. our proposed pr method was to train several arabic pre-trained models with qrcd dataset using sbert architecture and then ensemble the combination of these models. the ensemble method did not yield the best performance with the test set, although it had the best score with the development set. cl-arabert achieved the best results with a 0.124 map. our proposed mrc system is based on combining the transfer learning and ensemble approaches for the best-performing models. initially, we fine-tuned nine different arabic pretrained models using different data collections. we then applied the ensemble approach to the two bestperforming models. finally, we implemented appropriate post-processing steps. the combination of the base and large variants of arabert achieved the best results on the development set, with a 0.537 pap@10. the second-highest score was achieved by base arabert with a 0.49 pap@10. the results of applying these two models to the test set showed that the base arabert model was the best with a score of 0.5 pap@10, while the ensemble model achieved a score of 0.49 pap@10.