GYM at Qur’an QA 2023 Shared Task: Multi-Task Transfer Learning for Q uranic Passage Retrieval and Question Answering with Large Language Models (2023)

Abstract: This work addresses the challenges of question answering for vintage texts like the Quran. It introduces two tasks: passage retrieval and reading comprehension. For passage retrieval, it employs unsupervised fine-tuning sentence encoders and supervised multi-task learning. In reading comprehension, it fine-tunes an Electra-based model, demonstrating significant improvements over baseline models. Our best AraElectra model achieves 46.1% partial Average Precision (pAP) on the unseen test set, outperforming the baseline by 23%.

## introduction question answering (qa) for vintage, religious texts like the quran presents unique challenges for natural language understanding systems. understanding the concepts and connections in the quran requires deep semantic reasoning to map questions to relevant passages and surface correct answers. to advance research in this domain, the qur'anqa 2023 shared task 1 proposes two sub-tasks focused on machine comprehension of the quran @xcite . task a on passage retrieval requires matching modern standard arabic (msa) free-text questions to quran verses potentially containing the answer. this tests semantic similarity between questions and passages. we propose using sentence encoders @xcite to derive dense vector representations for questions and passages. these vectors can be indexed and searched efficiently to find relevant matches. task b on reading comprehension focuses on extracting span answers from a given passage. this is framed as a machine reading comprehension task. however, given its literary arabic and frequent need for theological reasoning, it is especially difficult for the quran. we formulate the task as ex-tractive qa and experiment with span prediction models like araelectra @xcite . the qur'anqa 2023 shared task includes two sub-tasks that form an end-to-end qa pipeline. task a retrieves candidate passages potentially containing answers. this narrows the search space from the entire quran to a small set of relevant verses. task b then extracts answer spans from these candidates. the tasks work sequentially: passage retrieval provides context to reading comprehension, which verifies answers. together they comprise an end-to-end qa system over the quran. our key contribution to this work is utilizing transfer learning and model adaptation techniques to develop customized qa models for the limited qur'anqa 2023 shared task dataset. after experimenting with several arabic and multi-lingual language models (lms) we choose araelectra and arabert (antoun et al.) as strong candidates. these models provide contextual representations of arabic text learned from broad domains. in this work, we aim to address these research questions: through experiments, we analyze different strategies for unsupervised sentence embeddings and supervised task-specific fine-tuning. despite the scarce training data, this allows the model to learn specialized embeddings for quranic comprehension. our work provides insights into adapting pretrained language models to new domains with limited labeled data. by combining broad pre-trained knowledge with targeted fine-tuning, we develop customized qa models capable of reasoning about the quran's abstract concepts and archaic language. the source code is available at github foot_0 . ## task a: passage retrieval for a free-text question in msa, the system must retrieve and rank quranic passages that potentially contain answers to the question from a corpus covering the entire quran. ## data for this work, we utilize the training and development datasets provided by the qur'anqa 2023 organizers @xcite @xcite , a summary of which is provided in to augment the limited size of the quran-specific data, we incorporate additional datasets during finetuning. for this passage retrieval task, we leverage the multi-lingual mr. tydi dataset, which contains monolingual question-passage pairs for information retrieval in 11 different languages @xcite . we utilized the arabic portion to fine-tune our proposed model.