Al-Jawaab at Qur’an QA 2023 Shared Task: Exploring Embeddings and GPT Models for Passage Retrieval and Reading Comprehension (2023)

Abstract: This paper introduces a comprehensive system designed to address two natural language processing tasks: Passage Retrieval (Task A) and Reading Comprehension (Task B), applied to datasets related to the Holy Qur’an. Task A was treated as a measurement of a textual similarity problem where the system leverages OpenAI’s “text-embedding-ada-002” embedding model to transform textual content into numerical representations, with cosine similarity serving as the proximity metric. Task B focuses on the extraction of answers from Qur’anic passages, employing the Generative Pre-trained Transformer-4 (GPT-4) language model. In Task A, the system is evaluated using the Mean Average Precision (MAP) metric, achieving MAP scores of 0.109438 and 0.06426543057 on the development and test datasets with an optimal similarity threshold set at 0.85. Task B evaluation employs partial Average Precision (pAP), where our system surpasses a baseline whole-passage retriever with pAP scores of 0.470 and 0.5393130538 on the development and test datasets, respectively.

## introduction establishing a dependable method for providing accurate responses and citing relevant passages from the holy qur'an within the framework of natural language processing represents a crucial and challenging endeavor. the creation of a reliable model capable of delivering precise answers to inquiries about islam and the holy qur'an holds substantial potential. it not only serves as a valuable resource for facilitating accurate information retrieval but also as a potent tool for automatically detecting and countering the dissemination of false information on the internet and social media platforms. qur'an qa 2023 shared task @xcite encourages researchers to work on two important tasks, task a: passage retrieval and task b: reading comprehension. task a: passage retrieval. this task involves providing a ranked list of passages from the holy qur'an that potentially contain answers to a given free-text question in modern standard arabic (msa). the task encompasses both factoid and non-factoid questions where factoid questions have short answers such as names and numerical values, and non-factoid questions need explanations, reasoning, or opinions to provide an answer @xcite . this task also includes certain questions within the dataset that lack corresponding answers in the holy qur'an. the system should return a ranked list containing up to 10 qur'anic passages believed to contain the answer(s) to the given question if any, and "no answers." in case there is no answer from the holy qur'an. ## data 2.1 task a: passage retrieval task a dataset @xcite @xcite comprises three main components: the qur'anic passage collection (qpc), the questions from the ayatec dataset, and query relevance judgements (qrels) as the assessments of how relevant these questions are to the passages within the qpc. the qpc was created by categorizing the 114 qur'anic chapters, each of varying lengths, based on thematic divisions as outlined in the thematic holy qur'an @xcite . this process led to a total of 1,266 distinct passages. the ayatec dataset has 199 questions and the qrels dataset consists of 1,132 gold qur'anic passage-ids that are deemed relevant to each question. the output format of the system that solves task a should be as follows, where tag is used to indicate a human-readable model name: '<question-@xmath0. the dataset was split as 70% for training, 10% for development, and 20% for testing, yielding 174 questions for the training, 25 for the development, and 52 for testing. from question-passage pairs point-of-view, the dataset had 972 pairs for training, 160 for development, and 427 for testing. ## system 3.1 task a: passage retrieval to solve this task, we measured the similarity between the question and all qur'anic passages and then selected the most similar passages, up to 10. we put a threshold to indicate whether the question and a passage are similar or not. if no passage has a similarity score of more than the threshold, then a "no answer" case is indicated by the system. similarity cannot be measured directly between two passages (the question and passage in our task). however, we can convert the passages to numerical representations and then measure the similarity between the resulting representations. embedding models, such as bert @xcite , word2vec @xcite , and glove @xcite , can be used to convert a given text into a numerical space. in this work, we used openai's embedding model which is called "text-embeddingada-002" (openai, 2023a). according to (ope-nai, 2023b), "text-embedding-ada-002" converts a given text into a 1536-dimension embedding vector with an 81.5% performance score on seneval, a tool designed to assess the effectiveness of sentence embeddings @xcite . to measure the distance between two embedding vectors, we used the cosine similarity @xcite . ## discussion the results demonstrate that the openai models utilized in this work provide a reasonable starting point for addressing the qur'an qa tasks. however, there is substantial room for improvement to achieve state-of-the-art performance. regarding task a, we initially attributed the low map score to a potential deficiency in arabic language support. to investigate this, we employed google translate to render both the questions and passages into english. subsequently, we applied the same methodology as described in section 3.1. surprisingly, the outcome proved to be notably inferior to the results obtained using the original arabic dataset. we attribute this disparity to the inherent limitations of translation, which struggle to convey the precise nuances of quranic passages accurately. unfortunately, since "text-embeddingada-002" embedding model is not open-sourced, it cannot be fine-tuned to fit our task. in the context of the reading comprehension task, it is noteworthy that the gpt-3.5 prompt engineering approach performs notably worse than a naive baseline model. conversely, the gpt-4 prompt engineering approach exhibits a significant performance improvement, surpassing the naive baseline by a considerable margin. however, it is essential to recognize that while gpt-4 demonstrates superior adherence to prompts compared to gpt-3.5, its behavior is not entirely deterministic, and variations can occur. additionally, we must address the issue of "prompt injection", wherein a prompt could be introduced after the initial prompt, potentially altering the model's behavior. while this behavior was more prevalent in gpt-3.5, it is less pronounced in gpt-4. for instance, when applying the gpt-4-based model to the test set, we encountered very few cases like the question , which yielded the answer indicating that gpt-4 ignored entirely the prompt we mentioned in section 3.2 and was appended before the question.