AHJL at Qur’an QA 2023 Shared Task: Enhancing Passage Retrieval using Sentence Transformer and Translation (2023)

Abstract: The Holy Qur’an is central to Islam, influencing around two billion Muslims globally, and is known for its linguistic richness and complexity. This article discusses our involvement in the PR task (Task A) of the Qur’an QA 2023 Shared Task. We used two models: one employing the Sentence Transformer and the other using OpenAI’s embeddings for document retrieval. Both models, equipped with a translation feature, help interpret and understand Arabic language queries by translating them, executing the search, and then reverting the results to Arabic. Our results show that incorporating translation functionalities improves the performance in Arabic Question-Answering systems. The model with translation enhancement performed notably better in all metrics compared to the non-translation model.

## introduction the holy qur'an holds significant relevance as it serves as the central holy book in islam, guiding the beliefs and practices of over 1.9 billion muslims worldwide. it provides essential spiritual guidance, imparts moral values, and establishes rules for living, exerting a profound influence on the lives of muslims and their communities. comprising 114 chapters (suras) and 6236 verses (ayas) of varying lengths, totaling approximately 80,000 arabic words, the qur'an, revealed over 1,400 years ago, is written in classical arabic @xcite . is considered to be linguistically complex because it uses a rich vocabulary, intricate sentence structures, and rhetorical devices like metaphors and allegories. its verses can have multiple meanings depending on the context, allowing for various interpretations @xcite . various studies have explored the holy qur'an for different nlp tasks, such as creating datasets, question answering (qa), retrieving related information, and and identifying topics @xcite @xcite . one recent study on applying nlp to the qur'an relies on the qur'an qa shared task (2022) @xcite . they propose a task defined as giving a group of verses from a particular part of the holy qur'an and a question about those verses; a system needs to find the answer to the provided question. the organizers continued to provide this shared task, qur'an qa 2023 shared task. however, they added a new task called the qur'anic passage retrieval (pr) task. pr is defined as participants will be given a question in modern standard arabic and a set of qur'anic passages that cover the entire holy qur'an. the system is required to return a list of these passages, ranked in order of how likely they are to contain the answer to the provided question. the question may vary in complexity, ranging from simple and direct to more intricate and nuanced. however, some questions might not have an answer in the holy qur'an to make the task more realistic and challenging. in such cases, an adequate system should recognize that there is no answer. otherwise, it should return a list of the top ten passages likely to contain the answer. this paper describes our participation in the pr task (task a) provided by the qur'an qa 2023 shared task. our proposed method is to translate the arabic questions into english and incorporate a paraphrasing module to enhance the retrieving process. the rest of the paper is organized as follows: section 2 discusses the related work. section 3 explains the data we used for our tests. section 4 provides details of our experiments. section 5 presents the results tied to our research queries. finally, we discuss potential next steps and conclude the paper. ## related work in the domains of natural language processing (nlp) and information retrieval (ir), the task of question answering (qa) involves finding accurate answers to questions within a body of text. qa combines these two fields by requiring an understanding of language, as in nlp, and the ability to find the proper documents, as in ir @xcite . a typical qa system consists of several steps, including understanding questions, finding relevant text passages, and extracting answers to deliver precise responses from extensive textual sources @xcite . in the field of information retrieval, using language models to rank documents based on their relevance to a query has been a popular method @xcite . earlier methods used count-based language models for each document to determine its likelihood of being relevant to a query @xcite . sentence similarity involves assessing the likeness between two texts, where each sentence pair is judged based on the notion that they have identical meanings @xcite . models for sentence similarity transform input texts into embeddings that capture the overall meaning and then compute their proximity according to some specific measure, such as cosine-similarity or dot product. in the al-bayan system by @xcite , the researchers utilized the holy qur'an and tafseer to identify verses with similar meanings using semantic analysis. they developed a semantic interpreter with machine learning to transform text into vectors representing qur'anic concepts. these vectors, built from terms in the relevant documents, are weighted using the tf-idf method. the system calculates the similarity between the vectors of a given question and terms in the qur'an, and then highlights the most relevant terms to that question. these methods had challenges, like dealing with limited data. using commercial search engines as external sources for paragraph retrieval is one of the methods used in the literature. the ewaq system, introduced by al-khawaldeh (2015), presents a novel passage retrieval (pr) method. this method fetches passages from search engines and calculates their relevance to a query based on "entailment similarity", employing cosine directional similarity as a metric. a similar method for passage retrieval was suggested by @xcite . initially, passages related to the query are fetched from google using the question's keywords. these passages are then refined, standardized, and divided. next, the questions and passages are examined linguistically, including identifying named entities, analyzing syntax, and assessing morphology. in the end, the main ideas of the question and the passage are presented logically. more modern techniques use advanced language models like bert to determine query relevance. such methods have an advantage over older sparse retrieval techniques because they recognize wordbased and more profound meaning similarities rather than just looking for exact keyword matches (nogueira dos santos et al., 2020). @xcite aimed to develop an effective dense embedding model by merging the bert pre-trained model with a dual-encoder setup. this model transforms text into a specific vector format and then indexes every passage for retrieval. they found that their model surpassed several other models in question-answer tests on various datasets like squad @xcite and triviaqa @xcite . ## result and discussion our findings indicate that the translationaugmented version exhibited significant improvements across all metrics when compared to the model without translation. for instance, the map score witnessed an increase from 0.003 using an arabic sentence transformer model 'medmediani/arabic-kw-mdel' 4 to 0.343 using the english sentence transformer model 'msmarcodistilbert-base-tas-b' on the dev set, suggesting that the translation component greatly enhanced the model's ability to retrieve more relevant documents. overall, integrating translation into the system substantially improved its performance, validating our hypothesis that translation is a crucial element for improving retrieval quality in the arabic questionanswering (qa) environment. building on this, we also introduced a second model that involved multiple paraphrased versions of the input question for even more precise retrieval. the results of the versions of the question produced by the paraphrasing component were sorted, and the duplication in the retrieved answers was deleted. the result of both t-test @xcite and mann-whitney u test @xcite shows no significant difference in map and mrr scores with adding the paraphrasing component to the base model. in the case of questions with no answer, the test set contained 7 questions with no answers, the best model was able to correctly say 'no answer' to four questions, 0.57 of the questions. the threshold value for eliminating irrelevant documents is set to -5, where documents with a score of -6 and below are considered irrelevant. the test set has in total 7 questions that did not have corresponding answers (no-answer questions). interestingly, out of these 7 questions, our bestperforming model accurately identified 'no answer' for 4 of them, giving us a 57% accuracy rate in this specific context. in order to filter out irrelevant documents, we established a threshold value of -5, which means that any documents scoring -6 or lower were considered irrelevant. some questions in the test set are not direct and cannot be solved with similarity measures but rather require some inference methodology to infer the question from the given context. certain questions within the test set are indirect and present challenges when addressed through similarity measures. to effectively tackle these questions, a more nuanced approach, specifically an inference methodology, is necessary in order to ascertain the intention of the question from the given context. ## conclusion in this study, we explored the linguistic complexity of the holy qur'an, which holds profound influence over approximately two billion muslims worldwide. our engagement in the qur'an qa 2023 shared task's pr task (task a) led us to employ two distinct models: the sentence transformer and openai's embeddings, both aimed at effective document retrieval. a significant feature of our approach was the integration of a translation mechanism to facilitate the interpretation of arabic queries. upon evaluation, the translation-enhanced model showcased superior performance across all metrics in comparison to its non-translation counterpart.