M aha SQ u AD : Bridging Linguistic Divides in M arathi Question-Answering (2023)

Abstract: Question-answering systems have revolutionized information retrieval, but linguistic and cultural boundaries limit their widespread accessibility. This research endeavors to bridge the gap of the absence of efficient QnA datasets in low-resource languages by translating the English Question Answering Dataset (SQuAD) using a robust data curation approach. We introduce MahaSQuAD, the first-ever full SQuAD dataset for the Indic language Marathi, consisting of 118,516 training, 11,873 validation, and 11,803 test samples. Challenges in maintaining context and handling linguistic nuances are addressed, ensuring accurate translations. Moreover, as a QnA dataset cannot be simply converted into any low-resource language using translation, we need a robust method to map the answer translation to its span in the translated passage. Hence, to address this challenge, we also present a generic approach for translating SQuAD into any low-resource language. Thus, we offer a scalable approach to bridge linguistic and cultural gaps present in low-resource languages, in the realm of question-answering systems. The datasets and models are shared publicly at https://github.com/l3cube-pune/MarathiNLP.

## introduction the stanford question answering dataset (squad) @xcite has emerged as a pivotal benchmark in the realm of natural language understanding and question answering and it has contributed to the development of cutting-edge machine learning models and techniques, enabling systems to pro-vide accurate and contextually relevant answers to a wide array of questions in english. question-answering systems have diverse applications in education, information retrieval, customer support, and more. these applications have the potential to bring significant benefits to marathi speakers, enabling them to access knowledge and services in their native language. question-answering systems are required for retrieving answers from a provided context, using the extractive qa approach, thus leading to efficient retrieval of information from the questions provided. however, as the world becomes increasingly interconnected and diversified, there is a growing need to make question-answering systems in natural language processing accessible and broadened across linguistic and cultural boundaries. marathi is not just a language but a carrier of culture and identity for millions. it is important to recognize its cultural, regional, and linguistic significance. the creation of a marathi question-answering dataset has the potential to impact a diverse population, both in india and among the marathi-speaking community, enabling marathi speakers to access and interact with information in their native language more effectively. although the indic language, marathi, is the third highest spoken language in india, after hindi and bengali, it doesn't have robust rich datasets in question answering despite these languages being spoken by several primarily in india, the country with the highest growing population @xcite . while some efforts have been made to develop nlp resources for marathi, there remains a substantial gap in terms of comprehensive question-answering datasets @xcite @xcite . existing research often lacks the depth and scale necessary for robust question answering. our research builds upon and surpasses previous efforts by focusing on the squad dataset, a gold standard in the field. motivated by this challenge, our research aims to cover this linguistic divide by translating the squad dataset into marathi, using a robust approach. our contributions can be summarised as follows: in the subsequent sections of this paper, we will provide a detailed account of our methodology for dataset translation, the challenges we encountered, the results, and the implications of our research for the field of natural language processing. our research extends beyond the immediate goal of creating a marathi dataset. by addressing the linguistic diversity challenge, we hope to inspire similar efforts for other languages and demonstrate the potential for cross-linguistic research and development in the nlp community. ## related work in this section, we provide an overview of related work in question-answering datasets, machine translation for natural language processing, and multilingual natural language processing. our discussion aims to contextualize our research and highlight the gaps in the existing literature that our proposed methodology addresses. the stanford question answering dataset (squad) has been a cornerstone in advancing the state of question-answering systems. @xcite introduced the original english version of squad, which has since served as a benchmark for evaluating machine comprehension models. numerous studies have leveraged squad for reading comprehension, question generation, and answer extraction @xcite . to address the scarcity of datasets for new languages, methods have been explored to create datasets based on existing ones. translating the context and answers within a dataset is a relatively straightforward task. however, a critical challenge arises when attempting to determine the exact index of the translated answer within the context, as the translated answer might not lie exactly in the same way as the answer in the context. this complexity stems from the translation of an answer, with or without surrounding context, which can vary, making it a hard task to directly pinpoint the index at which the answer resides within the context. the majority of squad-related research has focused on the english language, limiting its applicability in multilingual contexts. to our knowledge, few prior efforts have been made to translate squad into multiple languages, and those that exist often lack detailed methodologies for ensuring high-quality translations @xcite and also haven't been in large numbers to ensure better training on the models. this gap underscores the importance of our work in extending squad to a diverse set of indic languages. machine translation has witnessed remarkable progress, with models like transformer-based architectures @xcite significantly improving translation quality. however, translation in the context of natural language processing (nlp) poses unique challenges, including maintaining context, handling idiomatic expressions, and preserving semantic nuances. prior research has explored techniques to adapt machine translation for nlp tasks, such as paraphrasing @xcite , context-aware translation @xcite , and domain-specific translation (lample and @xcite . these studies have laid the foundation for addressing the translation challenges specific to question-answering datasets, which we further investigate in our proposed methodology. multilingual nlp has gained prominence due to its potential to foster cross-lingual communication and knowledge sharing. researchers have explored methods for cross-lingual transfer learning @xcite , zero-shot translation @xcite , and multilingual model development @xcite . these efforts have broadened the scope of nlp applications across languages but have yet to always address the nuanced requirements of questionanswering. while squad has been the benchmark for english question-answering, recent efforts have emerged to create multilingual questionanswering datasets. for instance, the xquad dataset consisted of 240 paragraphs and 1190 question-answer pairs from squad v1.1, which was translated by professional translators into ten different languages. @xcite it expands the squad concept to multiple languages, enabling the evaluation of question-answering models in a cross-lingual context. this work demonstrates the need and potential for multilingual question-answering resources, aligning with our objectives in the context of indic languages. however, it consists of limited data points to evaluate, and it has focused on indic languages such as spanish, german, greek, russian, turkish, arabic, vietnamese, thai, chinese, and hindi, not consisting of marathi, the language we aim to focus on. @xcite fine-tuned an mbertbased qa model by translating and transliterating qa samples of the target language into other languages, increasing the performance of the models. @xcite experiment with multilingual models on the task of machine comprehension, a part of question-answering (qa) for languages, hindi and english. they experiment with mbert on monolingual, zero-shot, and cross-lingual fine-tuning setups. @xcite , focuses on presenting the largest dataset question answering dataset for indic languages such as hindi and marathi, and they provide a performing model based on experimentation. we've aimed to provide a more robust and efficient technique for translating the dataset, that can provide an answer without the constraints of fixed word size in marathi and with a larger dataset consisting of 130,319 data points. our research bridges the gap between machine translation and question answering in the multilingual context, proposing a novel methodology that combines the strengths of both fields to translate and adapt the squad dataset effectively. ## methodology in this paper, our approach encompasses a comprehensive translation of the entire english question answering dataset, squad 2.0, into marathi, resulting in the creation of mahasquad. employing a robust methodology, we not only achieve translation but also extract the translated answer and its corresponding answer span for each entry, complemented by transliteration. furthermore, we meticulously construct distinct sets, including a training set, validation set, test set, and a gold set, each extensively examined in subsequent sections of the paper. to evaluate the efficacy of our translated dataset, we conduct training on both monolingual and multilingual models, encompassing ma-habert, maharoberta, mbert, muril-bert, and indic-bert. the detailed results of these training endeavors are systematically tabulated below, providing a comprehensive overview of our model performances. ## results in this study, we have used various monolingual and multilingual models in marathi. we have used the following evaluation metrics: the exact match (em) score in nlp measures the accuracy of question-answering systems by assessing whether their generated answers are identical, character by character, to the reference answers. it provides a binary metric, where a 1 indicates an exact match (correct answer) and a 0 indicates a mismatch (incorrect answer). the em score is a useful measure for evaluating the precision of qa systems. ## future scope our proposed pipeline has showcased enhanced efficiency in the creation of translated squad datasets. this improved pipeline can be extended to cater to other low-resource languages, facilitating the generation of questionanswer datasets for those languages as well. additionally, leveraging the accuracy in identifying the most relevant spans can be applied to generate datasets for various other applications. ## conclusion in this research, our objective was to address the scarcity of efficient question-answering (qna) datasets in low-resource languages, specifically focusing on marathi. our central research question, "can the english question answering dataset be effectively translated into marathi to foster linguistic accessibility?" guided our investigation into this crucial domain. employing a rigorous and robust methodology, we successfully translated the entire english question answering dataset into marathi, resulting in the creation of the 'mahasquad' dataset. our findings highlight the potential of the 'mahasquad' dataset to empower marathi speakers, providing accessibility to technology and knowledge in their native language. furthermore, by introducing a scalable approach for translating qna datasets into low-resource languages, we pave the way for enhanced information retrieval in marathi. notably, this dataset serves as a valuable benchmark dataset for developing marathi question-answering models. its applicability extends beyond research, proving useful in creating applications such as chatbots for lowresource languages. the notable success of ma-habert and maharoberta, as evidenced by state-of-the-art results, emphasizes the efficacy of these models in the realm of questionanswering datasets.