Kāraka-Based Answer Retrieval for Question Answering in I ndic Languages (2023)

Abstract: Kārakas from ancient Paninian grammar form a concise set of semantic roles that capture crucial aspect of sentence meaning pivoted on the action verb. In this paper, we propose employing a kāraka-based approach for retrieving answers in Indic question-answering systems. To study and evaluate this novel approach, empirical experiments are conducted over large benchmark corpora in Hindi and Marathi. The results obtained demonstrate the effectiveness of the proposed method. Additionally, we explore the varying impact of two approaches for extracting kārakas. The literature surveyed and experiments conducted encourage hope that kāraka annotation can improve communication with machines using natural languages, particularly in low-resource languages.

## introduction the web hosts a vast amount of information, including news articles, blogs, social media platforms, wikipedia, and other knowledge bases. the diverse population using this e-content, encompassing different languages and age groups, creates a demand for applications with native language interfaces to access these information sources. question answering (qa) systems play a significant role in addressing this demand by retrieving answers for natural language queries. india, as the second most populous nation, has officially recognized 121 distinct modern indian languages @xcite , and users of natural language interfaces prefer accessing applications in their native languages. a survey found that indian language internet users face challenges due to limited digital content and support in their languages @xcite . there is a dearth of application, and services in languages that have minimal digital presence and lack annotated corpora. moreover, indian languages are morphologically rich, exhibit flexibility in word or-der and possess a complex system of post-positions. there have been fewer efforts dedicated to the qa task in several indic languages. for open-domain qa, the task of answer retrieval holds significant importance. this article introduces a novel kārakabased answer retrieval approach for qa in indian languages, demonstrating that kāraka annotation captures text semantics at a level that can facilitate tasks such as answering questions and performing simple inferences. to understand the role of kārakas in the answer retrieval task, let's examine the following hypothetical question-answering scenario, with a question and two possible candidate sentences for choosing the answer: question: who created first effective covid-19 vaccine in india? possible answer 1: first effective covid-19 vaccine in india was created by bharat biotech. possible answer 2: the government in india created awareness regarding the administration of the first effective covid-19 vaccine. to address the aforementioned question, methods relying solely on word overlap or answer type would be inadequate to distinguish the answer sentence effectively. in this situation, obtaining a meaning representation from the surface form text, including the event and the different participants involved, along with their respective roles, would be beneficial. a system that can assign meaningful representations to diverse inputs that share similar or common contextual knowledge, independent of specific words or sentence structures is crucial. this is shown in the example below where the action and its direct participants involved in accomplishing it are labeled with their corresponding semantic roles in both the question and the candidate answer sentences. the first sentence in the example exhibits a higher similarity in terms of the argument's semantic roles from the question, making it an appropriate answer. question: who [agen t ] created [act ion ] first effective covid-19 vaccine [goal] in india? possible answer 1: first effective covid-19 vaccine [goal] in india was created [act ion ] by bharat biotech [agen t ] possible answer 2: the government [agen t ] in india created [act ion ] awareness [goal] regarding the administration of the first effective covid-19 vaccine. like this, for a qa system, understanding natural language utterances from the limited surface form involve dealing with a wide range of complex subject matters. literature highlights languagespecific resources like propbanks @xcite , framenets @xcite , and nombank @xcite developed for the task of identifying how different participants associate with events. these resources have been extended to a few languages mostly because each of these framework requires its process for corpus generation that is distributed across various resources. obtaining a comprehensive meaning representation for lowresource languages presents significant challenges due to extensive data requirements for training and evaluation. this research addresses the challenge by employing a set of fundamental and deep semantic roles known as "kārakas" that were first identified by ancient indian grammarian panini for sanskrit during 4 th century bc that symbolize the most widespread and concise form of speech during his era. by identifying the direct participants engaged in the action, kārakas effectively captures the fundamental meaning of utterances. these can be applied across various languages, even those with distinct grammatical structures, resulting in an abstraction that aligns with the cognitive processes of ordinary speakers, emulates their inference methods, and enables seamless interactions with machines through query-based interactions. additionally, it is observed that kārakas can be extracted from the surface form text based on syntactic and morphological information, without the need of any extralinguistic real-world knowledge; thus resulting in a scheme immensely valuable for low-resource languages. the remaining article is structured as follows: section 2 presents a concise overview of indic qa development. additionally, it presents a summary of nlp applications that demonstrate the usefulness of kāraka relations. section 3 provides de-tails on the proposed kāraka-based answer retrieval. section 4 outlines the experiment designed to validate the proposed approach, details on the dataset used, the evaluation metrics, and the result analysis. in section 5, the paper concludes and summarizes the main findings of the research. ## literature survey the origins of the indian qa system can be traced back to the early 2000s when hindi-english crosslingual qa became feasible @xcite . another system used relational databases and keywords to convert user queries into sql queries and present answers in the user's native language @xcite . several other approaches were proposed, including a natural language interface to relational databases using paninian grammar and kārakas @xcite , the use of universal networking language (unl) for representing the meaning of text in the source language without translation @xcite , and rule-based systems for hindi qa @xcite . additionally, there were developments in web-based qa systems @xcite , pattern matching algorithms for qa @xcite , question classification models @xcite , answer sentence selection models for qa @xcite and deep learning-based frameworks for cross-lingual @xcite and multi-lingual qa @xcite . recent experiments explored the use of transformer models pre-trained on multiple languages, with a focus on hindi and tamil qa, achieving improved performance in extractive qa tasks @xcite . a summary of question answering task for indic languages is presented in panini identifies six kārakas in as . the words that occur within the sentences and the role of these words within the given context, acting as a via media between the lexical/grammatical expression on one side and their semantics. kārakas have also been studied in the context of automatic question generation @xcite . we earlier attested the utility of kārakas as similarity measures in hindi and english extractive qa systems @xcite , and compared them to other known similarity features. therein we generated a feature representation for the entire passage by employing various similarity measures. the highest accuracy for selecting the best answer sentence in hindi was achieved when combining the kāraka features with cosine similarity and context word overlap. cosine similarity was computed based on vectors derived from large pre-trained models, that have limited availability. in this research, we investigate an alternative method that relies solely on kārakas for initial answer sentence classification and then employs the likelihood score ## kāraka-based answer sentence retrieval the task of answer retrieval holds significant importance in a qa system. when presented with a natural language query and a collection of sentences derived from an information extraction system, the answer retrieval module is responsible for identifying the appropriate phrase/sentence that precisely provides the answer to the user's query. the problem at hand involves a question q and a document or context (passage) containing multiple candidate answer sentences (s 1 , s 2 , ..., s n ) for the question. our primary goal is to locate the most appropriate sentence, denoted by s i (where if the identified sentence corresponds with the actual answer, then we deem the question q to be correctly answered. we do not consider the real world scenario of unrestricted questions. to accomplish this task, we employ a supervised learning approach that treats the task as a classification problem. the diagram presented in figure 1 illustrates the modules utilized in the kāraka-based answer retrieval process. ## conclusion in this work, a kāraka based answer sentence retrieval approach is presented and its effectiveness is demonstrated through experimental analysis on a large benchmark corpus. the results clearly show that the accuracy of kāraka extraction impact the performance of answer retrieval, which emphasizes the need for further research and investment to enhance it. for languages lacking extensive lexical resources like propbank and framenet and considering the scarcity of nlp resources for these lan-guages, the proposed approach holds promise. we do not evaluate or dispute the usefulness of these alternative approaches. however, we observe that the identification of verb-specific semantic roles, which requires the development of comprehensive language-specific verb frames, poses a challenge, especially in low-resource languages. our proposal is that kāraka annotation results into capturing semantic, facilitating tasks such as qa using smaller language models and lexicons.