Exploring Expected Answer Types for Effective Question Answering Systems for low resource language (2024)

Abstract: Question-answering (QA) systems play a pivotal role in natural language processing (NLP), powering applications such as search engines and virtual assistants by providing accurate responses to user queries. However, building effective QA systems for Dravidian languages, like Tamil, poses distinct challenges due to the scarcity of resources and the linguistic complexities inherent to these languages. This paper introduces a novel method to enhance QA accuracy by integrating answer-type features alongside traditional question and context inputs. We fine-tuned both mono- and multilingual pre-trained models on the Extended Chaii dataset, which comprises Tamil translations from the SQuAD dataset, as well as on the SQuAD-EAT-5000 dataset, consisting of English-language instances. Our experiments reveal that incorporating answer-type features significantly improves model performance compared to using only question and context inputs. Specifically, for the Extended Chaii dataset, the MuRIL model achieved the highest F1 score of 53.89, surpassing other pre-trained models, while RoBERTa outperformed BERT on the SQuAD-EAT-5000 dataset with a score of 82.07. This research advances QA systems for Dravidian languages and underscores the importance of integrating linguistic features for improved accuracy.

## introduction a question answering (qa) system is a challenging endeavor in the field of nlp, aiming to enable computers to derive final answers by reasoning from the semantic information of natural language queries @xcite . qa systems have become a cornerstone in nlp, finding applications in search engines, virtual assistants, and various other domains where accurate and efficient information retrieval is crucial. qa is a job in nlp and information retrieval that tries to automatically answer a human-posed question in natural language @xcite . these systems are designed to interpret and respond to user queries with precise answers, thereby enhancing the user experience and accessibility of information. while significant progress has been made in developing qa systems for widely spoken languages, creating effective qa systems for less-resourced languages, such as dravidian languages, poses unique challenges. tamil, a prominent dravidian language, exemplifies these challenges due to its rich linguistic structure and limited availability of annotated datasets. developing qa systems for tamil is particularly challenging because of the inherent linguistic complexities and the scarcity of resources @xcite . the traditional approach of relying solely on question and context inputs often falls short in addressing these challenges effectively. to overcome these obstacles, this paper proposes a novel approach that integrates answer-type features into the qa system, providing an additional layer of contextual understanding that enhances accuracy. in this study, we fine-tune several pre-trained models, both mono-lingual and multi-lingual, using a tamil dataset translated from the widely recognized squad dataset. by incorporating answer-type features, we aim to improve the performance of these models in predicting accurate answers. our experimental results demonstrate that models utilizing answer-type features significantly outperform those that rely solely on question and context inputs. ## related study in this section, we describe some related works regarding question answering systems in the tamil language. our literature review revealed that while english has several benchmark qa datasets, models have shown better performance in english than in other high-resource languages. our literature review revealed that few initiatives have been implemented for tamil question-answering tasks. there are few qa datasets available for indic languages @xcite . some of these works have been referenced here. karthik @xcite implemented mbert over google extended chaii dataset which is a multi-lingual question-answering dataset consisting of a total of 1114 records of multiple languages like hindi and tamil. the authors implemented mbert with a constrative training approach. authors evaluated the performance of mbert using the jaccard similarity metric. recent experiments @xcite have investigated the application of transformer models pre-trained on multiple languages, specifically focusing on hindi and tamil question-answering (qa). these studies have demonstrated enhanced performance in extractive qa tasks. ram vignesh (namasivayam and rajan, 2023) implemented four approaches to solve answer prediction over the chaii multi-linguaval dataset consisting of 746 data instances for hindi and 368 data instances for tamil. this work is implemented with fine-tuning of mbert, xml-roberta and muril multi-lingual pre-trained models and evaluated the performance using exact match and f1-score along with jaccard similarity metrics. rajendran et.al @xcite designed qa system for the tourism domain. this qa system uses dependency information while fetching the answers to the user queries. srivatsun et.al @xcite , implement the mbert model over extended chaii dataset and evaluate the performance of the model using the belu metric. the survey revealed that, despite the availability of several benchmark qa datasets for english, models perform well in english but not in other high-resource languages. there is a notable lack of benchmark datasets for indic languages, which hinders the development of efficient qa systems. in addition, from the literature we found that qa system provides a precise answer to the user question based on having prior knowledge about the expected answer type. it also leads to improve the performance of the qa system @xcite . so in this research, by incorporating additional information alongside the usual input parameters to the qa system, we are aiming to build an efficient qa system for the tamil language even with limited data. ## proposed method the proposed approach in this study aims to address the challenges of developing effective qa systems for dravidian languages, specifically tamil, by incorporating additional linguistic features into the model training process. the foundation of this approach lies in the creation of a tamil dataset translated from the well-known squad dataset. this translation ensures that the models are trained on high-quality, contextually relevant data, which is crucial for handling the unique linguistic characteristics of tamil. the key innovation in the proposed approach is the inclusion of answer-type features alongside the traditional question and context inputs. answer types classify the expected responses such as person, location, or date, providing the model with specific guidance that enhances its ability to generate accurate answers. this additional feature helps the model to better understand the na-ture of the question and focus its prediction efforts more effectively. fig 2 represents the architecture of the proposed qa system. the proposed qa system architecture consists of two modules: ## conclusion in conclusion, this study addresses the challenges of developing effective question-answering (qa) systems for dravidian languages, specifically tamil, by introducing a novel approach that incorporates answer-type features alongside traditional question and context inputs. by fine-tuning both mono-and multi-lingual pre-trained models on a extended chaii tamil qa dataset derived from the squad dataset, the research demonstrates that the inclusion of answer-type features significantly enhances the accuracy of qa systems. notably, the muril model achieved the highest f1 score of 53.89, outperforming other evaluated models. similarly, over squad-eat-5000 dataset, roberta model achieved highest f1-score with the inclusion of additional expected answer type information compared with bert. this work underscores the importance of integrating linguistic features to improve the performance of qa systems for underresourced languages, thereby contributing to the broader advancement of nlp technologies for dravidian languages.