T e Q u AD : T elugu Question Answering Dataset (2022)

Abstract: Recent state of the art models and new datasets have advanced many Natural Language Processing areas, especially, Machine Reading Comprehension tasks have improved with the help of datasets like SQuAD (Stanford Question Answering Dataset). But, large high quality datasets are still not a reality for low resource languages like Telugu to record progress in MRC. In this paper, we present a Telugu Question Answering Dataset - TeQuAD with the size of 82k parallel triples created by translating triples from the SQuAD. We also introduce a few methods to create similar Question Answering datasets for the low resource languages. Then, we present the performance of our models which outperform baseline models on Monolingual and Cross Lingual Machine Reading Comprehension (CLMRC) setups, the best of them resulting in an F1 score of 83 % and Exact Match (EM) score of 61 %.

## introduction mrc is one of the key tasks in nlp, where we test the ability of machines to understand and answer the questions using provided textual knowledge. in common machine reading comprehension tasks, for a given query, the machine needs to extract the answer from the context (paragraph) in the form of span indices. a popular large-scale annotated reading comprehension dataset -squad @xcite , revolutionised the research interest in this area for english. and though decent research work has been done in mrc for a few indian languages, for languages like telugu, which is a dravidian language, still need similar resources for such natural language understanding task. creating an rc dataset of good quantity & quality is difficult, requires manpower, and is timeconsuming. for a few languages, the dataset is created by translating squad and using few matching techniques to extract the span indices of answers in the target language @xcite , @xcite al. (2021), artetxe et al. (2019) ). for others, the dataset is created by using the methodology followed in the creation of squad @xcite al. (2019), efimov et al. (2020), cui et al. (2018), d'hoffschmidt et al. (2020) ). our idea is to introduce a few heuristics based approaches to create the datasets for a low resource language (telugu) using the resources from a high resource language via translation. an obvious challenge is to extract the span of the answers in the translated contexts. with translation, due to language divergences, the position and structure of the answer in the context will vary in the translated language, making it difficult to use straight-forward approaches, like translation candidate matching, to find the position of the answer in the context. we focused on the span extraction process, which is crucial for such a dataset creation after translation. we applied these methods to squad v1.1 and created tequad, a mrc dataset for telugu consisting of 82k parallel telugu-english triples (paragraphs, questions, and span indices of answers). the intention to create a parallel dataset is to exploit the advantage of cross-lingual reasoning. we also introduce a supervised approach to extract the span of the most probable answer from the target paragraph. this span extractor can later aid in data augmentation for mrc in low-resource languages. in cases where the heuristics do not work, our supervised method performs better than the matching techniques due to its ability to consider contextual semantic information using pre-trained language models. both monolingual and cross-lingual setups of multilingual bidirectional encoder representations from transformers (bert) were trained on tequad and evaluated on tidyqa @xcite and on two other test datasets, which we created manually by correcting a few samples from translated squad and by using wikipedia articles respectively. our dataset and code are available here foot_0 . ## related work several datasets such as squad @xcite , newsqa dataset @xcite and cnn/dailymail @xcite , etc fulfills the necessity of resources in english for qa tasks. although these datasets helped in attaining enormous progress for this specific language in nlp, other languages are still unexplored in this area due to the scarcity of high-quality annotated datasets in corresponding languages. while the generation of reading comprehension corpora in other languages is costly and time-consuming, few works such as @xcite al. (2019), efimov et al. (2020), cui et al. (2018 ), d'hoffschmidt et al. (2020) developed rc datasets natively. clark et al. (2020) presented a question answering dataset covering 11 typologically diverse languages including telugu. few others propose methods to boost the functioning of the model in low-resource settings. hsu et al. (2019) explored zero-shot cross-lingual transfer learning on reading comprehension tasks and suggested that translation from source to target languages is not necessary. bornea et al. (2020) presents translation-based data augmentation mechanism to improve multilingual transfer learning. liu et al. (2020) and cui et al. (2019) talks about leveraging translated information from the high resource languages to perform well in low resource languages. cui et al. (2019) presented several back translational approaches for cross-lingual experiments. they have also discussed techniques to align the answer phrases in the target language. stating the disadvantages of such approaches and the necessity to overcome them, they introduced a novel model called 'dual bert', which has the ability to learn semantic information from bilingual qa pairs and utilize the learned knowledge to improve mrc in low resource languages. yuan et al. (2020) introduced phrase boundary supervision tasks to improve the answer boundary detection capability in the low resource mrc models which are trained with training data from high resource languages to exploit cross-lingual transfer learning. post correction methods to improve the span of the extracted answer are addressed in @xcite . they added additional layers on top of a 3 corpus creation a simple and cost-efficient technique to create a dataset for an nlp task is to translate a wellannotated existing dataset. when it comes to mrc tasks, squad is favorite for its quality and adaptability to recent implementations of deep learning models. this span extractive qa data is created from english wikipedia articles by crowd workers. more than 100000 triples were generated in squad1.1. a triple consists of a question, an answer to the question in the form of span indices, and a context where the answer can be found. we translated the english squad triples to the telugu language using online google translator foot_1 , obtaining translated triples consisting of translated telugu paragraphs, questions, and answers. after translation, a well-known issue is difficulty in extraction of the span indices of the translated answers. considering the different possibilities of translated telugu answer phrase's presence in the translated telugu context, we followed multiple techniques to extract the span for answer phrases. the purpose of following different techniques is to create as much synthetic data as possible for telugu mrc. we also present a supervised span extraction technique to handle the cases where rulebased methods fail. ## mrc experiments we experimented on tequad in monolingual and cross-lingual setups. the pre-trained multilingual-bert (mbert) trained in 104 languages including telugu and english is employed for obtaining encoded representations for both languages. we use nltk tokenizer followed by bert word piece tokenizer to sub tokenize the tokens in all the experiments. experimented with a batch size of 64 and sequence length of 512. as in google's tensorflow implementation of bert, adam with weight decay optimizer is considered with different learning rates for different experimental setups. our models have been trained on google cloud tpu v2. ## results and observations the most important observation from the results is that the models fine tuned on tequad performed way better than the zero-shot mbert model. on average, a 40% increase in f1 and em scores were obtained in all setups. our experiments also show that performance on the wiki test dataset is better than others. it must be noted that wiki test dataset is well annotated, created from telugu wikipedia articles, and has better quality than the translated & corrected test dataset. in contrast, the quality of the tydiqa telugu samples is low and not recommended for fair telugu mrc evaluation. most of the queries in tydiqa revolve around the area of lands, zip codes, date of births/deaths etc. learning on a set of similar types of questions more often will make the model over- ## conclusion and future work as a move towards the creation of the qa dataset for indian languages, this work took a step forward in the mrc corpus creation using translation. to record decent performances in nlp tasks for low resource languages, sufficient resources are necessary. as we discussed, the creation of such resources is difficult. resources from high resource languages like english might be considered for the creation of datasets for low-resource languages like telugu to create abundant data for different nlp tasks. we introduce creation/correction techniques for such datasets improving the quality along with the quantity of the datasets along with providing mechanisms for further augmenting data. in the future, we would like to improve the mrc task for telugu, provide a collection of pre-trained models trained on openly available resources in the telugu language, as well as create additional data resources for the telugu language.