T amil ATIS : Dataset for Task-Oriented Dialog in T amil (2022)

Abstract: Task-Oriented Dialogue (TOD) systems allow users to accomplish tasks by giving directions to the system using natural language utterances. With the widespread adoption of conversational agents and chat platforms, TOD has become mainstream in NLP research today. However, developing TOD systems require massive amounts of data, and there has been limited work done for TOD in low-resource languages like Tamil. Towards this objective, we introduce TamilATIS - a TOD dataset for Tamil which contains 4874 utterances. We present a detailed account of the entire data collection and data annotation process. We train state-of-the-art NLU models and report their performances. The joint BERT model with XLM-Roberta as utterance encoder achieved the highest score with an intent accuracy of 96.26% and slot F1 of 94.01%.

## introduction task-oriented dialog (tod) systems enable a user to use natural language directions to complete specific tasks. recently, such systems have been successfully deployed in smart applications such as amazon's echo and spotify's car thing. there are several components that are critical to the performance of a tod system. these components are natural language understanding, dialogue state tracking (dst), and response selection. in this work, we focus on the nlu component. nlu aims to semantically parse an input utterance and typically has two tasks: intent classification and slot filling (refer intent classification deals with identifying the underlying motivation or the goal of the user query. it is modelled as a sequence classification task. the simplicity and conciseness of the utterances, paired with the necessity to scale to multiple domains, pose bottlenecks to intent detection. slot filling deals with identifying entities present in an utterance that corresponds to certain slots in the user find morning flights to chennai o b-period o o b-fromcity intent: find-flight intent classification and slot filling have been widely researched for the english language. the approaches presented in these works achieve excellent performance due to the availability of large amounts of high-quality and human-annotated datasets. however, such performance has not been achieved for several low-resource languages due to a lack of data. developing tod datasets for low-resource languages is essential to the proliferation of nlp technologies in these communities and contributes towards inclusivity and diversity of language resources. to facilitate this, we present a dataset named tamilatis, which contains 4874 utterances in tamil and their corresponding slots and intent annotations. the following are some of the main contributions of this work: the full dataset and the source code of the baseline models are available at https://github. com/ramaneswaran/tamil_atis utterance intent ex. 1 (morning flights from vadodara to vijayawada on april 20) ஏ ர 20 அ வேதாதராவ இ வ ஜயவாடாவ காைல வ மான க atis_flight slots b.mn b.dn o b.fcn o b.tcn b.pod ex. 2 (what is the fare for a taxi to agartala?) அக தலாவ ஒ டா ஸி எ ன க டண atis_ground_fare slots b.cn o b.tpt o o ex. 3 (what is the seat capacity of the 733?) 733 இ இ ைக திற எ ன? atis_capacity slots b.ac o o o o ex. 4 (i would like to take a flight from kochi to tiruchirappalli on saturday morning in vistara) வ தாராவ சன கிழைம காைல ெகா சிய லி தி சிரா ப ள வ மான ெச ல வ கிேற atis_flight slots b.an b.ddn b.dpd b.fcn b.tcn o o o b.mn month name b.dn day number b.fcn from city name b.tcn to city name b.pod period of day b.cn city name b.tpt transport type b.ac aircraft code b.an airline name b.ddn depart day name b.dpd depart period of day 2 related work intent classification and slot filling are two key challenges modelled separately or jointly for natural language understanding (nlu). joint modelling approaches have attained state-of-the-art performance, and have demonstrated that there exists a significant correlation between the two tasks. prior works have implemented cnn-crf @xcite , recnn @xcite , joint rnn-lstm @xcite , attentionbased birnn @xcite , and slot-gated attention-based model @xcite and more recently have used bert @xcite and bilstm based @xcite approaches. intent classification and slot filling functions are core modules for nlu in task-oriented dialogue (tod) systems @xcite @xcite @xcite @xcite . since these tasks are characterized as sequence classification and token tagging tasks, sentence encoder models have been utilized to solve them. the two extensively utilized large scale datasets in english (high resource language) for this purpose in nlu are: atis @xcite , which features audio recordings of individuals booking flight reservations, and snips @xcite , which is gathered from snips' personal voice assistant. dao et al. in indian languages, there have been works to synthesize training data by using google translate and proposed cnn+lstm based architecture @xcite . @xcite released a hindi dialogue restaurant search (hdrs) corpus consisting of 1.4k human-to-human typed dialogues collected using the wizard-of-oz paradigm and compared various state-of-the-art dst models. small sized datasets were constructed manually and used with indic and code-switched tod systems (jayarao and srivastava, 2018). @xcite used mbert based semantic tracking to associate the slot tokens to the respective tokens in the utterance and employed google translate api, morphological characteristics and semantics based heuristic slot aligner to publish a dataset for dravidian languages like kannada and tamil. the tamilatis corpus is collected to promote research and development in the field of task-oriented dialogue systems for the tamil language. it contains 4874 utterances related to airline-related enquiries. below, we describe the data collection and data annotation processes and give detailed statistics about the tamilatis dataset. ## experimental settings we benchmark the tamilatis dataset on eight state-of-the-art nlu models. in this section, we describe the models used and present the baseline results obtained. the problem of intent detection and slot filling can be cast as a generation task or a classification task, and in our baseline models, we include both of these types of architectures. ## result and analysis in this section, we report the results obtained by the baseline models. we evaluate the nlu performance for slot filling using the f1(micro) score and intent prediction using accuracy. the score obtained by each of the baselines is shown in since the focus of these experiments is to just establish baselines and provide a starting point for further exploration, we restrict ourselves from indepth error analysis. the baselines can be broadly classified into two approaches, generative approaches and classification-based approaches. the lowest scoring model is seq2seq, which uses a simple encoderdecoder architecture to generate intent and slot details. this approach scores a decent accuracy of 83.11 for intent detection but a low f1 score of 56.99 for slot filling. other generative approaches like stack-propagation, slotrefine and gl-gin achieve much better performances. these approaches have components in their architecture (like stack-propagation framework, graph interaction layers etc) that help them explicitly model the relationship between the slots and intent leading to superior performance. classification based approaches like sf-id, capsule nlu and slot-gated achieve performance similar to the three generative architectures mentioned before. all of these approaches we discussed use sophisticated techniques to better model the interaction between intent and slot information and yield noticeable improvements over a simple architecture like seq2seq. however, the best score is obtained by joint-bert. it obtains an intent accuracy of 96.26% and slot f1 score of 94.01%. this is an absolute improvement of 1.96% and 1.91% in intent accuracy and slot f1 over the second-best performing model (slotrefine). jointbert shows the effectiveness of transformer architectures pre-trained on large datasets when applied to downstream tasks like intent detection and slot filling. we further investigate the jointbert architecture by using different multilingual models as utterance encoders. ## conclusion in this paper, we presented tamilatis, a tod dataset in tamil with 4874 utterances. we benchmarked the dataset with eight state-of-the-art nlu models and reported their intent accuracy and slot f1. both generative and classification-based approaches perform similarly and achieve high intent accuracy and slot f1 score. we also highlighted the importance of modelling the relation between intent detection and slot labelling to yield performance improvement. the joint bert model with xlm-roberta as utterance encoder achieved the highest score with an intent accuracy of 96.26% and slot f1 of 94.01%. in future work, we plan to extend this dataset to other low-resource dravidian languages like malayalam, kannada and telugu. this would contribute towards the proliferation of tod technology in the communities that speak these languages and also promote the development of multi-lingual tod models for dravidian languages. having multidomain utterances is another important research direction.