Task-Oriented Dialog Systems for D ravidian Languages (2021)

Abstract: Task-oriented dialog systems help a user achieve a particular goal by parsing user requests to execute a particular action. These systems typically require copious amounts of training data to effectively understand the user intent and its corresponding slots. Acquiring large training corpora requires significant manual effort in annotation, rendering its construction infeasible for low-resource languages. In this paper, we present a two-step approach for automatically constructing task-oriented dialogue data in such languages by making use of annotated data from high resource languages. First, we use a machine translation (MT) system to translate the utterance and slot information to the target language. Second, we use token prefix matching and mBERT based semantic matching to align the slot tokens to the corresponding tokens in the utterance. We hand-curate a new test dataset in two low-resource Dravidian languages and show the significance and impact of our training dataset construction using a state-of-the-art mBERT model - achieving a Slot F1 of 81.51 (Kannada) and 78.82 (Tamil) on our test sets.

## introduction with the surge in popularity of digital assistants (google, siri, alexa) task-oriented dialog (tod) systems have become commonplace in nlp research today. tod systems are designed to complete a particular user goal by understanding requests from users and providing relevant information @xcite . they typically consist of four major components -natural language understanding @xcite , dialog state tracking ## related work tod systems -in the recent years, there has been a lot of work on building tod systems broadly around two themes -building out each component of the system independently @xcite @xcite and end-to-end tod systems @xcite @xcite . though end-to-end systems show better generalization, they usually require larger amounts of training data and may not be very feasible for low-resource settings. in this work, we primarily tackle the nlu component which performs intent detection and slot identification. since these tasks are framed as sequence classification and tagging tasks, sentence encoder models have been used to tackle them. prior works have utilized recurrent neural networks @xcite @xcite and more recently have used bert @xcite . multilingual tod -though there currently exist multiple large-scale datasets for tod @xcite @xcite , they are monolingual english corpora. @xcite released the facebook tod dataset which contains utterances from three languages -english, spanish, thai geared towards crosslingual transfer of dialog. on this dataset, there have been works on zero and few-shot transfer using latent variable transfer @xcite , mixed language training @xcite but they don't display consistent gains over augmentation with mt+word alignment data. on indian languages, there have been works towards indic and code-switched tod systems @xcite @xcite but these datasets are manually constructed and are smaller in size. furthermore, they're primarily for hindi and there currently exists no dataset for languages like kannada, telugu, etc. auto-construction of datasets -given the cost and difficulty in acquiring annotators, prior works have employed methods to create synthetic training data. @xcite use google translate to create synthetic utterances in indic languages towards the task of spoken intent detection but do not tackle the slot filling task. more recently lópez de lacalle et al. (2020) employed a seq2seq translation and word alignment to project the spanish slot tags to basque. however, we show that given the flexible word ordering and rich morphology, word alignment systems do not work particularly well for dravidian languages. furthermore, since dravidian languages do not have large open-source corpora to learn mt and alignment systems reliably, we utilize google translate api, morphology and semantics based heuristic slot aligner which does not require any parallel data. ## dataset construction the training dataset for the low-resource languages is constructed from the facebook multilingual task-oriented dialog dataset @xcite which contains utterances in english, spanish and thai tagged with the corresponding intent and slot labels by hand. ## model we learn the intent and slot filling tasks jointly with bert @xcite , more specifically, the multilingual variant -mbert which is pre-trained on 104 languages. we pass the [cls] vector representations onto a multi-layer perceptron to classify the intent onto one of the predefined intent classes. we pass the token representations produced by the mbert model onto another multi-layer perceptron to identify the slots. in the case of a word consisting of multiple tokens, we only use the first sub-word token and ignore the others during training and test. figure 2 shows the architecture of our system. since mbert is pre-trained across many languages, we follow the same architecture for our zero-shot and few-shot experiments. ## experiment setup we use the base model of bert and mbert in all our experiments with a batch size of 64 and a learning rate of 1e-5 with adam (kingma and ba, 2014) optimizer. on zero and few shot transfer experiments, the model is trained for about 10 epochs before evaluation/fine-tuning on few shot data.