Machine Translation for Multilingual Intent Detection and Slots Filling (2022)

Abstract: We expect to interact with home assistants irrespective of our language. However, scaling the Natural Language Understanding pipeline to multiple languages while keeping the same level of accuracy remains a challenge. In this work, we leverage the inherent multilingual aspect of translation models for the task of multilingual intent classification and slot filling. Our experiments reveal that they work equally well with general-purpose multilingual text-to-text models. Furthermore, their accuracy can be further improved by artificially increasing the size of the training set. Unfortunately, increasing the training set also increases the overlap with the test set, leading to overestimating their true capabilities. As a result, we propose two new evaluation methods capable of accounting for an overlap between the training and test set.

## introduction home assistants are omnipresent in everyday life. we expect to have an assistant at our disposal at any time using our phone, watch, or car -irrespective of our language. scaling home assistants to multiple languages brings additional challenges to nlu and asr components. there are two options: a single model per language or a shared model for all languages. a single model per language works well for resourcerich languages such as english. however, lower resource languages benefit from the cross-lingual knowledge transfer of a single model dealing with all languages @xcite . this trade-off applies to any multilingual system @xcite . while multilingual intent classification and slot filling datasets exist, their language coverage is limited, except for massive @xcite , a new dataset focused on multilingual intent detection and slot filling. the authors translated and localized an english-only dataset in 50 topologically diverse languages. massive provides a good base to scale existing intent detection and slot filling methods to multiple languages. the traditional way to tackle multilingual intents detection and slot filling is to use multilingual models such as xlm-r @xcite , or mt5 @xcite . these models are similar to their monolingual counterparts @xcite except for the multilingual data used to train them. 1 this approach has been shown to work in multiple studies @xcite . however, massive has an additional overlooked aspect: utterances are direct translations of one another. in this work, we approach the task of intent classification and slot filling as a translation task: we translate the original utterance into the annotated utterance. for example, we translate the utterance what is the temperature in new york? into the annotated utterance weather_query|what is the [weather_descriptor : temperature] in [place_name : new york]. 2 the typical use of translation models for intent detection and slot filling is to augment the size of an existing dataset @xcite . however, we believe the inherent multilingual capabilities of these models make them excellent candidates for multilingual intent detec-tion and slot filling. to this end, we leverage the recently released translation model no language left behind (nllb) @xcite capable of translating between 202 pairs of languages simultaneously using a shared encoder-decoder. we anticipate that the wide range of languages covered by the model will help us deal with lower resources languages present in the massive dataset. better modeling is only half the story. using more data also helps improve performance. for example, although the massive dataset displays a large training set of more than 500k training examples, the seed data is only around 10k training examples. therefore, we used gpt-3 @xcite to generate additional training data using a dual-model approach. we also leveraged a dataset close to the seed dataset of massive. as a result, after translating our new training examples to the 50 remaining languages, our training set contains more than 2m training examples -4x the size of the original training set. our experiments reveal that translation models such as nllb are a good fit for intent classification and slot filling. however, their performance sharply drops in languages that do not use spaces because of tokenization issues. unfortunately, the additional training data significantly overlaps with the massive test set. as a result, we propose two methods capable of dealing with overlaps: weighted exact match and logistic regression. we conclude this introduction by summarizing our contributions: we release our model foot_2 , utterance translation model foot_3 , and generated data foot_4 on the huggingface hub. ## related work the problem of multilingual intent detection and slot filling is not new. @xcite provides an excellent introduction to the subject. we divide our related work section into three parts. we start by reviewing the general problem of taskoriented semantic parsing (i.e., intent detection and slot filling). next, we review the models commonly used, and lastly, we review the available multilingual datasets. ## data there exist multiple alternative datasets to study multilingual intent detection and slot filling. however, in this work, we use the largest one available: the massive dataset. @xcite is a dataset assembled by translating and localizing an existing english-only dataset in 50 topologically different languages. ## experiments this section describes our experiments in applying nllb to the task of intent classification and slot filling. nllb is a translation model. while we could repurpose nllb to the task of intent classification and slot filling directly, we choose to first pre-train it on a translation task. ## results this section presents a high-level analysis of our results. ## training & test set overlap this section analyses the similarity between the training sets and the massive. next, we look for evaluation methods capable of correcting for the overlap between the training and test set. exact duplicates an analysis of the data reveals problematic overlaps between the training sets and the massive test set. however, this overlap is unequal across the training sets and languages. close duplicates some examples may not be exact duplicates but close duplicates. for example, call the dentist and olly please call the dentist now. we use character n-grams to measure the similarity between two utterances as similarity metric between two utterances. we search for the most similar training example for each example in the test and record their n-gram similarity. 11 figure 2 shows the distribution of maximum similarity between the test set and our three training sets for the english subset and across all locales. it is clear from figure 2 that the english synthetic dataset overlaps significantly with the english massive test set. however, as for the exact duplicates, the translation and localization process reduces this overlap but does not eliminate it. a naive solution would be to remove training examples that overlap with the test set. however, how does one decide what is a close duplicate? furthermore, as the training set grows, some overlap with the test is inevitable. we argue that the problem is not the training data but the evaluation metric. we need an evaluation metric capable of controlling for the overlap between the test and training sets. ## future work in this work, we estimated the similarity between two utterances using character n-grams. however, while this captures lexically similar utterances, it fails to capture utterances semantically similar but lexically different. for example, these two utterances are highly similar, although they only share a single common token: what time is it? and tell me the time. future work can tackle this by using multilingual sentence encoders such as laser3 @xcite , multilingual universal sentence encoder @xcite , or multilingual models on sentence transformers @xcite . this work did not explicitly address crosslingual training and instead relied on the crosslingual pre-training of the translation model. future work could combine a translation model with crosslingual training methods such as xtune @xcite , or x-mixup @xcite . section 8.1 showed the limitation of subword tokenization methods. future work could explore methods which do not uses subword tokenization such as byt5 @xcite . ## conclusion in this work, we showed that a translation model such as nllb can perform the task of intent classification and slot filling. because of tokenization issues, it is, however, suboptimal with non-spaced languages. moreover, we showed that artificially increasing the training sets' size leads to improved performance. unfortunately, we also show that this added data can overlap with the existing test set, distorting the true evaluation of these models. the normal way to overcome this problem is to remove the overlap from the training set. however, deciding on what constitutes an overlap remains an open question. therefore, we argued that the data overlap is not the problem -the evaluation metric is. as a result, we proposed two evaluation metrics that control the training/test overlap. both metrics reveal that the model trained on overlapped data improves the results on non-overlapped data. however, our analysis also reveals that these models struggle to beat random chance when evaluated on utterances dissimilar to the training set.