Multilingual and Cross-Lingual Intent Detection from Spoken Data (2021)

Abstract: We present a systematic study on multilingual and cross-lingual intent detection (ID) from spoken data. The study leverages a new resource put forth in this work, termed MInDS-14, a first training and evaluation resource for the ID task with spoken data. It covers 14 intents extracted from a commercial system in the e-banking domain, associated with spoken examples in 14 diverse language varieties. Our key results indicate that combining machine translation models with state-of-the-art multilingual sentence encoders (e.g., LaBSE) yield strong intent detectors in the majority of target languages covered in MInDS-14, and offer comparative analyses across different axes: e.g., translation direction, impact of speech recognition, data augmentation from a related domain. We see this work as an important step towards more inclusive development and evaluation of multilingual ID from spoken data, hopefully in a much wider spectrum of languages compared to prior work.

## introduction and motivation a crucial functionality of natural language understanding (nlu) components in task-oriented dialogue systems is intent detection (id) @xcite @xcite . in order to understand the user's current goal, the system must classify their utterance into several predefined classes termed intents. 1 scaling dialogue systems in general and intent detectors in particular to support a multitude of new dialogue tasks and domains is a challenging, timeconsuming, and resource-intensive process @xcite . this problem is further exacerbated in multilingual setups: it is extremely expensive to annotate sufficient task data in each of more than 7,000 languages @xcite . as a consequence, the current id work has been largely constrained only to english, and standard id benchmarks also exist only in english @xcite @xcite @xcite . the need to widen the reach of dialogue technology to other languages has been recognised only recently, and thus even text-based multilingual id datasets are still few and far between @xcite provide nlu data in three languages (english, spanish, thai), while a more recent multiatis++ dataset @xcite ) manually translates the atis dataset @xcite ) from english to 8 target languages, extending the work of @xcite which translated portions of the english atis data to hindi and turkish. 2 despite these efforts, there are still prominent gaps remaining: 1) a large number of (even major) languages is still uncovered, 2) there are no multilingual data for specialized and well-defined domains such as e-banking, and 3) most importantly, all intent detection datasets to date are text-based. in other words, current work completely ignores the fact that many conversational systems are inherently voice-based, and that telephony quality and errors in automatic speech recognition (asr) even prior to intent detection may have fundamental impact on the final intent detection performance. consequently, the impact of asr on multilingual intent detection has not been studied before. contributions. inspired by the current gaps, 1) we present the minds-14 dataset (multilingual intent detection from speech), a first multilingual evaluation resource for id from spoken data. it originates from the use of a commercial voice assistant and real-life industry needs: it covers 14 intents in the banking domain in 14 different language varieties, making it the most comprehensive multilingual id dataset to date. 2) we present a systematic evaluation and comparison of current state-of-the-art multilingual and cross-lingual id models, which rely on machine translation and current cutting-edge multilingual sentence encoders, multilingual use @xcite and labse @xcite . 3) we provide additional analyses to further profile the potential and current id gaps in multilingual voice-based contexts, including augmentation with data from a similar domain, target-only versus multilingual training, and aggregations of n-best asr hypotheses. our results demonstrate that strong id results can be achieved for all languages represented in minds-14, but we also indicate the crucial importance of in-domain model fine-tuning and fewshot learning, reporting strong gains over zero-shot transfer models. in hope to motivate and inspire further work on multilingual and voice-based id and future extensions to lower-resource languages, we release minds-14. the release includes the original speech data as well as the asr data, and is available online at: s3://poly-public-data/ minds-14/minds-14.zip. ## minds-14: dataset collection final dataset and languages covered. the final minds-14 dataset covers 14 intents in the banking domain with accompanying spoken and "asr-ed" text utterances. the intents were sampled from a set of 90+ fine-grained intents used by a commercial banking voice assistant, so that all intents have a clear and non-overlapping semantics, and are easy to understand by non-experts, i.e., crowdsourcers. around 50 examples for all 14 intents are collected in 14 different language varieties, with the exact numbers available in the appendix. the language set includes a) three varieties of english: british (en-gb), us (en-us), and australian (en-au); b) germanic and romance western european languages: french (fr), italian (it), spanish (es), portuguese (pt), german (de), and dutch (nl); c) slavic: russian (ru), polish (pl), and czech (cs); and d) asian languages: korean (ko) and chinese (zh). the choice of languages was driven by (a) the number of native speakers and (b) the number of participants on the used crowdsourcing websites, (c) combined with some typological diversity @xcite . disclaimer: we acknowledge that our language sample is typologically less diverse than in some recent evaluation sets for text-based multilingual language understanding @xcite : we consider the proposed dataset as only a first step towards more equitable research in this area, and our goal in this work was establishing and validating the data collection and benchmarking methodology with higher-resource languages before extending the focus to lower-resource ones. ## multilingual id: methodology a standard transfer learning paradigm @xcite fine-tunes a pretrained language model such as bert @xcite or roberta @xcite on the annotated task data. for the intent classification task in particular, @xcite have recently shown that full finetuning of the large pretrained model is not needed at all. in contrast, they propose a more efficient feature-based approach to intent detection. here, fixed universal sentence encoders such as use @xcite or convert @xcite are used "off-the-shelf" to encode utterances, and a standard multi-layer perceptron (mlp) classifier is then learnt on top of the sentence encodings. casanueva et al. (2020) demonstrate that the feature-based approach to intent classification yields performance on-par with the full-model finetuning, while offering improved training efficiency. therefore, due to the large number of executed experiments and comparisons in this work, and preliminary results which corroborated the findings from prior work @xcite , we opt for this efficient approach to id. we evaluate two widely used state-of-the-art multilingual sentence encoders, but remind the reader that decoupling mlp from the encoder allows for a wider exploration of other available multilingual sentence encoders @xcite . in what follows, we provide only brief descriptions of each encoder in our evaluation; for more details we refer the reader to the original work. muse @xcite is a multilingual version of the universal sentence encoder (use) model for english @xcite . it relies on a standard dual-encoder neural framework @xcite @xcite , features 16 languages, and learns a shared cross-lingual semantic space via translationbridging tasks @xcite . labse. language-agnostic bert sentence embedding (labse) @xcite adapts pretrained multilingual bert (mbert) @xcite using a dual-encoder framework @xcite with larger embedding capacity (i.e., it provides a shared multilingual vocabulary of 500k subwords). 5 labse is the current state-of-the-art mul5 in addition to the multi-task training objective of muse, tilingual encoder, and supports 109 languages. we keep pretrained sentence encoders fixed during mlp-based id training. formally, we pass a user utterance, that is, a sequence of input tokens @xmath0, x 1 , ..., x t ) through an encoder model θ enc , producing the sequence encoding s @xmath1, x 1 , ..., x t )). ## experimental setup speech transcription. for all language variants, we run the respective google asr model 6 to obtain n-best written transcriptions (i.e., asr hypotheses). unless noted otherwise, we work with the top (i.e., 1-best) transcription. ## results and discussion the main results are summarised in ## conclusion and future work we have presented a first study focused on multilingual and cross-lingual intent detection (id) from spoken data. to this end, we have presented minds-14, a first training and evaluation resource for the task with spoken data, covering 14 intents extracted from a commercial system in the e-banking domain, with spoken examples available in 14 language varieties. our key results have revealed that it is possible to build accurate id models in all target languages relying on a simple yet efficient paradigm based on current state-of-theart multilingual sentence encoders such as labse and machine translation. in future work we plan to expand the minds-14 dataset and put more focus on similar evaluations for truly low-resource languages, where reliable asr, mt, and even sentence encoders cannot be guaranteed. in the long run, we hope that our initiative will foster future developments and evaluation of multilingual id from spoken data, as one of the first steps towards truly multilingual voice-based dialogue systems.