MASSIVE : A 1 M -Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages (2023)

Abstract: We present the MASSIVE dataset–Multilingual Amazon Slu resource package (SLURP) for Slot-filling, Intent classification, and Virtual assistant Evaluation. MASSIVE contains 1M realistic, parallel, labeled virtual assistant utterances spanning 51 languages, 18 domains, 60 intents, and 55 slots. MASSIVE was created by tasking professional translators to localize the English-only SLURP dataset into 50 typologically diverse languages from 29 genera. We also present modeling results on XLM-R and mT5, including exact match accuracy, intent classification accuracy, and slot-filling F1 score. We have released our dataset, modeling code, and models publicly.

## introduction and description natural language understanding (nlu) is a machine's ability to understand the meaning and relevant entities from text. for instance, given the utterance what is the temperature in new york, an nlu model might classify the intent as weather_query and fill the slots as weather_descriptor: temperature and place_name: new york. our particular focus of nlu is one component of spoken language understanding (slu), in which raw audio is first converted to text before nlu is performed @xcite @xcite . slu is the foundation of voice-based virtual assistants like alexa, siri, and google assistant. though virtual assistants have advanced incredibly in the past decade, they still only support a small fraction of the world's 7,000+ languages @xcite . challenges *corresponding author, jgmf@amazon.com. all authors were associated with amazon at the time of publication. for multilingualism span the software stack and a variety of operational considerations, but one difficulty in creating massively multilingual nlu models is the lack of labeled data for training and evaluation, particularly data that is realistic for the task and that is natural for each given language. high naturalness typically requires human-based vetting, which is often costly. we present massive (multilingual amazon slu resource package (slurp) for slot filling, intent classification, and virtual assistant evaluation), a new 1m-example dataset composed of realistic, human-created virtual assistant utterance text spanning 51 languages, 60 intents, 55 slot types, and 18 domains. with the english seed data included, there are 587k train utterances, 104k dev utterances, 152k test utterances, and 153k utterances currently held out for the mmnlu-22 competition, which will be released after the competition. we have released our data, code, and models foot_0 . massive was created by localizing the slurp nlu dataset (created only in english) in a parallel manner. slurp is described further in section 2, linguistic analyses of the dataset in section 3, and the localization process in section 4.3. results for massively multilingual nlu (mmnlu) modeling, in which a single model can perform nlu on any of the incoming languages, are given in section 5. ## related work prior researchers have emphasized the need to explore the unique challenges of low-resource languages @xcite @xcite @xcite @xcite , while the growing number and size of language models (mbert @xcite , roberta @xcite , xlm @xcite , xlm-r @xcite , mbart @xcite , marge @xcite , and mt5 @xcite pre-trained on massively multilingual corpora have allowed for significant improvements in supporting them. however, the creation of evaluation datasets for specific tasks has not kept pace. some tasks, such as named entity recognition (ner) or translation, lend themselves to mining existing corpora @xcite @xcite , while others such as nlu, the focus here, require the creation of new data and schema-specific annotations. beyond the cost, even identifying a sufficient number of speakers for data generation and quality control can be difficult. most studies have thus focused on collecting data for one such low-resource language and determining the utility of multilingual models or cross-lingual learning from more readily available languages. moreover, such datasets are often isolated collections, creating an environment of multiple datasets not easily comparable across the different languages or tasks. there have been exceptions, such as squad @xcite and xquad @xcite , atis @xcite , its hindi and turkish extension @xcite , and multiatis++ @xcite , and snips @xcite with its addition of french @xcite , where researchers have extended popular english benchmark datasets to new languages. this work focuses on the general multi-domain nlu task and builds off the slurp @xcite ) benchmark dataset to extend to an unprecedented 50 new languages. for the task of nlu, the atis dataset has been popular in the nlp community since its first release. multiatis++ was one of the first efforts to extend an nlu dataset across a significant number of languages (nine), yet remained in the limited domain of airline bookings. while proving an asset, it has been questioned what is left to learn from such a dataset @xcite . facebook released a general intelligent virtual assistant (iva) dataset across the domains of alarm, reminder, and weather @xcite created for the purpose of demonstrating cross-lingual transfer learning; and so did not need to be parallel or have an equal number of datapoints, resulting in far fewer examples in thai (5k) compared to spanish (7.6k) and english (43k). the snips datasets (both the original english only and the english and french releases) are most similar to the nlu contained in the massive dataset, spanning smart home and music domains for a generic voice-based virtual assistant. the first iteration for the foundation of the mas-sive dataset was the nlu evaluation benchmarking dataset, with 25k utterances across 18 domains @xcite . the authors updated the dataset and added audio and asr transcriptions in the release of the spoken language understanding resource package (slurp) @xcite , allowing for full end-to-end spoken language understanding (slu) evaluation similar to the fluent speech commands dataset @xcite and chinese audio-textual spoken language understanding (catslu) @xcite . an overview of selected existing nlu datasets can be seen in we release the massive dataset along with baselines from large pre-trained models fine-tuned on the nlu slot and intent prediction tasks. early cross-lingual and multilingual nlu modeling approaches used projection or alignment methods @xcite , focusing on string matching, edit distance, or consonant signatures @xcite , lookup lexicons for lowresource languages @xcite , and aligning @xcite or jointly training word embeddings @xcite . more recently, researchers have borrowed encoders from pre-trained neural translation models before building subsequent classifiers and ner models @xcite , also focusing on language-agnostic and language specific features to learn what information to share between languages @xcite . generative parsing has been demonstrated using sequence-to-sequence models and pointer networks @xcite . with the rise of bert and large pre-trained language models, we have also seen impressive demonstrations of zero-shot performance, where subword tokenization wordpiece overlap helps but is not even necessary to realize improvements @xcite , as well as production multilingual nlu improvements with distillation and full fine-tuning @xcite . the translation task has then been incoporated in the pretraining @xcite of these models or even as part of the final nlu hypothesis for streamlined multilingual production systems (fitzgerald, name # lang utt per lang domains intents slots massive 51 19,521 18 60 55 slurp @xcite 1 16,521 18 60 55 nlu evaluation data @xcite 1 25,716 18 54 56 airline travel information system (atis) @xcite 1 5,871 1 26 129 atis with hindi and turkish @xcite 3 1,315-5,871 1 26 129 multiatis++ @xcite 9 1,422-5,897 1 21-26 99-140 snips @xcite 1 14,484 -7 53 snips with french @xcite 2 4,818 2 14-15 11-12 task oriented parsing (top) @xcite 1 44,873 2 25 36 multilingual task-oriented semantic parsing (mtop) @xcite 6 @xcite 3 5,083-43,323 3 12 11 microsoft dialog challenge @xcite 1 38,276 3 11 29 fluent speech commands (fsc) @xcite 1 30,043 -31 -chinese audio-textual spoken language understanding (catslu) @xcite 1 16,258 4 - 94 ## conclusion we have released a truly massive multilingual dataset for nlu spanning 51 typologically diverse languages. our hope is that massive will encourage many new innovations in massively multilingual nlu, other nlp tasks such as machine translation, and new linguistic analyses, such as with imperative morphologies.