MTOP : A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark (2021)

Abstract: Scaling semantic parsing models for task-oriented dialog systems to new languages is often expensive and time-consuming due to the lack of available datasets. Available datasets suffer from several shortcomings: a) they contain few languages b) they contain small amounts of labeled examples per language c) they are based on the simple intent and slot detection paradigm for non-compositional queries. In this paper, we present a new multilingual dataset, called MTOP, comprising of 100k annotated utterances in 6 languages across 11 domains. We use this dataset and other publicly available datasets to conduct a comprehensive benchmarking study on using various state-of-the-art multilingual pre-trained models for task-oriented semantic parsing. We achieve an average improvement of +6.3 points on Slot F1 for the two existing multilingual datasets, over best results reported in their experiments. Furthermore, we demonstrate strong zero-shot performance using pre-trained models combined with automatic translation and alignment, and a proposed distant supervision method to reduce the noise in slot label projection.

## introduction with the rising adoption of virtual assistant products, task-oriented dialog systems have been attracting more attention in both academic and industrial communities. one of the first steps in these systems is to extract meaning from the natural language used in conversation to build a semantic representation of the user utterance. typical systems achieve this by classifying the intent of the utterance and tagging the corresponding slots. with the goal of handling more complex queries, recent approaches propose hierarchical representations @xcite that are expressive enough to capture the task-specific semantics of complex nested queries. although, there have been sizable efforts around developing successful semantic parsing models for task-oriented dialog systems in english @xcite @xcite , we have only seen limited works for other languages. this is mainly due to the painstaking process of manually annotating and creating large datasets for this task in new languages. in addition to the shortage of such datasets, existing datasets @xcite are not sufficiently diversified in terms of languages and domains, and do not capture complex nested queries. this makes it difficult to perform more systematic and rigorous experimentation and evaluation for this task across multiple languages. building on these considerations and recent advancements on cross-lingual pre-trained models @xcite @xcite , this paper is making an effort to bridge the above mentioned gaps. the main contributions of this paper can be summarized as follows: task-oriented semantic parsing the majority of the work on task-oriented dialog systems has been centered around intent detection and slot filling -for example, the representations used on the atis dataset @xcite @xcite and in the dialog state tracking challenge @xcite . this essentially boils down to a text classification and a sequence labeling task, which works great for simple non-compositional queries. for more complex queries with recursive slots, state of the art systems use hierarchical representations, such as the top representation @xcite , that is modeled using recurrent neural network grammars @xcite or as a sequence to sequence task @xcite . ## data existing multilingual task-oriented dialog datasets, such as @xcite ; schuster et al. table. 1 shows a summary statistics of our mtop dataset. we release the dataset at https://fb.me/mtop_ dataset . ## experiments we conduct thorough experiments on the new dataset we describe in in §3. to further demonstrate the effectiveness of our proposed approaches, we provide reproducibility details and all hyperparameters in appendix a we also run additional experiments on the existing multilingual task-oriented semantic parsing datasets including multilingual atis @xcite and multilingual top @xcite . note that both these data sets only include flat representation, while our data set contains hierarchical representations. ## conclusion in this paper, we release a new multilingual taskoriented semantic parsing dataset called mtop that covers 6 languages, including both flat and compositional representations. we develop strong and comprehensive benchmarks for both representations using state-of-the-art multilingual pretrained models in both zero-shot and with target language settings. we hope this dataset along with proposed methods benefit the research community in scaling task-oriented dialog systems to more languages effectively and efficiently.