Don’t Parse, Insert: Multilingual Semantic Parsing with Insertion Based Decoding (2020)

Abstract: Semantic parsing is one of the key components of natural language understanding systems. A successful parse transforms an input utterance to an action that is easily understood by the system. Many algorithms have been proposed to solve this problem, from conventional rule-based or statistical slot-filling systems to shift-reduce based neural parsers. For complex parsing tasks, the state-of-the-art method is based on an autoregressive sequence to sequence model that generates the parse directly. This model is slow at inference time, generating parses in O(n) decoding steps (n is the length of the target sequence). In addition, we demonstrate that this method performs poorly in zero-shot cross-lingual transfer learning settings. In this paper, we propose a non-autoregressive parser which is based on the insertion transformer to overcome these two issues. Our approach 1) speeds up decoding by 3x while outperforming the autoregressive model and 2) significantly improves cross-lingual transfer in the low-resource setting by 37% compared to autoregressive baseline. We test our approach on three wellknown monolingual datasets: ATIS, SNIPS and TOP. For cross-lingual semantic parsing, we use the MultiATIS++ and the multilingual TOP datasets.

## introduction given a query, a semantic parsing module identifies not only the intent (play music, book a flight) of the query but also extracts necessary slots (entities) that further refines the action to perform (which song to play? where or when to go?). a traditional rulebased or slot-filling system classifies a query with one intent and tags each input token @xcite . however, supporting more complex queries that are composed of multiple intents and nested slots is a challenging problem @xcite . a relatively unexplored direction is the crosslingual transfer problem @xcite , where the parsing system is trained in a high-resource language and transfered directly to a low-resource language (zero-shot). the state-of-the-art model leverages the autoregressive decoder such as transformer @xcite and long-short-term memory (lstm) @xcite to generate the target sequence (representing the parse) from left to right. the left to right autoregressive generation constraint has two drawbacks: 1) generating a parse takes o(n) decoding time, where n is the length of the target sequence. this is further exacerbated when paired with standard search algorithms such as beam search. 2) in the cross-lingual setting, autoregressive parsers have difficulty transferring between languages. a recent direction in machine translation and natural language generation to speed up sequence to sequence models is non-autoregressive decoding @xcite @xcite . since the parsing task in the sequence to sequence framework only requires inserting tags rather than generating the whole sequence, an insertion based parser is both faster and more natural for language transfer than an autoregressive parser. in this paper, we leverage insertion based se-quence to sequence models for the semantic parsing problem that require only o(log(n)) decoding time to generate a parse. we enhance the insertion transformer @xcite with the pointer mechanism, since the entities in the source sequence are ensured to appear in the target sequence. our non-autoregressive based model can also boost the performance on the zero-shot and few-shot crosslingual setting, in which the model is trained on a high-resource language and tested on low-resource languages. we also introduce a copy source mechanism for the decoder to further improve the cross lingual transfer performance. in this way, the pointer embedding will be replaced by the corresponding outputs from the encoder. we test our proposed model on several well known datasets, top @xcite , atis @xcite , snips @xcite , multiatis++ @xcite and multilingual top @xcite . in summary, the main contributions of our work include: ## background in this section, we introduce the sequence generation via insertion operations and the pretrained models we leverage in our work. ## methodology in this section, we introduce our non-autoregressive sequence to sequence model for the semantic parsing problem. ## experiments in this section, we introduce the datasets and baseline models we experiment with. then we report the results of monolingual experiments and cross lingual transfer learning experiments. ## conclusion in this paper, we tackle two shortcomings of the autoregressive sequence to sequence semantic parsing models: 1) expensive decoding and 2) poor cross-lingual performance. we propose 1) insertion transformer with pointers and 2) a copy mechanism which replaces the pointer embeding with corresponding encoder out-puts, to mitigate these two problems. our model can achieve o(log(n)) decoding time with parallel decoding. for the specific task of semantic parsing, we can further reduce the decoding steps by initializing the decoder sequence with the whole source sequence. our model achieves new state-ofthe-art performance on both simple queries (atis and snips) and complex queries (top). in crosslingual transfer, our approach surpasses the baselines in the zero-shot setting by 9 em points on average across 9 languages.