Span Pointer Networks for Non-Autoregressive Task-Oriented Semantic Parsing (2021)

Abstract: An effective recipe for building seq2seq, non-autoregressive, task-oriented parsers to map utterances to semantic frames proceeds in three steps: encoding an utterance x, predicting a frame’s length |y|, and decoding a |y|-sized frame with utterance and ontology tokens. Though empirically strong, these models are typically bottlenecked by length prediction, as even small inaccuracies change the syntactic and semantic characteristics of resulting frames. In our work, we propose span pointer networks, non-autoregressive parsers which shift the decoding task from text generation to span prediction; that is, when imputing utterance spans into frame slots, our model produces endpoints (e.g., [i, j]) as opposed to text (e.g., “6pm”). This natural quantization of the output space reduces the variability of gold frames, therefore improving length prediction and, ultimately, exact match. Furthermore, length prediction is now responsible for frame syntax and the decoder is responsible for frame semantics, resulting in a coarse-to-fine model. We evaluate our approach on several task-oriented semantic parsing datasets. Notably, we bridge the quality gap between non-autogressive and autoregressive parsers, achieving 87 EM on TOPv2 (Chen et al. 2020). Furthermore, due to our more consistent gold frames, we show strong improvements in model generalization in both cross-domain and cross-lingual transfer in low-resource settings. Finally, due to our diminished output vocabulary, we observe 70% reduction in latency and 83% reduction in memory at beam size 5 compared to prior non-autoregressive parsers.

## introduction task-oriented conversational assistants typically first employ semantic parsers to map utterances to frames @xcite @xcite @xcite . due to performance constraints in real-world deployments, recent work in figure 1 : illustration of a span pointer network for task-oriented semantic parsing based on seq2seq, nonautoregressive, mask-predict models @xcite . in this example, we map the utterance x "message i'll be there at 6pm" to the frame y [in:send_message [sl:content_exact 1 5 ] ] where @xcite corresponds to the span "i'll be there at 6pm". our model operates in three stages: (1) an encoder consumes the utterance; (2) a length module creates |y| × [mask] tokens corresponding to the frame's length; and (3) a decoder swaps each @xcite token with an utterance or ontology (e.g., intent or slot) token. unlike typical methodology, we model span endpoints, which empirically simplifies the parsing task. one popular family of seq2seq, nonautoregressive, task-oriented parsers is based on the mask-predict algorithm @xcite , which operates in three steps: encoding, length prediction, and decoding. unlike machine translation, task-oriented semantic parsing does not benefit much from iterative refinement; therefore, frames are typically generated in one step. however, this increases the burden of upstream components, namely placing a major bottleneck on the length prediction module. therefore, even off-by-one length errors can change the syntactic and semantic characteristics of resulting frames. in this work, we create span pointer networks which model a seq2seq, non-autoregressive parsing task centered around span prediction as opposed to text generation. figure 1 illustrates an application of our model; when parsing the utterance "message i'll be there at 6pm", our model produces span endpoints @xcite as opposed to exact text "i'll be there at 6pm" in the appropriate leaf slot. as a result, our length module implicitly predicts the syntax of a frame (i.e., how many intents and slots does the utterance have?) and the decoder resolves leaf arguments with span endpoints (i.e., what utterance spans should be included in the frame?). this additionally creates a parallel among our approach and coarse-to-fine @xcite modeling as our length predictor predicts a general structure that our decoder further refines. despite this shift in behavior, our approach is largely compatible with the typical mask-predict methodology, as we largely modify the frame representation and model architecture to be span-based. we evaluate span pointer networks on three axes: quality, generalizability, and resources. first, we benchmark exact match (em) on multiple task-oriented semantic parsing datasets, including topv2 @xcite and top @xcite . our non-autoregressive parser is competitive, achieving 87 em on topv2, and matches or exceeds autoregressive parsers in many cases. second, we evaluate generalizability by setting up cross-domain and cross-lingual transfer experiments. our non-autoregressive outperforms strong baselines, notably outperforming autoregressive parsers by +15 em when averaged across 5 zeroshot multilingual settings. third, due to the restricted decoder vocabulary, we measure resource savings, in particular, latency and memory usage. compared to non-autoregressive models our parser achieves 70% reduction in latency and 83% reduction in memory with a beam size of 5. to summarize, our contributions are: (1) we propose span pointer networks, non-autoregressive parsers which use span prediction as opposed to text generation; (2) we evaluate several aspects of span pointer networks, conducting a series of experiments around quality, generalizability, and resources; (3) we empirically show improvements on all three axes, notably outperforming both nonautoregressive and autoregressive parsers. ## background and related work task-oriented semantic parsing broadly consists of mapping textual utterances to structured frames @xcite @xcite @xcite . frames are structured semantic representations of utterances, and are comprised of both ontology tokens (e.g., intents and slots) and utterance tokens. for example, we can map the utterance "message i'll be there at 6pm" into a frame [in:send_message [sl:content_exact i'll be there at 6pm ] ], which has several components: (1) a "send message" intent; (2) a "content exact" slot; and (3) a span "i'll be there at 6pm", which is an argument of the "content exact" slot. here, intents and slots are akin to functions and arguments, respectively, in an api call. modern parsers are typically based on seq2seq transformers which encode utterances and decode frames. these parsers, by virtue of being seq2seq models, can be learned either in an autoregressive or non-autoregressive fashion. autoregressive parsers generate frame components in sequence, while non-autoregressive parsers generate frame components in parallel. both paradigms have their advantages and disadvantages, primarily tradingoff latency and quality: while non-autoregressive models are optimized for inference and suitable in production settings, they are often qualitatively worse than autoregressive models due to strong independence assumptions during generation @xcite @xcite @xcite . our work focuses on improving the performance of one family of non-autoregressive parsers, namely those based on the mask-predict algorithm @xcite . these parsers operate in three stages: (1) on the source-side, an encoder consumes an utterance x; (2) using its representations, a length module predicts a frame's length |y| and creates |y| × [mask] tokens; and (3) a length-conditioned decoder produces a frame y with ontology and utterance tokens. frames are typically generated in one step as, unlike machine translation, task-oriented semantic parsing does not benefit much from iterative refinement @xcite . consequently, this increases the burden of accuracy on upstream components, especially placing a major bottleneck on the length prediction module. even off-by-one length errors can dramatically change resulting frames, which is one of the drivers behind the quality gap between nonautoregressive and autoregressive models. ## span pointer networks in this section, we introduce span pointer networks: seq2seq, non-autoregressive parsers which model a span-based, task-oriented semantic parsing task. our approach is based on the mask-predict algorithm, which requires an encoder, length module, and decoder to map utterances to frames. however, our core idea is shifting from text generation to span prediction when creating leaf arguments in frames; that is, replacing text spans (e.g., "i'll be there at 6pm") with index spans (e.g., @xcite ). text generation and span prediction are functionally identical; though span prediction requires converting index spans to text spans post-hoc, both resolve to the same frame. however, text generation and span prediction are technically different, as they present distinct targets for non-autoregressive parsing. note that our approach is different from @xcite , who develop a cky-based parser which maps spans to labels; in contrast, we explore span prediction with seq2seq modeling. to build intuition for how these paradigms affect mask-predict parsers, in particular, consider the utterance "message i'll be there at 6pm" and frames [in:send_message [sl:content_exact i'll be there at 6pm ] ] and [in:send_message [sl:content_exact 1 5 ] ], one with text spans (requiring text generation) and one with index spans (requiring span prediction). for accurate parsing, a length module needs to understand both frame syntax and semantics with text generation, but only frame syntax with span prediction. specifically, with text generation, the length module outputs 9 [mask] tokens, which requires it to implicitly guess the entire frame-both its syntactic structure and semantic arguments-before decoding. in contrast, with span prediction, the length module outputs 6 [mask] tokens, which requires it to implicitly guess syntactic structure but leave semantic arguments to the decoder, which will subsequently resolve the exact span indices. to make this argument more clear, consider the frames [sl:artist beyonce ] and sl:location seattle, washington ]: these frames both have the same length despite being semantically distinct, so during fine-tuning, the length module learns syntax and the decoder learns semantics. we view this type of span prediction as analogous to coarse-tofine modeling @xcite ; the length module predicts a coarse-grained frame structure and the decoder infills incomplete parts (e.g., leaf arguments) with fine-grained details. below, we elaborate on the core components required to implement span pointer networks: a span-based frame representation and the model architecture. ## experimental setup our goal is to benchmark span pointer networks on three axes: quality, generalizability, and resources. we benchmark quality by inspecting exact match (em) on semantic parsing datasets, generalizability by probing cross-domain and crosslingual performance, and resources by measuring latency and memory usage during inference. ## analysis having evaluated span pointer networks on several axes, we now turn towards understanding the driving factors behind its performance.