Retrieve-and-Fill for Scenario-based Task-Oriented Semantic Parsing (2023)

Abstract: Task-oriented semantic parsing models have achieved strong results in recent years, but unfortunately do not strike an appealing balance between model size, runtime latency, and cross-domain generalizability. We tackle this problem by introducing scenario-based semantic parsing: a variant of the original task which first requires disambiguating an utterance’s “scenario” (an intent-slot template with variable leaf spans) before generating its frame, complete with ontology and utterance tokens. This formulation enables us to isolate coarse-grained and fine-grained aspects of the task, each of which we solve with off-the-shelf neural modules, also optimizing for the axes outlined above. Concretely, we create a Retrieve-and-Fill (RAF) architecture comprised of (1) a retrieval module which ranks the best scenario given an utterance and (2) a filling module which imputes spans into the scenario to create the frame. Our model is modular, differentiable, interpretable, and allows us to garner extra supervision from scenarios. RAF achieves strong results in high-resource, low-resource, and multilingual settings, outperforming recent approaches by wide margins despite, using base pre-trained encoders, small sequence lengths, and parallel decoding.

## introduction task-oriented conversational assistants typically first use semantic parsers to map textual utterances into structured frames for language understanding @xcite @xcite @xcite . while these parsers achieve strong performance with rich supervision, they often face obstacles adapting to novel settings, especially ones with distinct semantics and scarce data. recent approaches address this by improving parsers' data efficiency, such as by using pretrained representations @xcite , optimizing loss functions @xcite , and supplying natural language prompts @xcite . however, these approaches typically rely on larger models and longer contexts, impeding their applicability in real-world conversational assistants. even though non-autoregressive models can alleviate some concerns @xcite @xcite , to the best of our knowledge, there exists no approach which strikes an appealing balance between model size, runtime latency, and cross-domain generalizability. we begin tackling this problem by introducing scenario-based task-oriented semantic parsing which is more closely tied to how new task domains are developed. a slight variation on original semantic parsing, we are given access to all supported scenarios apriori and have to parse the utterance given this scenario bank. here, a scenario is akin to a incomplete frame; it is, precisely, an intent-slot template with variables as leaf spans (e.g., in:get_weather [sl:location x 1 ] ]), indicating it maps to a family of linguistically similar utterances. as domain development and datacollection usually starts out with designing the set of supported scenarios, our intuition is that by giving the model access to this bank we can train it to more explicitly reason about scenarios and improve performance especially in the low data regime. concretely, we propose raf (retrieve-and-fill), a modular yet differentiable architecture for scenario-based task-oriented semantic parsing. guided by the definition of our task, raf also proceeds in two steps: (1) given an utterance, a retrieval module finds the highest ranking scenario and (2) given the utterance and retrieved scenario, a filling module imputes spans into the scenario, creating the final frame. this approach requires no extra supervision despite performing auxiliary inference: utterances and frames are typically provided, and scenarios are obtained by stripping leaf text in frames. we design raf to capitalize on the advantages of prior work but avoid their disadvantages; using base pre-trained encoders across-the-figure 1 : high-level overview. retrieve-and-fill (raf) consists of 4 steps: (1) encode the utterance via an utterance encoder (e.g., roberta); (2) encode all scenarios @xcite in the scenario bank into a cached index via a scenario encoder (e.g., roberta); (3) compute the dot product similarity amongst the incoming utterance and the index of all scenarios, obtaining the top-n candidates; (4) for each retrieved scenario, leverage the non-autoregressive span pointer decoder @xcite to impute each scenario's spans. we evaluate our approach in high-resource, lowresource, and multilingual settings using standard task-oriented semantic parsing datasets. raf achieves 87.52% em on topv2 @xcite and 86.14% em on top @xcite , outperforming recent autoregressive and non-autoregressive models. raf also excels in weakly supervised settings: on topv2-da, we outperform inventory @xcite on 4 domains (alarm, music, timer, weather) despite @xmath0, on topv2 low resource, we outperform bart @xcite , rine @xcite , and roberta + span pointer @xcite , and on mtop @xcite , we outperform xlm-r + span pointer, achieving 42% em averaged across @xmath1, fr, de, hi, th} transfer tasks. to summarize, our contributions are: (1) introducing scenario-based task-oriented semantic parsing, a novel task which requires disambiguating scenarios during typical @xmath2; (2) creating raf (retrieve-and-fill), a modular yet differentiable architecture composed of a retrieval and filling module for solving scenariobased task-oriented semantic parsing; and (3) achieving strong results in high-resource, low-resource, and multilingual settings, outperforming recent models such as span pointer, inventory, and rine by large margins, while also optimizing for model size, runtime latency, and cross-domain generalizability. ## scenario-based semantic parsing we formally introduce the task of scenario-based semantic parsing. task-oriented conversational assistants support a wide range of domains (e.g., calling, reminders, weather) to maximize coverage over users' needs. over time, in response to requests and feedback, developers often iterate on assistants' skill-sets by adding new domains. though the crowdsourcing process of collecting and annotating samples-utterances and frames-for new domains can be accomplished in many ways, we propose a two-step methodology where developers (a) develop scenarios which roughly describe a family of samples and (b) collect linguisticallyvaried samples consistent with each scenario. we elaborate more on our scenario-based methodology below. ## raf: retrieve and fill we propose a model called raf (retrieve and fill) for scenario-based semantic parsing that naturally decomposes the task into a coarse-to-fine objective where we (a) find the most likely scenario given the utterance and (b) find the most likely frame given the utterance and scenario. more concretely given an utterance u and scenarios s 1 , • • • , s n , as well as a gold scenario s * and gold frame f * , we learn our model as follows: raf is a composable yet differentiable model which implements coarse-to-fine processing: we first develop a coarse-grained sketch of an utterance's frame, then impute fine-grained details to achieve the final frame. in these types of approaches, there often exists a trade-off when creating the intermediate representation; if it is "too coarse", the filling module suffers, but if it is "too fine", the retrieval module suffers. we find scenarios, as defined in §2, offer the most appealing solution, as the retrieval module unearths rough syntactic-semantic structure, while the filling module focuses in on imputing exact leaf spans. in the following sub-sections, we discuss the technical details behind the retrieval and filling modules, as well as describe the training and inference procedures. ## experiments and results we evaluate raf in three settings: a high-resource setting (100,000+ training samples), low-resource setting (1-1,000 training samples), and multilingual setting (0 training samples). hyper parameter details are described in appendix §e. our goal here is to show that our system both achieves competitive performance on established benchmarks and offers substantial benefits in resource-constrained environments where training samples are limited. ## ablations and analysis we perform model ablations on raf, removing core retrieval-and filling-related components we originally introduced in §3 to better understand the design decisions. from tables 6 and negatives are important for accurate scenario retrieval. the metric learning objective for retrieval, as introduced in §3.1.2, precisely delineates between positive and negative samples. our ablations show model-based negatives and identity masking are critical to achieve best performance; when removing model-based negatives, for example, retrieval accuracy drops by 3%+. we also investigate training raf with heuristic-based negatives: a simple algorithm which finds top-k similar scenarios with string-based edit distance (appendix c). however, heuristic-based negatives regress both retrieval-only and end-to-end approach, suggesting model-based negatives are more informative. sharing parameters between retrieval encoders improves quality. our retrieval module has two encoders: an utterance encoder e u and a scenario encoder e s . an important design decision we make is tying both encoders' parameters together with roberta @xcite ; this improves end-to-end performance by roughly +0.6 em. we believe that parameter sharing among retrieval encoders improves generalizability: because there are more vastly more unique utterances than scenarios, the scenario encoder may overfit to a select set of scenarios, so weight tying enables the joint optimization of both encoders. ## related work scaling semantic parsing a critical theme in semantic parsing is reducing data requirements to stand up new domains and scenarios. existing works rely on leveraging large language models such as bart @xcite tions for scaling. in particular @xcite introduce a meta-learning approach to improve domain scaling in the low-resource setting. other works such as @xcite @xcite aim to improve scaling through new decoding formulations. @xcite introduce the concept of intrinsic modeling where we provide a human-readable version of the semantic parsing ontology as context to encoding to improve few-shot generalization. our work leverages the intrinsic modeling paradigm by building a function r to convert each intent-slot scenario into a readable representation via intent slot descriptions and example priming. furthermore, our bi-encoder based retrieval setup allows us to inject additional context into each scenario and cache it to an index in order to retain inference efficiency. retrieval based semantic parsing finally, there has been a recent trend towards dense retrieval in various nlp domains such as machine translation @xcite , question answering @xcite ), text generation @xcite and language modeling @xcite . recent works also introduce retrieval-based semantic parsing: retronlu @xcite and casper @xcite both leverage a retrieval step to provide examples as context to seq2seq models. our approach differs in two ways: (1) we phrase our problem as utterance-to-scenario retrieval rather than utterance-to-utterance retrieval. this allows us to look into supporting new scenarios with minimal-to-no-data required for retrieval. ## conclusion in this paper, we tackle scenario-based semantic parsing with retrieve-and-fill (raf), a coarse-tofine model which (a) retrieves a scenario with the best alignment to an utterance and (b) fills the scenario with utterance spans in leaf positions. experiments show our model achieves strong results in high-resource, low-resource, and multilingual settings. the modular nature of our architecture also lends itself well to interpretability and debuggability; we perform several case studies uncovering the inner-workings of our approach.