R etro NLU : Retrieval Augmented Task-Oriented Semantic Parsing (2022)

Abstract: While large pre-trained language models accumulate a lot of knowledge in their parameters, it has been demonstrated that augmenting it with non-parametric retrieval-based memory has a number of benefits ranging from improved accuracy to data efficiency for knowledge-focused tasks such as question answering. In this work, we apply retrieval-based modeling ideas to the challenging complex task of multi-domain task-oriented semantic parsing for conversational assistants. Our technique, RetroNLU, extends a sequence-to-sequence model architecture with a retrieval component, which is used to retrieve existing similar samples and present them as an additional context to the model. In particular, we analyze two settings, where we augment an input with (a) retrieved nearest neighbor utterances (utterance-nn), and (b) ground-truth semantic parses of nearest neighbor utterances (semparse-nn). Our technique outperforms the baseline method by 1.5% absolute macro-F1, especially at the low resource setting, matching the baseline model accuracy with only 40% of the complete data. Furthermore, we analyse the quality, model sensitivity, and performance of the nearest neighbor retrieval component’s for semantic parses of varied utterance complexity.

## proposed approach our proposed approach consists of four main steps: (a) index construction by embedding training examples and computing cosine similarity; (b) retrieval, where we extract the nearest neighbor utterances from the index given an example utterance; (c) augmentation, in which we append the nearestneighbor utterance ground truth semantic parse (semparse-nn) or the utterance itself (utterancenn) to the original input via a special separator token (such as '|'); and (d) semantic parsing, in which we train the parsing model using the retrieval-augmented input with output ground truth. figure 2 illustrates the retrieval augmented semantic parsing (retronlu) approach. indexing: to build an index we use a pre-trained bart model to get training utterance embeddings. more specifically, we get sentence embedding for all the training utterances. these sentence embeddings are obtained as average of token embeddings from last model layers of the bart models. 2 we then used the cosine similarity between embeddings to build a fast and efficient retrieval index with efficient faiss library @xcite . retrieval: next, given a new input (training or test row), we obtain embeddings by running it through same pre-trained bart, and then query the index with it to retrieve nearest neighbors text and their ground truth semantic parses based on cosine similarity. for training data, we exclude an example itself from the retrieved list. for example, for input utterance "please add 20 minutes on the lasagna timer", we retrieve the nearest neighbour "add ten minutes to the oven timer" along with the semantic parse as "[in:add_ time_ timer add [sl:date_ time ten minutes] to the [sl:timer_ name oven] [sl:method_ timer timer]]". augmentation: once we got a list of nearest neighbors, we can append either utterance text or semantic parse to the input, following the left to right order. 3 the closest neighbor appears to the immediate left of the input example utterance. one can also directly append the nearest neighbor utterance rather than the semparse, refer as utterance-nn. for the last example the final input would after augmentation would be "[in:add_ time_ timer add [sl:date_ time ten minutes ] to the [sl:timer_ name oven] [sl:method_ timer timer ]] | please add 20 minutes on the lasagna timer" for semparse-nn, and "add ten minutes to the oven timer | please add 20 minutes on the lasagna timer" for utterance-nn. here, the token '|' act as a separator between the input utterance and the neighbour's. ## experiment and analysis our experiments examines how our knowledge retrieval-based augmentation technique impacts model performance indicators such as accuracy and data efficiency. we study the following questions: rq1. can today's pre-trained models leverage non-parametric information in manner as described in §2 to enhance task-oriented semantic parsing? rq2. if only part of the dataset has semantic parses, i.e. limited supervision setting, can augmentation with unannotated instances (utterance_nn) enhance semantic parsing accuracy? rq3. how efficient is a retrieval-augmented model in terms of data? is it more accurate even with less training data than the baseline seq2seq model? rq4. does non-parametric memory benefit instances equally, e.g., do we notice greater benefits for (a) more complex (i.e. compositional) or (b) less frequent semantic frames (i.e. tail over head)? rq5. (a) does augmentation with more nearest neighbors benefits? (b) how sensitive is the model to retrieval noise? can the model predict right intent/slots for low-quality retrieve instances? our experiments are designed to demonstrate how non-parametric external information can be beneficial to a parametric model and to undertake an in-depth assessment of the impact. foot_3 ## comparison with related work task-oriented semantic parsing. sequence-tosequence (seq2seq) models have recently achieved state of the art results in semantic parsing @xcite , and they also provide a flexible framework for incorporating session-based, complex hierarchical semantic parsing @xcite @xcite and multilingual semantic parsing @xcite . architectures, such as t5 and bart @xcite , with large pre-trained language models pushed the performance even further. such models are quite capable of storing a lot of knowledge in their parameters @xcite , and in this work we explore the benefits of additional non-parametric knowledge in a form of nearest neighbor retrieval for the task of semantic parsing. to improve low resource seq2seq parsers @xcite have proposed looking at meta learning methods such as @xcite have introduced new fine-tuning objectives. our approach is focused on non-architecture changes to augment generation with retrieval and thus can be combined with either of these approaches. ## conclusion and future work we show that task-oriented semantic parsing performance can be enhanced by augmenting neural model-stored parametric information with nonparametric external memory. on the topv2 dataset, we demonstrated that adding instances derived from a nearest neighbor index greatly improves the semantic parsing performance of a bart model with copy mechanism. our retronlu model is able to achieve higher accuracy earlier with less training data (limited supervision setting), which allows maintaining a large index with annotated data, while using only a subset to train a model more efficiently. lastly, we performed an analysis of performance improvements on different slices, and found retronlu to be more effective on rarer complex frames, compared to a traditional seq2seq model. retronlu extensions, we focus on joint training of retrieval and parsing components. having task specific utterances representation can benefit i.e. finding utterances with similar semantic parse. exploring few/zero-shot performance could be interesting direction. having an easily-updateable index enables you to amend annotations, add new ones, or remove existing ones, without affecting the model. it will be useful to study other approaches of sentence embedding, such as @xcite . finally, using cross-lingual representations such as mbart @xcite , could help multilingual semantic parsing.