Generate-and-Retrieve: Use Your Predictions to Improve Retrieval for Semantic Parsing (2022)

Abstract: A common recent approach to semantic parsing augments sequence-to-sequence models by retrieving and appending a set of training samples, called exemplars. The effectiveness of this recipe is limited by the ability to retrieve informative exemplars that help produce the correct parse, which is especially challenging in low-resource settings. Existing retrieval is commonly based on similarity of query and exemplar inputs. We propose GandR, a retrieval procedure that retrieves exemplars for which outputs are also similar. GandR first generates a preliminary prediction with input-based retrieval. Then, it retrieves exemplars with outputs similar to the preliminary prediction which are used to generate a final prediction. GandR sets the state of the art on multiple low-resource semantic parsing tasks.

## introduction a common and successful approach to structured prediction problems @xcite is to treat the gold structure as a sequence and fine-tune a sequence-to-sequence model such as t5 @xcite or bart @xcite . however, the performance of fine-tuned models suffers in low resource scenarios where available training data is limited relative to the complexity of the task @xcite . existing work @xcite @xcite has found that retrieving related training samples, denoted exemplars, and appending the retrieved input-output pairs to the sample input before processing the sample can improve performance in low resource settings. in principle, all information from exemplars is available to the model during training and could be stored in model parameters. however, in practice the model may not successfully retain all information, and reminding the model of salient input-output patterns at test time appears to help. that raises the question: what exemplars are most informative for the model? existing work focuses on retrieving exemplars for which the input is partially similar to the test input, effectively answering "what is the output for similar inputs?". in this work we explore whether there is complementary information in exemplars that answer the inverse question, "what is the input for similar outputs?". we propose generate-and-retrieve (gandr), a method to retrieve exemplars with similar output as well as input. as the true output of a sample is in general unknown, gandr proceeds in two steps. first, a preliminary prediction is generated using retrievals with similar input only. then, a new set of exemplars is retrieved based on a relevance measure that balances the similarity of the inputs and the similarity of the preliminary prediction and the exemplar output. figure 1 provides an overview of the method. we evaluate gandr in the setting of taskoriented semantic parsing, a core component of widely used virtual assistants. we show that similarity in output space provides a complementary signal to input similarity, yielding retrievals that prove more informative for the model. moreover, for many structured prediction tasks the output space is more structured than the free-form input text, so that simple, non-learned distance measures work well for outputs even when inputs are lexically dissimilar. the proposed method strongly improves performance in low-resource settings for semantic parsing, achieving state of the art results for lowresource and transfer benchmarks in mtop @xcite and topv2 @xcite . ## method we approach semantic parsing as a conditional language generation task and apply a t5 sequence-tosequence model @xcite to predict a parse y given a query x. for each sample, we retrieve @xmath0. we append the retrieved input-output pairs to the sample input and apply the t5 model to the augmented input to predict a parse output. in particular, let (x 1 , y 1 ), . . . , (x k , y k ) denote the retrieved inputoutput pairs, then the augmented input is our approach closely follows that of @xcite , differing primarily in the choice of relevance function. during evaluation we retrieve the top k most relevant exemplars. during training, we sample retrievals according to a geometric distribution over the relevance score rank. in particular, the probability that we retrieve an exemplar is given by p(1 -p) r where r is the rank of the exemplar according to relevance score and p is a temperature hyperparameter. in @xcite , the relevance score is given by the inner product of universal sentence encoder @xcite encodings of the candidate input and the sample input. we found that a simple tf-idf @xcite similarity baseline achieves comparable or better results. our proposed approach, gandr, builds on the input-similarity baseline by constructing a hybrid similarity measure that takes into account not only the similarity between sample and candidate inputs, but also the similarity between the sequence predicted by the model and the candidate output. see figure 1 for an overview. first, gandr generates a preliminary prediction using an input augmented with exemplars with similar inputs. then, gandr retrieves exemplars based on a hybrid similarity measure over inputs and outputs, and generates a final prediction based on these exemplars. specifically, let ŷi be preliminary prediction, then the proposed output similarity between samples i and j is given by the tf-idf similarity between the predicted structure (in our case, the set of intents and slots) and the structure of the true parse y j . our proposed relevance score is a weighted sum of input and output similarity (see appendix a for detailed description on how output similarity is computed): ## related work sequence-to-sequence models @xcite have achieved state-of-the-art performance on task-oriented semantic parsing @xcite @xcite as well as other structured prediction tasks @xcite . the general approach is to pre-train on language modeling and perform fine-tuning on the specific domain of interest. several works augment the input with retrieved exemplars from the training data, with differing methods for selecting informative examples. pasupat et al. (2021) and @xcite retrieve exemplars with similar input encodings from a pretrained neural encoder, evaluating on semantic parsing. @xcite retrieves examplars for which the input has high bm25 similarity with the sample input, with good performance on language generation. we adopt a similar approach with tf-idf similarity as a baseline for semantic parsing. @xcite and @xcite learn dense retrievers in the spirit of @xcite , providing another path to incorporate label information for retrieval. @xcite proposes other methods to fine-tune a dense retriever. these approaches require training a separate model specifically for retrieval, possibly with additional learning signal. in contrast, we employ a sparse similarity measure over model predictions that are produced incidentally in the course of fine-tuning the main model. selecting relevant training exemplars is also important for in-context prompting @xcite . similar to related fine-tuning literature, work in this direction uses either a pre-trained @xcite or fine-tuned @xcite ## conclusion we propose gandr, a new method for structured prediction that generates a preliminary prediction, retrieves training exemplars with similar outputs (and similar inputs), and augments the input with the retrieved exemplars to generate a final prediction. we demonstrate that using output similarity yields improvements for semantic parsing in lowresource settings, achieving state of the art results on several semantic parsing benchmarks.