Controllable Semantic Parsing via Retrieval Augmentation (2021)

Abstract: In practical applications of semantic parsing, we often want to rapidly change the behavior of the parser, such as enabling it to handle queries in a new domain, or changing its predictions on certain targeted queries. While we can introduce new training examples exhibiting the target behavior, a mechanism for enacting such behavior changes without expensive model re-training would be preferable. To this end, we propose ControllAble Semantic Parser via Exemplar Retrieval (CASPER). Given an input query, the parser retrieves related exemplars from a retrieval index, augments them to the query, and then applies a generative seq2seq model to produce an output parse. The exemplars act as a control mechanism over the generic generative model: by manipulating the retrieval index or how the augmented query is constructed, we can manipulate the behavior of the parser. On the MTOP dataset, in addition to achieving state-of-the-art on the standard setup, we show that CASPER can parse queries in a new domain, adapt the prediction toward the specified patterns, or adapt to new semantic schemas without having to further re-train the model.

## introduction semantic parsing is the task of mapping input queries to their meaning representations. in practical applications of semantic parsing, such as conversational agents, we often want to rapidly control the behavior of the parser. we particularly focus on three scenarios: (1) domain bootstrapping: making the parser handle queries in a new domain @xcite @xcite . this requires predicting new semantic labels (e.g., intents and slots) unseen during training, and assigning correct values to such labels. (2) parse guiding: influencing the prediction toward a specific parse pattern. this can be used as an override for sensitive queries or queries that the model struggles on. (3) schema refactoring: adapting the a common way to control the parser's behavior is to construct training examples exhibiting the new behavior (e.g., examples from the new domain) and tune the model on them. however, model training requires computational resources, which can become unwieldy if we need to make multiple rapid changes. ideally, we want to control the behavior of the semantic parser without additional training. such an ability would enable many novel use cases. for example, developers could update the semantic parser's behavior and observe the results immediately, thus speeding up the development cycle. this can be used to quickly update the parser in time-critical scenarios while waiting for a fully retrained model. another use case is deploying a single model to service multiple clients. each client can manipulate the parser to fit its use without interfering with the central model or other clients, thus saving resources and preserving privacy. to this end, we propose controllable semantic parser via exemplar retrieval (casper). as illustrated in figure 1 , the parser first retrieves exemplars relevant to the input query (e.g., training examples resembling the input query) from a modifiable retrieval index. these retrieved exemplars are then augmented to the query. finally, a seq2seq generator model takes the augmented query as input and generates a meaning representation. the model takes inspiration from recent works that use modifiable retrieval indices @xcite and exemplar-augmented inputs @xcite . the retrieval and augmentation processes grant us several ways to control the behavior of the parser. for instance, in domain bootstrapping, we can add examples from the new domain to the retrieval index. when these added examples are retrieved, the generator can condition on them while generating the output. this allows the generator to, for instance, follow the semantic template of the exemplars and produce new semantic labels unseen during training. we evaluate our approach on the english portion of the mtop dataset @xcite . in our experiments, we show that casper preserves the generality and increases the performance of a seq2seq-based semantic parser, while also enabling new capabilities that are simply not possible with standard seq2seq parsers. our main results are: foot_0 we will elaborate on each step in the following subsections. we can view exemplar augmentation as a way to provide extra information to any seq2seq-based semantic parser, while still preserving its ability to generate complex outputs. the generator can learn to consider or ignore the provided exemplars, so casper can perform at least as well as the underlying generator in the standard setup (section 3). additionally, we will later show that by manipulating the retrieval index and how the augmented queries are constructed, we can control the behavior of casper for domain bootstrapping (sections 4), parse guiding (section 5), and schema refactoring (section 6). ## standard setup experiments we start with evaluating casper on the standard train-test setup of the english portion of the mtop dataset @xcite . 3 we show that the retrieved exemplars can aid the seq2seq generator even on in-distribution queries. ## domain bootstrapping experiments in addition to improving the parser on the standard setup, we will show that by manipulating the retrieval index, we can influence the parser's behavior. we first consider domain bootstrapping, where a new domain is being added to a previously trained parser, and we want to quickly update the model using a handful of examples in the new domain. ## parse guiding experiments in this section, we demonstrate casper's ability to guide the prediction toward the patterns specified in the exemplars. this parse guiding ability can be useful for correcting the parser's output on a set of problematic queries (e.g., sensitive queries, or queries that the model struggles on). in industrial semantic parsers, one common way to handle problematic queries is to add explicit "hotfix" filters and treat such queries as special cases. parse guiding enables us to also handle queries that are sufficiently similar to known problematic queries. concretely, we can use the similarity score from casper's retriever to identify whether the input is similar to any problematic examples, and apply parse guiding toward them if it is. ## schema refactoring experiments in this section, we show how casper can adapt to changes in the semantic schema. although the solution involves modifying the retrieval index like domain bootstrapping (section 4), schema refactoring presents a new challenge: the parser now needs to produce a different output for in-domain queries, and must resist the urge to produce semantic labels it has learned during training. ## conclusion we have presented casper, a retrieval-augmented semantic parser that uses the retrieved exemplars to influence the predictions. by manipulating the retrieval index and how the exemplars are augmented, we can control the parser's behavior, which is helpful for domain bootstrapping, parse guiding, and schema refactoring. future works include fine-tuning the retriever, possibly jointly with the generator, which has potential to improve the model (see section 3.3); introducing more fine-grained control on the faithfulness toward exemplars than the presence/absence of guiding tag; and pre-training the model on external resources to increase generalization.