Learning to Simulate Natural Language Feedback for Interactive Semantic Parsing (2023)

Abstract: Interactive semantic parsing based on natural language (NL) feedback, where users provide feedback to correct the parser mistakes, has emerged as a more practical scenario than the traditional one-shot semantic parsing. However, prior work has heavily relied on human-annotated feedback data to train the interactive semantic parser, which is prohibitively expensive and not scalable. In this work, we propose a new task of simulating NL feedback for interactive semantic parsing. We accompany the task with a novel feedback evaluator. The evaluator is specifically designed to assess the quality of the simulated feedback, based on which we decide the best feedback simulator from our proposed variants. On a text-to-SQL dataset, we show that our feedback simulator can generate high-quality NL feedback to boost the error correction ability of a specific parser. In low-data settings, our feedback simulator can help achieve comparable error correction performance as trained using the costly, full set of human annotations.

## introduction the state of nlp research has long been dominated by training and evaluating single-turn models, which, given a task input, produce the output and terminate the task immediately. however, in the more practical scenario of nlp applications (e.g., smart-home virtual assistance), users often anticipate multi-turn interactions, such as being able to provide feedback to the model output @xcite . in doing this, not only can the model obtain more information and guidance to improve its task performance, but it also provides human users a mechanism to intervene in the model decision-making for safety purposes. figure 1 : illustration of interactive semantic parsing, where the parser solicits nl feedback for error correction (example based on text-to-sql). in this work, we aim to simulate such nl feedback at scale to facilitate the error correction model training. to this end, we proposed a feedback evaluator for promoting this task, and experiment with different feedback simulator variants. however, training a neural model to understand human feedback requires a large number of human annotations, which has hindered the advancement of this line of research. in this paper, we investigate this problem under semantic parsing. semantic parsing is the task of translating nl sentences into their formal meaning representations (i.e., logical forms), which has been adopted for applications such as question answering @xcite @xcite and dialogue systems @xcite @xcite . the pressing need for further improving its application performance has motivated the research of interactive semantic parsing, where a semantic parser presents its parsing results to the user and requests user feedback for error correction @xcite @xcite . in this work, we follow @xcite ; @xcite to consider nl feedback, i.e., a sentence describing which parts of the generated logical form contain errors and how to correct them. we illustrate this paradigm in figure 1 . despite its promise, prior work has heavily relied on human-annotated feedback data to train the error correction model. for example, @xcite deployed the seq2struct parser @xcite and recruited 10 crowd workers to provide feedback annotations, which has been shown to be both costly and time-consuming (6 minutes per annotation as reported). moreover, since this feedback collection procedure is bound to a specific parser, the collected feedback may not generalize well to resolving errors made by different parsers. motivated by the above observations, in this paper, we propose the task of simulating nl feedback for interactive semantic parsing. specifically, given the initial user command, a model-generated incorrect logical form, the ground-truth logical form for the simulation purpose, as well as other contextual information, the goal is to generate an nl feedback sentence encoding the error correction information in a way that is close to the realuser feedback. we assume a small set of humanannotated feedback to bootstrap this task, but aim for an effective feedback simulator that can further simulate feedback for different semantic parsers at scale. while prior work has attempted a similar task @xcite @xcite , none of them carefully defined the task (e.g., how to evaluate simulated feedback) and investigated advanced simulation methods. to facilitate this research, we first propose a feedback evaluator that can be used to assess different simulators. in particular, our feedback evaluator is designed to evaluate whether the simulated feedback is logically consistent with the user error correction intent, a critical attribute that cannot be achieved by existing text evaluation metrics @xcite . instead of comparing the simulated feedback with the human-annotated one, we propose to compare it with the template feedback, which is not only logic-wisely less noisy but also scalable to cases when human annotations are not available. human evaluation shows that our feedback evaluator can more precisely assess the simulated feedback. we also propose a set of feedback simulators based on the pre-trained t5 model @xcite , and decide the best using our evaluator. to demonstrate the advantages of our feedback simulator, we conduct experiments on splash @xcite , a dataset containing humanannotated feedback to mistakes of the seq2struct parser @xcite in text-to-sql semantic parsing @xcite . we first show that our feedback simulator trained on splash can be used to simulate nl feedback for a different parser, using editsql @xcite ) as an example. the resulting simulated feedback, when being used to augment the splash training set, leads to improved error correction performance for both seq2struct and particularly editsql. we further demonstrate that even in the low-data setting (i.e., using a small portion of splash), our feedback simulator can still produce high-quality nl feedback, based on which we can train the error correction model to a comparable performance level as its counterpart trained using the full splash. this implies that our feedback simulator can be very helpful when there are limited annotation budgets. ## related work interactive semantic parsing. motivated by the need to further enhance its performance in practice, interactive semantic parsing emerged as a promising solution @xcite @xcite @xcite @xcite @xcite @xcite . among others, gur et al. (2018) and yao et al. (2019b) explained components in the generated logical form and, if they were wrong, requested users to select the correct ones as feedback. li et al. (2020) identified uncertain tokens in the language command and requested user choices on their paraphrases for clarification. while the multichoice feedback was shown to work well for correcting errors in semantic parsing, it suffers from the obvious drawbacks of being less user-friendly and inefficient, as users can only passively respond to the system-presented choices. labutov et al. (2018) and elgohary et al. (2020) have driven the research a step forward by introducing nl feedback. particularly, elgohary et al. ( 2020 ) annotated the splash feedback dataset and showed that an error correction model can learn to fix parsing mistakes from nl feedback. in @xcite , the authors further investigated a more advanced error correction model, which predicts the edits rather than the corrected logical form based on nl feedback. our work is complementary to the existing effort. instead of improving the error correction model architecture, we focus on simulating nl feedback to reduce the need for human annotations for training the error correction model. when constructing our feedback simulator, we also explore the use of "edits" to improve the model performance. ## conclusions in this work, we propose the task of simulating nl feedback for interactive semantic parsing and present two models for feedback evaluation and simulation, respectively. our experimental results have demonstrated the effectiveness of both models and show the promise of saving human-annotation effort with simulated feedback.