NL - EDIT : Correcting Semantic Parse Errors through Natural Language Interaction (2021)

Abstract: We study semantic parsing in an interactive setting in which users correct errors with natural language feedback. We present NL-EDIT, a model for interpreting natural language feedback in the interaction context to generate a sequence of edits that can be applied to the initial parse to correct its errors. We show that NL-EDIT can boost the accuracy of existing text-to-SQL parsers by up to 20% with only one turn of correction. We analyze the limitations of the model and discuss directions for improvement and evaluation. The code and datasets used in this paper are publicly available at http://aka.ms/NLEdit .

## introduction major progress in natural language processing has been made towards fully automating challenging tasks such as question answering, translation, and summarization. on the other hand, several studies have argued that machine learning systems that can explain their own predictions @xcite and learn interactively from their endusers @xcite can result in better user experiences and more effective learning systems. we develop nl-edit-an approach that employs both explanations and interaction in the context of semantic parsing. most existing systems frame semantic parsing as a one-shot translation from a natural language question to the corresponding logical form (e.g., sql query) @xcite @xcite . a growing body of recent work demonstrates that semantic parsing systems can be improved by including users in the parsing loop-giving them the affordance to examine the parses, judge their correctness, and provide feedback accordingly. the feedback often comes in the form of a binary correct/incorrect figure 1 : example human interaction with nl-edit to correct an initial parse through natural language feedback. in the semantic parsing phase (top), an offthe-shelf parser generates an initial sql query and provides an answer paired with an explanation of the generated sql. in the correction phase (bottom), the user reviews the explanation and provides feedback that describes how the explanation should be corrected. the system parses the feedback as a set of edits that are applied to the initial parse to generate a corrected sql. unlike other frameworks for interactive semantic parsing that typically expect users to judge the correctness of the execution result or induced logical form, @xcite introduced a framework for interactive text-to-sql in which induced sql queries are fully explained in natural lan-guage to users, who in turn, can correct such parses through natural language feedback (figure 1 ). they construct the splash dataset and use it to evaluate baselines for the semantic parse correction with natural language feedback task they introduce. we present a detailed analysis of the feedback and the differences between the initial (incorrect) and the correct parse. we argue that a correction model should be able to interpret the feedback in the context of other elements of the interaction (the original question, the schema, and the explanation of the initial parse). we observe from splash that most feedback utterances tend to describe a few edits that the user desires to apply to the initial parse. as such, we pose the correction task as a semantic parsing problem that aims to convert natural language feedback to a sequence of edits that can be deterministically applied to the initial parse to correct it. we use the edit-based modeling framework to show that we can effectively generate synthetic data to pre-train the correction model leading to clear performance gains. we make the following contributions: (1) we present a scheme for representing sql query edits that benefits both the modeling and the analysis of the correction task, (2) we present nl-edit, an edit-based model for interactive text-to-sql with natural language feedback. we show that nl-edit outperforms baselines in @xcite ) by more than 16 points, (3) we demonstrate that we can generate synthetic data through the edit-based framing and that the model can effectively use this data to improve its accuracy and (4) we present a detailed analysis of the model performance including studying the effect of different components, generalization to errors of state-of-the-art parsers, and outline directions for future research. ## background in the task of text-to-sql parsing, the objective is given a database schema (tables, columns, and primary-foreign key relations) and a natural language question, generate a sql query that answers the question when executed against the database. several recent text-to-sql models have been introduced @xcite @xcite as a result of the availability of spider @xcite , a large dataset of schema, questions and gold parses spanning several databases in different domains. the task of sql parse correction with natural language feedback @xcite aims to correct an erroneous parse based on natural language feedback collected from the user. given a question, a database schema, an incorrect initial parse, natural language feedback on the initial parse, the task is to generate a corrected parse. to study this problem, elgohary et al. (2020) introduced the splash dataset. splash was created by showing annotators questions and a natural language explanation of incorrect parses and asking them to provide feedback, in natural language, to correct the parse. the dataset contained 9,314 question-feedback pairs. like the spider dataset, it was split into train-dev-test sets by database to encourage the models to generalize to new unseen databases. they contrast the task with conversational semantic parsing @xcite @xcite and show that the two tasks are distinct and are addressing different aspects of utilizing context. they establish several baseline models and show that the task is challenging for state-of-the-art semantic parsing models. we use these as baselines for this work. ## sql edits we define a scheme for representing the edits required to transform one sql query to another. we use that scheme both in our model and analysis. our goal is to balance the granularity of the editstoo fine-grained edits result in complex structures that are challenging for models to learn, and too coarse-grained edits result in less compact structures that are harder for models to generate. we view a sql query as a set of clauses (e.g, select, from, where), each clause has a sequence of arguments (figure 2 ). we mirror the sql clauses select, from, where, group-by, order-by, having, and limit. for subqueries, we define a clause subs whose arguments are recursively defined as sets of clauses. subqueries can be linked to the main query in two ways: either through an ieu clause (mirrors sql intersect/except/union) whose first argument is one of the keywords intersect, except, union and its second argument is a pointer to a subquery in subs. the second is through nested queries where the arguments of some of the clauses (e.g., where) can point at subqueries in subs (e.g., "id not in subs 1 "). with such view of two queries p source and p target , we define their edit d @xmath0: arg 1 :"id", arg 2 :"max(grade)" from: arg 1 :"assignments" group-by: arg 1 :"id" subs: arg 1 : where: arg 1 :"@xmath1, arg 2 : "id not in subs 1 " select: arg 1 : "id" from: arg 1 : "graduates" select: arg 1 :"id", arg 2 :"avg(grade)" from: arg 1 :"assignments" group-by: arg 1 :"id" order-by: arg 1 :"id" where: arg 1 :"@xmath2: remove: "max(grade)", add: "avg(grade)" where: remove: "id not in subs 1 " order-by: add: "id" source target @xmath3: edit for transforming the source query "select id, max(grade) from assignments where @xmath4, avg(grade) from assignment where @xmath5. the source and target are represented as sets of clauses (left and middle). the set of edits and its linearized form (section 4) are shown on the right. removing the condition "id not in subs 1 " makes the subquery unreferenced, hence pruned from the edit. the set of clause-level edits {d c @xmath6. to compare two clauses of type c, we simply exact-match their arguments: unmatched arguments in the source (e.g., max(grade) in select) are added as toremove arguments to the corresponding edit clause, and unmatched arguments in the target (e.g., "id" in the order-by) are added as to-add arguments. our current implementation follows spider's assumption that the number of subqueries is at most one which implies that computing edits for different clauses can be done independently even for the clauses that reference a subquery (e.g., where in figure 2 ). the edit of the subs clause is recursively computed as the edit between two queries (any of them can be empty); the subquery of source and the subquery of target, i.e., d subs @xmath7:subs 1 →target:subs 1 . we keep track of the edits to the arguments that reference the subquery. after all edit clauses are computed, we prune the edits of the subs clause if the subquery will no longer be referenced (subs 1 in figure 2 ). we follow the spider evaluation and discard the values in where/having clauses. throughout this paper, we refer to the number of add/remove operations in an edit as the edit size, and we denote it as |d @xmath8. for example, the edit in figure 2 is of size four. ## model we follow the task description in section 2: the inputs to the model are the elements of the interaction-question, schema, an initial parse p ˜, the gold parse p ˆis available for training. our model is based on integrating two key ideas in an encoder-decoder architecture. we start with a discussion of the intuitions behind the two ideas followed by the model details. ## synthetic feedback in this section, we describe our process for automatically synthesizing additional examples for training the correction model. recall that each example consists of a question about a given schema paired with a gold parse, an initial erroneous parse, and feedback. starting with a seed of questions and their corresponding gold parses from spider's training set (8,099 pairs) 1 , our synthesis process applies a sequence of sql editing operations to the gold parse to reach an altered parse that we use as the initial parse (algorithm 1). by manually inspecting the edits (section 3) we induce for the initial and gold parses in splash training set, we define 26 sql editors and pair each editor with their most frequent corresponding feedback template(s) (examples in algorithm 1 summarizes the synthesis process. we start by creating n (controls the size of the dataset) clones of each seed example. @xcite 's analysis of splash shows that multiple mistakes might be present in the initial sql, hence we allow our synthesis process to introduce up to four edits (randomly decided in line:4) to each clone p. for each editing step, we sample a feasible edit for the current parse (line:5) with man1 we ensure there is no overlap between examples in the seed and the dev set of splash. we use bert-base-uncased @xcite ) in all our experiments. we set the number of layers in the relational-aware transformer to eight @xcite and the number of decoder layers to two. we train with batches of size 24. we use the adam optimizer (kingma and ba, 2015) for training. we freeze bert parameters during the first 5,000 warm-up steps and update the rest of the parameters with a linearly increasing learning rate from zero to 5 × 10 -4 . then, we linearly decrease the learning rates from 5 × 10 -5 for bert and 5 × 10 -4 for the other parameters to zero. 2 we use beam search with a beam of size 20 and take the top-ranked beam that results in a valid sql after applying the inferred edit. evaluation: we follow @xcite ) and use the correction accuracy as our main evaluation measure: each example in splash test set contains an initial parse p ˜and a gold parse p ˆ. with a predicted (corrected) parse by a correction model p ¯, they compute the correction accuracy using the exact-set-match @xcite between p ¯and p ˆaveraged over all test examples. while useful, correction accuracy also has limitations. it expects models to be able to fully correct an erroneous parse with only one utterance of feedback as such, it is defined in terms of the exact match between the corrected and the gold parse. we find ( to combine edit ↓ and edit ↑ in one measure and account for the relative reduction (increase) in the number of edits, we define progress(s) = 1 |s| ∑︂ p ˜,p ¯,p ˆ∈s |d p ˜→p ˆ| -|d p ¯→p ˆ| |d p ˜→p ˆ| . given a test set s, the progress of a correction model is computed as the average relative edit reduction between the initial parse p ˜and the gold parse p ˆby predicting a correction p ¯of p ˜. a perfect model that can fully correct all errors in the initial parse would achieve a 100% progress. a model can have a negative progress (e.g., rulebased re-ranking in results: we compare ( ## related work and discussion natural language to sql: natural language interfaces to databases have been an active field of study for many years @xcite @xcite . the development of new large scale datasets, such as wikisql @xcite and spider @xcite , has reignited the interest in this area with several new models introduced recently @xcite @xcite . another related line of work has focused on conversation semantic parsing, e.g. sparc @xcite , cosql @xcite , and smcalflow @xcite , where parsers aim at modeling utterance sequentially and in context of previous utterances. interactive semantic parsing: several previous studies have looked at the problem of improving semantic parser with feedback or human interactions @xcite . interactions are supported in multiple ways including binary correct/incorrect signal @xcite , answers to a yes/no or a multiple-choice question posed by the system @xcite or suggestions of edits that can be applied to the parse @xcite . @xcite and @xcite ask yes/no and multiple-choice questions and use the answers in generating the pars. @xcite introduce splash (section 2), a dataset for correcting semantic parsing with natural language feedback. using language as a medium for providing feedback enables the human to provide rich open-form feedback in their natural way of communication giving them control and flexibility specifying what is wrong and how it should be corrected. our work uses splash and proposes to pose the problem of semantic parse correction as a parser editing problem with natural language feedback input. this is also related to recent work on casting text generation (e.g. summarization, grammatical error correction, sentence splitting, etc.) as a text editing task @xcite @xcite where target texts are reconstructed from inputs using several edit operations. semantic parsing with synthetic data: semantic parsing systems have frequently used synthesized data to alleviate the challenge of labeled data scarcity. in their semantic parser overnight work, @xcite proposed a method for training semantic parsers quickly in a new domain using synthetic data. they generate logical forms and canonical utterances and then paraphrase the canonical utterances via crowd-sourcing. several other approaches have demonstrated the benefit of adopting this approach to train semantic parsers in low-resource settings @xcite @xcite . most recently, synthetic data was used to continue to pre-train language models for semantic parsing tasks @xcite . we build on this line work by showing that we can generate synthetic data automatically without human involvement to simulate edits between an erroneous parse and a correct one. ## conclusions and future work we introduced a model, a data augmentation method, and analysis tools for correcting semantic parse errors in text-to-sql through natural language feedback. compared to previous models, our model improves the correction accuracy by 16% and boosts the end-to-end parsing accuracy by up to 20% with only one turn of feedback. our work creates several avenues for future work: (1) improving the model by better modeling the interaction between the inputs and exploring different patterns for decoder-encoder attention, (2) evaluating existing methods for training with synthetic data (e.g., curriculum learning @xcite ), (3) optimizing the correction model for better user experience using the progress measure we introduce, and (4) using the sql edits scheme in other related tasks such as conversational text-to-sql parsing.