Correcting Semantic Parses with Natural Language through Dynamic Schema Encoding (2023)

Abstract: In addressing the task of converting natural language to SQL queries, there are several semantic and syntactic challenges. It becomes increasingly important to understand and remedy the points of failure as the performance of semantic parsing systems improve. We explore semantic parse correction with natural language feedback, proposing a new solution built on the success of autoregressive decoders in text-to-SQL tasks. By separating the semantic and syntactic difficulties of the task, we show that the accuracy of text-to-SQL parsers can be boosted by up to 26% with only one turn of correction with natural language. Additionally, we show that a T5-base model is capable of correcting the errors of a T5-large model in a zero-shot, cross-parser setting.

## introduction the task of parsing natural language into structured database queries has been a long-standing benchmark in the field of semantic parsing. success at this task allows individuals without expertise in the downstream query language to retrieve information with ease. this helps to improve data literacy, democratizing accessibility to otherwise opaque public database systems. many forms of semantic parsing datasets exist, such as parsing natural language to programming languages @xcite @xcite , prolog assertions for exploring a database of geographical data @xcite , or sparql queries for querying a large knowledge base @xcite . the current work discusses parsing natural language into a structured query language (sql), perhaps the most well-studied sub-field of semantic parsing. most text-to-sql works frame the task as a oneshot mapping problem. methods include transitionbased parsers @xcite , grammarbased decoding @xcite , an incorrect parse from a neural text-to-sql model is paired together with natural language feedback commenting on how the parse should be corrected. and the most popular approach as of late, sequence to sequence (seq2seq) models @xcite @xcite . in contrast to the one-shot approach, conversational text-to-sql aims to interpret the natural language to structured representations in the context of a multi-turn dialogue @xcite . it requires some form of state tracking in addition to semantic parsing to handle conversational phenomena like coreference and ellipsis @xcite @xcite . interactive semantic parsing frames the task as a multi-turn interaction, but with a different objective than pure conversational text-to-sql. as a majority of parsing mistakes that neural text-to-sql parsers make are minor, it is often feasible for humans to suggest fixes for such mistakes using natural language feedback. displayed in figure 1 , splash (semantic parsing with language assistance from humans) is a text-to-sql dataset containing erroneous parses from a neural text-to-sql system alongisde human feedback explaining how the interpretation should be corrected @xcite . most similar to splash is the inspired dataset @xcite , which aims to correct errors in sparql parses from the complexwebquestions dataset @xcite . while the interactive semantic parsing task evaluates a system's ability to incorporate human feedback, as noted in @xcite , it targets a different modeling aspect than the traditional conversational paradigm. hence, good performance on one does not guarantee good performance on the other task. we make the following contributions: (1) we achieve a new state-of-the-art on the interactive parsing task splash, beating the best published correction accuracy @xcite by 12.33% using destt5 (dynamic encoding of schemas using t5); (2) we show new evidence that the decoupling of syntactic and semantic tasks improves text-to-sql results @xcite , proposing a novel architecture which leverages a single language model for both tasks; (3) we offer a new small-scale test set for interactive parsing 1 , and show that a t5-base interactive model is capable of correcting errors made by a t5-large parser. ## dataset in this work, we evaluate our models on the splash dataset as introduced in @xcite . it is based on spider, a large multidomain and cross-database dataset for text-to-sql parsing @xcite . incorrect sql parses were selected from the output of a seq2struct model trained on spider @xcite . seq2struct achieves an exact set match accuracy of 42.94% on the development set of spider. alongside the incorrect parse, an explanation of the sql query is generated using a rule-based template. annotators were then shown the original question q alongside the explanation and asked to provide natural language feedback f such that the incorrect parse p ′ could be resolved to the final gold parse p. each item in the splash dataset is associated with a relational database d. each database has a schema s containing tables @xmath0, t 2 , ... ## conclusion we present a new model, destt5 (dynamic encoding of schemas using t5), which achieves a new state-of-the-art correction accuracy on the interactive parsing dataset splash. by using t5 as a schema prediction model, we display better performance compared to classification-based methods. we validate our results on a new test set for interactive semantic parsing based on a modern parser, and offer recommendations for evaluating future systems.