Text-to- SQL Error Correction with Language Models of Code (2023)

Abstract: Despite recent progress in text-to-SQL parsing, current semantic parsers are still not accurate enough for practical use. In this paper, we investigate how to build automatic text-to-SQL error correction models. Noticing that token-level edits are out of context and sometimes ambiguous, we propose building clause-level edit models instead. Besides, while most language models of code are not specifically pre-trained for SQL, they know common data structures and their operations in programming languages such as Python. Thus, we propose a novel representation for SQL queries and their edits that adheres more closely to the pre-training corpora of language models of code. Our error correction model improves the exact set match accuracy of different parsers by 2.4-6.5 and obtains up to 4.3 point absolute improvement over two strong baselines.

## introduction text-to-sql parsing is a classic semantic parsing task that finds wide applications @xcite . since the release of spider @xcite , a cross-database text-to-sql benchmark, many semantic parsers with decent performance have been developed @xcite @xcite @xcite . nonetheless, state-of-the-art semantic parsers are still not accurate enough. as a result, their users need to constantly correct wrongly predicted sql queries, which can be as time-consuming and errorprone as writing a sql query from scratch @xcite . therefore, in this paper, we study the problem of automatic text-to-sql error correction to better assist users in querying complex databases. we first highlight that it is essential to factor in the compositional substructures within sql queries, such as abstract syntax trees @xcite and data-flow graphs @xcite , instead of treating code snippets as string sequences. compared to individual tokens, substructures (e.g. sql clauses) include more context of the entire program and are more semantically meaningful. consequently, edit patterns of such substructures are more intuitive for humans to understand and easier for language models to learn. moreover, while the pre-training corpora for language models of code, such as codet5 @xcite , do not include many sql queries based on their documentation, they naturally contain abundant examples of common data structures like dictionaries. therefore, we hypothesize that transforming unfamiliar sql queries into familiar data structures can help language models of code better perform structural editing of sql queries. based on these observations, we develop our error correction model and make two contributions. first, we propose considering sql clauses instead of tokens as basic semantic units for editing. using a context-free grammar, we can decompose a sql query and identify its clauses by traversing its abstract syntax tree. second, we propose a new representation of sql queries and their edits that adheres more closely to common code pre-training corpora, including codesearchnet @xcite , and makes the structures of a sql query more explicit. with a decomposed sql query, we pair each clause with its sql keyword and represent the entire query as a python dictionary. then, we format edits on a wrong sql query as a program that modifies data of the query's corresponding dictionary. unlike token-level edits in existing work @xcite , such dictionary operations define all edits unambiguously and can be directly executed with a python interpreter. through comprehensive experiments with different representations, we show that: (1) our proposed representation has the lowest zero-shot perplexity query representation edit representation sql select tweets.text from tweets order by tweets.text token-@xmath0.@xmath1.@xmath2.@xmath3.@xmath4: "select tweets.text", "from": "from tweets", "orderby": "order by tweets.text" } clause-@xmath5: "order by tweets.text" <@xmath6: "order by tweets.createdate" <@xmath7.createdate" ## text-to-sql error correction given a natural language utterance u, a database schema s, and a wrong sql query q -produced by an existing parser, our goal is to develop an error correction model that predicts a sequence of edit actions e and the correct query q + . following previous work @xcite , we formulate our task as sequence-to-sequence generation: ## related work since the release of codebert @xcite , many language models of code have emerged for program understanding and generation @xcite @xcite @xcite @xcite . in addition to programrelated tasks, recent work shows they also excel at processing natural language structures. using code as meaning representations (mrs), we can leverage language models of code in various tasks, such as commonsense reasoning @xcite , action planning @xcite , and event extraction @xcite . in fact, how to design mrs to reduce model learning difficulty is a salient research question in semantic parsing @xcite @xcite . our work demonstrates that program-related tasks themselves can also benefit from code-based mrs. specifically, we apply such mrs to sql error correction, a variant of automatic program repair tasks @xcite @xcite . although sql is a code-based mr, it is much harder for models to learn compared to other mrs, such as funql and lambda calculus @xcite . consequently, without many sql queries in their pre-training corpora, language models of code can underperform state-of-the-art text-to-sql parsers. by converting sql queries into python dictionaries, we can explicitly represent their compositional substructures and define edit actions as programs, which reduces the learning difficulty for language models of code and yields better performance. ## conclusion and future work this paper presents a study on developing a text-to-sql error correction model with clause-level edits and different representations. our comprehensive experiments demonstrate that clauses are better semantic units than tokens for editing sql queries and mimicking patterns in code pre-training corpora helps better leverage language models of code. as a future direction, we plan to incorporate our model into interactive semantic parsing frameworks @xcite @xcite ) by suggesting possible edits to users once a wrong parse is identified. in this way, users would more efficiently correct parse errors and get better assistance. we also plan to experiment with other language models of code @xcite ) and text-to-sql datasets @xcite to verify the generalizability of our method.