Error Detection for Text-to- SQL Semantic Parsing (2023)

Abstract: Despite remarkable progress in text-to-SQL semantic parsing in recent years, the performance of existing parsers is still far from perfect. Specifically, modern text-to-SQL parsers based on deep learning are often over-confident, thus casting doubt on their trustworthiness when deployed for real use. In this paper, we propose a parser-independent error detection model for text-to-SQL semantic parsing. Using a language model of code as its bedrock, we enhance our error detection model with graph neural networks that learn structural features of both natural language questions and SQL queries. We train our model on realistic parsing errors collected from a cross-domain setting, which leads to stronger generalization ability. Experiments with three strong text-to-SQL parsers featuring different decoding mechanisms show that our approach outperforms parser-dependent uncertainty metrics. Our model could also effectively improve the performance and usability of text-to-SQL semantic parsers regardless of their architectures.

## introduction recent years have witnessed a renewed interest in text-to-sql semantic parsing @xcite @xcite @xcite @xcite @xcite , which allows users with a limited technical background to access databases through a natural language interface. although state-of-the-art semantic parsers have achieved remarkable performance on spider @xcite , a large-scale cross-domain text-to-sql benchmark, their performance is still far from satisfactory for real use. while syntax errors can be automatically caught by sql execution engines, detecting semantic errors in executable sql queries can be non-trivial and time-consuming even for experts @xcite . therefore, an accurate error detector that can flag parsing issues and accordingly trigger error correction procedures @xcite can contribute to building better natural language interfaces to databases. researchers have proposed multiple approaches for error detection in text-to-sql parsing. @xcite detect errors by setting a threshold on the prediction probability or dropout-based uncertainty of the base parser. however, using these parser-dependent metrics requires the base parser to be calibrated, which limits their applicability. several interactive text-to-sql systems detect parsing errors based on uncertain span detection @xcite @xcite . despite having high coverage for errors, this approach is reported to be of low precision. finally, text-to-sql re-rankers @xcite @xcite , which estimate the plausibility of sql predictions, can be seen as on-the-fly error detectors. nevertheless, existing re-rankers are trained on in-domain parsing errors, limiting their generalization ability. in this work, we propose a generalizable and parser-independent error detection model for textto-sql semantic parsing. since syntax errors can be easily detected by an execution engine, we focus on detecting semantic errors in executable sql predictions. we start developing our model with codebert @xcite , a language model pre-trained on multiple programming languages. on top of that, we use graph neural networks to capture compositional structures in natural language questions and sql queries to improve the performance and generalizability of our model. we train our model on parsing mistakes collected from a realistic cross-domain setting, which is indispensable to the model's strong generalization ability. furthermore, we show that our model is versatile and can be used for multiple tasks, including er-ror detection, re-ranking, and interaction triggering. to summarize, our contributions include: ## experiments in this section, we first evaluate the performance (section 4.2.1) and generalization ability (section 4.2.2) of our error detection model on the binary error detection task. then we investigate our model's effectiveness when used for re-ranking (section 4.2.3) and action triggering (section 4.2.4). ## conclusion in this work, we propose the first generalizable parser-independent error detection model for textto-sql semantic parsing. through learning compositional structures in natural language and sql queries, the proposed model significantly outperforms parser-dependent uncertainty metrics and could generalize well to unseen parsers. we further demonstrate the versatility of our approach in error detection, re-ranking, and action triggering through a case study with three state-of-the-art text-to-sql parsers featuring different decoding mechanisms. our experiments highlight the important role of structural features and cross-domain training data in building strong and generalizable error detec-tors for semantic parsing. potential future work includes (1) developing more advanced architectures to better evaluate the semantic correctness of synthesized sql queries, (2) exploring data synthesis strategies to automatically create high-quality training data for error detection models.