Re-examining the Role of Schema Linking in Text-to- SQL (2020)

Abstract: In existing sophisticated text-to-SQL models, schema linking is often considered as a simple, minor component, belying its importance. By providing a schema linking corpus based on the Spider text-to-SQL dataset, we systematically study the role of schema linking. We also build a simple BERT-based baseline, called Schema-Linking SQL (SLSQL) to perform a data-driven study. We find when schema linking is done well, SLSQL demonstrates good performance on Spider despite its structural simplicity. Many remaining errors are attributable to corpus noise. This suggests schema linking is the crux for the current text-to-SQL task. Our analytic studies provide insights on the characteristics of schema linking for future developments of text-to-SQL tasks.

## introduction structured query language (sql), while exact and powerful, suffers from a complex grammar presenting significant challenges for laymen to write queries. automatically parsing natural language into sql (text-to-sql) thus has huge potential, as it would enable lay users to mine the world's structured data using natural language queries. to achieve practical text-to-sql workflow, a model needs to correlate natural language queries with the given database. therefore, schema linking is considered helpful for text-to-sql parsing @xcite @xcite . here, schema linking means identifying references of columns, tables and condition values in natural language queries. for example, for the question "find the names of schools that have a donation with amount above 8.5" (shown with relevant tables in figure 1 ), "name" is a column reference to school.name, "donation" a table reference to endowment, and "8.5" and "sale" are value references, corresponding to the condition values in the sql query. existing solutions largely treat schema linking as a minor component implemented with simple string matching @xcite @xcite heuristics to support sophisticated textto-sql models. an exception is @xcite , which framed schema linking as a task to be solved by sequential tagging. while they did show the importance of schema linking, how it contribute to text-to-sql task performance remains unanswered as there is no annotated corpus to analyze. to address these shortcomings, we perform an indepth study on the role of schema linking in text-to-sql parsing. intuitively, schema linking helps both cross-domain generalizability and complex sql generation, which have been identified as the current bottlenecks of the text-to-sql task @xcite . by crossdomain generalizability, we refer to the proper separation of training and testing instances and databases, requiring a model to infer against with arbitrary databases where the schema and the domain are previously unknown. this means the model must be aware of what tables and columns are involved in the question -exactly what schema linking does. schema linking indirectly addresses the complex sql generation challenge: the writing of sql queries comprising a mixture of select, group by, and nested clauses. generating such queries requires the modeling of complex semantic dependencies in the input and to manage complex sql grammar during decoding. as discussed in @xcite , detecting and removing domainspecific words from the model's purview allows the model to focus on learning syntactic conversion between natural language and sql, reducing the figure 1 : two examples of schema linking. column, table and value references are marked in red, yellow and green, respectively. the arrows of column and table references indicate their respective referents in the schema. for value references, the arrows point to the columns they compare with. to perform a systematic data-driven study, we annotate and contribute ground truth schema linking data for the publicly-available training and development set of the spider dataset @xcite . we then build a simple bert baseline, named schema-linking sql (slsql), which links the schema in a natural language query and parses the sql query with awareness of the predicted schema linking results. we systematically compare several variants of slsql, each of which utilizes schema linking differently. we find that schema linking always leads to better sql parsing performance. but the performance of schema linking is far from perfect, even under supervised learning (≤ 0.83 f1). to maximize the potential of schema linking, we study how slsql performs during inference when provisioned with oracle schema linking results (i.e., ground truth annotation). as such, we find our simple slsql model performs impressively on spider -the remaining gap is largely due to corpus noise, including inconsistent patterns and errors in the dataset. this evidence points to schema linking as a critical task for text-to-sql parsing, and also provides an indicative upper bound on performance on the spider dataset. interestingly, our analyses on the failure cases caused by model deficiencies reveal advanced challenges on the text-to-sql task like deep logical reasoning and extremely complex structure (c.f. section 5). our annotated data enables us to address these challenges without the interference of the current noisy schema linker. in summary, we contribute an annotation of schema linking and in-depth analyses on the role of schema linking in the text-to-sql task. we identify schema linking as a crux for the further improvement of text-to-sql parsing. our analyses provide insights to advance the understanding of text-to-sql parsing, facilitating future research on the areas of problem identification, dataset construction and model evaluation. ## schema linking annotation to support a data-driven and systematical study, we annotate the schema linking information for each instance in the training and development set of spider @xcite (the test set is hidden), the largest and most challenging text-to-sql dataset so far. a simple way is conducting automatic annotation by matching table/column names and values in a ground truth sql query against its corresponding natural language query. however, such automatic annotation method can bring much noise that would potentially hinder the model performance. therefore, we annotate the dataset combining both automatic and manual processing. ## model to conduct an in-depth study of the role of schema linking, we develop the slsql model, as depicted in figure 2 . it comprises a base model (the left part), which is based on the encoder-decoder structure, and an explicit schema linking component (the right part). while many sophisticated models have been proposed, we adopt this structure to leverage its simplicity to systematically analyze the factor of schema linking. in our experiments, we will study different variants of this structure by configuring the model in various ways. before the detailed model description, we first introduce our mathematical notations. we denote the natural language query q as @xmath0, where q i indicates the i th token of q and |q| indicates the length. similarly, we denote the database schema ## conclusion we critically examine the role of schema linking for the text-to-sql task. to support modelindependent and thorough studies, we invest human resources to annotate schema references and contribute a high-quality, large-scale schema linking corpus. experimenting with our designed schema linking sql (slsql) model, we demonstrate that more accurate schema linking conclusively leads to better text-to-sql parsing performance. importantly, given oracular schema references, a simple bert model like slsql can achieve an impressive performance. our experiments show that schema linking, often overlooked as simple preprocessing, is actually a requisite for good sql parsing performance, providing an intriguing perspective for future improvements on this task. our study sheds light on the characteristics of text-to-sql parsing for future efforts including advanced modeling, problem identification, dataset construction and model evaluation.