Multitask Pretraining with Structured Knowledge for Text-to- SQL Generation (2023)

Abstract: Many machine learning-based low-code or no-code applications involve generating code that interacts with structured knowledge. For example, one of the most studied tasks in this area is generating SQL code from a natural language statement. Prior work shows that incorporating context information from the database schema, such as table and column names, is beneficial to model performance on this task. In this work we present a large pretraining dataset and strategy for learning representations of text, tables, and SQL code that leverages the entire context of the problem. Specifically, we build on existing encoder-decoder architecture by introducing a multitask pretraining framework that complements the unique attributes of our diverse pretraining data. Our work represents the first study on large-scale pretraining of encoder-decoder models for interacting with structured knowledge, and offers a new state-of-the-art foundation model in text-to-SQL generation. We validate our approach with experiments on two SQL tasks, showing improvement over existing methods, including a 1.7 and 2.2 percentage point improvement over prior state-of-the-arts on Spider and CoSQL.

## introduction tables, relational databases, and other forms of structured knowledge (sk) encompass a massive amount of data across a wide range of applications. extracting insights held in such data often requires proficiency in query languages like sql, making it only accessible to the minority of people with the technical skills. a natural language interface, however, would expand access to these information exponentially. likewise, querying via natural language allows users quickly hone in on an answer to their particular question, rather than visually scanning dense tables where the majority of the information is irrelevant to the user. to that end, we explore pretraining techniques for large language models that focus on the challenging interplay between structured and unstructured knowledge, and target a variety of downstream text-to-sql tasks. recently there have been significant advancements in learning representations for tables @xcite @xcite @xcite @xcite , which advanced the state-of-the-art in a range of table-totext tasks, like table question-answering @xcite , fact verification @xcite , data-to-text @xcite , and semantic parsing @xcite . while better table understanding benefits a range of tasks, pretraining focused on text-to-sql has thus far received less attention. pretrained encoders, such as tabert and tapas @xcite @xcite , show that pretraining bert-style encoders @xcite on tables with mask language modeling (mlm) loss produces a strong foundation model that can be extended for text-to-sql. grappa includes small amount of synthetic sql code in the pretraining data to more specifically target the text-to-sql task @xcite . these encoder-only approaches are, however, restricted in their generative capabilities as they must be combined with an additional module that is carefully designed to generate valid sql code @xcite . encoder-decoder architectures like t5 @xcite , on the other hand, exhibit better performance on text-to-sql to-date when constraining the decoder with rules that check for syntactic correctness @xcite . however, the t5based models with exceptional text-to-sql performance @xcite have still only been pretrained on natural language (nl) -begging the question, can text-to-sql encoderdecoders benefit from pretraining on structured in-11067 for our diverse of pretraining data. in (1-4) we show the context-to-output format for four data sources, however in pretraining mlm objectives are also applied. for example, (5) shows the mlm objectives applied to the same data as (4), showing the combination of the t5 style of masking with masked column recovery. for data sources ( 1 ) and in this work we introduce our sql and table aligned multi-task pretraining (stamp) framework, which explores pretraining encoder-decoder models for text-to-sql. starting from text-only t5 @xcite checkpoints, our multi-stage pretraining framework refines previous text-only models by continuing training on a collection of large multi-modal datasets that combine structured knowledge with natural language and sql. additionally, inspired by the impressive generalization of large language models incorporating code in pretraining data @xcite @xcite @xcite , we apply our pretraining framework to codet5 @xcite checkpoints that are trained on code. building on recent work in multi-task pretraining @xcite @xcite , we combine masked language modeling (mlm) with taskaware context-to-output objectives that vary across tasks and datasets. for pretraining datasets with multiple modalities (i.e. combinations of nl, sql, and structured knowledge) or intrinsic splits (e.g. question and answer), we explore the benefit of the dual learning objectives @xcite . we assess our pretraining strategy on a variety of sql benchmarks following the unifiedskg framework @xcite . our approach outperforms previous text-and code-only pretraining, and gives a new state-of-the-art on a range of benchmarks. to better understand our strategy, we present ablation studies on the optimal objective mix, the impact of linearizing structured knowledge into row-versus column-centric tables, and the effect of building on previously pretrained text-versus code-only checkpoints. our work shows that continued pretraining with multi-task learning is a promising direction for advancing the capacity of language models. ## related work encoder-only encoder-only transformer architectures like bert and its successors @xcite @xcite @xcite optimize masked language modeling (mlm) objectives while using a bidirectional receptive field covering the whole input sequence. the encoderonly architectures perform well across a variety of tasks like classification, regression, sentiment analysis, question-answering, and retrieval. however, recent work @xcite @xcite shows that tasks like table-to-text and text-to-sql require additional pretraining on structured knowledge for good generalization, and adapting mlm objectives to the unique structure of tabular data improves learning. prior to bert, text-to-sql models like sql-net and seq2sql @xcite encoded inputs with bidirectional lstms @xcite and generated queries via slot-filling. text-to-sql performance improved with the adoption of bert-based encoders, for example @xcite attach feed forward networks and lstms to the bert-style encoder to generate queries. because encoder-only architectures are restricted in their ability to generate sequences, they require careful design to generate valid sql queries and limit the complexity of those queries. encoder-decoder alternatively, encoderdecoders like bart @xcite and t5 @xcite combine the bidirectional encoder with a causal decoder are naturally suited for sequence-to-sequence tasks like text-to-sql, and are quickly becaming the mainstream approach due to the reduced need for domain specific solutions @xcite . t5 @xcite in particular achieves impressive performance on a range of table-to-text and text-to-sql tasks @xcite despite pretraining that is limited to nl. moreover, @xcite and @xcite leverage a bart-style encoder-decoder to improve the performance of pretrained models for text-to-sql and table-to-text tasks, respectively. we follow this line, proposing a strategy that builds on top of t5 and codet5 @xcite . multi-task training @xcite explore various self-supervised objectives, and found the fill-in-the-blank style of denoising objective as most effective. additionally, combining mlm objectives with small amounts of auxiliary objectives is effective @xcite . for encoder-decoder models @xcite ; @xcite show the benefit of multi-task pretraining on a mix of the t5 span corruption objective @xcite along with a the causal language modeling (clm) style of objective, similar to those used in decoder-only architectures @xcite . in the domain of textto-sql, @xcite ; tao @xcite perform multitask learning by combining mlm with sql specific objectives. lastly, @xcite al. (2022); aghajanyan et al. (2021); aribandi et al. (2021); sanh et al. (2022); fitzgerald et al. (2022); chen et al. (2022) demonstrate that multi-task learning across a variety of datasets can improve performance relative to the single-task, single-dataset paradigm. @xcite show that an objective mix specific to programming languages (pl) along with dual learning on bimodal data promotes generation on tasks combining pl and nl. ## multi-task pretraining on structured knowledge our sql and table aligned multi-task pretraining (stamp) model builds on the t5 encoder-decoder architecture and pretraining checkpoints @xcite , and similarly our codestamp models build on the codet5 architecture and checkpoints @xcite . we develop a multitask pretraining framework specifically designed to leverage our large and unique collection of data that combine various data modalities, namely natural language (nl), stuctured knowledge (sk), and sql. stamp introduces a new stage of pretraining that transitions t5 from being a purely nl programing language (pl) trained model to a backbone model that excels at text-to-sql generation. next, we present the construction of our pretraining dataset in section 3.1, the mixture of objectives designed to learn the unique structure of our data and align the nl, sk, sql data modalities in section 3.2, and our unified format for representing tasks and structured knowledge in section 3.3. ## conclusion we present stamp, a pretraining framework for encoder-decoders on sql tasks. we introduce a large scale pretraining dataset of tables, sql code, discussions on stack overflow, and a modified tapex dataset @xcite . we complement our data with a multi-task learning framework to align the data modalities, finding that an equal mix of the objectives is optimal. we explore both row-and column-centric approaches to linearizing tables, creating a unified format across training stages. a column-centric format is often superior, challenging the conventional row-centric approach. lastly, while pl pretraining may help generalization @xcite , stamp models based on t5 yield better performance.