T a BERT : Pretraining for Joint Understanding of Textual and Tabular Data (2020)

Abstract: Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language (NL) understanding tasks. Such models are typically trained on free-form NL text, hence may not be suitable for tasks like semantic parsing over structured data, which require reasoning over both free-form NL questions and structured tabular data (e.g., database tables). In this paper we present TaBERT, a pretrained LM that jointly learns representations for NL sentences and (semi-)structured tables. TaBERT is trained on a large corpus of 26 million tables and their English contexts. In experiments, neural semantic parsers using TaBERT as feature representation layers achieve new best results on the challenging weakly-supervised semantic parsing benchmark WikiTableQuestions, while performing competitively on the text-to-SQL dataset Spider.

## introduction recent years have witnessed a rapid advance in the ability to understand and answer questions about free-form natural language (nl) text @xcite , largely due to large-scale, pretrained language models (lms) like bert @xcite . these models allow us to capture the syntax and semantics of text via representations learned in an unsupervised manner, before fine-tuning the model to downstream tasks @xcite @xcite @xcite . it is also relatively easy to apply such pretrained lms to comprehension tasks that are modeled as text span selection problems, where the boundary of an answer span can be predicted using a simple classifier on top of the lm @xcite . however, it is less clear how one could pretrain and fine-tune such models for other qa tasks that involve joint reasoning over both free-form nl text and structured data. one example task is semantic parsing for access to databases (dbs) @xcite @xcite , the task of transducing an nl utterance (e.g., "which country has the largest gdp?") into a structured query over db tables (e.g., sql querying a database of economics). a key challenge in this scenario is understanding the structured schema of db tables (e.g., the name, data type, and stored values of columns), and more importantly, the alignment between the input text and the schema (e.g., the token "gdp" refers to the gross domestic product column), which is essential for inferring the correct db query @xcite . neural semantic parsers tailored to this task therefore attempt to learn joint representations of nl utterances and the (semi-)structured schema of db tables (e.g., representations of its columns or cell values, as in @xcite ; @xcite ; @xcite , inter alia). however, this unique setting poses several challenges in applying pretrained lms. first, information stored in db tables exhibit strong underlying structure, while existing lms (e.g., bert) are solely trained for encoding free-form text. second, a db table could potentially have a large number of rows, and naively encoding all of them using a resource-heavy lm is computationally intractable. finally, unlike most text-based qa tasks (e.g., squad, @xcite ) which could be formulated as a generic answer span selection problem and solved by a pretrained model with additional classification layers, semantic parsing is highly domain-specific, and the architecture of a neural parser is strongly coupled with the structure of its underlying db (e.g., systems for sql-based and other types of dbs use different encoder mod-els). in fact, existing systems have attempted to leverage bert, but each with their own domainspecific, in-house strategies to encode the structured information in the db @xcite @xcite , and importantly, without pretraining representations on structured data. these challenges call for development of general-purpose pretraining approaches tailored to learning representations for both nl utterances and structured db tables. in this paper we present tabert, a pretraining approach for joint understanding of nl text and (semi-)structured tabular data ( § 3). tabert is built on top of bert, and jointly learns contextual representations for utterances and the structured schema of db tables (e.g., a vector for each utterance token and table column). specifically, tabert linearizes the structure of tables to be compatible with a transformer-based bert model. to cope with large tables, we propose content snapshots, a method to encode a subset of table content most relevant to the input utterance. this strategy is further combined with a vertical attention mechanism to share information among cell representations in different rows ( § 3.1). to capture the association between tabular data and related nl text, tabert is pretrained on a parallel corpus of 26 million tables and english paragraphs ( § 3.2). tabert can be plugged into a neural semantic parser as a general-purpose encoder to compute representations for utterances and tables. our key insight is that although semantic parsers are highly domain-specific, most systems rely on representations of input utterances and the table schemas to facilitate subsequent generation of db queries, and these representations can be provided by tabert, regardless of the domain of the parsing task. we apply tabert to two different semantic parsing paradigms: (1) a classical supervised learning setting on the spider text-to-sql dataset @xcite , where tabert is fine-tuned together with a task-specific parser using parallel nl utterances and labeled db queries ( § 4.1); and (2) a challenging weakly-supervised learning benchmark wikitablequestions @xcite , where a system has to infer latent db queries from its execution results ( § 4.2). we demonstrate tabert is effective in both scenarios, showing that it is a drop-in replacement of a parser's original encoder for computing contextual representations of nl utterances and db tables. specifically, systems augmented with tabert outperforms their counterparts using bert, registering state-of-the-art performance on wikitable-questions, while performing competitively on spider ( § 5). depending on the underlying data representation schema used by the db, a table could either be fully structured with strongly-typed and normalized contents (e.g., a table column named distance has a unit of kilometers, with all of its cell values, like 200, bearing the same unit), as is commonly the case for sql-based dbs ( § 4.1). alternatively, it could be semi-structured with unnormalized, textual cell values (e.g., 200 km, § 4.2). the query language could also take a variety of forms, from general-purpose db access languages like sql to domain-specific ones tailored to a particular task. given an utterance and its associated tables, a neural semantic parser generates a db query from the vector representations of the utterance tokens and the structured schema of tables. in this paper we refer schema as the set of columns in a table, and its representation as the list of vectors that represent its columns foot_0 . we will introduce how tabert computes these representations in § 3.1. ## tabert: learning joint representations over textual and tabular data we first present how tabert computes representations for nl utterances and table schemas ( § 3.1), and then describe the pretraining procedure ( § 3.2). given an utterance u and a table t , tabert first creates a content snapshot of t . this snapshot consists of sampled rows that summarize the information in t most relevant to the input utterance. the model then linearizes each row in the snapshot, concatenates each linearized row with the utterance, and uses the concatenated string as input to a transformer (e.g., bert) model, which outputs row-wise encoding vectors of utterance tokens and cells. the encodings for all the rows in 3 example adapted from stanford.io/38iz8pf ## example application: semantic parsing over tables we apply tabert for representation learning on two semantic parsing paradigms, a classical supervised text-to-sql task over structured dbs ( § 4.1), and a weakly supervised parsing problem on semistructured web tables ( § 4.2). ## experiments in this section we evaluate tabert on downstream tasks of semantic parsing to db tables. ## conclusion and future work we present tabert, a pretrained encoder for joint understanding of textual and tabular data. we show that semantic parsers using tabert as a general-purpose feature representation layer achieved strong results on two benchmarks. this work also opens up several avenues for future work. first, we plan to evaluate tabert on other related tasks involving joint reasoning over textual and tabular data (e.g., table retrieval and table-totext generation). second, following the discussions in § 5, we will explore other table linearization strategies with transformers, improving the quality of pretraining corpora, as well as novel unsupervised objectives. finally, to extend tabert to cross-lingual settings with utterances in foreign languages and structured schemas defined in english, we plan to apply more advanced semantic similarity metrics for creating content snapshots. language modeling (mlm) objective for utterance tokens, the masked column prediction (mcp) objective for columns, and the column value recovery (cvr) objective for their cell values. an exception is pretraining the tabert(@xmath0. since there are no additional vertical attention layers, we do not use the cvr objective, and the mcp objective reduces to the vanilla mlm objective over encodings from the base transformer model. our largest model tabert large (@xmath1.