TABBIE : Pretrained Representations of Tabular Data (2021)

Abstract: Existing work on tabular representation-learning jointly models tables and associated text using self-supervised objective functions derived from pretrained language models such as BERT. While this joint pretraining improves tasks involving paired tables and text (e.g., answering questions about tables), we show that it underperforms on tasks that operate over tables without any associated text (e.g., populating missing cells). We devise a simple pretraining objective (corrupt cell detection) that learns exclusively from tabular data and reaches the state-of-the-art on a suite of table-based prediction tasks. Unlike competing approaches, our model (TABBIE) provides embeddings of all table substructures (cells, rows, and columns), and it also requires far less compute to train. A qualitative analysis of our model’s learned cell, column, and row representations shows that it understands complex table semantics and numerical trends.

## introduction large-scale self-supervised pretraining has substantially advanced the state-of-the-art in natural language processing @xcite @xcite . more recently, these pretraining methods have been extended to jointly learn representations of tables as well as text @xcite , which enables improved modeling of tasks such as question answering over tables. however, many practical problems involve semantic understanding of tabular data without additional text-based input, such as extracting tables from documents, retrieving similar columns or cells, and filling in missing information @xcite . in this work, we design a pretraining methodology specifically for tables (tabular information embedding or tabbie) that resembles several core tasks in table extraction and decomposition pipelines and figure 1 : tabbie is a table embedding model trained to detect corrupted cells, inspired by the elec-tra @xcite objective function. this simple pretraining objective results in powerful embeddings of cells, columns, and rows, and it yields stateof-the-art results on downstream table-based tasks. our tabbie architecture relies on two transformers that independently encode rows and columns, respectively; their representations are pooled at each layer. this setup reduces the sequence length of each transformer's input, which cuts down on its complexity, while also allowing us to easily extract representations of cells, rows, and columns. additionally, tabbie uses a simplified training objective compared to masked language modeling: instead of predicting masked cells, we repurpose electra's objective function @xcite for tabular pretraining by asking the model to predict whether or not each cell in a table is real or corrupted. we emphasize that this pretraining objective is a fundamental task in table structure decomposition pipelines @xcite @xcite , in which incorrectly predicting row/column separators or cell boundaries leads to corrupted cell text. unlike @xcite , we do not require a separate "generator" model that produces corrupted candidates, as we observe that simple corruption processes (e.g., sampling cells from other tables, swapping cells within the same column) yield powerful representations after pretraining. in a controlled comparison to tabert (pretraining on the same number of tables and using a similarly-sized model), we evaluate tabbie on three table-based benchmarks: column population, row population, and column type prediction. on most configurations of these tasks, tabbie achieves state-of-the-art performance, outperforming tabert and other baselines, while in others it performs competitively with tabert. additionally, tabbie was trained on 8 v100 gpus in just over a week, compared to the 128 v100 gpus used to train tabert in six days. a qualitative nearest-neighbor analysis of embeddings derived from tabbie confirms that it encodes complex semantic properties about textual and numeric cells and substructures. we release our pretrained models and code to support further advances on tablebased tasks. foot_1 2 model tabbie is a self-supervised pretraining approach trained exclusively on tables, unlike prior approaches @xcite ) that jointly model tables and associated text snippets. at a high level, tabbie encodes each cell of a table using two different transformer models @xcite , one operating across the rows of the table and the other across columns. at each layer, the representations from the row and column transformers are averaged and then passed as input to the next layer, which produces a contextualized representation of each cell within the table. we place a binary classifier over tabbie's final-layer cell representations to predict whether or not it has been corrupted, or replaced by an intruder cell during preprocessing, inspired by the electra objective of @xcite . in the remainder of this section, we formalize both tabbie's model architecture and pretraining objective. ## experiments we validate tabbie's table representation quality through its performance on three downstream tablecentric benchmarks (column population, row population, and column type prediction) that measure semantic table understanding. in most configurations of these tasks, tabbie outperforms tabert and other baselines to set new state-of-the-art numbers. note that we do not investigate tabbie's performance on table-and-text tasks such as wikitable-questions @xcite , as our focus is not on integrating tabbie into complex taskspecific pipelines @xcite , although this is an interesting avenue for future work. ## analysis the results in the previous section show that tab-bie is a powerful table representation method, outperforming tabert in many downstream task configurations and remaining competitive in the rest. in this section, we dig deeper into tabbie's representations by comparing them to tabert across a variety of quantitative and qualitative analysis tasks, including our own pretraining task of corrupt cell classification, as well as embedding clustering and nearest neighbors. taken as a whole, the analysis suggests that tabbie is able to better capture fine-grained table semantics. ## related work the staggering amount of structured relational data in the form of tables on the internet has attracted considerable attention from researchers over the past two decades @xcite @xcite @xcite , with applications including retrieval @xcite , schemamatching @xcite , and entity linking @xcite . similar to popular large-scale language models pretrained on tasks involving unstructured natural language @xcite @xcite , our work is part of a recent trend of self-supervised models trained on structured tabular data. tabert @xcite and tapas @xcite jointly model tables with text (typically captions or questions), and are thus more suited for tasks like question answering @xcite . for pretraining, tabert attempts to recover the name and datatype of masked column headers (masked column prediction), in addition to contents of a particular cell (cell value recovery). the pretraining objectives of tapas, on the other hand, encourage tabular textual entailment. in a concurrent work, the tuta model @xcite uses masked language modeling, cell-level cloze prediction, and tablecontext retrieval as pretraining objectives. further, in addition to traditional position embeddings, this work accounts for the hierarchical nature of tabular data using tree-based positional embeddings. similiarly, in @xcite , the authors perform a variant of mlm called masked entity recovery. in contrast, tabbie is pretrained strictly on tabular data and intended for more general-purpose tablebased tasks, and uses corrupt-cell classification as its pretraining task. ## conclusion in this paper, we proposed tabbie, a selfsupervised pretraining method for tables without associated text. to reduce the computational cost of training our model, we repurpose the electra objective for corrupt cell detection, and we use two separate transformers for rows and columns to minimize complexity associated with sequence length. on three downstream table-based tasks, tabbie achieves competitive or better performance to existing methods such as tabert, and an analysis reveals that its representations include a deep semantic understanding of cells, rows, and columns. we publicly release our tabbie pretrained models and code to facilitate future research on tabular representation learning.