T able F ormer: Robust Transformer Modeling for Table-Text Encoding (2022)

Abstract: Understanding tables is an important aspect of natural language understanding. Existing models for table understanding require linearization of the table structure, where row or column order is encoded as an unwanted bias. Such spurious biases make the model vulnerable to row and column order perturbations. Additionally, prior work has not thoroughly modeled the table structures or table-text alignments, hindering the table-text understanding ability. In this work, we propose a robust and structurally aware table-text encoding architecture TableFormer, where tabular structural biases are incorporated completely through learnable attention biases. TableFormer is (1) strictly invariant to row and column orders, and, (2) could understand tables better due to its tabular inductive biases. Our evaluations showed that TableFormer outperforms strong baselines in all settings on SQA, WTQ and TabFact table reasoning datasets, and achieves state-of-the-art performance on SQA, especially when facing answer-invariant row and column order perturbations (6% improvement over the best baseline), because previous SOTA models’ performance drops by 4% - 6% when facing such perturbations while TableFormer is not affected.

## introduction recently, semi-structured data (e.g. variable length tables without a fixed data schema) has attracted more attention because of its ubiquitous presence on the web. on a wide range of various table reasoning tasks, transformer based architecture along with pretraining has shown to perform well @xcite . in a nutshell, prior work used the transformer architecture in a bert like fashion by serializing tables or rows into word sequences @xcite title length screwed up 5:02 ghetto queen 5:00 question: of all song lengths, which one is the longest? gold answer: 5:02 tapas: 5:00 tapas after row order perturbation: 5:02 tableformer: 5:02 (a) tapas predicts incorrect answer based on the original table, while it gives the correct answer if the first row is moved to the end of the table. table-former predicts correct answers for both questions. @xcite , where original position ids are used as positional information. due to the usage of row/column ids and global position ids, prior strategies to linearize table structures introduced spurious row and column order biases @xcite @xcite @xcite . therefore, those models are vulnerable to row or column order perturbations. but, ideally, the model should make consistent predictions regardless of the row or column ordering for all practical purposes. for instance, in figure 1 , the predicted answer of tapas model @xcite for question (a) "of all song lengths, which one is the longest?" based on the original table is "5:00", which is incorrect. however, if the first row is adjusted to the end of the table during inference, the model gives the correct length "5:02" as an-swer. this probing example shows that the model being aware of row order information is inclined to select length values to the end of the table due to spurious training data bias. in our experiments on the sqa dataset, tapas models exhibit a 4% -6% (section 5.2) absolute performance drop when facing such answer-invariant perturbations. besides, most prior work @xcite did not incorporate enough structural biases to models to address the limitation of sequential transformer architecture, while others inductive biases which are either too strict @xcite or computationally expensive @xcite . to this end, we propose tableformer, a transformer architecture that is robust to row and column order perturbations, by incorporating structural biases more naturally. tableformer relies on 13 types of task-independent table↔text attention biases that respect the table structure and table-text relations. for question (a) in figure 1 , tableformer could predict the correct answer regardless of perturbation, because the model could identify the same row information with our "same row" bias, avoiding spurious biases introduced by row and global positional embeddings. for question (b), tapas predicted only partially correct answer, while tableformer could correctly predict "spain, ukraine" as answers. that's because our "cell to sentence" bias could help table cells ground to the paired sentence. detailed attention bias types are discussed in section 5.2. experiments on 3 table reasoning datasets show that tableformer consistently outperforms original tapas in all pretraining and intermediate pretraining settings with fewer parameters. also, tableformer's invariance to row and column perturbations, leads to even larger improvement over those strong baselines when tested on perturbations. our contributions are as follows: ## preliminaries: tapas for table encoding in this section, we discuss tapas which serves as the backbone of the recent state-of-the-art tabletext encoding architectures. tapas @xcite uses transformer architecture in a bert like fashion to pretrain and finetune on tabular data for table-text understanding tasks. this is achieved by using linearized table and texts for masked language model pre-training. in the finetuning stage, texts in the linearized table and text pairs are queries or statements in table qa or tabletext entailment tasks, respectively. specifically, tapas uses the tokenized and flattened text and table as input, separated by [sep] token, and prefixed by @xcite . besides token, segment, and global positional embedding introduced in bert @xcite , it also uses rank embedding for better numerical understanding. moreover, it uses column and row embedding to encode table structures. concretely, for any table-text linearized sequence @xmath0, v 2 , • • • , v n }, where n is the length of table-text sequence, the input to tapas is summation of embedding of the following: as for the model, tapas uses bert's selfattention architecture @xcite offthe-shelf. each transformer layer includes a multihead self-attention sub-layer, where each token attends to all the tokens. let the layer input @xmath1, h 2 , • • • , h n ] ∈ r n×d corresponding to s, where d is the hidden dimension, and h @xmath2. for a single-head self-attention sub-layer, the input h is projected by three matrices w @xmath3, w @xmath4, and w @xmath5, k, and v : learnable structure enforced attention bias scalars. we use 13 types of attention biases based on task independent relation between header, row, column, text, etc. then, the output of this single-head selfattention sub-layer is calculated as: ## tableformer: robust structural table encoding as shown in figure 2 , tableformer encodes the general table structure along with the associated text by introducing task-independent relative attention biases for input of tableformer. tableformer uses the same token embeddings w , segment embeddings g, and rank embeddings z as tapas. however, we make 2 major modifications: positional encoding in tableformer. note that the transformer model either needs to specify different positions in the input (i.e. absolute positional encoding of @xcite ) or encode the positional dependency in the layers (i.e. relative positional encoding of @xcite ). tableformer does not consume any sort of column and row order information in the input. the main intuition is that, for cells in the table, the only useful positional information is whether two cells are in the same row or column and the column header of each cell, instead of the absolute order of the row and column containing them. thus, inspired by relative positional encoding @xcite and graph encoding @xcite , we capture this with a same column/row relation as one kind of relative position between two linearized tokens. similarly, we uses 12 such table-text structure relevant relations (including same cell, cell to header and so on) and one extra type representing all other relations not explicitly defined. all of them are introduced in the form of learnable attention bias scalars. formally, we consider a function φ(v i , v j ) : v × @xmath6, which measures the relation between v i and v j in the sequence (v i , v @xmath7. the function φ can be defined by any relations between the tokens in the table-text pair. attention biases in tableformer. in our work, φ(v i , v j ) is chosen from 13 bias types, corresponding to 13 table-text structural biases. the attention biases are applicable to any table-text pair and can be used for any downstream task: note that, each cell can still attend to other cells in the different columns or rows through "others" instead of masking them out strictly. we assign each bias type a learnable scalar, which will serve as a bias term in the self-attention module. specifically, each self-attention head in each layer have a set of learnable scalars {b 1 , b 2 , • • • , b 13 } corresponding to all types of introduced biases. for one head in one self-attention sub-layer of tableformer, equation 2 in the transformer is replaced by: ## conclusion in this paper, we identified the vulnerability of prior table encoding models along two axes: (a) capturing the structural bias, and (b) robustness to row and column perturbations. to tackle this, we propose tableformer, where learnable task-independent learnable structural attention biases are introduced, while making it invariant to row/column order at the same time. experimental results showed that tableformer outperforms strong baselines in 3 table reasoning tasks, achieving state-of-the-art performance on sqa dataset, especially when facing row and column order perturbations, because of its invariance to row and column orders.