MATE : Multi-view Attention for Table Transformer Efficiency (2021)

Abstract: This work presents a sparse-attention Transformer architecture for modeling documents that contain large tables. Tables are ubiquitous on the web, and are rich in information. However, more than 20% of relational tables on the web have 20 or more rows (Cafarella et al., 2008), and these large tables present a challenge for current Transformer models, which are typically limited to 512 tokens. Here we propose MATE, a novel Transformer architecture designed to model the structure of web tables. MATE uses sparse attention in a way that allows heads to efficiently attend to either rows or columns in a table. This architecture scales linearly with respect to speed and memory, and can handle documents containing more than 8000 tokens with current accelerators. MATE also has a more appropriate inductive bias for tabular data, and sets a new state-of-the-art for three table reasoning datasets. For HybridQA (Chen et al., 2020), a dataset that involves large documents containing tables, we improve the best prior result by 19 points.

## introduction the transformer architecture @xcite is expensive to train and run at scale, especially for long sequences, due to the quadratic asymptotic complexity of self-attention. although some work addresses this limitation @xcite @xcite , there has been little prior work on scalable transformer architectures for semi-structured text. 1 however, although some of the more widely used benchmark tasks involving semi-structured data have been restricted to moderate size tables, many semi-structured documents are large: more than 20% of relational tables on the web have 20 or more rows @xcite figure 1 : sparse self-attention heads on tables in mate are of two classes: row heads attend to tokens inside cells in the same row, as well as the query. column heads attend to tokens in the same column and in the query. query tokens attend to all other tokens. here we study how efficient implementations for transformers can be tailored to semi-structured data. figure 1 highlights our main motivation through an example: to obtain a contextual representation of a cell in a table, it is unlikely that the information in a completely different row and column is needed. we propose the mate architecture foot_1 (section 3), which allows each attention head to reorder the input so as to traverse the data by multiple points of view, namely column or row-wise (figure 2 ). this allows each head to have its own data-dependent notion of locality, which enables the use of sparse attention in an efficient and context-aware way. this work focuses on question answering (qa) and entailment tasks on tables. while we apply our model to several such tasks (see section 6), hybridqa @xcite is particularly interesting, as it requires processing tables jointly with long passages associated with entities mentioned in the table, yielding large documents that may not fit in standard transformer models. ## the mate model following tapas @xcite , the transformer input in mate, for each table-qa example, is the query and the table, tokenized and flattened, separated by a [sep] token, and prefixed by a @xcite . generally the table comprises most of the the input. we use the same row, column and rank embeddings as tapas. to restrict attention between the tokens in the table, we propose having some attention heads limited to attending between tokens in the same row (plus the non-table tokens), and likewise for columns. we call these row heads and column heads respectively. in both cases, we allow attention to and from all the non-table tokens. formally, if @xmath0, the k-th position of the output of the i-th attention head is: in mate, we use two types of attention patterns. the first h @xmath1: one possible implementation of this is an attention mask that selectively sets elements in the attention matrix to zero. (similar masks are used for padding tokens, or auto-regressive text generation.) the ratio of row and column heads is a hyperparameter but empirically we found a 1 : 1 ratio to work well. in section 6 we show that attention masking improves accuracy on four table-related tasks. we attribute these improvements to better inductive bias, and support this in section 7 showing that full attention models learn to approximate this behavior. ## the pointr architecture many standard table qa datasets @xcite @xcite , perhaps by design, use tables that can be limited to 512 tokens. recently, more datasets @xcite requiring parsing larger semi-structured documents have been released. ## experimental setup we begin by comparing the performance of mate on hybridqa to other existing systems. we focus on prior efficient transformers to compare the benefits of the table-specific sparsity. we follow @xcite ; @xcite in reporting error bars with the interquartile range. ## results in figure 4 we compare inference speed of different models as we increase the sequence length. similar results showing number of flops and memory usage are in appendix a. the linear scaling of linformer and the linear-time version mate can be seen clearly. although linformer has a slightly smaller linear constant, the pre-training is 6 times slower, as unlike the other models, lin-former pretraining must be done at the final sequence length of 2048. we also applied mate to three tasks involving table reasoning over shorter sequences. in among the non near-misses, the majority predictions were either numerically incorrect, or were referencing an incorrect entity but still in an relevant context-especially the questions involving more than 2 hops. (e.g. q: in which sport has an award been given every three years since the first tournament held in 1948-1949? g: "badminton", p: "thomas cup"). reassuringly, for a huge majority (> 80%), the entity type of the predicted answer (person, date, place, etc.) matches the type of the gold answer. the observed errors suggest potential gains by improving the entity @xcite and numerical @xcite reasoning skills. ## conclusion we introduce mate, a novel method for efficiently restricting the attention flow in transformers applied to tabular data. we show in both theory and practice that the method improves inductive bias and allows scaling training to larger sequence lengths as a result of linear complexity. we improve the state-of-the-art on tabfact, sqa and hybridqa, the last one by 19 points.