Permutation Invariant Strategy Using Transformer Encoders for Table Understanding (2022)

Abstract: Representing text in tables is essential for many business intelligence tasks such as semantic retrieval, data exploration and visualization, and question answering. Existing methods that leverage pretrained Transformer encoders range from a simple construction of pseudo-sentences by concatenating text across rows or columns to complex parameter-intensive models that encode table structure and require additional pretraining. In this work, we introduce a novel encoding strategy for Transformer encoders that preserves the critical property of permutation invariance across rows or columns. Unlike existing state-of-the-art methods for Table Understanding, our proposed approach does not require any additional pretraining and still substantially outperforms existing methods in almost all instances. We demonstrate the effectiveness of our proposed approach on three table interpretation tasks: column type annotation, relation extraction, and entity linking through extensive experiments on existing tabular datasets.

## introduction representation learning of natural language text within relational databases, spreadsheets, and other structured content has received particular interest in recent times due to the advances made in transformer-based language models @xcite @xcite . there is a growing need for developing automated systems to generate insights and make decisions with information in these sources. along with quantitative data, these sources also have textual content that play a crucial role in tasks such as retrieval from data catalogs @xcite , question answering @xcite @xcite , and automating business intelligence tasks @xcite . the availability of large pretrained language models using transformers has enabled researchers to encode text within relational tables in many ways. in one approach, existing works @xcite on question answering over relational tables build a pseudo-sentence by concatenating row/column entries, for example, which corresponds to the first column and the first row respectively in consider the pseudo-sentence [s3] which is built by randomly shuffling the values in [s1], if we follow above strategy, the final representation for [s1] and [s3] will be different. this would be sub optimal, since it would be at odds with the tabular structure where rows and columns may be shuffled without losing semantic meaning. another approach used to encode text within a relational table is introduced by @xcite . here, the authors employ table metadata and the row-column structure to constrain self-attention only over structurally related elements. for example, the mention brad cathers in in this paper, we propose a permutation invariant position encoding strategy that we call pi strategy, which can be used with existing transformer-based language models. the pi strategy uses the pseudosentence construction approach while allowing the model to be insensitive to the ordering of cells within a row or column. we evaluate our approach on three relational table interpretation tasks: column type annotation, column relation linking, and cell entity linking. compared to existing state-ofthe-art approaches, we show that: ## related work the task of modeling set-input problems, i.e., data instances that behave as a set rather than a sequence, using neural networks has been slowly gaining traction. recent works such as @xcite ; @xcite propose a three step strategy. first, each set element is encoded independently to a fixed-size embedding. the second step comprises of a commutative pooling operation, and finally, the pooled embedding is processed through a non-linear layer. because each set element is processed independently, information regarding possible interactions between the set elements has to be necessarily discarded @xcite . @xcite handle set inputs by pooling them via a weighted average operation with weights computed via an attention mechanism. @xcite employ a similar strategy for multi-view 3d reconstruction, where a dot-product attention operator is used for weighted average pooling. @xcite use attention-based weighted sum-pooling for multiple-instance learning. @xcite propose the relational network, an architecture that sum-pools all pairwise interactions between set elements, but not the higher order interactions. compared to these approaches, ours uses multi-head attention for aggregation. also, our approach uses stacks of selfattention modules within the transformer encoder, enabling us to model higher-order interactions between textual instances in a table row/column. @xcite introduce set transformer, a novel transformer architecture, wherein the encoder module does not contain positional embedding and drop-out features. such an approach is not apt for encoding rows/columns within a table because the ordering of individual tokens within a particular cell value must still be encoded. in the area of the representation learning strategies on tables, recent works such as tapas @xcite , and tabert @xcite encode the natural language question, query table pair jointly, and employ a transformer encoder model to assist in semantic parsing or question answering (qa) over tables. tapas proposes multiple additional embeddings to encode the entire tabular structure. however, none of these embeddings is designed to encode permutation invariance. for example, the position embeddings in tapas are monotonically increasing from left to right, thereby encoding a sense of ordering across the cell mentions. moreover, the row id and the column id embeddings within tapas impart a sense of ordering across the rows and columns. another table encoding model mate @xcite is still vulnerable to row and column perturbations. on the other hand, understanding relational tables by mapping them to entities, types, and relations in a semantically rich knowledge base (kb) is a well-studied problem @xcite . the main table interpretation tasks involve table cell linking, column type annotation, and relation extraction between column pairs. recent works such as tcn @xcite and doduo @xcite employ multi-task training objective for jointly learning column types and relation labels. tabbie @xcite uses figure 1 : a graphical representation of our permutation invariant position encoding strategy. given a query column with a header we first construct a pseudo sentence, and then tokenize it. while standard transformer-based language models such as bert, albert, gpt2, etc. assign position ids from left-to-right (default approach), our approach assigns them in a piece-wise monotonically increasing manner. turl @xcite introduces a structureaware transformer that employs a pretraining/finetuning strategy for the table understanding tasks. in contrast, our approach does not require additional pretraining. therefore it saves upon the computational overhead associated with pretraining. furthermore, we also show that our method performs better compared to turl on the table interpretation tasks. currently, turl is the published stateof-the-art on the three table interpretation tasks and tcn @xcite improves on turl for a subset of tasks with data on both column type annotation and relation extraction. ## permutation invariance in this section, we first describe our permutationinvariant position encoding strategy, and then explain how it ensures permutation invariance. figure 1 illustrates a graphical representation of our encoding strategy for a given query column. current approaches @xcite @xcite use absolute position ids from left-to-right as indicated by the "default approach". in contrast, we propose an alternative strategy for assigning position ids that work as follows, once we build the position ids for a given pseudo-sentence ψ according to our strategy, we use it to encode ψ via a transformer-based language model. a transformer based language model typically consists of the encoder module of a transformer, which generally takes in the following as inputs, a) sub-word embeddings for each subword within the tokenized sentence, b) token type embeddings for each subword token, and c) position embeddings for each subword token. the position embeddings are responsible for assigning an ordering to the sequence of sub-word tokens since a transformer encoder module is unaware of a token's position. this ordering influences the context of the subword token. these three embeddings are added for each sub-word token in ψ, the transformer encoder then processes the resulting embedding sequence. following the strategy used by @xcite to fine-tune on glue benchmark, we consider the [cls] vector as the aggregated representation of the query column. for a given column c, let ψ 1 and ψ 2 denote two pseudo sentences due to two independent orderings of cell values. under our encoding strategy, we observe that the vector corresponding to any token t (in either ψ 1 or ψ 2 ) attends foot_0 over the same set of other token vectors regardless of cell ordering. at each layer of the transformer encoder, both ψ 1 and ψ 2 generate the same set of intermediate vectors, just with a different ordering. moreover, the first vector in either ordering corresponds to the [cls] token and is equal for both ψ 1 and ψ 2 . therefore, the [cls] vector at the final layer, which we refer to as our column representation, remains the same for ψ 1 and ψ 2 , thereby yielding a permutation invariant encoding. our encoding strategy ensures that all the cell values are positionally equidistant to the [cls] token and expected to have comparable impact on its encoding. consider the example column in figure 1 . this column contains four cell values yielding twentyfour permutations in total. for each permutation, we computed its column representation as described above. finally, we calculated the variance of all possible l2 distances between any two column representations. the variance measure for our proposed permutation invariant position encoding strategy was 1.5e-12, whereas, for the default approach, this value was 0.41. this indicates that our proposed approach yields embeddings that are invariant to the ordering of the cell-values. ## tasks we first define the table interpretation tasks and then describe our approach using the pi strategy. ## evaluation to be comparable to turl, we implement our pi strategy using a tinybert @xcite model. for comparing our approach against existing state-of-the-art methods, we use the datasets released by @xcite . this paper introduces three datasets, one each for column type annotation (ct), relation extraction (re), and entity linking (el) tasks. we refer to them as wt-turl-{ct,re,el} respectively in this work. for the column type annotation task, we use two additional small scale datasets, namely, t2d foot_2 and efthymiou. we follow the same experimental setting as introduced in @xcite . for this task, the type labels belong to freebase for the wt-turl-ct dataset and dbpedia for the t2d and efthymiou datasets. for the zero-shot column type annotation (zs-ct) task, we rearranged the wt-turl-ct dataset into a new dataset called the zs-wt-turl-ct dataset. we ensure that individual columns within the train, valid, and test folds do not overlap within this new dataset. for evaluation, we use the trained model to generate a ranked list l c of type labels for a given column q c and evaluate based on the mean reciprocal rank (mrr) and hits@1 scoring metrics. as q c can have more than one true label, we use the best-ranked label for evaluation. for the wt-turl-el dataset, we use the same set of candidates as used by @xcite 4 . this dataset only contains instances with the ground truth label within the candidate list. therefore, we focus on entity disambiguation only and use mrr and hits@1 metrics for evaluation. additionally, for the t2d dataset @xcite , we do not train a separate model. rather we use our model trained on the wt-turl-el dataset and apply it to t2d's test set. moreover, following @xcite , we use two different experimental settings, i.e., a) using entity mentions only, and b) using extra metadata. ## analysis: encoding the entire table in this section, we compare our column-centric approach against doduo @xcite on the ct task. the doduo model encodes the entire table and implements a multi-task learning objective for jointly learning column types and relations between column pairs. the dosolo model encodes the entire table, whereas the dosolo † model encodes the query column only. in comparison, our approach is column-centric, i.e., it does not encode the entire table. moreover, unlike doduo, the last three rows in ## analysis: ablation tests the results indicate that removing the position id parameter altogether performs the same as the default approach. we expected it to be the case, since there is no notion of an ordering between cell-values across a single column. nevertheless, using our proposed permutation invariant strategy results in the best overall performance. we also ran additional experiments wherein we created five random permutations of the test set corresponding to the supervised ct task. within each newly created test set, the cell values within each column were randomly permuted and then scored against both trained default tinybert and tiny-bert+pi models. compared to a constant macro f1 score of 0.832 for each of the five tries by the tinybert+pi model, the default tinybert model had a macro f1 score of 0.81 with a standard devi- the experimental settings, hyper-parameters for our experiments, error analysis, and additional ablation experiments are described in the appendix. ## conclusion this paper introduces a permutation invariant position encoding strategy for transformer-based language models that can encode a set of textual instances independent of its ordering. we argued that such a strategy generates meaningful representations for the rows/columns in a relational table. we show that our proposed method is flexible enough to apply to many downstream tasks. unlike existing state-of-the-art approaches on table interpretation tasks, our method yields models that do not require additional pretraining yet achieve a new state-of-the-art on almost all the benchmarks. bert-large models ( as we move from tinybert to bert-base to bert-large, we observe that the number of trainable parameters increases by an overall factor of 24x (comparing tinybert to bert-large). this increase in the model size is why we believe that the performance improvement due to the pi strategy decreases from 2% (for tinybert) to 0.1% (for bert-large). or, in other words, bert-large has too many parameters that the learner can finetune, which effectively offsets any gain in performance obtained due to the pi strategy. on the other hand, training bert-large is quite cumbersome, i.e., it takes at least one order of magnitude longer to train compared to tinybert and is computationally (and environmentally) expensive. since tinybert is smaller in size, faster in inference, and competitive in performance compared to bert-base, a higher performance improvement due to the usage of pi strategy is highly significant in the context of building and deploying practical systems on real-world use cases.