Permutation Invariant Strategy Using Transformer Encoders for Table Understanding
Abstract
Representing text in tables is essential for many business intelligence tasks such as semantic retrieval, data exploration and visualization, and question answering.
Existing methods that leverage pretrained Transformer encoders range from a simple construction of pseudosentences by concatenating text across rows or columns to complex parameter-intensive models that encode table structure and require additional pretraining.
In this work, we introduce a novel encoding strategy for Transformer encoders that preserves the critical property of permutation invariance across rows or columns.
Unlike existing state-of-the-art methods for Table Understanding, our proposed approach does not require any additional pretraining and still substantially outperforms existing methods in almost all instances.
We demonstrate the effectiveness of our proposed approach on three table interpretation tasks: column type annotation, relation extraction, and entity linking through extensive experiments on existing tabular datasets.
Introduction
Representation learning of natural language text within relational databases, spreadsheets, and other structured content has received particular interest in recent times due to the advances made in
Transformer-based language models (Devlin et al., 2019; Lan et al., 2020; Raffel et al., 2020).
There is a growing need for developing automated systems to generate insights and make decisions with information in these sources.
Along with quantitative data, these sources also have textual content that play a crucial role in tasks such as retrieval from data catalogs (Zhang and Balog, 2018), question answering (Glass et al., 2021;
Yin et al., 2020; Herzig et al., 2020), and automating business intelligence tasks (Sallam and Idoine, 2019).
The availability of large pretrained language models using Transformers has enabled researchers
[Table]
[Table] to encode text within relational tables in many ways.
In one approach, existing works (Glass et al., 2021;
Yin et al., 2020) on Question Answering over Relational Tables build a pseudo-sentence by concatenating row/column entries, for example,
[Table]
which corresponds to the first column and the first row respectively in Table 1.
Here, the tokens | and : act as delimiters.
Such pseudo-sentences are then processed through a Transformer encoder model, and the [CLS] vector at the final layer is treated as its representation (Devlin et al., 2019).
While such a strategy provides valuable contextbased information, it is sensitive to the ordering of the cell values used in the encoding process.
Consider the pseudo-sentence [S3] which is built by randomly shuffling the values in [S1] ,
[Table]
If we follow above strategy, the final representation for [S1] and [S3] will be different.
This would be sub optimal, since it would be at odds with the tabular structure where rows and columns may be shuffled without losing semantic meaning.
Another approach used to encode text within a relational table is introduced by Deng et al.
Here, the authors employ table metadata and the row-column structure to constrain self-attention only over structurally related elements.
For example, the mention Brad Cathers in Table 1 can only attend to entries within the same row/column and to the table metadata such as headers and captions.
Such constraints introduce additional table pretraining objectives for representation learning, which comes with a relatively high up-front computational cost.
These constraints and representation are also inflexible to the explicit consideration of structural relationships across rows/columns, such as relations between column-pairs or hierarchies between rows/columns that may need to be leveraged in some end tasks.
In this paper, we propose a permutation invariant position encoding strategy that we call PI Strategy, which can be used with existing Transformer-based language models.
The PI Strategy uses the pseudosentence construction approach while allowing the model to be insensitive to the ordering of cells within a row or column.
We evaluate our approach on three relational table interpretation tasks: column type annotation, column relation linking, and cell entity linking.
Compared to existing state-ofthe-art approaches, we show that:
• Our approach is less parameter intensive.
• Our approach can also adapt the pseudosentence representation to the requirements of the downstream tasks.
• Our approach without any pretraining outperforms existing state-of-the-art approaches on almost all instances.
Related Work
The task of modeling set-input problems, i.e., data instances that behave as a set rather than a sequence, using neural networks has been slowly gaining traction.
Recent works such as Edwards and Storkey (2017); Zaheer et al.
(2017) propose a three step strategy.
First, each set element is encoded independently to a fixed-size embedding.
The second step comprises of a commutative pooling operation, and finally, the pooled embedding is processed through a non-linear layer.
Because each set element is processed independently, information regarding possible interactions between the set elements has to be necessarily discarded (Lee et al., 2019).
Vinyals et al.
(2016) handle set inputs by pooling them via a weighted average operation with weights computed via an attention mechanism.
Yang et al.
(2020) employ a similar strategy for multi-view 3D reconstruction, where a dot-product attention operator is used for weighted average pooling .
Ilse et al.
(2018) use attention-based weighted sum-pooling for multiple-instance learning.
Santoro et al.
(2017) propose the relational network, an architecture that sum-pools all pairwise interactions between set elements, but not the higher order interactions.
Compared to these approaches, ours uses multi-head attention for aggregation.
Also, our approach uses stacks of selfattention modules within the Transformer encoder, enabling us to model higher-order interactions between textual instances in a table row/column.
Lee et al.
(2019) introduce Set transformer, a novel Transformer architecture, wherein the encoder module does not contain positional embedding and drop-out features.
Such an approach is not apt for encoding rows/columns within a table because the ordering of individual tokens within a particular cell value must still be encoded.
In the area of the representation learning strategies on tables, recent works such as TAPAS (Herzig et al., 2020), and TaBERT (Yin et al., 2020) encode the natural language question, query table pair jointly, and employ a transformer encoder model to assist in semantic parsing or question answering (QA) over tables.
TAPAS proposes multiple additional embeddings to encode the entire tabular structure.
However, none of these embeddings is designed to encode permutation invariance.
For example, the position embeddings in TAPAS are monotonically increasing from left to right, thereby encoding a sense of ordering across the cell mentions.
Moreover, the Row ID and the Column ID embeddings within TAPAS impart a sense of ordering across the rows and columns.
Another table encoding model MATE (Eisenschlos et al., 2021) is still vulnerable to row and column perturbations.
On the other hand, understanding relational tables by mapping them to entities, types, and relations in a semantically rich knowledge base (KB) is a well-studied problem (Zhang and Balog, 2020;
Ritze et al., 2015).
The main table interpretation tasks involve table cell linking, column type annotation, and relation extraction between column pairs.
Recent works such as TCN (Wang et al., 2021) and
DODUO (Suhara et al., 2021) employ multi-task training objective for jointly learning column types and relation labels.
TABBIE (Iida et al., 2021) uses two transformers, one to encode rows and the other to encode columns, and employs a corrupt cell detection objective function for pretraining.
TABBIE was pretrained using eight v100 GPUs for a week.
TURL (Deng et al., 2020) introduces a structureaware Transformer that employs a pretraining/finetuning strategy for the table understanding tasks.
In contrast, our approach does not require additional pretraining.
Therefore it saves upon the computational overhead associated with pretraining.
Furthermore, we also show that our method performs better compared to TURL on the table interpretation tasks.
Currently, TURL is the published stateof-the-art on the three table interpretation tasks and
TCN (Wang et al., 2021) improves on TURL for a subset of tasks with data on both column type annotation and relation extraction.
Permutation Invariance
In this section, we first describe our permutationinvariant position encoding strategy, and then explain how it ensures permutation invariance.
Figure 1 illustrates a graphical representation of our encoding strategy for a given query column.
Current approaches (Devlin et al., 2019; Lan et al., 2020;
Brown et al., 2020) use absolute position IDs from left-to-right as indicated by the “Default Approach".
In contrast, we propose an alternative strategy for assigning position IDs that work as follows,
• We introduce two special tokens [d] and [|] .
As shown in Figure 1, the [|] token acts as a delimiter between the cell values in a column, whereas the [d] separates the column header from the column values.
Both [d] and [|] are initialized at random, and their representations are learned during training.
• Starting from a position ID of zero for the [CLS] token, we increment by one till we reach the [d] token.
This assigns position IDs zero through two for this example.
• If the position ID assigned to the [d] token is n , then for all tokenized cell values, we start assigning position IDs from n + 1 onwards, in a left-to-right manner.
• In this example, the cell values lake laberge (four tokens) gets position IDs of three to six , whereas Riverdale North and Porter Creek Centre (three tokens each) gets position IDs of three to five .
The cell value Klondike with two tokens gets position ID of three and four .
• If m is the max value of position ID assigned after all cells are processed, then all the [|] tokens get a position ID of m + 1 ,
i
e., seven for this example.
• Finally, the [SEP] token gets a position ID of m + 2 , i.e. a value of eight in this example.
Once we build the position IDs for a given pseudo-sentence ψ according to our strategy, we use it to encode ψ via a Transformer-based language model.
A Transformer based language model typically consists of the encoder module of a transformer, which generally takes in the following as inputs, a ) Sub-word embeddings for each sub– word within the tokenized sentence, b )
Token type embeddings for each subword token, and c ) Position Embeddings for each subword token.
The position embeddings are responsible for assigning an ordering to the sequence of sub-word tokens since a transformer encoder module is unaware of a token’s position.
This ordering influences the context of the subword token.
These three embeddings are added for each sub-word token in ψ , the transformer encoder then processes the resulting embedding sequence.
Following the strategy used by Devlin et al.
(2019) to fine-tune on GLUE benchmark, we consider the [CLS] vector as the aggregated representation of the query column.
For a given column c , let ψ 1 and ψ 2 denote two pseudo sentences due to two independent orderings of cell values.
Under our encoding strategy, we observe that the vector corresponding to any token t (in either ψ 1 or ψ 2 ) attends 1 over the same set of other token vectors regardless of cell ordering.
At each layer of the transformer encoder, both ψ 1 and ψ 2 generate the same set of intermediate vectors, just with a different ordering.
Moreover, the first vector in either ordering corresponds to the [CLS] token and is equal for both ψ 1 and ψ 2 .
Therefore, the [CLS] vector at the final layer, which we refer to as our column representation, remains the same for ψ 1 and ψ 2 , thereby yielding a permutation invariant encoding.
Our encoding strategy ensures that all the cell values are positionally equidistant to the [CLS] token and expected to have comparable impact on its encoding.
Consider the example column in Figure 1. This column contains four cell values yielding twentyfour permutations in total. For each permutation, we computed its column representation as described above. Finally, we calculated the variance of all possible l2 distances between any two column representations.
The variance measure for our proposed permutation invariant position encoding strategy was 1.5e-12, whereas, for the default approach, this value was 0.41.
This indicates that our proposed approach yields embeddings that are invariant to the ordering of the cell-values.
We first define the table interpretation tasks and then describe our approach using the PI Strategy.
4. Column Type Annotation
The column type annotation task is defined as,
Definition 1.
Given a table T and a set of semantic types L , the column type annotation task refers to the task of annotating a column c ∈ T with a type label l ∈ L so that all entities in c have the type l .
Note that a column can have multiple type labels.
Column type annotation is a crucial task for Table understanding and can provide vital information in many downstream tasks, such as Question
Answering over Tables, Knowledge Discovery, etc.
Earlier works (Mulwad et al., 2010; Ritze et al., 2015; Zhang, 2017) often coupled this task together with entity linking.
The entities within a column were first linked to a KB, and then a majority voting strategy was employed on the types of the linked entities.
Recently, Chen et al.
(2019a,b); Hulsebos et al.
; Deng et al.
(2020) have studied this task based on only the available information in a given table without doing entity linking first.
In this work, we adopt a similar setting and compare against Deng et al.
(2020), which is the current state of the art on this task.
Our Approach.
Given a column c , for example, let’s say the second column in Table 1, we encode it via the PI Strategy to obtain column vector h .
The probability of predicting the class label l is,
[Equation]
We use a binary cross-entropy loss function for training.
4. Relation Extraction
The relation extraction task is defined as follows,
Definition 2.
Given a table T and a set of relations
R in a KB, for a subject-object column pair in T , the goal is to annotate it with r ∈ R , so that r holds between all the entity pairs.
Similar to the Column Type Annotation task, existing approaches Mulwad et al.
Ritze et al.
; Zhang (2017) perform entity linking first, followed by a KB lookup to obtain the list of applicable relations.
These approaches rely on the entity linking performance and assume that
the
KB in question is complete, i.e., all relations between entity pairs are present in the KB.
Such an assumption may not be true in general.
Therefore, following Deng et al.
(2020), the goal is to classify a given subject-object column pair into one or more relations, without explicitly linking table cell mentions to entities.
This is important as it allows an user to extract new knowledge from web tables for knowledge base population tasks.
Our Approach.
To predict the relation label(s) for a given column pair p, q (let’s say columns one and two in Table 1), we first concatenate the two columns to obtain a pseudo-column z , i.e. the i -th cell value of column z is the concatentation (separated by a special delimiter token [:] ) of the i -th cell values of columns p and q .
We employ two different strategies to build a model for this task, based on whether we are using entity mentions only or using extra metadata too (See Table 2 below).
For the case of entity mentions only, we encode the column z based on our
PI Strategy to obtain an unique representation h for the column pair.
Additionally, if we are using extra metadata too, then we encode columns p, q and z separately using the PI Strategy to obtain representations h p , h q , h z , and concatenate them to obtain the final representation h .
The goal of the concatenation step is to ensure that as long as the cells in columns p, q are shuffled in tandem, the overall column pair representation remains the same.
Finally, for predicting the class labels, we follow the same expression as in equation 1, and also employ binary cross-entropy loss function for training.
4. Zero shot Column Type Annotation
To further test the effectiveness of PI Strategy in generating meaningful column representations, we construct a Zero-shot column type annotation task.
To predict the type(s) for a given column in the zero-shot setting, we employ a Siamese network architecture, by using two separate transformer encoders.
Each training instance consists of a (table column, list of type labels pair), i.e., I = ( c, Y c ) where c is a particular column, and Y c is a list of type labels associated with c .
We encode column c using the PI Strategy and the first transformer encoder.
Concurrently, we encode the true type label
y ∈
Y c using the PI Strategy and the second transformer encoder to obtain the label representation h y .
Finally, we calculate the positive score s + as the dot product between h c and h y .
The negative type labels are sampled at random , and a similar process is used to calculate the negative score
s − .
We use both s + and s − , together with a binary cross-entropy loss for training.
4. Entity Linking
The entity linking task is defined as follows,
Definition 3.
Given a table T and a knowledge base K , entity linking aims to assign each potential mention of cells in T to its referent entity e ∈ K .
Entity linking is usually addressed in a two-step approach: a ) Candidate generation, and b ) Entity disambiguation.
The Candidate generation module proposes a set of potential candidates, whereas the
Entity disambiguation module is a re-ranking step that selects the best candidate entity matching the cell mention for a given table.
Recent methods for table entity linking include
T2K (Ritze et al., 2015) that uses an iterative matching approach, combining schema and entity matching; Hybrid II (Efthymiou et al., 2017) that combines a lookup and entity embedding method.
Following Deng et al.
(2020), we use the same Wikidata lookup service for candidate generation and focus only on the disambiguation step.
Our approach.
Entity disambiguation is a matching problem that requires us to match a given table cell mention to a particular entity within a candidate set of entities.
We employ a bi-encoder architecture to rank the candidate set of entities for a query mention.
Given a cell mention m within a table T m , we extract the corresponding row R m , which consists of a set of column header, cell-value pairs representing different attributes of m .
Consider the mention Lake Laberge in the first row of Table 1 as an example.
We extract out this row and build the following pseudo-sentence:
[Table]
For this task, we introduce another special delimiter token [m] that separates the metadata information 2 from the rest of the pseudo sentence.
In this case, the page title , section title , and the table caption (See Table 1) constitute the metadata information.
Moreover, following Soares et al.
(2019), we introduce two additional special tokens [s] and [e] that surround the query mention.
The rest of pseudo-sentence after [d] consists of the remaining entries within R m .
This pseudo-sentence is then encoded according to our proposed PI Strategy where entries after [d] are permutationally invariant.
This generates the representation h m , which is the output vector at the final layer, corresponding to the [s] token.
On the other hand, following Deng et al.
(2020), for each candidate entity e ∈ K , we use its name
N , description D , and a set of type labels T .
We concatenate N, D together with the labels T , to obtain the pseudo-sentence, which is then encoded using PI Strategy to obtain a representation h e for the candidate entity e .
Given the mention vector h m and the candidate entity vector h e , we employ a similar training strategy as the zero-shot column type annotation task.
Evaluation
To be comparable to TURL, we implement our
PI Strategy using a TinyBERT (Jiao et al., 2020) model.
For comparing our approach against existing state-of-the-art methods, we use the datasets released by Deng et al.
This paper introduces three datasets, one each for Column Type
Annotation (CT), Relation Extraction (RE), and
Entity Linking (EL) tasks.
We refer to them as
WT-TURL-{CT,RE,EL} respectively in this work.
For the Column Type Annotation task, we use two additional small scale datasets, namely, T2Dand Efthymiou.
We follow the same experimental setting as introduced in Chen et al.
(2019b).
For this task, the type labels belong to Freebase for the WT-TURL-CT dataset and DBPedia for the T2D and Efthymiou datasets.
For the zero-shot Column Type Annotation (zsCT) Task, we rearranged the WT-TURL-CT dataset into a new dataset called the zs-WT-TURL-CT dataset.
We ensure that individual columns within the train, valid, and test folds do not overlap within this new dataset.
For evaluation, we use the trained model to generate a ranked list L c of type labels for a given column q c and evaluate based on the
Mean Reciprocal Rank (MRR) and Hits@1 scoring metrics.
As q c can have more than one true label, we use the best-ranked label for evaluation.
For the WT-TURL-EL dataset, we use the same set of candidates as used by Deng et al.
This dataset only contains instances with the ground truth label within the candidate list.
Therefore, we focus on Entity Disambiguation only and use MRR and Hits@1 metrics for evaluation.
Additionally, for the T2D dataset (Lehmberg et al., 2016), we do not train a separate model.
Rather we use our model trained on the WT-TURL-EL dataset and apply it to T2D’s test set.
Moreover, following Deng et al.
(2020), we use two different experimental settings, i.e., a ) Using
Entity Mentions Only, and b ) Using Extra metadata.
Table 2 illustrates a comparison of different data artifacts used by both TURL and our approach, in either of the experimental settings.
The dataset statistics are available in Tableof the Appendix.
Also, we will release the zs-WTTURL-CT dataset once the anonymity period ends.
[Table]
See Table 1 for an
5. Results: Column Type Annotation
Table 3 shows the results on the Column Type annotation task for the WT-TURL-CT dataset.
The first entry illustrates the performance of Sherlock (Hulsebos et al., 2019), a model that usesfeatures describing statistical properties, character distributions, word embeddings and paragraph vectors of the cell mentions in a column.
TURL (Deng et al., 2020) uses a structure-aware Transformer encoder model, and a pretraining/finetuning strategy to attain the current state of the art for this task
[Table]
The third row and the fifth row in Table 3 illustrates the performance of our approach on this task, which is an original contribution of this work.
Our approach substantially outperforms TURL on the
Macro F1, under both the settings, i [Table] e., “Only Entity Mentions" and “Extra metadata".
In contrast, for Micro F1, our approach generates a statistically significant gain of 1.4% (p-value <
1e-3 using McNemar’s test) under the former setting [Table] and achieves similar performance on the latter setting.
Next, we compare our approach to the TCN model as introduced by Wang et al.
This model learns unique representation for tables by aggregating information from both a single table and across different tables.
Moreover, it employs a supervised multi-task training objective for jointly learning column types and relations between column pairs.
In comparison, our approach does not use a multi-task training objective, and is columncentric, i.e., encodes the query column only for learning the column types.
TCN is evaluated on the intersection of the WTTURL-CT and the WT-TURL-RE datasets, i.e., columns present in both these datasets are only considered.
The intersection yields a smaller dataset having 55,318 tables with 201 type labels andrelation labels.
Table 4 illustrates the results.
[Table] Table 5 illustrates a comparison of Accuracy on T2D and Efthymiou datasets.
Following Chen et al.
(2019b), we train a single model using 70% of the T2D dataset and then score separately on both T2D and Efthymiou datasets.
We observe that our approach achieves the new state-of-the-art on
T2D, whereas on Efthymiou, it is almost similar in performance ( lags behind by 0.7%) to TURL.
On the other hand, when evaluated on entity mentions only, our approach greatly improves over TURL.
[Table]
5. Results: Zero shot Column Type Annotation
Table 6 shows the results on Zero shot Column
Type Annotation task when the choice of both the
Transformer encoder models is either a ) TinyBERT (Jiao et al., 2020), or b ) BERT-Base (Devlin et al., 2019).
In either case, we observe that using our PI
Strategy yields a substantially higher performance, compared to without it
[Table]
5. Results: Relation Extraction
Table 7 shows the results on the Relation Extraction task for the WT-TURL-RE dataset.
The first block of two rows compares the performance of TURL with our approach when only entity mentions from a given column pair are used.
In comparison, the second block illustrates relative performance when extra metadata information is also used
[Table]
model.
We observe that when using entity mentions only, our approach improves over TURL by 3.2% on the
Macro F1, whereas using additional table metadata results in an almost similar Macro F1 by both methods.
The Micro F1 remains practically identical in either case, i.e., ± 1% between both approaches.
Following Table 2, we notice that, unlike TURL, our approach does not use Page Title, Section Tile, nor Table Caption while operating under the “Extra metadata" setting, yet achieves comparable performances to TURL.
How do we incorporate this additional contextual information within our approach is something that we leave as future work.
Results: Entity Linking
Table 8 shows the results on the Entity Linking task for the WT-TURL-EL and the T2D dataset, wherein the final row illustrates the performance of our approach.
[Table]
Note that the Wikidata lookup baseline achieves a higher score than TURL on the T2D dataset, which is in line with the findings reported by Deng et al.
Nevertheless, our approach substantially outperforms over both Wikidata lookup as well as TURL, on both the datasets.
We observe that running this experiment without our Permutation Invariant (PI) strategy yields similar performance compared to using the PI strategy.
We encode the cell values across a row, unlike the
CT and RE tasks.
Each row within the WT-TURLEL dataset has roughly three textual cell values on average, as opposed to hundreds for the CT and RE tasks.
We believe that due to this small number of values to encode, we cannot maximize the efficacy of the PI strategy for this task compared to CT and
RE tasks.
Thus we conclude that the PI Strategy does not provide gains over the default approach when the number of permutation invariant cells being encoded is small.
Analysis: Encoding the entire Table
In this section, we compare our column-centric approach against DODUO (Suhara et al., 2021) on the CT task.
The DODUO model encodes the entire table and implements a multi-task learning objective for jointly learning column types and relations between column pairs.
Table 9 illustrates the comparison results.
The DOSOLO model encodes the entire table, whereas the DOSOLO † model encodes the query column only.
In comparison, our approach is column-centric, i.e., it does not encode the entire table.
Moreover, unlike DODUO, the last three rows in Table 9 are finetuned only on the CT Task. [Table]
We observe that our approach substantially outperforms DOSOLO † , wherein both these approaches are evaluated under similar experimental settings.
Moreover, our method performs slightly below DODUO, even though the latter encodes the entire table as additional context information and performs multi-task learning.
These results demonstrate the effectiveness of our approach and open up the possibility of employing a multi-task learning strategy within our method, which we leave as future work.
Analysis: Ablation Tests
Table 10 illustrates the ablation experiments performed on the supervised CT and RE tasks respectively.
The first row in the top block corresponds to using the PI Strategy for the Position IDs.
Replacing PI Strategy with the default approach (Section 3) yields the second row.
In the third row, we donot use any Position IDs akin to the SetTransformer model (Lee et al., 2019), whereas the fourth row uses a constant Position ID of zero throughout.
The results indicate that removing the Position
ID parameter altogether performs the same as the default approach.
We expected it to be the case, since there is no notion of an ordering between cell-values across a single column.
Nevertheless, using our proposed permutation invariant strategy results in the best overall performance.
We also ran additional experiments wherein we created five random permutations of the test set corresponding to the supervised CT task.
Within each newly created test set, the cell values within each column were randomly permuted and then scored against both trained default TinyBERT and TinyBERT+PI models.
Compared to a constant macro f1 score of 0.832 for each of the five tries by the
TinyBERT+PI model, the default TinyBERT model had a macro f1 score of 0.81 with a standard devi-
[Table]
ation of 8e-4. These results further reinforce our hypothesis that the performance gains are indeed due to PI. For the default approach to mirror the PI results, one needs to augment the training dataset with additional shuffled training instances. Such a strategy is not optimal as it increases the runtime complexity for training.
In comparison, the PI approach learns a consistent model regardless of the ordering of the cells within a column, thereby demonstrating its efficacy.
The experimental settings, hyper-parameters for our experiments, error analysis, and additional ablation experiments are described in the Appendix.
Conclusion
This paper introduces a Permutation Invariant position encoding strategy for transformer-based language models that can encode a set of textual instances independent of its ordering.
We argued that such a strategy generates meaningful representations for the rows/columns in a relational table.
We show that our proposed method is flexible enough to apply to many downstream tasks.
Unlike existing state-of-the-art approaches on table interpretation tasks, our method yields models that do not require additional pretraining yet achieve a new state-of-the-art on almost all the benchmarks.
A Appendix A. Hyperparameters
This section enumerates the experimental details and the hyperparameters used in all of our experiments.
For all the methods, we do a grid search over the hyper-parameter space using the validation set, and report the test set results corresponding to the best performing settings.
For choosing a threshold value to compute the Macro/Micro F1 metrics, we employ a search space of [0.05, 1.0] with a step size of 0.05.
For the learning rate, we performed a grid search over the following values {2e-5, 3e-5, 5e-5}.
For the WT-TURL dataset, the supervised CTA task uses a learning rate of 2e-5.
In contrast, the supervised RE task uses a value of 5e-5.
These tasks use a single CPU and a GPU, a batch size of 32, are evaluated for a max of 40 epochs, and have early stopping criteria of 5 epochs.
The CTA task on T2D and Efthymiou datasets follow the same experimental configuration as the WT-TURL dataset, except that it runs for a max of 100 epochs and has early stopping criteria of 20 epochs.
Similarly, the zero-shot CTA (zs-CTA) and the
Entity linking tasks also follow the same experimental configuration as the CTA task, except for the following.
The transformer models for this task use a max tokenizer length of 128.
The zs-CTA task on TinyBERT/BERT-Base uses eight/sixteen [-x
-]x
CPUs for concurrent data loading, whereas the Entity linking task uses eight x86 CPUs for the same purpose.
Moreover, the zs-CTA/Entity linking task on TinyBERT uses a single GPU for training (with a batch size of 32), whereas on BERT-Base, it uses four GPUs with a combined batch size of 24 for faster training.
We used PyTorch v1.8.1+cu102 for running our experiments, using Intel x86 CPU and Nvidia [-v
-]v
GPU machines.
All of our experiments used a random seed of 73.
A. Comparison of Model sizes
Table 11 illustrates a comparison of model sizes in terms of the number of trainable parameters of
TURL against our proposed approach.
[Table]
The TURL method uses a pretrained TinyBERT model.
It performs a pretraining step, to begin with, wherein it learns a total of 303 million parameters 5 .
Out of these, 289 million are kept fixed during finetuning.
In comparison, we do not perform additional pretraining for our approach.
Moreover, our approach requires roughly 200K less trainable parameters for fine-tuning on the CT and RE tasks.
For the EL task, we employ a siamese network architecture, therefore our approach uses roughly 2x trainable parameters compared to TURL.
However, the overall number of model parameters, involving both pre-training and fine-tuning, required by our approach is one order of magnitude less.
A. Using Larger Transformer Encoder Models
Our PI Strategy also yields a performance improvement when evaluated against BERT-Base and
BERT-Large models (Table 12). [Table]
Default
[Table]
With PI
[Table]
Params
[Table]
As we move from TinyBERT to BERT-Base to
BERT-Large, we observe that the number of trainable parameters increases by an overall factor of 24x (comparing TinyBERT to BERT-Large).
This increase in the model size is why we believe that the performance improvement due to the PI strategy decreases from 2% (for TinyBERT) to 0.1% (for
BERT-Large).
Or, in other words, BERT-Large has too many parameters that the learner can finetune, which effectively offsets any gain in performance obtained due to the PI strategy.
On the other hand, training BERT-Large is quite cumbersome, i.e., it takes at least one order of magnitude longer to train compared to TinyBERT and is computationally (and environmentally) expensive.
Since TinyBERT is smaller in size, faster in inference, and competitive in performance compared to BERT-Base, a higher performance improvement due to the usage of PI Strategy is highly significant in the context of building and deploying practical systems on real-world use cases.
A. Error Analysis
In this section, we perform error analysis for the
Column Type Annotation and the Relation Extraction tasks.
The goal is to identify the type labels that are most difficult to predict using our approach.
Note that, for this analysis, we use PI Strategy over a TinyBERT model.
In addition, we also use column headers and canonical labels corresponding to the linked entity associated with the table cell mention.
We selected the top five type labels sorted by the sum of the false positives and false negatives on the test set.
Table 13 illustrates these results.
The fine grained types such as sports.pro_athlete and government.politician in Table 13 have a
[Table]
classification error rate of 26.5% and 25.3% respectively, whereas coarse-grained types such as people.person have a 2.5% error rate.
Furthermore, out of 138 false positives for the label location.citytown , 135 of them have location.location as a true label.
On the other hand, all 89 test instances that get marked as False negatives (for location.citytown ) have location.location as one of the ground truth labels, and we can identify 74 of them correctly.
Table 14 illustrates the result of a similar analysis on the Relation Extraction task.
If we generalize these observations, we can conclude that our approach a ) Makes lot more errors on fine-grained type labels as opposed to coarse– grained labels, and b ) Confuses often between labels that are semantically related.
This behavior is expected since it is difficult for any model to resolve these issues based only on tabular data, i.e., without additional table contextual information.
As mentioned before, we leave the strategy of inculcating contextual information into our learning approach for future work.
[Table] [Table]
[Table] [Table]
CT task, the number within the brackets (for the train/dev/test folds) indicates the number of class labels available for that fold.
For the EL task, we have a total of roughly 5 million candidate entities due to Wikidata lookup across all the instances.