Table Fact Verification with Structure-Aware Transformer (2020)

Abstract: Verifying fact on semi-structured evidence like tables requires the ability to encode structural information and perform symbolic reasoning. Pre-trained language models trained on natural language could not be directly applied to encode tables, because simply linearizing tables into sequences will lose the cell alignment information. To better utilize pre-trained transformers for table representation, we propose a Structure-Aware Transformer (SAT), which injects the table structural information into the mask of the self-attention layer. A method to combine symbolic and linguistic reasoning is also explored for this task. Our method outperforms baseline with 4.93% on TabFact, a large scale table verification dataset.

## introduction table fact verification aims at classifying whether a textual hypothesis is entailed or refuted by the given table. it could benefit downstream tasks such as fake news detection, misinformation detection, etc. compared to fact verification over textual evidence @xcite , verification on semi-structured data further requires 1) the ability to encode and understand structural information of tables, and 2) the ability to perform symbolic reasoning over structured data, such as counting, comparing, and numerical calculation. although large-scale pre-trained language models @xcite achieved dominant results on textual entailment datasets @xcite , they could not be directly used to encode semi-structured data as they are pre-trained on unstructured natural language. @xcite eliminate the discrepancy by serializing tables into word sequences, and then table fact verification could be processed as a natural language inference task. the most straightforward method for table serialization is linearizing * the first two authors contribute equally to this work. the table contents via horizontal scan. however, this would destroy structural information within tables, i.e. the alignments between table cells. in figure 1 , the value "533" and "733" is meaningless digits without the column name "core clock", and it is hard for the model to recover the alignments from the flattened word sequence. therefore, table-bert @xcite includes the column name into cell representation using natural language templates during the linearization. however, comparing or counting column contents of different rows over the flattened word sequence remains a hard task, and simply duplicating the column name multiple times does not achieve satisfying results. to better utilize the transformer architecture for table representation, we propose to inject the table's structural information into the mask of the self-attention layer. figure 2 illustrates the pattern commonly adopted when human read or write a table. usually, each table row describes a record, and cell c 1,2 describes a record property with the attribute name clarified in the corresponding column name c 0,2 . besides, values of the same column are usually compared or aggregated for analysis. so, the colored row and column are most crucial to the representation of cell c 1,2 . in the long flattened sequence obtained by horizontal/vertical scan, the alignments between table cells would be disturbed by other unimportant words. to tackle this problem, we have the representation of cell c 1,2 only depend on the colored cells in figure 2 by zeroing the attention weights to other ones. figure 3 illustrates the representation of cell c 1,2 utilizing transformer. through masking, only two pseudo sentences, i.e. the corresponding row and column, that share some common words are considered in the representation of each cell. that is, the flattened word sequence is implicitly decomposed into a series of small readable sentences so as to unleashes the power of large pre-trained language model. pre-trained transformers are good at semanticlevel understanding, i.e. capturing the identical meaning between different expressions. however, one limitation is that they are not doing perfectly in symbolic reasoning @xcite . to tackle this, we perform first-order aggregation over each column and append the result as a special row into the table. an improvement of 1% is achieved, indicating that the ability of hard symbolic reasoning requires further studying. our contributions are summarized as follows: ## methodology as the examples shown in figure 1 , given a statement s, table fact verification aims to classify whether the statement is entailed or refuted by the evidence table t . the table t consists of a caption t and cells {c i,j } of m × n, where m and n are the numbers of rows and columns. since pre-trained transformer could only take word sequences as input, we feed it with a concatenation of the statement s, the [sep] token, the table caption t, and the flattened table t f . the table could be serialized by the horizontal or vertical scan. figure 3 shows an example of horizontal scanning. the representation of the word sequence follows the general encoding procedure of the pre-trained transformers @xcite , so we only describe the self-attention layer in which an attention mask is introduced for table representation. as illustrated in figure 2 , understanding the table requires both horizontal and vertical views. that is, if the table is flattened by a horizontal scan, the vertical alignment information will be lost, and vise versa. for example, the column name c 0,2 is crucial to the representation of c 1,2 , but its signal could be perturbed by other cells in grey, since all c 0, * and c 2, * cells are far from c 1,2 in the flattened sequence and are processed equally. therefore, we propose to recover the alignment information by masking signals of unimportant cells during self-attention. the attention mask @xmath0: it could be observed that if w j ∼ w i , then a i,j is reset to zero and h l j will not contribute to the representation of w i , i.e. h l+1 i . figure 3 sketches the representation learning of tokens in cell c 1,2 leveraging the masked selfattention. in the lower layers, the token representation of each cell considers information from four aspects: a) tokens of the same row that describe the same entry, b) its column title that clarifies the attribute name, c) the table caption which provides global background, and d) the statement for verification. in the upper layers, cross row attention among cells of the same column is further enabled. in this manner, lower layers focus on capturing low-level lexical information and upper layers are capable of simple cross-row reasoning. note that tokens of the statement s and the table caption receive information from all cells. another preferred ability of sat is to perform symbolic reasoning such as counting, comparing, and numerical calculation. pre-trained models like bert are good at semantic-level understanding, but not symbolic reasoning @xcite . we explore to enhance the performance of counting verification by converting the counting problem into a semantic matching problem. specifically, for every column, the frequency of duplicate cell contents is counted as a summary cell, leading to a summary row which is then appended to the table. for example, the summary cell of the second column in figure 1 is "count desktop:2", so the second refuted statement could be verified via semantic matching. ## related work to encourage the study on table fact verification, wenhu et al. (2020) construct a large scale table fact checking dataset and study two promising approaches, table-bert and latent program algorithm (lpa) respectively. table-bert transforms the problem into a natural language inference task to leverage the power of the pre-trained language models. lpa formulates the task as a program synthesis problem and it is good at symbolic reasoning. our work aligns with the direction of table-bert. inspired by existing work weijie et al. 2020); nguyen et al. (2020); dong et al. (2019); yang et al. (2019 that manipulates self-attention masks, we devise a structure-aware transformer to attain better table representation. there are several recent works that table fact verification could benefit from. @xcite and @xcite study to improve the pre-trained model in numerical reasoning and logical comparisons. the enhanced pre-trained model could be directly used in our approach. herzig et al. (2020) extend bert's architecture to encode tables for the table question answering task @xcite , where additional embeddings identifying the row and column number are added. the proposed architecture is potentially applicable to table fact checking but requires expensive pre-training. ## conclusion we propose sat to enhance the pre-trained transformer's ability on table representation by injecting structural information into the mask of selfattention layers. significant improvements on tab-fact demonstrate its effectiveness. we further enhance sat by appending a summary row to the table, the results show that it is promising to solve the fact verification that requires both symbolic reasoning and semantic understanding by feeding symbolic reasoning results into sat. overall, an improvement of 4.93% is achieved compared to the state-of-the-art method. the proposed method can further contribute to other semi-structured data (table, graph, etc.) related tasks, e.g. wikitable-questions @xcite and common-senseqa @xcite . there still exists plenty of potentials that require future studies in this direction.