Structural Encoding and Pre-training Matter: Adapting BERT for Table-Based Fact Verification (2021)

Abstract: Growing concern with online misinformation has encouraged NLP research on fact verification. Since writers often base their assertions on structured data, we focus here on verifying textual statements given evidence in tables. Starting from the Table Parsing (TAPAS) model developed for question answering (Herzig et al., 2020), we find that modeling table structure improves a language model pre-trained on unstructured text. Pre-training language models on English Wikipedia table data further improves performance. Pre-training on a question answering task with column-level cell rank information achieves the best performance. With improved pre-training and cell embeddings, this approach outperforms the state-of-the-art Numerically-aware Graph Neural Network table fact verification model (GNN-TabFact), increasing statement classification accuracy from 72.2% to 73.9% even without modeling numerical information. Incorporating numerical information with cell rankings and pre-training on a question-answering task increases accuracy to 76%. We further analyze accuracy on statements implicating single rows or multiple rows and columns of tables, on different numerical reasoning subtasks, and on generalizing to detecting errors in statements derived from the ToTTo table-to-text generation dataset.

## introduction the rapid growth in the amount and sources of online textual content has raised concerns about misinformation and its potential harmful impacts on society when quickly spread to a massive audience. for example, a study on enhancing medical education with wikipedia in 2015 found that 97% of medical students completing the survey disclosed that they found mistakes in wikipedia medical entries @xcite . concerns about mis-information have stimulated extensive research on automatic fact verification, i.e., verifying whether a given textual statement is entailed or refuted by the given evidence. figure 1 : example from the tabfact dataset: the top table contains the structured evidence; the bottom two boxes contain the statements entailed and refuted by the evidence. errors are highlighted in red. while most existing fact verification work focuses on unstructured textual evidence, fact verification with structured evidence is still underexplored. recently, @xcite introduced a new large-scale dataset, tabfact, for verifying statements based on structured evidence in tables. figure 1 presents an example table and the corresponding entailed and refuted statements from tabfact. the task of fact verification based on structured evidence is challenging in two aspects. first, traditional language models trained on unstructured text are not directly applicable to learn representations for structured text. it is difficult for such language models to understand a sentence like "round pick player college 1 1 ralph sampson virginia ..." by directly concatenating the table cells in figure 1 . second, detecting misinformation with structured evidence involves not only linguistic inference but also numerical reasoning such as addition, subtraction, sorting, and counting over records. for example, to verify the statement "ralph sampson was two picks ahead of rodney mccray", we need first to find in which order each was picked and subtract them. table representation learning is important for utilizing table data as evidence for fact verification. most existing methods apply bert @xcite model to learn table representations. table-bert @xcite uses simple templates to transform tables into "somewhat natural" sentences, and fine-tunes bert on pairs of statement sentence and corresponding table "sentence". however, this model adds many extra tokens to the tables, sometimes doubling the length of the original table token sequence. @xcite propose to first derive logical forms from the table and the statement. a heterogeneous graph is then constructed to capture connections between table cells, functions and arguments and statement tokens. a graph-enhanced contextual representation is learned for each token by only paying attention to the neighbor nodes in the graph when applying bert model. however, as we show in §4.2, bert is less effective when it is pre-trained on unstructured data but applied to structured data such as tables. table-based fact verification also requires numerical reasoning over table records. @xcite propose a latent program algorithm (lpa), where the statements are parsed into potential programs and a weakly supervised discriminator is trained to assign confidence score to each program. the output from the latent programs are aggregated or ranked according to their confidence score as the final prediction. @xcite propose to learn a representation for the program with a program-driven neural module network, where semantic compositionality is dynamically modeled along the program parsing structure in a bottomup style. the program representation and the token representation for the table, statement and linearized program are then combined to make the final prediction. however, these two models depends on weakly-supervised labels, which could be noisy, to derive potential programs. recently, the same authors of table-bert released a new model, gnn-tabfact foot_0 , applying numerical-aware graph neural networks (numgnn) on top of table-bert to learn to compare numerical cells in the same col-umn. nonetheless, it requires constructing a graph neural network and conducting iterative message passing among the nodes in the graph. we propose to adapt the table parsing (tapas) model @xcite , which has proven effective in question answering over tables, to model tables for fact verification. tapas concatenates all table cells without adding extra tokens and then extends bert's architecture with additional embedding layers to capture the table structure and numerical comparison information for each token in the table. we replace its original two top classification layers for answer generation with a single classification layer on the [cls] token to classify whether a given statement is entailed or refuted by the table. our experimental results show that with proper pre-training, tapas outpeforms the state-of-theart gnn-tabfact model, increasing the accuracy of statement classification from 72.2% to 73.9% on the tabfact test set even without modeling numerical information. by further adding ranking information for numerical rows and pre-training on the question answering task, tapas achieves 76% accuracy, with 89% on the simple statements and 69.8% on the complex statements. we also perform further analysis to examine: (1) how the numerical comparison information improves tapas's performance when ranking information is needed; structured data modeling modeling structured data is essential for multiple tasks, e.g., table classification, table population, table retrieval, question answering , data-to-text generation, and table-based fact verification. ghasemi-gol and szekely (2018), trabelsi et al. (2019) and deng et al. (2019) propose to embed tabular data into a vector space, using table structure to classify tables into different categories, adding more rows or columns to tables and retrieving tables given query keywords. @xcite employ rnns to encode cell content and cnns to encode table structure to better capture semantic features for table classification. @xcite propose to generate candidate logical forms from a question, convert logical forms to paraphrases, and rank them according to their similarity to the original question to generate the answer for the questions. @xcite add additional embedding layers to a bert model to capture table structure and numerical information, and add two classification layers to predict aggregation functions and corresponding table cells to generate an answer for a given question. much current table -to-text generation work focus on generating biographies from wikipedia infoboxes @xcite @xcite or summarizing basketball games @xcite according to the box-and line-score tables. parikh et al. (2020) release a dataset for controlled table-to-text generation, where table cells are highlighted for the target sentences. chen et al. (2020) propose a new natural language generation task, where the model tasked with generating statements that entailed by a given table. chen et al. (2019) introduce a dataset for fact verification given tabular data as evidence and propose a bert-based table-bert model and a latent program algorithm (lpa) model for this task. zhong et al. (2020) propose to first drive the program from the table and statement, and then learn graph-enhanced contextual representations for both the table tokens and the program to classify the statements. after this paper was submitted, four related papers were published, which explore additional questions in table-based fact verification. zhang et al. (2020) propose to utilize masking in the self attention layer to model table structure. shi et al. (2020) and yang et al. (2020) explore how to effectively combine both linguistic information and symbolic information for table-based fact verification. eisenschlos et al. (2020) generate synthetic datasets to pre-train a tapas model to better understand tables for downstream tasks such as table-based fact verification and question answering. ## methods we describe adapting the tapas model to fact verification over tables and then introduce different pre-trained models on which we fine-tune the tapas for the table verification task. ## experiments in this section, we first introduce the experimental settings in detail ( §4.1). then we present the results of comparing two variants of tapas model and other state-of-the-art models on the table-based fact verification task in §4.2. more discussions about the models are provided in §4.3. ## conclusions we adapted the table parsing (tapas) model, with efficient encoding of table content, structure, and numerical comparison information for the task of table-based fact verification. we compared two variant models: tapas-row-col, which models table contents and structure, and tapas-row-col-rank, which adds numerical comparison information. experiments showed that both tapas-row-col and tapas-row-col-rank outperform the state-of-the-art numerically-aware graph neural net-work model by pre-training on tabular data. we also examined how ranking information helps improve tapas's performance on superlative, comparative, sum, and count statements. models trained on different datasets were compared to study how question complexity affects model performance. we also constructed two synthetic datasets to examine the generalization of these models. we find models trained on tabfact perform well on errors arising from simple cell replacement but not on digit-editing errors. in future, we aim to extend tapas to explicitly model more numerical operations for fact verification and to correct false statements automatically given tabular evidence.