Joint Verification and Reranking for Open Fact Checking Over Tables (2021)

Abstract: Structured information is an important knowledge source for automatic verification of factual claims. Nevertheless, the majority of existing research into this task has focused on textual data, and the few recent inquiries into structured data have been for the closed-domain setting where appropriate evidence for each claim is assumed to have already been retrieved. In this paper, we investigate verification over structured data in the open-domain setting, introducing a joint reranking-and-verification model which fuses evidence documents in the verification component. Our open-domain model achieves performance comparable to the closed-domain state-of-the-art on the TabFact dataset, and demonstrates performance gains from the inclusion of multiple tables as well as a significant improvement over a heuristic retrieval baseline.

## introduction verifying whether a given fact coheres with a trusted body of knowledge is a fundamental problem in nlp, with important applications to automated fact checking @xcite and other tasks in computational journalism @xcite . despite extensive investigation of the problem under different conditions including entailment and natural language inference @xcite as well as claim verification @xcite @xcite , relatively little attention has been devoted to the setting where the trusted body of evidence is structured in nature -that is, where it consists of tabular or graph-structured data. recently, two datasets were introduced for claim verification over tables @xcite . in both datasets, claims can be verified figure 1 : example query to be evaluated against two retrieved tables. named entities represent a strong baseline for retrieval, but ultimately a more complex model is required to distinguish highly similar tables. here, we investigate fact verification over tables in the open setting. we take inspiration from similar work on unstructured data @xcite @xcite , proposing a two-step model which combines ad-hoc retrieval with a neural reader. drawing on preliminary work in open question answering over tables @xcite , we perform retrieval based on simple heuristic modeling of individual table cells. we combine this retriever with a roberta-based @xcite joint reranking-and-verification model, performing ## open-domain table fact verification formally, the open table fact verification problem can be described as follows. given a claim q and a collection of tables t , the task is to determine whether q is true or false. as such, we approach the task by modeling a binary verdict variable v as p(v|q, t ). this is in contrast to the closed setting, where a single table t @xmath0, and the task is to model p(v|q, t q ). since there are large available datasets for the closed setting @xcite , it is reasonable to expect to exploit t q during training; however, at test time, this information may not be available. we follow a two-step methodology that is often adopted in open-domain setting for unstructed data @xcite @xcite to our setting. namely, given a claim query q, we retrieve a set of evidence tables d q ⊂ t (section 3), and subsequently model p(v|q, d q ) in place of p(v|q, t ) (section 4). ## entity-based retrieval we first design a strategy for retrieving an appropriate subset of evidence tables for a given query. for question answering over tables, @xcite demonstrated strong performance on retrieving relevant tables using entity linking information, following the intuition that many table cells contain entities. we take inspiration from these results. in their setting, claim entities are linked to freebase entities, and string matching on the alias list is used to map entities to cells. to avoid reliance on a knowledge graph, we instead use only the textual string from the claim to represent entities, and perform approximate matching through dot products of bi-and tri-gram tf-idf vectors. we pre-compute bi-and trigram tf-idf vectors z(c foot_0 t ), ..., z(c m t ) for every table @xmath1, ..., c m t . then, we identify the named entities e 1 q , ..., e n q within the query q. for our experiments, we use the named entity spans for tabfact, provided by @xcite as part of their lpa-model. 1 we compute bi-and trigram tf-idf vectors z(e 1 q ), ..., z(e n q ) for the surface forms of those entities. to retrieve d q given q, we then score every @xmath2. since we are approximating entity linking between claim entities and cells, we let the score between an entity and a table be the best match between that entity and any cell in the table. that is: in other words, we compute for every entity the best match in the table, and score the table as the sum over the best matches. to construct the set of evidence tables d q , we then retrieve the top-k highest scoring tables. our choice to use bi-and tri-gram tf-idf as the retrieval strategy was determined empirically -see section 5.1 and ## neural verification to model p(v|q, d q ), we employ a robertabased @xcite late fusion strategy (see figure 2 for a diagram of our model). given a query q with a ranked list of k retrieved tables d @xmath3, ..., d k q ), we begin by linearising each table. our linearisation scheme follows @xcite . we first perform sub-table selection by excluding columns not linked to entities in the query. here, we reuse the entity linking obtained during the retrieval step (see section 3), and retain only the three columns in which cells received the highest retrieval scores. we linearise each row separately, encoding entries and table headers. suppose r is a row with cell entries c 1 , c 2 , ..., c m in a table, where the corresponding column headers are we construct a final linearisation l q,t for each query-table pair q, t by prepending the query to the filtered table linearisation. we then encode each l q,t with roberta, and obtain a contextualised embedding f (d k q ) ∈ r n for every table by using the final-layer embedding of the cls-token. we construct the sequence of embeddings f (d 1 q ), ...f (d k q ) for all k tables. when the model attempts to judge whether to rely on a given table for verification, other highlyscored tables represent useful contextual information (e.g., in the example in figure 1 , newspapers belonging to the same owner may be likely to also share political leanings). nevertheless, each table embedding f (d k q ) is functionally independent from the embeddings of the other tables. as such, contextual clues from other tables cannot be taken into account. to remedy this, we introduce a crossattention layer between all tables corresponding to the same query. we collect the embeddings f (d k q ) of each table into a tensor f (d q ). we then apply a single multi-head self-attention transformation as defined by @xcite to this tensor, and concatenate the result. that is, we compute an attention score for head h from table i to table j with query q as: we subsequently use f * (d q ), i.e. the tensor containing f * (d 1 q ), ..., f * (d k q ), for downstream predictions. we note that our approach can be viewed as an extension of the table-bert algorithm introduced by @xcite to the multi-table setting, using an attention function to fuse together the information from different tables. ## conclusion we have introduced a novel model for fact verification over large collections of tables, along with two strategies for exploiting closed-domain datasets to increase performance. our approach performs on par with the current closed-domain state of the art, with larger gains the more tables we include. when using an oracle to retrieve a reference table, our approach also represents a new closed-domain state of the art. finally, we have made an initial foray into wikipedia-scale open-domain table fact verification, demonstrating improvements from multiple tables also when using a full set of wikipedia tables as the knowledge source. our results indicate that the use of multiple tables can provide contex-tual clues to the model even when those tables do not explicitly verify or refute the claim, because they can provide evidence for the probability of the claim. this is a double-edged sword, as reliance on such clues can increase performance while also inducing biased claims of truthfulness. care will be needed in future work to disentangle the positive and negative aspects of this phenomenon.