Dual-Channel Evidence Fusion for Fact Verification over Texts and Tables (2022)

Abstract: Different from previous fact extraction and verification tasks that only consider evidence of a single format, FEVEROUS brings further challenges by extending the evidence format to both plain text and tables. Existing works convert all candidate evidence into either sentences or tables, thus often failing to fully capture the rich context in their original format from the converted evidence, let alone the context information lost during conversion. In this paper, we propose a Dual Channel Unified Format fact verification model (DCUF), which unifies various evidence into parallel streams, i.e., natural language sentences and a global evidence table, simultaneously. With carefully-designed evidence conversion and organization methods, DCUF makes the most of pre-trained table/language models to encourage each evidence piece to perform early and thorough interactions with other pieces in its original format. Experiments show that our model can make better use of existing pre-trained models to absorb evidence of two formats, thus outperforming previous works by a large margin. Our code and models are publicly available.

## introduction the task of fact extraction and verification aims to extract evidence and verify a given claim. previous efforts focus on dealing with text format evidence from unstructured documents @xcite @xcite or evidence from a single given table @xcite @xcite . recently, @xcite propose a new realistic setting, feverous, i.e., fact extraction and verification over unstructured and structured information. in feverous, models should not only extract evidence sentences/table cells from millions of passages, but also combine the evidence in different formats to verify a given claim. previous works on feverous generally convert all evidence pieces into either plain text @xcite @xcite or several tables @xcite . however, format conversions inevitably lose rich context information for the converted evidence, thus may mislead the subsequent encoding and interaction steps. for example, in figure 1 , the entire top two rows are indispensable to understand the table cell won. it is difficult to identify all related context cells and design a general conversion method to render them into sentences, but these connections can be easily caught by pre-trained table models @xcite . on the other side, identifying/re-organizing crucial elements in a sentence to construct a table is also challenging. simply inserting a whole sentence in a table cell @xcite will make the new cells much larger (and unique) than normal ones, thus can not make the most of general pre-trained table models @xcite as we expect. considering the inevitable expense in format conversion, we believe that each evidence in its original format can contribute necessary information to final verification, thus should be better encoded in its original format. this further indicates that we should design both sentence-to-table and table(cell)-to-sentence conversion methods to obtain all evidence in both formats, and maintain two parallel encoders to absorb the two formats, respectively. an advantage of doing so is to maximally encourage early interaction, which proves more effective than pair-wise encoding @xcite when converting table evidence into sentences, previous works either convert table cells to a concatenation of key-value pairs @xcite , or construct sentences in a coordinate-description style @xcite . they pay less attention to the conventional organization of tables structures. we observe that, in a table, the column headers usually represent the types/properties and the row headers often denote entities or scopes. we argue that one should consider these conventions to convert a table cell evidence into more natural sentences, and later pretrained language models will be able to better capture the contextualized semantics of the table cells from generated sentences. on the other hand, existing pre-trained table models are trained to analyze one table at one time, while previous evidence conversion methods produce several small tables for one instance. it would be necessary to properly organize all evidence in one table so that pre-trained table models can allow the most interactions among all evidence pieces. in this paper, we propose a dual channel unified format verification model (dcuf) to allow each evidence piece encoded in its original format and maximally maintain its original rich context, while encouraging further interactions with other evidence. dcuf converts each evidence into both textual and tabular formats in respective channels, and apply corresponding pre-trained models to learn the representations for the final verification. with the dual channel setting and carefully designed evidence conversion methods, dcuf makes better use of pre-trained language/table models to perform early and thorough interactions among all evidence and also between the claim and evidence. in summary, we make the following contribu-tions in this paper: (1) we propose dcuf, a novel model to maintain various evidence in two unified formats and allow each evidence piece to interact with other evidence in its best form. experiments show that dcuf outperforms all previous works in literature. ( 2 ) we propose a context-aware evidence conversion method that can properly organize evidence of different formats, which fits current pre-trained language/table models hence take their most advantage to obtain accurate and focused representations. ## our model the feverous task can be formalized as, given a claim q and wikipedia dump, a model is asked to find the evidence set consisting of sentences s and table cells c, and predict the veracity label of the claim accordingly. the veracity label set includes "supports", "refutes" and "not enough information". ## experiments we evaluate our models on the feverous dataset, where each claim is annotated with a gold veracity label and several gold evidence sets. any one of the evidence sets is sufficient to verify the claim. more details about the feverous dataset are in appendix. the feverous dataset provides two official metrics, namely label accuracy (acc.) and fever-ous score. label accuracy calculates the ratio of the instances whose veracity label is correctly predicted. feverous score is the ratio of the instances whose veracity label is correct and the extracted evidence set is sufficient. here, sufficient evidence sets are defined as the evidence sets covering one of the gold evidence sets provided in the feverous dataset. note that there are at most 25 table cells and 5 sentences to calculate the scores. ## analysis few-shot results figure 4 shows the fever-ous score on the development set when we train unified dcuf, dcuf without text format encoding and dcuf without table format encoding verdict predictors on 1%, 5% and 10% instances of the training set respectively. we find that, training on only 10% instances, all of the three settings outperform previous sota results, with the dual channel predictor achieving the best feverous score of 33.9%. meanwhile, with only 1% of the training set, namely, 713 instances, the unified text format predictor outperforms previous sota result by 0.9%. these improvements may contribute to our carefully-designed format conversion methods. the text format evidence converted from cells is similar to sentences in natural language when keeping much context information in the conversion procedure. and the concatenated claim-evidence string is the same as previous fact verification tasks on which the roberta checkpoint is fine-tuned. meanwhile, a given claim, and a single global evidence table are consistent to the input requirements in the pre-training step of tapas. with few new parameters introduced and an input form strictly complying with the requirements of pre-trained models and previous single format fact-checking tasks, dcuf could make the most of the pre-training stage and, thus, learn well in the few-shot setting. we also find that when training with only 1% instances, the unified text format predictor outperform other two settings. as the number of training instances increase, dual-channel predictor learns how to combine information from the two channel, thus achieving better results. ## conclusion in this paper, we propose dcuf, a dual channel unified format model for fact verification over structured and unstructured data. with context-aware evidence format conversion, dcuf gets a unified text format representation of all evidence and a global evidence table of them at the same time. the dual channel design helps us make the most of existing pre-trained language/table models to encourage all evidence pieces to interact with each other in their best forms as early as possible. experiments show that, with dual-channel unified format encoding, our proposed dcuf achieves state-ofthe-art performance and also comparable results in few-shot settings.