F a BULOUS : Fact-checking Based on Understanding of Language Over Unstructured and Structured information (2021)

Abstract: As part of the FEVEROUS shared task, we developed a robust and finely tuned architecture to handle the joint retrieval and entailment on text data as well as structured data like tables. We proposed two training schemes to tackle the hurdles inherent to multi-hop multi-modal datasets. The first one allows having a robust retrieval of full evidence sets, while the second one enables entailment to take full advantage of noisy evidence inputs. In addition, our work has revealed important insights and potential avenue of research for future improvement on this kind of dataset. In preliminary evaluation on the FEVEROUS shared task test set, our system achieves 0.271 FEVEROUS score, with 0.4258 evidence recall and 0.5607 entailment accuracy.

## introduction in the past year, concerns about the spread of misinformation have risen dramatically. around 54% of people claim that they have seen fake news on covid-19 in a week (reuters institute for the study of journalism) resulting in a global lack of trust in our political institutions and the media. considering the amount of data there is on the internet, the need for a trustworthy, efficient, automated fact-checking tool is undeniable. research has made great stride in automating fact verification, driven for instance by the successive fever datasets and challenges @xcite , for which new techniques have been developed for both retrieval @xcite and entailment @xcite . the retrieval part has been handled through hard sample mining and an adjusted embedding based model training. this allows very fast inference, leveraging approximate nearest neighbors searches on large dimension embeddings. given the dataset size, this is necessary to * equal contribution achieve fast enough running times. on the entailment end, one of the state-of-the-art approaches is called kgat @xcite . this method leverages graph structure over the evidences to allow for both more accurate entailment as well as provide interpretability to the result. however, this method was designed for text, and is therefore not suitable for out of the box handling of structured data. this is where our interest can go towards methods like tapas @xcite , which is suitable for structured data. it was developed for tabfact @xcite , on which it exhibits excellent performance. however, while the previous fever challenges @xcite , and other well-known reference datasets like hot-potqa @xcite only focused on text, the feverous dataset introduced multi-modality through added structured data. this brings the dataset closer to real-world data and the task closer to fact verification in a practical setting. many difficulties inherently arise from the diverse nature of the data, and we want to shed some light on both the data and how to reason on it. our method achieved the top result on the feverous shared task and gave us insights on how to progress further by exploring the nature of the data and where our pipeline fails. we also developed the reinforced adaptive retrieval embedding (rare) and noisy entailment through adapted training (neat) methods to address certain hurdles inside the training itself. in this paper, we aim to share both the insights we gained on the feverous dataset @xcite during the workshop competition, as well as how we designed the top scoring pipeline of the competition. we will first present the dataset, what we learned from it, as well as the issues we encountered while designing an algorithm for it and how we devised our approach. in the second section, we will explain how our pipeline works and the tweaks we made to the classical pipeline defined by the baseline method @xcite . following, we will present the unique training methods that were used to ensure both stable predictions as well as state-of-the-art performance in both retrieval and entailment. in short, our contributions with this paper are the following: ## dataset insights the dataset has been designed to introduce multimodality @xcite . this has been done by adding structured data, i.e. text data which has an underlying structure to support it, like tables and lists. as we say one of these clues is not like the others, and we will explain what make cells so unlike sentences, as well the particularities involving the label annotation philosophy. the difficulty with those is that their entries are not self-contained. indeed, the value of a cell in a table depends on other values of the table. for instance, in in order to understand the value of a given cell, you might need to access values of headers and other values of the same row/column. this makes the retrieval particularly hard as it might miss the value or the header which are needed for the full set to be retrieved. in addition, linearising the table might not be enough, as the header to retrieve might be on a different row altogether. this could also cause oversampling of a certain row or column. this issue of context can be solved more easily for entailment as you can keep the extra information present of all headers available. indeed, we can reconstruct the full table from the retrieved cells for entailment, as long as the model for entailment allows dealing with the full table size. as it was the case with the tapas (eisenschlos et al., 2020) method we used for entailment, one important part might be to find the right position embedding space to encode that structure. another peculiar aspect of the data that proved hard to tackle was the handling of refuting evidence sets. we have found that often refuting relies on small details which have been changed. the issue with this is that training for retrieval will most of the time converge towards trying to get as much information out of the selected evidence as possible. it will therefore not rank the small information overlap of the evidence refuting as high as needed for the evidence retrieval and as a consequence not get the full accurate result. an iterative approach to build up supporting, refuting and non relevant evidence sets might prove successful to address this. as described in figure 1 , we would build a set of supporting evidence up until we verify everything, find a refuting evidence or reach a termination heuristic, and then re-rank evidences properly based on their entailment usefulness. this paradigm would fit the way the dataset has been annotated. the main drawback to this is the need to entangle the entailment and retrieval in one model that may not scale with data as easily as the embedding-based retrieval models that rely on cosine similarity. same goes for not enough information (nei) examples, where the piece of information missing might be very small. hence making the difference between "support" and "nei" is hard for classical approaches. an example of this can be found in appendix a where the claim is not completely covered by supporting evidences and therefore does not count as supported. in the setup of figure 1 , nei labels would appear naturally from not verifying the full claim. finally, it is noteworthy to point out that the document retrieval is relying heavily on name entities, which makes bm25 quite good at it. an easy way to improve it would be to disambiguate the nicknames, abbreviations and the likes to make it more able to find certain pages. ## our pipeline as depicted in figure 2 , for the full pipeline, we first perform document retrieval, then passage retrieval and finally entailment. on a single gpu machine described in appendix b, inference takes figure 1 : example of an iterative retrieval and entailment process. ideally we would take the candidates from retrieval, eliminate the non-supporting passages, and retrieve again, à la multi-hop dense retriever @xcite . organisation carrots/day children/year hogwarts 0.5 0.98 phoenix order 0.98 0.5 for each claim to verify, we first retrieve documents using bm25 and filter the tables of those documents to only keep the most relevant ones according to bm25. from these documents we can extract sentence and cell passages, on which, in turn, we run our passage retriever to get the most relevant passages. using input normalisation techniques, we feed this to the ensemble entailer model which predicts the entailment class. this full pipeline achieves a significant improvement of 53% improvement over the feverous baseline @xcite . we will detail in the next subsections each sub-part of the pipeline with its respective performance. ## training methodology in this section, we will give additional detail on our training methodology and how it helped us achieve state-of-the-art performance on the feverous dataset. ## conclusion overall, this multi-modal task creates a plethora of new challenges to overcome and opens exciting avenues of research for the future of automated fact verification. our design improved on the classical retrievethen-entail approach by giving special treatment to both modalities before putting them in a common multi-hop entailment model, that is necessary for multi-hop entailment. thereby, we reached improvement of the state-of-the-art for that specific task. through novel and adapted training schemes, we overcame certain challenges like maximising the full set evidence recall, and allowing to have entailment work with a fixed number of evidence, discarding irrelevant pieces of evidence for its decision. this paradigm allows us to re-think the future of entailment on this type of real-world dataset, so as to see it as a selection mechanism too. a direct consequence of this is the belief that doing entailment in an interpretable way will boost both the evidence recall and accuracy. indeed, using the interpretation will allow deriving the passages that were useful for entailment and not those that were matching the most in terms of information overlap. this might prove to be the only way to get the right evidence set even when the refuting evidence set only deals with a small detail of the claim. the way the dataset is constructed gives rise to the idea of an iterative process that starts with finding evidence, then performing entailment and then going back to retrieve more. such an approach would provide a more natural way of distinguishing between nei and support labels and allow for reasonable runtimes. ultimately, this will be accompanied by both more interpretable entailment and better scalability. this will in turn make the algorithm even more useful in the real world as a tool against misinformation and a general researching tool for fact verification. finally, a model that can deal with multiple modalities more universally and scalably is a question left for the future, making room for new research possibilities. those intuitions are entailed by both the insights we gained on this new dataset, and the fact that our best effort with a more classic approach could not yield human-level results on this complex problem.