M eta S um P erceiver: Multimodal Multi-Document Evidence Summarization for Fact-Checking (2024)

Abstract: Fact-checking real-world claims often requires reviewing multiple multimodal documents in order to assess the claim’s truthfulness, a highly laborious and time-consuming task. In this paper, we present a summarization model crafted to generate claim-specific summaries useful for fact-checking from multimodal multi-document datasets. The model takes inputs in the form of documents, images, and a claim, with the objective of assisting in fact-checking tasks. We introduce a dynamic perceiver-based model that is able to handle inputs from multiple modalities of arbitrary lengths. To train our model, we leverage a novel reinforcement learning-based entailment objective in order to generate summaries that provide evidence distinguishing between different truthfulness labels. To assess the efficacy of our approach, we conduct experiments on both an existing benchmark as well as a new dataset of multi-document claims which we contribute. Our approach outperforms the SOTA approach by 4.6% in the claim verification task on the MOCHEG dataset and demonstrates strong performance on our new Multi-News-Fact-Checking dataset.

## introduction fact-checking claims on social media platforms poses a significant challenge due to the large volume of new claims constantly being posted without sufficient methods for verification @xcite . research indicates that manually verifying all aspects of a 200-word claim can require up to four hours of dedicated effort @xcite . further, despite the exceptional capabilities of large language models (llms) in natural language processing tasks, they still generate content with factual errors. given that llms can produce convincing statements and thus influence beliefs, the potential for hallucinations poses a serious risk of misleading users when deployed for fact-checking @xcite ## multi-news-fact-checking dataset to train our system, we need a dataset of claims whose facts are drawn from multiple documents along with the entailment label of each claim. we build our dataset on top of the multi-news summarization dataset @xcite , which con-tains sets of multiple text documents along with human-written summaries of each set. because the multi-news dataset lacks claims specifically tailored for fact-checking tasks, we prompt llama 2 @xcite to generate labeled claims from each set of documents. within each group of multi-news documents, we leverage the humanwritten multi-document summary to generate 30 claims (ten of each entailment type), resulting in a dataset of 1,291,168 labeled claims. the specific prompts contain sections containing a task description, example, and instructions, which are fully detailed in the appendix 9.1. our dataset contains 111,905 images we obtained by retrieving images from the original articles. to assess the quality and effectiveness of claim generation, we conducted an evaluation using a scale from 1 to 5, where 1 indicates low quality and 5 indicates high quality. we tested 60 claims and obtained an average score of 3.61 for claim generation quality and effectiveness. in comparison, the claims generated from primera summaries scored an average of 3.21. to ensure impartiality, the annotators were blinded and asked to rate the claims alongside their respective articles and summaries. additionally, in to assess the checkworthiness of our generated claims (i.e., to ensure that they are factual claims worth fact-checking), we use a pre-trained model trained on the claimbuster dataset @xcite , as illustrated in ## approach in this section, we explain the details of our approach, msp as illustrated in figure 2 . we also describe the preprocessing steps for both text and image data, the components of our model, and the reinforcement learning methodology we applied to train msp. our approach is capable of summarizing multiple multimodal documents consisting of arbitrarily long texts and images. specifically, we use x c , x d , and x i to represent embeddings for claims, documents, and images, respectively. ## conclusion we present metasumperceiver (msp), a summarization model crafted to generate concise and informative summaries specifically tailored for factchecking claims in intricate multimodal datasets. the model's adaptable architecture can handle varying numbers of documents and input types, encompassing documents, images, and claims, by leveraging a perceiver-based design. we train our model using a rl approach, aiming to produce summaries that are instrumental in verifying the accuracy of claims. moreover, our reward function is designed to generate more concise and clear summaries, aiding in the verification of diverse claims with the assistance of the lm critic. our experimental assessments on the mocheg and our multi-news-fact-checking datasets highlight msp's robust performance in claim verification and explanation generation tasks and demonstrate its effectiveness in real-world fact-checking scenarios. this contribution underscores msp's potential to streamline fact-checking processes in today's multimodal information landscape. finally, we release the publicly accessible multi-news-fact-checking dataset, aimed at assisting researchers in developing multi-document fact-checking methods.