COVID - VTS : Fact Extraction and Verification on Short Video Platforms (2023)

Abstract: We introduce a new benchmark, COVID-VTS, for fact-checking multi-modal information involving short-duration videos with COVID19- focused information from both the real world and machine generation. We propose, TwtrDetective, an effective model incorporating cross-media consistency checking to detect token-level malicious tampering in different modalities, and generate explanations. Due to the scarcity of training data, we also develop an efficient and scalable approach to automatically generate misleading video posts by event manipulation or adversarial matching. We investigate several state-of-the-art models and demonstrate the superiority of TwtrDetective.

## introduction the proliferation of misinformation in social media poses a serious threat to our society, especially during the covid-19 pandemic. therefore, it is necessary to develop automatic fact-checking tools to verify the claims propagated online. recent fact-checking work @xcite @xcite @xcite investigated automatic misinformation detection by developing various benchmark datasets as well as state-of-theart neural network architectures involving sources such as wikipedia pages, tables, news articles, and scientific articles. however, due to the lack of publicly available benchmarks, fact checking is still challenging on short video platforms, such as tik-tok, twitter, and instagram. verifying the factual correctness of claims in short video platforms poses new challenges. firstly, multi-modal misinformation that leads to short video posts is more misleading than using just textual content @xcite , since the claims usually only tamper with subtle elements of the factual information from the source video. furthermore, short videos @xcite containing diverse scene shifts, human activities, and crossmodal information, greatly increase the complexity and ambiguity of the video content. moreover, current methods explored the authenticity of gangenerated video with unimodal analysis @xcite . nevertheless, they cannot be directly applied to the short video platform, where the video content is often intentionally manipulated. to tackle these challenges, we introduce covid-vts, a new benchmark dataset with trustworthy claims and corresponding good-quality videos, collected from twitter video posts. covid-vts contains different inconsistent taxonomies and the inconsistency comes from different modalities. examples are shown in figure 1 . we additionally introduce a novel approach to generate fake video posts automatically by manipulating event elements or adversarial matching @xcite . the advantage of our event manipulation tool is that it's able to control the polarization of semantics and track the manipulation object by editing a small component of the factual information. we apply quality control to delete unqualified generations and address the linguistic bias. we propose twtrdetective, a multimodal factchecking model, where the input consists of a claim with the paired video, and the goal is to predict the consistency. twtrdetective takes advantage of the event alert module to precisely extract finegrained factual details from the claim, as well as pairwise consistency aggregation module to effectively measure the consistency between each modality. twtrdetective is also able to point out the inconsistent modality (e.g., see figure 3 ). experimental results show that twtrdetective achieves higher detection accuracy over baselines on two datasets. our main contributions are summarized as: • we introduce covid-vts, a fact-checking dataset for short video platforms, consisting of 10k video text pairs with diverse scenes, more accessible ## related work the spread of misinformation has led to a growing body of research in automatic fact-checking. many large scale datasets collected from wikipedia and fact-checking websites were introduced, including fever @xcite , scifact @xcite , ukp snopes @xcite , multifc @xcite . however, the fake claims in these datasets are manually generated by humans, making it expensive to deploy at scale. recently, some synthetic datasets were proposed to address this limitation. @xcite generated complex fake claims using word substitutions. @xcite took advantage of the token-infilling ability from masked language model to replace the salient tokens. also, @xcite formulated a novel data synthesis method by manipulating knowledge elements within the multi-modal knowledge graph. in order to alleviate the linguistics bias within the machinegenerated claims, @xcite constructed the out-of-context captions by retrieving the realworld sentence with the similar semantics. in a similar vein our dataset, @xcite constructed violin for the video-language inference while all the statements are written by experts. @xcite collected social media video posts from twitter but the fake claims are constructed by random swap. in comparison, covid-vts is the first covid-19 fact-checking dataset for short video platforms, containing rich information with diverse scenes, more accessible modalities as well as misleading claims from both the real world and machine generation. traditional fact-checking approaches @xcite @xcite are mainly based on text. they fall short if the evidence stems from visual information. recent multi-modal models @xcite @xcite are equipped with the ability to check consistency according to the information conveyed across modalities. in contrast, we propose a fact-checking model for the short video platforms with multimedia explanations that achieves higher accuracy to detect misinformation. ## covid-vts dataset construction covid-vts contains 10k well-formed claims with the paired videos to support or refute the claims. in this section, we first describe the procedure to select the trustworthy and consistent video/claims pairs from twitter video posts. second, we also present our approach to automatically construct well-formed inconsistent video-claim pairs. ## experiments in this section, we first introduce details of our implementation and compare the results to competing methods. lastly, we present comprehensive experiment results and discussions. ## conclusions in this paper, we release a new benchmark, covid-vts, for fact-checking in short video platforms, consisting of 10k video-claim pairs. we develop an efficient tool to automatically generate large-scale trustworthy inconsistent pairs with different semantic meanings. furthermore, our proposed fact-checking model twtrdetective achieves state-of-the-art detection accuracy. we hope this work paves the way for future studies in multi-modal fact-checking as well as other related research areas in video and language.