Not all Fake News is Written: A Dataset and Analysis of Misleading Video Headlines (2023)

Abstract: Polarization and the marketplace for impressions have conspired to make navigating information online difficult for users, and while there has been a significant effort to detect false or misleading text, multimodal datasets have received considerably less attention. To complement existing resources, we present multimodal Video Misleading Headline (VMH), a dataset that consists of videos and whether annotators believe the headline is representative of the video’s contents. After collecting and annotating this dataset, we analyze multimodal baselines for detecting misleading headlines. Our annotation process also focuses on why annotators view a video as misleading, allowing us to better understand the interplay of annotators’ background and the content of the videos.

## introduction social media platforms are used by half of us adults for everyday news consumption @xcite . they have supplanted television as the most common purveyor of news @xcite . however, content created on these online platforms are often lower quality than traditional sources and more prone to false stories. @xcite contend that false news spreads six times faster online than offline. this work focuses on one part of this problem: does a video headline match its content. we call this misleading video headline detection. in text, this is called incongruent headline detection @xcite and is an important problem because the headline is the first step to a reader accessing content @xcite . while there has been work to automatically detect misleading headlines from text (section 6), users are more likely to believe fake news when it is accompanied by videos @xcite -and there are no datasets to train models for misleading video headline detection. ## video misleading heading dataset vmh a misleading headline is when the headline distorts the underlying content @xcite and facts in the news body, leading the audience to infer more or less than what was actually presented in the content. for example, in our task, the headline "obama: i'm proud to be leaving without scandal" exaggerates the view of the content; the video plays obama's speech that he left the administration without a significant scandal. this distortion makes detecting misleading video headlines even more arduous because the audience has to watch the video to know if the headline is representative or-as in this case-has a subtle exaggeration or misrepresentation. vmh consists of 2,247 video posts from 2014 to 2016. we focus on this period because it coincided with the 2016 us presidential election, which was rife with disinformation, and is distant enough from current events that we believe annotators can be more confident about determining whether claims are true; as even news organizations are not immune to false news @xcite . our facebook video posts come from rony et al. (2017), where we manually filtered any video that exceeded five minutes or had low-quality video or sound. the videos in vmh ( we further collect venue-related information such as venue credibility 2 (e.g., high) and venue kind 3 (e.g., broadcast). also, we manually assigned news topics (e.g., politics) inspired by news was there anything other than what you expected in the video? ## dataset analysis out of 2,247 video headlines, 1,906 headlines are annotated as representative, while 341 headlines are annotated as misleading, suggesting a high class imbalance. this section investigates vmh to understand what features contribute to (or correlate with) a headline being classified as misleading. misleading features figure 4 suggests that the venues trutv and wearechange.org are strong indicators for misleading headlines. more generally, videos from the venue kind website (as opposed to traditional media) are likely to be misleading (29%). the specific venue and the kind of venue may help detect misleading headlines (appendix e). clickbait misleading videos and clickbait both have the same goal: to entice more people to click on the underlying content. a reasonable hypothesis is that they would use similar tricks to lure in users. thus, we reproduce the features found by @xcite to be associated with clickbait headlines such as the number of demonstrative adjectives, numbers, and wh-words (e.g., what, who, how) for the headlines in vmh. demonstrative adjectives do appear more frequently in misleading headlines, while numbers and superlative word features are less frequent ( ## experiments the misleading headline detection task is challenging because of the inherent subjectivity of the task. it also requires multimodal approaches that can consider both the headline and the video to make inferences about whether the headline is representative or not. thus, this section benchmarks both textonly and multimodal approaches typically used for detecting video-text similarity and video-text entailment tasks. ## experimental results and model analyses experiment results next, we validate the utility of the multimodal features in a partial-input setting. we explore how the subjectivity can affect the detection. ## related work one of the major factors of misinformation is inaccurate headlines, which pervade social media platforms @xcite . clickbait is characterized by misleading headlines, depending on the degree of deception the audience experiences @xcite . however, clickbait detection problems are distinguished from misleading headlines as they may exaggerate the content but are not particularly misleading @xcite . as the spread of fake news appears in many forms of multimedia @xcite , several works are on constructing datasets to enable research on multimodal misleading headline detection @xcite . @xcite introduces an image-based dataset and focuses on misrepresented headlines on instagram. also, @xcite introduces a dataset of youtube videos with manual annotations generated by misleading seed videos from the youtube recommendation system. zannettou et al. (2018) proposes a misleading-labeling mechanism with both manual and automatic. in this case, annotated videos could be biased as manual and automatic annotation may not be in consensus; they can lead to erroneous annotations of misleading headlines. apart from dataset research, previous works focus on detecting multimodal fake news by including multimedia features such as false videos, images, audio, and caption @xcite @xcite . however, these works feature general forms of fake news (i.e., deep-fake videos), not misleading headlines. for multimodal models built for misleading headline detection tasks, song et al. (2016) headlines transcripts entail score answer the sounds of emotions . . . we use the principles of music to work with rhythm and melody to regain the functional use of language. phrase is if we. . . nice job. let's all. well you wanna skip this up? okay. do you wanna skip it or singing it? you wanna try to sing it? let's jump to the chorus. okay? so darling then. music is what emotions sound like. . . ✓ 0.71 m there is a double standard . . . is there a double standard when it comes to transparency between trump and clinton? well, of course, there's a double standard. . . he's doing over a hundred foreign deals and he wants to be both the commander chief and the representative in the world for the united states. . . i mean, the difference between telling somebody you had pneumonia on sunday instead of friday is not even in the same league really. . . ✗ 0.20 r honor a vet i warfighters . . . having worked with veterans throughout my career, i know firsthand the importance of honoring our troops. this veterans day our series the war fighters and history are partnering with team rub con to create honor event. . . honor the vets and more fighters in your life, and share a photo and a story today. learn more history dot com honor that. . . ✓ 0.53 r ## conclusion and future work we present vmh, a dataset of misleading headlines from social media videos. our annotation scheme reduces the task's subjectivity, and we verify the reliability of the annotations. we believe incorporating the crowd workers' distinct opinions (e.g., headline types and rationales) on misleading headlines allows crude reflection of the current social media misinformation phenomenon. through their lenses, we anticipate a better understanding of how people perceive misinformation in misleading video headlines and for future work, use it to generalize the detection models that are soon to be deployed. to obtain even more realistic examples for this task, we encourage applying a dynamic adversarial generation pipeline. motivated by @xcite , misleading headlines could be authored by humans guided to break the existing video headline detection models. for example, while they are writing a misleading headline, if the model falsely predicts the headline as representative, it would become an adversarial, realistic example @xcite . these examples can prevent the model from learning superficial patterns @xcite and further be developed to become a robust tool for journalists to prevent them from making "honest" mistakes when writing video headlines @xcite .