Assessing News Thumbnail Representativeness: Counterfactual text can enhance the cross-modal matching ability (2024)

Abstract: This paper addresses the critical challenge of assessing the representativeness of news thumbnail images, which often serve as the first visual engagement for readers when an article is disseminated on social media. We focus on whether a news image represents the actors discussed in the news text. To serve the challenge, we introduce NewsTT, a manually annotated dataset of 1000 news thumbnail images and text pairs. We found that the pretrained vision and language models, such as BLIP-2, struggle with this task. Since news subjects frequently involve named entities or proper nouns, the pretrained models could have a limited capability to match news actors’ visual and textual appearances. We hypothesize that learning to contrast news text with its counterfactual, of which named entities are replaced, can enhance the cross-modal matching ability of vision and language models. We propose CFT-CLIP, a contrastive learning framework that updates vision and language bi-encoders according to the hypothesis. We found that our simple method can boost the performance for assessing news thumbnail representativeness, supporting our assumption. Code and data can be accessed at https://github.com/ssu-humane/news-images-acl24.

## introduction this study investigates the representativeness of news thumbnails, which are images displayed as previews of news articles. visual content often carries a more persuasive impact and leaves a longlasting impression compared to text @xcite @xcite . consequently, the misrepresentation in news thumbnails can cause more critical consequences. despite its importance, there have been a few studies addressing the representativeness of news images @xcite . achieving the goal requires a model to understand the cross-modal relationship between visual and textual content. researchers in natural language processing and computer vision communities have explored multimodal technologies by addressing general vision and language (v+l) tasks, including visual question answering @xcite , imagetext retrieval @xcite , and visual entailment @xcite . recent approaches have leveraged contrastive learning techniques for representing distinct modalities in the same vector space. clip @xcite stands out as one of the pioneering methods that achieve remarkable performance, aimed at increasing the similarity between image-text pairs (referred to as positive samples) compared to non-paired image and text (referred to as negative samples). while contrastive learning was not a new concept @xcite , clip outperformed preexisting methods by utilizing a vast collection of web-based image and text data for pretraining. recent advancements have further enhanced the performance by refining learning objectives @xcite , incorporating crossattention layers into the model architecture @xcite , and augmenting the training data with high-quality samples @xcite . despite the progress made in multimodal methods for general vision-and-language benchmarks, we hypothesize that evaluating the representativeness of thumbnails remains difficult, even for the most advanced models, due to two primary challenges. (1) preference for general descriptions in popular benchmarks: general v+l datasets, such as conceptual captions @xcite or coco-caption @xcite , were primarily designed to enhance a model's ability to comprehend general behaviors or scenes. consequently, these datasets deliberately avoid specific descriptions about entities (e.g., "the man at bat readies to swing at the pitch"). in contrast, news events typically revolve around political actors or organizations, leading to news texts that frequently reference named entities @xcite . (2) lack of dataset labeled according to an objective definition: representativeness is an inherently abstract and subjective concept, where even humans may disagree with each other. developing an objective definition is imperative to obtain high-quality labels that different human annotators can agree on and facilitate the development of proper technologies. while there was a human-labeled dataset @xcite , they asked for subjective opinions of human annotators without providing objective definitions grounded by journalism principles. to address (1) and ( 2 ), borrowing the principle of five ws in journalism (the associate press, 2022), we introduce the evaluation dataset of news thumbnails and text pairs, newstt. the annotators were instructed to label the 1000 pairs through the lens of who, on whether an image represents news actors. using the high-quality labels reflecting the objective definition of representativeness, we evaluated the zero-shot ability of vision and language models to understand news actors' visual portrayals. we found that clip performed better at the task than more recent methods, such as blip-2, suggesting the efficiencies of its dual encoder architecture and contrastive objective. to improve its matching ability, this study introduces cft-clip, a counterfactual text-guided contrastive language-image pretraining framework. as illustrated in figure 1 , the key idea of the proposed framework is to generate counterfactual news text where the actor of a news event is changed to be used as hard negatives for contrastive learning. in the news text example, the actor is identified as 'biden,' and its corresponding token is replaced with 'trump' by the counterfactual text generator. thus, the generated sentence no longer represents the key actor of the news event depicted in the thumbnail image. our research hypothesis is that learning to contrast the generated counterfactual text with the original news text will enhance the ability of vision language models to assess the representativeness of a thumbnail image. we summarize the contributions of this study. ## target problem: assessing news thumbnail representativeness we aim to address the problem of automatically evaluating the representativeness of a news thumbnail image for its article. the task is critical due to the two reasons. first, the sharing of news online is abundant @xcite @xcite . second, the thumbnail often serves as the initial and sole visual interaction for social media users when a news link is shared. visual content is perceived as more credible and leaves a long-lasting impression than text @xcite @xcite . given these considerations, thumbnail images that do not accurately represent their content can misguide the reader's understanding in online environments. we formally define the problem as a binary classification. given a news thumbnail i and its news text t , we aim to develop a classifier f θ (i, t ) predicting a binary label l on predicting whether i represents t . we assign 1 to l if i represents t ; otherwise, 0. i is deemed representative of t if i portrays at least one of the actors of a news event, which can be identified from t . this study aims to develop a vision language model that predicts l from a pair of i and t . we assume the task in a zero-shot setting, where the labeled data is limited. the reference news text t can be either a title or a summary. ## newstt: a dataset of thumbnail representativeness for news text 4.1 raw data collection nela-gt-2021 is a headline-oriented foot_0 news corpus that encompasses nearly all news articles published by 367 news sources throughout the year 2021 @xcite . since the dataset does not provide web links to thumbnail images, we conducted a supplementary data collection step. by referencing the urls of news articles available in the dataset, we parsed the html document and extracted the thumbnail image url from the meta tag with the og:image attribute. as a result, we obtained 442,741 pairs of news headlines and thumbnail images, which were sourced from 81 different news media outlets. according to the media ratings provided by media bias/fact check (mbfc) 2 , the datasets were well-balanced for the political orientation and trustworthiness ratings. subsequently, we partitioned this unlabeled dataset into 427,741/5,000/10,000 pairs for train/validation/test purposes while keeping the distribution of the number of articles over different media sources. the validation split was reserved for hyperparameter optimization, and the test split served as the source for labeled data construction. ## methods to tackle the task of assessing thumbnail representativeness in a zero-shot setting where no labeled data is available for training, we use a thresholding classifier based on the cross-modal similarity, ## evaluation we conducted evaluation experiments to understand the effects of the proposed method and test the research hypothesis. in section 6.2, we examined how the pretrained and proposed vision language models perform for the target task. in ablation experiments (section 6.3), we analyzed the effects of each module of the proposed method in more detail. lastly, in section 6.4, we conducted qualitative analyses to identify the error patterns where the proposed method fails. ## discussion and conclusion automatically evaluating a news thumbnail image's representativeness is important for holding journalistic standards and building a trustworthy online environment. to address the important but underexplored problem in the research community, this study introduced newstt, a paired dataset of news thumbnail images and text with high-quality labels on whether the image represents the actor of the news event. we investigated the use of vision language models for the zeroshot assessment. our proposed cft-clip outperformed larger pretrained vision language models and domain-adapted methods. this supports the research hypothesis that counterfactual news text, of which named entities are replaced by a masked language model, could enhance the cross-modal matching ability by contrastive learning. in ablation experiments, we found that using the counterfactual text generated by an autoregressive transformer language model, gpt-3.5 turbo, as negative samples could not achieve a better outcome than cft-clip. while this supports the effectiveness of the proposed method, we do not conclude that masked language models are clear winners for the counterfactual news generation over autoregressive language models. after manually examining the results, we observed that gpt could produce plausible counterfactual news. by contrast, the generation by a masked language model sometimes led to imperfect generations, including broken grammar (table a4 ). as shown in a recent study on text embedding @xcite , an additional step might be required to ensure the quality of the generated text. this study has several future directions. first, according to the rule of 5ws, this study focused on the who aspect of news thumbnails. future research could extend its focus to cover other aspects of 5ws, such as whether the image represents the subject's action, i.e., what. question-answering approaches could be investigated, as done in a recent study @xcite . second, the proposed method could be improved to address the challenges identified in the error analysis (section 6.4). future studies could investigate the use of multimodal language models, such as instructblip @xcite , llava @xcite , or gpt-4v (openai, 2023), to allow for handling the errors involving deep visual understanding. the proposed cft-clip could assist them with aligning the vision and language representation by adopting the counterfactual text. third, this study could be extended to the broader research on computational social science. as similarly done in previous studies @xcite , one could extend the target problem by addressing it as a ranking task rather than binary classification. it could enable a news application that recommends a suitable thumbnail image for a given news article. additionally, the proposed cft-clip could be adopted for automated fact verification involving image evidence @xcite @xcite .