COVE : CO ntext and VE racity prediction for out-of-context images (2025)

Abstract: Images taken out of their context are the most prevalent form of multimodal misinformation. Debunking them requires (1) providing the true context of the image and (2) checking the veracity of the image’s caption. However, existing automated fact-checking methods fail to tackle both objectives explicitly. In this work, we introduce COVE, a new method that predicts first the true COntext of the image and then uses it to predict the VEracity of the caption. COVE beats the SOTA context prediction model on all context items, often by more than five percentage points. It is competitive with the best veracity prediction models on synthetic data and outperforms them on real-world data, showing that it is beneficial to combine the two tasks sequentially. Finally, we conduct a human study that reveals that the predicted context is a reusable and interpretable artifact to verify new out-of-context captions for the same image. Our code and data are made available.

## introduction an out-of-context (ooc) image is a form of multimodal misinformation where the caption misrepresents the image in one or several dimensions, including the date, location, or event depicted @xcite . in 2023, more than 40% of the visual misinformation verified by fact-checkers consisted of ooc images @xcite . to debunk them, human fact-checkers follow two objectives: identifying the true context of the image, which usually takes the form of a fixed set of items @xcite @xcite @xcite , and deciding on the veracity of the caption. identifying the true context of the image first is often beneficial when checking the caption, as it may reveal inconsistencies with the image. however, it goes beyond that objective, as most context items are neither consistent nor inconsistent with the caption and provide more details than needed to verify the caption. in figure 1 , the context, shown in blue, reveals that the location in the caption is inconsistent: malta instead of indonesia. it also discusses the source and date, which are absent from the caption, and provides a precise location at the city level. many methods have been proposed to facilitate fact-checking (fc) of ooc images. however, they study either context prediction @xcite or veracity prediction @xcite @xcite . they do not consider automating them sequentially, leveraging the comprehensive context to predict the veracity of the caption. in this work, we introduce cove, the first method that both predicts a comprehensive context for the image and a veracity label for the caption. the method consists of two steps, shown in figure 1 . first, the context is predicted as a set of seven items, three of which are first introduced in this work. compared to prior work, cove leverages a more diverse set of evidence to predict the context, including web search, knowledge bases, and the parametric knowledge of large language models (llms). afterward, the caption is compared with the context to predict its veracity. we summarize our contributions as follows. (1) leveraging a more diverse set of evidence, cove outperforms the context prediction sota @xcite for all context items, including three items introduced in this work, by 0.3 to 18.9 percentage points. (2) cove is competitive with the best models for veracity prediction on synthetic data and outperforms them on real-world data by up to 4.5 percentage points in macro f1, highlighting the benefits of automating veracity prediction based on the predicted context. (3) our experiments show that the predicted context is an interpretable artifact for human users, which they can reuse multiple times to verify new captions about the same image. ## context and veracity (cove) cove is a method to predict the true context of an image first and then the veracity of its caption. the overall architecture can be divided into six steps, as illustrated in figure 2 . the first three steps are concerned with the collection of a diverse set of evidence, i.e., web captions and visual entities ( §3.1), wikipedia entities ( §3.2), and automated captions ( §3.3). based on their similarity with web images and captions, the veracity of certain instances is already predicted during evidence retrieval ( §3.1). after collecting the evidence, a first version of the context is predicted with an llm ( §3.4). then, some missing context items are updated by searching relevant wikipedia passages ( §3.5). eventually, a model predicts the veracity of the caption based on the predicted context ( §3.6). ## conclusion we propose cove, a new method to combat ooc misinformation that predicts first the true context of an image and leverages it to predict the veracity of its caption. cove outperforms the sota on context prediction for all context items while being competitive with the best veracity models, even outperforming them on real-world data by up to 4.5 percentage points in macro f1, showcasing the benefits of sequentially performing context and veracity prediction. furthermore, our human study shows that the predicted context is a useful and reusable artifact for human users to verify new captions for the same image.