Dataset of News Articles with Provenance Metadata for Media Relevance Assessment (2025)

Abstract: Out-of-context and misattributed imagery is the leading form of media manipulation in today’s misinformation and disinformation landscape. The existing methods attempting to detect this practice often only consider whether the semantics of the imagery corresponds to the text narrative, missing manipulation so long as the depicted objects or scenes somewhat correspond to the narrative at hand. To tackle this, we introduce News Media Provenance Dataset, a dataset of news articles with provenance-tagged images. We formulate two tasks on this dataset, location of origin relevance (LOR) and date and time of origin relevance (DTOR), and present baseline results on six large language models (LLMs). We identify that, while the zero-shot performance on LOR is promising, the performance on DTOR hinders, leaving room for specialized architectures and future work.

## introduction over the last few years, the use of manipulated imagery for disinformation and misinformation has grown steadily @xcite @xcite . many believe this is largely due to the abundance of ai-powered tools that allow users to edit or generate media from scratch, including images (text-to-image @xcite @xcite , in-painting @xcite ), audio @xcite @xcite , voice cloning @xcite ), and video (deepfakes @xcite @xcite ), text-to-video @xcite ). these tools have not only become easily accessible online but also increasingly intuitive to use, often requiring only textual descriptions @xcite . consequently, a large body of work has emerged focusing on the detection of ai-manipulated or ai-generated content @xcite . however, despite the proliferation of ai tools, a simpler form of image-based manipulation remains prevalent in misinformation and disinformation @xcite : the use of outof-context or misattributed imagery to frame events in misleading ways @xcite . for example, in april 2020, images of body bags from ecuador were falsely presented as deceased covid-19 patients in new york hospitals (news literacy project, 2025), sparking confusion and controversy online. studies indicate that this type of manipulation appears in over 40% of online misinformation containing images, whereas ai-generated media is used in approximately 30% @xcite . despite this, the literature has not responded to the threat of out-of-context and misattributed imagery with the same urgency as ai-manipulated and ai-generated content. as a result, there is a scarcity of specialized resources-methods, tools, datasets, and benchmarks-for studying this phenomenon from the perspective of natural language processing (nlp). some existing work evaluates whether an image is relevant to the article in which it appears, it primarily considers whether the depicted object or scene aligns with the textual narrative @xcite . while this analyzes one aspect of media-based manipulation, it misses cases where the imagery and text appear semantically consistent but were captured at times or places that may be irrelevant or outright deceptive. @xcite , therefore, suggest a new formulation of this task. rather than asking "is this image relevant to the news story?", they instead ask "was this image captured at a time and place that is relevant to the news story?". to this end, they hypothesize that provenance metadata-a record of a file's existence from its creation through edits to distribution-could help answer this question. hence, they conduct some exploratory experiments with large language models (llms) to analyze the metadata of images used in news articles. however, they identify two major limitations: in response, we introduce a dataset of news articles with provenance-tagged images and annotations regarding their relevance to the article. since the news outlets from which the articles were sourced do not yet incorporate provenance metadata (consistent with the limitation identified above), we simulate it. specifically, we gather annotations for relevant locations and dates and embed them into the images using c2pa (rosenthol, 2022), a widely used provenance metadata library. we then use an llm to generate alternative, non-relevant dates and locations, constructing a balanced dataset containing relevant, partially relevant, and irrelevant images based on provenance. while provenance metadata is not limited to images, our dataset and evaluations focus exclusively on news articles with images. other modalities, such as video or audio, are not included, since the modality of the file from which provenance metadata is extracted does not affect the included information. the primary contributions of this paper can be summarized as follows: ## related work this section reviews existing nlp literature connected to image and video relevance assessment in news articles. first, we provide an overview of the broader area of study, which involves news articles in nlp. we then proceed specifically to existing work on image and video relevance and data provenance. ## news media provenance dataset this section presents the news media provenance dataset, comprising 637 news articles with simulated image provenance metadata, which is labeled either as relevant or not relevant. the provenance is inserted into the images using the c2pa library @xcite by us: the relevant information is provided by annotators and the not relevant is generated using an llm. two example data points are shown in figure 1 . ## baseline models we evaluate the following off-the-shelf llms to establish baseline results: chatgpt-4o @xcite , deepseek v3 @xcite , gemma 2 27b instruct @xcite , llama 3.1 8b instruct @xcite , mistral 7b instruct @xcite , and phi 3.5 vision instruct @xcite . these are some of the most prominent models in the community, chosen based on their popularity on hugging face transformers @xcite and overall benchmark performance at the time of writing. note that the parameter size and training scope of these models vary, and one can, of course, expect the larger models to outperform the smaller ones. for example, it is reasonable to expect that chatgpt-4o or deepseek v3 will outperform the much smaller llama 3.1 8b instruct. the results of this analysis should serve as a baseline for future work investigating methods designed specifically for lor and dtor. the chatgpt-4o inference was performed using openai's api. the remaining models were implemented using the hugging face transformers (wolf, 2020) library. to preserve some comparability across models, all inference parameters were left at their default values, and thus mimicking off-the-shelf use. the full prompt is presented in appendix b. the binary responses to lor and dtor were converted from text to corresponding boolean representations. whenever the llm returned a response that did not conform to the json format specified in the prompt, the inference was repeated. the inference code and prediction data is open-sourced foot_5 to maximize reproducibility. ## evaluation this section presents both quantitative and qualitative results of the baseline models evaluated on the news media provenance dataset. ## discussion this section first discusses the significance of the results attained and then shifts to examining the broader implications of our dataset and the newly proposed tasks. the quantitative results presented in section 5.1 suggest that current llms can perform lor assessment in a zero-shot setting and achieve strong baseline performance. this is not the case, however, for dtor, where the evaluated models struggled to achieve accuracy above 50%. these findings are supported by our qualitative results, reported in section 5.2, which show that while the models can reason soundly about the location presented in the article, they struggle with reasoning about dates and timelines. this highlights a broader limitation of llms and underscores the need for further research into improving temporal reasoning capabilities. in addition to challenges with representing time, we also observed that more recent news articles were often more difficult for the models to reason about. we hypothesize that this may stem from the nature of the models' training, as the most recent events are typically not included in their training datasets, making it harder for them to process or contextualize such information. as expected, larger models outperformed smaller models in our evaluation. the performance of each model could likely be improved by optimizing its parameters and customizing the instruction prompts. we, however, chose to pursue minimal optimization to maintain a level of comparability necessary for measuing baseline results. the relatively low baseline performance nonetheless reinforces the need for developing new architectures tailored to the lor and dtor tasks. we expect our dataset to play a critical role in this effort, as, to the best of our knowledge, there are no other datasets explicitly designed for the tasks of lor and dtor. expanding the dataset to include non-western news contexts and additional languages will also be essential to ensure inclusive support for underserved communities, who are often at greater risk of media manipulation. ## limitations despite the benefits of provenance metadata for assessing the relevance of media in news articles, some limitations remain. one major issue is that, even when an image or video presented alongside an article matches the scope and timeline of the story, the article can still be inaccurate or outright manipulative. we, therefore, see our method as just one tool that should be a part of a broader suite of techniques aimed at discerning problematic practices in news articles. c2pa, the employed provenance metadata framework, also has some drawbacks. older photos usually lack provenance data, limiting the use of our method on historical images. moreover, there are articles in which the presence of time-and location-matched media is not necessarily an indicator of relevance. an example of this would be articles reporting on events without clearly bounded locations and/or time frames, such as natural disasters, which often span broad regions and extended periods. additionally, certain media can be used for illustrative purposes, where strict provenance alignment is less critical to the integrity of the article (e.g., historical illustrations or generic portraits). in such cases, assessing metadata relevance requires a more flexible, nuanced approach. future work could explore automatic methods for detecting when precise alignment is necessary. furthermore, as c2pa is still a new technology, its adoption among media organizations is still limited. with many outlets pledging to join, however, its use is expected to grow. ## ethical and societal implications the use of provenance metadata for assessing the relevance of media in news articles raises ethical concerns pertaining privacy. embedding provenance metadata includes potentially sensitive information, such as location and device information, that could put journalists and activists reporting from unsafe regions at risk. sharing any information that could reveal identity or location of individuals in such contexts may be undesirable and, we believe, should take priority over establishing trustworthy news channels. this also leads to a broader point, which we wish to highlight. even though we gathered feedback on our approach from both practitioners and scholars of journalism, there may be additional implications for journalists and their readers. we, therefore, recommend that before this method (or its derivatives) are put in use at a news organization, they should be first extensively scrutinized by its staff to uncover any additional concerns. simultaneously, we remain optimistic that this method will introduce an effective tool to support individuals in an increasingly less credible and transparent information ecosystem. to that end, we believe our dataset will serve as a critical tool to improve and evaluate approaches to lor and dtor moving forward. ## conclusion this paper defined the tasks of location of origin relevance (lor) and date and time of origin relevance (dtor) for media (images and videos) presented alongside news articles, based on their provenance metadata. since no suitable datasets existed for these tasks, we introduced the news media provenance dataset-a collection of news articles with provenance-tagged images-containing both human-annotated relevant metadata and irrelevant metadata generated by a large language model (llm). we presented baseline zero-shot results for six prominent llms and found that, while out-of-the-box lor performance is strong, dtor performance remains limited, as models struggle to reason about time relevance and temporal relationships.