Learning to Describe Implicit Changes: Noise-robust Pre-training for Image Difference Captioning (2025)

Abstract: Image Difference Captioning (IDC) methods have advanced in highlighting subtle differences between similar images, but their performance is often constrained by limited training data. Using Large Multimodal Models (LMMs) to describe changes in image pairs mitigates data limits but adds noise. These change descriptions are often coarse summaries, obscuring fine details and hindering noise detection. In this work, we improve IDC with a noise-robust approach at both data and model levels. We use LMMs with structured prompts to generate fine-grained change descriptions during data curation. We propose a Noise-Aware Modeling and Captioning (NAMC) model with three modules: Noise Identification and Masking (NIM) to reduce noisy correspondences, Masked Image Reconstruction (MIR) to correct over-masking errors, and Fine-grained Description Generation (FDG) to produce coherent change descriptions. Experiments on four IDC benchmarks show that NAMC, pre-trained on our large-scale data, outperforms streamlined architectures and achieves competitive performance with LLM-finetuned methods, offering better inference efficiency.

## introduction image difference captioning (idc) involves discerning fine-grained changes between two similar images and generating change descriptions that summarize these changes @xcite . unlike general image captioning task @xcite , which describes the content of a single image, idc requires finegrained semantic understanding to accurately detect and articulate subtle changes between two visually and semantically similar images. the ability of this task to model these nuanced changes proves invaluable in various applications, like medical case comparison @xcite ), * corresponding author remote sensing monitoring @xcite , and security monitoring @xcite . recently, idc approaches have made long-term progress @xcite . however, their performance is often constrained by limited training data. this limitation arises because acquiring idc training data-comprising similar image pairs and their corresponding change descriptions-is particularly challenging. the process heavily relies on manual annotations, making data curation both labor-intensive and time-consuming. recent efforts @xcite have attempted to address this challenge by leveraging large multi-modal models (lmms) to generate change descriptions for pre-collected image pairs @xcite . however, these generated descriptions often introduce noise in the form of obvious, irrelevant, or redundant details, resulting in weak annotations compared to those created by human annotators, undermining model performance. in particular, such descriptions tend to summarize changes in a coarse manner, failing to capture fine-grained details from both images @xcite . this makes it challenging to identify and handle noisy correspondences between images and descriptions during subsequent model training, especially when considering noise-aware learning. to this end, this work focuses on noise-robust image difference captioning at both the data and model levels, as depicted in figure 1 . at the data level, we aim to leverage lmms to generate finegrained implicit-change descriptions for similar image pairs, which provides the critical data foundation required for enabling model-level noise identification mechanisms. at the model level, we propose namc: a noise-aware modeling and captioning model designed to mitigate cross-modal noisy correspondences introduced by the generated data. namc incorporates three trainable modules: a noise identification and masking (nim) module to mitigate cross-modal noisy correspondences, a masked image reconstruction (mir) module to mitigate over-masking errors on images and enhance image comprehension, and a fine-grained description generation (fdg) module to enable model to generate accurate change descriptions. data level. we curate the pre-training data generation process by using structured prompts to guide lmms in generating descriptions that capture fine-grained differences between two images. specifically, we aim to generate detailed descriptions for each image at a granular level, such as "the first image depicts a daytime scene with ..., while the second image is a night scene with ...". these descriptions implicitly capture changes between the images-for instance, the phrase "change it to night" can be inferred through the contrast between the two descriptions. moreover, these descriptions can be explicitly aligned with corresponding image regions, facilitating noise-robust learning in subsequent model training. model level. we develop three modules in the proposed namc. nim module uses dual encoders-with contrastive loss-to project images and their corresponding descriptions into a shared space. within this space, cross-modal token relevance computation guides a noise masking (nm) strategy to generate masks for both modalities. these masks filters noise from both images and descriptions, producing denoised images and descriptions. mir module mitigates potential over-masking-relevant information is mistakenly masked-by reconstructing missing image content through a masked image encoder, while simultaneously modeling cross-image interactions that ignore noisy regions. this not only compensates for masking errors but also captures cross-image relationship modeling. fdg module leverages the cross-image relationships learned by mir and em-ploys a multi-modal decoder for the generation of change descriptions. it is pre-trained using the combined denoised description as supervision, enabling description of subtle inter-image distinctions through joint visual-textual decoding. the main contributions of this work are summarized as: (1) we explore a noise-robust image difference captioning, using lmms to generate largescale change descriptions while mitigating the impact of noise from these generated descriptions during pre-training. (2) we introduce a pre-training dataset generation method that automatically produces fine-grained implicit-change descriptions, capturing subtle and implicit differences between image pairs. (3) we propose a noise-aware modeling and captioning model to address cross-modal noisy correspondences and enhance the comprehension of images and descriptions during pre-training. (4) experiments on four idc datasets show that our proposed namc achieves competitive performance with the llm-finetuned state-of-the-arts, while maintaining superior inference efficiency. ## related work within the domain of multi-modality learning @xcite @xcite , image difference captioning (idc) was first introduced by @xcite and refined by @xcite with distractors like viewpoint or brightness changes. both followed a single-step training from scratch paradigm including an image encoder and a text decoder, typically with fewer parameters. leveraging this paradigm, subsequent works mitigated the impact of distractors @xcite and concentrated on fine-grained difference recognition @xcite , providing a thorough exploration of the idc task. on the other hand, recent approaches have adopted a two-step training that either (1) integrates auxiliary tasks alongside the main idc objective, or (2) implements a pre-alignment stage prior to caption generation, as opposed to employing end-toend training from scratch. @xcite introduced an additional retrieval task as an auxiliary objective to enhance the performance of image difference captioning. both yao et al. (2022) and guo et al. (2022) proposed a pretraining phase to align the image pairs with change descriptions before performing difference captioning. particularly, yao et al. (2022) used a cross-modal transformer architecture to jointly process the images and captions, leveraging multiple contrastive learning tasks. guo et al. (2022) utilized a cliplike architecture, aligning the [cls] token from the encoder output with image-text retrieval objectives. with the advent of lmms @xcite , research focus has shifted toward applying these models to idc and fully leveraging their rich prior knowledge. @xcite and @xcite partially finetuned blip2 @xcite model for idc task, with @xcite specifically introducing retrieval-augmented generation to improve captioning. other methods @xcite @xcite utilized knowledge from lmms to generate additional training data for finetuning. despite promising results, handling noise interference is still a challenge for models using lmms to generate additional data. inspired by the success of noise-robust techniques across various domains @xcite @xcite , we propose a noise-aware idc model to address cross-modal noisy correspondences in lmm-generated data. ## method figure 2 shows our proposed noise-robust image difference captioning framework, which begins with generating fine-grained implicit-change descriptions using lmms on curated image pairs. we then introduce the namc model, composed of three modules, nim for noise mitigation, mir for reconstruction, and fdg for change description generation, followed by the pre-training strategies. ## conclusion in this work, we focus on noise-robust image difference captioning at both the data and model levels. at the data level, we aim to leverage lmms to gen-erate fine-grained implicit-change descriptions for similar image pairs. at the model level, we propose namc: a noise-aware modeling and captioning model designed to mitigate cross-modal noisy correspondences introduced by the generated data. namc incorporates three trainable modules: a noise identification and masking (nim) module, a masked image reconstruction (mir) module, and a fine-grained description generation (fdg) module. experiments on four idc benchmarks demonstrate the effectiveness of our namc.