Change Entity-guided Heterogeneous Representation Disentangling for Change Captioning (2025)

Abstract: Change captioning aims to describe differences between a pair of images using natural language. However, learning effective difference representations is highly challenging due to distractors such as illumination and viewpoint changes. To address this, we propose a change-entity-guided disentanglement network that explicitly learns difference representations while mitigating the impact of distractors. Specifically, we first design a change entity retrieval module to identify key objects involved in the change from a textual perspective. Then, we introduce a difference representation enhancement module that strengthens the learned features, disentangling genuine differences from background variations. To further refine the generation process, we incorporate a gated Transformer decoder, which dynamically integrates both visual difference and textual change-entity information. Extensive experiments on CLEVR-Change, CLEVR-DC and Spot-the-Diff datasets demonstrate that our method outperforms existing approaches, achieving state-of-the-art performance. The code is available at https://github.com/yili-19/CHEER

## introduction change captioning aims to describe the differences between two images using natural language. unlike conventional image captioning that describes main content of a single image, change captioning requires understanding both the semantic correspondence and the differences between a pair of images. this task has garnered significant attention due to its wide-ranging applications in fields such as visual monitoring @xcite , remote sensing image analysis @xcite , and medical image comparison @xcite . existing methods @xcite @xcite mainly follow an encoderdecoder framework, which first extracts patch features from a pair of images, then models the difference features in between, and finally decodes these features to generate change captions. to accurately locate the change regions, current works @xcite mostly match similar features between the two images and then disentangle the difference features. additionally, to generate higher-quality captions, some studies @xcite introduce large language models (llms) into this task. they primarily replace the lstm/transformer structure with pre-trained llms, and further fine-tune the llms with different strategies to make them adapt to change captioning. despite the progress, there are two major limitations in existing approaches. first, viewpoint variation (figure 1 (c )) between image pairs often leads to deformation of objects in the images (i.e., pseudo changes @xcite ). such pseudo changes make the distinguishing of really semantic changes more challenging. existing works attempt to reduce the influence of irrelevant factors through introducing additional mechanisms in the visual encoder, such as using contrastive learning to align the visual features @xcite . this approach, however, does not demonstrate significant effectiveness under extreme viewpoint changes (figure 1 (c )), as pseudo changes in these scenarios become more pronounced. this leads to difficulties in feature matching for unchanged objects, which affects subsequent change localization. we have observed that despite the challenge of distinguishing real changes from distractions based solely on visual features, the similarity between the object representation in the image and its corresponding textual representation remains relatively unaffected. some works @xcite attempt to introduce full-sentence descriptions as prior knowledge. however, directly using an entire sentence as prior knowledge introduces a lot of redundancy and even incorrect information. in fact, it is sufficient to focus only on the change entity provided by the text (such as a red cylinder, a green cube, etc.) which indicates what has changed. secondly, previous studies @xcite typically rely solely on visual features as input to the decoder. some existing works incorporate additional information, such as part-of-speech tags @xcite , to generate higher-quality descriptions. there are also methods based on llms @xcite that yield good results, but they come with considerable computational cost. if we can incorporate some semantic prior information to guide the model, it could improve model performance without introducing significant cost. in this paper, we propose a novel change entity-guided heterogeneous representation disentangling (cheers) network, which explicitly models and uses textual change entities to guide both feature disentanglement and caption generation. specifically, we first design a change entity retrieval module, to locate what has changed based on the similarity between change entities and images. second, we design a heterogeneous representation disentangling module to decouple the genuine differences between two images and generate the representations that encapsulate the difference information. here, we devise a commonality representation enhancement module (cre) that strengthens visual features in similar regions to decouple the difference information from the similar features. then, we use a difference representation enhancement (dre) module, to highlight the difference regions. meanwhile, the change entities are further used to enhance the difference features, while enforcing consistency in the enhanced regions between the image-image and image-entity pairs to constrain the model. finally, to generate more accurate change captions, we design a gated transformer decoder that dynamically fuses the change entity textual information with the difference visual features. through the gating mechanism, the model can adjust the fusion ratio of the textual information containing the change entity and the visual information representing the change based on context when generating the next word. the key contributions of this work are threefold: (1) we propose a novel cheers that identifies changed objects from a textual perspective, providing explicit guidance for representation learning. further, cheers uses hrd to effectively separate differences and similarities while mitigating viewpoint variations and enhancing subtle change perception. ( 2 ) we design a gated transformer decoder, which dynamically adjusts the fusion of textual and visual information based on context, prioritizing textual entity information for subject descriptions and visual features for change details. (3) extensive experiments on the three public datasets demonstrate that our approach significantly outperforms state-of-the-art change captioning models. ## related work change captioning is an emerging task in the multi-modal community @xcite @xcite , which generates natural language descriptions of the differences between two images. early works, such as jhamtani @xcite , approach this task by approximating object-level differences through pixel-wise clustering based on the difference between images. park @xcite uses dynamic attention maps to localize the changes, while shi @xcite extracts both changed and unchanged features to input into a caption decoder. however, in real-world scenarios, viewpoint variation often introduces interference, reducing the model's ability to accurately identify changes. to enhance the robustness of models to such viewpoint changes, tu @xcite designs neighboring feature aggregation to capture contextual information and common feature distillation to learn contrastive information between images. liao @xcite attempts to model the relative spatial relationships of objects in a 3d scene to eliminate interference based on this contextual information. tu @xcite utilizes contrastive learning to align the representations of two images, thereby learning a stable difference representation. additionally, to generate higherquality captions, several works have attempted to incorporate additional information to assist the decoding process. tu @xcite introduces part-of-speech information during decoding and uses a dynamic switch to control the fusion of this information. more recent works have leveraged large pre-trained llm for this task. for instance, hu @xcite employs learnable query tokens that probe the multi-level encoded features of both images to effectively capture their differences and assist the llms in learning these differences. zhang @xcite fine-tunes large models and incorporates a relevant corpus as additional assistance to generate more accurate captions. overall, previous works have primarily focused on identifying differences from visual information, generating difference representations, and then using a decoder to produce captions. in contrast, this paper shifts the focus to discovering differences from a textual perspective, leveraging additional textual information to guide the visual encoder in more accurately localizing differences. furthermore, during the caption generation process, we fuse textual information to produce higher-quality descriptions. ## method given a pair of images (i bef , i af t ), we first employ the change entity retrieval module, as shown in figure 3 , as to extract textual change entities, denoted as e, which provide explicit guidance for identifying key differences. next, the heterogeneous representation disentangling module processes (i bef , i af t ) to separate difference features, denoted as (d bef , d af t ) respectively. finally, we utilize a gated transformer decoder, which dynamically fuses e and d based on context to generate the final change description s cap . ## conclusion this paper proposes cheers, which leverages change entities to guide difference localization and caption generation. cheers first determines the change entity by maximizing the similarity difference between two images and candidate subjects. then, guided by the change entity, a representation enhancement mechanism is applied to disentangle difference features from distraction. additionally, we design a gated transformer that dynamically fuses visual difference information with the textual change entity features. extensive experiments show that cheers achieves state-of-the-art results on multiple benchmark datasets, demonstrating its effectiveness in various change scenarios.