Context-aware Difference Distilling for Multi-change Captioning (2024)

Abstract: Multi-change captioning aims to describe complex and coupled changes within an image pair in natural language. Compared with single-change captioning, this task requires the model to have higher-level cognition ability to reason an arbitrary number of changes. In this paper, we propose a novel context-aware difference distilling (CARD) network to capture all genuine changes for yielding sentences. Given an image pair, CARD first decouples context features that aggregate all similar/dissimilar semantics, termed common/difference context features. Then, the consistency and independence constraints are designed to guarantee the alignment/discrepancy of common/difference context features. Further, the common context features guide the model to mine locally unchanged features, which are subtracted from the pair to distill locally difference features. Next, the difference context features augment the locally difference features to ensure that all changes are distilled. In this way, we obtain an omni-representation of all changes, which is translated into linguistic sentences by a transformer decoder. Extensive experiments on three public datasets show CARD performs favourably against state-of-the-art methods. The code is available at https://github.com/tuyunbin/CARD.

## introduction change captioning aims to describe differences between a pair of similar images, which enables many important applications, such as automatic report generation about change conditions of surveillance areas @xcite and pathological changes between medical images @xcite . on the other hand, this task is more challenging than image captioning @xcite @xcite . this is because machines need to understand the contents of * corresponding authors a car is approaching the parking lot from the right the blue truck is gone @xcite house and bushes are removed and a road is built with villas built along on both sides @xcite someone changed location of the small gray metal cylinder someone added a large yellow rubber sphere someone removed the large gray metal cube the small blue rubber sphere changed its location (c) two images simultaneously, and further reason and caption all genuine changes between them, while resisting irrelevant viewpoint/illumination changes. recently, single-change captioning has made remarkable progress @xcite @xcite @xcite . in a dynamic environment, however, the changes are usually the many-in-one, where multiple changes exist in an image pair. for instance, there are multiple object/background changes (figure 1 (a) (b)). in other cases, object and viewpoint changes simultaneously appear (figure 1 (c )). in above cases, unchanged objects commonly mingle with changed ones and even appear position misalignment under viewpoint changes. such distractors pose a great challenge to identify and caption the genuine changes. there are a few attempts to address multi-change captioning. the pioneer work @xcite ) computed pixel differences of two images, which is sensitive to noise. latest works tried to capture differences at representation space: some of them @xcite @xcite computed difference features by subtraction, while the others @xcite built the correlations between the patches of two images to model the change features for caption generation. despite progress, the above endeavors in multichange captioning have several limitations. (1) direct subtraction between two images generalizes poorly to unaligned image pairs under viewpoint changes (figure 1 (c) ). (2) directly correlating two images fails to sufficiently mine locally unchanged features as multiple objects change, because such features might mingle with the features of changed objects. (3) these methods focus on modeling locally difference features, which are useful to catch conspicuous changes. nevertheless, certain local changes with weak features might be overlooked, e.g., the car occluded by its shadows in figure 1 (a) . these limitations would result in obtaining unreliable difference features for the language decoder. we notice that the above methods capture differences between two images only based on local features, while neglecting the use of more comprehensive features. we argue that, to learn locally unchanged/changed features of two images, the model should first encapsulate their context features of commonality and difference. such context features aggregate all similar/dissimilar semantics, termed common/difference context features. the former can help correlate and mine locally common features for deducing locally difference features, while the latter can augment the locally difference features to ensure all changes are distilled. in this paper, we propose a context-aware difference distilling (card) network to learn the robust difference features under multi-change scenes. specifically, given the featuers of two images, we first build intra-image interaction to help the model understand each image content of the pair. then, we use card to decouple the common/difference context features from the image pair. herein, the common context features of two images summarize joint semantics in between; the difference context feature in each image provides an independent space to preserve its all changed semantics. besides, the consistency and independence constraints are designed to enforce the alignment and discrepancy of common and difference context features, respectively. next, guided by the common context features, card models inter-image interaction to mine locally common features, which are removed from the pair to distill locally difference features. subsequently, card augments the locally difference features via the difference context features, so as to construct an omni-representation of all changes, for generating descriptions by a transformer decoder. our key contributions are: (1) we propose card to first decouple common and difference context features, and then use them to facilitate modeling an omni-representation of all changes for multi-change captioning. (2) the consistency and independence constraints are customized to guarantee the alignment and discrepancy of decoupled common and difference context features. (3) extensive experiments show our method achieves the state-of-the-art results on three public datasets. ## related work change captioning is an emerging task in the community of multi-modal learning @xcite @xcite . in the following, we introduce the relevant works about single-change captioning and multi-change captioning, respectively. single-change captioning has been widely studied by most existing methods. the prior work @xcite collects a dataset about geometric objects under viewpoint changes. this work computes the difference representation by direct subtraction, which generalizes poorly between two unaligned images. to remedy this limitation, m-vam @xcite , vacc @xcite and r 3 net @xcite ) match local features to predict difference features, which has been a classic paradigm. the latest work scorer+cbr @xcite further improves this paradigm by maximizing cross-view contrastive alignment between two images, so as to learn a more stable difference representation. in addition, these works @xcite @xcite introduce the idea of cross-modal consistency constraint to improve captioning quality. besides improving architecture, recent works @xcite propose the strategy of pre-training to fine-tuning for facilitating change location and caption. however, it is seldom that only one change appears in a dynamic environment, so a powerful model should has the capability to describe multiple changes. multi-change captioning has been explored by a few attempts. the pioneer work @xcite proposes to describe multiple changes between the image pairs from the surveillance cameras, where it captures changes at pixel level. recent works @xcite @xcite propose to caption changes between remote sensing images, where they first compute the difference features and then use them for change detection and description. nevertheless, the above methods only describe differences between two well-aligned images, while ignoring the cases of unaligned image pairs under varied viewpoints. to this end, qiu et al. @xcite collect a dataset to caption multiple changes under viewpoint changes, where they adopt the classic paradigm of local feature matching to detect changes for caption generation. however, such a matching paradigm may fail to mine the fine-grained common features and thus learn error-prone difference features. meanwhile, the current methods focus on modeling local difference features, which risks ignoring certain local changes with weak features. on the other hand, qiu et al. @xcite develop a new synthetic dataset to describe the multiple changes and their orders. however, recording the order of changed objects is laborious in the real world, making it hard to test such a capability in a real-world scene. in short, different from previous methods computing difference features only based on local features, card first decouples common and difference context features from an image pair. the common context features guide the model to fully extract locally unchanged features for computing the features of local differences, while the difference context features augment the locally difference features to construct an omni-representation of all changes, for generating accurate sentences. ## methodology the overall architecture of our method is shown in figure 2 . given a pair of images, our method is to generate linguistic sentences that detail all of the changes. architecture-wise, our method contains three components: ## caption generation after obtaining the omni-representation @xmath0, we use a transformer decoder @xcite to decode it into sentences. first, we obtain the embedding features of all m words of these sentences, where each sentence is separated by a special token @xcite . then, we use the masked self-attention to model relationships among these word features. next, we model the interaction between the word features and omni-representation by cross-attention, so as to locate the most related difference features during word generation. subsequently, we feed the selected features into a feedforward network to obtain the enhanced difference representation, denoted as ĥ ∈ r m×d . finally, the probability distributions of words in these sentences are calculated via a single hidden layer: ## conclusion in this paper, we propose the card to reason and describe genuine changes under various multichange scenarios. card first decouples the common and difference context features from the image pair. then, two kinds of constraints are designed to ensure the alignment and discrepancy of the common and difference context features, respectively. further, we use the common context features to guide the mining of locally common features for deducing locally difference features. in addition, we leverage the difference context features to augment the locally difference features, thereby constructing an omni-representation of all changes for multi-change captioning. extensive experiments conducted on the three datasets show that the proposed card outperforms the current state-of-the-art methods by a large margin.