R ˆ3 N et:Relation-embedded Representation Reconstruction Network for Change Captioning (2021)

Abstract: Change captioning is to use a natural language sentence to describe the fine-grained disagreement between two similar images. Viewpoint change is the most typical distractor in this task, because it changes the scale and location of the objects and overwhelms the representation of real change. In this paper, we propose a Relation-embedded Representation Reconstruction Network (Rˆ3Net) to explicitly distinguish the real change from the large amount of clutter and irrelevant changes. Specifically, a relation-embedded module is first devised to explore potential changed objects in the large amount of clutter. Then, based on the semantic similarities of corresponding locations in the two images, a representation reconstruction module (RRM) is designed to learn the reconstruction representation and further model the difference representation. Besides, we introduce a syntactic skeleton predictor (SSP) to enhance the semantic interaction between change localization and caption generation. Extensive experiments show that the proposed method achieves the state-of-the-art results on two public datasets.

## introduction change captioning aims to generate a natural language sentence to detail what has changed in a pair of similar images. it has many practical applications, such as assisted surveillance, medical imaging, and computer assisted tracking of changes in media assets @xcite . different from single-image captioning @xcite @xcite , change captioning addresses two-image captioning, which requires not only to understand both the tiny red metal block that is to the right of the matte cylinder moved. <@xmath0 the black car has moved slightly down the road. figure 1 : two examples of change captioning about an object move. the first example shows that the viewpoint changes the scale and location of the objects in the "after" image; the second example shows mostly well-aligned a pair of images with underlying illumination changes from surveillance cameras. in this paper, we propose a relation-embedded representation reconstruction network (r 3 net) to handle viewpoint changes and model the finegrained difference representation between two images in the process of representation reconstruction. concretely, for "before" and "after" images, the relation-embedded module respectively performs semantic relation reasoning among object features via the self-attention mechanism. this can enhance the fine-grained representation ability of original object features. to model the difference representation, a representation reconstruction module (rrm) is designed, where a "shadow" representation ("after" or "before") is used to reconstruct a "source" representation ("before" or "after"). the rrm first leverages every location in the "source" to stimulate the corresponding locations in the "shadow" to judge their semantic similarities, i.e., "response signals". further, under the guidance of the signals, the rrm picks out the unchanged features from the "shadow" as the "reconstruction" representation. the "difference" representation is computed with the changed features between the "source" and "reconstruction". next, a dual change localizer is devised to use the representation of difference as the query to localize the changed object feature on the "before" and "after", respectively. finally, the localized features are fed into an attention-based caption decoder for caption generation. besides, we introduce a syntactic skeleton predictor (ssp) to enhance the semantic interaction between change localization and caption generation. as observed in figure 1 , a caption mainly consists of a set of nouns, adjectives, and verbs. these words can convey main information of the changed object and its surrounding references, called syntactic skeletons. the skeletons, which are predicted based on a global semantic representation derived from the r 3 net, can supervise the modeling of difference representation and provide the decoder with high-level semantic cues about change type. this makes the learned difference representation more relevant to the target words and enhances the quality of generated sentences. the main contributions of this paper are as follows: (1) we propose the r 3 net to learn the finegrained change from the large amount of clutter and overcome viewpoint changes by embedding semantic relations into object features and performing representation reconstruction with respect to the two images. (2) the ssp is introduced to enhance the semantic interaction between change localization and caption generation via predicting a set of syntactic skeletons based on a global semantic representation derived from the r 3 net. (3) extensive experiments show that the proposed method outperforms the state-of-the-art approaches by a large margin on two public datasets. skeleton prediction in captioning. syntactic skeletons can provide the high-level semantic cues (e.g., attribute, class) about objects, so they are widely used in image/video captioning works. these methods either used skeletons as main information to generate captions @xcite 2015; <@xmath1, red, metal, block, moved syntactic skeleton predictor representation reconstruction a tiny red metal block moved. ⨀ ⊖ 𝑋 𝑑𝑖𝑓𝑓 𝑎𝑓𝑡 𝑋 𝑎𝑓𝑡 𝑋 𝑏𝑒𝑓 𝑋 𝑑𝑖𝑓𝑓 𝑏𝑒𝑓 ⊕ 𝜎 figure 2 : the architecture of the proposed method, consisting of a relation-embedded representation reconstruction network, a syntactic skeleton predictor, a dual change localizer and an attention-based caption decoder. @xcite or leveraged them to bridge the semantic gap between vision and language @xcite . although the skeletons played different roles in the above methods, the common point was that they only represent basic information of objects in images or videos. different from them, besides basic information, we try to use skeletons to capture the changed information among objects. ## methodology as shown in figure 2 , the architecture of our method consists of four main parts: (1) a relationembedded representation reconstruction network (r 3 net) to learn the fine-grained change in the presence of viewpoint changes; ## conclusion in this paper, we propose a relation-embedded representation reconstruction network (r 3 net) and a syntactic skeleton predictor (ssp) to address change captioning in the presence of viewpoint changes, where the r 3 net can explicitly distinguish semantic changes from viewpoint changes and the ssp is to enhance the semantic interaction between change localization and caption generation. extensive experiments show that the state-ofthe-art results are achieved on two public datasets, clevr-change and spot-the-diff.