L 2 C : Describing Visual Differences Needs Semantic Understanding of Individuals (2021)

Abstract: Recent advances in language and vision push forward the research of captioning a single image to describing visual differences between image pairs. Suppose there are two images, I_1 and I_2, and the task is to generate a description W_1,2 comparing them, existing methods directly model I_1, I_2 -> W_1,2 mapping without the semantic understanding of individuals. In this paper, we introduce a Learning-to-Compare (L2C) model, which learns to understand the semantic structures of these two images and compare them while learning to describe each one. We demonstrate that L2C benefits from a comparison between explicit semantic representations and single-image captions, and generalizes better on the new testing image pairs. It outperforms the baseline on both automatic evaluation and human evaluation for the Birds-to-Words dataset.

## introduction the task of generating textual descriptions of images tests a machine's ability to understand visual data and interpret it in natural language. it is a fundamental research problem lying at the intersection of natural language processing, computer vision, and cognitive science. for example, single-image captioning @xcite @xcite has been extensively studied. recently, a new intriguing task, visual comparison, along with several benchmarks @xcite @xcite has drawn increasing attention in the community. to complete the task and generate comparative descriptions, a machine should understand the visual differences between a pair of images (see figure 1 ). previous methods @xcite often consider the pair of pre-trained visual features such as the resnet features @xcite as a whole, and build end-to-end neural networks to predict the description of visual comparison directly. in contrast, humans can easily reason about the visual components of a single image and describe the visual differences between two images based on their semantic understanding of each one. humans do not need to look at thousands of image pairs to describe the difference of new image pairs, as they can leverage their understanding of single images for visual comparison. therefore, we believe that visual differences should be learned by understanding and comparing every single image's semantic representation. a most recent work @xcite conceptually supports this argument, where they show that low-level resnet visual features lead to poor generalization in vision-and-language navigation, and high-level semantic segmentation helps the agent s e g m e n ta ti o n h e a d (f c n ) feature map f 1 p o o li n g h e a d gcn ⊙ fc binary map b 1 part map c 1 feature map f 2 same operations for f 2 image i 1 generalize to unseen scenarios. motivated by humans, we propose a learning-to-compare (l2c) method that focuses on reasoning about the semantic structures of individual images and then compares the difference of the image pair. our contributions are three-fold: ## l2c model we present a novel framework in figure 2 , which consists of three main components. first, a segmentation encoder is used to extract structured visual features with strong semantic priors. then, a graph convolutional module performs reasoning on the learned semantic representations. to enhance the understanding of each image, we introduce a single-image captioning auxiliary loss to associate the single-image graph representation with the semantic meaning conveyed by its language counterpart. finally, a decoder generates the visual descriptions comparing two images based on differences in graph representations. all parameters are shared for both images and both tasks. ## conclusion in this paper, we present a learning-to-compare framework for generating visual comparisons. our segmentation encoder with semantic pooling and graph reasoning could construct structured image representations. we also show that learning to describe visual differences benefits from understanding the semantics of each image.