Recent advances in language and vision push forward the research of captioning a single image to describing visual differences between image pairs . Suppose there are two images , I 1 and I 2 , and the task is to generate a description W 1,2 comparing them , existing methods directly model ⟨I 1 , I 2 ⟩ → W 1,2 mapping without the semantic understanding of individuals . In this paper , we introduce a Learningto-Compare ( L2C ) model , which learns to understand the semantic structures of these two images and compare them while learning to describe each one . We demonstrate that L2C benefits from a comparison between explicit semantic representations and singleimage captions , and generalizes better on the new testing image pairs . It outperforms the baseline on both automatic evaluation and human evaluation for the Birds-to-Words dataset . The task of generating textual descriptions of images tests a machine 's ability to understand visual data and interpret it in natural language . It is a fundamental research problem lying at the intersection of natural language processing , computer vision , and cognitive science . For example , single-image captioning ( Farhadi et al . , 2010 ; Kulkarni et al . , 2013 ; Vinyals et al . , 2015 ; Xu et al . , 2015 ) has been extensively studied . Recently , a new intriguing task , visual comparison , along with several benchmarks ( Jhamtani and Berg-Kirkpatrick , 2018 ; Tan et al . , 2019 ; Park et al . , 2019 ; Forbes et al . , 2019 ) has drawn increasing attention in the community . To complete the task and generate comparative descriptions , a machine should understand the visual differences between a pair of images ( see Figure 1 ) . Previous methods ( Jhamtani and Berg-Kirkpatrick , 2018 ) as the ResNet features ( He et al . , 2016 ) as a whole , and build end-to-end neural networks to predict the description of visual comparison directly . In contrast , humans can easily reason about the visual components of a single image and describe the visual differences between two images based on their semantic understanding of each one . Humans do not need to look at thousands of image pairs to describe the difference of new image pairs , as they can leverage their understanding of single images for visual comparison . Therefore , we believe that visual differences should be learned by understanding and comparing every single image 's semantic representation . A most recent work ( Zhang et al . , 2020 ) conceptually supports this argument , where they show that low-level ResNet visual features lead to poor generalization in vision-and-language navigation , and high-level semantic segmentation helps the agent Image I 1 Image I 2 V 1 V 2 Relation- enhanced features V g 1 V g 2 LSTM h t In this paper , we present a learning-to-compare framework for generating visual comparisons . Our segmentation encoder with semantic pooling and graph reasoning could construct structured image representations . We also show that learning to describe visual differences benefits from understanding the semantics of each image .

Challenge: To generate a different description between two images, in contrast to how humans reason, existing models do not understand the semantics of each image.
Approach: They propose a Learning-to-Compare model that can consider the semantic structures of each image using graph convolutional networks before comparing them and generating a description.
Outcome: The proposed graph-based learning-to-compare model outperforms the baseline models using automatic and human evaluations and also generalizes better.