  Recent advances in language and vision push forward the research of
captioning a single image to describing visual differences between image pairs.
Suppose there are two images, I_1 and I_2, and the task is to generate a
description W_{1,2} comparing them, existing methods directly model { I_1, I_2
} -> W_{1,2} mapping without the semantic understanding of individuals. In this
paper, we introduce a Learning-to-Compare (L2C) model, which learns to
understand the semantic structures of these two images and compare them while
learning to describe each one. We demonstrate that L2C benefits from a
comparison between explicit semantic representations and single-image captions,
and generalizes better on the new testing image pairs. It outperforms the
baseline on both automatic evaluation and human evaluation for the
Birds-to-Words dataset.
