CLIP 4 IDC : CLIP for Image Difference Captioning (2022)

Abstract: Image Difference Captioning (IDC) aims at generating sentences to describe differences between two similar-looking images. Conventional approaches learn an IDC model with a pre-trained and usually frozen visual feature extractor. Accordingly, two major issues may arise: (1) a large domain gap usually exists between the pre-training datasets used for training such a visual encoder and that of the downstream IDC task, and (2) the visual feature extractor, when separately encoding two images, often does not effectively encode the visual changes between two images. Due to the excellent zero-shot performance of the recently proposed CLIP, we thus propose CLIP4IDC to transfer a CLIP model for the IDC task to address those issues. Different from directly fine-tuning CLIP to generate sentences, we introduce an adaptation training process to adapt CLIP’s visual encoder to capture and align differences in image pairs based on the textual descriptions. Experiments on three IDC benchmark datasets, CLEVR-Change, Spot-the-Diff, and Image-Editing-Request, demonstrate the effectiveness of CLIP4IDC.

## introduction tasks involving understanding and expressing visual contents are hard for machines because modelling relationships between the visual and textual domains requires sophisticated computational reasoning. as one of the tasks, image captioning (ic) @xcite aims at generating a coherent description given an image. extended from image captioning, image difference captioning (idc) @xcite describes the subtle changes that appear in a pair of two similar images. it is more challenging as a machine is required to recognize both visual objects and nuances in the pair. a conventional approach to idc is shown in figure 1a . first, the visual features of an image pair are extracted offline with pre-trained models @xcite . then a captioning network generates sentence(s) to describe the changes in the pair. even though such approaches have made great progress @xcite @xcite @xcite , they suffer from the fact that the visual features do not account for the domain gap between the pre-training and idc tasks. @xcite demonstrated that the purpose of the feature extractor trained on the original task introduces a gap with that of the subsequent tasks. for example, the features extracted by models trained on image classification task focus on high-level context and lose fine-grained information required for idc. moreover, the extracted visual representations of single modality are uncorrelated with the textual ones. as an effective approach to deal with the drawbacks, fine-tuning models on the target dataset narrows the gap between the tasks. @xcite showed that a transformer @xcite model that was pre-trained and fine-tuned on the same offline-extracted features achieves state-ofthe-art results in idc. however, it does not yet fully exploit the knowledge from the large-scale dataset as in the recent advancements in vision-language (vl) pre-training @xcite (vlp). in particular, clip @xcite , a contrastive vlp model has demonstrated its zeroshot superiority in numerous vl downstream tasks @xcite . we set out experimenting with a typical clip fine-tuning strategy on the idc task as shown in figure 1b , where clip's visual encoder is learned and fine-tuned on raw pixels. however, gaps still exist not only between the objectives of clip pretraining and idc, but also between the collected image-text pairs for pre-training and the image difference pairs in idc. these gaps throttle the model in adapting for the idc task. to tackle these problems, we study how to efficiently transfer a pre-trained clip for idc. the overview of the proposed clip4idc model is shown in figure 2 . compared to directly finetuning clip for the idc task, clip4idc employs "adapt-and-fine-tune" strategy. to adapt, the clip encoder learns to capture the fine-grained differences in the image pair rather than to produce only high-level semantic information separately for these two images. the visual and textual representations for the image pairs and the sentences are learned to be aligned with a retrieval loss in this stage. to fine-tune, the learned vision encoder is followed by a captioning transformer trained from scratch. extensive experiments are conducted on synthetic and real benchmark datasets clevr-change @xcite and spot-the-diff (jhamtani and berg-kirkpatrick, 2018), respectively. in addition, results on image-editing-request (tan et al., 2019), a mixed real-synthetic dataset, are also reported. clip4idc outperforms the strong baselines on all the metrics on these three datasets. the main contributions of this work are: ## clip4idc as shown in figure 1a , the canonical idc approach generates sentences on pre-extracted features. the bottleneck lies in three aspects: 1) the stopped gradient flow in the feature extraction, 2) the mismatched objectives and data domains between the pre-training and idc fine-tuning, and 3) the visual features being "purely visual", i.e. they reside in the visual domain, far apart from the textual domain. in the following sections, we introduce clip4idc, a clip-based approach to address these bottlenecks. ## assessments of idc adaptation we study how the retrieval accuracy is affected by different architectural options in clip4idc on clevr-change test split. to further study the relationships between the retrieval-based adaptation and the captioning accuracy, we fine-tune the models from the adaptation stage on the captioning task with the frozen image encoder. it can be observed in ## conclusion and future work in this work, we studied how to fine-tune clip for image difference captioning. retrieval-based adaptation was introduced to improve the visual representations for captioning and to narrow the gap between the purposes and data domains of clip pre-training and idc. experimental results demonstrated the effectiveness of the clip4idc model and the applied domain adaptation. in the future work, we will further explore enhancing the relationships between the vision and language domains. specifically, clip4idc adapts clip which does not involve cross-modal interactions as early as other pre-trained vl models @xcite @xcite that allow the interactions from the ground up. adapting other vl models for idc is naturally one interesting future direction. moreover, exploring other means than our contrastive approach, such as domain confusion @xcite , to bridge vision and language domains is another plausible direction. @xcite is a synthetic dataset generated by clevr engine. geometric differences between the objects in the images are annotated. it is divided into the training, validation and test splits which have 67,660, 3,976 and 7,970 image pairs, respectively. spot-the-diff @xcite describes multiple scene changes in the real 13,192 image pairs sampled from the virat ground video dataset with human-annotated captions. on an average, there are 1.86 sentences to describe the differences for each image pair. two decoding strategies containing single-sentence decoding and multi-sentence decoding are set for captioning. following jhamtani and berg-kirkpatrick (2018), we evaluate models in the single-sentence decoding by setting the ground truth description as multiple reference captions. @xcite is a dataset consisted of camera shots, paintings and animations, and most of the images are realistic. it contains 3,939 image pairs with instructions written by human annotators.