Fine-tuning CLIP Text Encoders with Two-step Paraphrasing (2024)

Abstract: Contrastive language-image pre-training (CLIP) models have demonstrated considerable success across various vision-language tasks, such as text-to-image retrieval, where the model is required to effectively process natural language input to produce an accurate visual output. However, current models still face limitations in dealing with linguistic variations in input queries, such as paraphrases, making it challenging to handle a broad range of user queries in real-world applications. In this study, we introduce a straightforward fine-tuning approach to enhance the representations of CLIP models for paraphrases. Our approach involves a two-step paraphrase generation process, where we automatically create two categories of paraphrases from web-scale image captions by leveraging large language models. Subsequently, we fine-tune the CLIP text encoder using these generated paraphrases while freezing the image encoder. Our resulting model, which we call ParaCLIP, exhibits significant improvements over baseline CLIP models across various tasks, including paraphrased retrieval (with rank similarity scores improved by up to 7.6% and 9.6%), Visual Genome Relation and Attribution, as well as seven semantic textual similarity tasks.

## introduction contrastive language-image pre-training (clip) models @xcite have gained significant attention in the fields of computer vision and natural language processing for their remarkable capacity to understand the relationship between text and images. they have been widely used in various vision-language applications, including image classification @xcite , image retrieval @xcite , and text-toimage generation @xcite , where the model should return desired visual outputs for a given text, and vice versa. ## method our objective is to refine the clip model's training process, enabling its text encoder to produce consistent representations for various semantically similar textual inputs that the model might encounter in real-world scenarios. certain image-captioning datasets provide multiple captions for a single image @xcite , which might be utilized as semantically similar text pairs during training. however, the volume of these datasets is limited, which presents a challenge in terms of exposing models to diverse language patterns. therefore, we automatically generated semantically similar pairs (i.e., paraphrases) for millions of image captions sourced from the web. ## experimental setups we obtained image-caption pairs using laion-400m @xcite . we initially generated 300k paraphrases using chatgpt and instruction-tuned an open-sourced llm named llama (7b) @xcite using these 300k data to generate additional paraphrases. 1 our final dataset comprises 5m examples of x i , x t , x ′ t , and x ′′ t . more details and hyperparameters are described in appendix a. ## conclusion in this study, we proposed a two-step paraphrasing approach for enhancing the representations of clip for paraphrases that may occur in text inputs in realworld applications. our paraclip models, finetuned using synthetic paraphrases, outperformed baseline models by a large margin on various tasks requiring language semantics and compositional understanding, including paraphrased retrieval.