Distilling Knowledge from Text-to-Image Generative Models Improves Visio-Linguistic Reasoning in CLIP (2024)

Abstract: Image-text contrastive models like CLIP have wide applications in zero-shot classification, image-text retrieval, and transfer learning. However, they often struggle on compositional visio-linguistic tasks (e.g., attribute-binding or object-relationships) where their performance is no better than random chance. To address this, we introduce SDS-CLIP, a lightweight and sample-efficient distillation method to enhance CLIP’s compositional visio-linguistic reasoning. Our approach fine-tunes CLIP using a distillation objective borrowed from large text-to-image generative models like Stable-Diffusion, which are known for their strong visio-linguistic reasoning abilities. On the challenging Winoground benchmark, SDS-CLIP improves the visio-linguistic performance of various CLIP models by up to 7%, while on the ARO dataset, it boosts performance by up to 3%. This work underscores the potential of well-designed distillation objectives from generative models to enhance contrastive image-text models with improved visio-linguistic reasoning capabilities.

## introduction in recent years, multimodal models like clip @xcite have excelled in tasks such as zero-shot classification, image-text retrieval, and image-captioning @xcite @xcite . these models are also crucial components in various state-ofthe-art pipelines for tasks like segmentation and object detection @xcite @xcite . however, they struggle with visiolinguistic reasoning tasks, such as determining the spatial relationships between objects in an image @xcite . notably, clip's performance on the challenging winoground @xcite , a benchmark designed to assess visiolinguistic reasoning, is close to random chance. this shortcoming is attributed to clip's contrastive objective which prioritizes shortcuts for retrieval, and thus impacts its ability to understand finegrained object details and their positions @xcite . in contrast, text-to-image models like stable diffusion @xcite excel in visiolinguistic tasks, likely due to their text conditioning enhanceing semantic consistency in its crossattention maps @xcite . @xcite recently demonstrated this on the winoground benchmark, reliably matching captions to images with fine-grained spatial differences using denoising diffusion scores (see fig 1 ). similar results have been shown for other text-toimage models, including imagen @xcite , with almost all of these methods outperforming clip variants on the same tasks. while these works have shown the potential of using generative text-to-image models for visiolinguistic tasks, it remains computationally intensive. for instance, computing the denoising diffusion score for image-text matching involves multiple passes through a unet model (approximately 892m parameters) with varying noise levels and time-steps. on an entry-level gpu, this can take up to a minute for a single image-text matching task, making it impractical for real-world and real-time applications. in contrast, clip models can classify images up to 18 times faster (see fig 1 ), requiring only one pass through both image and text encoders. a promising research direction, therefore, lies in finding methods that combine the strong visio-linguistic capabilities of text-to-image models with the rapid inference of clip. to this end, we introduce sds-clip, a lightweight and sample-efficient fine-tuning approach for clip which distills knowledge from stable diffusion, and enhances clip's visio-reasoning capabilities. specifically, we add a regularization term to clip's standard contrastive loss based on score-distillation sampling (sds) @xcite . this regularization encourages clip's embeddings to be aligned with the denoising diffusion loss from a text-to-image model. by fine-tuning clip with this regularized objective on a small paired image-text dataset, specifically 118k imagetext pairs from ms-coco, we demonstrate an 1.5-7% performance gain compared to vanilla clip on winoground and aro, two highly challenging visio-linguistic reasoning benchmarks. notably, this is achieved by only updating clip's layer-norm parameters. furthermore, we show that sds-clip's zero-shot performance is not impacted on a wide range of downstream datasets. in summary, our contributions are as follows: the winoground benchmark establishes a challenging image-text matching task to measure a model's visio-linugistic reasoning abilities: given an image x, the model must match it with the correct caption c * from a set of captions @xmath0, where all caption contains the same words but each describes a different spatial arrangement of the objects, with only one being correct. concurrent works @xcite @xcite to this paper have showed that it is possible to use the denoising diffusion score from text-to-image generative models to perform such an image-matching task. this can be formalized as follows: for an image x and caption c, the denoising diffusion score, denoted by d(x, c), is defined as: ## sds-clip: our method the core idea of our approach is to regularize the contrastive objective in clip with the denoising diffusion score from stable diffusion (see eq.( 1 )). our method builds on the recent work of @xcite which maps the output of a 3d nerf model into the input space of stable diffusion's unet and optimizes its parameteres with the denoising diffusion loss, also known as the scoredistillation sampling (sds). in a similar vein, we fine-tune the parameters of clip using sds. intuitively, our set-up can be viewed as a form of knowledge distillation where the teacher is the textto-image model and the student is clip. as a result, in inference, clip can benefit from the visiolinguistic reasoning capabilities that are already learned by text-to-image diffusion models. formally, we map the output of clip's image encoder to the input space of stable diffusion's unet. specifically, we pass a given image x through ## experiments in this section, we empirically validate our proposed method sds-clip on two types of tasks: i) visio-linguistic reasoning using two challenging benchmarks (winoground, aro) and ii) zero-shot image classification using a suite of downstream datasets (imagenet, cifar-100, and others). overall, we show that our method improves clip's performance significantly on winoground and some key tasks in aro, while also marginally improving downstream zero-shot classification performance. implementation details. due to computational limit, we fine-tune clip from a publicly available checkpoint instead of training from scratch. notably, we only fine-tune clip's layernorm parameters following @xcite along with the linear transformation h w -accounting for only ≈ 8m trainable parameters. we finetune these parameters using image-text pairs from mscoco @xcite . in particular, we choose mscoco as it is relatively small and less noisy than other image-text datasets such as cc-12m @xcite . both these factors make our fine-tuning method extremely sample-figure 2 : our fine-tuning method does not harm the zero-shot abilities of clip. in fact for certain downstream datasets (e.g., imagenet, cifar-10, mnist, aircraft) -we observe an improvement in the zero-shot performance between 1% -8% for vit-b/16. for other clip models, we find no drop in zero-shot performance. and parameter-efficient. baselines. we compare our method with two different baselines: (i) pre-trained (vanilla) clip checkpoints; and (ii) clip fine-tuned on ms-coco with the standard contrastive loss without the regularization term. ## related works while clip models @xcite are renowned for their robust zero-shot classification, recent research @xcite has exposed their limitations in visio-linguistic reasoning. in contrast, recent studies have demonstrated that text-to-image models @xcite @xcite outperform clip in reasoning tasks. these models in fact leverage scores computed from the diffusion objective. we note that while @xcite use scoredistillation sampling for text to 3d generation, ours is the first work to adapt the formulation as a regularizer and improve compositional abilities in clip. ## conclusion our paper introduces sds-clip, a novel data and parameter-efficient method that effectively enhances clip's visio-linguistic reasoning abilities by distilling knowledge from text-to-image models, without compromising its zero-shot abilities.