Updating CLIP to Prefer Descriptions Over Captions (2024)

Abstract: Although CLIPScore is a powerful generic metric that captures the similarity between a text and an image, it fails to distinguish between a caption that is meant to complement the information in an image and a description that is meant to replace an image entirely, e.g., for accessibility. We address this shortcoming by updating the CLIP model with the Concadia dataset to assign higher scores to descriptions than captions using parameter efficient fine-tuning and a loss objective derived from work on causal interpretability. This model correlates with the judgements of blind and low-vision people while preserving transfer capabilities and has interpretable structure that sheds light on the caption–description distinction.

## introduction the texts that accompany images online are written with a variety of distinct purposes: to add commentary, to identify entities, to enable search, and others. one of the most important purposes is (alt-text) description to help make the image non-visually accessible, which is especially important for people who are blind or low-vision (blv) @xcite @xcite . the ability to automatically evaluate descriptions of images would mark a significant step towards making the web accessible for everyone. unfortunately, present-day metrics for imagetext similarity tend to be insensitive to the text's purpose @xcite , as they don't distinguish between accessibility descriptions that are intended to replace the image from captions, which supplement them (see figure 1 ). thus current metrics fall short when it comes to making genuine progress towards accessibility @xcite . the contrastive language-image pre-training (clip) model of @xcite is an impor-figure 1 : visualization of a single training step on concadia, updating clip to prefer descriptions to captions. though both objectives update clip to be sensitive to the description-caption distinction, iit-das localizes this distinction to a subspace of clip's activations. the clipscore metric of @xcite ) inherits this limitation; clipscore is referenceless in that it makes no use of human-generated, groundtruth texts, but rather depends only on image-text pairs, and so it too is not sensitive to the purpose of the text. indeed, @xcite find that clipscores correlate with neither sighted nor blv user evaluations of image alt-descriptions. thus, despite high performance on many image-text classification tasks, clipscore is unsuitable for alt-text evaluation. the goal of this paper is to update clip to assign higher scores to descriptions than captions when they are both relevant to an image, while preserving the model's ability to select the most relevant text for a particular image. to this end, we fine-tune clip on the concadia dataset @xcite , which consists of 96,918 images with corresponding descriptions, captions, and textual context. our core goal in fine-tuning is to teach the model to prefer descriptions over captions. to this end, we use a contrastive loss objective, which updates clip to produce a higher score for the description than the caption of an image in concadia. in addition, we propose an extension of this objective that seeks to amplify the core distinction and create more interpretable models. the guiding idea here is to use concadia to approximate the counterfactual: what if the text for this image were a description instead of a caption (or vice versa)? these counterfactuals enable us to use a novel combination of two recent ideas from causal interpretability research: interchange intervention training (iit; @xcite with a distributed alignment search (das; @xcite ) to localize the description-caption concept to an activation vector. our experiments lead to a few key findings that have implications beyond the accessibility use case. first, we find that lora @xcite is superior to standard fine-tuning at raising the clip-score assigned to descriptions compared to captions while preserving the original capabilities of clip. second, our analysis shows that improved performance on concadia results in stronger correlations with blv user judgements, affirming the value of our update. third, we find that the iit-das objective results in a more stable fine-tuning process. fourth, the iit-das objective produces a more interpretable model; to show this, we use mediated integrated gradients @xcite to characterize how the description-caption distinction is computed in our fine-tuned models. the key role of the iit-das objective in these results illustrates one way that interpretability research can lead directly to more performant and understandable models. image captioning models provide an opportunity to generate such accessibility descriptions at scale, which would promote equal access @xcite . but the resulting models have remained largely unsuccessful in practice @xcite @xcite . @xcite argue that this is partly due to the general approach of treating all image-based text generation problems as the same underlying task, and instead highlight the need for a distinction between accessibility descriptions and contextualizing captions. descriptions are needed to replace images, while the purpose of a caption is to provide supplemental information. @xcite find that the language used in descriptions and captions categorically differs and that sighted participants tend to learn more from captions but can visualize the image better from descriptions. ## methods our goal is to fine-tune a clip model c θ to prefer a description over a caption when the two texts are relevant to an image, while preserving c θ 's ability to select the most relevant text for a particular image. to this end, we use concadia, a dataset that consists of (x image , x description , x caption ) triplets. we consider two different contrastive learning objectives and two common fine-tuning methods that minimally update c θ 's sensitivity to the descriptioncaption distinction. ## conclusion limitations our results serve as proof concept for using iit-das to update clip with the concadia dataset. this is one model and one dataset, so general conclusions about the use of iit-das for updating a pretrained model should not be drawn. we hope future work will shed further light on the value of iit-das. the concadia dataset provides textual context for each image-description-caption triple. we do not use the context in our experiments, but we are excited about future work that incorporates this data. whereas our work focuses on the specific purposes of describing and captioning an image, the context of an image can illuminate many other purposes (e.g. search, geolocation, social communication) and models that incorporate it can enrich our work.