PRISM : A New Lens for Improved Color Understanding (2024)

Abstract: While image-text pre-trained models, such as CLIP, have demonstrated impressive capabilities in learning robust text and image representations, a critical area for substantial improvement remains—precise color understanding. In this paper, we address this limitation by introducing PRISM, a simple yet highly effective method that extends CLIP’s capability to grasp the nuances of precise colors. PRISM seamlessly adapts to both recognized HTML colors and out-of-vocabulary RGB inputs through the utilization of our curated dataset of 100 image-text pairs, which can be effortlessly repurposed for fine-tuning with any desired color. Importantly, PRISM achieves these enhancements without compromising CLIP’s performance on established benchmarks. Furthermore, we introduce a novel evaluation framework, ColorLens, featuring both seen and unseen test sets that can be readily repurposed to assess a model’s precision in understanding precise colors. Our comprehensive evaluation and results demonstrate significant improvements over baseline models.

## introduction vision-language foundation models (vlms), such as contrastive language-image pre-training (clip) @xcite , learn transferable rich knowledge in a joint space for vision and language with remarkable zero-shot and few-shot capability in 2d visual recognition tasks such as classification @xcite , detection @xcite , retrieval @xcite , and text-conditioned image generation @xcite . recently, several techniques have been proposed to improve the finetuning stability of clip, enabling it to adapt and this figure illustrates the contrasts between standard color shades and the unique, brand-specific shades used by wellknown brands. the juxtaposition highlights the significance of precise color recognition in brand identity and consumer perception. the significance of precise color understanding resonates profoundly in practical domains, particularly in advertising and branding, where it plays a pivotal role in establishing brand recognition and influencing consumer perceptions. colors not only significantly influence consumer buying decisions, enhancing brand recognition and impacting visual appeal, but also evoke specific emotional and psychological responses crucial for brand identity. several brands have invested significantly in establishing brand identity by designing unique (or non-standard html) color palettes, creating a visual language that is instantly recognizable worldwide, as shown in figure 1 . failure to accurately recognize these unique shades in vision-language models would lead to significant shortcomings in downstream generation tasks (see (c) in figure 2 ), such as automated advertising or brand-related content creation. therefore, enhancing the color discernment capabilities of these models is not just a 1. brune top in beige 2. brune top in dark salmon 3. brune top in khaki 4. brune top in burly wood 5. brune top in snow 6. brune top in peru technical challenge, but a necessity for maintaining brand integrity in digital representations. as illustrated in figure 2 , when tasked with retrieving images based on exact rgb colors (e.g., d2b48c representing tan color), clip frequently struggles to accurately retrieve images that align with the specified color, particularly when colors exhibit subtle resemblances. this limitation not only impacts the performance of image retrieval tasks but also extends to downstream applications reliant on vlms, including image generation models, which face difficulties with generating images consistently adhering to the precise color palette. the direct fine-tuning of vlms for this purpose encounters inherent challenges, including the risks of overfitting and mode-collapse, primarily stemming from the limited availability of image-text pairs explicitly describing precise colors. in this work, we introduce prism, to address these limitations. at its core, our principal objective revolves around expanding the pre-trained representational domain, ensuring effective encapsulation of a one or more desired rgb color values, all the while retaining the vlm performance on established benchmarks. to achieve this, we meticulously construct a training set comprising 100 diverse and high-quality image-text pairs. we show that our curated training set can be seamlessly repurposed for fine-tuning, facilitating the implantation of any desired rgb triplet with remarkable ease. in order to enhance the efficiency of fine-tuning, especially with the constraint of a relatively small set of examples, we introduce explicit hard negatives and encourage the learning of a disentangled embedding for the desired color. for rgb triplets not recognized as standard html colors, we employ a rare-token lookup in the vocabulary @xcite . additionally, we construct a new benchmark colorlens that can be readily repurposed to measure a model's precision in understanding precise colors. our empirical findings demonstrate a significant enhancement over baseline models in retrieval tasks. ## method in this section, we first describe the construction of our repurposable training and testing datasets, then present our fine-tuning paradigm in detail. our primary objective is to enhance clip's nuanced understanding of colors by learning disentangled embeddings for the desired color using our curated small set of training examples, all while simultaneously preserving the semantic context of images and text. ## experiments in the section, we present a comprehensive evaluation of our proposed method, prism, on retrieval tasks using our newly introduced colorlens test splits specifically designed for assessing precision in color-based retrieval tasks. we evaluate our approach from both quantitative and qualitative perspectives. through ablation studies, we systematically dissect the contributions of each component within our framework, highlighting their individual effectiveness in enhancing the model's performance. our experiments consistently demonstrate the superiority of prism over state-of-theart methods, including clip and albef @xcite , both in fine-tuning and adapter tuning settings foot_2 . additionally, to provide a more comprehensive perspective, we extend the comparison to include models such as vit-l-14 and vit-b-32 for all the models. in tables 1 and 2 we compare prism against established methods across both seen and unseen col-orlens test splits. the evaluation is multi-faceted: the first column compares the original image-text matching performance using precision and rank metrics against a backdrop of 20 randomly selected negatives from the test set. the second column extends this challenge by using the entire test suite as potential negatives. the third column specifically targets color-based retrieval performance, introducing 'hard-negatives' that are identical in every aspect except for distinct differences in rgb color values of specific objects. these hard-negatives are crafted to vary in their deviation from the true color values, with smaller delta values (δ ă 1 ) and larger ones ( 1 ă δ ă 2 ) to escalate the retrieval difficulty (see section a in supplementary for more details). furthermore, in the final two columns, we increase the number of negatives from 27 to 64, testing the models' robustness under more challenging conditions. as we can see, while most baseline models, both in their zero-shot and fine-tuned forms, exhibit strong performance in standard image-text matching (as evidenced in the first two columns of the results), there is a noticeable tendency for direct finetuning to lead to overfitting. this is particularly evident in the performance dip observed from clip to its fine-tuned variant (ft clip) in traditional matching tasks. contrasting this, prism stands out by not only improving precision and ranking in the color-focused retrieval tasks but also maintaining robust performance in standard image-text matching. this clearly indicates prism's unique ability to enhance color-specific understanding while preserving the foundational semantics of the models. further strengthening our findings, similar trends of prism's efficacy are observed in the unseen test split. ablations ## conclusion we have presented prism, a novel and effective framework designed to address the critical challenge of precise color understanding. leveraging a carefully curated training dataset comprising 100 image-text pairs, prism enables the seamless implantation of any desired rgb color value while preserving the core performance of clip on established benchmarks. through the incorporation of explicit hard negatives, disentangled color embeddings, and rare-token lookup mechanisms, we have ensured the robustness and generalization of our approach. furthermore, we introduced the col-orlens benchmark, encompassing both seen and unseen test sets, which provides a comprehensive evaluation of a model's ability to understand pre