Getting More Juice Out of Your Data: Hard Pair Refinement Enhances Visual-Language Models Without Extra Data (2025)

Abstract: Contrastive Language-Image Pre-training (CLIP) has become the standard for cross- modal image-text representation learning. Improving CLIP typically requires additional data and retraining with new loss functions, but these demands raise resource and time costs, limiting practical use. In this work, we introduce HELIP, a cost-effective strategy that improves CLIP models by exploiting challenging text-image pairs within existing datasets in continuous training. This eliminates the need for additional data or extensive retraining. Moreover, HELIP integrates effortlessly into current training pipelines with minimal code modifications, allowing for quick and seamless implementation. On comprehensive benchmarks, HELIP consistently boosts existing models. In particular, within just two epochs of training, it improves zero-shot classification accuracy on ImageNet for SLIP models pre-trained on CC3M, CC12M, and YFCC15M datasets by 3.05%, 4.47%, and 10.1% , respectively. In addition, on fine-grained classification datasets, HELIP improves the zero-shot performance of CLIP and SLIP by an average of 8.4% and 18.6%, and their linear probe performance by an average of 9.5% and 3.0%.

## introduction contrastive language-image pretraining (clip) @xcite is quickly becoming the standard for foundation models @xcite due to its effectiveness for a variety of vision-language tasks without task-specific finetuning @xcite . * equally contributed to this work, † corresponding author. § affiliated with department of systems engineering and engineering management, and shun hing institute of advanced engineering, the chinese university of hong kong, hong kong however, web-crawled image-text pairs used for the clip model pretraining are often loosely connected, resulting in multiple plausible matches beyond the assigned ones @xcite . several methods have been presented to improve clip models by investigating appropriate matches and utilizing widespread supervision among image-text pairs for training @xcite @xcite . efforts to improve contrastive language-image pretraining models have primarily taken two directions: (1) the addition of objectives to improve the efficacy of supervision @xcite ; and (2) the employment of intra-and inter-modality similarity to select and retrain models using data deemed challenging at the sample level @xcite . however, those approaches inevitably require retraining, and those identified as challenging data struggle to bring benefits to model performance. this challenge is partly due to their reliance on finding challenging data within a single batch during training, where truly beneficial challenging data is rare. and, clip models' original contrastive loss is not optimally configured to exploit the nuances of difficult data. these limitations restrict the practical application of these methods, especially considering the substantial investments already made in pretraining numerous clip models @xcite ; retraining for minimal gains is inefficient. this aspect underscores the need for efficient enhancement strategies that do not rely on additional data collection to improve existing pretrained models. to improve the existing clip models, we introduce the helip framework, which involves further training the models with challenging data selected from their original training dataset. helip defines and identifies the challenging data at the pair level, distinguishing it from traditional methods that compare sample-level similarities between images and texts. specifically, helip treats each text-image pair as a distinct entity within the joint vision-language space, and defines pairs in close proximity as hard pairs. furthermore, helip introduces the hard pair mining (hpm) strategy, a novel approach that moves beyond the traditional use of representation spaces learned by clip models. note, the clip space is primarily designed for evaluating sample-level similarities-for instance, comparing an image and text (individually, not as a pair)-lacking in evaluating characteristics at the pair level. hpm transforms the task of discovering pairs in close proximity into a solvable proxy task, with the goal of selecting a pair set that optimally supports the target pair's text-image agreement. helip enhances clip models not just with the original text-image contrastive loss @xcite , which uniformly pushes all negative samples away from their positive counterpart but also incorporates the hard negative margin loss (hnml) into the loss function. as depicted in figure 2 , hnml imposes an additional geometric structure on the representation space, reflecting the pair-level similarity. through this approach, helip effectively leverages the information within challenging data to boost model performance. empirical evidence shows that helip improves the performance of existing clip models, including pre-trained clip, slip, and declip, across a variety of benchmarks, such as zero-shot classification, text-image retrieval, and fine-grained linear probing. for zero-shot classification on imagenet, cifar-10, and cifar-100, helip consistently boosts the performance of all six pre-trained models. particularly, using helip to boost slip models pre-trained on cc3m, cc12m, and yfcc15m results in imagenet zero-shot accuracy gains of 3.05%, 4.47%, and 10.14%, respectively. further, on seven fine-grained image classification datasets, those pre-trained models achieve better zero-shot and linear probe performance with helip. specifically, the average zero-shot accuracy of cc3m pre-trained clip and slip are improved by 8.4% and 18.6%. the average linear probe accuracy of cc3m pre-trained clip and slip are improved by 9.5% and 3.0% respectively. additionally, the performance gain is also valid in terms of zero-shot retrieval, with 1.1 of r@1 on flickr30k, and 2.2 of r@1 on coco for slip with helip. ## related work vision-language pre-training. vision-language pretraining (vlp) leverages large-scale image-text datasets to learn joint representations transferable to downstream tasks. vlp models are typically classified into single-stream and dual-stream architectures. single-stream models concatenate text and visual features processed by a single transformer @xcite @xcite . dual-stream models use separate encoders for image and text, performing crossmodal interactions at a higher level @xcite @xcite @xcite . clip @xcite , a dual-stream model, employs contrastive learning with 400m web-crawled image-text pairs to achieve remarkable zero-shot recognition performance. recent works enhance clip's performance by applying self-supervision within the visual modality @xcite or incorporating nearest neighbor supervision @xcite . while these methods improve performance, they introduce additional computational costs due to data augmentations. contrastive learning with hard negative samples. contrastive learning aims to learn representations by bringing similar examples closer and pushing dissimilar ones apart @xcite . incorporating hard negative samples into the loss function has been shown to improve performance @xcite @xcite @xcite @xcite . in language-image contrastive learning, approaches like @xcite and @xcite mine hard negatives using intra-or intermodality similarity, selecting samples with high cosine similarity in visual or textual features. however, due to the loose alignment in web-crawled data, high similarity in these features doesn't necessarily indicate that pairs are difficult to distinguish. in contrast, we propose a hard sample mining method that discovers similar pairs in the joint vision-language space, efficiently selecting truly challenging samples to improve learning. negative margin loss (hnml), designed to efficiently exploit hard pairs. ## experiments in section 4.2, we empirically investigate he-lip's efficacy in improving zero-shot classification, image-text retrieval, and linear probing performance of existing vision-language models. section 4.3 examines helip's performance with scaled training data. we provide in-depth studies on hard positive mining (hpm) and hard negative mining with margin loss (hnml) in sections 4.4 and 4.5, respectively. discussions on robustness over noisy datasets and additional empirical analyses of hard mining methods are deferred to the appendix. ## conclusion in this work, we present helip, a framework that enhances pre-trained clip models by more effectively utilizing their original training datasets. helipoffers a cost-effective and easily integrable solution for improving existing models without extensive retraining or additional data. specifically, it treats each text-image pair as a point in the joint vision-language space and identifies hard pairs, those that are close together, using the hard pair mining (hpm) strategy. furthermore, to efficiently leverage these hard pairs, we introduce the hard negative margin loss (hnml). empirically, we found that helipboosts the performance of existing checkpoints within a few epochs of continuous training. evaluations across various benchmarks, including zero-shot classification, imagetext retrieval, and linear probing, demonstrate the effectiveness and efficiency of our method. these findings highlight that in the era of large-scale models and datasets, performance improvement can be achieved not only by collecting more data or scaling up models, but also by intelligently maximizing the utility of the data we already have.