Enhancing Conceptual Understanding in Multimodal Contrastive Learning through Hard Negative Samples (2024)

Abstract: Current vision-language models leveraging contrastive learning often face limitations in developing fine-grained conceptual understanding. This is due to random negative samples during pretraining, causing almost exclusively very dissimilar concepts to be compared in the loss function. Consequently, the models struggle with fine-grained semantic differences. To address this problem, we introduce a novel pretraining method incorporating synthetic hard negative text examples. The hard negatives replace terms corresponding to visual concepts, leading to a more fine-grained visual and textual concept alignment. Further, we introduce InpaintCOCO, a new challenging dataset for assessing the fine-grained alignment of colors, objects, and sizes in vision-language models. We created the dataset using generative inpainting from COCO images by changing the visual concepts so that the images no longer match their original captions. Our results show significant improvements in fine-grained concept understanding across various vision-language datasets, including our InpaintCOCO dataset.

## introduction recent advancements in vision-language (vl) modeling have demonstrated the effectiveness of contrastive learning in various multimodal tasks @xcite @xcite . however, this training method does not provide sufficient training signals for several important visual concepts @xcite . we attributed it to the objective function's use of random and, therefore, too dissimilar negative samples, which prevents the model from learning fine-grained semantic representations of the concepts. therefore, we propose a novel approach to address the issue of poorly represented concepts in contrastive learning. we introduce a mechanism to incorporate hard negative samples into the contrastive learning loss. specifically, we generate synthetic hard negative samples by substituting keywords in the captions of original image-text pairs, disrupting the alignment between the image content and its description. this paper presents three key contributions: in this work, we conduct extensive evaluations of four basic concepts -color, object, location, and size. these concepts were selected as examples to demonstrate the effectiveness and robustness of our proposed approach in capturing nuanced semantic relations, but it is important to note that the choice of concepts is flexible and can be tailored to specific applications. furthermore, our methodology is easy to construct, requiring only minimal domain expertise and the simple usage of regular expressions. this study shows that simple tweaks in contrastive learning can significantly enhance multimodal understanding and model performance. i1 t pos 1 t neg 2 t neg 3 t hn 1 a white cat sits under a black open umbrella. a person standing on a loading platform next to a train. a man leans near the door of a creatively painted tour bus. a brown cat sits under a black open umbrella. figure 1 : classical contrastive learning approaches use (i 1 , t pos 1 ) as positive pairs in combination with negative samples like t neg 2 and t neg 3 to learn an image-text alignment. a bag of words (e.g., nouns) is often sufficient to extract the correct text that matches a given image, resulting in only broad concepts learned. we also use hard negatives like t hn 1 so that fine-grained semantic concepts are learned for visual and textual alignment. insert hard-negatives at random position for u i a small yellow bird on a branch o a tree. ## vision-language representation learning contrastive learning. the objective of contrastive representation learning is to learn representations that are close to each other for similar samples and distant from each other for dissimilar samples. while many objectives originally addressed a single modality @xcite @xcite , the idea can also be extended to multimodal training as well. a successful example of multimodal learning is clip @xcite . clip is a transformerbased model that consists of an image encoder and a text encoder, which are trained simultaneously. the objective is to maximize the cosine similarity of the image and text embeddings from the correct image-text pairs and to minimize the similarity between the incorrect pairs. a batch of n training samples (i.e., matching image-text pairs) results in a similarity matrix for each image-text combination. the main diagonal indicates the correct pair matches; the remaining entries correspond to negative entries. the symmetric cross-entropy loss is applied on n × n similarity scores. this heuristical construction of negative samples has several issues. negative captions can still match the given image in some cases, especially if the text is short and lacks details. additionally, the negative pairs are often very dissimilar, which causes the model to decide only on coarse-grained features. we illustrate this in figure 1 with text-image pair i 1 and t pos 1 . the original learning process only uses negative text samples, such as t neg 2 and t neg 3 . here, it is sufficient that the model can assign or negate objects from texts (i.e., nouns) to the objects in the image. the fact that no "person" and no "door" are present in image i 1 is sufficient for the model to discard these image descriptions. as a result, fine-grained concepts like object-color alignment, size, or spatial details ("cat under umbrella") are not necessary for reaching low loss. in this example, the model can rely solely on the presence and absence of specific objects. in this scenario, the caption can be seen as a bag of words without linguistic structure. we address this problem by creating new hard negative data samples to learn more fine-grained concepts. see § 3 for details. related work. several approaches incorporate hard negative samples in multimodal learning. @xcite use importance sampling (based on @xcite , which up-samples hard negative samples and down-samples or ignores simple negatives. yet, they reweight the simple negative samples per batch only but do not create new, challenging samples requiring fine-grained understanding. @xcite propose keyword permutation to create hard negative samples to learn spatial concepts. they added a spatial understanding classifier as an auxiliary pretraining objective and evaluated the models on visual question answering (vqa). their model is based on lxmert @xcite and not on a contrastive learning approach like clip. @xcite generate hard negatives using a rule-based procedure where they replace keywords. moreover, they implement an approach where they randomly parse parts of speech and fill the mask using a bert encoder with a plausible but wrong word. unlike us, they do not incorporate hard negatives into the similarity matrix but use an auxiliary loss summed with the original contrastive loss. they use four distinct loss functions in total, introducing an additional layer of complexity to the overall procedure. in contrast, our approach uses the original loss function, with minimal modifications limited to the text inputs. ## contrastive learning with hard negatives we present a novel contrastive learning approach using sampled negative pairs and artificially generated textual hard negatives. instead of training with one positive sample and several weak negative samples, we create a scenario where models also minimize the similarity to hard negative textual samples. this forces the model to learn fine-grained concepts during training. we use keyword substitutions for different concepts to break the correct meaning of an image caption. see figure 1 for an example of the concept color. as a result, the new caption still lists, e.g., correct objects (e.g., "cat" and "umbrella") and actions (e.g., "cat sits") from the image but is no longer correct. we call this a "hard negative sample". using these samples during training, we ensure that fine-grained concepts are learned. the idea to inject hard negative samples in contrastive learning is highlighted in figure 2 . ## inpaintcoco: challenge set from the visual perspective many multimodal tasks, such as vl retrieval and visual question answering, present results in terms of overall performance. unfortunately, this approach overlooks more nuanced concepts, leaving us unaware of which specific concepts contribute to the success of current models and which are ignored. more recent benchmarks attempt to assess particular aspects of vision-language models in response to this limitation. some existing datasets focus on linguistic concepts utilizing one image paired with multiple captions; others adopt a visual or cross-modal perspective. in this study, we are particularly interested in fine-grained visual concept understanding, which we believe is not covered in existing benchmarks in sufficient isolation. therefore, we create the inpaintcoco dataset with image pairs with minimum differences that lead to changes in the captions. ## experiments we run several experiments to evaluate the proposed method. we compare the original openai model (orig.), with continued pretraining clip model using the classical contrastive learning approach (clas.) and our method with 1 up to 3 hard negative values per batch (hn1, hn2, hn3). we measure both how well concepts are learned and whether the general image retrieval capability of the model changes on the coco dataset ( § 5.1). additionally, we evaluate the method using our in-paintcoco challenge set ( § 5.2), and several other datasets ( § 5.3). ## conclusion we introduce a robust method for enhancing finegrained concept understanding with minimal impact on general retrieval capabilities using hard negative sampling in contrastive learning. we show that various concepts can be learned efficiently with minor text input adjustments. moreover, improvements in concept understanding are observable after continued pretraining on only 10% of our data. furthermore, one hard negative sample per image in a batch of 64 proves sufficient to incorporate the concept of interest into the model. we comprehensively evaluate our method on several datasets, including our new challenge set. our method outperforms classical contrastive learning on all investigated concepts. existing datasets often focus on linguistic perturbations or use dissimilar images, precluding a structured evaluation of permuted visual concepts in isolation. to address this gap, inpaintcoco represents the first dataset adjusting minor image parts in a controlled setting, facilitating cross-model fine-grained understanding. this ensures that the model's output is influenced only by one object and not the rest of the scene. the results show that fine-grained concept understanding also generalizes to images of different styles when using inpaintcoco and domainspecific datasets. our method is data-efficient and requires only a little domain knowledge to design the hard negatives. this makes it particularly suitable for domain adaptation in image retrieval, as well as for developing new clip-based models, e.g., for object detection @xcite .