KDMCSE : Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learning (2024)

Abstract: Previous work on multimodal sentence embedding has proposed multimodal contrastive learning and achieved promising results. However, by taking the rest of the batch as negative samples without reviewing when forming contrastive pairs, those studies encountered many suspicious and noisy negative examples, significantly affecting the methods’ overall performance. In this work, we propose KDMCSE (Knowledge Distillation Multimodal contrastive learning of Sentence Embeddings), a novel approach that enhances the discrimination and generalizability of multimodal representation and inherits the knowledge from the teacher model to learn the difference between positive and negative instances and via that, can detect noisy and wrong negative samples effectively before they are calculated in the contrastive objective. Furthermore, to overcome the limitation of modeling the variation within negative pairs, we introduce a new contrastive objective, AdapACSE (Adaptive Angular Margin Supervised Contrastive Learning for Multimodal sentence embeddings), that enhances the discriminative representation by strengthening the margin within the angular space while capturing varying semantics within the negative. Experimental results on widely used Semantic Textual Similarity (STS) benchmarks demonstrate the effectiveness of our approach.

## introduction learning sentence embeddings, which involves translating sentences into consistent-length vectors that capture their semantic connections, remains a pivotal task in the nlp field. while pretrained language models like bert @xcite and roberta @xcite have witnessed immense success, studies indicate that the default sentence embeddings from these plms, without specific fine-tuning, may not perform as well as simply averaging glove vectors @xcite in capturing semantic similarity @xcite . consequently, recent works @xcite @xcite are geared towards refining sentence embeddings from plms without supervision. while text-centric models have made significant strides, their depth of understanding of sentence semantics remains a point of contention. true semantic understanding often stems from associations in the real world rather than mere textual statistics @xcite @xcite . numerous recent studies have explored the enhancement of language representations through visual information @xcite @xcite . the recent multimodal contrast in sentence embedding learning @xcite offers an encouraging direction. this method augments simcse with a multimodal contrastive objective, which seeks congruence between sentences and their associated images within a unified space. however, samples drawn randomly from grounded datasets during training can present semantic differences, subtle similarities, or overlapping attributes. for example, in figure 1, two captions b.1 and b.2 describe the second image b, but also correctly represent the first image a. due to their inherent noise, such samples can hinder the learning process. mitigating this issue requires preemptively filtering out these semantically similar samples before loss calculation, ensuring a noise-reduced training environment. in addition, comparing two captions, a.1 and a.2, for the left image reveals that the latter offers a more comprehensive description, detailing aspects such as the individual's gender, attire, sneakers, and even the material of the stairs (concrete). thus, using only visual images cannot capture the depth of language representations. furthermore, mcse did not consider the distinctions between negative sample pairs while focusing on constructing pairs of positive and negative representations. as shown in figure 2 , although none of the captions perfectly describes the image, their levels of discrimination vary between pairs. on the contrary, the caption c registers a score of 0.49, hinting at its partial relevance to the image. caption d, with a very weak score of 0.03, is completely unrelated. consequently, within the feature space, both captions c and d should be substantially distanced from the accurate depiction. furthermore, the gap between the true caption and caption d should considerably exceed that between the true caption and the caption c. to address the aforementioned limitation of noisy negative sampling and complexity of linguis-tic representation, we present kdmcse: a knowledge distillation multimodal contrastive learning framework for sentence embeddings that utilizes clip @xcite as the teacher model for both image and text modalities. our approach harnesses images for multimodal objectives and textual representations derived from the clip text encoder. relying solely on images falls short of capturing the nuances in sentence representation. emulating the representation of the clip's textual encoder -especially when paired with image matching -can augment the richness of linguistic representations, transcending a mere alignment with visual features. moreover, in response to the identified challenge of diversity among negative pairs, we introduce adapacse: adaptive angular margin contrastive learning. this method bolsters discriminative representation by amplifying the margin within the angular space, specifically accounting for the various semantics present in negative samples. leveraging the teacher model, clip, we produce soft labels that signify the similarity between samples. instead of merely utilizing feature representations as positive or negative samples during the multimodal objective, we employ clip's soft labels to intensely penalize samples exhibiting low similarity and provide leniency to pairs exhibiting certain resemblances. our contributions can be summarized as follows. deep metric learning objectives contrastive learning @xcite has gained traction across multiple domains @xcite @xcite @xcite @xcite @xcite . renowned training objectives such as n-pair loss @xcite , triplet margin loss @xcite , and arccon @xcite are deeply rooted in metric learning principles. in supervised tasks, objectives that leverage softmax have shown efficacy, especially when they integrate class centers and impose penalties on the distances between deep features and their corresponding centers. among these, center loss @xcite , sphereface @xcite , cosface @xcite , and arcface @xcite stand out and are frequently used in computer vision and natural language processing applications. nonetheless, these loss functions primarily aim at classification tasks, rendering them unsuitable for regression labels. ar-ccse @xcite presents training objectives using additional margin tailored to enhance the discriminative prowess in pairwise relations and capture the entailment dynamics within triplet sentence structures. ## analysis 6.1 exploring the impact of angular margin in our adapacse loss function, the angular margin, represented as m, is critical to determining the discriminative capability. we conducted an experiment to better understand the impact of m by varying it in steps of 0.025 radians from 0.025 radians to 0.225 radians. the performance metrics averaged on all tasks are depicted in figure 7 . from the data, we discern an optimal performance when m is set to 0.125. any deviation from this optimal point, be it an increase or decrease, results in a performance drop. this aligns with our preliminary hypotheses: a smaller value of m could have a negligible effect, whereas an overly large m could negatively skew the representations within the fusion space. ## conclusion in this paper, we introduce kdmcse, a novel approach to sentence embedding learning that transfers knowledge from a vision and language model with a multimodal contrastive objective to not only align sentences and the corresponding image and text representations of the teacher model but also avoid noisy negative samples. additionally, our innovative contrastive objective, adapacse, ad-dresses the challenges associated with capturing nuanced differences within negative pairs, strengthening discriminative representation. our framework lays the groundwork for continued exploration in this domain, fostering advancements in vision and language applications.