Bootstrap Your Own PLM : Boosting Semantic Features of PLM s for Unsuperivsed Contrastive Learning (2024)

Abstract: This paper aims to investigate the possibility of exploiting original semantic features of PLMs (pre-trained language models) during contrastive learning in the context of SRL (sentence representation learning). In the context of feature modification, we identified a method called IFM (implicit feature modification), which reduces the tendency of contrastive models for VRL (visual representation learning) to rely on feature-suppressing shortcut solutions. We observed that IFM did not work well for SRL, which may be due to differences between the nature of VRL and SRL. We propose BYOP, which boosts well-represented features, taking the opposite idea of IFM, under the assumption that SimCSE’s dropout-noise-based augmentation may be too simple to modify high-level semantic features, and that the features learned by PLMs are semantically meaningful and should be boosted, rather than removed. Extensive experiments lend credence to the logic of BYOP, which considers the nature of SRL.

## introduction contrastive learning has been successfully adopted in the field of vrl by constructing contrastive pairs (drawing positive pairs and repelling negative pairs) based on the sufficient background of augmentation strategies @xcite . after that, srl (sentence representation learning) followed the literature established by the baseline simcse @xcite , which proposed to construct contrastive pairs based on dropout-noise. recent studies have generally confirmed the effectiveness of this method @xcite @xcite . one interesting point is that simcse significantly improves the performance of plms (pretrained language models) on the sentence representation benchmark, named sts benchmark @xcite where plms showed poor performance before the introduction of simcse. at the same time, vanilla plms have shown comparable or even better performances on several transfer tasks than plms trained by simcse. we also observed these performance trends, each reported in based on these empirical results, we hypothesize that plms indeed learn several well-represented features, considering their success in the transfer tasks even without the contrastive framework proposed by simcse. and such meaningful features would be utilized in contrastive learning of simcse, which may partly contribute to the performance improvement in the sts benchmark. therefore, if there is a way to boost these wellrepresented features, it would make simcse perform even better. in this context, we identified a method, named ifm (implicit feature modification) @xcite from the vrl literature, which tries to remove some well-represented features, for the purpose of avoiding shortcut learning @xcite ) -a model tends to depend on a subset of features that is easier to learn during training @xcite . we interpret ifm to be the opposite of our idea, although ifm ultimately seeks to improve performance as we do. considering that vrl models are initialized and trained from scratch while plms already capture semantic features before contrastive learning, taking a contrary approach to ifm will work better for srl, rather than following ifm as is. this study first conducts a pilot study applying the vanilla ifm to simcse. contrary to its success in vrl, we observe a performance degradation, especially for a larger size of plms. we interpret that these results come from the fact that plms already learn several meaningful features, which are indeed helpful in srl and are not the shortcut features that harm the generalization performance. then, we propose byop (bootstrap 1 your own plm), which boosts the well-represented features, contrary to the intuition of ifm from the vrl perspective. experimental results demonstrate the effectiveness, robustness, and extensibility of our byop. ## preliminary unsupervised contrastive learning for srl simcse followed the literature of the nt-xent (normalized temperature cross entropy) loss @xcite with in-batch negatives: to solve this problem, @xcite proposed a simple method, called ifm, which accelerates instances to avoid well-represented features by applying adversarial perturbations toward the gradient ascent of the contrastive loss. considering the similarity function of equation 1 as a simple ℓ 2 -normalized dot product 2 , each gradient with respect to the positive (∇ z ′ i l i ) and the negative instance (∇ z ′ j l i ) can be defined as: ## pilot study despite the effectiveness of ifm in vrl, we assume that boosting the well-represented features, contrary to ifm, will fit in srl, due to the differences between vrl and srl; e.g., the use of plms that may learn several well-represented features. in this pilot study, we empirically show the failure of the vanilla ifm applied to simcse, and provide further analyses to point out differences in the two fields. ## conclusion we have proposed byop based on the intuition that plms' semantic features are useful for sentence representation. our pilot study, which observes unexpected experimental artifacts in terms of uniformity, also motivates re-examining the logic of the original ifm by boosting the gradient of loss. we have conducted the sts benchmark of which the results back up the assumption of byop by testing several variants. we hope that these approaches shed new light on the deeper analysis of the contrastive learning of srl.