R obust S ent E mbed: Robust Sentence Embeddings Using Adversarial Self-Supervised Contrastive Learning (2024)

Abstract: Pre-trained language models (PLMs) have consistently demonstrated outstanding performance across a diverse spectrum of natural language processing tasks. Nevertheless, despite their success with unseen data, current PLM-based representations often exhibit poor robustness in adversarial settings. In this paper, we introduce RobustSentEmbed, a self-supervised sentence embedding framework designed to improve both generalization and robustness in diverse text representation tasks and against a diverse set of adversarial attacks. Through the generation of high-risk adversarial perturbations and their utilization in a novel objective function, RobustSentEmbed adeptly learns high-quality and robust sentence embeddings. Our experiments confirm the superiority of RobustSentEmbed over state-of-the-art representations. Specifically, Our framework achieves a significant reduction in the success rate of various adversarial attacks, notably reducing the BERTAttack success rate by almost half (from 75.51% to 38.81%). The framework also yields improvements of 1.59% and 0.23% in semantic textual similarity tasks and various transfer tasks, respectively.

## introduction pre-trained language models (plms) have demonstrated state-of-the-art performance in learning contextual word embeddings @xcite , contributing to significant advancements in various natural language processing (nlp) tasks @xcite @xcite . plms, including prominent models like bert @xcite and gpt-3 @xcite , have revolutionized text classification, sentence representation, and machine translation among a plethora of diverse nlp tasks. while plms have expanded their focus to include universal sentence embeddings, which effectively capture the semantic representation of input text, plm-based sen-tence representations lack two crucial characteristics: generalization and robustness. extensive research efforts have been dedicated to the development of universal sentence embeddings employing plms @xcite @xcite . although these embeddings have demonstrated proficiency in generalization across various downstream tasks @xcite , they exhibit limitations when subjected to adversarial settings and remain vulnerable to adversarial attacks @xcite . existing research has highlighted the limited robustness of plm-based representations @xcite @xcite . the vulnerability arises when these representations can be easily deceived by making small, imperceptible modifications to the input text. to address these limitations, we propose a method to obtain robust sentence embeddings called robustsentembed. the main idea is to generate small adversarial perturbations and employ an efficient contrastive objective @xcite . the goal is to enhance the adversarial resilience of the sentence embeddings. specifically, our framework involves an iterative collaboration between an adversarial perturbation generator and the plmbased encoder to generate high-risk perturbations in both token-level and sentence-level embedding spaces. robustsentembed then employs a contrastive learning objective in conjunction with a token replacement detection objective to maximize the similarity between the embedding of the original sentence and the adversarial embedding of a positive pair (the former objective) as well as its edited sentence (the latter objective). we have conducted comprehensive experiments to substantiate the efficacy of the robustsentembed framework. the tasks encompass textattack @xcite assessments, adversar-ial semantic textual similarity (sts) tasks, nonadversarial sts tasks @xcite , and transfer tasks @xcite . two initial series of experiments were designed to evaluate the robustness of our sentence embeddings against various adversarial attacks and tasks. subsequently, we conducted two final series of experiments to assess the quality of our embeddings in the contexts of semantic similarity and natural language understanding. robustsentembed demonstrates significant improvements in robustness, reducing the attack success rate from 75.51% to 38.81% against the bertattack attack and from 71.86% to 12.80% on adversarial sts. moreover, the framework outperforms existing methods in ten out of thirteen tasks while obtaining comparable results with the other three, showcasing improvements of 1.59% and 0.23% on sts tasks and nlp transfer tasks, respectively. ## related work recently, self-supervised methods using contrastive objectives have become prominent for learning effective and robust text representations: simcse, as outlined by @xcite , introduced a minimal augmentation method involving the application of two distinct dropout masks to predict the input sentence. the consert model @xcite employed four unique data augmentation techniques, namely adversarial attacks, token shuffling, cut-off, and dropout, to generate a variety of perspectives in order to carry out a contrastive objective. @xcite utilized adversarial training to improve the robustness of contrastive learning. they achieved this by incorporating regularization into their learning objective, combining benign contrastive learning with an adversarial contrastive scenario. @xcite proposed a novel method for training language processing models, combining adversarial training and contrastive learning. their approach incorporates linear perturbations to input embeddings and uses contrastive learning to minimize the distance between the original and perturbed representations. @xcite introduced a simple technique to improve the fine-tuning of transformer-based encoders. their method involves regularization by generating adversarial examples through word embedding perturbations and using contrastive learning to obtain noise-invariant representations. unlike existing approaches for training text representation through contrastive adversarial learning @xcite @xcite , our framework generates more efficient, high-risk perturbations at both the token-level and sentence-level within the embedding space. furthermore, our framework utilizes a robust contrastive objective and incorporates an adversarial replaced token detection method, leading to high-quality text representations that yield improved generalization and robustness characteristics. ## the proposed framework we extended our previous robust text representation (i.e., robustembed, see @xcite ) by utilizing adversarial perturbation at various levels, including token-level and sentence-level. we introduce robustsentembed, a straightforward yet highly effective method for generating robust text representation. given a plm f θ (.) as the encoder and a raw dataset d, our framework aims to pretrain f θ (•) on d to enhance the efficacy of sentence embeddings across a wide range of nlp tasks (improved generalization) and to fortify its resilience against various adversarial attacks (improved robustness). figure 1 presents an overview of our framework. the framework involves an iterative interaction between the perturbation generator and the f θ (.) encoder to produce high-risk adversarial perturbations in both token-level and sentence-level embedding spaces. these perturbations provide the ## conclusion and future work this paper introduces robustsentembed, a selfsupervised sentence embedding framework enhancing robustness against adversarial attacks while achieving state-of-the-art performance in text representation and nlp tasks. current sentence embeddings are vulnerable to attacks, and robustsentembed addresses this by generating high-risk perturbations at token and sentence levels. these perturbations are incorporated into novel contrastive and difference prediction objectives. the framework is validated through comprehensive experiments on semantic textual similarity and transfer learning tasks, confirming its robustness against adversarial attacks and semantic similarity tasks. in future research, we aim to investigate the use of hard negative examples to further enhance the effectiveness of text representations.