R obust E mbed: Robust Sentence Embeddings Using Self-Supervised Contrastive Pre-Training (2023)

Abstract: Pre-trained language models (PLMs) have demonstrated their exceptional performance across a wide range of natural language processing tasks. The utilization of PLM-based sentence embeddings enables the generation of contextual representations that capture rich semantic information. However, despite their success with unseen samples, current PLM-based representations suffer from poor robustness in adversarial scenarios. In this paper, we propose RobustEmbed, a self-supervised sentence embedding framework that enhances both generalization and robustness in various text representation tasks and against diverse adversarial attacks. By generating high-risk adversarial perturbations to promote higher invariance in the embedding space and leveraging the perturbation within a novel contrastive objective approach, RobustEmbed effectively learns high-quality sentence embeddings. Our extensive experiments validate the superiority of RobustEmbed over previous state-of-the-art self-supervised representations in adversarial settings, while also showcasing relative improvements in seven semantic textual similarity (STS) tasks and six transfer tasks. Specifically, our framework achieves a significant reduction in attack success rate from 75.51% to 39.62% for the BERTAttack attack technique, along with enhancements of 1.20% and 0.40% in STS tasks and transfer tasks, respectively.

## introduction recent research has demonstrated the state-of-theart performance of pre-trained language models (plms) in learning contextual word embeddings @xcite , leading to improved generalization in various natural language processing (nlp) tasks @xcite @xcite . the focus of plms has extended to acquiring universal sentence embeddings, such as universal sentence encoder (use) @xcite and sentence-bert @xcite , which effectively capture the semantic representation of the input text. this representation learning facilitates feature generation for classification tasks and enhances large-scale semantic search @xcite . the assessment of plm-based sentence representation relies on two crucial characteristics: generalization and robustness. while considerable research efforts have been dedicated to developing universal sentence embeddings using plms @xcite @xcite @xcite , it is worth noting that despite their promising performance across various downstream classification tasks @xcite , demonstrating proficiency in generalization, these representations exhibit limitations in terms of robustness in adversarial settings and are vulnerable to diverse adversarial attacks @xcite . existing research @xcite @xcite highlights the poor robustness of these representations, such as bertbased representations, which can be deceived by replacing a few words in the input sentence. in this paper, we propose robustembed, a robust sentence embedding framework that takes both of these essential characteristics into account. the core concept involves introducing a small adversarial perturbation to the input text and employing the contrastive objective @xcite to learn high-quality sentence embeddings. robustembed perturbs the embedding space rather than the raw text, which exhibits a positive correlation with generalization and promotes higher invariance. our framework utilizes the original embedding along with the perturbed embedding as "positive pairs," while other sentence embeddings in the same minibatch serve as "negatives." the contrastive objective identifies the positive pairs among the negatives. by incorporating norm-bounded adversarial perturbation and contrastive objectives, our method enhances the robustness of similar sentences and disperses sentences with different semantics. this straightforward and efficient approach yields superior sentence embeddings in terms of both generalization and robustness benchmarks. we conduct extensive experiments on a wide range of text representation and nlp tasks to verify the effectiveness of robustembed including semantic textual similarity (sts) tasks @xcite , transfer tasks @xcite , and textattack @xcite . two first series of experiments evaluate the quality of sentence embeddings on semantic similarity and natural language understanding tasks, while the last series assess the robustness of the framework against state-of-the-art adversarial attacks. robustembed demonstrates significant improvements in robustness, reducing the attack success rate from 75.51% to 39.62% for the bertattack attack technique and achieving similar improvements against other adversarial attacks. additionally, the framework achieves performance improvements of 1.20% and 0.40% on sts tasks and nlp transfer tasks, respectively, when employing the bert base encoder. contributions. our main contributions in this paper are summarized as follows: ## related work the early work in text representations focused on applying the distributional hypothesis to predict words based on their context @xcite . there are extensive studies on learning universal sentence embeddings using supervised and unsupervised approaches, such as doc2vec @xcite , skipthought @xcite , universal sentence encoder @xcite , and sentence-bert @xcite . more recently, self-supervised approaches have emerged, employing contrastive objectives to learn effective and robust text representations: simcse @xcite introduced a minimal augmentation strategy to predict the input sentence by applying two different dropout masks. the consert model @xcite utilized four distinct data augmentation techniques to generate diverse views for the purpose of executing a contrastive objective: adversarial attacks, token shuffling, cut-off, and dropout. qiu et al. (2021) introduced two adversarial training methods, carl and rar, to strengthen the ml model's defense against gradient-based adversarial attacks. carl aims to acquire a resilient representation at the sentence level, whereas rar focuses on enhancing the robustness of individual word representations. rima et al. (2022) proposed adversarial training with contrastive learning for training natural language processing models. it involves applying linear perturbations to input embeddings and leveraging contrastive learning to minimize the distance between original and perturbed representations. pan et al. (2022) presents a straightforward approach for regularizing transformer-based encoders during the fine-tuning step. the model achieves noise-invariant representations by generating adversarial examples perturbing word embeddings and leveraging contrastive learning. in comparison to several existing contrastive adversarial learning approaches in the text representation area @xcite @xcite @xcite , our framework stands out by generating more efficient high-risk iterative perturbations in the embedding space. furthermore, our framework leverages a more powerful contrastive objective approach, leading to high-quality text representations that demonstrate enhanced generalization and robustness properties. empirical results substantiate the superiority of our approach across various generalization and robustness benchmarks. ## background in this section, we present an overview of the recent progress in adversarial perturbation generation and self-supervised contrastive learning. ## the proposed adversarial self-supervised contrastive learning we introduce robustembed, a simple yet effective approach for generating universal text representations through adversarial training of a selfsupervised contrastive learning model. given a plm f θ (.) as the encoder and a large unsupervised dataset d, robustembed aims to pre-train f θ (.) on d to enhance the efficiency of sentence embeddings across diverse nlp tasks (improved generalization) and increase resilience against various adversarial attacks (enhanced robustness). algorithm 1 demonstrates our framework's approach to generating a norm-bounded perturbation using an iterative process, confusing the f θ (.) model by treating the perturbed embeddings as different instances. plm encoder plm embedding layer sentence 1 sentence n … ... embedding 1 embedding n … ... iterative perturbation generator perturbed embedding n … ... sentences mini-batch benign embedding n benign embedding 1 perturbed embedding 1 mapping from text to embedding space high-risk perturbation generating & positive pair forming contrastive learning with benign and perturbed representations plm encoder layer our framework then employs a contrastive learning approach to maximize the similarity between the embedding of an input instance and the adversarial embedding of its positive pair. moreover, figure 1 provides an overview of our robustembed framework, which aims to achieve adversarial robustness in representations. the framework involves an iterative collaboration between the perturbation generator and the f θ (.) model to generate high-risk perturbations for adversarial contrastive learning during the final training step. the subsequent sections delve into the main components of our framework and provide a detailed analysis of the training objective. ## evaluation and experimental results this section presents a comprehensive set of experiments aimed at validating the effectiveness of our proposed framework in terms of generalization and robustness metrics. in the first two series of experiments, we investigate the performance of our framework on seven semantic textual similarity (sts) tasks and six transfer tasks within the senteval framework foot_0 to assess the generalization capability of our framework in generating efficient sentence embeddings. in the final series of experiments, we measure the resilience of the embeddings against five state-of-the-art adversarial attacks to assess the robustness capability of our framework in generating robust text representation. appendices a and b provide training details and ablation studies that illustrate the effects of hyperparameter tuning. ## conclusion and future work in this paper, we proposed robustembed, a selfsupervised sentence embedding framework that significantly enhances robustness against various adversarial attacks while achieving state-of-the-art performance in a wide range of text representation and nlp tasks. current sentence embeddings are vulnerable to adversarial attacks. robustembed fills this gap by leveraging high-risk adversarial perturbations within a novel contrastive objective approach. we demonstrated the effectiveness of our framework through extensive experiments on semantic textual similarity and transfer learning tasks. furthermore, empirical findings substantiate the robustness of robustembed against diverse adversarial attacks. as future work, we aim to explore the use of hard negative examples in the supervised setting to further enhance the efficiency of text representations.