CLINE : Contrastive Learning with Semantic Negative Examples for Natural Language Understanding (2021)

Abstract: Despite pre-trained language models have proven useful for learning high-quality semantic representations, these models are still vulnerable to simple perturbations. Recent works aimed to improve the robustness of pre-trained models mainly focus on adversarial training from perturbed examples with similar semantics, neglecting the utilization of different or even opposite semantics. Different from the image processing field, the text is discrete and few word substitutions can cause significant semantic changes. To study the impact of semantics caused by small perturbations, we conduct a series of pilot experiments and surprisingly find that adversarial training is useless or even harmful for the model to detect these semantic changes. To address this problem, we propose Contrastive Learning with semantIc Negative Examples (CLINE), which constructs semantic negative examples unsupervised to improve the robustness under semantically adversarial attacking. By comparing with similar and opposite semantic examples, the model can effectively perceive the semantic changes caused by small perturbations. Empirical results show that our approach yields substantial improvements on a range of sentiment analysis, reasoning, and reading comprehension tasks. And CLINE also ensures the compactness within the same semantics and separability across different semantics in sentence-level.

## introduction pre-trained language models (plms) such as bert @xcite and roberta @xcite have been proved to be an effective way to improve various natural language processing tasks. however, recent works show that plms suffer from poor robustness when encountering adversarial examples @xcite @xcite @xcite . as shown in to improve the robustness of plms, recent studies attempt to adopt adversarial training on plms, which applies gradient-based perturbations to the word embeddings during training @xcite @xcite or adds high-quality adversarial textual examples to the training phase @xcite . the primary goal of these adversarial methods is to keep the label unchanged when the input has small changes. these models yield promising performance by constructing high-quality perturbated examples and adopting adversarial mechanisms. however, due to the discrete nature of natural language, in many cases, small perturbations can cause significant changes in the semantics of sentences. as shown in the phenomenon makes us wonder can we train a bert that is both defensive against adversarial attacks and sensitive to semantic changes by using both adversarial and contrastive examples? to answer that, we need to assess if the current robust models are meanwhile semantically sensitive. we conduct sets of pilot experiments (section 2) to compare the performances of vanilla plms and adversarially trained plms on the contrastive examples. we observe that while improving the robustness of plms against adversarial attacks, the performance on contrastive examples drops. to train a robust semantic-aware plm, we propose contrastive learning with semantic negative examples (cline). cline is a simple and effective method to generate adversarial and contrastive examples and contrastively learn from both of them. the contrastive manner has shown effectiveness in learning sentence representations @xcite @xcite , yet these studies neglect the generation of negative instances. in cline, we use external semantic knowledge, i.e., wordnet @xcite , to generate adversarial and contrastive examples by unsupervised replacing few specific representative tokens. equipped by replaced token detection and contrastive objectives, our method gathers similar sentences with semblable semantics and disperse ones with different even opposite semantics, simultaneously improving the robustness and semantic sensitivity of plms. we conduct extensive experiments on several widely used text classification benchmarks to verify the effectiveness of cline. to be more specific, our model achieves +1.6% absolute improvement on 4 contrastive test sets and +0.5% absolute improvement on 4 adversarial test sets compared to roberta model @xcite . that is, with the training on the proposed objectives, cline simultaneously gains the robustness of adversarial attacks and sensitivity of semantic changes foot_0 . ## pilot experiment and analysis to study how the adversarial training methods perform on the adversarial set and contrastive set, we first conduct pilot experiments and detailed analyses in this section. ## method as stated in the observations in section 2, we explore strategies that could improve the sensitivity of plms. in this section, we present cline, a simple and effective method to generate the adversarial and contrastive examples and learn from both of them. we start with the generation of adversarial and contrastive examples in section 3.1, and then introduce the learning objectives of cline in section 3.2. ## experiments we conduct extensive experiments and analyses to evaluate the effectiveness of cline. in this section, we firstly introduce the implementation (section 4.1) and the datasets (section 4.2) we used, then we introduce the experiments on contrastive sets (section 4.3) and adversarial sets (section 4.4), respectively. finally, we conduct the ablation study (section 4.5) and analysis about sentence representation (section 4.6). ## conclusion in this paper, we focus on one specific problem how to train a pre-trained language model with robustness against adversarial attacks and sensitivity to small changed semantics. we propose cline, a simple and effective method to tackle the challenge. in the training phase of cline, it automatically generates the adversarial example and semantic negative example to the original sentence. and then the model is trained by three objectives to make full utilization of both sides of examples. empirical results demonstrate that our method could considerably improve the sensitivity of pre-trained language models and meanwhile gain robustness.