Differentiable Data Augmentation for Contrastive Sentence Representation Learning (2022)

Abstract: Fine-tuning a pre-trained language model via the contrastive learning framework with a large amount of unlabeled sentences or labeled sentence pairs is a common way to obtain high-quality sentence representations. Although the contrastive learning framework has shown its superiority on sentence representation learning over previous methods, the potential of such a framework is under-explored so far due to the simple method it used to construct positive pairs. Motivated by this, we propose a method that makes hard positives from the original training examples. A pivotal ingredient of our approach is the use of prefix that attached to a pre-trained language model, which allows for differentiable data augmentation during contrastive learning. Our method can be summarized in two steps: supervised prefix-tuning followed by joint contrastive fine-tuning with unlabeled or labeled examples. Our experiments confirm the effectiveness of our data augmentation approach. The proposed method yields significant improvements over existing methods under both semi-supervised and supervised settings. Our experiments under a low labeled data setting also show that our method is more label-efficient than the state-of-the-art contrastive learning methods.

## introduction learning universal and effective sentence representations is an enduring problem in natural language processing (nlp). the objective of learning sentence representations is similar to that of learning word embeddings @xcite , i.e., we expect the embeddings of sentences with similar semantics to be close to each other, while the embeddings of sentences with different meanings to be sufficiently far away from each other. recently, we have witnessed the success of pre-trained language models in various nlp 1 our code and model checkpoints are available at https: //github.com/tianduowang/diffaug . tasks. however, the quality of the sentence representations directly obtained from non fine-tuned language models remains unsatisfactory @xcite . one approach to addressing this problem is to fine-tune a pre-trained language model via the contrastive learning objective on labeled or unlabeled sentences. several recent research efforts @xcite @xcite have shown that a contrastive learning based finetuning stage can produce state-of-the-art results on the sentence embedding learning task. some previous contrastive learning works on other modalities, e.g., image @xcite , have shown that an effective data augmentation (da) method is crucial for the success of contrastive learning. however, due to the discrete nature of language, applying commonly-used sentence augmentation strategies, e.g., word deletion and replacement, over the input sentences for contrastive learning leads to suboptimal results @xcite . instead, @xcite propose to perform data augmentation via dropout, showing that this extremely simple approach can yield better sentence representations than previous da methods that are based on discrete transformations. although this dropout-based da method outperforms its discrete counterparts, the potential of the contrastive learning framework is under-explored due to its simple treatment of the positive pair construction process. previous works have shown that contrastive learning benefits from strong data augmentations that can produce hard positives with meaningful differences. specifically, @xcite demonstrate that contrastive learning requires stronger data augmentation than traditional supervised learning. @xcite further show that, for contrastive learning, a good pair of positive instances should only share task-relevant information while discarding irrelevant information as much as possible. motivated by this idea, we propose diffaug, a differentiable data augmentation method for contrastive sentence representation learning. this method prepends two separate prefix modules @xcite to a common pre-trained language model. the goal is to obtain hard positive pairs with the help of these two prefix modules for contrastive learning, where supervised signals can be used to guide the tuning process of such prefix modules. such a design essentially requires us to fine-tune both language model and newly-added prefix. however, how to effectively train them jointly to best serve our sentence representation learning purpose is a non-trivial research question. our observations show that, though tempting, it can be undesirable to fine-tune language model and prefix jointly from the beginning of the training process. rather, we propose an effective two-stage tuning strategy, prefix-tuning followed by joint tuning, as illustrated in figure 1 . specifically, we argue that it is crucial to perform prefix-tuning first, until a module-compatible state is achieved, before we move to the second stage to perform joint tuning for both modules. our main contributions can be summarized as follows: we further design a mechanism that allows this module to be carefully optimized with labeled data first, allowing hard positives with meaningful differences to be constructed, which can benefit contrastive learning. through extensive analysis we identify the crucial elements required to ensure the success of our proposed approach. ## background our method is based on the state-of-the-art contrastive learning framework proposed by @xcite . we first introduce this framework. next, we discuss the mechanism of prefix-tuning @xcite . ## method in this section, we describe the proposed data augmentation method, and our two-stage tuning strategy. we then discuss reasons why our proposed approach works. ## related work learning sentence embeddings as a fundamental nlp problem has been extensively studied. many works @xcite @xcite @xcite achieve good results based on pretrained language models. among these works, contrastive learning methods achieve the state-ofthe-art results. in this work, we improve the data augmentation module of the contrastive learning framework by a prompting method. contrastive sentence representation learning. contrastive learning aims to learn effective representations by pulling together semantically close neighbors and pushing apart non-neighbors. a crucial problem in contrastive learning is how to generate positive instances. the commonly-used data augmentation methods in nlp include word deletion, reordering, and substitution @xcite . however, data augmentation in nlp is inherently difficult because of the discrete nature of language. @xcite shows that data augmentation via dropout is consistently better than previous discrete transformations. prompting. prompting means querying a pretrained language model by adding natural language tokens into the input sentences @xcite . this method alleviates the discrepancy between a language model's pre-training stage and fine-tuning stage, and is shown to be useful when the labeled data for downstream tasks is scarce @xcite . although adding natural language tokens (i.e., hard prompt) is effective for certain tasks, it takes efforts to design good prompts. motivated by this problem, soft prompt methods @xcite are proposed. unlike hard prompt, soft prompt tokens can be fine-tuned continuously. therefore, it is more expressive. some recent works further improve the soft prompt method by adding deep prompt modules on language models @xcite @xcite , and they are called parameter-efficient methods. with more tunable parameters, the parameter-efficient methods can handle complex natural language tasks, e.g., language understanding @xcite ), text generation @xcite , and structured prediction @xcite . such methods usually assume the language model is fixed during fine-tuning, and only the newly-added prompting modules can be tuned. ## conclusion in this paper, we propose a differentiable data augmentation method for contrastive sentence representation learning. unlike previous discrete transformations for sentences (e.g., token shuffling and synonyms replacement) and continuous methods (e.g., dropout @xcite ), the proposed method learns how to do data augmentation implicitly based on supervised signals from the natural language inference (nli) tasks. in this way, our method produces hard positives with meaningful differences for contrastive learning. we demonstrate the effectiveness of our method on several semantic textual similarity tasks, and our method achieves new state-of-the-art performance in both semi-supervised and supervised settings on average. we also conduct experiments where only limited labeled data is available. the results demonstrate the proposed method is robust when is labeled data is scarce. to the best of our knowledge, our method is the first work that provides a successful solution for combining the conventional pre-trained language model fine-tuning with parameter-efficient methods via a two-stage tuning process. the approach of combining the two yields better results than using any of them alone on the sentence representation learning tasks. one potential future research direction is to generalize this method to other nlp research areas, e.g., domain adaptation and lowresource tasks.