Virtual Augmentation Supported Contrastive Learning of Sentence Representations (2022)

Abstract: Despite profound successes, contrastive representation learning relies on carefully designed data augmentations using domain-specific knowledge. This challenge is magnified in natural language processing, where no general rules exist for data augmentation due to the discrete nature of natural language. We tackle this challenge by presenting a Virtual augmentation Supported Contrastive Learning of sentence representations (VaSCL). Originating from the interpretation that data augmentation essentially constructs the neighborhoods of each training instance, we, in turn, utilize the neighborhood to generate effective data augmentations. Leveraging the large training batch size of contrastive learning, we approximate the neighborhood of an instance via its K-nearest in-batch neighbors in the representation space. We then define an instance discrimination task regarding the neighborhood and generate the virtual augmentation in an adversarial training manner. We access the performance of VaSCL on a wide range of downstream tasks and set a new state-of-the-art for unsupervised sentence representation learning.

## introduction universal sentence representation learning has been a long-standing problem in natural language processing (nlp). leveraging the distributed word representations @xcite @xcite as the base features to produce sentence representations is a common strategy in the early stage. however, these approaches are tailored to different target tasks, thereby yielding less generic sentence representations @xcite @xcite . this issue has motivated more research efforts on designing generic sentence-level learning objectives or tasks. among them, supervised learning on the natural language inference (nli) datasets @xcite @xcite has established benchmark transfer learning performance on various downstream tasks @xcite @xcite . despite promising progress, the high cost of collecting annotations precludes its wide applicability, especially when the target domain has scarce annotations but differs significantly from the nli datasets @xcite . on the other hand, unsupervised learning of sentence representations has seen a resurgence of interest with the recent successes in self-supervised contrastive learning. these approaches rely on two main components, data augmentation and an instance-level contrastive loss. the popular contrastive learning objectives @xcite ; @xcite and their variants thereof have empirically shown their effectiveness in nlp. however, the discrete nature of the text makes it challenging to establish universal rules for effective text augmentation generation. various contrastive learning based approaches have been proposed for sentence representation learning, where the main difference lies in how the augmentations are generated @xcite @xcite @xcite @xcite . somewhat surprisingly, a recent work @xcite shows that dropout @xcite , i.e., augmentations obtained by feeding the same instance to the encoder twice, outperforms common data augmentations obtained by operating on the text directly, including cropping, word deletion, or synonym replacement. again, this observation validates the inherent difficulty of attaining effective data augmentations in nlp. this paper tackles the challenge by presenting a neighborhood-guided virtual augmentation strategy to support contrastive learning. in a nutshell, data augmentation essentially constructs the neighborhoods of each instance, with the semantic content being preserved. we take this interpretation in the opposite direction by leveraging the neighborhood of an instance to guide augmentation generation. benefiting from the large training batch of contrastive learning, we approximate the neighborhood of an instance via its k-nearest in-batch neighbors. we then define an instance discrimination task within this neighborhood and generate the virtual augmentation in an adversarial training manner. we run in-depth analyses and show that our vascl model leads to a more dispersed representation space with the data semantics at different granularities being better captured. we evaluate our model on a wide range of downstream tasks and show that our model consistently outperforms the previous state-of-the-art results by a large margin. more recent efforts focus on directly designing the sentence-level learning objectives or tasks. on the supervised learning regime, @xcite ; @xcite empirically show the effectiveness of leveraging the nli task @xcite to promote generic sentence representations. the task involves classifying each sentence pair into one of three categories: entailment, contradiction, or neutral. @xcite further bolster the performance by using the pre-trained transformer @xcite as backbone. on the other end of the spectrum, @xcite ; @xcite propose using the denoising or variational autoencoders for sentence representation learning. @xcite ; @xcite extend the distributional hypothesis to the sentence level and train an encoder-decoder to construct the surrounding context for each sentence. alternatively, @xcite present a model that learns to discriminate the target context sentences from all contrastive ones. contrastive learning contrastive learning has been the pinnacle of recent successes in sentence representation learning. @xcite ; @xcite substantially advance the previous state-of-the-art results by leveraging the entailment sentences in nli as positive pairs for optimizing the properly designed contrastive loss functions. nevertheless, we focus on unsupervised contrastive learning and form the positive pairs via data augmentation since such methods are more costeffective and applicable across different domains and languages. along this line, several approaches have been proposed recently, where the augmentations are obtained via dropout @xcite , back-translation (fang and xie, 2020), surrounding context sampling @xcite , or perturbations conducted at different semantic-level @xcite @xcite . consistency regularization our work is also closely related to consistency regularization, which is often used to promote better performance by regularizing the model output to remain unchanged under plausible input variations that are often induced via data augmentations. @xcite ; @xcite ; @xcite ; @xcite show randomized data augmentations such as dropout, cropping, rotation, and flipping yield effective regularization. berthelot et al. (2019, 2020); verma et al. (2019) improve the performance by applying mixup @xcite and its variants on top of stochastic data augmentations. however, data augmentation has long been a challenge in nlp as there are no general rules for effective text transformations. an alternative that comes to light when considering the violation of consistency regularization can in turn be used to find the most sensitive perturbation for a model. therefore, we utilize consistency regularization to promote informative virtual augmen- for each instance x i in a randomly sampled batch, we optimize (i) an instance-wise contrastive loss with the dropout induced augmentation obtained by forwarding the same instance twice, i.e., x i and x i denote the same text example; and (2) a neighborhood constrained instance discrimination loss with the virtual augmentation proposed in section 3.2. tation for a training instance in the representation space while leveraging its approximated neighborhood to regularize the augmentation sharing similar semantic content as its original instance. ## experiment in this section, we mainly evaluate vascl against simcse @xcite which leverages the dropout @xcite induced noise as data augmentation. we show that vascl consistently outperforms simcse on various downstream tasks that involve semantic understanding at different granularities. we carefully study the regularization effects of vascl and empirically demonstrate that vascl leads to a more dispersed representation space with semantic structure better encoded. please refer to appendix a for details of our implementations and the dataset being used. ## conclusion in this paper, we present a virtual augmentationoriented contrastive learning framework for unsupervised sentence representation learning. our key insight is that data augmentation can be interpreted as constructing the neighborhoods of each training instance, which can, in turn, be leveraged to generate effective data augmentations. we evaluate vascl on a wide range of downstream tasks and substantially advance the state-of-the-art results. moreover, we conduct in-depth analyses and show that vascl leads to a more dispersed representation space with the data semantics at different granularities being better encoded. on the other hand, we observe a performance drop of both simcse and vascl when combined with the explicit text augmentations. we suspect this is caused by the linguistic patterns generated by explicit augmentations being less informative yet noisy. we hypothesize effective data augmentation operations on the discrete texts could complement our virtual augmentation approach if new and informative linguistic patterns are generated. models with simcse. we found 3e-05 yields better performance. for both simcse and vascl, we set the batch size to 1024, train all models over five epochs and evaluate the development set of sts-b every 500 iterations. we report all our evaluations on the downstream tasks with the associated checkpoints attaining the best performance on the validation set of sts-b.