An End-to-End Contrastive Self-Supervised Learning Framework for Language Understanding (2022)

Abstract: Self-supervised learning (SSL) methods such as Word2vec, BERT, and GPT have shown great effectiveness in language understanding. Contrastive learning, as a recent SSL approach, has attracted increasing attention in NLP. Contrastive learning learns data representations by predicting whether two augmented data instances are generated from the same original data example. Previous contrastive learning methods perform data augmentation and contrastive learning separately. As a result, the augmented data may not be optimal for contrastive learning. To address this problem, we propose a four-level optimization framework that performs data augmentation and contrastive learning end-to-end, to enable the augmented data to be tailored to the contrastive learning task. This framework consists of four learning stages, including training machine translation models for sentence augmentation, pretraining a text encoder using contrastive learning, finetuning a text classification model, and updating weights of translation data by minimizing the validation loss of the classification model, which are performed in a unified way. Experiments on datasets in the GLUE benchmark (Wang et al., 2018a) and on datasets used in Gururangan et al. (2020) demonstrate the effectiveness of our method.

## introduction self-supervised learning @xcite @xcite @xcite , which learns data representations by solving prediction tasks defined on input data without leveraging humanprovided labels, has achieved broad success in nlp. many nlp-specific self-supervised learning (ssl) methods have been proposed, such as neural language models @xcite , word2vec @xcite , bert @xcite , gpt @xcite , bart @xcite , and so forth, with various ssl tasks defined. for example, in word2vec and bert, the ssl task is predicting the identities of masked tokens based on their contexts. in neural language models including gpt, the ssl task is language modeling: given a history of tokens, predict the next token. recently, contrastive self-supervised learning @xcite has been borrowed from vision domains into nlp and has shown promising success in predicting semantic textual similarity @xcite , machine translation @xcite , relation extraction @xcite , and so on. the key idea of contrastive self-supervised learning (cssl) is: create augments of original examples, then learn representations by predicting whether two augments are from the same original data example. in existing cssl approaches, data augmentation and contrastive learning are performed separately. as a result, augmented data may not be optimal for contrastive learning. for example, considering a back-translation @xcite -based augmentation method, if the translation model is trained using news corpora, it is not suitable for augmenting data for movie review data. in this paper, we aim to address this issue. we propose a four-level optimization framework that performs data augmentation and contrastive learning end-to-end in a unified way, to allow the data augmentation models to be guided by the contrastive learning task and make augmented data suitable for performing contrastive learning. we assume the end task is text classification. our framework consists of four learning stages. at the first stage, we train four translation models to perform sentence augmentation based on back translation. to account for the fact that translation data used at this stage and text classification data used in later stages have a domain discrepancy, we perform reweighting of translation data; these weights are tentatively fixed at this stage and will be updated at a later stage. at the second stage, we pretrain a text encoder using contrastive learning on augmented sentences created by the translation models. at the third stage, we finetune a text classification model, using the text encoder pretrained at the second stage as regularization. at the fourth stage, we measure the performance of the text classifier on a validation set and update weights of translation data by maximizing the validation performance. each level of optimization problem in our framework corresponds to a learning stage. these stages are performed end-to-end. experiments on datasets in the glue benchmark @xcite and on datasets used in @xcite demonstrate the effectiveness of our method. the major contributions of this paper include: ## method in this section, we present our proposed end-to-end contrastive learning framework.