Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning (2021)

Abstract: We introduce EfficientCL, a memory-efficient continual pretraining method that applies contrastive learning with novel data augmentation and curriculum learning. For data augmentation, we stack two types of operation sequentially: cutoff and PCA jittering. While pretraining steps proceed, we apply curriculum learning by incrementing the augmentation degree for each difficulty step. After data augmentation is finished, contrastive learning is applied on projected embeddings of original and augmented examples. When finetuned on GLUE benchmark, our model outperforms baseline models, especially for sentence-level tasks. Additionally, this improvement is capable with only 70% of computational memory compared to the baseline model.

## introduction many state-of-the-art language models involve the paradigm of unsupervised pretraining followed by fine-tuning on downstream tasks @xcite @xcite . however, pretraining a language model from scratch with a huge corpus has high computational costs. one way to cut down the cost is to use continual training of a pretrained model which could improve the language model with less computation @xcite . contrastive learning is effective for selfsupervised learning for image classification @xcite , and it works by allowing the model to put similar examples close and different examples far from one another. often in contrastive learning, data augmentation is used to make the positive pairs. recent papers describe how to apply contrastive learning to the language domain @xcite @xcite , and even a combination of contrastive learning with continual pretraining @xcite . however, because of the sequential nature of language, it is difficult to apply data augmentation methods used in images directly to language modeling. additionally, curriculum learning is a powerful training technique for deep networks @xcite . by training easy to hard examples in order, it facilitates faster convergence, leading to better performance. other studies show that it is also effective for language modeling @xcite @xcite . we propose an efficient yet powerful continual pretraining method using contrastive learning and curriculum learning. the contribution of this paper is as follows: ## method the overall learning process of efficientcl is illustrated in figure 1 . ## conclusion in this paper, we propose efficientcl, a pretrained model with efficient contrastive learning method utilizing data augmentation and curriculum learning. our data augmentation process is divided into cutoff and pca jittering. rather than using one, combining two augmentation methods significantly boosts the performance. additionally, by incrementing the augmentation level for each difficulty step, the model achieves faster convergence, resulting in better performance on sentence-level tasks. our method is also memory efficient and applicable for a wide range of corpora at pretraining stage.