Constructing contrastive samples via summarization for text classification with limited annotations (2021)

Abstract: Contrastive Learning has emerged as a powerful representation learning method and facilitates various downstream tasks especially when supervised data is limited. How to construct efficient contrastive samples through data augmentation is key to its success. Unlike vision tasks, the data augmentation method for contrastive learning has not been investigated sufficiently in language tasks. In this paper, we propose a novel approach to construct contrastive samples for language tasks using text summarization. We use these samples for supervised contrastive learning to gain better text representations which greatly benefit text classification tasks with limited annotations. To further improve the method, we mix up samples from different classes and add an extra regularization, named Mixsum, in addition to the cross-entropy-loss. Experiments on real-world text classification datasets (Amazon-5, Yelp-5, AG News, and IMDb) demonstrate the effectiveness of the proposed contrastive learning framework with summarization-based data augmentation and Mixsum regularization.

## introduction learning a good representation has been an essential problem in the deep learning era. especially, in the area of natural language processing, the language model pre-training techniques, such as bert @xcite , have been overwhelming in a wide range of tasks by learning contextualized representations. however, the success of these pre-trained models hinge largely on plenty of labeled data for fine-tuning. with limited labels on the target task, fine-tuning bert has been shown unstable @xcite . in practice, it is costly to gather labeled data for a new task, and lack of training data is still a big challenge in many real-world problems. recently, contrastive learning methods have become popular self-supervised learning tools and gained big progress in few-shot learning due to its better discriminative ability @xcite . various contrastive learning methods have been developed and lead to state-of-the-art performance in many computer vision tasks. they are also extended to the fully supervised setting by leveraging label information to make further improvement. in natural language processing, contrastive learning has not been fully investigated but it is attracting more and more attentions. a contrastive learning method generally consists of two components: finding positive samples and negative samples for each anchor sample; and building up an effective objective function to discriminate them. in many contrastive learning frameworks, how to efficiently find the contrastive samples has been the key to their success. for example, in moco @xcite , the contrastive pairs are constructed by matching an encoded query with a dynamic dictionary; in simclr @xcite , the contrastive pairs are created by applying two different data augmentation operators, and it was shown that composition of data augmentation operations is crucial for learning good representations. in supervised contrastive learning, essentially the positive sample space has been augmented. instead of only using the anchor sample and its own transformation, all samples in the same class can be further regarded as positive pairs. in this paper, we focus on using contrastive learning to assist the text classification tasks with limited labels. considering the specialty of the text classification task, we propose two novel strategies to further enhance the performance of supervised contrastive learning. we assume that a good summarization system can keep the most critical information of original texts and the generated summary tends to belong to the same category as the original text. thus we utilize text summarization as a data augmentation method to create more positive and negative samples for supervised contrastive learning. furthermore, we propose mixsum, an idea similar to the methodology of mix-up @xcite , which combines texts from different categories and creates new summary samples to further augment the data for contrastive learning. we adapt the supervised contrastive loss to the mixsum setting, and show that it brings great benefit for text classification when training data is extremely scarce. our main contributions are listed as below: ## conclusion we proposed a novel data augmentation technique for constructing contrastive samples in supervised contrastive learning-summarization. besides, we also proposed a mixsum method based on using summarization to construct the contrastive samples. we demonstrate the effectiveness of the proposed two new techniques on text classification task under the limited annotation setting. the experiment results on four datasets show that mixsum and using summarization as contrastive samples can improve the performance of text classification under the limited annotations setting. besides, we show that the proposed mixsum methods can be generalized to different summarization methods and text mixing methods. our work also opens up several possibilities for future work, since using summarization to construct contrastive samples has shown the effectiveness in supervised contrastive learning. we may investigate whether using summarization as data augmentation can improve unsupervised text classification @xcite , and the robustness and performance of other nlp applications like question answering, commonsense reasoning and semantic code retrieval @xcite .