S up CL - S eq: S upervised C ontrastive L earning for Downstream Optimized Sequence Representations (2021)

Abstract: While contrastive learning is proven to be an effective training strategy in computer vision, Natural Language Processing (NLP) is only recently adopting it as a self-supervised alternative to Masked Language Modeling (MLM) for improving sequence representations. This paper introduces SupCL-Seq, which extends the supervised contrastive learning from computer vision to the optimization of sequence representations in NLP. By altering the dropout mask probability in standard Transformer architectures (e.g. BERT-base), for every representation (anchor), we generate augmented altered views. A supervised contrastive loss is then utilized to maximize the system’s capability of pulling together similar samples (e.g., anchors and their altered views) and pushing apart the samples belonging to the other classes. Despite its simplicity, SupCL-Seq leads to large gains in many sequence classification tasks on the GLUE benchmark compared to a standard BERT-base, including 6% absolute improvement on CoLA, 5.4% on MRPC, 4.7% on RTE and 2.6% on STS-B. We also show consistent gains over self-supervised contrastively learned representations, especially in non-semantic tasks. Finally we show that these gains are not solely due to augmentation, but rather to a downstream optimized sequence representation.

## introduction sequence classification is a fundamental problem in natural language processing (nlp), as it has a wide range of applications, including but not limited to the tasks such as sentiment analysis, inference and question answering @xcite . cross-entropy loss is generally the default loss function in training neural networks for nlp downstream tasks @xcite . recently, thanks to the simplicity of augmentation methods in computer vision (e.g., zooming, cropping, rotation, etc.), selfsupervised and supervised variants of contrastive learning proved to be effective training approaches in image classification tasks @xcite @xcite . these methods aim at optimizing the representations by minimizing the distance between similar samples and maximizing it between diverse samples @xcite . @xcite proposed to leverage the built-in dropout masks in attention and fullyconnected layers of transformers @xcite to introduce noise in the embedding representations. this is obtained by simply passing twice the same input and using different dropout masks. in this way, for every representation (anchor), altered views are generated. @xcite applied this augmentation approach to improve the semantic representation of a sequence in a self-supervised fashion, by taking an input sentence and contrasting its similarity against its augmented version and the remaining samples in a batch. the authors further extended this approach by employing positive (i.e., entailment) and negative (i.e., contradiction) examples from natural language inference (nli) datasets. the resulting sentence embeddings achieved large gains in semantic textual similarity (sts) tasks. to the best of our knowledge, however, contrastive learning has not yet been applied in a supervised fashion to optimize sequence representations towards downstream tasks. 1 inspired by the recently proposed supervised contrastive learning in computer vision @xcite , in this paper we introduce supcl-seq, which extends the selfsupervised contrastive method by @xcite to a supervised contrastive learning approach, in which anchors and altered views, along with their classification labels, are used to learn downstream the book was written by john encoder books were sent to each other by the students. she voted for herself. positive class negative class different views in two forward passes with different dropout masks supcl-seq dropout @xmath0.1 dropout @xmath1.0 positive class embedding negative class embedding anchor positives negatives figure 1 : supcl-seq applied to cola @xcite . supcl-seq first forward propagates the input n times (in this example @xmath2.g. bert base ) with n different dropout masks (e.g. @xmath3.1 respectively) and obtains their corresponding noisy embedding views. the noisy embeddings that belong to the same class are then employed as the positive pairs for the original input (anchor with dropout mask of @xmath4. in this way, the samples belonging to the negative class effectively are used as negatives. the main contributions of this paper are: the augmentation step is obtained by forward propagating the input batch n times in the same encoder with n distinct dropout masks (i.e., different dropout probabilities). the generated altered views, along with their anchor's label, are then used to optimize the sequence representations through a supervised contrastive loss function @xcite . figure 1 details our training approach. formally, our pipeline consists of a single encoder transformer, enc(.) (i.e., bert base with ≈110m parameters @xcite ). this encoder generates n altered embeddings, @xmath5, p n ), for each input x and dropout probability p n . foot_1 a contrastive loss function is then applied in a supervised fashion to maximize the encoder's capability of building downstream optimized sequence representations (see section 2.1). after this contrastive training, the encoder parameters are frozen and a linear classification layer is then trained with cross-entropy. in the remainder of this section, we review the self-supervised contrastive function @xcite and its extended supervised counterpart inspired by @xcite . ## experiments we performed a set of experiments to i) evaluate the effect of number and level of dropout ## discussion and conclusion in this paper, we introduced supcl-seq a supervised contrastive learning framework for optimizing sequence representations for downstream tasks. in a series of experiments, we showed that supcl-seq leads to large performance gains in almost all glue tasks when compared to both a standard bert base architecture and an augmented bert base (i.e., improvements are not only due to augmentation). we also investigated the effect of number and level of dropout passes, finding that this has to be treated as a task-dependent hyper-parameter, to be fine tuned in a validation set. finally, we compared our supervised approach to the selfsupervised method by @xcite , showing consistent performance improvements, especially in non-semantic tasks, where the self-supervised approach is weaker. these encouraging results open the door to multi-task learning applications of supcl-seq, where the optimization needs to be constrained towards multiple objectives.