Label Anchored Contrastive Learning for Language Understanding (2022)

Abstract: Contrastive learning (CL) has achieved astonishing progress in computer vision, speech, and natural language processing fields recently with self-supervised learning. However, CL approach to the supervised setting is not fully explored, especially for the natural language understanding classification task. Intuitively, the class label itself has the intrinsic ability to perform hard positive/negative mining, which is crucial for CL. Motivated by this, we propose a novel label anchored contrastive learning approach (denoted as LaCon) for language understanding. Specifically, three contrastive objectives are devised, including a multi-head instance-centered contrastive loss (ICL), a label-centered contrastive loss (LCL), and a label embedding regularizer (LER). Our approach does not require any specialized network architecture or any extra data augmentation, thus it can be easily plugged into existing powerful pre-trained language models. Compared to the state-of-the-art baselines, LaCon obtains up to 4.1% improvement on the popular datasets of GLUE and CLUE benchmarks. Besides, LaCon also demonstrates significant advantages under the few-shot and data imbalance settings, which obtains up to 9.4% improvement on the FewGLUE and FewCLUE benchmarking tasks.

## introduction in recent years, contrastive learning (cl) has been widely applied to self-supervised representation learning and led to major advances across computer vision (cv) @xcite , speech @xcite , and natural language processing (nlp) @xcite @xcite . the basic idea behind these works is to pull together an anchor and a "positive" sample in the embedding space, and to push apart the anchor from many "negative" samples. since no labels are available, a positive pair often consists of data augmentations of the sample (a.k.a "views"), and negative pairs are formed by the anchor and randomly chosen samples from the mini-batch. in visual representations, an effective solution to generate data augmentations is to take two random transformations of the same image (e.g., cropping, flipping, distortion and rotation) @xcite @xcite . for natural language, similar approaches are adopted such as word deletion, reordering, substitution, and back-translation etc. @xcite however, data augmentation in nlp is inherently difficult because of its discrete nature. therefore, some previous works @xcite also use dropout technique @xcite to obtain sentence augmentations. unlike self-supervised setting, some researchers propose supervised contrastive learning (scl) @xcite @xcite which can construct positive pairs by leveraging label information. examples from the same class are pulled closer than the examples from different classes, leveraging the semantics of labels to construct negatives and positives rather than shallow lexical information via data augmentation. despite the aforementioned advantages brought by scl, we argue that cl under supervised learning is not fully explored because the label information can be better utilized. on the one hand, labels are usually not merely categorical indices in the label vocabulary, but also contain specific semantic meanings, especially in the language understanding tasks. thus labels can be used as positive/negative samples or anchors when calculating contrastive loss. on the other hand, label embedding enjoys a built-in ability to leverage alternative sources of information related to labels, such as class hierarchies or textual descriptions. once we obtain representative label embeddings, they can be utilized to enhance the image/text representations, and finally facilitate the classification task. previous la-bel embedding based classification models @xcite have demonstrated the effectiveness of leveraging label information. motivated by above analysis, we propose a novel label anchored supervised contrastive learning approach (denoted as lacon), which combines the advantages of both contrastive learning and label embedding techniques. specifically, we have the following three novel designs: 1) instance-centered contrastive loss (icl), which uses the infonce @xcite to encourage each text representation and its corresponding label representation to be closer while pushing far away mismatched instance-label pairs. we further apply a multi-head mechanism to catch different aspects of text semantics. 2) label-centered contrastive loss (lcl), which takes label as anchor, and encourages the label representation to be more similar to the corresponding instances belonging to the same class in a mini-batch than the instances with different labels. 3) label embedding regularizer (ler), which keeps the inter-label similarity as low as possible thus the feature space of each class is more dispersed to prevent representation degeneration. by combining above three losses, lacon can learn good semantic representations within the same space for both input instances and labels. it's also well aligned with the two key properties related to cl: alignment and uniformity @xcite , where alignment favors encoders that assign similar features to similar samples. uniformity prefers a feature distribution that preserves maximal information, i.e., the uniform distribution on the unit hypersphere. to validate the effectiveness of lacon, we perform extensive experiments on eight language understanding tasks. we take the popular pre-trained language model bert-base @xcite ) as text encoder without loss of generality. for simplicity, we predict the classification label by matching the instance representation with label embeddings directly. since our approach does not require any specialized network architecture or any extra data augmentation, lacon can be easily plugged into other pre-trained language models. additionally, we also explore the capability of lacon under more difficult task settings, including few-shot learning and data imbalance situations. to summarize, our contributions are as follows: ## model in this section, we introduce the details of lacon. we focus on the language understanding classification tasks. for a multi-class classification problem with c classes, we work with a batch of training examples {x i , y i }, where 1 ≤ @xmath0. our target is to learn discriminative representations for both instances and class labels. as figure 1 shows, we propose three supervised cl based objectives, including the instance-centered contrastive loss, the label-centered contrastive loss, and the label embedding regularizer loss. ## discussion in this section, we conduct further experiments under more challenging few-shot and data imbalance settings. we also discuss the hyper-parameter tuning and the impact of class number on lacon. ## conclusions in this paper, we proposed a novel supervised contrastive learning approach for language understanding. to utilize the class labels sufficiently, we devise three novel contrastive objectives, including a multi-head instance-centered contrastive loss, a label-centered contrastive loss, and a label embedding regularizer. extensive experiments were conducted on eight public datasets from glue and clue benchmarks, showing the competitiveness of lacon against various strong baselines. besides, we also demonstrate the strong capacity of lacon on more challenging few-shot and data imbalance settings, which leads up to 9.4% improvement on the fewglue and fewclue benchmarks. lacon does not require any complicated network architecture or any extra data augmentation, and can be easily plugged into mainstream pre-trained language models. in the future, we will explore more advanced representation fusion approaches to enhance the capability of lacon and plan to extend lacon to the computer vision and speech fields.