Learning Label Hierarchy with Supervised Contrastive Learning (2024)

Abstract: Supervised contrastive learning (SCL) frameworks treat each class as independent and thus consider all classes to be equally important. This neglects the common scenario in which label hierarchy exists, where fine-grained classes under the same category show more similarity than very different ones. This paper introduces a family of Label-Aware SCL methods (LA-SCL) that incorporates hierarchical information to SCL by leveraging similarities between classes, resulting in creating a more well-structured and discriminative feature space. This is achieved by first adjusting the distance between instances based on measures of the proximity of their classes with the scaled instance-instance-wise contrastive. An additional instance-center-wise contrastive is introduced to move within-class examples closer to their centers, which are represented by a set of learnable label parameters. The learned label parameters can be directly used as a nearest neighbor classifier without further finetuning. In this way, a better feature representation is generated with improvements of intra-cluster compactness and inter-cluster separation. Experiments on three datasets show that the proposed LA-SCL works well on text classification of distinguishing a single label among multi-labels, outperforming the baseline supervised approaches. Our code is publicly available ^1 .

## introduction supervised contrastive learning (scl) @xcite aims to learn generalized and discriminative feature representations given labeled data. it relies on the construction of positive pairs from the same class and negative pairs from different classes, thereby encouraging similar data points to have similar representations while pushing dissimilar data points apart in the feature space. this method considers each class to be independent and 1 https://github.com/rxlian/la-scl figure 1: supervised v.s. label-aware supervised contrastive loss: the supervised contrastive loss (left) contrasts the set of all samples from the same class as positives against the negatives from the remainder of the batch @xcite . the label-aware supervised contrastive loss (right) proposed in our work incorporates label hierarchy by considering class similarities. hierarchical text classification (htc) is one way to structure textual data into a tree-like category or label hierarchy, representing a taxonomy of classes @xcite . existing htc can be divided into global and local approaches. global approaches treat the problem as a flat classification, while local approaches build classifiers for labels at each level of the hierarchy. @xcite propose fcdc, which aims to transfer information from coarse-grained levels to fine-grained categories and thus adapt models to categories of different gran-ularity. besides, @xcite incorporate label hierarchy information extracted from a separate encoder. some other works leverage additional hierarchical information @xcite @xcite . other than that, @xcite augment the classification loss by the cophenetic correlation coefficient (cpcc) @xcite as a standalone regularizer to maximize the correlation between the label tree structure and classconditioned representations. @xcite propose a protonce loss, a generalized version of the infonce loss @xcite to learn a representation space by encouraging each instance to become closer to an assigned prototype such as the clustering centroid. in this way, the underlying semantic structure of the data can be encoded. based on these studies, the hierarchical structure of the labels suggests that learning methods could be enhanced if the learning mechanism can be made aware of the class taxonomy. we explore several ways of exploiting such hierarchical relationships between classes by proposing to augment the scl loss function as depicted in fig. 1 . since this incorporates class taxonomy information, we call it label-aware scl (lascl). this is achieved by first using pairwise class similarities to scale the temperature in the scl to encourage samples under the same branches to cluster more closely while driving apart samples with different labels under different coarse clusters. in addition, we add instance-center-wise contrastive with learned label representations as the center of the sentence embeddings from the corresponding class. these result in making sub-classes under the same coarsegrained classes closer to each other and generating more discriminative representations by making intra-class samples closer to their centers. to utilize intrinsic information from label and data hierarchies, we encode the textual label information to be class centers and compute pairwise class cosine similarities on top of that. this quantifies the proximity between classes and forms the basis for instantiating variations of lascl objectives. since the dimension of these label representations is the same as the linear classifier, we show that it can be applied directly to downstream classification without further finetuning. to the best of our knowledge, we are the first to work on leveraging the textual hierarchical label and integrating it into the scl to improve the representations. our methods can be transferred to various backbone models, and are simple yet effective across different datasets. the only changes we make are in the cost function so that the method can be applied in any situation where labels in a hierarchy exist. our contributions are summarized as follows: • lascl integrates label hierarchy information into scl by leveraging the textual descriptions of the label taxonomy. ## method this section describes our proposed label-aware supervised contrastive learning objectives. overview: in the embedding space, we hypothesize that sentences from different fine-grained classes under the same coarse-grained class are closer to each other in comparison to sentences from different high-level categories. given this intrinsic information provided by the label and data hierarchy, we use the pairwise cosine similarities of a set of learnable parameters representing label features to quantify the proximity between classes, which are used to instantiate variants of label-aware supervised contrastive learning objectives. ## results and analysis to demonstrate the effect of the amount of labeled data to lascl, we perform experiments with both the few-shot setup and full dataset in §5.1 and §5.2. in §5.3, we visually show how the proposed methods generate a more well-structured and discriminative embedding space by visualizations. we discuss how the size of the hierarchy plays a role by constructing a bottom-up label hierarchy with different depths in §5.4. the experimental results are reported with linear probes (lp) and with direct testing (dt). for lp, a randomly initialized linear layer was trained on a small number of labeled samples with the encoder frozen. we denote dt as directly applying the learned label parameters as the classifier ( §3.4). ## related work learning label hierarchy hierarchical text classification is a task involving assigning samples to specific labels (most commonly fine-grained levels) arranged in a structured hierarchy, which is typically represented as a tree or directed acyclic graph, where each node corresponds to a label @xcite . recent studies have suggested integrating the label structure into text features by encoding them with a label encoder. for instance, @xcite embed the word and label hierarchies jointly in the hyperbolic space. @xcite propose a hierarchy-aware global model to extract the label structural information. @xcite design a label-based attention module to extract information hierarchically from the labels on different levels. @xcite propose a network to embed label hierarchy to text encoder with contrastive learning. @xcite propose a matching network to match labels and text at different abstraction levels. other than these studies on network structure, @xcite propose a hierarchical triplet loss, which is useful for finding hard negatives by hierarchically merging sibling branches. recent work by @xcite introduces a hierarchy-preserving loss, applying a hierarchical penalty to contrastive loss with the preservation of a hierarchical relationship between labels on images by using images under the same branch as positive pairs. our lascl, in contrast, exploits a small number of known labels and their hierarchical structure to improve the learning process. it differs from these works in constructing penalties from the hierarchical structure and exploiting it in the contrastive loss. contrastive learning self-supervised contrastive learning is a representation learning approach that maximizes agreement between augmented views of the same instance and pushes different instances far apart. works on text data @xcite constructing various augmentations on text level @xcite @xcite , embedding level @xcite @xcite , and via language models @xcite @xcite , etc. scl effectively learns meaningful representations and improves classification performance by combining supervised and contrastive learning advantages. it was initially introduced in simclr @xcite . other following works introduce novel insights to improve the representation learning such as moco @xcite , byol @xcite , and swav @xcite . scl has also been applied to nlp tasks such as sentence classification @xcite , relation extraction @xcite ) and text similarity @xcite , where it has shown promising results in learning effective representations for text @xcite @xcite . multi-label classification multi-label text classification is to assign a subset of labels to a given text @xcite . it acknowledges that a document can belong to more than one category simultaneously, and is especially useful when dealing with complex and diverse content that may cover multiple topics or themes. the modeling dependencies amongst labels in this work only consider assigning a single category to each sequence, and our future study is to extend this method to multi-label classification. ## conclusion in this work, we propose lascl to include information about the label hierarchy by introducing scaling to the scl loss to penalize distances between negative example pairs using the class similarities constructed from the learned label feature representations. an additional instance-center-wise contrastive is introduced. these bring instances with similar semantics or belonging to the same high-level categories closer to each other, encourage each instance to become closer to its centers, and the underlying hierarchical structures can be encoded. a better-structured and discriminative feature space is generated by improving the intracluster compactness and inter-class separation. the learned labeled parameters can be directly applied as a nearest neighbor classifier without further tuning. their effectiveness is demonstrated with experiments on three text classification datasets.