Balancing Methods for Multi-label Text Classification with Long-Tailed Class Distribution (2021)

Abstract: Multi-label text classification is a challenging task because it requires capturing label dependencies. It becomes even more challenging when class distribution is long-tailed. Resampling and re-weighting are common approaches used for addressing the class imbalance problem, however, they are not effective when there is label dependency besides class imbalance because they result in oversampling of common labels. Here, we introduce the application of balancing loss functions for multi-label text classification. We perform experiments on a general domain dataset with 90 labels (Reuters-21578) and a domain-specific dataset from PubMed with 18211 labels. We find that a distribution-balanced loss function, which inherently addresses both the class imbalance and label linkage problems, outperforms commonly used loss functions. Distribution balancing methods have been successfully used in the image recognition field. Here, we show their effectiveness in natural language processing. Source code is available at https://github.com/blessu/BalancedLossNLP .

## introduction multi-label text classification is one of the core topics in natural language processing (nlp) and is used in many applications such as search @xcite and product categorization @xcite . it aims to find the related labels from a fixed-set of labels for a given text that may have multiple labels. figure 1 from the reuters-21578 multi-label text classification dataset @xcite . here, for the document with the title penn @xmath0.k. unit, the aim is to find the labels acq (acquisitions), strategic-metal, and nickel from 90 labels. multi-label classification becomes complicated when there is a long-tailed distribution (class imbalance) and linkage (co-occurrence) of labels. class imbalance occurs when a small subset of the labels (namely head labels) have many instances, while majority of the labels (namely tail labels) have only a few instances. for example, half of the labels in the reuters dataset, including copper, strategicmetal, and nickel, occur in less than 5% of the training data. label co-occurrence or label linkage is a challenge when some head labels co-occur with rare or tail labels, resulting in bias for classification to the head labels. for example, even though the la-bel nickel occurs less frequently, the co-occurrence information of nickel/copper, nickel/strategic-metal is important for accurate modeling (figure 1 ). solutions such as resampling of the samples with lessfrequent labels in classification @xcite , using co-occurrence information in the model initialization @xcite , or providing a hybrid solution for head and tail categories with a multi-task architecture @xcite have been proposed in nlp, however they are not suitable for imbalanced datasets or they are dependent on the model architecture. multi-label classification has been widely studied in the computer vision (cv) domain, and recently has benefited from cost-sensitive learning through loss functions for tasks such as object recognition @xcite , semantic segmentation @xcite , and medical imaging @xcite . balancing loss functions such as focal loss @xcite , class-balanced loss @xcite and distribution-balanced loss @xcite provide improvements to resolve the class imbalance and co-occurrence problems in multi-label classification in cv. loss function manipulation has also been explored @xcite in nlp as it works in a model architecture-agnostic fashion by explicitly embedding the solution into the objective. for example, @xcite has borrowed dice-based loss function from a medical image segmentation task @xcite and reported significant improvements over the standard cross-entropy loss function in several nlp tasks. in this work, our major contribution is the introduction of the use of balancing loss functions to the nlp domain for the multi-label text classification task. we perform experiments on reuters-21578, a general and small dataset, and pubmed, a biomedical domain-specific and large dataset. for both datasets, the distribution balancing methods not only outperform the other loss functions for the total metrics, but also lead to significant improvement for the tail labels. we suggest that the balancing loss functions provide a robust solution for addressing the challenges in multi-label text classification. ## loss functions figure 2 : the long-tailed distribution and label cooccurrence for the reuters-21578 dataset. the cooccurence matrix is color coded based on the conditional probability p(i|j) of class in the i th column on class in the j th row. the plain bce is vulnerable to label imbalance due to the dominance of head classes or negative instances @xcite . below, we describe three alternative approaches that address the class imbalance problem in long-tailed datasets in multi-label text classification. the main idea of these balancing methods is to reweight bce so that rare instance-label pairs intuitively get reasonable "attention". ## results a summary of the results of different loss functions are listed in experiments with reuters-21578 dataset. the loss functions fl, cb, r-fl and ntr-fl perform similar to bce in head classes, yet outperform bce in medium and tail classes, indicating the advantage of handling imbalance. db provides the biggest improvement in tail class assignment; the tail micro-f1 score gains 21.49 from fl and 25.81 from cb. it outperforms prior works that also used this commonly used dataset, including approaches based on binary relevance, encdec, cnn, cnn-rnn, optimal completion distillation or attention-based gnn, that achieved micro-@xmath1.9 @xcite @xcite experiments with pubmed dataset. pubmed is a biomedical domain specific, larger dataset with bigger class imbalance. for this dataset, bce does not work efficiently, therefore we use fl as a strong baseline. with fl, the medium and tail micro-f1 scores are 26 and 9. all other loss functions outperform fl in medium and tail classes, indicating the advantage of balancing label distribution. db again has the highest performance for all classes but the most significant improvement is achieved for the medium (micro-f1:41) and tail (micro-f1:24) classes. ablation study. we further investigate the contribution of the three layers of db by comparing db results with r-fl, ntr-fl and db without the focal layer (db-0fl). as shown in ## conclusion we propose and compare the application of a series of balancing loss functions to address the class imbalance problem in multi-label text classification. we first introduce the loss function db to nlp and design a novel loss function cb-ntr. the experiments show that the db outperforms other approaches by considering long-tailed distribution and label co-occurrence, and its performance is robust to different datasets such as reuters (90 labels, general domain) and pubmed (18,211 labels, biomedical domain). this study demonstrates that addressing challenges such as class imbalance and label co-occurrence through loss functions is an effective approach for multi-label text classifica-tion. it does not require additional information and can be used with all types of neural network-based models. it may also be a powerful strategy for other nlp tasks, such as part-of-speech tagging, named entity recognition, machine reading comprehension, paraphrase identification and coreference resolution, all of which usually suffer from longtailed distribution.