Long-tailed Extreme Multi-label Text Classification by the Retrieval of Generated Pseudo Label Descriptions (2023)

Abstract: Extreme Multi-label Text Classification (XMTC) has been a tough challenge in machine learning research and applications due to the sheer sizes of the label spaces and the severe data scarcity problem associated with the long tail of rare labels in highly skewed distributions. This paper addresses the challenge of tail label prediction by leveraging the power of dense neural retrieval model in mapping input documents (as queries) to relevant label descriptions. To further enhance the quality of label descriptions, we propose to generate pseudo label descriptions from a trained bag-of-words (BoW) classifier, which demonstrates better classification performance under severe scarce data conditions. The proposed approach achieves the state-of-the-art (SOTA) performance of overall label prediction on XMTC benchmark datasets and especially outperforms the SOTA models in the tail label prediction. We also provide a theoretical analysis for relating the BoW and neural models w.r.t. performance lower bound.

## introduction extreme multi-label text classification (xmtc) is the task of tagging documents with relevant labels in a very large and often skewed candidate space. it has a wide range of applications, such as assigning subject topics to news or wikipedia articles, tagging keywords for online shopping items, classifying industrial products for tax purposes, etc. the most difficult part in solving the xmtc problem is to train classification models effectively for the rare labels in the long tail of highly skewed distributions, which suffers severely from the lack of sufficient training instances. efforts addressing this challenge by the text classification community include bayesian modeling of graphical dependencies among labels @xcite , novel loss or regularization of label embeddings @xcite ; wei et al., 2021), clustering-based algorithms @xcite @xcite , and so on. despite the remarkable progresses made so far, the problem is still very far from being well solved. figure 1 shows the performance of x-transformer @xcite , one of the state-of-the-art (sota) xmtc models, on the wiki10-31k benchmark dataset (with over 31k labels). the horizontal axis is the ranks of the labels sorted from rare to common and the vertical axis is the text classification performance measured in macro-averaged f 1@19 (higher the better) for binned labels (100 labels per bin). the blue curve is the result of x-transformer, which has the scores close to 0 (worst possible score) for nearly half of the total labels. in other words, sota methods in xmtc still perform poorly in tail label prediction. in this paper, we seek solutions for tail label prediction from a new angle: we introduce a novel framework, namely the dual encoder with pseudo label (depl). it treats each input document as a query and uses a neural network model to retrieve relevant labels from the candidate space based on the textual descriptions of the labels. the underlying assumption is, if the label descriptions are highly informative for text-based matching, then the retrieval system should be able to find relevant labels. the system would be particularly helpful for tail label prediction as the retrieval effectiveness does not necessarily rely on the availability of a large number of training instances, which is what the tail labels are lacking. the next research question that we tackle is how to obtain highly informative descriptions for each label without human annotation. in reality, class names are often available but they are typically one or two words, which cannot be sufficient for retrieval-based label prediction. therefore, we propose to augment the label description with statistical learning algorithms. specifically, we train linear support vector machine (svm) model with the bag-of-words (bow) features, such as tf-idf, to automatically generate informative keywords for each label, which we call the pseudo description of the label. since the learned label embeddings of the bow classifier encode token importance information, it is natural and efficient to leverage them for keywords extraction. in sections 4 and 6, we further provide theoretical motivations and empirical evidence to show the advantage of unsupervised statistical features for classification under extreme scarce data conditions. the result of our approach (depl) is shown as the red curve in figure 1 , which significantly outperforms the blue curve of x-transformer not only in the tail-label region but also in all other regions. we also observed similar improvements by depl over strong baselines on other benchmark datasets (see section 6). our main contributions are summarized as the following: to compensate for the lack of semantics in bow features, deep learning models were proposed for xmtc. examples include cnn-based models such as xml-cnn @xcite and slice @xcite , rnn-based models such as atten-tionxml @xcite and transformer-based models such as x-transformer @xcite , lightxml @xcite and aplc-xlnet @xcite . ## evaluation results our experiments reveal the effectiveness of our model on the tail label prediction and we also include and discuss the performance on the overall prediction in appendix a.1.4. ## conclusion in this paper, we propose a novel neural retrieval framework (depl) for the open challenge of taillabel prediction in xmtc. by formulating the problem as to capture the semantic mapping between input documents and system-enhanced label descriptions, depl combines the strengths of neural embedding based retrieval and the effectiveness of a large-marge bow classifier in generating informative label descriptions under severe data sparse conditions. our extensive experiments on very large benchmark datasets show significant performance improvements by depl over strong baseline methods, especially in tail label description.