Cluster-Guided Label Generation in Extreme Multi-Label Classification (2023)

Abstract: For extreme multi-label classification (XMC), existing classification-based models poorly per- form for tail labels and often ignore the semantic relations among labels, like treating”Wikipedia” and “Wiki” as independent and separate labels. In this paper, we cast XMC as a generation task (XLGen), where we benefit from pre-trained text-to-text models. However, generating labels from the extremely large label space is challenging without any constraints or guidance. We, therefore, propose to guide label generation using label cluster information to hierarchically generate lower-level labels. We also find that frequency-based label ordering and using decoding ensemble methods are critical factors for the improvements in XLGen. XLGen with cluster guidance significantly outperforms the classification and generation baselines on tail labels, and also generally improves the overall performance in four popular XMC benchmarks. In human evaluation, we also find XLGen generates unseen but plausible labels. Our code is now available at https://github.com/alexa/xlgen-eacl-2023 .

## introduction extreme multi-label classification (xmc) is a task to predict multiple relevant labels for a given input where the label space is extremely large. conventional approaches for xmc decompose the problem into a set of binary classifications, training one-vs-all classifiers for each label. however, they encounter several issues in practical use cases. first, the labels in xmc are long-tail distributed. in other words, only a few labels have sufficient positive samples, thereby the other infrequent labels could be rarely predicted during inference as we see the heavily right-skewed distribution in the long-tail in figure 2a . second, multi-label classification techniques such as one-by-one and label powerset (gibaja, 2015) assume independent wikipedia wiki science interesting fun video funny food humor weird humour wtf #aftedarkclub soda eruption wikipedia fun science diet wiki funny coke tv video health interesting humor food diet coke and mentos eruption diet coke and mentos eruption is a reaction of diet coke and mint mentos candies, a bottle of diet coke (other carbonated beverages may be used instead) and dropping some mentos. this causes the coke to foam at a rapid rate and spew into the air. and identically distributed labels, while the usergenerated labels in xmc are dependent on each other. moreover, annotated labels are only a portion of possible labels, thus, resulting in positive and unlabeled (pu) setting @xcite . in this paper, we tackle extreme multi-label classification with a generative approach, called extreme multi-label generation (xlgen). in particular, we fine-tune a pre-trained transformer-based encoder-decoder model @xcite with input documents and their known positive labels. this (label) generation approach is more intuitive and closely similar to how humans tag documents with text labels without a fine-grained ontology or guideline. however, the generated labels from the extremely large label space without any constraints and/or guidance can be noisy and not cover infrequent labels. to address this issue, we propose a method to leverage label clusters into generation: first, generate cluster ids of semantically similar labels, and then generate text labels utilizing the cluster ids as additional contextual inputs. specifically, we propose two xlgen architectures (xlgen-bcl, xlgen-mcg) in which such clusters are jointly trained with labels in different ways. using clusters for label generation is motivated by showing label categories to human annotators. as an example, humans often start by setting highlevel topics first and then hierarchically create actual tags under each high-level topic. the clusters, however, are treated as additional guidance rather than a constraint since we do not restrict the model to only predict labels under the given clusters. similarly to xlgen, @xcite proposed groov, which fine-tunes t5 to generate labels in xmc. in particular, groov aims to implement a label order invariant training objective by randomly shuffling label orders and using multisoftmax function, which does not penalize if any first tokens of true labels are predicted regardless of the label orders. however, it does not outperform classification baselines consistently, and we empirically find that label order by frequency helps alleviate the issue in our ablation study. our experiment shows that xlgen (and its variants) outperforms classification baselines on four xmc benchmarks. furthermore, xlgen with cluster guidance (xlgen-bcl and -mcg) significantly and consistently outperforms the classification and generation baseline (xlgen-base) on tail labels, respectively. the effect on tail labels from xlgen-bcl is demonstrated in figure 2b . figure 1 shows predicted or generated labels from different models. a wikipedia page of diet coke and mentos eruption has true labels such as "beverage", "fun", and "eruption". we find xlgen can also generate a new positive label "soda" based on the context of "carbonated beverage" in the input text. from a human evaluation (s6.1), we find newly generated labels by xlgen are highly associated with the input texts, which potentially helps automatically find new labels without manual tagging. we also show the generated labels from large language models (llms) like gpt-3 @xcite : we find that the overall performance of incontext learning is significantly less than xlgen (see §4.4 for details), but llms could generate reasonable labels with a few examples as xlgen does. lms and generative approach in xmc. for xmc, pre-trained lms such as xlnet @xcite and transformer @xcite are used but only for encoding input texts, thus, it still relies on the classification approach for label prediction. previously, other works address multi-label classification with generative approaches @xcite @xcite @xcite but in much smaller label spaces. recently, @xcite also used t5 @xcite for directly generating labels in end-to-end manners for xmc, but its performance was not convincing compared to classification-based models. positive and unlabeled data. in practice, xmc is inherently with positive and unlabeled (pu) setting as the label space is extremely large and it is infeasible to manually review all the labels @xcite . multi-label performances on pu tasks can be simulated by leaving only a few labels per train instance (e.g., leaving 8 out of 10 positive labels in one instance for label deficit rate 20%) positive @xcite . in this work, we show how xlgen works on such pu settings in §4.3. ## extreme multi-label generation (xlgen) with cluster guidance classifying a document with multiple labels can be regarded as tagging a document with possible topical labels, which is basically decoding the freeformed text labels in an encoder-decoder setting. moreover, if encoder and decoder are trained on large text corpora, labels are generated with an understanding of their lexical variations and semantic similarities. our baseline framework fine-tunes a pre-trained transformer using the input text as an encoder input and the label sequences as a decoder output. in addition, to more effectively address a huge number of labels in a long-tail distribution, we propose two different architectures, xlgen-bcl ( §3.2) and xlgen-mcg ( §3.3), generating labels guided by pre-computed cluster information, inspired by class-based language models @xcite . for each label, we use the averaged embedding of positive documents including the label in train set and compute the label cluster assignment matrix (top). for training, we assign label clusters for each training document based on the true labels using the label cluster assignment matrix (bottom). pre-computed clustering. we compute label clusters using k-means algorithm, as depicted in figure 4 . we first obtain label features using average embedding of positive documents in a train set, following @xcite , and compute the label-cluster assignment matrix. label clusters are assigned to each training document using this matrix and ground-truth labels, and used as a multicluster prediction layer for xlgen-bcl training or sequence of cluster ids for xlgen-mcg training. ## ablation study we explore various factors that impact the performance of xlgen on wiki10-31k, such as label orders ( §5.1) and sampling strategies ( §5.2) in order to reduce training costs, we mainly train xlgen-base on the base size model with epoch 5 for ablation tests. we then investigate the model performance by clustering sizes and algorithms ( §5.3). ## qualitative analysis lastly, we evaluate the quality of generated labels via human evaluation ( §6.1) and visualization of the semantic relations among labels ( §6.2). ## conclusion we apply text-to-text transformers to extreme multi-label classification, by tweaking the classification problem as generation of label texts. as we do not control the vocabulary space of generated labels, xlgen can create completely unseen but still relevant labels, inferred from the input context and semantic relationship from the previously generated labels. our experiments show that xlgen outperforms the classification baselines in general, and significantly improves the long-tail performance and pu setting. also, we observe utilizing label cluster information helps improve the performance in various settings. xlgen is expected to more benefit from pre-trained models as they become larger and powerful @xcite .