Open-world Multi-label Text Classification with Extremely Weak Supervision (2024)

Abstract: We study open-world multi-label text classification under extremely weak supervision (XWS), where the user only provides a brief description for classification objectives without any labels or ground-truth label space. Similar single-label XWS settings have been explored recently, however, these methods cannot be easily adapted for multi-label. We observe that (1) most documents have a dominant class covering the majority of content and (2) long-tail labels would appear in some documents as a dominant class. Therefore, we first utilize the user description to prompt a large language model (LLM) for dominant keyphrases of a subset of raw documents, and then construct a (initial) label space via clustering. We further apply a zero-shot multi-label classifier to locate the documents with small top predicted scores, so we can revisit their dominant keyphrases for more long-tail labels. We iterate this process to discover a comprehensive label space and construct a multi-label classifier as a novel method, X-MLClass. X-MLClass exhibits a remarkable increase in ground-truth label space coverage on various datasets, for example, a 40% improvement on the AAPD dataset over topic modeling and keyword extraction methods. Moreover, X-MLClass achieves the best end-to-end multi-label classification accuracy.

## introduction multi-label text classification (mltc) aims to assign one or more labels to each input document in the corpus. traditional methods @xcite @xcite require a complete list of class names, which is challenging to provide beforehand given the massive number of documents and diverse topics. this work focuses on a new problem, open-world foot_0 mltc un-der extremely weak supervision (xws), where the user only provides a brief description for classification objectives without any labels or ground-truth label space. despite the considerable technical challenges posed by this task, its practical significance in real-world applications cannot be underestimated. for instance, the need for product tagging and categorization is ubiquitous in online shopping platforms. it requires identifying multiple labels for each product without access to a predefined label space, a challenge our model adeptly addresses. the most related xws problems are text clustering @xcite and topic modeling @xcite , where those methods are typically only capable of assigning a single label to each document. these single-label methods cannot be easily adapted for multi-label. we observe that (1) most documents have a dominant class covering the majority of content and (2) long-tail labels would appear as the dominant class in some documents. experiments reveal that over 90% of documents contain a dominant class, and 100% of labels appear as the dominant class in at least one document, as analyzed further in section 4. based on these observations, we propose a novel method, x-mlclass, to discover a pragmatic label space by iteratively adding (longtailed) labels and construct a multi-label text classification classifier with the assistance of llm (i.e., llama-2-13b-chat in our experiments), as shown in figure 1 . our approach requires only a brief user description about the classification objective as prompt for llm, which significantly reduces the cost of model training. the first step in x-mlclass is to construct a high-quality label space. we start with a reasonably large subset of the raw documents. for each document, we partition it into chunks to better align with the context length of llm and also ensure that each chunk contains a single topic, and then prompt the only required supervision from the user is a brief description of the classification objective. during the first llm prompting stage for keyphrases, x-mlclass leverages this description as a part of the prompt, so it will be helpful if the description includes some demonstrations. the llm to generate the most dominant keyphrases for each chunk. as previous llm-based text clustering work has suggested @xcite , there are very likely some semantically redundant yet lexically different keyphrases among the generated ones. we cluster these keyphrases, and within every cluster, we pull together the corresponding chunks of the keyphrases closest to the cluster center to prompt the llm once again, generating one single label for each cluster. after eliminating labels with high similarity scores, we constitute an initial label space. we then apply the state-of-the-art textual entailment-based classification methods @xcite to construct a classifier that revisits the documents and identifies long-tail labels. specifically, we query every text chunk against all the labels for the entailment score, identifying chunks with low top predicted scores that lack a dominant class. we revisit the keyphrases generated by these chunks to unveil more long-tail labels, selectively choosing keyphrases with a modest presence in the entire keyphrase set but absent in the original label space. these new keyphrases are included in the label set, and documents are reassessed with this updated set. by repeating these steps for a fixed number of iterations, the final label space contains a substantial number of long-tail labels. extensive experiments on 5 benchmark datasets reveal the superiority of x-mlclass outperforming all compared methods. remarkably, compared with baselines, x-mlclass achieves a significant enhancement of 40% and 30% in ground-truth label space coverage on the aapd and rcv1-v2 datasets. furthermore, it achieves the highest accu-racy in zero-shot mltc, surpassing the top-ranking models on huggingface across all datasets. our contributions are summarized as: • we attack a challenging problem, open-world mltc with xws, where the user only provides a brief description for classification objectives without any labels or ground-truth label space. multi-label text classification: numerous approaches have been proposed to tackle the complexities of multi-label text classification (mltc) problems. bhatia and jain @xcite employ embedding-based methods, leveraging the power of embeddings to train individual classifiers for each label. later, xml-cnn @xcite uses a convolutional neural network (cnn) to learn text representations, demonstrating improvements in mltc accuracy. recent works have started to tackle mltc problems using a small amount of labeled data or even with no labels at all. for example, @xcite achieves impressive results by using only class names and taxonomies. @xcite train a neural architecture with both true labels and their natural language descriptor. however, these methods still require access to the ground-truth label space. ## problem formulation given an unlabeled corpus @xmath0, d 2 , . . . , d n }, where d @xmath1. our task is to (1) identify class names @xmath2, where k is the unknown number of classes, and (2) build a text classifier f (•) to map any raw document d i to its target labels y @xmath3, where y j i is the single label name and p is the number of target labels for d i . the definition of "open world" denotes the absence of labeled information during training, which is more stringent than the original definition where new labels only appear in the test phase. several existing works @xcite adopt a similar "open world" definition to ours. to the best of our knowledge, this is the first work that explores open-world multi-label text classification without the presence of a ground-truth label space. given the challenging nature of the problem, we assume that human experts are willing to devote minimal effort, i.e., extremely weak supervision, typically in the form of brief classification objective descriptions. ## our observations our two observations mentioned in section 1 are confirmed by experiments based on 5 benchmark datasets: aapd @xcite , reuters-21578 @xcite , rcv1-v2 @xcite , dbpedia-298 @xcite , and amazon-531 @xcite . specifically, we prompt a large language model (llm) to check if any of the ground truth labels of a given document is dominant, i.e., covering more than 50% of the content; and if it exists, which one is the dominant label. 3 we randomly sample 1000 examples from each dataset and calculate the percentage of documents with a dominant class. the dominance percentages across datasets are aapd: 92%, rcv1-v2: 91.7%, dbpedia-298: 95.2%, reuters-21578: 96.4%, and amazon-531: 87.3%. notably, reuters dataset exhibits a higher proportion of dominant labels due to its mainly consisting of single-labeled examples. conversely, amazon presents a lower dominance percentage, attributed to the mix of fine-grained and coarse-grained labels, posing challenges in determining dominance. overall, our analysis indicates that across all datasets, more than 90% of documents contain a dominant class. for our second observation, we identify labels present in less than 1% of the dataset as long-tail labels. notably, we find that instances of these long-tail labels emerge as dominant classes in at least one document, demonstrating that 100% of the labels serve as dominant classes in some instances. these observations highlight the potential for generating labels, particularly long-tail labels, from raw documents and guide the design of our framework. ## our x-mlclass framework x-mlclass consists of three key steps. first, every document is split into chunks and transformed into keyphrases by prompting an llm to construct an initial label space. we further assign labels to each raw document d i using a custom keyphrase-chunk zero-shot textual entailment classifier. finally, we iteratively enhance the label space by incorporating additional long-tail labels. the framework overview is depicted in figure 1 , and the below sections provide a detailed discussion. ## experiments we assess the performance of x-mlclass through two primary criteria: label space quality and zeroshot mltc accuracy. our evaluation involves a comparison of our model's label coverage with that of four topic modeling and three keyword extraction methods. in terms of end-to-end classification accuracy, we test our method with several top-ranking models available on huggingface. the subsequent section provides comprehensive details on the datasets, baseline methods, evaluation metrics, implementation specifics, and performance analysis. ## conclusion and future work we attack a novel and challenging problem, openworld mltc with extremely weak supervision. in this scenario, user only provides a brief description for classification objectives without any labels or ground-truth label space. our llm-based framework, x-mlclass, is designed to overcome this challenge by discovering a practical label space and constructing an mltc classifier for label prediction. notably, it excels in identifying long-tail labels, arguably the most challenging aspect in mltc problems. our experiment results show that x-mlclass surpasses baselines in terms of ground-truth label coverage and exhibits higher zero-shot text classification performance compared to top-ranking models. despite our model's success in generating some tail labels, a considerable number of tail labels remain undiscovered. future work should focus on refining our approach to capture more long-tail labels. subsequent studies could explore methodologies tailored for datasets featuring significantly larger label spaces, contributing to the broader applicability of our model.