Label-semantics Aware Generative Approach for Domain-Agnostic Multilabel Classification (2025)

Abstract: The explosion of textual data has made manual document classification increasingly challenging. To address this, we introduce a robust, efficient domain-agnostic generative model framework for multi-label text classification. Instead of treating labels as mere atomic symbols, our approach utilizes predefined label descriptions and is trained to generate these descriptions based on the input text. During inference, the generated descriptions are matched to the predefined labels using a finetuned sentence transformer. We integrate this with a dual-objective loss function, combining cross-entropy loss and cosine similarity of the generated sentences with the predefined target descriptions, ensuring both semantic alignment and accuracy. Our proposed model LAGAMC stands out for its parameter efficiency and versatility across diverse datasets, making it well-suited for practical applications. We demonstrate the effectiveness of our proposed model by achieving new state-of-the-art performances across all evaluated datasets, surpassing several strong baselines. We achieve improvements of 13.94 % in Micro-F1 and 24.85 % in Macro-F1 compared to the closest baseline across all datasets.

## introduction text classification automates the analysis and organization of large datasets, enabling efficient labeling, categorization, and valuable insights. in text classification, two main categories arise: multiclass and multi-label classification. multi-class classification assigns a single category to a text, while multi-label classification (mltc) @xcite assigns multiple relevant labels to a document. real-world applications include topic recognition @xcite , question answering @xcite , sentiment analysis @xcite , information retrieval @xcite , and text categorization @xcite , among others. in the field of multi-label text classification, various approaches including traditional machine learning, deep learning, and hybrid models have been proposed @xcite . several state-ofthe-art methods @xcite emphasize the importance of label descriptions or metadata in improving model performance and capturing label correlations. however, these approaches often face limitations in their generalizability and adaptability. some models rely on label hierarchies or meta-path graphs @xcite , which, although effective in certain contexts, hinder scalability and flexibility. additionally, some models are designed for specific tasks, such as legal or financial applications, integrating label descriptions for specialized classifiers @xcite . despite their advancements, these methods remain limited to particular domains, highlighting the need for a more generalizable approach to mltc. large language models (llms), with their extensive pretraining, are capable of understanding similarities and relationships based on textual patterns @xcite . this motivated us to propose a novel domain-agnostic pipeline that leverages recent generative models in a parameterefficient setup employing low-rank adaptation (lora) @xcite , for the mltc task. unlike previous approaches that treat label descriptions as metadata, our proposed approach lagamc trains the generative model to generate these descriptions directly. this enables us to harness the full potential of llms, providing a more nuanced representation of label relationships and their connections to the corresponding text. obtaining label descriptions manually for multilabel datasets is both labor-intensive and subjective. to overcome this, we automate the process by leveraging gpt-3.5 @xcite and wikipedia, enabling the efficient creation of label descriptions with minimal human intervention. by leveraging a generative model while providing semantic information about labels, our method consistently outperforms state-of-the-art models, achieving a 9.32% improvement in micro-f1 and 15.25% in macro-f1 over the closest baseline. we further integrate a dual-objective loss function, combining cross-entropy loss with a cosine similarity-based loss. this results in an additional increase of 4.62% and 9.60% in micro-f1 and macro-f1 respectively. using such a hybrid loss helps bring the embeddings of the outputs of the generative model and that of the predefined label description closer in the representation space, making it easier to map the generated outputs with final labels. this hybrid approach ensures the model comprehends label distinctions and avoids overfitting to token-level matches, contributing to improved performance across diverse datasets. empirical evaluations on diverse datasets spanning social media, news, academic, and healthcare domains demonstrate the effectiveness of our approach lagamc. across various datasets, our method consistently outperforms state-of-theart models, achieving an overall improvement of 13.94% in micro-f1 and 24.85% in macro-f1 over the closest baseline. additionally, our model shows strong performance on rare labels and exhibits zeroshot capability, further enhancing its applicability. in summary, the key contributions of this paper are as follows: ## related works multi-label text classification has been approached through a range of techniques, including extending single-label classifiers, employing neural network architectures, and, more recently, utilizing transformer and pretrained language models based works. neural network-based approaches have shown great success, with methods leveraging cnns @xcite , rnns @xcite , and hybrid cnn-rnn models @xcite . attention-based models @xcite @xcite have also improved document representation, but often overlook label semantics and dependencies. more recent approaches utilize pretrained transformers such as bert @xcite and roberta @xcite , which have been adapted for multi-label tasks. for instance, @xcite applies multiple attention layers over roberta's final layer to enhance label correlation learning. @xcite explores various loss functions designed to mitigate the impact of class imbalance in datasets. most of these do not leverage the information contained in label descriptions. some of the works that utilize label descriptions to enhance performance @xcite are not scalable as they require extra information like label hierarchy. additionally, some models are built for specific tasks @xcite , making it difficult to generalize to datasets from different domains. there also exists popular extreme multi-label classification frameworks like galaxc @xcite , siamesexml @xcite , dexa @xcite , renee @xcite , inceptionxml @xcite etc. these works focus more on efficiency given the large set of labels to predict from, but rarely utilize label descriptions to enhance performance. ## datasets and label description generation we curated popular multi-label classification datasets from social media, news, academic, and healthcare domains for our experiments. ## baselines we validate our model with different baselines ranging from traditional rnn and cnn based approaches like textcnn @xcite , textrnn @xcite , attentive convnet @xcite to transformer based approaches like bert @xcite , xlnet @xcite , roberta @xcite , @xcite . we also compared the performance of lagamc against various popular extreme multi-label classification frameworks like (attentionxml @xcite , (galaxc @xcite , siamesexml @xcite , dexa @xcite , deepxml @xcite and renee @xcite . we also created our own generative baselines t5-base @xcite and t5-large @xcite . lastly, with the emergence of chatgpt @xcite , we were curious to check it's performance for this task using the same instruction prompt for 500 random samples using gpt-3.5-turbo foot_3 api. ## proposed framework for multi-label classification the proposed framework lagamc for multi-label classification (figure 1 ), is divided into two stages: a supervised generative phase, and an unsupervised description-to-label matching phase. problem formulation: given an input sequence x @xmath0, x 2 , ..., x n }, the task is to assign textclass labels y @xmath1, y 2 , ...., y k } ⊂ y to x input (the text to classify) where @xmath2, y 2 , ...., y p } contains all possible labels of that dataset. we adopt prompt-based learning paradigm, generating text conditioned on a given input prompt. ## experimental setup for all our model variants (performed on nvidia a100 80g gpus), we obtain the pre-trained checkpoints from the huggingface library foot_4 . for training the models with lora, the rank for the trainable decomposition matrices was set to 2. flan-t5-large model is instruction-tuned for 20 epochs, with batch-size of 8 and with an lr of 2e -4 with lora (training time: 56 minutes/epoch, inference time: 2 minutes/sample). these hyperparameters were selected based on the best macro-f1 results on the validation set. the input length was set by the average number of input tokens per dataset, while the output length was based on the average label description length for each dataset. ## results and discussion we report the results of our proposed generative model variants and various baselines in ## analysis we now present different analyses and ablations of lagamc. ## conclusion in this work, we propose a parameter-efficient generative approach equipped with a dual loss objective to tackle the challenging problem of multilabel classification. our method introduces a novel and domain-agnostic framework that is flexible enough to be adapted across various applications. by leveraging both generative modeling and discriminative supervision, the approach effectively captures label correlations and enhances prediction robustness. through extensive experiments, we compare our method against several state-of-the-art models and strong baseline systems specifically designed for the task. our results show that lagamc achieves significant performance gains, demonstrating its superiority across multiple evaluation metrics. furthermore, we conduct detailed ablation studies and empirical analyses to validate the contribution of each component within the framework.