Enhancing Extreme Multi-Label Text Classification: Addressing Challenges in Model, Data, and Evaluation (2023)

Abstract: Extreme multi-label text classification is a prevalent task in industry, but it frequently encounters challenges in terms of machine learning perspectives, including model limitations, data scarcity, and time-consuming evaluation. This paper aims to mitigate these issues by introducing novel approaches. Firstly, we propose a label ranking model as an alternative to the conventional SciBERT-based classification model, enabling efficient handling of large-scale labels and accommodating new labels. Secondly, we present an active learning-based pipeline that addresses the data scarcity of new labels during the update of a classification system. Finally, we introduce ChatGPT to assist with model evaluation. Our experiments demonstrate the effectiveness of these techniques in enhancing the extreme multi-label text classification task.

## introduction extreme multi-label text classification (xmtc) refers to the task of assigning to each document its most relevant labels from a taxonomy, where the number of labels could reach hundreds of thousands or millions @xcite . xmtc plays a crucial role in various industry applications such as search systems, recommendation systems, and social media analysis. by enabling accurate categorization of documents, it facilitates making it easier to search, filter, and organize the content effectively @xcite . however, the existing approaches often face inherent challenges pertaining to the model, data, and evaluation aspects. first, classification models typically serve as the default choice for this task @xcite . nonetheless, these models struggle to scale to a large number of labels as the increasing size of feature space causes the number of parameters to explode quickly. second, when building a new classification model, labeled data is often unavailable, and the available data can be imbalanced. moreover, our taxonomy data, from which the labels originate, undergoes yearly updates. consequently, both the training and test data, as well as the model, require regular updates. third, the evaluation process is timeconsuming. evaluations are typically performed offline using a test set, which necessitates subject matter expertss (smes) to spend significant time labeling samples. these existing issues have direct consequences for businesses, leading to prolonged release times, limited innovation, increased efforts for the sales team, and dissatisfied clients. in this work, we aim to replace our existing classification pipeline with a new solution that addresses the aforementioned issues. first, we introduce a label ranking model to replace the scibertbased classification model used in production. this new model comprises a bi-encoder model and a cross-encoder model @xcite . the bi-encoder model offers benefits such as high recall and low computational cost, while the cross-encoder model enhances precision by re-ranking the top (i.e., 100) documents. second, we propose an active learningbased pipeline for model updates and data collection. since active learning needs an initial pool of positive documents, we use an unsupervised training strategy to train a bi-encoder that can adapt to our target domain. for new labels without labeled data, we use this bi-encoder model to identify potentially positive documents for annotation. human annotators are then involved in the annotation loop to label the training data. finally, we introduce chatgpt to assist with model evaluation. we generate prompts for documents that require annotation and utilize chatgpt (openai) to obtain label answers along with confidence scores and explanations. subsequently, smes manually verify these answers. we assess our pipeline's performance by considering model effectiveness, training costs, and manual annotation costs. the predicted labels of our pipeline exhibit greater correctness and specificity compared to the production baseline. for a newly introduced label, it requires on average 100 human-annotated samples for the updated model to achieve a recall@10 of 0.8. additionally, with the help chatgpt, smes' annotation effort is reduced from 15 mins to 5 mins for annotating a single document with 10 labels. as a result, our proposed pipeline enables multiple releases within a single year, significantly enhancing efficiency and productivity. ## related work in the field of multi-label text classification, numerous studies have contributed to the development of effective models and techniques @xcite . previous research has explored a variety of methodologies, including traditional machine learning algorithms, deep learning architectures, and hybrid models, to address the complex nature of multi-label classification tasks @xcite . notable work has been conducted on feature engineering @xcite , neural network architectures @xcite , and loss functions tailored for multi-label scenarios @xcite , aiming to enhance the predictive accuracy and interpretability of models. furthermore, recent advancements in pre-trained language models, such as bert @xcite and its variants @xcite , have demonstrated substantial results in multi-label classification, opening up new possibilities for transfer learning in this domain. additionally, research efforts have delved into handling imbalanced label distributions @xcite , leveraging auxiliary information, and adapting models for specific domains. the existing work provides a comprehensive foundation upon which our current research builds, with a focus on the capabilities of introducing new labels in a multi-label text classification setting. ## web interface to facilitate efficient model updates and data annotation, we have developed a web application (figure 3 ). this application enables multiple users to seamlessly interact with the model simultaneously, with all interactions logged and stored. it employs a microservices architecture for scalability, consisting of a front-end react client application and two fastapi server applications. one server manages user and project management, while the other focuses on the al component. communication between the api and al is facilitated through rab-bitmq message queues, and all data is stored in a mongodb instance. the application can be hosted on a p3.2xlarge or a g4dn.xlarge amazon ec2 instance. at the beginning of the al process, the bicross-encoder model provides a list of ranked documents by relevancy. these documents are shown one by one to all users without repetition. the users will be able to decide if the label matches the content of the abstract. once a batch of positive results (label matches abstract) is obtained, it is sent to the model for training, and a new list of ranked abstracts is provided. the application's asynchronous nature ensures that users are unaffected by any time delays caused by these model processes. additionally, user responses and time spent on annotations are stored and linked to project and abstract data. prompt which of the following 0. fuzzy neural networks ... are relevant topics for this abstract. for each just provide a relevance score between 0 and 2, and an explanation. 0 means not relevant and 2 means highly relevant. -> title: ... abstract: ... the determination of the rail voltage for a 1500 v dc-fed rail system by means of the adaptive neuro-fuzzy inference system ... response 0. fuzzy neural networks: 2 -the study uses an adaptive neuro-fuzzy inference system (anfis), which combines fuzzy logic and neural networks ... chatgpt evaluation model update & data collection via active learning finetune finetune retrieve is the doc relevant to the new label? test_docs predict prompt label 1 label 2 … label 1 label 2 … doc pool the application also allows users to track model performance during their annotation, once they are satisfied with the performance they can terminate model training. ## conclusion in this work, we propose an approach to enhance our pipeline for the extreme multi-label text classification task. we replace the traditional scibertbased classification model with a label ranking model based on a bi-encoder and a cross-encoder, enabling efficient handling of large-scale labels. moreover, we present an active learning-based pipeline that addresses the data scarcity of new labels during the update of a classification model. finally, we demonstrate the effectiveness of using chatgpt for model evaluation when faced with limited time and monetary budget for annotation.