An Empirical Study on Active Learning for Multi-label Text Classification (2023)

Abstract: Active learning has been widely used in the task of text classification for its ability to select the most valuable samples to annotate while improving the model performance. However, the efficiency of active learning in multi-label text classification tasks has been under-explored due to the label imbalanceness problem. In this paper, we conduct an empirical study of active learning on multi-label text classification and evaluate the efficiency of five active learning strategies on six multi-label text classification tasks. The experiments show that some strategies in the single-label setting especially in imbalanced datasets.

## introduction active learning (al) has been applied in many natural language processing (nlp) tasks due to its efficiency in improving model performance with limited annotation cost. most works in al have focused on developing strategies for single-label text classification @xcite , named entity recognition @xcite @xcite and neural machine translation @xcite @xcite . more recently, multi-label text classification @xcite @xcite has received considerable attention since many text classification tasks are multi-labeled, i.e., each document can belong to more than one category. take news classification as an example, a news article talking about the effect of the olympic games on the tourism industry might belong to the following topic categories: sports, economy and travel. the challenge of multi-label text classification lies in three aspects: (i) heavily imbalanced labels, i.e. only a small amount of labels have high frequency while others exhibit extremely low frequency; (ii) sparse label correlation, where some labels may be correlated with others, but the correlation is weak; and (iii) hierarchical label structures, this is prevalent in many scientific document indexing, e.g. arxiv or pubmed @xcite . given the above challenges, we raise the research questions: are the commonly used strategies in single-label text classification still applicable for the multi-label setting? will they always benefit classification performances? to answer these questions, we conducted an empirical study to evaluate the effectiveness of five al strategies on six prevalent multi-label text classification datasets. our experiments show that the strategies commonly used in single-label text classification can have some effectiveness under multi-label settings. however, their performance is not consistent and highly dependent on the label distribution of the datasets. the main findings of our work are as follows: ## active learning on multi-label text classification we consider multiple widely-used al strategies to investigate their different performance on multilabel text classification, including least confidence (lc) @xcite , kmeans @xcite , max entropy (lewis and gale, 1994), deep bayesian active learning(bald) @xcite , monte carlo (mc) dropout @xcite and coreset @xcite ; sener and savarese, algorithm 1 pool-based multi-label active learning input: initial labeled set l, unlabeled set u , query budget b, model parameter θ, annotation cost per round b, query strategy q output: the final classifier θ 1: initialize θ 0 with l 2: for @xmath0, . . . , do 3: {(x i , y i ) b @xmath1, q, θ t-1 ) ▷ use strategy q to select b examples 4: @xcite . random sampling, also known as passive learning, randomly selects instances for annotation and serves as a baseline for comparison with other al strategies. lc is one of the most common approach to select queries in active learning, in which it uses the probability to measure how uncertain the model is towards the instances. kmeans clustering unlabeled data samples based on their feature representations, and then selecting the samples closest to the cluster centres for labeling. this strategy can help improve the efficiency and effectiveness of the active learning process by focusing on the most representative samples in each cluster. max entropy measures the confidence of the model using entropy @xcite . it ranks all instances in u by the posterior class entropy under the model and selects the top unlabelled instances to be labelled by the expert. bald @xcite is another commonly used uncertainty-based al strategy, which maximizes the mutual information between the predictions and model posterior to achieve maximum information gain. mc dropout selects samples based on their representativeness. as its name, it uses the mc dropout on inference circles, where the uncertainty is measured by the fraction of models across mc samples that disagree with the most popular choice @xcite . coreset @xcite , is one of the most popular diversity-based querying criteria, which selects the best representation of the dataset using the farthest-first traversal algorithm. algorithm 1 shows the pseudo code of our al loop, given a fixed budget and an initial labeled set l, we try each strategy for the multi-label text classification tasks. in each al iteration, we acquire b labelled examples, this process is repeated until the budget is exhausted. ## experiments datasets ## conclusion in this paper, we explored different active learning strategies and its performance on multi-label text classification using a basic neural network model. our goal is to understand if the popular active learning strategies can prove effective in a multi-label text classification tasks under al setting. to the best of our knowledge, our work presented the first systematic and comparative study in this context. we observed that unlike single-label text classification, not all strategies can outperform the random baseline. in future work, we plan to perform a deeper analysis of the fairness issue for multi-label text classification under al setting while exploring more strategies recently published. learning strategies and the balance of selected data samples in future study. this inquiry is crucial, as unbalanced data acquisition may lead to fairness issues that can significantly affect the validity and reliability of machine learning models.