Meta- LMTC : Meta-Learning for Large-Scale Multi-Label Text Classification (2021)

Abstract: Large-scale multi-label text classification (LMTC) tasks often face long-tailed label distributions, where many labels have few or even no training instances. Although current methods can exploit prior knowledge to handle these few/zero-shot labels, they neglect the meta-knowledge contained in the dataset that can guide models to learn with few samples. In this paper, for the first time, this problem is addressed from a meta-learning perspective. However, the simple extension of meta-learning approaches to multi-label classification is sub-optimal for LMTC tasks due to long-tailed label distribution and coexisting of few- and zero-shot scenarios. We propose a meta-learning approach named META-LMTC. Specifically, it constructs more faithful and more diverse tasks according to well-designed sampling strategies and directly incorporates the objective of adapting to new low-resource tasks into the meta-learning phase. Extensive experiments show that META-LMTC achieves state-of-the-art performance against strong baselines and can still enhance powerful BERTlike models.

## introduction large-scale multi-label text classification (lmtc) is a fundamental and practical task in natural language processing @xcite . lmtc can be found in several domains, such as organizing documents in wikipedia articles @xcite , annotating medical records with diagnostic and procedure labels @xcite , assigning legislation with relevant legal concepts @xcite . different from multi-class classification, the lmtc task aims to assign multiple labels from a large predefined set (typically thousands) to each instance. due to the large predefined label set and limited annotated resources, lmtc tasks usually face the challenges of long-tailed label distribution, i.e., many labels have few or even no annotated samples. for example, in eurlex15k @xcite , about 70% of seen labels have been assigned to less than 20 documents (i.e., few-shot labels); and more than 40% of the predefined labels are not associated with any document (i.e., zeroshot labels). to make matter worse, new labels continually emerge as the field evolves. though few/zero-shot labels may not contribute heavily to the overall performance, correct prediction of such labels is crucial in some cases @xcite . for instance, when assigning the diagnosis labels to electronic health records, incorrect predictions of these labels either bring unnecessary financial burdens or make patients ignore potential health risks. these factors require models to utilize few or no samples to accurately assign labels. to cope with these few/zero-shot labels, current models typically match texts to feature vectors for each label obtained by exploiting prior label information. specifically, @xcite utilizes label textual descriptors to generate a feature vector for each label. also, it employs a 2layer graph convolutional neural network @xcite to take advantage of the structured knowledge of label spaces to enhance label representations. apart from that, @xcite finds that label similarity graphs based on pre-trained word embeddings and co-occurrence frequency are also beneficial. nonetheless, these approaches neglect the potential meta-knowledge contained in the dataset that can guide the models to learn with only a small amount of samples. meta-learning has been suggested as an efficient strategy to acquire this knowledge. to acquire meta-knowledge, meta-learning constructs the tasks of few-shot learning scenarios and aims to learn how to achieve maximum performance by utilizing a limited amount of samples @xcite . following the idea of meta-learning (qian and yu, 2019), for the first time, we propose to investigate the problem of few/zero-shot labels in lmtc tasks from a meta-learning perspective. illustrated in fig. 1 , we simulate some few/zero-shot scenarios, which are faithful to the lmtc task and thereby provide chances for models to learn how to adapt fast and efficiently with a limited amount of data. however, most meta-learning algorithms are designed for multi-class classification under the fewshot setting @xcite , and it is critical for meta-learned models' generalization to construct faithful and diverse tasks @xcite . we argue that the simple extension of these approaches to multi-label classification is sub-optimal for the lmtc tasks in that (1) lmtc tasks need to cope with few-and zero-shot scenarios, while existing methods only consider few-shot ones, which is not faithful to the lmtc tasks. (2) lmtc tasks often face the challenge of long-tailed data distribution. however, these algorithms are not designed for specific data distribution and thereby makes the rare labels in the training set less involved in the meta-learning process, which reduces the diversity of the tasks. to address the above two issues, we propose an optimization-based meta-learning algorithm, namely meta-lmtc, which contains the metalearning phase and fine-tuning phase. we design a task sampling strategy when considering the characteristics of lmtc tasks (i.e., the coexistence of few-and zero-shot scenarios, long-tailed data distribution). during the meta-learning phase, this strategy not only constructs more faithful metalearning tasks (i.e., the zero-and few-shot scenarios coexist) but also provides more diverse labels and more various instances. then the model acquires meta-knowledge on these tasks through the alternating meta-training process and meta-evaluation process. during the fine-tuning phase, the metalearned model is fine-tuned on the original lmtc dataset to further improve performance. in summary, our contributions are as following: our work is a synthesis of two research directions: large-scale multi-label text classification and metalearning. we review them in this section. ## meta-learning for lmtc in this section, we first define lmtc tasks from the meta-learning perspective. then we present a detailed description of the proposed meta-lmtc. ## experiments in this section, we conduct several experiments to evaluate the efficacy of our method in lmtc tasks. the experimental result shows that our method can bring performance improvements to all of the few/zero-shot lmtc base models. ## discussion in this section, we report some experimental results of evaluating few-and/or zero-shot labels in the lmtc tasks in stricter settings. ## conclusion and future work in this paper, we proposed an optimization-based meta-learning framework, namely meta-lmtc, along with several task sampling strategies. we are the first study to address the lmtc tasks from a meta-learning perspective. extensive experimental results showed that our method is able to significantly improve the performance of all the base models. the further analysis presented that our method is also applicable to the strong bertlike model, and revealed the source of the performance boost our method brings. as future work, we will further explore the meta-learning approaches to handle the generalized zero-shot learning problem (gzsl) in the lmtc tasks.