An Empirical Study on Large-Scale Multi-Label Text Classification Including Few and Zero-Shot Labels (2020)

Abstract: Large-scale Multi-label Text Classification (LMTC) has a wide range of Natural Language Processing (NLP) applications and presents interesting challenges. First, not all labels are well represented in the training set, due to the very large label set and the skewed label distributions of datasets. Also, label hierarchies and differences in human labelling guidelines may affect graph-aware annotation proximity. Finally, the label hierarchies are periodically updated, requiring LMTC models capable of zero-shot generalization. Current state-of-the-art LMTC models employ Label-Wise Attention Networks (LWANs), which (1) typically treat LMTC as flat multi-label classification; (2) may use the label hierarchy to improve zero-shot learning, although this practice is vastly understudied; and (3) have not been combined with pre-trained Transformers (e.g. BERT), which have led to state-of-the-art results in several NLP benchmarks. Here, for the first time, we empirically evaluate a battery of LMTC methods from vanilla LWANs to hierarchical classification approaches and transfer learning, on frequent, few, and zero-shot learning on three datasets from different domains. We show that hierarchical methods based on Probabilistic Label Trees (PLTs) outperform LWANs. Furthermore, we show that Transformer-based approaches outperform the state-of-the-art in two of the datasets, and we propose a new state-of-the-art method which combines BERT with LWAN. Finally, we propose new models that leverage the label hierarchy to improve few and zero-shot learning, considering on each dataset a graph-aware annotation proximity measure that we introduce.

## introduction large-scale multi-label text classification (lmtc) is the task of assigning a subset of labels from a large predefined set (typically thousands) to a given document. lmtc has a wide range of applications in natural language processing (nlp), figure 1 : examples from lmtc label hierarchies. ∅ is the root label. l l is the number of labels per level. yellow nodes denote gold label assignments. in eurlex57k, documents have been tagged with both leaves and inner nodes (gap: 0.45). in mimic-iii, only leaf nodes can be used, causing the label assignments to be much sparser (gap: 0.27). in amazon13k, documents are tagged with leaf nodes, but it is assumed that all the parent nodes are also assigned, leading to dense label assignments (gap: 0.86). apart from the large label space, lmtc datasets often have skewed label distributions (e.g., some labels have few or no training examples) and a label hierarchy with different labelling guidelines (e.g., they may require documents to be tagged only with leaf nodes, or they may allow both leaf and other nodes to be used). the latter affects graph-aware annotation proximity (gap), i.e., the proximity of the gold labels in the label hierarchy (see section 4.1). moreover, the label set and the hierarchies are periodically updated, thus requiring zero-and few-shot learning to cope with newly introduced labels. figure 1 shows a sample of label hierarchies, with different label assignment guidelines, from three standard lmtc benchmark datasets: eur-lex @xcite , mimic-iii @xcite , and amazon @xcite ). current state-of-the-art lmtc models are based on label-wise attention networks (lwans) @xcite , which use a different attention head for each label. lwans (1) typically do not leverage structural information from the label hierarchy, treating lmtc as flat multi-label classification; (2) may use the label hierarchy to improve performance in few/zero-shot scenario, but this practice is vastly understudied; and (3) have not been combined with pre-trained transformers. we empirically evaluate, for the first time, a battery of lmtc methods, from vanilla lwans to hierarchical classification approaches and transfer learning, in frequent, few, and zero-shot learning scenarios. we experiment with three standard lmtc datasets (eurlex57k; mimic-iii; ama-zon13k). our contributions are the following: we propose new lwan-based models with improved performance in these settings, taking into account the labelling guidelines of each dataset and a graph-aware annotation proximity (gap) measure that we introduce. ## bert-base-lwan (new) ## conclusions we presented an extensive study of lmtc methods in three domains, to answer three understudied questions on (1) the competitiveness of plt-based methods against neural models, (2) the use of the label hierarchy, (3) the benefits from transfer learning. a condensed summary of our findings is that (1) tf-idf plt-based methods are definitely worth considering, but are not always competitive, while attention-xml, a neural plt-based method that captures word order, is robust across datasets; (2) transfer learning leads to state-of-the-art results in general, but bert-based models can fail spectacularly when documents are long and technical terms get over-fragmented; (3) the best way to use the label hierarchy in neural methods depends on the proximity of the label assignments in each dataset. an even shorter summary is that no single method is best across all domains and label groups (all, few, zero) as the language, the size of documents, and the label assignment strongly vary with direct implications in the performance of each method. in future work, we would like to further investigate few and zero-shot learning in lmtc, especially in bert models that are currently unable to cope with zero-shot labels. it is also important to shed more light on the poor performance of bert models in mimic-iii and propose alternatives that can cope both with long documents @xcite and domain-specific terminology, reducing word over-fragmentation. pretraining bert from scratch on discharge summaries with a new bpe vocabulary is a possible solution. finally, we would like to combine plts with bert, similarly to attention-xml, but the computational cost of fine-tuning multiple bert encoders, one for each plt node, would be massive, surpassing the training cost of very large transformerbased models, like t5-3b @xcite and megatron-lm @xcite with billions of parameters (30-100x the size of bert-base).