Multi-label Few/Zero-shot Learning with Knowledge Aggregated from Multiple Label Graphs (2020)

Abstract: Few/Zero-shot learning is a big challenge of many classifications tasks, where a classifier is required to recognise instances of classes that have very few or even no training samples. It becomes more difficult in multi-label classification, where each instance is labelled with more than one class. In this paper, we present a simple multi-graph aggregation model that fuses knowledge from multiple label graphs encoding different semantic label relationships in order to study how the aggregated knowledge can benefit multi-label zero/few-shot document classification. The model utilises three kinds of semantic information, i.e., the pre-trained word embeddings, label description, and pre-defined label relations. Experimental results derived on two large clinical datasets (i.e., MIMIC-II and MIMIC-III ) and the EU legislation dataset show that methods equipped with the multi-graph knowledge aggregation achieve significant performance improvement across almost all the measures on few/zero-shot labels.

## introduction multi-label learning is a fundamental and practical problem in computer vision and natural language processing. many tasks, such as automated medical coding @xcite @xcite , recommender systems @xcite , image classification @xcite , law study @xcite , and stance detection @xcite can be formulated as a multi-label learning problem. different from multiclass classification, an instance in multi-label learning is often associated with more than one class label, which makes the task even more challenging due to the combinatorial nature of the label space. in real-world applications, there are often insufficient or even unavailable training data of ever emerging classes @xcite . for instance, more than half of the international classification of diseases (icd) codes are not associated with a discharge summary in the mimic-iii dataset @xcite . as a solution, zero-shot learning @xcite aims to generalize classifiers to unseen classes by leveraging various label semantics. those classifiers are required to recognise instances of classes that have never been seen in the training set, which becomes more difficult in multi-label learning. moreover, the number of classes can reach hundreds of thousands. the icd-9-cm taxonomy contains 17k diagnosis/procedure codes foot_0 , where the majority occurs less than 10 times in mimic-iii; the eu legislation corpus (eurlex57x) @xcite ) contains about 7k labels, 70%of which have been assigned to less than 10 documents. the power-law distribution of labels @xcite @xcite leads to the few-shot learning challenge, where each label has a few training instances. classes come naturally with structures, which capture different relationships between individual classes. for example, codes in the icd-9-cm taxonomy are organised in a rooted tree with edges representing is-a relationships between parents and children @xcite . we can compute a code similarity graph using the code description and a code co-occurrence graph using the annotated discharge summaries in mimic-ii/iii. these two graphs can capture label relationships that are missing in the taxonomy. for example, the sim-ilarity graph can reveal the relationship between "hypertensive chronic kidney disease" and "acute kidney failure"; the co-occurrence graph can give us information about that "coronary atherosclerosis of native coronary artery" frequently co-occurs with "coronary arteriography using two catheters". it has been shown that ignoring this structured information and assuming all classes to be mutually exclusive are insufficient @xcite @xcite . in this paper, we present a simple but effective multi-graph knowledge aggregation model that can transform and fuse the structural information from multiple label graphs while utilising three kinds of semantics: the pre-trained word embeddings, label description, and the label relations. to demonstrate its efficacy, we adapt the model as a sub-module to several existing neural architectures @xcite for multilabel few/zero-shot learning. however, this model can work as a self-contained module and be flexibly adapted to most existing multi-label learning models @xcite ) that use gcns to leverage the label structures. experiments on three real-world datasets show that neural classifiers equipped with our multi-graph knowledge aggregation model can significantly improve the few/zero-shot classification performance. ## related work leveraging structural label information via gcns @xcite has become a promising approach of tackling the few/zero-shot problem, attracting increasing attention in recent years. @xcite ; @xcite , and @xcite have used gcns to learn visual classifiers for multi-class image classification. these ideas can be generalised to multi-label learning @xcite @xcite @xcite . however, none of these methods can be directly adapted to multi-label few/zero-shot text classification. using the label-wise attention mechanism @xcite , @xcite introduced an attention-based cnn to convert each document into a feature matrix, each row of which is a label-specific document feature vector. the multi-label document classifiers were learned from a gcn over the label hierarchy. while considering only the efficiency of the document encoder, @xcite ; li and yu (2020); @xcite further proposed to replace the simple cnn with bigru, multi-filter residual cnn and densely-connected cnn respectively. in contrast, our work focuses on the learning of the classifiers from multiple label graphs. existing work on multiple graphs learning often proposed to either fuse multiple graphs before fed into a gcn @xcite or consider the multi-dimensionality of graphs @xcite for only note classification/link prediction. ## learning with knowledge aggregation problem formulation let c s and c u be disjoint sets of seen and unseen labels. c s is further divided into frequent labels c r s and few-shot labels c f s such that c @xmath0. given a training set {(x 1 , y 1 ), . . . , (x n , y n )}, where x i indicates the i-th document and y i ⊂ c s is the subset of labels assigned to x i , the goal is to predict ŷi for each test document in generalised zero-shot settings @xcite , where ŷi is a subset of c s ∪ c u . note that: i) every label has a description; ii) the label relationships encoded in graphs can be computed from various resources; iii) documents associated with any label from c u are excluded from training. ## experiments in this section, several experiments were conducted to evaluate the efficacy of kamg in classifying discharge summaries and legislative documents. we compared our methods with several state-of-theart multi-label classifiers in a few/zero-shot setting, and studied how kamg behaves by varying label graphs in a set of ablation experiments. datasets we used two benchmark medical datasets (mimic ii and iii) and the eu legislation dataset (eurlex57k) to evaluate our method in the few/zero-shot settings. statistics of these datasets are shown in experiment settings and metrics for mimic-ii/iii, we used the neuralclassifier @xcite as a base framework to implement our methods. we used 200 filters with kernel size 10 to setup the cnns by following @xcite and the gcns' hidden layer size was set to 200. for eurlex57k, we leveraged chalkidis et al. results on mimic-ii/iii we compared kamg, which uses all three label graphs (h g , h s and h c ), with the following baselines: cnn, rcnn (the best model in @xcite ), caml, dr-caml, zacnn and zagcnn. table-2 shows the performance of all those models. kamg outperforms the other models in all the metrics across almost all the settings on both datasets with a notable margin, due to our multi-graph knowledge aggregation model. specifically, while classifying zero-shot labels, acnn-kamg outperforms zagcnn, which uses only the label hierarchy (i.e., h g ), by 8% in r@10 and 6.5% in ndcg@10 on mimic-ii and 4.1% in r@10 and 10.5% in ndcg@10 on mimic-iii. similarly, acnn-kamg gains 4% in r@10 and 2.7% in ndcg@10 on mimic-ii and 3.7% in r@10 and 6.5% in ndcg@10 on mimic-iii over zagcnn on few-shot labels. ## conclusion we have proposed a multi-graph aggregation method that can effectively fuse knowledge from multiple label graphs. experiments on mimic-ii/iii and eurlex57k have shown that the classifiers derived from the multi-graph aggregation have achieved substantial performance improvements particularly on few/zero-shot labels. as future work, we will further study our method's ability of extreme multi-label learning @xcite and different document encoders. frequent few zero overall r@1 r@5 r@10 r@1 r@5 r@10 r@1 r@5 r@10 r@1 r@5 r@10