Hierarchical Multi-Label Classification of Scientific Documents (2022)

Abstract: Automatic topic classification has been studied extensively to assist managing and indexing scientific documents in a digital collection. With the large number of topics being available in recent years, it has become necessary to arrange them in a hierarchy. Therefore, the automatic classification systems need to be able to classify the documents hierarchically. In addition, each paper is often assigned to more than one relevant topic. For example, a paper can be assigned to several topics in a hierarchy tree. In this paper, we introduce a new dataset for hierarchical multi-label text classification (HMLTC) of scientific papers called SciHTC, which contains 186,160 papers and 1,234 categories from the ACM CCS tree. We establish strong baselines for HMLTC and propose a multi-task learning approach for topic classification with keyword labeling as an auxiliary task. Our best model achieves a Macro-F1 score of 34.57% which shows that this dataset provides significant research opportunities on hierarchical scientific topic classification. We make our dataset and code for all experiments publicly available.

## introduction with the exponential increase of scientific documents being published every year, the difficulty in managing and categorizing them in a digital collection is also increasing. while the enormity of the number of papers is the most important reason behind it, the problem can also be assigned to the large number of topics. it is a very difficult task to index a paper in a digital collection when there are thousands of topics to choose from. fortunately, the large number of topics can be arranged in a hierarchy because, except for a few general topics, all topics can be seen as a sub-area of another topic. after arranging the topics in a hierarchy tree, the task of categorizing a paper becomes much simpler since now there are only a handful of topics to choose from at each level of the hierarchy. however, manual assignment of topics to a large number of papers is still very difficult and expensive, making an automatic system of hierarchical classification of scientific documents a necessity. after arranging the topics in a hierarchy, the classification task no longer remains a multi-class classification because in multi-class classification, each paper is classified into exactly one of several mutually exclusive classes or topics. however, if the topics are arranged in a hierarchy, they no longer remain mutually exclusive: a paper assigned to a topic node a in the hierarchy also gets assigned to topic node b, where b is a node in the parental history of a. for example, if a paper is classified to the area of natural language processing (nlp), it is also assigned to the topic of artificial intelligence (ai) given that ai is in the parental history of nlp. this non-mutual exclusivity among the topics makes the task a multi-label classification task. despite being an important problem, hierarchical multi-label topic classification (hmltc) has not been explored to a great extent in the context of scientific papers. most works on hierarchical and/or multi-label topic classification focus on news articles @xcite and use the rcv-1 dataset @xcite for evaluation. this is partly because of a lack of datasets for hierarchically classified scientific papers, which hinders progress in this domain. precisely, the existing multi-label datasets of scientific papers are either comparatively small @xcite or the label hierarchy is not deep @xcite . therefore, we address the scarcity of datasets for hmltc on scientific papers by introducing a new large dataset called scihtc in which the papers are hierarchically classified based on the acm ccs tree. 2 our dataset is large enough to allow deep learning exploration, comprising 186, 160 re-search papers that are organized into 1, 233 topics, which are arranged in a six-level deep hierarchy. we establish several strong baselines for both hierarchical and flat multi-label classification for scihtc. in addition, we conduct a thorough investigation on the usefulness of author specified keywords in topic classification. furthermore, we show how multi-task learning with scientific document classification as the principal task and its keyword labeling as the auxiliary task can help improve the classification performance. however, our best models with scibert @xcite achieve only 34.57% macro-f1 score which shows that there is still plenty of room for improvement. ## related work to date, several datasets exist for topic classification of scientific papers. @xcite created a hierarchically classified dataset of scientific papers from the web of science (wos). 3 however, their hierarchy is only two levels deep and the size of their dataset is 46, 985, which is much smaller than its counterpart from news source data. in addition, there are only 141 topics in the entire hierarchy. the cora foot_3 dataset introduced by @xcite is also hierarchically classified with multiple labels per paper and contains about 50, 000 papers. the hierarchy varies in depth from one to three and has 79 topics in total. however, the widely used version of cora foot_4 contains only 2, 708 papers @xcite and is not hierarchical. similarly, the labeled dataset for topic classification of scientific papers from cite-seer foot_5 @xcite is also very small in size containing only 3, 312 papers with no hierarchy over the labels. @xcite created a dataset of 55, 840 arxiv papers where each paper is assigned multiple labels using a two-level deep topic hierarchy containing a total of 54 topics. similar to us, santos and rodrigues (2009) proposed a multi-label hierarchical document classification dataset using the acm category hierarchy. however, our dataset is much larger in size than this dataset (which has ≈ 15, 000 documents in their experiment setting). furthermore, the dataset by @xcite is not available online and cannot be reconstructed as the acm paper ids are not provided. recently, @xcite released a dataset of 25, 000 papers collected from the microsoft academic graph foot_6 (mag) as part of their proposed evaluation benchmark for document level research on scientific domains. although the papers in mag are arranged in a five level deep hierarchy @xcite , only the level one categories (19 topics in total) are made available with the dataset. in contrast to the above datasets, scihtc has 1, 233 topics arranged in a six level deep hierarchy. the total number of papers in our dataset is 186, 160 which is significantly larger than all other datasets mentioned above. the topic hierarchy of each paper is provided by their respective authors. since each paper in our dataset is assigned to all the topics on the path from the root to a certain topic in the hierarchy tree, our dataset can be referred to as a multi-label dataset for topic classification. for multi-label classification, there are two major approaches: a) training one model to predict all the topics to which each paper belongs @xcite @xcite ; and b) training one-vs-all binary classifiers for each of the topics @xcite . the first approach learns to classify papers to all the relevant topics simultaneously, and hence, it is better suited to leverage the interlabel dependencies among the labels. however, despite that it is simpler and more time efficient, it struggles with data imbalance @xcite . on the other hand, the second approach gives enough flexibility to deal with the different levels of class imbalance but it is more complex and not as time efficient as the first one. in general, the second approach takes additional steps to encode the inter-label dependencies among the co-occurring labels. for example, in hierarchical classification, the parameters of the model for a child topic can be initialized with the parameters of the trained model for its parent topic @xcite @xcite . in this work, we take both approaches and compare their performance. besides these approaches, another approach for hierarchical and/or multi-label classification in recent years is based on sequence-to-sequence models @xcite , which we explored in this work. however, these models failed to show satisfactory performance on our dataset. we also explored the hierarchical classification proposed by @xcite where a local classifier is trained at every node of the hierarchy, but this model also failed to gain satisfactory performance. @xcite proposed the use of keywords together with traditional ensemble methods to classify scientific papers. however, since ground truth keywords were not available for the papers in their dataset, the authors explored a frequency based keyword selection, which gave the best performance. therefore, their application of keyword extraction methods for the classification task can be seen as a feature selection method. our scihtc dataset, in addition to being very large, multi-labeled, and hierarchical, contains the author-provided keywords for each paper. in this work, we present a thorough investigation of the usefulness of keywords for topic classification and propose a multi-task learning framework @xcite that uses keyword labeling as an auxiliary task to learn better representations for the main topic classification task. ## the scihtc dataset we constructed the scihtc dataset from papers published by the acm digital library, which we requested from acm. precisely, the dataset provided by acm has more than 300, 000 papers. however, some of these papers did not have their author-specified keywords, whereas others did not have any category information. thus, we pruned all these papers from the dataset. finally, there were 186, 160 papers which had all the necessary attributes and the category information. the final dataset was randomly divided into train, development and test sets in an 80 : 10 : 10 ratio. the number of examples in each set can be seen in category hierarchy score @xmath0, data generation, invariant, pairing function, algebra, exhaustive testing, random testing, lazy evaluation, program inversion, dsl, scife topics such as "hardware," "networks," "security and privacy," etc. note that ccs itself does not represent any topic (or category). it is simply the root of the acm hierarchy tree. there are 6 levels in the hierarchy tree apart from the root 'ccs'. that is, the maximum depth among the leaf nodes in the tree is 6. however, note that the depths of different sub-branches are not uniform and there are leaf nodes in the tree with a depth less than 6. each paper in our dataset is assigned to one or more sub-branches of the hierarchy tree by their respective authors with different depth levels and relevance scores among {100, 300, 500} with 500 indicating the most relevant. the authors also provide a set of keywords relevant to their paper. there are 1, 233 different topics in total in our final dataset. however, we find that the distribution of the number of papers over the topics is very imbalanced and a few topics (especially in the deeper levels of the hierarchy) had extremely low support (i.e., rare topics). thus, for our experiments, we only consider the topics up to level 2 of the ccs hierarchy tree which had at least 100 examples in the training set. figure 1 shows the number of papers in each of the 95 topics up to level 2 of the hierarchy tree in our dataset. we also report the explicit topic distribution (i.e., topic name vs. support) in appendix a. note that since there are 12 topics (among the 95 topics up to level 2 of the hierarchy) with less than 100 examples in the training set, we remove them and experiment with the remaining 83 topics. although we do not use the topics with low support in our experiments, we believe that they can be potentially useful for hierarchical topic classification of rare topics. therefore, we make available not only the two-level hierarchy dataset used in our experiments but also all relevant topics for each paper from the six-level hierarchy tree. ## methodology this section describes the hierarchical and flat multi-label baselines used in our experiments ( §4.1); after that, it introduces our simple incorporation of keywords into the models ( §4.2); lastly, it presents our multi-task learning framework for topic classification ( §4.3). problem definition let p be a paper, t be a topic from the set of all topics t , and n be the number of all topics in t ; and let x p denote the input text and y p denote the label vector of size n corresponding to p. for our baseline models, x p is a concatenation of the title and abstract of p. the goal is to predict the label vector y p given x p such that, if p belongs to topic t, y p @xmath1, and y p @xmath2, i.e., identify all topics relevant to p. ## experiments and results we perform the following experiments. first, we study the difficulty of classifying topics for scientific papers from our dataset in comparison with related datasets ( §5.1). second, we show the impact of the hierarchy of topics on models' performance and how incorporating keywords can help improve the performance further ( §5.2). third, we evaluate the performance of our proposed multi-task learning approach ( §5.3). implementation details are reported in appendix d. ## analysis and discussion from our experiments, it is evident that all of our hierarchical baselines can outperform their flat counterparts. but it is not clear whether the performance gain comes from using the hierarchy to better learn the parent-child dependency or it is because we allow the models to focus on each class individually by training one-vs-all binary classifiers in our hierarchical setting as opposed to one flat model for all the classes. in addition, our experiments also show that keywords can be used in multiple ways to improve topic classification performance. however, it is unclear whether or not keywords by themselves can achieve the optimal performance. thus, we analyze our models in these aspects with the following experiments. hierarchical vs. n-binary we conduct an experiment with scibert where we train a binary classifier for each class similar to the hierarchical scibert model but do not initialize it with its parent's model parameters, i.e., we do not make use of the topic hierarchy. we compare the performance with keywords vs. only keywords we experiment with flat bilstm, bert and scibert models with only keywords as the input. a comparison of these only keywords models with the models which use title, abstract and keywords can be seen in ## conclusion in this paper, we introduce scihtc, a new dataset for hierarchical multi-label classification of scientific papers and establish several strong baselines. our experiments show that scihtc presents a challenging benchmark and that keywords can play a vital role in improving the classification performance. moreover, we propose a multi-task learning