Evaluating Hierarchical Document Categorisation (2021)

Abstract: Hierarchical document categorisation is a special case of multi-label document categorisation, where there is a taxonomic hierarchy among the labels. While various approaches have been proposed for hierarchical document categorisation, there is no standard benchmark dataset, resulting in different methods being evaluated independently and there being no empirical consensus on what methods perform best. In this work, we examine different combinations of neural text encoders and hierarchical methods in an end-to-end framework, and evaluate over three datasets. We find that the performance of hierarchical document categorisation is determined not only by how the hierarchical information is modelled, but also the structure of the label hierarchy and class distribution.

## introduction document categorisation is a core task in information retrieval and natural language processing, whereby documents are categorised relative to a pre-defined set of labels. while the majority of research on document categorisation assumes a flat label structure, in practice in large-scale document categorisation tasks, there is often hierarchical label structure, in the form of either a tree or directed acyclic graph @xcite , where "child" labels inherit the properties of their parents. the goal of hierarchical document categorisation is to classify documents into a set of labels, where there is a hierarchical relationship among the labels. hierarchical document categorisation methods explicitly capture the label structure during training. there has been a resurgence of interest in document categorisation in recent years, in part driven by breakthroughs in representation learning and pre-trained language models @xcite @xcite @xcite , which generate more expressive, general-purpose representations, thereby leading to performance gains across a range of nlp tasks. despite this, there has been relatively little recent work specifically on hierarchical document categorisation. what recent work does has varied wildly in the choice of text encoder and dataset, with no systematic, controlled cross-dataset evaluation to be able to make solid conclusions as to whether the reported performance gains are attributable to the proposed hierarchical document categorisation method or just the text encoders used. our work focuses on examining the capacity of existing methods dealing with labels with a hierarchical structure, which is different from the work of @xcite , which focuses on modelling documents in a hierarchical way to perform classic document classification task. in this work, we carry out systematic evaluation of a range of contemporary hierarchical document categorisation approaches, using a range of neural text encoders, based on three document collections with hierarchical label sets. ## related work hierarchical document categorisation methods can be grouped into: flat approaches, local approaches, global approaches, and hybrid methods, based on how they utilise the label hierarchy. ## conclusions we examine various combinations of text encoders and hierarchical categorisation methods in an endto-end fashion over three datasets. we find that the choice of text encoder is a strong determinant of performance than the choice of hierarchical method, and indeed that local hierarchical methods don't consistently outperform baseline flat classification methods. with regards to hierarchical methods, rr improves macro-f 1 at the cost of micro-f 1 on rcv1 and shinra, while hmcn improves micro-f 1 at the cost of macro-f 1 on rcv1 and shinra. an opposite trend is observed on wos, namely an improvement for rr and deterioration for hmcn. these different behaviours are determined by how the hierarchical label information is modelled during training. the global model hi-gcn achieves superior performance in terms of both micro-f 1 and macro-f 1 on all three datasets, indicating the necessity of capturing the hierarchy label structure holistically.