Revisiting Hierarchical Text Classification: Inference and Metrics (2024)

Abstract: Hierarchical text classification (HTC) is the task of assigning labels to a text within a structured space organized as a hierarchy. Recent works treat HTC as a conventional multilabel classification problem, therefore evaluating it as such. We instead propose to evaluate models based on specifically designed hierarchical metrics and we demonstrate the intricacy of metric choice and prediction inference method. We introduce a new challenging dataset and we evaluate fairly, recent sophisticated models, comparing them with a range of simple but strong baselines, including a new theoretically motivated loss. Finally, we show that those baselines are very often competitive with the latest models. This highlights the importance of carefully considering the evaluation methodology when proposing new methods for HTC

## introduction text classification is a long-studied problem that may involve various types of label sets. in particular, hierarchical text classification (htc) includes labels that exhibit a hierarchical structure with parent-child relationships. the structure that emerges from these relationships is either a tree @xcite @xcite @xcite or a directed acyclic graph (dag) @xcite . each input text then comes with a set of labels that form one or more paths in the hierarchy. a first crucial challenge in htc lies in accurately evaluating model performance. this requires metrics that are sensitive to the severity of prediction errors, penalizing mistakes with larger distances within the hierarchy tree. while pioneering efforts have been made by @xcite , @xcite , @xcite and @xcite , evaluation in the ## evaluation metrics the aforementioned inference rule used in recent htc literature corresponds to a classical multilabel evaluation methodology: computing a f1score (micro and macro) with τ = 0.5. in what follows, we show that this thresholding scheme is suboptimal and we introduce the metrics we use in our experiments. we will then motivate the use of an inference-free evaluation methodology. ## simple conditional loss-based methods as a counterpart to the existing state-of-the-art consisting mainly of bce-based approaches, we introduce several loss-based methods that incorporates local information, all relying on estimating conditional probabilities. ## experimental settings in this section, we introduce the existing datasets and models we experiment with; we also present our new dataset, hierarchical wikivitals (hwv). ## conclusion in this paper, we come back upon recent progress in htc, and propose to investigate its evaluation. to do so, we begin by showing the limitations of the inference and metrics that are commonly used in the recent literature. we instead propose to use existing hierarchical metrics, and an associated inference method. then, we introduce a new and challenging dataset, hierarchical wikivitals; our experiments show that recent sophisticated hierarchy-aware models have trouble integrating hierarchy information in any better way than simple baselines. we finally propose simple hierarchical losses, able to better integrate hierarchy information on our dataset. in the future, we plan to investigate the inference mechanism for hierarchical metrics, through which we will aim to make a direct contribution to improving models on htc tasks.