C o PHE : A Count-Preserving Hierarchical Evaluation Metric in Large-Scale Multi-Label Text Classification (2021)

Abstract: Large-Scale Multi-Label Text Classification (LMTC) includes tasks with hierarchical label spaces, such as automatic assignment of ICD-9 codes to discharge summaries. Performance of models in prior art is evaluated with standard precision, recall, and F1 measures without regard for the rich hierarchical structure. In this work we argue for hierarchical evaluation of the predictions of neural LMTC models. With the example of the ICD-9 ontology we describe a structural issue in the representation of the structured label space in prior art, and propose an alternative representation based on the depth of the ontology. We propose a set of metrics for hierarchical evaluation using the depth-based representation. We compare the evaluation scores from the proposed metrics with previously used metrics on prior art LMTC models for ICD-9 coding in MIMIC-III. We also propose further avenues of research involving the proposed ontological representation.

## introduction the large-scale multi-label text classification (lmtc) is a multi-label document classification task with a large (in the order of thousands) and potentially structured label space. lmtc tasks with structured label spaces can be found in the domain of medicine, legislation, product categorisation, etc. individual label spaces may follow different labelling rules, e.g., the eurlex @xcite dataset's labels can be any node from the eurovoc tree of concepts, while amazon13k (mcauley and leskovec, 2013) assumes a dense assignment, where if a predicted (leaf) node's truth value is positive, so is the truth value of all its ancestors -also known as the true path rule. a representative dataset for medical lmtc are mimic-iii @xcite discharge summaries. these are weakly labelled with leaf nodes of the icd-9 ontology foot_0 . icd-9 is a tree-structured ontology of medical conditions and procedures. since the release of the mimic-iii dataset there have been several attempts at training neural models for automated coding of medical documents @xcite @xcite @xcite . while some prior art has made use of ontological structure @xcite @xcite , the task has mostly been treated as a flat prediction of the assigned leaves. this is reflected in the evaluation metrics that are used across previous work -precision, recall and f 1 score on flat predictions. these metrics applied to flat vectors disregard the rich ontological structure. a notable exception is manginas et al. (2020) -they fine-tune a bert @xcite ) model, such that different layers within the model learn to represent the different depths of the target label space. each layer undergoes flat evaluation with respect to labels on its depth. the aim of this study is to review the use of ontological structure within the prior lmtc art with a focus on automated icd-9 coding, point out issues in hierarchy representation in prior approaches, describe methods more suited for addressing the structured label space including a hierarchical evaluation metric, and propose further avenues of research utilising structured label spaces. our implementation of evaluation and the representation of icd-9's graph are available to the community foot_1 . ## background within a structured label space some labels are inherently closer to one another than to others -e.g., in icd-9 425.0 endomyocardial fibrosis is closer to 425.3 endocardial fibroelastosis than to 305.1 tobacco use disorder. flat prediction and standard precision/recall/f 1 score (from hereon referred to as standard metrics) of individual prediction level (leaf) codes treat all mispredictions equally -e.g., having 425.0 be mispredicted as 425.3 is penalised the same way as mispredicting it as 305.1. this phenomenon has been addressed in information extraction (ie) by @xcite through the use of distance metrics. the ie setting assumes both the gold standard and predictions to be associated with specified spans within the input text. this means an individual prediction can be associated with a true label, allowing direct comparison between them. the lmtc setting uses weak labels -predictions and true labels appear on the document level, without exact association to spans within the text. due to the absence of information regarding associated spans, direct links between individual predictions and true labels do not exist. label comparison is performed on full vectors representing multiple labels, hence the ie approach is not directly usable. @xcite address hierarchical label spaces in document classification with set-based measures. the gold standard and prediction vectors are extended to include ancestor nodes within the hierarchical label-space and augmented according to its structure, and the true-path rule. let @xmath0, ..., m } and let @xmath1, ..., n } represent the set of predicted and true codes for a certain document respectively. assume we have access to an augmentation function an j (x) which returns the ancestors of x up to the j th order. let x @xmath2, ..., m } and let y @xmath3, ..., n } represent the set of predicted and gold ancestor codes for x and y respectively. standard metrics can then be applied to the x aug and y aug sets. a correct assignment of predicted lower-level (leaf) codes results in correct assignment of their ancestors. in the case of incorrect prediction-level assignments, the closer a mismatched predicted code is to the gold standard leaf within the ontology, the more matches will occur across the levels of the hierarchy. on the leaf level each code appears at most once per document. duplicates can occur when an j (x) produces the same ancestor for multiple codes. as x aug is a set, duplicates are removed. hence, the set-based approach captures whether an ancestor is present, but not how many of its descendants were predicted. this results in loss of information regarding over-/under-predictions of classes on the ancestral level. over-and under-prediction is a valuable phenomenon to track, particularly if the label-space includes inexplicit rules -for instance, for some nodes only a single descendant can be predicted at a time, as individual siblings are mutually exclusive (e.g., a patient can be assigned at most one of codes 401.0, 401.1, and 401.9, which represent malignant, benign, and unspecified hypertension respectively -concepts that are mutually exclusive). furthermore, retaining this numeric data on ancestral levels enables analyses on higher levels, e.g., performance of a family of codes in the case of a semi-automated code-assignment application. for this reason we propose using a metric that retains the descendant counts for these ancestor codes. to correctly define the augmentation function we need to ensure our representation of the hierarchy fits the setting. previous work involving the icd-9 hierarchy @xcite @xcite represents it through the relation of direct ancestry considering parents and grandparents of the leaf nodes. as the icd-9 has leaves at different depths, this representation results in structural issues, such as one code being both in the position of a parent and a grandparent for different leaves. for instance, code 364 has a parent relation to the leaf 364.3 and grandparent to leaf 364.11 (figure 1 ). this poses an issue to aggregation and evaluation. we aim to address this issue by producing a representation of the hierarchy through the levels of the label space with each level representing all the nodes at a certain depth in the tree structure. ## metric analysis we produced evaluation results with cophe on three prior art models on mimic-iii. 3 caml @xcite is the first notable lmtc model developed on mimic-iii whose contribution to the task was the introduction of label-wise attention. @xcite proposed an alteration on caml called bigru-lwan, (here re3 since bert based models are yet to be well adapted to mimic-iii icd coding @xcite , we leave bert models for a future study. ferred to as bglwan) swapping caml's cnn encoder with a bidirectional gru, and making use of zero-shot-motivated label description embeddings proposed by @xcite . @xcite further added label-wise word-and sentence-level attention and incorporated label embeddings pre-trained from the training labelsets (instead of the label descriptions) into the prediction and attention layers producing the hlan model. in the context of mimic-iii, results tend to be presented on two label sets -the 50 most frequent codes (top50) and the full codeset (full) -this stems from the big-head long-tail distribution of labels in mimic-iii with many labels being infrequent, or not appearing within the training set at all. we have used the pre-processing and top50 codeset dataset split of @xcite . our results are averaged across 10 runs and serve as a comparative example for the evaluation metrics. we have applied cophe up to the lowest chapter levels (one above e 0 ), which we simply refer here to as the chapter level (c). this choice was made due to the level above c already including the root of the ontology for some codes, leading to structural inconsistency. we compared the flat metric results against the results of the set-based and cophe hierarchical measures on the three models ( the scores of cophe are consistently lower than that of set-based evaluation, showing that cophe can capture more over-and/or under-prediction of the labels, as the true positives of the ancestor codes are better manifested in cophe. if all (binary) true positives within higher levels of the hierarchy reported by the set-based measure were true positives in cophe (no over-or under-predictions), this would result in a greater proportion of true positives for cophe than in the set-based measure, and hence higher scores (see the change of evaluation measure has not affected the ranking of the models on f 1 . this is likely due to a lack of representation of the label space structure within the explored models. while hierarchical measures enrich the evaluation with structural information, they do not serve as a replacement of the flat measures, but rather should be viewed side-by-side. an approach may not surpass the state-of-the-art on the flat measures, whilst being better at modelling the structure of the label space based on the hierarchical score. indication of better modelling of the structure should prompt further analysis of the model. ## conclusion we have proposed the use of hierarchical evaluation measures in the lmtc task involving hierarchical label spaces and provided an example in the task of automated icd coding. unlike the approaches in prior art of icd coding, which penalise all mispredictions equally, the proposed hierarchical evaluation measures adjust the penalty based on the performance on the ancestral levels. we have described a means to represent the hierarchy according to depth within the ontology. finally we have proposed the use of count-preserving evaluation which captures data on both over and underprediction in ancestral levels, as opposed to an existing set-based (binary) hierarchical evaluation approach.