Extensive Error Analysis and a Learning-Based Evaluation of Medical Entity Recognition Systems to Approximate User Experience (2020)

Abstract: When comparing entities extracted by a medical entity recognition system with gold standard annotations over a test set, two types of mismatches might occur, label mismatch or span mismatch. Here we focus on span mismatch and show that its severity can vary from a serious error to a fully acceptable entity extraction due to the subjectivity of span annotations. For a domain-specific BERT-based NER system, we showed that 25% of the errors have the same labels and overlapping span with gold standard entities. We collected expert judgement which shows more than 90% of these mismatches are accepted or partially accepted by the user. Using the training set of the NER system, we built a fast and lightweight entity classifier to approximate the user experience of such mismatches through accepting or rejecting them. The decisions made by this classifier are used to calculate a learning-based F-score which is shown to be a better approximation of a forgiving user’s experience than the relaxed F-score. We demonstrated the results of applying the proposed evaluation metric for a variety of deep learning medical entity recognition models trained with two datasets.

## introduction named entity recognition (ner) in medical texts involves the automated recognition and classification of relevant medical/clinical entities, and has numerous applications including information extraction from clinical narratives @xcite , identifying potential drug interactions and adverse affects @xcite , and de-identification of personal health data @xcite . in recent years, medical ner systems have improved over previous baseline performance by incorporating developments such as deep learning models @xcite , contextual word embeddings @xcite , and domain-specific word embeddings @xcite @xcite . typically, research groups report their results using common evaluation metrics (most often precision, recall, and f-score) on standardized data sets. while this facilitates exact comparison, it is difficult to know whether modest gains in f-score are associated with significant qualitative differences in the system performance, and how the benefits and drawbacks of different embedding types are reflected in the output of the ner system. this work aims to investigate the types of errors and their proportion in the output of modern deep learning models for medical ner. we suggest that an evaluation metric should be a close reflection of what users experience when using the model. we investigate different types of errors that are penalized by exact f-score and identify a specific error type where there is high degrees of disagreement between the human user experience and what exact f-score measures: namely, errors where the extracted entity is correctly labeled, but the span only overlaps with the annotated entity rather than matching perfectly. we obtain expert human judgement for 5296 such errors, ranking the severity of the error in terms of end user experience. we then compare the commonly used f-score metrics with human perception, and investigate if there is a way to automatically analyze such errors as part of the system evaluation. the code that calculates the number of different types of errors given the predictions of an ner model and the corresponding annotations is available upon request and will be released at https://github.com/nrc-cnrc/ nrc-medner-eval after publication. we will also release the collected expert judgements so that other researchers can use it as a benchmark for further investigation about this type of errors. ## what do ner evaluation metrics an output entity from an ner system can be incorrect for two reasons: either the span is wrong, or the label is wrong (or both). although entity-level exact f-score (also called strict f-score) is established as the most common metric for comparing ner models, exact f-score is the least forgiving metric in that it only credits a prediction when both the span and the label exactly match the annotation. other evaluation metrics have been proposed. the message understanding conference (muc) used an evaluation which took into account different types of errors made by the system @xcite . building on that work, the semeval 2013 task 9.1 (recognizing and labelling pharmacological substances in biomedical text) employed four different evaluations: strict match, in which label and span match the gold standard exactly, exact boundary match, in which the span boundaries match exactly regardless of label, partial boundary match, in which the span boundaries partially match regardless of label, and type match, in which the label is correct and the span overlaps with the gold standard @xcite . the latter metric, also commonly known as inexact match, has been used to compute inexact or relaxed f-score in the i2b2 2010 clinical ner challenge @xcite . relaxed f-score and exact f-score are the most frequently used evaluation metrics for measuring the performance of medical ner systems @xcite . other biomedical ner evaluations have accepted a span as a match as long as either the right or left boundary is correct @xcite . in bionlp shared task 2013, the accuracy of the boundaries is relaxed or measured based on similarity of entities @xcite . another strategy is to annotate all possible spans for an entity and accept any matches as correct @xcite , although this detailed level of annotation is rare. here, we focus on the differences between what f-score measures and the user experience. in the case of a correct label with a span mismatch, it is not always obvious that the user is experiencing an error, due to the subjectivity of span annotations @xcite . existing evaluation metrics treat all such span mismatches equally, either penalizing them all (exact f-score), rewarding them all (relaxed f-score), or based on oversimplified rules that do not general-ize across applications and data sets. we use both human judgement and a learning-based approach to evaluate span mismatch errors and the resulting gap between what f-score measures and what a human user experiences. we only consider the information extraction task and not any specific downstream task. ## types of errors in ner systems while the semeval 2013 task 9.1 categorized different types of matches for the purpose of evaluation, we further categorize mismatches for the sake of error analysis. we consider five types of mismatches between annotation and prediction of the ner system. reporting and comparing the number of these mismatches alongside an averaged score such as f-score can shed light on the differences of ner systems. an entity is predicted by the ner model, but is not annotated in the hand-labelled text. we focus on type-5 errors and show that treating these mismatches is not a trivial task. previous works have shown that some type-5 mismatches are completely wrong predictions while others are fully acceptable predictions resulting from the subjectivity and inconsistency of span annotations @xcite . figure 1 shows several examples of error type-5. in the first example, an adenosine -thallium stress test is annotated as a test, while the ner system extracts thallium stress test as a test. here, what ner extracted is partially correct but misses an figure 1 : examples of type-5 error. we used the visualisation tool developed in @xcite important part of the entity. whether the extracted entity is acceptable may depend on the downstream task. in the next sentence, patchy consolidation in all lobes is annotated as a problem, but the ner system extracted patchy consolidation in all lobes of both longs as the problem. here, the system's prediction is more complete than the annotated entity, and so it appears to be a fully acceptable prediction. in the last example, according to human annotation, 1cm cyst in the right lobe of the liver is a problem, but the ner system extracts two entities from the same phrase, 1) 1cm cyst in the right lobe as a problem and 2) liver as another problem. while the first extracted entity is correct and may be acceptable the second one is completely wrong. ## datasets and models we consider two medical text datasets, one clinical and the other biomedical. we analyse the errors of three models for each dataset to cover a variety of deep learning models. the i2b2 dataset was annotated by community annotators with carefully crafted guidelines. the ground truth generated by the community obtained f-measures above 0.90 against the ground truth of the experts @xcite . medmentions dataset: the medmentions dataset was released in 2019 and contains 4,392 abstracts from biomedical articles on pubmed @xcite . the abstracts are annotated for umls concepts and semantic types. the fully annotated dataset contains 127 semantic types and these classes are highly-imbalanced. the creators of the dataset also provide a version which has been annotated with only a subset of the most relevant concepts, called 'st21pv' (21 semantic types from preferred vocabularies); we consider this version in the current work. while fewer papers have been published on medmentions to date, it represents an interesting challenge to nlp systems due to its imbalanced and high number of classes, and some observed inconsistencies in the annotations @xcite . there are 3513 documents (162,908 entities) in the st21pv train set and 879 documents (40,101 entities) in the test set. medmentions was annotated by a team of profes-sional annotators with rich experience in biomedical content curation. the precision of the annotation in medmention is estimated as 97.3% @xcite . ## analysis of error types across models and datasets further investigation of type-5 errors is only worthwhile if a significant proportion of the errors belong to this group. we looked at the distribution of error types across datasets and ner models, described in section 4, and visualized the results in figure 2 . by calculating the distribution of error types, we observed that for all assessed models at least 20% of the errors are recognized as type-5 mismatches. moreover, for both datasets, we observed that better ner models generate more type-5 errors. models based on general bert outperform glovebased models in terms of both exact and relaxed f-score and they also generate relatively more type-5 errors. same pattern is observed when comparing domain-specific bert models with general bert models. this observation may be explained with the fact that contextualized embeddings combine the meaning of words through attention mechanism and the span information might be more vague in the resulting representation. figure 3 shows exact f-score, relaxed f-score and the proportion of type-5 mismatches to the total number of errors, for all the models and datasets. this analysis implies that proper handling of type-5 errors becomes more important for comparison of modern strong ner systems. ## expert judgement on type-5 errors we considered an information extraction task and asked a medical doctor to assess the type-5 errors made by the biobert ner model on the st21pv dataset and either confirm or reject the extracted entity with granular scores. our goal is to: 1) investigate the proportion of type-5 extracted entities that are acceptable, 2) set a benchmark of human experience from type-5 errors. human judgement scheme : the following scoring scheme is used by the expert for scoring the acceptability of type-5 mismatches for the biobert-based model trained with the st21vp dataset. the type-5 mismatches are identified and the expert is given the original sentence in the test set, the annotated (gold-standard) entity, and the entity predicted by the ner model for all 5296 type-5 mismatches. ## entity classifier for automatic refining of type-5 mismatches we propose that an entity classifier can be trained to predict the tag of entities extracted by the ner model and the predicted tag can be used to distinguish between acceptable and unacceptable type-5 errors. figure 5 shows the workflow of the proposed method. using the training dataset of the ner model, we train an entity classifier with gold standard entities as inputs and their assigned tags as outputs. for this classifier, the span is given and the tag is the only information that has to be learned. although the full context of the sentence helps the ner model to learn a better representation of the entity, many entities can be classified without seeing the full sentence and this is what the entity classifier learns. for type-5 entities, the human annotators and the ner already agree on the tag and it is only the span that is in disagreement. so, the intuition here is that the entity classifier can confirm or reject the tag predicted by ner, given the identified span. this classifier is meant to play a third party role that has seen the variety of span annotations in the training dataset and performs the task that the human expert did in section 6. this classifier is trained once for each dataset and is not dependent on the type of the ner model. ## refining type-5 mismatches across datasets and models figure 7 shows how the entity classifier refines type-5 errors across models and datasets. consistently, a significant proportion of type-5 errors are accepted by the entity classifier. for example, for the glove-based model trained on i2b2 dataset, the entity classifier accepts 90% of type-5 errors which is 26.6% of the total number of the errors penalized by the exact f-score. the proportion of accepted type-5 mismatches to the total number of errors is 31.11% for i2b2+bert, 33.23% for i2b2+clinicalbert, 17.95% for st21pv+glove, 19.55% for st21pv+bert and 19.36% for st21pv+biobert. to sum up, about 20% to 30% of the mismatches penalized by exact f-score are accepted by the entity classifier. ## learning-based f-score the trained entity classifier can be leveraged for f-score calculation. here, instead of penalizing all the type-5 mismatches as in exact f-score or rewarding all of them in relaxed f-score, we penalize the type-5 mismatches that are rejected by the classifier and reward the rest of them. in other words, this f-score penalizes errors of type-1, type-2, type-3, type-4 and the rejected type-5 mismatches. accepted type-5 mismatches and exact matches are rewarded. ## discussion we highlighted the fact that when we evaluate ner systems by comparing extracted and annotated entities across a test set, for a significant part of the errors that are penalized by the exact f-score, the la-bel is recognized correctly and the span has overlap with the annotated entity. we referred to this type of error as type-5 mismatch and for six ner models (3 model structures and 2 datasets) showed that at least 20% of the errors belong to this category. the previous literature has raised the issue that in the case of medical ner, many such predictions are valid and useful entity extractions and penalizing them is a flaw of evaluation metrics. however, distinguishing between acceptable and unacceptable predictions when the label is correct and the span overlaps is not trivial. we argue that the best evaluation metric is the one that reflects the human experience of the system best. we collected human judgement about a all type-5 errors made by a ner model based on biobert embeddings, trained with st21pv dataset and showed that almost 70% of such errors are completely acceptable and only 10% of them are rejected by the user. the rest of the predictions are acceptable entities for the associated tags but lack important information when seen in the context. setting human experience as a benchmark, we suggested that expert judgement can be approximated by a decision made by an entity classifier. the entity classifier can be trained using the training set of an ner. while the ner model looks at the context and identifies the type of the entity and a partially correct span, this classifier looks at the extracted entities out of context and decides whether with the partially correct span, the extracted entity can still belong to the predicted class or not. the entity classifier trained on st21pv dataset accepts more than 80% of type-5 errors made by biobert-based ner model trained with the same dataset, 96% of which is also accepted by the expert user. the proposed entity classifier is trained for each ner training set once and can be used to evaluate any ner model trained on that dataset, regardless of the structure of the ner model being evaluated. we used a computationally inexpensive model structure and encourage researchers to use this model in order to automatically evaluate type-5 mismatches. reporting the distribution of errors across all error types and also accepted and rejected type-5 errors, will allow us to compare our models in a variety of dimensions and sheds light on how these models behave differently for detecting labels and spans. accepting some type-5 errors as useful predictions can be translated to f-score calculation by not penalizing the accepted entity extractions. we did this calculation separately for the cases that were accepted by human expert or the classifier, and showed that the f-score resulting from the classifier is closer to the judgement of a forgiving user than both the exact and the relaxed f-score. in cases where a strict evaluation of the system is desired, exact f-score is a better approximation of human experience, due to the fact that the entity classifier is a forgiving one and accepts most of the cases that are partially accepted by the expert. we only collected human judgement on the decisions made by ner model for one model and one dataset. further investigation is needed to confirm or reject our observations and to investigate the limitations and potential capabilities of training an entity classifier alongside a ner model and using that for error analysis. also, further research is needed to find a way of distinguishing between partially accepted and accepted entity extractions, which is a necessary tool for measuring the experi-ence of a strict user. using extra sources of training data other than the ner training dataset may be a way to improve the judgements of the entity classifier. we used this classifier for error analysis and refining of type-5 errors. in future work, we can look at the possibility of using this classifier as a refining tool for all types of mismatches or a postprocessing tool without the need for annotation to identify the types of mismatches. ## conclusion medical ner systems that are based on most recent deep learning structures generate a high amount of outputs that match with the hand-labelled entities in terms of tag but only overlap in the span. while the exact f-score penalizes all of these predictions and relaxed f-score credits all of them, a human user accepts a significant proportion of them as valid entities and rejects the rest. a reformatted version of the ner training dataset can be used to train an entity classifier for evaluation of extracted entities with right label and overlapping span. we showed that there is a high degree of agreement between human expert and this entity classifier in accepting or rejecting span mismatches. this classifier is used to calculate a learning-based evaluation metric that outperforms relaxed f-score in approximating the experience of a forgiving user.