Not a cute stroke: Analysis of Rule- and Neural Network-based Information Extraction Systems for Brain Radiology Reports (2020)

Abstract: We present an in-depth comparison of three clinical information extraction (IE) systems designed to perform entity recognition and negation detection on brain imaging reports: EdIE-R, a bespoke rule-based system, and two neural network models, EdIE-BiLSTM and EdIE-BERT, both multi-task learning models with a BiLSTM and BERT encoder respectively. We compare our models both on an in-sample and an out-of-sample dataset containing mentions of stroke findings and draw on our error analysis to suggest improvements for effective annotation when building clinical NLP models for a new domain. Our analysis finds that our rule-based system outperforms the neural models on both datasets and seems to generalise to the out-of-sample dataset. On the other hand, the neural models do not generalise negation to the out-of-sample dataset, despite metrics on the in-sample dataset suggesting otherwise.

## introduction information extraction (ie) from radiology reports is of great interest to clinicians given its potential for automating large scale data linkage, targeted cohort selection, retrospective statistical analyses, and clinical decision support @xcite . accurate ie from radiology reports has also received a surge of attention due to the insatiable demand of deep learning medical image classifiers for more labelled training data @xcite . while ie from radiology reports is of increasing value, the scarcity of annotated data and limited transferability of previously developed models is currently hindering progress. despite recent breakthroughs in learning contextual representations for clinical and biomedical text from large amounts of unlabelled text @xcite @xcite , labelled data scarcity remains the bottleneck to improvements and wider adoption of deep learning methods. data scarcity is even more prominent in the general clinical domain with its vast quantity of possible entity labels. existing approaches to overcome the lack of labelled data include using a rule-based system to annotate more data @xcite or propose labels in an annotation tool @xcite @xcite , leveraging semi-supervised learning to speed up annotation @xcite and creating artificial data @xcite . it is also common for rule-based systems to be developed alongside statistical models to contrast their performance @xcite @xcite . we need to understand the shortcomings and benefits of rule-based and neural models to improve annotation decisions and system evaluation, a comparison which we explore in this paper both on in-and out-of-sample data. the use of end-to-end learning for document labelling has been a recent trend in analysing radiology reports @xcite . contextual representations of a document such as bidirectional encoder representations from transformers (bert) @xcite are used as input to a multi-label classifier to label the report directly without first recognising named entities and negation. while such approaches make annotation simpler and faster and rely less on complex modelling decisions, they have various shortcomings. firstly, they lack in interpretability, as it is hard to probe which parts of a document the model uses when making predictions. some models employ an attention mechanism highlighting tokens in the input used to arrive at the decision @xcite . however, they are opaque as to the exact sub-decisions that lead to the labels, which is unsatisfactory in the clinical domain where interpretability is of paramount importance. secondly, they are not data-efficient. for example, @xcite predict four labels per entity type @xcite . to scale such approaches to more entity types a lot of annotated data is needed, the absence of which is currently a limiting factor. lastly, a significant drawback of end-to-end approaches is that no part of the system other than the encoder is reusable in any domain that has a different non-overlapping set of output labels. for that new domain, the labelling procedure needs to be initiated from scratch, leading to a duplication of effort. in this work, as in some previous neural approaches @xcite and as is common in rule-based approaches @xcite , we employ a bottom up approach to document labelling by factoring the problem into sub-tasks. this way document labels are interpretable as a sequence of decisions with some sub-tasks being extendable and reusable on other datasets. our three ie systems, edie-r, edie-bilstm and edie-bert (a rule-based and two neural models), recognise mentions of stroke, stroke sub-types and other related findings such as tumours and small vessel disease in text. they also identify related temporal modifiers (recent or old) and location modifiers (deep or cortical). for downstream document classification by phenotype, the systems also mark findings and modifier entities for negation (negation detection). the contributions of our work are three-fold: approaches for ner in this domain, while not mutually exclusive, can broadly be categorised into the following: approaches leveraging lexicons, such as ctakes @xcite and radlex @xcite ; ontologies, such as metamap @xcite ; rule-based systems and pattern matching @xcite ; feature based machine learning such as conditional random fields (crfs) @xcite ; and more recently, deep learning @xcite . negation detection is commonly framed as identifying negation or speculation cues and their matching scopes in sentences @xcite . in the clinical domain, however, it is common for approaches to tackle negation assertion, namely, to verify whether each identified entity mention in the text is negated or affirmed @xcite , and in some cases, whether it is uncertain @xcite , conditionally present, hypothetically present or relating to some other patient @xcite . as with ner, some of the earlier negation detection approaches were rule-based. negex @xcite relies on regular expressions to detect negation patterns, and has been successfully applied to discharge summaries. @xcite and @xcite use negex for negation detection on extracted entities. context @xcite extends negex to capture hypothetical mentions, experiencer information and temporality, albeit with limited success on the latter. negbio @xcite , another rule-based negation and uncertainty detection system extended through dependency parsing information, has been shown to outperform negex. similarly, @xcite demonstrated that enhancing negex with stanford dependencies outperformed their bidirectional lstm (bilstm) negation model. bilstm approaches for negation detection have been successful, with @xcite reporting state of the art results for bio-scope @xcite abstracts. @xcite outperformed the latter using pretrained transformer models. despite the amount of progress on negation detection for clinical texts, however, there is still ample evidence that while fitting systems on a particular dataset is straightforward, generalising negation detection across datasets is challenging @xcite . this is true both for out-of-domain evaluation, such as training on a dataset of medical articles with evaluation on a dataset of clinical text @xcite , as well as for out-of-sample evaluation, where the training and test datasets are from the same domain but may have differences due to different annotation style, or distribution of named entities @xcite . for the in-domain but out-of-sample case, a domain fine-tuned rule based system seems to transfer well @xcite . for all other cases, transfer is challenging, both for rule-based and machine-learning models @xcite @xcite , with machine learning models benefiting from the addition of in-domain data to the training set. @xcite demonstrate that a pretrained bert model can improve the results of domain transfer for negation detection, but the results are still lower for outof-domain datasets than in-domain datasets if we compare to the results of earlier models in @xcite . in our work we concur with previous findings: our neural models do not generalise negation detection across datasets, despite both datasets comprising radiology reports with stroke findings, such as acute ischemic stroke (ais). ## system descriptions the rule-based system, edie-r, and the neural systems, edie-bilstm and edie-bert, all factor the document labelling task into the same three subtasks. namely, extracting finding mentions, extracting modifier mentions and using negation detection to assert whether the mentions imply their presence or absence in the brain imaging report. all three systems work at a sentence level granularity. ## in-sample evaluation in this section we describe the dataset we used to develop and fit our systems and report in-sample performance of our models on the unseen test set. ## out-of-sample evaluation we now test how well each of our systems generalises to unseen radiology reports from a different source, highlighting that our neural models do not generalise negation detection to this other dataset. by out-of-sample, we mean this dataset has similar labels but comes from a different distribution than the one the systems were developed on. we emphasise that we have not trained or adapted our models to this dataset. ## error analysis in this section we provide a fine-grained breakdown of the types of errors made by edie-r, edie-bilstm and edie-bert, arguing that not all error types are equally detrimental to the downstream task of document labelling. next, we investigate the variability in error types between our systems by exploiting bluebert's context-aware embeddings to group together training and evaluation examples that are similar. we then compare their labels to identify annotation artefacts that influence system errors. lastly, we investigate how our systems handle spelling errors. ## summary and conclusions access to annotated clinical text is a bottleneck to progress in clinical ie. while it is vital to strive for high quality gold datasets that are annotated from scratch with clear annotation guidelines, the reality of the situation is that many teams face data accessibility issues, strict time constraints and limited access to expert annotators, whose time is extremely valuable. given finite resources, it is therefore common to leverage output from previously developed systems to speed up annotation. through extensive error analysis, we exposed artefacts of annotations originating from experts correcting system output and recommend exploiting context-aware embedding models, such as bert, to improve recall and ameliorate annotation inconsistencies. we are not suggesting that standards of the annotation procedure should be overlooked, but we highlight that our approach may be of value for many teams that are not in a position to label a dataset from scratch: semi-automated expert data is extremely useful under low resource settings, and therefore having a way to guide such annotation processes is valuable. we also highlighted the pitfall of blindly trusting well-established metrics, both for ranking systems on subtasks that do not directly match the downstream task and, more importantly, in the case of generalisation, where metrics on in-sample data were misleading as to how well our neural models were capturing negation. we concur with the findings in @xcite , negation detection is straightforward to optimise for an in-domain sample of data, but generalisation to other datasets without any adaptation is still challenging. therefore, negation detection models should be tested across multiple datasets for generalisation. to conclude, our rule-based system outperforms our neural network models on the limited sized insample dataset and generalises to an unseen dataset of radiology reports. through a manual error analysis, we found that a large proportion of errors of our systems are due to ambiguities in annotation. given the fairly high performance of our models, we extrapolate that we have likely distilled most of the information available in our limited labelled dataset. in future work we plan to extend our annotations to a larger dataset to further assess generalisation.