Assertion Detection in Clinical Notes: Medical Language Models to the Rescue? (2021)

Abstract: In order to provide high-quality care, health professionals must efficiently identify the presence, possibility, or absence of symptoms, treatments and other relevant entities in free-text clinical notes. Such is the task of assertion detection - to identify the assertion class (present, possible, absent) of an entity based on textual cues in unstructured text. We evaluate state-of-the-art medical language models on the task and show that they outperform the baselines in all three classes. As transferability is especially important in the medical domain we further study how the best performing model behaves on unseen data from two other medical datasets. For this purpose we introduce a newly annotated set of 5,000 assertions for the publicly available MIMIC-III dataset. We conclude with an error analysis that reveals situations in which the models still go wrong and points towards future research directions.

## introduction the clinical information buried in narrative reports is difficult for humans to access for clinical, teaching, or research purposes @xcite . to provide high-quality patient care, health professionals need to have better and faster access to crucial information in a summarized and interpretable format. in this paper, we focus on english discharge summaries and the task of assertion detection, which is the classification of clinical information as demonstrated in figure 1 . given a piece of text, we need to identify two pieces of information -a medical entity and textual cues indicating the presence or absence of that entity. medical entity extraction has been studied extensively @xcite , we thus focus our work on the task of predicting the present / possible / absent class over a medical entity, addressing an important information need of health professionals. this setting is reflected in the dataset released by the 2010 i2b2 challenge assertions task @xcite , on which we base our main evaluation. clinical assertion detection is known to be a difficult task @xcite due to the free-text format of considered clinical notes. detecting possible assertions is particularly challenging, because they are often vaguely expressed, and they occur far less frequently than present and absent assertions. language models pre-trained on medical data have shown to create useful representations for a multitude of tasks in the domain @xcite . we apply them to our setup of assertion detection to evaluate whether they can increase performance (especially on the minority class) and where they still need improvement. we argue that clinical assertion detection models must be transferable to data that differs from the training data, e.g. due to different writing styles of health professionals from other clinics or from other medical fields. as existing datasets do not represent such diversity, we manually annotate 5,000 assertions in clinical notes from several fields in the publicly available mimic-iii dataset. we then use these annotated notes as an additional evaluation set to test the transferability of the best performing model. our contributions are summarized as follows: ## related work one of the earliest approaches to assertion detection is negex @xcite , where hand-crafted word patterns are used to extract the absent category of assertions in discharge summaries. in 2010, the i2b2 challenge assertions task (de bruijn et al., 2011a) was introduced, and an accompanying corpus was released. there is a variety of prior work focused on scope resolution for assertions, which differs from our setting in that it does not consider medical concepts but scopes of a certain assertion cue. representative current approaches for this task setup include a cnn-based (convolutional neural network) one by @xcite , reaching an f1 of 0.858 on the more challenging possible category. @xcite propose a lstm-based (long short-term memory) approach to detect only absent scopes. when "gold negation cues" are made available to the model and synthetic features are applied, an f1 of 0.926 is reached. negbert @xcite is another approach to detect absent scopes. as its name suggests, it is bert-based and reaches an f1 of 0.957 on bioscope abstracts. in contrast to these approaches we focus our work on entity-specific assertion detection, the results of which are of more practical help for supporting health professionals. bhatia et al. (2019 and chen (2019 ) apply a bidirectional lstm model with attention to the task and evaluate it on the i2b2 corpus. while these models reach f1-scores above 0.9 on the majority classes, the challenging possible class does not surpass 0.65. we show that medical language models outperform these scores especially regarding the minority class. furthermore, @xcite compared then state-of-the-art approaches for negation detection and found a lack of generalisation to arbitrary clinical text. we thus want to examine the transfer capabilities of recent language models to understand whether they can mitigate the phenomenon. ## methodology we want to understand the abilities of medical language models on the task of assertion detection. we hence fine-tune various (medical) language models on the i2b2 corpus described below. we further apply the best performing model to the bioscope dataset and our newly introduced mimic-iii assertion dataset without further fine-tuning to test their performance on unseen medical data. ## evaluation and discussion we start by evaluating the mentioned models on the i2b2 corpus. we use training and test data as defined by in the i2b2 challenge and compare our results to previous state-of-the-art approaches in ## conclusion and future work in this work, we present an evaluation on medical language models to detect assertions in clinical texts and experimental results which show that they outperform baseline approaches. we further provided a new corpus of assertion annotations on the mimic-iii dataset that will augment existing data collections and shows the model's capability to be transferred to other sources -if the text styles do not strongly differ. we suggest future work to investigate generalization to unseen data and expressions. we further encourage work on multi-task learning of entity extraction and assertions to support health professionals with systems that learn jointly in an end-to-end fashion.