A Visualization Approach for Rapid Labeling of Clinical Notes for Smoking Status Extraction (2021)

Abstract: Labeling is typically the most human-intensive step during the development of supervised learning models. In this paper, we propose a simple and easy-to-implement visualization approach that reduces cognitive load and increases the speed of text labeling. The approach is fine-tuned for task of extraction of patient smoking status from clinical notes. The proposed approach consists of the ordering of sentences that mention smoking, centering them at smoking tokens, and annotating to enhance informative parts of the text. Our experiments on clinical notes from the MIMIC-III clinical database demonstrate that our visualization approach enables human annotators to label sentences up to 3 times faster than with a baseline approach.

## introduction deep learning algorithms achieve state-of-the-art accuracy on a range of natural language processing tasks. however, to achieve high accuracy, deep learning algorithms typically require a lot of labeled data. in extremely error-sensitive applications, such as those in the medical domain, the trade-off between labeling effort and prediction accuracy is strongly skewed towards maximizing the accuracy. in such applications, data labeling arises as the most costly and human-intensive step during the development of deep learning models. in this paper, we focus on a scenario where the requirement is to label all available data because the goal is to maximize the accuracy using the available corpus of documents. in such a scenario, none of the labeling shortcuts developed in the machine learning community such as active learning are of much help on their own. our focus is on presenting textual information to human annotators in a way that minimizes their cognitive load, thus improving their focus, and maximizes their labeling speed, thus reducing the cost of labeling. our proposed visualization approach is fine-tuned to enable text labeling in the specific application where the objective is to extract information about smoking status of patients from their medical notes. smoking status of patients is critical information in many practical applications, ranging from recruiting participants in clinical trials to determining medical and life insurance premiums for prospective customers. smoking status extraction is a specific instance of information extraction problems. our visualization approach relies on several key observations about this particular type of problem. we first observed that smoking status could typically be extracted from sentences that contain one of the smoking keywords such as smoke, smoking, tobacco, nicotine. thus, our first step was to extract from the corpus only sentences containing one of those keywords. our second observation was that smoking status can typically be deduced from several words surrounding the keyword. thus, it might be possible to prune very long sentences to subsentences surrounding the keyword without loss of information. this observation allows reserving only a single line to display each relevant sentence. our third observation is that the space of possible smoking-related sentences occurring in clinical notes is relatively limited and that for any smokingrelated sentence there are likely very similar sentences in the corpus. we hypothesized that displaying similar sentences next to each other would allow human annotators to process the text much faster than if sentences are shown in random order. our fourth observation is that some common discriminative keywords reveal the smoking status, such as denies, quit, former, packs. we hypothesized that highlighting those keywords in the text could allow a human annotator to work faster. our final observation was that by training a predictive model on the currently available labels, even when the number of available labels is relatively small, would likely result in prediction accuracy that is significantly higher than a baseline that assigns labels randomly or based on the majority class labels. thus, providing labels obtained by the current prediction model would allow a human annotator to skip the correctly labeled sentences and only enter the labels for the incorrectly labeled ones. as the number of labels grows, the accuracy of the prediction model is expected to increase, and the effort to correct the labels would decrease, thus increasing the speed of labeling. the resulting visualization approach developed by exploiting the stated observations is illustrated in figure 1 . a panel at the top shows 7 randomly selected smoking-related sentences from our corpus. a panel at the bottom shows the same sentences displayed using our approach. the main features of our visualization approach are (1) sentence ordering, (2) sentence centering around the smoking keyword, (3) text annotation to emphasize discriminative keywords, and (4) displaying of the predicted labels. we are claiming, and our user study (described in section 4) confirms it, that the bottom panel makes it much easier and faster for a human annotator to label a large corpus of smoking related sentences for the smoking status of a patient. to produce the bottom panel in figure 1 , we had to decide (1) what are the smoking keywords, (2) what keywords are discriminative of the smoking status, (3) how to order the sentences, (4) how to provide predicted labels, (5) what to do during the cold start when no or very few sentences are labeled, and (6) how to implement the visualization approach. details about the proposed approach are provided in section 3. in section 2 we provide a brief overview of the related work. in section 4 we describe the experimental design, explain our user study, and provide experimental results that convincingly indicate the usefulness of the proposed approach. ## related work extracting smoking status of patients from electronic health records [ehr] has been crucial in clinical settings, and especially useful to health care providers to select the best care plan for patients at risk of smoking-related diseases. (rajendran and topaloglu, 2020) investigates the application of three deep learning models on ehr data to extract the smoking status of patients. authors compare their approach with traditional machine learning models on both binary (smoker vs non-smoker) and multi-class classification (current smoker vs. former smoker vs. non-smoker) tasks. @xcite extracts smoking status from three different sources such as narrative texts, patient-provided-information, and diagnosis codes. they conclude that narrative text proves to be the most useful source for smoking status extraction. @xcite develop rule-based algorithms to determine tobacco use by patients. @xcite further identify the cessation date and smoking intensity of patients. common for the aforementioned work on smoking status extraction is a need to label sentences and train an appropriate machine learning model. none of those papers discuss issues related to labeling nor attempt to reduce labeling costs. a common approach to annotate a large amount of data is through crowdsourcing @xcite @xcite . it has been used in variety of tasks such as image classification @xcite , bioinformatics @xcite , and text mining @xcite . although crowdsourcing is a cost-effective way to collect labeled data, it can still be costly when the required labeling effort is significant. moreover, when using imperfect annotators with varying levels of expertise, it is important to develop appropriate label integration approaches @xcite . beyond the crowdsourcing issues, one popular approach to reduce labeling costs is to apply active learning and label only the most informative examples @xcite . more recently, human-in-the-loop [hil] approaches were proposed to improve the efficiency of annotation @xcite . @xcite present a hil system for sound event detection, which directs the annotator's attention to the most promising regions of an audio clip for labeling. @xcite apply a similar technique on entity linking [el] task, in which the machine learning component makes recommendations about the most relevant entries in a knowledge base, and the annotator selects the correct candidate. the recommender improves itself based on the obtained feedback. in addition, @xcite present an interface for entity normalization annotation in which they measure the number of clicks in a tool to quantify the human effort. while many papers attempt to minimize labeling effort, a vast majority of them are measuring the effort by counting the number of labeled examples. there are very few papers @xcite ) that measure labeling effort in terms of elapsed time. the uniqueness of our work is in demonstrating that annotation speed can be significantly impacted by the way data is presented to an annotator. furthermore, our work is specific in its focus on an extreme labeling scenario where the task is to label the complete corpus in order to maximize the prediction accuracy. ## methodology problem definition: given a document corpus d representing clinical notes of patients from which a set of n unlabeled smoking-related sentences s 1 , s 2 , ..., s n is extracted, the goal is to ask human annotators to label all n sentences for smoking status. there are 4 types of labels: smoker (s), non-smoker (n), former smoker (f), and other (o), where others refer to sentences that do not reveal the smoking status. in this section, we describe a visualization approach that improves human annotation speed. the main components of the approach are sequence ordering, label prediction, and text visualization. the details are explained in the following subsections. ## experimental design we performed our experiments using 52,726 discharge notes from the mimic-iii dataset @xcite , which contains de-identified records of the beth israel deaconess medical center's intensive unit emergency department patients from 2001 to 2012. we defined smoking-related keywords by selecting keyword smoke and its selected word2vec nearest neighbors. we collected 26 unique keywords. using those keywords, we found 34,149 unique matching sentences. ## conclusion we presented a visualization approach that enables rapid annotation of sentences for smoking status of patients. our framework contains three main components: sentence ordering, sentence presentation, and sentence labeling by the prediction model. our approach does not depend on high-quality ml predictors to provide initial labels. the display has a significant impact on speeding up the annotation process. we evaluated our visualization approach with a user study on sentences from mimic-iii discharge summaries. we achieved close to 3× faster annotation rate compared to the baseline method that displayed sentences randomly in their original shape. as the annotation progressed, as the batches of unlabeled sentences became larger, and as the prediction models improved, the annotation speed kept increasing in our user experiments. the proposed visualization approach is applicable to similar text classification tasks. it is a topic of further research to study how to modify the presented approach to make it applicable to a large number of text annotation tasks in natural language processing.