Automatic Patient Note Assessment without Strong Supervision (2022)

Abstract: Training of physicians requires significant practice writing patient notes that document the patient’s medical and health information and physician diagnostic reasoning. Assessment and feedback of the patient note requires experienced faculty, consumes significant amounts of time and delays feedback to learners. Grading patient notes is thus a tedious and expensive process for humans that could be improved with the addition of natural language processing. However, the large manual effort required to create labeled datasets increases the challenge, particularly when test cases change. Therefore, traditional supervised NLP methods relying on labelled datasets are impractical in such a low-resource scenario. In our work, we proposed an unsupervised framework as a simple baseline and a weakly supervised method utilizing transfer learning for automatic assessment of patient notes under a low-resource scenario. Experiments on our self-collected datasets show that our weakly-supervised methods could provide reliable assessment for patient notes with accuracy of 0.92.

## introduction sponsored by the federation of state medical boards (fsmb) and the national board of medical examiners (nbme), the united states medical licensing examination (usmle) is a "three-step examination for medical licensure in the u.s. that assesses a physician's ability to apply knowledge, concepts, and principles, and to demonstrate fundamental patient-centered skills, that are important in health and disease and that constitute the basis of safe and effective patient care." 1 prior to 2020, the usmle step 2 exam included a second component, step 2 clinical skills, that used a simulated clinical examination with standardized patients to assess various clinical competencies, including the ability to document relevant patient 1 https://www.usmle.org/ history and differential diagnoses in a written patient note. after the discontinuation of the usmle step 2 clinical skills examination, medical schools may have more motivation to include a clinical skills examination that requires patient note writing after observing standardized patients @xcite . patient notes, as one type of health documents, document clinical findings and reflect examinees' ability to gather information and communicate their findings to patients and colleagues. therefore, in step 2 clinical skills, examinees' written patient notes were assessed manually by experienced physician raters. more than 30,000 examinees took this examination each year, resulting in more than 330,000 patient notes that were graded by more than 100 raters @xcite . the case-specific nature of the patient notes and large volume of exams make the human scoring process time-consuming and tedious. additionally, it is well-documented that human judgement in general is prone to bias and errors @xcite . training of qualified physician graders also requires assessment and feedback from medical experts, costing significant amounts of time. the manual effort required in grading medical examinations makes this a challenging problem to be addressed with the addition of nlp techniques. nlp has been applied to automatically process health documents, including assessing practical clinical content from patient notes @xcite . specifically, patient notes after simulated patient encounters are required to contain specific information, which is specified by items in a checklist created through faculty consensus. figure 1 shows an example of patient note and checklist items. the task of automatic patient note assessment aims to judge if the given checklist items are included in the patient notes by exactly same expressions or synonymous expressions. equivalents may be true synonyms, acceptable abbreviations, or answer alternatives 28 year old male with no pmhx who presents with ha for past 3 months. initially, he had headaches once every couple of weeks, but now they occur 1-2 times weekly. he describes pain in his left forehead and behind his l eye which radiates to the back of his neck. the headaches last 6-8 hours and interfere with his focus and concentration. they start 30 min after he wakes up in the morning. he has some associated nausea but is able to keep food down without vomiting. he has no auditory or visual aura, and has no tingling in his extremities. taking tylenol extra strength helps, as well as coffee and naps. he denies any tearing of his eye, denies cp, sob. he attributes the ha partly to his stressful job as an accountant. he normally has a chronic runny nose during this time of year from allergies, which is relieved by flonase normally, but he has not been using it lately. he denies cough, sore throat, nausea/ vomiting/abd pain. history: past medical history of allergies pain discribed as pounding unilateral headache severity 8/10 nausea photophobia aggravated by stress relieved by coffee resolves after work no other neurologic symptoms physical examination: no sinus tenderness to palpation diagnose: migraine headache tension headache seasonal allergies cluster headache depression deemed acceptable. notes are further complicated by indications of body side (right or left), frequent negations, strings of positive or negative findings, and nonstandard abbreviations used by learners. learners may use medical terms to describe findings (cholelithiasis) or lay terms (gall stones) and are typically judged the same if correct. ideally, the nlp model would directly identify the phrases in patient notes correlated with the given checklist items for the most granular grading analysis and feedback to learners in formative settings. therefore, we study automatic patient note assessment as two tasks: (i) directly judging if the given checklist items are entailed in the patient notes (a natural language inference task), and (ii) identifying the phrases in patient notes correlated with the given checklist items (a named entity recognition task). despite its importance, the task of automatic grading of patient notes remains under-explored with only a few works that have studied it @xcite . traditional supervised models have been utilized for this task @xcite , but are limited in scope because they rely on large scale annotated datasets. the significant manual effort associated with labeled dataset creation makes these methods difficult and impractical. besides, the traditional supervised models trained on data with prior clinical cases will be less effective for new clinical cases. another challenge lies in the inconsistency between the checklist item and the corresponding phrase(s) in the patient note owing to their being non-exact matches occurring as, for instance, synonyms or abbreviations. to overcome the limitations of previous works and the challenges of traditional supervised models for a low-resource scenario, we propose our method without strong supervision. first we propose a simple baseline unsupervised method with a pipeline framework which could be used in a zeroresource scenario. then we propose our weakly supervised method utilizing multi-level transfer learning, including data-level and task-level. data-level transfer learning refers to the ability of transferring knowledge learned from data in one domain to another domain. task-level transfer learning refers to the ability of transferring knowledge learned from one task to another task. a bert model @xcite pretrained on biomedical texts and a publicly available dataset foot_0 are used for data-level transfer learning. a key assumption is that judging if the checklist item is entailed in the patient note and identifying the corresponding phrases in the patient note are mutually related and thus we treat the automatic grading as a multi-task learning problem. with experiments on our self-collected datasets, we show that our weakly supervised method achieves a state-of-the-art performance. overall, the main contributions are as follows: ## related work being a research task that is currently underexplored, there are very few works studying automatic patient note assessment. the most closely related task that past works focus on is automatic short answer grading (asag) for scientific topics @xcite @xcite @xcite @xcite @xcite , which is different from complex domain-specific answer assessment (e.g. medical domain in our work). asag aims to grade free text that answers to a prompt categorically or numerically. produced by ets, c-rater @xcite is one example system for asag focusing on grading school-level examinations based on the presence or absence of required answers. text goes through a sequence of nlp modules for spelling correction, syntactic analysis, pronoun resolution, morphological analysis and synonym detection. generated canonical representations are then fed into a maximum entropy model for classification. @xcite also focused on a similar task of awarding content points for specific items for college biology essays. two text analytic platforms are utilized: spss text analysis 3.0 (spssta) relying on hand-crafted vocabulary and rules and summarization integrated development environment (side) using a classic bag-of-words representation and support vector machine. with the development of transformers, different transformers and large pre-trained models including bert and roberta have also been applied @xcite . while there are some works on asag for scientific topics, only three works studied automatic patient note assessment @xcite @xcite . inspired by the works on asag, the first two @xcite studied two systems: a feature based system including an n-gram feature extraction followed by a svm and a simple bert based neural network. the third @xcite followed previous works on asag and leveraged the pipeline framework. their system employs a sequence of modules including text normalization, lexicon-based matching, fuzzy matching and supervised concept detection all utilizing significant manual annotation and brute force exhaustive searches. inspired by these works, we also proposed a pipeline model without supervision, which could be used under zero-resource scenario. a key departure from the prior pipeline efforts is our nonreliance on task-specific manual annotation. however, the methods proposed in previous works are insufficient for the task of automatic patient note assessment. n-gram features and svms are limited for extracting linguistic and semantic features, especially for complex domain-specific text. bert based model requires a huge amount of annotated data for training, which is usually unavailable for our task, whereas, pipeline models have the obvious problem of error propagation. therefore, we also proposed an end-to-end model, which utilizes multi-level transfer learning to alleviate the dependence on annotated data. ## datasets we used two datasets in our study. the first is a selfcollected dataset on two clinical cases-headache and abdominal pain-collectively referred to as case-specific datasets. data from the same clinical case is referred to as in-domain data and data from a different clinical case is referred to as outof-domain data. collected data pertains to patient notes written by examinees, where each note covers three sections: (i) history, (ii) physical exam and (iii) differential diagnosis. patient note in each section should pertain to items from the same domain in the checklist, which includes 17 checklist items for the headache case and 19 for the abdominal pain case. the checklist item may contain fine-grained medical concepts (e.g., 'headache') and general descriptions (e.g., 'pain started two weeks ago'). the medical concepts included in the checklist items for different cases may be similar or vastly different, depending on the clinical condition being portrayed by the patient. as part of the grading process two expert raters, typically physician faculty members, are asked to judge if the checklist items are stated in the patient notes. interannotator agreements on both cases are reported in a second dataset is the usmle® step 2 clinical skills patient note @xcite , which contains a total of 43,985 patient note history portions from 10 clinical cases, where 2,840 patient notes (284 notes per case) were annotated with concepts from the exam scoring rubrics. at the time of the writing of this paper the dataset was used for a kaggle competition on automated scoring of clinical patient notes foot_1 , and only a subset of 100 patient notes from the annotated data were made available to the public (the remaining 184 notes per case were used as a test set for the competition). therefore, the study presented here has used a subset of 100 annotated patient notes per case, which was not large enough to be directly used for training or fine-tuning the model but was still considered as a diverse but related dataset. this dataset is referred to as generic dataset in the rest of the paper. ## baseline method in this section, we introduce our proposed unsupervised method used as a baseline model for written patient note assessment. this approach utilized amazon comprehend medical foot_2 for the purpose of medical entity extraction. amazon comprehend medical is an api that performs various types of text analysis for the medical domain, and is a service that is provided by amazon web services (aws). we made use of the medical entity detection feature of this api, that allowed extraction ## experiments 6.1 baselines due to the fact that related prior works did not release their codes and did not provide enough details for reproduction, we only test one baseline model for comparison with our proposed unsupervised and weakly supervised methods on the nli-related task. besides, we also use one baseline model for comparison on the ner-related task. for both the baselines, the experimental settings and the parameters are set to be the same as those in our weakly supervised method. in addition, the baseline models and our weakly supervised model are trained on the same data but with different la ## results the performances of the different models on the nli task of judging if the checklist item is entailed by the patient note are summarized in for the ner task of identifying the corresponding phrases, our weakly supervised method achieves an f1 score of 0.61 on the headache dataset and 0.62 on abdominal pain dataset, which is better than the performance of the baseline ner model as shown in ## analysis in this section, we provide some ablation studies to analyze the contribution of data and the different modules used in our weakly supervised method. ## practical impact our system is currently undergoing pilot testing by learners and faculty to assess the perceived impact on providing more immediate and automated feedback. the immediacy is important so that the case is fresh, and it will likely impact debriefing of simulation cases, potentially making debriefing more focused on areas of learner struggles identified in the notes. with our system grading all learners, automated feedback could be provided to the users and learners in time, which can be used to help their study of patient notes writing. ## conclusion and future work in this paper, we study the problem of automatic written medical examination assessment. the complexity and huge manual effort required make data resources for this task very limited. therefore, traditional nlp systems relying on large annotated corpora are impractical. with these factors in mind, we proposed a weakly supervised method. our weakly supervised method utilizes multi-level transfer learning including data-level transfer learning and task-level transfer learning to simultaneously judge if the checklist item is stated in the patient note and also identify spans of relevant phrases. experiments on two self-collected datasets show that our weakly supervised method is able to achieve the sota performance on both tasks. therefore, our weakly supervised method can correctly judge if the checklist item is stated in the given patient note and can also find the relevant phrases most of the time. our future work involves developing more effective transfer learning mechanisms to improve the performance on identifying the relevant phrases.