The USMLE ® Step 2 Clinical Skills Patient Note Corpus (2022)

Abstract: This paper presents a corpus of 43,985 clinical patient notes (PNs) written by 35,156 examinees during the high-stakes USMLE® Step 2 Clinical Skills examination. In this exam, examinees interact with standardized patients - people trained to portray simulated scenarios called clinical cases. For each encounter, an examinee writes a PN, which is then scored by physician raters using a rubric of clinical concepts, expressions of which should be present in the PN. The corpus features PNs from 10 clinical cases, as well as the clinical concepts from the case rubrics. A subset of 2,840 PNs were annotated by 10 physician experts such that all 143 concepts from the case rubrics (e.g., shortness of breath) were mapped to 34,660 PN phrases (e.g., dyspnea, difficulty breathing). The corpus is available via a data sharing agreement with NBME and can be requested at https://www.nbme.org/services/data-sharing .

## introduction large clinical text corpora are both one of the most needed and one of the least available resources in biomedical nlp, largely due to patient confidentiality considerations and expert annotation cost. this has been identified as a main reason for lagging progress in biomedical nlp compared to the general nlp domain @xcite , and is evidenced by the fact that mimic-iii @xcite is the only freely available large corpus of clinical notes to date (section 2). as a result, biomedical nlp is heavily reliant on corpora of pubmed scientific abstracts, foot_0 whose academic language is in stark contrast to the often ungrammatical and telegraphic text constructions found in clinical notes. a known example of how the lack of shared clinical note corpora affects application development is the task of nlp-assisted scoring of clinical patient notes (pns) written during exams. in medical education, students are often assessed through encounters with standardized patients -people trained to portray simulated scenarios called clinical cases. for each such encounter, the student is expected to perform a history and physical examination, determine differential diagnoses, and then document their findings in a pn. this assessment format is ubiquitous in medical education due to the important clinical skills it measures (van der @xcite , however, there is a significant cost associated with the manual scoring of the produced pns by expert physician raters, as well as potential for human error and bias @xcite . there has been fragmented effort by individual institutions to train in-house nlp systems for clinical text scoring, with no fully transparent evaluation on public data @xcite @xcite . this has raised questions from a key stakeholder -the medical student community -about potential algorithmic bias and its implications for fairness @xcite . overall, the lack of shared data (here, mainly for exam security reasons) has slowed down innovation and limited public support, despite nlp's potential to alleviate financial burden and improve reliability. the goal of this paper is to advance pn automated scoring specifically, and biomedical nlp in general, through the development and public release of a large corpus of examinee-written pns. the corpus consists of 43,985 pn history portions from 10 clinical cases, where 2,840 pns (35k phrases) were annotated with concepts from the exam scoring rubrics (section 3). the main, but not sole, application for this data is the development of interpretable, transparent, and cost-effective systems for clinical text scoring, thus improving educational assessment in the field of medicine. the two key contributions of this paper are the development of a large corpus of examinee-written pns, made available for research purposes, and the expert annotation of a subset of 2,840 pns to advance automated scoring of clinical pns. ## related datasets large corpora of clinical patient notes (e.g. > 2k) are scarcely available as shared resources. as noted in two overview articles by @xcite and @xcite , such large corpora include clef (565k notes), which is "currently restricted" , awaiting "a governance framework in which it can be made more widely available" @xcite ; and a corpus related to the trec shared task, where "the university of pittsburgh distributes the records only to track participants" @xcite . among the larger ehr databases, the eicu database specifically excludes clinical note text: "to minimize risk of including phi" (protected health information) foot_1 . these restrictions make mimic-iii @xcite the only freely available large corpus of clinical notes to date. as a result of patient confidentiality considerations, the use of patient notes describing fictional patients is not new in the field of biomedical nlp. this type of data has shown promise in several shared tasks: the ntcir10 foot_2 ntcir11 foot_3 , ntcir12 foot_4 , and ntcir16 foot_5 mednlp tasks use de-scriptions of fictional patients written in japanese. as reported in the ntcir10 task overview paper, "we asked physicians to write down fictional medical reports of imaginary patients (...) we offered 50 collected medical reports for this task, which include 3,365 sentences in all: about 40,000 words" @xcite . in addition to its small size, limitations of this dataset include the lack of clarity around the procedure the physicians followed to create these patient notes. nevertheless, given the lack of publicly available data from real patients, this dataset contributed to the field by enabling the evaluation of tasks such as patient anonymization and detection of complaint and diagnosis. the next section describes the high-stakes clinical examination context in which the patient notes from our corpus were written. ## context the united states medical licensing examination ® (usmle ® ) is a series of examinations to support medical licensure decisions in the united states that is developed by the national board of medical examiners (nbme ® ) and federation of state medical boards (fsmb). until 2020, one of the exams was the usmle step 2 clinical skills examination, which used standardized patients to assess examinee ability to gather information, perform physical examinations, and interpret data, as documented in the pns examinees completed after each encounter (an example of a full pn is presented in appendix a). annually, the exam resulted in more than 330,000 pns graded by more than 100 raters. the pns are scored by licensed physicians using case-specific rubrics that were developed by physireal-mednlp/ cians on a test development committee. the rubrics outline each case's important concepts (henceforth called features) which should appear in an appropriately documented pn (figure 1 , feature column). for example, for a clinical case about a patient with constant headaches, it may be important that the examinee asks questions leading to the information that the patient has photophobia. in a case like this, photophobia would be listed as one of the rubric features, and pns that do not mention that specific symptom (or some expression of it such as sensitive to light ) will receive a lower score. a main challenge for developing an interpretable system that can identify expressions of the features in the pns is the variety of ways in which features are expressed, with examples such as loss of interest in activities expressed as no longer plays tennis, or shortness of breath expressed as dyspnea. there is often a need to map concepts by combining multiple text segments, or resolve ambiguous negation as in no cold intolerance, hair loss, palpitations or tremor corresponding to the feature lack of other thyroid symptoms. in addition, automated scoring systems should employ a dynamic threshold to determine whether a given feature has been found in a pn, i.e., whether the f1 score for a given identified phrase is high enough for the phrase to be considered a match @xcite . finally, to be comparable to human rater performance and thus operationally usable, such systems need to be highly accurate. this requirement is crucial because of the high societal cost of passing an examinee with insufficient knowledge, and the high personal cost of failing an examinee who should have passed. as will be seen in section 5, the average inter-rater agreement on whether a feature is mentioned in the corpus is @xmath0.97. ## data the dataset consists of the history portions foot_6 of 43,985 pns from ten clinical cases (average # per @xmath1,398; @xmath2, @xmath3,936) and the corresponding features for each case. the cases cover diverse clinical areas: women's health (2), gastrointestinal (2), neurological (1), psychiatric (2), and cardiovascular (3); as well as patients from diverse age groups: < 18 (2), 18-44 (6), 45-64 (1), 65+ (1). the number of tokens in the dataset is 5,958,464, with a type-token ratio of 0.022. the average length of each history portion is 135.47 tokens (@xmath4.27), and average number of history portion features per case is 14.3 (3.34). data were collected between 2017 and 2020 from 35,156 us or international medical students and graduates who took the exam under standardized conditions in one of five testing locations in the us. each examinee-patient encounter resulted in a unique pn. the dataset includes pns only from examinees who, during registration, indicated that they agreed to have their data used in research. all pns were assigned new ids that cannot be linked to operational ids used in scoring. the pns do not include identifying information such as name, affiliation, or descriptions of personal experiences. finally, the dataset features only the history portions of the pns as opposed to complete pns, and no information is given on which pns belong to an individual examinee. this limits the inferences that can be made about the performance of individual examinees, while allowing the use of this data for advancing automated scoring and biomedical nlp research. ## annotation a total of 2,840 pns (284 per case) were annotated by 10 experienced us medical practitioners -nine with a medical doctorate degree and one with a degree in nursing. the annotators were divided in five pairs of two, such that each pair would contain one experienced "senior" annotator. the annotation was performed using brat. 8 the annotators were instructed to first read the entire pn and then 1) identify all phrases that are expressions of a feature and link them to their corresponding feature (figure 1 ), 2) mark fragmented annotations by excluding the text that is not relevant to the feature, and 3) mark each feature as a separate annotation (see detailed annotation guidelines in appendix b). for example, if the feature was "no blood in stool", only the underlined text of the following excerpt was annotated: "no blood or mucus in stool". unlike other features, gender and age were annotated only once for the first mention, with subsequent relevant phrases such as "she" or "his" not marked. for each case, 284 notes were randomly selected for annotation and each annotator pair annotated notes from two cases over a period of six weeks. two of the notes were annotated jointly as part of an initial discussion on the specifics of each clinical case. during this discussion, the annotators would develop consistent case-specific understanding of the requirements for phrases to be considered a match (e.g., for the feature visual hallucinations, is the mention of hallucinations sufficient or does it need to be specified as visual?). next, both annotators would annotate the same set of 5 notes independently and have a follow-up meeting to discuss potential discrepancies. after these were resolved, each annotator would proceed to independent work, where 29 notes per case (18% of the data) were double-rated foot_8 and used to compute inter-annotator agreement and the remaining 124 notes per annotator per case were single-rated. the annotators would receive a new set of notes weekly, to ensure an even work pace and mitigate fatigue. the produced data were cleaned by fixing instances of wrong feature attribution (81) and correcting: leading and trailing spaces (167), punctuation (533), extra characters (115), and missing characters (e.g., as in "ot flashes") (64). f1 agreement scores were computed based on character position overlap, with a substantial agreement across all cases of @xmath5.84 (@xmath6.075); jaccard distance of 86.55% (9.89); and cohen's κ of 0.89 (0.057) (see individual case agreement scores in appendix c). finally, the annotators had an even higher agreement (binary f1) on whether an expression of a given feature was found in a pn or not (mean @xmath7.97 (0.014)). the final corpus includes 43,985 pns, of which 2,840 (284 per case) were annotated and contains 34,660 annotated phrases linked to 143 features. ## baselines to quantify the number of phrases from the gold standard that can be matched using simple heuristics and a small amount of annotated data, we compute three baselines. first, we divide the annotated portion of the data into ten folds. then, we apply 10-fold cross-validation such that we take the phrases from one fold and see how many of them can be found in the remaining nine folds foot_9 using three approaches: i) direct match between a string from the "training" fold and those from the nine "test" folds, ii) fuzzy match with a window of two characters, iii) fuzzy match with a window of two characters and synonyms from wordnet @xcite and the unified medical language system (umls) @xcite . the evaluation metric is micro-averaged f1 of character span overlap between the predicted and gold-standard phrases, where a character span is a pair of indexes representing a range of characters within a text. for each instance, there is a collection of ground-truth spans (the phrases identified by the annotators) and a collection of predicted spans (the phrases identified by an automated system, in this case one of the three baselines). each character within that span is identified as a true positive (tp) if it is within both a ground-truth and a prediction, a false negative (fn) if it is within a ground-truth but not a prediction, and a false positive (fp) if it is within a prediction but not a ground truth. the overall f1 score is computed from the tps, fns, and fps aggregated across all instances. as shown in figure 2 , the fuzzy + synonyms approach outperformed exact and fuzzy match with a mean f1 of .64 (.074), compared to .53 (.073), and .62 (.075). this result compares to an average inter-annotator agreement of .84 (.075) for character location overlap between phrases, showing a need for considerable improvement to match human performance. this gap varies between cases, with some having more than 20 points difference in f1 (e.g., case 204). it is also seen that the variance in responses for certain cases (e.g., 201) is easier to capture computationally compared to others (e.g., 203). finally, the results show that including a list of synonyms in fuzzy + synonyms does not lead to significant improvement, with the task requiring more sophisticated semantic processing. a binary f1 score of whether a given feature was expressed in a pn (1 if found, 0 otherwise) reveals a very strong agreement between the annotators (.97 (.014)) and a significantly worse performance for the best baseline (.86 (.048) for fuzzy + synonyms match). therefore, to be comparable to human performance and thus operationally usable, automated approaches need to show a significant improvement over the baseline results presented here. ## discussion the goal of this paper was to advance pn scoring and biomedical nlp through the development and annotation of a large pn corpus. for pn scoring, this data can aid the development and evaluation of interpretable systems that identify feature expressions rather than black-box modeling of rater scores. having a shared dataset can guarantee transparency, informing stakeholders on various aspects of system performance. it is also conceivable that the semantic mapping solutions enabled by this data could scale to scoring other constructed-response items, such as short-answer questions assessing clinical knowledge. as noted in the introduction, real clinical notes are scarcely available, which creates a bottleneck in the development of biomedical nlp. this corpus can help bridge this gap, since the pns in it share many characteristics with real clinical notes -medical jargon, typos, abbreviations, and telegraphic style, among others. moreover, having thousands of pns written by different examinees that correspond to the same clinical case allows the development of robust nlp models exposed to a large-scale, real-life variation of clinical language. such models would be trained to recognise the various ways in which, say, thyroid symptoms are described in clinical pns, rather than their expressions in scientific abstracts. beyond that, the corpus is relevant to machine reading comprehension and automated question answering, where the features are treated as yes/no questions ("is photophobia present in this document"), and the identified phrases are supporting information. the strengths of this data for some applications represent limitations for others. for example, all pns in the corpus pertain to a set of ten cases, which excludes the possibility of using this data for patient cohort identification or phenotyping, typically performed with electronic health record (ehr) data. in addition, the exam is a simulation of patient visits. nevertheless, because of its highstakes nature, the cases were treated as real. unlike ehr data, this corpus poses no risks for real patients, which is why the final data is less "sanitized" compared to deidentified ehr records; in addition, the cases were created by a team of licensed physicians ensuring that they are accurate representations of cases found in clinical practice. including anonymized, partial data (history portions only) prevents risks for examinee identification or inferences about individual performance. responsible use of the data for research purposes is further ensured by its distribution via a data use agreement. this is done following application to nbme's data sharing and collaboration program at https://www.nbme.org/services/data-sharing . it is our hope that the public release of this data will spur the development of interpretable and transparent solutions for pn scoring and related tasks, improving technology-assisted educational assessment in the field of medicine.