Automated Scoring of Clinical Patient Notes: Findings From the K aggle Competition and Their Translation into Practice (2024)

Abstract: Scoring clinical patient notes (PNs) written by medical students is a necessary but resource-intensive task in medical education. This paper describes the organization and key lessons from a Kaggle competition on automated scoring of such notes. 1,471 teams took part in the competition and developed an extensive, publicly available code repository of varying solutions evaluated over the first public dataset for this task. The most successful approaches from this community effort are described and utilized in the development of a PN scoring system. We discuss the choice of models and system architecture with a view to operational use and scalability, and evaluate its performance on both the public Kaggle data (10 clinical cases, 43,985 PNs) and an extended internal dataset (178 clinical cases, 6,940 PNs). The results show that the system significantly outperforms a state-of-the-art existing tool for PN scoring and that task-adaptive pretraining using masked language modeling can be an effective approach even for small training samples.

## introduction a core practice in assessing the clinical skills of medical students is the use of objective structured clinical examinations (osces) -a type of exam, where test-takers interact with standardized patients, who are trained to portray a set of clinical symptoms. after examining the patients, the test-takers are asked to describe their findings in a clinical patient note (pn), similar to those found in electronic health records (see an example pn in appendix a). the pn serves as a documentation of the encounter and is used to assess examinee ability to gather information, record physical examinations, and interpret clinical data. osces are widely used in medical schools in various countries, with around 90% of us schools requiring their students to pass such exams @xcite . a major bottleneck for scaling osce assessment is the time, cost, and effort associated with the expert grading of large amounts of pns, especially given limited faculty time. for example, in the former united states medical licensing examination ® (usmle ® ) step 2 clinical skills exam (discontinued in 2020), more than 100 licensed physician raters were needed every year to grade ≈ 330,000 pns from ≈ 35,000 us and international test-takers @xcite . while there is interest among medical educators to address the above limitations using automated grading methods, the exploration of such methods has been slow and fragmented due to exam security concerns, which limit data sharing. this has resulted in small-scale, predominantly internal explorations of automated scoring, with no shared datasets or code to foster collaborative research. to address this gap, we organized a kaggle competition on clinical pn scoring foot_0 as a community effort to move this field forward. we then used the most successful approaches for the development of an interpretable and transparent pn scoring system. the contributions of this paper are as follows: the data used in this study originated from the united states medical licensing examination ® (usmle ® ) -a series of examinations used to support medical licensure decisions in the united states. until 2020, the usmle step 2 clinical skills examination was a part of the usmle step sequence and involved interactions with standardized patients portraying different clinical scenarios. the resulting pns were graded using rubrics specific to each clinical case, which contain a set of features -important concepts, which should appear in an appropriately documented pn (figure 1 ). for example, for a clinical case about a patient with anxiety, it may be important that the examinee discovers that the patient has insomnia, in which case insomnia would be listed as a rubric feature. pns that do not mention that symptom or some expression of it such as difficulty falling asleep would receive a lower rater score. key challenge for automated scoring is the variety of ways features are expressed (e.g., evaluation for coronary risk factors expressed as father with mi at age 50, or denies depressed mood expressed as (-) anhedonia). there are cases of ambiguous negation as in denies nausea, vomiting for the feature no nausea and or vomiting or temporal aspects such as recent uri for uri one week ago. to be operationally usable, a pn scoring system needs to provide interpretable evidence and be highly accurate. these requirements are crucial to ensure exam fairness and protect the health of the public. ## related work the vast majority of work on automated scoring has been done in the field of writing evaluation (see @xcite for an overview). studies on scoring clinical text include @xcite , who use a feature-based system for scoring short responses to clinical decision-making questions, @xcite the work most relevant to ours is the incite system @xcite , which was developed for operational scoring of pns from the usmle step 2 cs exam and which we use as a baseline. the system is a modular pipeline which outputs a binary score of "found" or "not found" for each rubric feature, utilizing custom-built lexicons and annotations. the first two modules perform direct and fuzzy matching between a feature or a lexicon variant and the pn text using a fixed or dynamic levenshtein ratio threshold. any features whose expressions are found using this method are removed from the pipeline to optimize running time. next, matching is performed against combinations of lexicon variants and annotations, which "often leads to an explosion of the number of eventual entries" @xcite as terms in the annotations are replaced with variants from the lexicons. to limit this search space, there is a cap of 10,000 randomly sampled combinations per feature. matching is then done using these phrases as sequences and as bag-of-words to cope with fragmented entries 2 . advantages of the incite system include its high performance, ability to be tuned for precision and recall by varying the thresholds, as well as its speed -it is capable of processing over 50,000 pns per day on a desktop computer. however, the rule-based nature of the system limits improvement from more training data, especially because more annotations would greatly increase the search space for supervised concept detection. ## task description and evaluation the task of developing an interpretable system for automated scoring of pns is one where features from the rubric are mapped to expressions from the pn. if an expression of the feature is identified in the pn then the feature is considered "found", else it is "not found". the more features are found, the higher the score for that pn. we perform two types of model evaluation, as described below. token-level evaluation: this type of evaluation answers the question "what phrase spans in the pn correspond to a given rubric feature?". this evaluation is identical to the one used in the kaggle competition and comparable to its leaderboard. for each instance, the system predicts a set of character spans that it considers to correspond to that feature, where a character span is a pair of indexes representing a range of characters within a text. these predicted spans are then compared to ground-truth spans from the annotation and scored as: a character is considered true positive if it is within both a ground-truth and a prediction; false negative if it is within a ground-truth but not a prediction; and false positive if it is within a prediction but not a ground truth. an overall f1 score is computed from the tps, fns, and fps aggregated across all instances 3 . ## kaggle competition: key lessons the kaggle competition on scoring clinical pns resulted in a total of 28,049 code entries from 1,471 participating teams. after the end of the competition, many teams posted their notebooks in the competition's code repository, making them publicly available. in this section, we analyze the top 15 publicly shared solutions 6 (the teams ranking from 1st to 11th place, and those that ranked #13, #14, #18, #19, and #20), as well as insights from other notebooks and key forum discussions. the final leaderboard rankings can be seen at https://tinyurl.com/p9mwfu8c and corresponding code contributions can be accessed at https://tinyurl.com/3h8p5a67 . results many of the top-performing teams reached a token-level f1 score of .89, with minor differences between solutions (e.g., #1 @xmath0.89456, #2 @xmath1.89432, and #3 @xmath2.89384), indicating that there are different, equally successful ways of addressing this task. this result also suggests potential ceiling effects arising from annotation inconsistencies such as not capturing every instance of a phrase that can be mapped to a feature 7 or not identifying the correct character span of a phrase (average inter-annotator agreement @xmath3.84). such inconsistencies resulting from human error are inevitable in spite of rigorous training and data cleaning efforts, further showcasing the need for improved reliability in scoring. @xcite as the backbone and performed taskadaptive pretraining @xcite ) by using masked language modeling (mlm) over the 6 detailed solution descriptions for first place: https: //tinyurl.com/2p8afa94 , second place: https:// tinyurl.com/yc77s4rk , and third place https:// tinyurl.com/3yf4u6hr . 7 the 2 nd place winner hypothesised that annotators were more likely to miss repeated annotations than first occurrence and noted that the use of recursive neural networks (rnns) could be useful to capture such sequence dependencies. unannotated portion of the data. one solution (#2) additionally pretrained on the squad 2.0 question answering dataset @xcite , drawing a parallel between the two tasks: the feature text in pn scoring corresponds to the question in the squad data, the patient history is the context, and the annotations are all answers to the question. ## models two key differences between real-world applications and the competition are that: i) newly developed cases do not come with large amounts of unannotated pns (which makes pseudo-labeling not suitable), and ii) the trade-off between performance gain and resource requirements such as speed and compute power is an important aspect of model selection (making the ensembling of a large number of models impractical). with these prerequisites in mind, the following approaches were trained and evaluated. ## error analysis for deberta + mlm, there were 990 errors (594 fns and 396 fps), distributed across all 178 clinical cases foot_5 . however, the errors were only distributed across 36% of the 1815 features. we hy-pothesize that this may be due to differences in annotation length for different features. indeed, the mean annotation length differs between the correct predictions and the errors: it is µ = 19.6 (@xmath4.7) for correct and µ = 13.2 (@xmath5.2) for the errors (mann-whitney @xmath6, @xmath7.001). this is somewhat counter-intuitive, as it suggests that the shorter features and shorter annotations are more difficult to detect. further content-specific analysis is needed to illuminate the potential causes for this phenomenon. annotation length affected incite inversely and to a much greater extent, where the annotations for the correct class (µ = 15.9, @xmath8.8) are on average twice as short as the errors (µ = 29.12, @xmath9.9), (@xmath10.5, @xmath11.0), potentially due to limitations from its window-based approach. spearman correlation between annotation length and correct/incorrect predictions further supports this finding: @xmath12.08 for deberta + mlm model and @xmath13.36 for in-cite. a likely explanation for this result is that incite's window-based approach is challenged by long phrases, while deberta's multi-head selfattention layers, where the encoder reads the entire sequence bidirectionally, enables it to cope well with these. in addition, since the objective of the neural models was to decide whether a given character belongs to a relevant phrase, the higher character count of longer phrases increases the available information for making a prediction. further analysis of the differences between correct and erroneous predictions did not reveal a specific pattern. this extended analysis is presented in appendix d together with examples of specific features. ## deployment a system based on the deberta + mlm model was deployed on the amazon web services (aws) platform. a graph depicting the aws architecture can be seen in appendix c. figure 2 provides a visualization of the system output. speed, resource efficiency, and scalability are ensured by the use of sagemaker and eliminating the need for human interference via event triggers: placing incoming data in an initial s3 bucket triggers a series of lambda functions, which initiate preprocessing, training, and scoring. ## discussion the results presented above showed that the best model, deberta + mlm, led to significant improvements over incite for a diverse set of 178 clinical cases (binary @xmath14.95 for deberta + mlm compared to .76 for incite), as well as the kaggle data (.96 vs .89). incite was significantly more challenged by lengthy phrases and the smaller number of training instances in the proprietary dataset. by contrast, as shown when evaluating in the limited annotation scenario, deberta + mlm continues to yield meaningful gains when trained on as few as 12 pns. these experiments add evidence that task-adaptive pretraining can be beneficial even for small training samples, making the approach applicable to a wide range of practical scenarios. while the incite system struggled to identify lengthy expressions (i.e., the annotations of the errors were twice as long as those of the correctly identified instances), the deberta + mlm model coped well with long sequences. this is likely due to the multi-head self attention layers of deberta, where the encoder reads the entire sequence in a bidirectional manner. in addition, since the task was to decide whether a given character belongs to a relevant phrase or not, the higher character count of longer phrases increases the available information for making a prediction. at the same time, incite's window-based approach limits the length of the text spans being considered at a time, making the capturing of long dependencies less feasible. the ability of the neural approaches to output the relevant pn phrases that correspond to each feature greatly improves the interpretability of the scoring process by making explicit the relationship between the assigned score and its supporting evidence. importantly, this is an improvement not only upon incite but also upon human scoring, as raters rarely have the time capacity to mark specific expressions. as each human rater scores hundreds of patient notes, it is not practically feasible for them to link specific phrases to rubric features for a large volume of data. in addition to improving interpretability, outputting the phrases enables applications of these tools that go beyond summative assessment. such information can serve to provide pointed learner feedback in osce assessment, especially in cases where students are still learning how to document their clinical findings in an appropriately detailed and organized manner. when discussing the development of this system, it is important to mention community competitions as an important source of innovation. the benefits from sharing data for such purposes are not limited to the organization or the data science community, but extend to improving transparency -a crucial prerequisite for building stakeholder trust. when applying these creative approaches to a real-world scenario, important considerations such as speed and scalability limit the use of large model ensembles that are typically widely used in competitions. other practical considerations include data availability for training (e.g., newly developed cases rarely have large numbers of pns associated with them) and the need for weak supervision. ## limitations and ethical considerations some of the limitations of this research relate to the small within-case sample size of the annotated notes (which is somewhat mitigated by the large number of clinical cases) and the fact that not all notes could be double-rated due to resource constraints. while the scoring method is interpretable in that it can be traced to specific phrases within the pns, the neural algorithms that identify the phrase boundaries are black-box models which needs to be carefully scrutinized for bias. in addition, it is still not fully apparent why certain features are easier to detect than others. future work includes development of scoring approaches for other segments from the pns such as the physical examination and data interpretation sections, deeper exploration of challenges related to specific features, experimentation with adversarial training, as well as further investigation of the operational use of the system. like many other products, automated scoring tools are socio-technical systems, whose impact is determined not solely by their technical capabilities but also by their use and output interpretation. misuse and incorrect interpretation of the model outputs can lead to unethical practices of serious consequence. in a summative setting, the models described here are intended to be used as hybrid systems, where borderline cases and the notes from examinees below the passing standard are always reviewed by human raters. in a formative setting, it is paramount to carefully examine the relationship between use of the system and learning outcomes as necessary validity evidence. another ethical consideration for this study is the transparency of the approaches when developing technology for highly consequential decisions. as @xcite write: "decisioning software with the potential to profoundly affect the career of a medical student should be examined closely. transparency of implementation is critical for such a high-stakes application". this is particularly important in automated scoring, where the scores only have value if all stakeholders (e.g., faculty, students, and residency selection programs, to name a few) trust that they are fair, reliable, and valid. having public datasets and code such as the ones shared in the kaggle competition go a long way in building trust by increasing transparency and accountability. as per the rules of the kaggle competition foot_6 , all code shared publicly is licensed under an open source initiative-approved license. it is important to note that the benefits of system transparency go hand-in-hand with risks associated with using that knowledge to "game" the system. these include reverse-engineering a strategy that would result in a higher score, as well as the occurrence of negative "washback" @xcite )over-focus on developing only those skills that are currently covered by the scoring tool. limiting these negative consequences while also building trust through transparency requires a delicate balance. in the case of this study, we foster transparency via organizing the competition, describing the main approaches, and evaluating our system on a dataset we made public. at the same time, we do not publish the code behind the system, limiting potential efforts reverse-engineer it or "game" it. the data used in the kaggle competition was released following strict adherence to ethical practice. it contains pns only from examinees who explicitly indicated that they agreed to have their data used in research as part of the official exam registration process; use of the anonymized data was considered "exempt" following an irb review. the pns were assigned a new set of ids that cannot be linked to operational ids used in scoring. none of the pns include names, affiliations or personal descriptions (note that the names and clinical data associated with the standardized patients do not belong to real people; they are part of carefully constructed clinical cases that aim to resemble realworld clinical practice). in addition, the dataset does not feature complete pns (only history portions are included), and no identifying information is given on which pns were written by an individual examinee. according to kaggle's terms and conditions, data can only be accessed for partici-pating in the competition. for purposes unrelated to the competition, access to the data is subject to an application process and a data use agreement as a way to ensure ethical use. a few important aspects remain to be examined before the system can be used in practice. this includes analyses related to differential functioning of the system for users with different backgrounds, e.g., ensuring that non-native english speakers are not disproportionally penalized due to differences in language proficiency, as well as continuous monitoring for issues such as drift or latency.