Large Language Models Provide Human-Level Medical Text Snippet Labeling (2024)

Abstract: This study evaluates the proficiency of Large Language Models (LLMs) in accurately labeling clinical document excerpts. Our focus is on the assignment of potential or confirmed diagnoses and medical procedures to snippets of medical text sourced from unstructured clinical patient records. We explore how the performance of LLMs compare against human annotators in classifying these excerpts. Employing a few-shot, chain-of-thought prompting approach with the MIMIC-III dataset, Med-PaLM 2 showcases annotation accuracy comparable to human annotators, achieving a notable precision rate of approximately 92% relative to the gold standard labels established by human experts.

## introduction advanced natural language processing (nlp) tools especially generative language models have recently made a big difference in healthcare @xcite @xcite @xcite . one key way nlp is used is to find important medical details, like diagnoses, within a patient's unstructured data. clinicians can quickly search for medical conditions in these documents, speeding up their understanding of a patient's medical history. in this work, we focus on identifying both potential and confirmed medical conditions throughout the various text snippets of information found in patients' medical records. particularly, we establish a mapping between a large comprehensive list of possible medical condition or procedures queries c and text snippets from clinical documents s. we visualize the core task in figure 4 in the appendix. when establishing a connection between a medical condition or procedure and a snippet of medical text, we do not expect the text to include "supporting" components that are directly related to the condition or procedure. instead, we anticipate that the labeler (here llm) recognizes significant medical patterns, medications, and symptoms that point to a potential diagnosis (i.e. medical condition) or medical procedure. a straightforward example of this is as follows: text snippet: "the patient has been taking metformin 2500mg a day since last year." possible llm condition/procedure labeling: ## methodology we provide in figure 1 the general framework of our proposed labeling pipeline. pre-filtering. the first step in the pipeline involves pre-filtering a comprehensive list using costeffective filtering strategies. this step aims to reduce the number of expensive calls to the llm and avoid quality label loss (see appendix i.1). there are several methods for implementing a pre-filtering step, such as embedding similarity, medical search engines, etc. we encourage researchers to explore other available and easy alternatives. in this work, we employed a keyword search algorithm. this technique expands the input queries (through query expansion) and looks for the matched text in the input document, which we regard as reference snippets. more details can be found in appendix b. text chunking. we broke down the patient's medical record into more manageable and informative text segments (i.e. medical snippets). we performed different chunking strategies (see appendix d), and settled with a hybrid method involving a sentence-based (3-4 sentences) chunking algorithm with a constraint of 10-70 word tokens (figure e ). alignment. at this stage, we matched the remaining medical conditions and procedures to the corresponding text snippet. in particular, we opted for fuzzy matching. this can be considered as a secondary pre-filtering step at the snippet level. in our work, since our pre-filtering step outputs a reference snippet per condition or procedure, we attempted to locate these snippets within the different text chunks we have produced. this way, the condition becomes associated with the chunked snippet. * llm labeling. in the final stage of our framework, we paired the text snippets and their corresponding medical conditions. these pairs are then sent to the llm using appropriate prompting strategies. the llm assesses the relevance of the text snippet and medical condition in each pair. if it determines a condition to be relevant, the condition label is included as one of the final labels for that snippet. ## experimental setup dataset and pre-processing. we used the publicly available de-identified dataset mimic-iii @xcite . it is a collection of deidentified medical records and notes of more than 40,000 critical care patients at a large tertiary care hospital. it contains over two million unstructured clinical documents from nurses, physicians, etc. in our work, we randomly sampled 1000 patients and fetched all of their corresponding clinical records. our pre-processing of the dataset was kept simplistic. we used simple regular expressions to identify formatting inconsistencies, such as extra spaces or tabs, in the clinical documents. we provide basic statistics in section f about the sampled subset from the mimic-iii dataset. human labeling workflow. the human labeling process was carried out in three separate rounds. in each round, a different group of medical expert raters was recruited to evaluate a distinct set of medical text snippets paired with a condition. overall, we had 14 different medical experts as human annotators: 3 experts on the first round, 5 on the second, and 6 on the third round of labeling. specifically, the raters were given a set of multiple-choice options ("relevant", "irrelevant", and "not sure") and were asked to answer the following question: "is the following text snippet relevant to the following medical condition/procedure?". the raters were given a random sample of snippets. in total, we collected 14,470 labeled snippet-condition pairs. ## results we provide details about the basic statistics on both human labeled data and the sampled data from mimic-iii in appendix f. agreement between human raters. before relying on human labels, it is essential to assess their reliability and validity, especially when there is no clear or accessible ground truth label. to do this, we start by plotting the response distribution of each rater at each labeling round. figure 2 exhibits significant variations within the different raters' responses. in round 1, for instance, two raters (raters 1 and 2) demonstrated a tendency to provide answers skewed towards the "irrelevant" category. in contrast, rater 3 maintained a balanced approach, assigning an equal number of responses to both the "irrelevant" and "relevant" categories. during the second round of the labeling process, raters 5, 6, and 8 exhibited a similar pattern of providing more "irrelevant" labels. in contrast, raters 4 and 7 produced more "relevant" responses. in the third round, we observe a similar distribution trend, which is predominantly characterized by a skew towards the "irrelevant" side. in figure 3 , we assess inter-rater reliability using cohen's kappa statistics @xcite and we provide in appendix h the agreement interpretations. we observe that the level of agreement between raters varies across different rounds. in round 1 of labeling, the agreement ranges from "fair" to "moderate," indicating a practical level of consensus. however, in rounds 2 and 3, substantial variations emerge. in round 2, raters 5 and 6 exhibit a stronger agreement compared to other raters. in the third round, we observe a notable agreement between raters 11 and 12 and a moderate agreement between raters 10 and 11. ## conclusion we proposed a framework for labeling clinical notes. our findings suggest that llms can produce high-quality medical data labels, which can serve as a valuable dataset for nlp tasks, such as information retrieval systems. these systems can help clinicians to be more efficient in their daily workflow by finding the key information faster and focus on pertinent facts within a clinical note.