Comparing Two Model Designs for Clinical Note Generation; Is an LLM a Useful Evaluator of Consistency? (2024)

Abstract: Following an interaction with a patient, physicians are responsible for the submission of clinical documentation, often organized as a SOAP note. A clinical note is not simply a summary of the conversation but requires the use of appropriate medical terminology. The relevant information can then be extracted and organized according to the structure of the SOAP note. In this paper we analyze two different approaches to generate the different sections of a SOAP note based on the audio recording of the conversation, and specifically examine them in terms of note consistency. The first approach generates the sections independently, while the second method generates them all together. In this work we make use of PEGASUS-X Transformer models and observe that both methods lead to similar ROUGE values (less than 1% difference) and have no difference in terms of the Factuality metric. We perform a human evaluation to measure aspects of consistency and demonstrate that LLMs like Llama2 can be used to perform the same tasks with roughly the same agreement as the human annotators. Between the Llama2 analysis and the human reviewers we observe a Cohen Kappa inter-rater reliability of 0.79, 1.00, and 0.32 for consistency of age, gender, and body part injury, respectively. With this we demonstrate the usefulness of leveraging an LLM to measure quality indicators that can be identified by humans but are not currently captured by automatic metrics. This allows scaling evaluation to larger data sets, and we find that clinical note consistency improves by generating each new section conditioned on the output of all previously generated sections.

## introduction as a part of a physician's workload, the electronic health record (ehr) has become an important tool for documenting patient information that is used for care and billing purposes. a soap note is a common framework for structuring a record of a doctor patient conversation (dopaco) that consists of subjective, objective, assessment, and plan sections. two common components inside of the subjective section are the "chief complaint" (cc) and "history of present illness" (hpi) sections. chief complaint is normally a brief one sentence statement about the reason for the patient's visit to the physician. for example: "patient presents for evaluation of left foot pain". this information is mentioned in the dopaco, but may also be provided as a part of the patient's admission documentation. history of present illness is usually a multisentence or paragraph description of relevant patient information that was discussed in the dopaco. for example, it may contain snippets such as: "patient is a 60-year-old male who reports left foot pain after having his foot run over by a tractor in 2021 . . . he had surgery on his foot in 2022 and is still experiencing pain . . . the patient reports a history of osteoporosis." the assessment section is normally a brief description or list of the doctor's assessment, e.g. "assessment: left foot pain" that may be mentioned during the dopaco or directly entered or dictated into an ehr system following the encounter. the plan section is a description of a path forward and commonly has a more narrative style such as: "plan: i have personally reviewed the findings with the patient today. he is scheduled for a left total knee arthroplasty soon. i anticipate that this procedure will also help with his left foot pain. at this time, we will hold off on a new orthotic prescription. he will follow up with me at the end of february." for our experiment we combine assessment and plan to be considered as a single section. in this work we present clinical notes that contain the chief complaint (cc), history of present illness (hpi), and assessment & plan (a&p) sections. we omit the objective section because at the time of writing, the objective section commonly contains information from a physical examination that has not been directly verbalized during the dopacos that are used in our dataset. we leave to future work the exploration of generating the objective section of a soap note using a dopaco and incorporating additional information that was not verbalized during the encounter. in order to ease the burden on a physician and accelerate workflows, recent research in natural language processing (nlp) techniques are being explored to automatically generate soap notes using variants of the transformer architecture @xcite . a common approach to the automatic generation of a soap note is to use an automatic speech recognition (asr) system to create a transcript of the dopaco, based upon an audio recording of the encounter. in this work, we present a comparison of two designs for generating a soap note. from the pretrained pegasus-x model @xcite , we train a single fine-tuned model (genmod) to produce a clinical note, as well as 3 individual fine-tuned models (specmod) to each produce a single section. in order to compare these two note generation designs, rouge @xcite is a common automatic metric for measuring the performance of summarization models. however, it is not always a reliable proxy for human preference, and a model with a lower rouge score may be preferred by humans @xcite . in particular, for comparing the two model approaches we expect that the spec-mod design would be more likely than genmod to have conflicting content between sections, since in specmod the output of a section such as a&p is not conditioned on the output text of any other section, e.g. hpi. for example, if the dopaco does not make clear reference to the gender of the patient, we expect that specmod would be more likely than genmod to refer to the patient as female in hpi and mistakenly refer to the patient as a male in a&p. in this work we present an approach to use the llama2 llm @xcite as an additional measure of model quality for criteria not clearly captured by existing automatic metrics. ## related work clinical note generation can be viewed as a summarization problem, since it can involve the use of a dopaco as input to summarize the content into a document that uses the appropriate clinical terminology and style. recent discoveries in deep learning based nlp have enabled advancements in the creation of clinical notes from dopacos @xcite @xcite . @xcite investigate the use of a fine-tuned bart model @xcite for generation of the hpi section of a clinical note. similarly, @xcite generate a clinical note that contains an hpi, a&p, and physical examination section, and train a separate model for each section. @xcite seeks to improve the consistency of soap notes through the integration of section tokens and section-specific cross-attention parameters to encoder-decoder models. this approach uses a single bart model with a modified cross-attention mechanism to generate a soap note based on extracted segments of the dopaco, and produces a single section at a time based upon the special token pre-pended to the input conversation. in order to evaluate the quality of these generated clinical notes, it can be difficult to find a reliable proxy for human evaluation. with the advent of chatgpt/gpt-4 (openai, 2023) and llama2 @xcite , the use of llms as a proxy for human evaluation is a popular subject of recent research @xcite @xcite . @xcite found llm based evaluation results to be consistent with human evaluation across several nlp tasks in terms of cohesiveness, among other criteria. @xcite finds that gpt-4 has the ability to match human preferences when comparing two different llm generated answers to a question. @xcite presents a framework called g-eval which uses chain-of-thought prompting and form-filling to evaluate text outputs for coherence. the llm prometheus @xcite is a fine-tuned llama2-13b llm designed to act as an evaluator llm that is aligned with human preferences. @xcite investigate the use of gpt-4 to evaluate medical notes in terms of factuality. to the best of our knowledge, this work is the first to evaluate the performance of an llm in reviewing clinical notes for specific criteria of consistency. aci-bench @xcite provides results from comparison of full-note vs division-based note generation techniques, which corresponds to our genmod and specmod designs, respectively. their work finds that specmod results in higher performance then genmod. however, aci-bench uses a smaller dataset of 207 encounters with an average conversation length of 1,300 tokens. using our larger proprietary dataset with more average tokens per conversation, we seek to understand whether the comparison of genmod to specmod is sensitive to the size of the dataset as well as the dimension upon which the two models are measured (i.e. in terms of full note consistency). ## data the dopacos in the dataset for this experiment come from an asynchronous scribing configuration, similar to that which is described in detail in @xcite . the physician is aware that the audio is being provided to a human scribe after the encounter, and as a result, the audio may contain dictated portions where the doctor directly instructs the scribe regarding information that should be included in the note. the physicians in the dataset represent a mix of 20 different specialties, identified using the national provider identification (npi) registry ( https://npiregistry.cms.hhs.gov ) we create three dataset splits for the experiment: training, validation, and test. information about these splits can be found in the mediqa @xcite task uses the aci-bench @xcite dataset for clinical note summarization, which contains 207 encounters of average length 1,300 tokens. this precludes it from being substituted as a training dataset for this experiment due to the small dataset size and shorter average token length of the conversations. further, 112 of the 207 aci-bench encounters come from the ambient clinical intelligence setting, which does not match the asynchronous scribing domain that our experiment addresses. ## conclusion the paper presents an empirical study on the quality of two distinct pegasus-x based designs for generating a clinical note. the genmod design reduces the rouge by less than 1% compared to specmod, and has no difference in factuality. based on human review, our findings indicate that genmod improves the measured age, gender, and body part consistency by 4.5%, 2.5%, and 8.5%, respectively, when compared to specmod. we highlight the difficulties of measuring the consistency of topics such as note coherence, as the criteria can be difficult to clearly specify in a way that is reliably understood by humans as well as llms. lastly, we observe a cohen kappa inter-rater reliability of 0.79, 1.00, and 0.32 for consistency of age, gender, and body part injury, which shows that llama2 can be a valuable proxy for human evaluation in specific evaluation criteria. this finding supports the usage of the llama2 llm as an evaluator on large datasets that would be impractical for comprehensive human review. this proves a promising area for future work to explore development of a robust evaluation suite that utilizes an llm for clinical note analysis on an expanded variety of criteria.