Consultation Checklists: Standardising the Human Evaluation of Medical Note Generation (2022)

Abstract: Evaluating automatically generated text is generally hard due to the inherently subjective nature of many aspects of the output quality. This difficulty is compounded in automatic consultation note generation by differing opinions between medical experts both about which patient statements should be included in generated notes and about their respective importance in arriving at a diagnosis. Previous real-world evaluations of note-generation systems saw substantial disagreement between expert evaluators. In this paper we propose a protocol that aims to increase objectivity by grounding evaluations in Consultation Checklists, which are created in a preliminary step and then used as a common point of reference during quality assessment. We observed good levels of inter-annotator agreement in a first evaluation study using the protocol; further, using Consultation Checklists produced in the study as reference for automatic metrics such as ROUGE or BERTScore improves their correlation with human judgements compared to using the original human note.

## introduction while electronic health record systems are a necessity in modern healthcare, they are burdening primary care clinicians with significant clerical work that distracts them from patient care and increases their dissatisfaction and burnout rates @xcite . since a significant part of the required documentation involves note writing, there has been a mounting interest in assisting clinicians by automatically generating consultation notes @xcite @xcite . a common approach involves passing the recording of the consultation through a speech-to-text system, then using a sequence-to-sequence model trained on parallel transcript and note datasets to automatically generate the note @xcite patient: hi there. well, i have this like really crazy headache that's been going on for days. clinician: ohh dear, ok. when, when did it exactly start, this headache? patient: eh, around three days ago, maybe. clinician: three days ago, ok. and whereabout in your head, is this pain? evaluating the output of such systems is challenging @xcite , as it is often the case in natural language generation (nlg). widely used automatic metrics, such as rouge @xcite and bleu @xcite , often fail to capture relevant aspects of generated text @xcite , and human evaluation, the best practice in nlg, is not only expensive and hard to reproduce @xcite but also highly subjective @xcite @xcite . even in the field of note generation where evaluators tend to be medical experts rather than crowd-sourced workers, inter-annotator agreement is low, as there is no explicit ground truth and the annotators have dif-fering opinions on the importance of each patient statement and whether it should be included in a consultation note @xcite . in this work, we propose an evaluation protocol that uses consultation checklists (cc), itemised reference of all facts discussed during doctorpatient consultations. we report good agreement between clinicians building ccs from the same consultation, which indicates good consistency of the reference creation process. since consultation checklists act as an approximation of the ground truth, they reduce evaluator subjectivity, which is reflected in the high inter-annotator agreement observed in our first study. we also show that correlation with human judgements increases when using ccs instead of the original clinician note as the reference for automatic evaluation metrics. ## related work there are a number of different approaches to quantitative human evaluation in nlg. rating or likert scales work well with few criteria, but lack explanatory power and fail to capture text quality @xcite . adding more criteria partly resolves this, but at the cost of the evaluation task becoming more difficult and subjective (van der @xcite . for example, @xcite use a 30 item rating scale and report that subjects found it too difficult to use. ranking methods, where evaluators are asked to rank the output of text generation systems along a specified criterion, are an alternative to rating scales. some studies have shown ranking to be more reliable and consistent (van der @xcite ; however, ranking methods do not scale well when comparing multiple models. extrinsic measures, such as measuring post-edit time of generated text @xcite provide a better estimate of how useful the generated text may be to the final user, but are often expensive and subjective @xcite . a common shortcoming among all methods described above is that none of them provide granular insights into the text generation systems' errors and how to address them. this is particularly important when evaluating automatically generated medical notes, where the factual accuracy and completeness of the generated note are critical, as well as identifying the situations where the system fails. as with all summarisation tasks, note generation has an element of content selection that is highly subjective. for this reason, evaluating system-generated notes against a single reference summary would penalise those notes that diverge from the reference in their content selection. one way to address this is by using multiple reference summaries foot_0 , as for example in the pyramid evaluation protocol @xcite . in a similar way to our proposed protocol, it splits the evaluation into two independent steps: extracting summarization content units (scus) from multiple references, then using these scus to evaluate generated text. another way to address the subjectivity of using single references comes from reference-less approaches that compare the generated text against the source text directly rather than against reference summaries. for example, @xcite once more split the evaluation in two steps: highlighting annotations in the source document, then comparing each generated summary to these annotations. in the domain of note generation, however, the format of the source documents (consultation transcripts) and the generated summaries (consultation notes) is different enough (see in the medical domain @xcite give evaluators the consultation audio recording instead of a reference, and ask them to identify the missing and incorrect items in a generated note, providing the required insight into how the generated text is wrong. however, even though the evaluators are experts (medical practitioners), agreement between them is very low. while this could be improved by better evaluator training, we believe that the evaluation task itself inherently inhibits agreement. there is no standard way of recognising or mapping facts from the audio recording of the consultation, which is the 'ground truth', to the consultation note -generated or otherwise. this makes it very hard to align multiple evaluators and get consistent results. ## proposed protocol we propose consultation checklists -a protocol using an expert-crafted ground truth approximation to evaluate the quality of system-generated medical notes with human raters. the evaluation protocol consists of a reference creation step followed by a notes evaluation step (see figure 2 ). ## checklist creation pilot based on our initial assumption that a consultation checklist should capture the salient points of a consultation in an itemised format, we ran a pilot study with 2 clinicians expert in ai annotation, a & b. the goal of the study was to define best practices for creating consultation checklists and to evaluate the consistency of their creation between clinicians. to investigate the agreement on checklist creation, we asked the two clinicians to produce consultation checklists for the same 10 mock consultations taken from papadopoulos @xcite (see figure 3 in the appendix for an example). in order to quantify the alignment between them, two of the authors checked whether the information of each item in clinician a's version was present in clinician b's version, and vice-versa. based on this analysis, we define the fact coverage for each consultation checklist as the number of matching facts divided by the total number of facts. we found that 93.7% of the items in clinician a's consultation checklists were also present in clinician b's, and 78.3% the other way around. this may indicate that the two clinicians agree on the basic facts to include but clinician b tended to add more details. as part of the pilot, we refined the following process for creating consultation checklists: 1. listen to the consultation audio and take notes on every patient statement. the pilot study also highlighted the cognitive effort of producing consultation checklists. on average, a consultation checklist for a 10 minutes consultation contains 56 items (excluding sub-headings) and it takes the clinician around 1 hour to complete. also, the clinicians found that they would be able to produce 4 consultation checklists in a row before requiring a break, and that the first one would be the quickest to make (in as little as 30 minutes) but the following ones would require progressively more time. ## consultation notes evaluation to test the utility of our protocol, we used a single expert clinician to create 20 consultation checklists from real-life patient consultations. we then generated history and examination notes for each of the consultations using a bart @xcite encoder-decoder transformer model foot_2 . the model was pre-trained on the cnn/dailymail dataset @xcite , and fine-tuned on a proprietary dataset of 10,000 (speech-to-text generated) consultation transcripts and human-written history and examination notes. finally, we hired three clinicians (two women and one man from ethnically diverse backgrounds) to evaluate the 20 generated notes by following the evaluation process defined in section 3. the training for this task involved: (i) task instructions (see appendix); (ii) two evaluation practice tasks; (iii) an alignment session between the three clinicians, where they investigated all cases of disagreement on the practice tasks and came to a joint decision. in addition, we wanted to quantify how often evaluators misjudge the generated note because of information that is omitted or misrepresented in consultation checklists. in order to do this, we asked each evaluator to listen to the audio of the consultation and review whether a generated note item was correct or incorrect. ## conclusion in this work we proposed a novel reference data structure called consultation checklists (cc), and a protocol that uses it for evaluating automatically generated consultation notes. our experiments show good inter-annotator agreement levels when parallel ccs are created from the same set of clinical consultations. we also report good agreement when the consultation checklist protocol is used by different clinicians to evaluate the same consultations. finally, we showed that expertly-crafted consultation checklists are better than human-written notes when used as references for automatic evaluation. while we have tested our protocol only on note generation for primary care consultations, we postulate that consultation checklists would apply to a number of medical domains, including secondary care and