Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation (2022)

Abstract: In recent years, machine learning models have rapidly become better at generating clinical consultation notes; yet, there is little work on how to properly evaluate the generated consultation notes to understand the impact they may have on both the clinician using them and the patient’s clinical safety. To address this we present an extensive human evaluation study of consultation notes where 5 clinicians (i) listen to 57 mock consultations, (ii) write their own notes, (iii) post-edit a number of automatically generated notes, and (iv) extract all the errors, both quantitative and qualitative. We then carry out a correlation study with 18 automatic quality metrics and the human judgements. We find that a simple, character-based Levenshtein distance metric performs on par if not better than common model-based metrics like BertScore. All our findings and annotations are open-sourced.

## introduction modern electronic health records (ehr) systems require clinicians to keep a thorough record of every patient interaction and management decision. while this creates valuable data that may lead to better health decisions, it also significantly increases the burden on the clinicians, with studies showing this is a major contributor to burnout @xcite . in most primary healthcare practices, the universal record of a clinician-patient interaction is the soap (subjective, objective, assessment, plan) note, which captures the patient's history, and the clinician's observations, diagnosis, and management plan @xcite . at the end of a consultation, the clinician is required to write up a soap note of the encounter. with the exception of the clinician's internal observations on how the patient looks and feels, most of the soap note is verbalised and could be automatically constructed from the transcript of the consultation. a number of recent studies @xcite @xcite propose using summarisation systems to automatically generate consultation notes from the verbatim transcript of the consultation-a task henceforth referred to as note generation. yet, there is very limited work on how to evaluate a note generation system so that it may be safely used in the clinical setting. where evaluations are present, they are most often carried out with automatic metrics; while quick and cheap, these metrics were devised for general purpose summarisation or machine translation, and it is unclear whether they work just as well on this new task. in the field of automatic summarisation and natural language generation (nlg) in general, human evaluation is the gold standard protocol. even in cases where the cost of using human evaluation is prohibitive, it is essential to establish the ground truth scores which automatic metrics should aim for. our contributions are: (i) a large-scale human evaluation performed by 5 clinicians on a set of 285 consultation notes, (ii) a thorough analysis of the clinician annotations, and (iii) a correlation study with 18 automatic metrics, discussing limitations and identifying the most suitable metrics to this task. we release all annotations, human judgements, and metric scores. of the 18 studies on the subject that we could identify, 13 present an automatic evaluation (typically based on rouge and sometimes on medical entity linking) and 12 carry out a small-scale intrinsic human evaluation. in particular, moen et al. (2016a) employ three domain experts to review 40 generated notes with likert scales along 30 criteria (including 'long-term diagnosis', 'reason for admission', 'assessment'), but report that the subjects found the 30 item scale too difficult and detailed to assess. macavaney et al. (2019) use one domain expert to review 100 notes and report likert scale values for 'readability', 'accuracy', and 'completeness'. moramarco et al. (2021) employ three clinicians and compare the times to post-edit generated notes with those of writing them from scratch, reporting that, while faster, post-editing may be more cognitively intensive than writing. outside of the medical domain, our work is comparable to @xcite , who run an automatic metrics correlation study for news article summaries for the cnn/dailymail dataset @xcite . they also release code foot_0 for evaluating text with a suite of common metrics, some of which we include in our own list of metrics to evaluate. ## dataset and models our evaluation study is based on a dataset of 57 pairs of mock consultation transcripts and summary notes @xcite . 3 the data was produced by enacting consultations using clinical case cards. the clinicians that conducted the mock consultations also wrote the corresponding soap note. the consultations span common topics within primary healthcare and are about 10 minutes long. to mimic a live clinical environment, the audio of the consultations was transcribed with google speech-to-text engine foot_2 . these transcripts form the input to the note generation models. the aim is to generate the subjective part of a soap note. the base models are large bart architectures pretrained on the cnn/dailymail dataset foot_3 . since our focus is on evaluation, the aim was to obtain models which would produce different outputs to cover a wider range of errors. the differences between the models included: fine-tuning on different sized datasets; using pre-processing techniques such as filtering the transcripts for relevant sentences; and using post-processing techniques such as filtering the generated notes for irrelevant sentences. ## human evaluation setup under the supervision of one of the authors (a clinician expert in ai development henceforth referred to as the lead clinician) we design the following evaluation tasks: following the tasks described above, each clinician evaluated the entire dataset of 57 mock consultations. each consultation included 5 notes to evaluate, 4 of which were sampled from our 10 models and 1 was written by the consulting doctor (human_note). we shuffled these for every consultation and-to avoid biases-did not specify that one of the notes was not synthetic. the evaluation study took circa 30 working hours per evaluator to complete over a period of 8 weeks. before commencing, each evaluator went through a training and practice process conducted by the lead clinician, who explained the evaluation interface and guided them through the annotation of a practice note. a copy of the evaluator instructions can be found in appendix a. throughout the study, the authors and the lead clinician held weekly sessions with each evaluator where we shadowed the evaluation tasks through screen sharing. this helped us understand the difficulties in performing the tasks while ensuring the evaluators followed the guidelines set out for them. criterion agree. word overlap post-edit times 0.542 * -incorrect statements 0.541 * 0.431 † omissions 0.374 * 0.268 † ## conclusions we conducted a human evaluation study for the task of consultation note generation, computed agreement between evaluators, and quantified the extent to which human error impacts the judgements. we then carried out a correlation study with 18 automatic metrics, discussing their limitations and identifying the most successful ones. we found that the choice of human reference has a significant effect on all automatic metrics and that simple character-based metrics like levenshtein distance can be more effective than complex model-based metrics for the task of note generation. based on our findings, character-based levenshtein distance, bertscore, and meteor are the most suitable metrics to evaluate this task. we release all the data and annotations and welcome researchers to assess further metrics. table of examples given to the evaluators for reference.