A Preliminary Study on Evaluating Consultation Notes With Post-Editing (2021)

Abstract: Automatic summarisation has the potential to aid physicians in streamlining clerical tasks such as note taking. But it is notoriously difficult to evaluate these systems and demonstrate that they are safe to be used in a clinical setting. To circumvent this issue, we propose a semi-automatic approach whereby physicians post-edit generated notes before submitting them. We conduct a preliminary study on the time saving of automatically generated consultation notes with post-editing. Our evaluators are asked to listen to mock consultations and to post-edit three generated notes. We time this and find that it is faster than writing the note from scratch. We present insights and lessons learnt from this experiment.

## introduction in modern ehr (electronic health records) systems, at the end of a medical consultation the physician is required to file a consultation note detailing symptoms, examination, and treatment discussed. this is a pain point for physicians, who, according to a us study in 2017-2018 @xcite spend up to 44.2% of their time on clerical tasks, and this is a major contributor to physician burnout @xcite . a number of recent studies @xcite @xcite propose to use summarisation systems to automatically generate the consultation note from the transcript of the consultation. yet there is limited work on how to properly evaluate such a system so that it may be used in the clinical setting. intrinsic evaluation metrics through likert scales or ranking methods may help select the best model, but they * equal contribution don't ensure the model will never hallucinate information or miss key items when generating the consultation note. in this study we propose to evaluate generated consultation notes with an extrinsic measure based on post-editing time. we ask our evaluators (primary healthcare physicians) to listen to a consultation, write a consultation note, and post-edit a number of generated notes. we then compare the timings of each task to determine whether post-editing a note is faster than writing one from scratch. we focus on post-editing time because (i) it's simple to measure, and (ii) it provides a gate for adoption of the technology (i.e. post-editing a note should be faster than writing it from scratch). there are other extrinsic metrics which we intend to investigate in the future, such as patient satisfaction, doctor cognitive load, doctor-patient engagement, and usefulness for the next doctor accessing the note. ## related work post-editing has a long history in machine translation (mt) @xcite @xcite @xcite , with a number of production systems and tools using a semi-automatic approach to fix errors and check the output of the system before it is shown to the users @xcite . outside of mt, @xcite carry out a study on post-editing an nlg system for generating weather forecast from data. as an evaluation metric, @xcite define productivity as "the quantity of text an experienced translator could translate in a given period of time @xcite with the quantity of text gen-erated by [the system] that the same person could edit in the given time." to the best of our knowledge, post-editing is not widely used in document summarisation. we speculate this is partly because a post-editor of document summaries would need to read the entire document in order to accurately post-edit the summary, and this may minimise the benefit of having a generated summary compared to writing it from scratch. this is not the case, however, with consultation note generation, whereby the physician in charge of writing the note is the same physician who has conducted the consultation. here post-editing may be very valuable in saving physician time. ## data we partner with a uk healthcare provider, babylon health, which gives us access to a dataset of 800 proprietary consultation transcripts (automatically transcribed) and notes. the consultations span various topics within primary healthcare and are 10 minutes long on average. the notes are written by the physician who carried out the consultation and are in patient-friendly format, meaning they are in the same language the doctor used while talking to the patient and don't contain any abbreviations or acronyms. each note is made up of three sections: history & examination, diagnosis, and management. for our evaluation, we design a dataset of 57 mock consultations produced in a similar manner. we ask five babylon health physicians working in primary healthcare to act as doctors and a number of lay people (employees at babylon) to act as patients. participation is entirely voluntary and all participants sign a consent form explaining what the study would involve and the intended use of the data produced. they are given the choice to withdraw consent at any point. we give each patient a case card, prepared by a physician, that contains the condition they need help with and a list of medical details and symptoms. we record the audio of each mock consultation and ask the doctor to write a patient-friendly note as described above. figure 1 shows a mock patient-friendly note. we then employ a transcription agency to transcribe the recordings on an utterance level. figure 2 shows a transcript snippet from the same consultation. history & examination you developed lower abdominal pain 2 days ago. the pain came on gradually, is burning in nature, constant and is worsening. you have no bowel symptoms or pain on urination, but have noticed a pink colour to your urine. you have not noticed and blood in your urine. you feel some nausea, but have not vomited. you feel hot and sweaty. you are sexually active with a long term partner. your last sexual health check-up was 6 months ago. you last had unprotected sex 2 days ago. your last period was 2 weeks ago. you have no other symptoms. you have no past medical history, but use implanon for contraception. diagnosis: urinary tract infection. must rule out pregnancy management take a pregnancy test. give urine sample for a urine dip and to check for bacteria. treat with antibiotics. regular paracetamol for pain. review in 1 -2 days if no improvement, or earlier if symptoms are worsening. figure 1: mock consultation note written by a locum doctor, from our evaluation dataset. ## experimental setup we use the proprietary dataset of 800 consultations to finetune two automatic summarisation models based on bart @xcite . we feed the transcripts as inputs and the consultation notes as outputs. we then apply the models on the mock consultation dataset, using them to generate the history & examination section of the consultation note. for our experiment, we consider the generated notes from these two models (model a, model b) together with the original reference note (ref). we shuffle these for each task and tell the evaluators that all three notes are generated. the task is presented to the evaluators using heartex @xcite , a configurable annotation platform that allows us to customise the design of the evaluation task. our evaluators are three primary healthcare physicians. they are employed at babylon health [...] doctor: hello? good morning, tim. um, how can i help you this morning? patient: um, so i'm having some, some pain, uh, in my tummy, like the lower part of my tummy. um and i've just been feeling, quite, hot and sweaty. doctor: ok. right, i'm sorry to hear that. when, when did your symptoms all start? patient: about two days ago. doctor: ok. and whereabouts in your tummy is the pain, exactly? patient: uh, like below my belly button, it's like quite, sore when i press on it. doctor: ok. did the pain come on quite suddenly, or was it more gradual? patient: it hasn't been, it's more gradual and it's just, it is getting a bit worse now. doctor: ok, ok. and can you describe the pain to me? [...] and have experience in ai research annotation. the task we submit to them consists of the following steps: we let evaluators note down any key symptoms on a piece of paper as they would normally do during a consultation. we then present a number of questions to evaluate the quality of the given note. our criteria are correctness, completeness @xcite , and coherence. we agreed these criteria with the lead physician, who drafted definitions and a scoring guidance for the evaluators (figure 4 ). figure 5 shows the questions we ask for scoring these criteria and a sample annotation. we also ask evaluators to record their screen for the duration of the task. we use these recordings to calculate how long they took to write the note (step 2) and to edit each generated note (step 3). we use the difference of these two timings as our extrinsic measure to check whether editing a generated note is faster than writing one from scratch. ## results and discussion for this experiment, we run our evaluation on 3 of the 57 mock consultations. ## future work in this paper, we presented our preliminary evaluation study of consultation note generation with post-editing. based on the insights from this study, we plan to: if the issues described in this paper are addressed, we believe post-editing time can be a metric that is both valuable for evaluating model performance and relevant for use in production systems. finally, it is important to mention that while automation of medical note taking might help reduce physician burnout and allow the doctors to spend more time with the patients, there are ethical considerations associated to the use of such a technology. for example, time pressures or unwarranted trust in an automated system could potentially result in doctors not properly reviewing and editing the automated notes. also, post-editing is a very different cognitive task from writing a note from scratch, and that might put extra strain on doctors' already cognitively demanding workflows. in order to mitigate the above concerns in a production system, user experience design, system evaluation, and clinician on-boarding and training are crucially important.