Towards Objectively Evaluating the Quality of Generated Medical Summaries (2021)

Abstract: We propose a method for evaluating the quality of generated text by asking evaluators to count facts, and computing precision, recall, f-score, and accuracy from the raw counts. We believe this approach leads to a more objective and easier to reproduce evaluation. We apply this to the task of medical report summarisation, where measuring objective quality and accuracy is of paramount importance.

## introduction natural language generation in the medical domain is notoriously hard because of the sensitivity of the content and the potential harm of hallucinations and inaccurate statements @xcite . this informs the human evaluation of nlg systems, selecting accuracy and overall quality of the generated text as the most valuable aspects to be evaluated. in this paper we carry out a human evaluation of the quality of medical summaries of clinical reports generated by state of the art (sota) text summarisation models. our contributions are: (i) a re-purposed parallel dataset of medical reports and summary descriptions for training and evaluating, (ii) an approach for a more objective human evaluation using counts, and (iii) a human evaluation conducted on this dataset using the approach proposed. ## related work a recent study by @xcite gives a comprehensive view on different approaches to text summary evaluation. while many of these can * equal contribution be wholly or partly translated between different domains, the medical domain remains particularly problematic due to the sensitive nature of its data. @xcite and @xcite try to establish if there is a correlation between automatic and human evaluations of clinical summaries. a 4-point and 2-point likert scales are used for the human evaluation. in @xcite the authors generate free-text summary letters from the data of 31 different patients and compare them to the respective original physician-composed discharge letters, measuring relative completeness, quantifying missed data items, readability, and functional performance. closest to our approach is the pyramid method by @xcite , which defines semantically motivated, sub-sentential units (summary content units) for annotators to extract in each reference summary. scus are weighed according to how often they appear in the multiple references and then compared with the scus extracted in the hypothesis to compute precision, recall, and f-score. ## data the mtsamples dataset comprises 5,000 sample medical transcription reports from a wide variety of specialities uploaded to a community platform website foot_0 . the dataset has been used in past medical nlp research @xcite @xcite including as a kaggle dataset foot_1 . there are 40 medical specialties in the dataset, such as 'surgery', 'consult -history and phy.', and 'cardiovascular / pulmonary'. each specialty contains a number of sample reports ranging from 6 to 1103. the reports are free text with headings, which change according to the specialty. however, all reports also have a description field, which is a good approximation of a summary of the report. the length of each report varies greatly according to the specialty, with an average of 589 words for the body of the report, and 21 words for the description. figure 1 shows an example of mtsamples reports, inclusive of description. given the brevity of some descriptions, we discard reports with descriptions shorter than 12 words and consider a dataset of 3242 reports. by examining the dataset, we note that descriptions are mostly extractive in nature, meaning they are phrases or entire sentences taken from the report. to quantify this we compute n-gram overlap with rouge-1 (unigram) and rouge-l (longest common n-gram) @xcite precision scores, which are 0.989 and 0.939 respectively. we split the dataset into 2 576 reports for training (80%), 323 for development (10%) and 343 for testing (10%). we perform the split separately for each medical specialty to ensure they are ade-quately represented and then aggregate the data. the dataset, models, and evaluation results can be found on github foot_2 . ## experimental setup for our experiment, we consider one baseline and three sota automatic summarisation models (extractive, abstractive, and fine-tuned on our training set respectively). more specifically: we generate descriptions with these 4 models using the entire clinical report text as input. ## human evaluation protocol we select 10 clinical reports and summary descriptions from our mtsamples test set. our subjects are three general practice physicians. they are employed at babylon health and have experience in ai research evaluation. the task is implemented with the heartex annotation platform foot_4 , which lets researchers define tasks in an xml language and specify the number of annotators. it then generates each individual task and collates the results. the task involves (i) reading the clinical report, (ii) reading the reference description (supplied by the dataset, see figure 1 ), (iii) then evaluating 4 generated descriptions by answering 5 questions (for a total of 40 generated descriptions). we ask the evaluators to count the "medical facts" in each generated description and to compare them against those in the reference. initially, we considered listing the types of facts to be extracted, as done by thomson and reiter (2020), but the sheer diversity in the structure and content across the specialties in our dataset made this approach impractical. instead, we give evaluators instructions containing two examples and ask them to extrapolate a process for fact extraction. figure 2 shows the instructions we give them. the evaluation consists of reading a clinical report and a number of short descriptions, then quantifying how many "medical facts" were correctly reported. we understand that the definition of a "medical fact" is vague, and so it's up to your interpretation. as an example, in the following description: there are (arguably) 4 facts: the patient had a syncopal episode last night. she did not have any residual deficit. she had a headache at that time. she denies chest pains or palpitations. here there are (arguably) 5 facts: based on this set of questions, we gather the following raw counts: • r: facts in the reference description • g: facts in the generated description • r&g: facts in common • c: correct facts in the generated description we use these raw counts to compute four derived metrics: ## results and discussion patient's weight has dropped from 201 pounds to 201 pounds. she has lost a total of 24 pounds in the past month. whereas, the source report states: weight today is 201 pounds, which is down 3 pounds in the past month. she has lost a total of 24 pounds. ## future work in this paper we presented an evaluation of the quality of medical summaries using fact counting. the results of this study help us to identify a number of insights to guide our future work: