A Survey of Evaluation Methods of Generated Medical Textual Reports (2023)

Abstract: Medical Report Generation (MRG) is a sub-task of Natural Language Generation (NLG) and aims to present information from various sources in textual form and synthesize salient information, with the goal of reducing the time spent by domain experts in writing medical reports and providing support information for decision-making. Given the specificity of the medical domain, the evaluation of automatically generated medical reports is of paramount importance to the validity of these systems. Therefore, in this paper, we focus on the evaluation of automatically generated medical reports from the perspective of automatic and human evaluation. We present evaluation methods for general NLG evaluation and how they have been applied to domain-specific medical tasks. The study shows that MRG evaluation methods are very diverse, and that further work is needed to build shared evaluation methods. The state of the art also emphasizes that such an evaluation must be task specific and include human assessments, requesting the participation of experts in the field.

## introduction medical report generation (mrg) 1 @xcite is a subfield of natural language generation (nlg) and aims to present information from various sources in textual form to synthesize salient information, with the goal of reducing the time spent by domain experts in writing medical reports and providing supporting information for decisionmaking. mrg includes all systems used to generate medical documentation, such as the generation of radiology reports @xcite , discharge summaries or soap notes @xcite , etc. the evaluation of automatically generated texts is important for the validity of the systems, especially in the era of chatgpt and its increased usage in medical domain @xcite . however, it has often been reported that the two main approaches of nlg evaluation -human-based and automatic-based -need to be improved @xcite . on the one hand, the use of automatic metrics for system quality assessment has been criticized for two main points: they are uninterpretable and do not correlate well with human evaluations @xcite . on the other hand, the deployment of human evaluation is sometimes too complex. indeed, crowdsourcing solutions are not always reliable or adequate while using experts, when available, can lead to high costs. furthermore, there is a lack of unified framework/criteria (van der @xcite . although there have been many surveys on nlg evaluation @xcite @xcite , there is no systematic study on report generation evaluation in the medical domain. it is worth mentioning @xcite , which has done survey work in the area of automatic report generation from medical images, including an analysis of evaluation methods. in this paper, we focus on mrg tasks and their evaluations. we include more than 20 papers in this study, classified into two broad categories: text-totext and data-to-text. we summarize the evaluation methods currently in use and make recommendations for future evaluation of mrg systems. ## evaluation in nlg in this section, we will briefly introduce automatic evaluation in nlg and then will look at human evaluation with some current practices. ## evaluation for text-based medical report generation systems in the following subsections, we summarize the automatic measures and human evaluation used in the literature in ## conclusion automating medical report generation can save time for experts and provide crucial information for decision-making. however, the evaluation process is necessary for validation and adoption of mrg systems in the real world. due to the specificity of domain-specific nlg tasks like mrg, their evaluation requires more investigation and subtlety. mrg evaluation shares similarities with general nlg evaluation, but it differs in its focus on domain knowledge and task-specific concerns, especially in the assessment of conceptual accuracy of medical concepts. however, the question of which medical facts to pay attention to (correlation, consensus, etc) is an open question, requiring close collaboration with experts in the field. in addition, the evaluation of mrg systems requires both intrinsic and extrinsic evaluation. intrinsic evaluation focuses on properties of the system's output, while extrinsic evaluation involves professional experts in the design acceptability process, developing a list of key questions, and post-editing. future research should prioritize extrinsic evaluation, particularly in scenarios where references are unavailable, and developing efficient, medical task-specific automated measures.