D oc L ens: Multi-aspect Fine-grained Evaluation for Medical Text Generation (2024)

Abstract: Medical text generation aims to assist with administrative work and highlight salient information to support decision-making.To reflect the specific requirements of medical text, in this paper, we propose a set of metrics to evaluate the completeness, conciseness, and attribution of the generated text at a fine-grained level. The metrics can be computed by various types of evaluators including instruction-following (both proprietary and open-source) and supervised entailment models. We demonstrate the effectiveness of the resulting framework, DocLens, with three evaluators on three tasks: clinical note generation, radiology report summarization, and patient question summarization. A comprehensive human study shows that DocLens exhibits substantially higher agreement with the judgments of medical experts than existing metrics. The results also highlight the need to improve open-source evaluators and suggest potential directions. We released the code at https://github.com/yiqingxyq/DocLens.

## introduction medical text generation has been widely applied to various scenarios, including clinical note generation @xcite , report summarization @xcite , and patient question summarization (abacha and demner-fushman, 2019). in report summarization, for example, text generation systems aim to assist medical experts by automatically summarizing the salient findings in a ct or mr report, which reduces the time on paperwork and supports decision-making @xcite . to help medical experts decide whether to adopt text generation systems or which system to use, it is imperative to have a reliable evaluation methodology. one line of work on medical evaluation conducts human evaluation under multiple aspects, reflecting different criteria of an ideal generation result @xcite @xcite . to capture which exact information is inaccurate or omitted, other human evaluation methods conduct more fine-grained evaluations by examining each fact individually @xcite . due to the high cost and poor scalability of human evaluation, another line of work focuses on automatic evaluation. however, existing automatic medical evaluation methods typically assign a coarse-level score for the entire system output @xcite , without indicating the aspects or criteria the score reflects. recent general-domain evaluation methods focus on more fine-grained units such as sentences @xcite or atomic facts @xcite . however, existing methods neglect evaluation aspects critical to medical generation or require external knowledge sources in evaluation. in this paper, we propose doclens, which automatically conducts evaluation of medical text generation at a fine-grained level, including both reference-based and reference-free aspects. as shown in figure 1 , to evaluate the recall and precision of clinically significant information in the generation, we conduct a reference-based evaluation for the completeness and conciseness of the system-generated output. specifically, we break down the system output (e.g., generated clinical note) and reference (e.g., human-written note) into subclaims and assign a binary score for each subclaim. in real-world scenarios, ai systems are typically used in an assisting role, where the medical experts need to judge the reliability of the generated information as part of their process using the ai systems. as a result, we evaluate attribution, which checks whether each piece of generated information is properly grounded in the input. specifically, we conduct a reference-free evaluation and check whether each generated sentence contains accurate references back to the input. doclens can be computed with various types of evaluator models and we experiment with three representatives: a proprietary model (gpt-4 (ope-nai, 2023)), an open-source instruction-following model (mistral @xcite ), and a supervised natural language inference (nli) model (true @xcite ). we apply do-clens with the three evaluators to benchmark multiple medical generation systems on three tasks: clinical note generation, radiology report summarization, and patient question summarization. to compare the quality of doclens with existing metrics and to compare different evaluators of do-clens, we conduct a human study to investigate how well each metric aligns with medical experts' judgment. experiments show that doclens exhibits substantially higher agreement with medical experts than existing metrics commonly used in the medical domain. the results also reveal the gap between open-source and proprietary evaluators. our analyses further suggest potential directions to improve open-source evaluators. contributions. @xcite we identify crucial aspects of medical text evaluation and design corresponding metrics for conducting a fine-grained evaluation. ## evaluation framework of doclens there are two special characteristics of clinical text: (a) criticality: every piece of information is essential. medical documentation must be free from inaccuracies and omissions. (b) assistance: the generated text will be examined by experts. as decisions can be life-critical, the generated text is positioned as a resource rather than a final product. in line with these considerations, our proposed framework employs three evaluation aspects: completeness ( §3.1), conciseness ( §3.2), and attribution ( §3.3) and design a set of corresponding metrics. the proposed metrics can be automatically computed using a variety of evaluator models ( §3.5). as shown in figure 2 , we formulate the three evaluation aspects as verification of entailment relations between the input, system output, and humanwritten references. an illustrative example of each metric is shown in figure 3 . ## conclusions and future works we present doclens, a medical evaluation framework that judges three aspects with a set of finegrained level metrics. doclens can be computed with various types of evaluator models, including proprietary and open-source instruction-following models and nli models. we apply doclens to three tasks: clinical note generation, radiology report summarization, and patient question generation. human study shows that doclens exhibits substantially higher agreement with human judgments than existing metrics. the results also reveals the substantial gap between proprietary and open-source evaluators. to close the gap, as suggested by our case study, future works could improve open-source evaluators by ( 1 ) continuous pretraining the model on medical corpora, and (2) instruction-tuning the model for entailment, where we can construct training data by adapting existing nli datasets or leveraging the model itself to generate silver labels. another potential direction is to train the evaluator to generate multiple forms of feedback, such as the explanation of its judgement used in this work. then evaluator model can then be applied to further improve medical text generation models.