An Investigation of Evaluation Methods in Automatic Medical Note Generation (2023)

Abstract: Recent studies on automatic note generation have shown that doctors can save significant amounts of time when using automatic clinical note generation (Knoll et al., 2022). Summarization models have been used for this task to generate clinical notes as summaries of doctor-patient conversations (Krishna et al., 2021; Cai et al., 2022). However, assessing which model would best serve clinicians in their daily practice is still a challenging task due to the large set of possible correct summaries, and the potential limitations of automatic evaluation metrics. In this paper we study evaluation methods and metrics for the automatic generation of clinical notes from medical conversation. In particular, we propose new task-specific metrics and we compare them to SOTA evaluation metrics in text summarization and generation, including: (i) knowledge-graph embedding-based metrics, (ii) customized model-based metrics with domain-specific weights, (iii) domain-adapted/fine-tuned metrics, and (iv) ensemble metrics. To study the correlation between the automatic metrics and manual judgments, we evaluate automatic notes/summaries by comparing the system and reference facts and computing the factual correctness, and the hallucination and omission rates for critical medical facts. This study relied on seven datasets manually annotated by domain experts. Our experiments show that automatic evaluation metrics can have substantially different behaviors on different types of clinical notes datasets. However, the results highlight one stable subset of metrics as the most correlated with human judgments with a relevant aggregation of different evaluation criteria.

## introduction in recent years, the volume of data created in healthcare has grown considerably as a result of record keeping policies @xcite . the documentation requirements for electronic health records significantly contribute to physician burnout and work-life imbalance @xcite . automatic generation of clinical notes can help healthcare providers by significantly reducing the time they spend on documentation, and allowing them to spend more time with patients @xcite . it can also improve the clinical notes' accuracy by reducing errors and inconsistencies in documentation, leading to patient records with higher quality. a reliable evaluation methodology is necessary to build and improve clinical note generation systems, but faces the two traditional limitations of evaluating natural language generation (nlg) systems. on one hand, human-expert evaluation, considered to be the most reliable way to evaluate nlg systems, can be both time-consuming and expensive. on the other hand, evaluating the performance of natural language generation (nlg) systems automatically can be challenging due to the complexity of human language. several metrics have been proposed to evaluate the performance of nlg systems, including lexical n-gram based metrics and embedding-based metrics that measure the similarity between a system's generated text and one or more reference texts using pre-trained language models. while several research efforts studied and compared automatic evaluation metrics on many opendomain and domain-specific datasets such as the cnn/dailymail and tac datasets @xcite @xcite @xcite , very few research works addressed the adequacy of evaluation metrics to the task of clinical note generation, where e.g., omitting critical medical facts in the generated text is a more significant failure point. to the best of our knowledge, only one research paper addressed this task based on one synthetic dataset of 57 mock consultation transcripts and summary notes @xcite . in this paper, we study evaluation methods and metrics for the automatic generation of clinical notes from medical conversations, including their correlations with human assessments of factual omissions and hallucinations. we also propose new task-specific metrics and we compare them to sota evaluation metrics across several clinical text summarization datasets. our contributions are as follows: ## related work different evaluation metrics are commonly used to evaluate text summarization and generation including rouge-n @xcite , bertscore @xcite , moverscore @xcite , bartscore @xcite , and bleurt @xcite . other metrics have been also proposed for evaluating factual consistency and faithfulness @xcite @xcite @xcite . to study their effectiveness, several efforts focused on comparing automatic metrics such as rouge and bleu based on their correlation with human judgments @xcite , and showed that automatic evaluation of generated summaries still has several limitations and biases @xcite . furthermore, in @xcite , the authors showcase that the effectiveness of an evaluation metric depends on the task (e.g. summarization) and on the application scenario (e.g. system-level/ summary level). despite observations of frequent disagreements in manual evaluation campaigns @xcite , expert-based evaluation remains an effective method to assess the performance of automatic metrics, especially in specialized domains. however, it relies on the availability of domain experts to rate the summaries and relevant datasets. recently, @xcite studied the task of medical note generation on a small set of 57 transcriptnote pairs, manually annotated by clinicians. their experiments showed that character-based levenshtein distance, bertscore, and meteor performed best for evaluating automatic note generation in that dataset. ## evaluation methodology to assess the relevance and suitability of automatic evaluation metrics for the task of clinical note generation, we create expert-based annotations for critical aspects such as factual consistency, hallucinations, and omissions. we then assess each metric in light of its correlation with manual scores generated from the expert annotations. ## task-specific evaluation metrics we study four different types of evaluation metrics for the task of automatic clinical note generation, that take into account the specificities of the medical domain by: (i) using embeddings built from medical knowledge graphs (e.g, umls), (ii) adapting model-based metrics (e.g., bertscore) by increasing the weights of medical terms, (iii) fine-tuning a model-based metric on a large collection of clinical notes, and (iv) building linear ensembles based on normalization and averaging of different metrics. ## evaluation setup we used the deberta-xlarge-mnli model @xcite as the base model for bertscore and the bleurt-20 checkpoint for the bleurt metric, that correlate better with human judgment than the default variants based on recent experiments. for bartscore metric, we used the bart model that was trained on the parabank2 dataset @xcite which was provided by the authors. from the designed and tested 50+ metrics and variants (e.g. our new metrics and variants, opendomain metrics, ensemble metrics), we selected the top 21 metrics to study and analyze their performance on the different datasets. the selection was based on the performance of these metrics and their pearson correlation scores with human judgments on the mts-dialog and the consult-full datasets. our first tests also included open-domain fact-based metrics such as factcc @xcite (trained on the cnn/dailymail dataset) and qa metrics such as quals @xcite (developed using xsum and cnn/dailymail) but they did not perform well due to the differences between open-domain and clinical questions/answers. the experiments were performed on one 80gb nvidia a100 gpu. ## performance of evaluation metrics we compute the pearson correlation scores between the automatic metrics and the reference scores. when both manual factual scores (f ), hallucination (h), and omission rates (o) are available, we compute an aggregate score: the intuition behind this score is that both omissions (o) and hallucinations (h) are critical criteria but they need to be taken into account in the context of factual correctness (f). on the larger consult-full dataset, rouge-n results followed a similar pattern on the consult hpi , consult assessment , consult exam , and consult results subsets, as presented in the fine-tuned clinicalbleurt metric achieves the highest correlation scores for the hallucination rate in the consult hpi , consult exam , and consult results datasets. the new medical metrics medbertscore-p and medbertscore-ps have the highest correlations for hallucination and omission rates on the consult assessment and consult exam datasets, respectively. ## conclusion while finding a relevant and generic evaluation metric for nlg systems remains a challenging task, our study shows that the solution to the problem is likely to be domain-and task-specific. in particular, metrics that did well on capturing factual accuracy did not necessarily capture critical aspects in clinical note generation such as hallucinations and key medical fact omissions. our experiments also show that language-model based metrics and metric ensembles can outperform sota n-gram based measures such as rouge when reference summaries are not biased. the extensive measurements and new metrics evaluated in this paper are valuable for guiding decisions on which metrics will be most effective for researchers to use going forward in their automated medical note generation scenarios.