imap S core: Medical Fact Evaluation Made Easy (2024)

Abstract: Automatic evaluation of natural language generation (NLG) tasks has gained extensive research interests, since it can rapidly assess the performance of large language models (LLMs). However, automatic NLG evaluation struggles with medical QA because it fails to focus on the crucial correctness of medical facts throughout the generated text. To address this, this paper introduces a new data structure, imap , designed to capture key information in questions and answers, enabling evaluators to focus on essential details. The imap comprises three components: Query, Constraint, and Inform, each of which is in the form of term-value pairs to represent medical facts in a structural manner. We then introduce imap Score, which compares the corresponding medical term-value pairs in the imap to score generated texts. We utilize GPT-4 to extract imap from questions, human-annotated answers, and generated responses. To mitigate the diversity in medical terminology for fair term-value pairs comparison, we use a medical knowledge graph to assist GPT-4 in determining matches. To compare imap Score with existing NLG metrics, we establish a new benchmark dataset. The experimental results show that imap Score consistently outperforms state-of-the-art metrics, demonstrating an average improvement of 79.8% in correlation with human scores. Furthermore, incorporating imap into n-gram, embedding, and LLM metrics boosts the base versions, increasing correlation with human scores by averages of 89.9%, 81.7%, and 32.6%, respectively.

## introduction llms, assimilated extensive knowledge, have demonstrated impressive potential as medical consultants. they can act as professional experts to aid doctors in diagnosis, treatment decision-making, and offering second opinions on complex cases * equal contribution † corresponding author however, most of the metrics fail to recognize sma's alternate name, werdnig-hoffmann disease. gpt-4 correctly identifies this alternate name and should receive equal recognition and votes as palm-2. upon unifying the disease name in the 3 rd round, all metrics accurately identified and correctly voted for the disease. @xcite . they also assist patients by providing diagnostic information, treatment options, and medical popular science @xcite . however, the practical application of llms in clinical settings, where medical factual correctness is critical, encounters challenges. one significant issue is the tendency of llms to generate "hallucinated" content-information that is plausible but incorrect or unfounded in evidence @xcite . meanwhile, many llms are lauded for their proficiency in medical questionanswering (qa), as evidenced by their high scores on standardized single-choice or multiple-choice medical examinations @xcite . however, excelling in such exams does not necessarily equate to being a competent medical professional. recently, bleu @xcite ) and rouge @xcite have become popular in medical response evaluation @xcite . however, they fall short in effectively assessing content quality or capturing medical language nuances and factual accuracy @xcite . llm-based metrics @xcite also face challenges due to the diversity and complexity of medical terminology, impacting their ability to verify medical facts accurately. furthermore, these models' uniform treatment of all tokens in the responses neglects the crucial aspect of medical correctness, affecting the reliability of llm-based evaluations in healthcare settings. as shown in figure 1 , an example of automatic evaluation for disease diagnosis qa generated by three llms: chatgpt, palm-2 and gpt-4. in this scenario, chatgpt mistakenly receives the majority of support from six metrics (votes 6) in the first voting on raw generated text, indicating that most metrics are incapable of accurately evaluating disease diagnoses in their entirety. when the focus shifted to disease names in the second round, the majority of metric votes leaned towards the correct answer from palm-2. this suggests key information extraction could be a viable approach, mirroring the practice of highlighting and aligning essential details in the generated answer and the label during manual assessment. however, the results still neglect the fact that sma (the disease mentioned in the example) is also known as werdnig-hoffmann disease, a detail overlooked by most metrics. if we further unify the disease name and conduct the third round of vote, all metrics unanimously select the correct answer (generated by palm-2 and gpt-4), underscoring the importance of terminology standardization in evaluation. these considerations guide a more refined evaluation paradigm for medical qa, involving: 1) identifying core needs and critical evidence, 2) unifying terminology and ensuring alignment, and 3) prioritizing key information and scoring. to extract the key information, we introduce a new data structure, imap, which captures the core requirements and informs from questions and an-swers, parsing them into medical term-value pairs based on primary needs, constraints, and informed points. gpt-4 can be readily instructed to generate the imap for questions and answers. additionally, a medical knowledge graph aids in unifying medical expressions. leveraging the imap of the question, reference, and generated response, we proposed a new metric, imapscore, which compares corresponding medical term-value pairs within the imap to score generated texts. this metric can be flexibly applied to prompt llms to evaluate text from various perspectives, integrating principles from human scoring to align more closely with expert assessments. we quantitatively evaluate imapscore by comparing the correlation between imapscore and human score for medical qa. since there are no public available benchmark for medical qa with human ratings, we created a new benchmark that can also be utilized for other medical qa-related research. we enlisted three medical professionals to rate the generated outputs of two public datasets, hmedqa in chinese and icliniq foot_0 in english from three perspectives: factual accuracy, completeness, and specific. 2 after performing extensive experiments and conducting in-depth analysis from diverse perspectives, we found that imapscore surpasses other metrics, significantly improving the correlation with human scores. furthermore, the integration of imap into n-gram, embedding, and llm metrics notably enhances the base versions. our contributions are as follows: metrics for evaluating generated response. various metrics have been proposed to measure the semantic equivalence of the generated texts against the reference. n-gram-based metrics are most commonly used; they rate the quality of generated text by computing scores derived from the lexical overlap between the predictions and the reference. representatives include bleu which emphasizes n-gram precision, and rouge which focuses on n-gram recall. nist (doddington, 2002), a variant of bleu, emphasizes the importance of informative n-grams, offering a more nuanced assessment. embedding-based metrics offer better semantic understanding compared to n-grambased methods that depend on superficial text overlap. bertscore (zhang et al., 2019), a prominent embedding-based metric, employs bert to derive features from sentences, producing similarity scores through word pair inner products. another noteworthy metric is bartscore (yuan et al., 2021), which formulates evaluating generated text as a text generation task from bart (lewis et al., 2019). moverscore (zhao et al., 2019) advances by evaluating semantic similarity, tracking "information units" from reference to generated text. in the realm of fact-based summarization evaluation, questeval (scialom et al., 2021) and qafacteval (fabbri et al., 2021) employ question generation models and question answering models to facilitate a detailed comparison of facts, while alignscore (zha et al., 2023) evaluates text alignment through chunk comparison. however, these general task methods may require specific dataset training and lack specificity evaluation. recently, llm-based approaches such as gptscore (fu et al., 2023) and g-eval (liu et al., 2023b) have been employed for evaluating generated responses. leveraging llms as evaluators presents significant promise in appraising responses across a range of tasks. however, these methods often struggle to measure finegrained medical facts and are heavily dependent on predefined evaluation protocols and prompts. factscore @xcite proposes an approach to evaluate factual accuracy by dividing generated text into short sentences. in contrast, our method delves into term-level relationships and explores additional dimensions, including comprehensiveness and specificity, providing a more nuanced and detailed evaluation. medical qa benchmarks. numerous significant studies have enriched the benchmarks of medical qa. medqa @xcite provides a trilingual, open-domain qa dataset derived from medical exams. medmcqa @xcite a benchmark primarily based on questions from indian medical institutions. multimedqa @xcite encompasses a wide range of healthcare topics through multiple-choice questions. mlec-qa @xcite is the largest chinese biomedical qa dataset from national exams. care-mi @xcite is a chinese benchmark focusing on misinformation in maternity and infant care. cmexam (liu et al., 2023a) is a dataset with explanations for model reasoning from chinese medical exams. recently, cai et al. (2023) proposed medbench, a comprehensive chinese medical benchmark that employs bleu and rouge for self-assessment. ## experiment results our research aims to explore the following: 1) the efficacy of imapscore in medical qa evaluations; 2) the impact of incorporating imap on improving current leading approaches. main results of imapscore. other interesting findings include: 1) llmbased metrics surpass other baselines, highlighting their effectiveness in medical qa evaluation. 2) on hmedqa, palm-2 surpasses gpt-4 through better medical language alignment with labels, but imapscore boosts gpt-4's human score alignment, surpassing palm-2. 3) differences in hmedqa and icliniq correlations hint at dataset variability, yet imapscore consistently enhances evaluation performance across both. ## conclusions this paper presented imap, a data structure that parses medical qa into term-value pairs, enhancing evaluation by capturing essential needs and evidence from qas. besides, we proposed imapscore, a novel metric that uses imap for text scoring, aligning closely with expert assessments. furthermore, we built a new benchmark for evaluating nlg in medical qa, focusing on the medical factual correctness generated by llms. experimental results demonstrated that imapscore is in better agreement with humans compared to existing metrics.