Automatic Evaluation of Healthcare LLM s Beyond Question-Answering (2025)

Abstract: Current Large Language Models (LLMs) benchmarks are often based on open-ended or close-ended QA evaluations, avoiding the requirement of human labor. Close-ended measurements evaluate the factuality of responses but lack expressiveness. Open-ended capture the model’s capacity to produce discourse responses but are harder to assess for correctness. These two approaches are commonly used, either independently or together, though their relationship remains poorly understood. This work is focused on the healthcare domain, where both factuality and discourse matter greatly. It introduces a comprehensive, multi-axis suite for healthcare LLM evaluation, exploring correlations between open and close benchmarks and metrics. Findings include blind spots and overlaps in current methodologies. As an updated sanity check, we release a new medical benchmark–CareQA–, with both open and closed variants. Finally, we propose a novel metric for open-ended evaluations –Relaxed Perplexity– to mitigate the identified limitations.

## introduction the growing use of large language models (llms) in public domains, such as healthcare, shows promise for improving global quality of life @xcite . at the same time, the reliability and evaluation of llms in such sensitive topics requires extreme caution due to the potential impact on people's rights and well-being. llm evaluation today is approached through various perspectives, which consider different types of llm assessment: automatic evaluation (scalable and factual), user evaluation (utility and usability) @xcite , and expert evaluation (support and coherence) @xcite . while each of these evaluation perspectives serves distinct roles that contribute to a holistic assessment, automatic evaluation remains the most prevalent one due to its lack of dependency on human effort. within automatic evaluation, there are two types of tests. those which include closed-ended responses @xcite , namely multiple-choice question answering (mcqa), and those which have open-ended responses @xcite . close-ended mcqa validation enables the automatic verification of response factuality, but it does not reflect the complex nature of real world situations (e.g., clinical settings @xcite ). as such, mcqa alone often fails to identify critical short-comings of model performance @xcite @xcite @xcite . to incorporate a broader range of tasks relevant to the medical field @xcite , one typically has to rely on open-ended answers. that is, reference responses are not the only valid outputs. since these cannot be completely assessed for factuality without human expert supervision, approximate measures based on n-grams and model perplexity remain in place, which limits the reliability of these evaluations @xcite . efforts have been dedicated to analyze the relation between automatic evaluations and either user or expert evaluations, showing a lack of direct correspondence @xcite . this is explained by the difference in the model features these assess (e.g., factuality vs usability vs support capacity), pointing at their complementary nature. nonetheless, a similar analysis within the family of automatic evaluations is still pending; a study of the relations between openended and close-ended benchmarks and metrics, to understand which of these tests should be used, and when. for that purpose, we focus on the healthcare domain, providing the following contributions: ## methodology this study considers four different close-ended healthcare tasks, which include nine different datasets (e.g., medqa). these are all assessed using the accuracy metric. at the same time, six open-ended tasks are studied, based on nine distinct datasets (e.g., medtext). in this case, eleven different metrics are extracted. further details are shown in the open-ended version (english only) was created by rephrasing the questions from the closeended version using the qwen2.5-72b-instruct model. after the rephrasing process, the number of suitable questions was reduced to 3,730 qa pairs. this set retains the same categories as the closedended version. to ensure the validity of both the translations and rephrasing, 10 annotators conducted a manual review of a total of 360 samples, each reviewed by at least three evaluators. this process achieved a confidence level of 95% and a margin of error of 5% approximately. the translation results were positive, with all three evaluators agreeing on 83.1% of the questions as correct. based on this, we considered the translation to be of good quality. however, the percentage of rephrased qa pairs labeled as correct by the three evaluators was 65.8%. to address this, we conducted a second iteration incorporating feedback from human reviewers. the main issue identified was that while the rephrased answers might differ from the ground truth, they could still be considered valid. as a result, a new rephrasing iteration was carried out, explicitly prompting the model to account for this nuance, and questions with multiple valid answers were excluded. this led to the removal of 961 samples, leaving the final careqa (open-ended) dataset with 2,769 qa pairs. consequently, the percentage of correct labels increased to 73.6%. see appendix a for further details. ## relaxed perplexity: a novel metric by being optimized for next token prediction on the ground truth, llm's are optimized for perplexity. however, as seen before, this does not necessarily entail good performance on open or close-ended downstream tasks. additionally, perplexity can be greatly impacted by instruct-tuning and alignment techniques @xcite . on the other hand, it has been widely noted that models are more likely to arrive at the correct answer after outputting intermediate tokens, commonly known as chain of thought (cot) @xcite , and that this happens even without specific cot prompting (wang and zhou, 2024). however, perplexity fails to capture this improvement, and can be negatively impacted by the presence of intermediate tokens. to evaluate factuality in open-ended benchmarks, with no dependence on confounders or exact formulation while accounting for the potential benefits of intermediate tokens, we propose relaxed perplexity. given a question and a target, we wish to estimate for the ℓ more likely n-token sequences sampled from the model given question, because the events seq i n and seq j n are mutually exclusive. in this notation, p(seq this allows to evaluate correctness in the model's answers probability distribution, with no regard for the exact formulation. further, for a given prompt and fixed sampling parameters, the metric is perfectly self consistent. we thus test it with the olaph (jeong et al., 2024) medical factuality dataset. in contrast to perplexity, we observe that relaxed perplexity assigns higher scores to models finetuned on healthcare datasets. more details on the mathematical formulation, implementation and results of relaxed perplexity can be found in appendix d. ## conclusions this study finds very weak correlations between close-ended and open-ended benchmarks. these results highlight the complementary roles of closeended and open-ended approaches, and the limited insights provided by individual tests. it thus advocates for broader evaluation setups. even within open-ended benchmarks targeting the same task (e.g., aci-bench and mts-dialog), no consistently high correlations were found. this indicates that different benchmarks assess distinct model capabilities, underscoring the significance of the benchmark's design. the analysis of evaluation metrics for openended benchmarks identified three distinct clusters that are particularly relevant for assessing medical models: (1) perplexity-based metrics, (2) n-grambased metrics combined with semantic similarity metrics, and (3) llm-as-a-judge metrics. notably, none of these clusters showed strong correlations with the close-ended mcqa evaluation. additionally, differences in resilience to answer rephrasing and self-consistency were observed, due to the distinct ways these metrics are computed. the findings highlight the importance of selecting appropriate benchmarks and evaluation metrics designed for specific tasks. in this regard, the introduced careqa benchmark, featuring both closed-and open-ended formats, serves as a sanity check of existing tests, while the proposed relaxed perplexity metric fills a gap in evaluation by focusing on factuality and being resistant to exact formulations in an open-ended setting.