Using Large Language Models to Evaluate Biomedical Query-Focused Summarisation (2024)

Abstract: Biomedical question-answering systems remain popular for biomedical experts interacting with the literature to answer their medical questions. However, these systems are difficult to evaluate in the absence of costly human experts. Therefore, automatic evaluation metrics are often used in this space. Traditional automatic metrics such as ROUGE or BLEU, which rely on token overlap, have shown a low correlation with humans. We present a study that uses large language models (LLMs) to automatically evaluate systems from an international challenge on biomedical semantic indexing and question answering, called BioASQ. We measure the agreement of LLM-produced scores against human judgements. We show that LLMs correlate similarly to lexical methods when using basic prompting techniques. However, by aggregating evaluators with LLMs or by fine-tuning, we find that our methods outperform the baselines by a large margin, achieving a Spearman correlation of 0.501 and 0.511, respectively.

## introduction biomedical question answering (qa) is concerned with building systems that automatically answer biomedical questions posed by humans in natural language @xcite . to develop and optimise these systems, we must use metrics that evaluate the quality of their output. automatic metrics such as rouge @xcite and bleu @xcite have been shown to correlate poorly with human evaluation @xcite , and human annotations are prohibitively expensive and impractical in the biomedical domain @xcite . to rectify this problem, recent research has suggested using medium-sized model-based evaluators and large language models (llms). model-based evaluators such as bertscore @xcite or bleurt @xcite have demonstrated improvements over n-gram based metrics in various natural language generation (nlg) evaluation contexts such as summarisation and qa @xcite . however, their evaluation capability is still far below that of humans. the recent improvement of llms for various tasks has fostered research on their use for the evaluation of the performance of text generation tasks such as summarisation and dialogue generation @xcite . in this paper, we experiment with using llms to evaluate biomedical query-focused summarisation systems. to the best of our knowledge, this is the first study for such a task. in particular, we compare the correlation between human judgements and llm-based evaluators for the evaluation of several systems participating in the "ideal answer" task of bioasq 2021 and 2022 @xcite . our study examines different prompting strategies for such evaluations. llms, through prompting, have also been used to evaluate text via likert scale scoring (a fivelevel scale) @xcite , leading to greater performance than n-gram and model-based evaluation techniques. however, @xcite showed that outputting only a number could be suboptimal but that asking the llm to explain its rating can lead to an increase in correlation to human ratings. the pairwise ranking also showed promising results @xcite , with accuracy outperforming n-gram and model-based evaluators. llms can also leverage emergent abilities which can be used to incorporate in-context learning (icl) and chain-of-thought (cot) in their evaluation strategies. in icl @xcite , a model is provided with input-output examples of a downstream task instead of being trained or fine-tuned on the task. in cot @xcite , a complex task is broken down into multiple intermediate steps to improve reasoning in llms. @xcite used gpt-4 to achieve the highest correlation with human evaluations in comparison to other model and n-gram based metrics. @xcite , using gpt-3, showed that using few-shot prompting -a small number of examples added to the prompt -can reach or exceed the state-of-the-art on multi-dimensional evaluation and that this is robust to the sampling method of in-context examples whether it be random or representative of the range of scores in the example pool. however, @xcite showed using a small llm (orca-mini-v3-7b), that oneshot prompting doesn't bring significantly greater results than zero-shot. the use of llms as meta-evaluators who use their reasoning capabilities to combine diverse evaluation techniques has also seen promise. in @xcite , various llms were given supplementary evaluation metrics like nli score @xcite , bleurt and probability-based llm techniques to aid with their judgements, with the meta-evaluation outperforming all individual evaluators. the above research, however, was not used in query-focused summarisation tasks. this paper is the first that tests the use of llms for the evaluation of biomedical query-focused summarisation. ## methodology we use the human judgements of runs participating in the "ideal answers" question answering task of bioasq @xcite . such "ideal answers" are multi-sentence answers, and therefore the task is about biomedical query-focused summarisation. in particular, training and development is based on systems (runs) participating at bioasq 2020 1 , whereas testing is on runs participating at bioasq 2021 2 and bioasq 2022 3 . the rationale for using bioasq 2020 for training is that it is the 1 run mq1 in each batch submitted by @xcite . 2 run mq1 in each batch submitted by @xcite . ## experiments we investigated several evaluation strategies, from baseline well-known token-based metrics to the use of llms as evaluators in different settings. ## results and discussion similar to previous work, we find that tokenbased methods perform worse than llms in general @xcite . possible causes of the relatively poor performance of token-based approaches is, as mentioned by @xcite , that rouge is unable to incorporate information on context and semantic meaning, whereas sci-bertscore is less sensitive to errors in text, especially if the candidate is lexically or stylistically similar, and both are insensitive to negation. these limitations have increasingly larger impacts on abstractive over extractive systems, making evaluating outputs using these metrics potentially unreliable. still, ## conclusions we compared the use of traditional evaluation metrics (rouge, bertscore) with the use of llms for the evaluation of query-focused summarisation of biomedical questions. for this, we used system runs that participated in bioasq challenge, and computed correlation between automatic evalua-tions and human judgements. our experiments show that, while llms with basic prompting do not outperform rouge or bertscore, approaches that use fine-tuning or that combine llms with additional scorers, significantly improve correlation with human judgements.