GPT -4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions (2023)

Abstract: There is growing interest in systems that generate captions for scientific figures. However, assessing these systems’ output poses a significant challenge. Human evaluation requires academic expertise and is costly, while automatic evaluation depends on often low-quality author-written captions. This paper investigates using large language models (LLMs) as a cost-effective, reference-free method for evaluating figure captions. We first constructed SCICAP-EVAL, a human evaluation dataset that contains human judgments for 3,600 scientific figure captions, both original and machine-made, for 600 arXiv figures. We then prompted LLMs like GPT-4 and GPT-3 to score (1-6) each caption based on its potential to aid reader understanding, given relevant context such as figure-mentioning paragraphs. Results show that GPT-4, used as a zero-shot evaluator, outperformed all other models and even surpassed assessments made by computer science undergraduates, achieving a Kendall correlation score of 0.401 with Ph.D. students’ rankings.

## introduction there has been increasing interest in automating caption generation for scientific figures in scholarly articles. the scicap dataset @xcite , an extensive collection of scientific figures and captions from arxiv papers, encouraged more work in this area and led to several innovative approaches @xcite @xcite . however, despite advances in caption generation for scientific figures, evaluating the results is still challenging. human evaluation is costly as only domain experts can assess captions, making crowdsourced evaluation impractical. meanwhile, automatic evaluation is not reliable. @xcite found that 1 scicap-eval is accessible at https://github.com/ crowd-ai-lab/scicap-eval . domain-specific ph.d. students did not prefer captions rated highly by automatic scores. fortunately, recent findings in this area may provide solutions to the challenge. figure captions were found to be similar to summaries of figure-mentioning paragraphs @xcite , i.e., most words in captions can be semantically traced back to these paragraphs, and captions can be effectively generated using text-summarization models. a missing piece from previous research is the use of this finding for evaluation, which we address in this paper. this paper explores the cost-effective, reference-free evaluation of scientific figure captions using pre-trained large language models (llms). figure 1 overviews the process. the intuition is that since a scientific figure's caption in scholarly articles often functions as a condensed summary of all figure-mentioning paragraphs @xcite , llms, when given the appropriate context, can effectively evaluate how well the caption captures the essential information from these contexts @xcite @xcite . we first introduced scicap-eval, a human evaluation dataset for scientific figure captions that contains 600 figures from arxiv papers. for ev-ery figure, we collected six captions: one authored by the original paper authors and five machinegenerated. two participant groups independently assessed these captions. the first group, ph.d. students from the figure's domain, ranked the six captions for each figure. the second group, undergraduate computer science and informatics students, rated each caption's helpfulness. we used ph.d. students for their in-depth understanding but recognized their limited availability. undergraduates were easier to recruit, but their knowledge might be insufficient. we aimed to examine if undergraduates could replace ph.d. students in building a human evaluation dataset for scientific figure captions. equipped with scicap-eval, we then prompted llms like gpt-4 and gpt-3.5 to assign a score (1-6) to each caption, based on how helpful they could be for readers to understand the paper. the results showed gpt-4's effectiveness as a zero-shot evaluator. it achieved a 0.401 kendall correlation with ph.d. students' rankings, surpassing all other models and even exceeding evaluations by computer science and informatics undergraduates. this paper presents three contributions. firstly, we have validated the effectiveness of using llm to evaluate figure captions, exploring an avenue previously untouched with such techniques. secondly, we have developed a new dataset, cultivated over a few months with an investment of over @xmath03,300. we engaged 20 undergrad-uate participants, each responsible for 20 batches. each batch consisted of 30 figures, and the time taken to finish each batch ranged from 30 minutes to an hour. additionally, 15 ph.d. students were recruited, with each student tasked to rank 4 batches. each batch comprised 10 figures, each having 6 different captions. the time taken by ph.d. students for each batch varied from 30 minutes to an hour. the authors' institute's institutional review board (irb) approved this study. human assessments by ph.d. students on a sixitem ranking task. we recruited 15 ph.d. students (who are not coauthors of this paper) specializing in cv, nlp, and hci to rank six captions for each of the 40 figures in their respective fields. specifically, we formed a ranking task with six items, where each ph.d. student manually ranked 6 different captions for a figure. these rankings were based on how helpful the captions were in understanding the figures. the annotation interface is shown in figure 2 in appendix a. participants were compensated at @xmath115 per batch, with annotation times ranging from 30 minutes to one hour. to ensure annotation quality, we provided each participant with a tutorial using an additional 30 figures. ## methods in this section, we describe different approaches used for caption rating, including zero-shot, fewshot, and chain-of-thought prompting with llms, and fine-tuning a classifier using the data collected. ## experimental results experimental setups. we conducted experiments with four llms: gpt-4 (openai, 2023), gpt-3.5 @xcite , falcon @xcite , and llama-2 @xcite , across various settings. we parsed the prediction scores from the outputs. when the llms failed to predict, we allocated the lowest score of 1. for the scibert classifier, we split the dataset into model setting/ training data (a) correlation with phd students' reversed rank [good to bad caption: 6, 5, ..., 2, 1] correlation with phd students' reciprocal rank (b) original order (picking good samples) [good to bad caption: 1 1 , 1 2 , ..., 1 5 , 1 6 ] (c) reversed order (spotting bad samples) [good to bad caption: 1 6 , 1 5 , ..., 1 2 , 1 1 ] ρ τ rs ρ ρ gpt-4 zero-shot .501 .401 .491 .391 -.492 few-shot .531 .429 .523 .425 -.513 cot (qa) .283 .270 .315 .277 -.223 cot (qa-yn) .357 .334 .400 .324 -.307 gpt-3.5 zero-shot .465 .370 .462 .383 -.433 few-shot .462 .371 .462 .403 -.402 cot (qa) .270 .276 .347 .249 -.226 cot (qa-yn) .365 .329 .404 .317 -.327 llama 2-70b zero-shot .407 .342 .413 .326 -.392 few-shot .424 .353 .430 .335 -.405 falcon-7b zero-shot .026 .048 .055 .018 -.030 few-shot .044 .048 .058 .044 -.035 falcon-40b zero-shot .169 .137 .167 .152 -.136 few-shot .150 .119 .143 .126 -.137 scibert* undergrad .329 .290 .329 .261 -.319 phd .372 .308 .379 .372 -.372 human undergrad .221 .195 .221 .206 -.196 gpt-4 prompting achieved the highest correlations with ph.d. students' assessments. ## conclusion and future work in conclusion, we have shown the capacity of llms, specifically gpt-4, to evaluate scientific figure captions effectively. this work produced scicap-eval, a human evaluation dataset of 3,600 captions, both original and machinegenerated, for 600 arxiv figures. using scicap-eval, we confirmed gpt-4's ability to assess caption quality when supplied with context from figure-mentioning paragraphs. gpt-4 even outperformed computer science and informatics undergraduates, achieving a kendall correlation score of 0.401 against ph.d. students' ratings. future work will leverage this evaluation capability to cleanse noisy training data, a well-known challenge in figure caption generation @xcite . furthermore, we plan to investigate the personalized captions generation to cater to individual reader requirements. last, we will explore ways to consider caption factuality in the evaluation metrics.