C ap A rena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era (2025)

Abstract: Image captioning has been a longstanding challenge in vision-language research. With the rise of LLMs, modern Vision-Language Models (VLMs) generate detailed and comprehensive image descriptions. However, benchmarking the quality of such captions remains unresolved. This paper addresses two key questions: (1) How well do VLMs actually perform on image captioning, particularly compared to humans? We built CapArena, a platform with over 6000 pairwise caption battles and high-quality human preference votes. Our Arena-style evaluation marks a milestone, showing that leading models like GPT-4o achieve or even surpass human performance, while most open-source models lag behind. (2) Can automated metrics reliably assess caption quality? Using human annotations from CapArena, we evaluate traditional and recent captioning metrics, as well as VLM-as-a-Judge. Our analysis reveals that while some metrics (e.g., METEOR) show high caption-level agreement with humans, their systematic biases lead to inconsistencies in model ranking. In contrast, VLM-as-a-Judge demonstrates robust discernment at both the caption and model levels. Building on these insights, we release CapArena-Auto, an accurate and efficient automated benchmark for detailed captioning, achieving 93.4% correlation with human rankings at just $4 per test. All data and evaluation resources have been open-sourced.

## introduction image captioning, the task of generating textual descriptions for images, has long been a fundamental challenge in both the computer vision and natural language processing communities @xcite @xcite . it has broad and valuable applications, such as assisting visually impaired individuals and supporting multimedia retrieval. driven by progress in llms, modern vision-language models (vlms) are capable of generating long, detailed descriptions of image content (openai, 2023; chen et al., 2024b), moving beyond the short captions of traditional methods. the proliferation of vlms presents new opportunities for the image captioning field. however, image captioning has not advanced as expected. current vlms focus on tasks like visual question answering @xcite and multimodal reasoning @xcite , bypassing the essential task of image captioning. a few works still rely on mscoco @xcite )-a dataset with an average caption length of 10 words, which is clearly outdated for evaluating advanced vlms. this obstacle stems from the inherent difficulty in evaluating detailed captions. unlike multiple-choice questions or mathematical reasoning, captioning lacks explicit answers, resulting in the absence of reliable evaluation benchmarks. researchers are unable to assess the captioning capabilities of existing vlms, nor effectively evaluate and improve their own models. this paper addresses two key questions to drive the evolution of image captioning in the llm era: to explore the first question, we developed caparena, which includes over 6000 high-quality human annotations to evaluate the detailed captioning capabilities of 14 advanced vlms and humans. in our preliminary study, we found that traditional scoring methods are unsuitable for annotating detailed captions' quality due to their fine-grained intricacy and diversity. inspired by llm evaluation @xcite , we adopted a pairwise caption battles paradigm. the resulting model rankings are shown in figure 1 . for the first time, we observe that state-of-the-art models, such as gpt-4o, are comparable to or even surpass human-level performance, marking a pivotal milestone in image captioning. while open-source vlms achieve competitive results on general benchmarks, ca-parena reveals a persistent performance gap between them and commercial models. an exception is internvl2-26b @xcite , a smaller open-source model that stands out with its strong performance, underscoring the potential of compact and efficient vlms for detailed captioning. in response to the second question, we conducted a comprehensive analysis of traditional and recent captioning metrics, as well as the ability of vlmas-a-judge @xcite to assess caption quality. we compared these metrics against human preferences from caparena. our results reveal that most metrics designed for short captions, such as clipscore @xcite , fail entirely in the detailed captioning task. although some rule-based metrics, such as meteor @xcite , exhibit decent agreement with human judgments at the caption level, they suffer from systematic biases across different vlms. this results in low agreement at the model level, where the rankings produced by these metrics deviate significantly from human rankings. in contrast, we introduce vlm-as-a-judge with reference captions, which demonstrates robust discernment for detailed captions. it achieves the highest alignment with human judgments at both the caption and model levels. in light of these findings, we release caparena-auto, an automated benchmark for detailed captioning. it comprises 600 samples and innovatively adopts the pairwise battle paradigm to improve evaluation reliability. vlm-as-a-judge is employed to estimate human preferences by comparing captions against three baseline models. with 94.3% correlation to human rankings and just @xmath04, making it an effective and affordable evaluation benchmark for detailed captioning. ## conclusion in this paper, we explore the task of detailed image captioning in the llm era. we conducted the first large-scale human-centered empirical study to benchmark advanced vlms performance. the results demonstrate that, for the first time, leading models (e.g., gpt-4o) achieve or surpass humanlevel performance, while open-source models lag behind. next, we provided an in-depth analysis of existing captioning metrics. our experiments reveal that vlm-as-a-judge sets a new standard, highlighting that systematic biases in existing metrics are the key factor affecting alignment with human preferences. finally, we release caparena-auto, an automated benchmark that closely aligns with human preferences, offering a cost-effective and efficient tool for detailed captioning evaluation. our findings lay the foundation for the future development of image captioning.