FLEUR : An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model (2024)

Abstract: Most existing image captioning evaluation metrics focus on assigning a single numerical score to a caption by comparing it with reference captions. However, these methods do not provide an explanation for the assigned score. Moreover, reference captions are expensive to acquire. In this paper, we propose FLEUR, an explainable reference-free metric to introduce explainability into image captioning evaluation metrics. By leveraging a large multimodal model, FLEUR can evaluate the caption against the image without the need for reference captions, and provide the explanation for the assigned score. We introduce score smoothing to align as closely as possible with human judgment and to be robust to user-defined grading criteria. FLEUR achieves high correlations with human judgment across various image captioning evaluation benchmarks and reaches state-of-the-art results on Flickr8k-CF, COMPOSITE, and Pascal-50S within the domain of reference-free evaluation metrics. Our source code and results are publicly available at: https://github.com/Yebin46/FLEUR.

## introduction evaluating image captions is essential as it provides a significant indicator of the model's ability to understand visual and language information effectively @xcite . however, there are two primary challenges with existing image captioning evaluation metrics. existing methods 1) require reference captions to evaluate candidate captions 2 and 2) lack explainability. ## method the overall framework of fleur is depicted in figure 2 . fleur derives a raw score based on the prompt that accurately reflects grading criteria (orange speech bubble on the left of figure 2 ), and makes the score continuous by using the probabilities computed by an lmm for score smoothing (blue dashed line box in figure 2 ). words marked in italics will be explained in detail in the following sections. in this paper, we opt for llava @xcite as our lmm to introduce score smoothing. please refer to appendix a for a specification of llava that we used. ## discussion and conclusion we propose an explainable reference-free metric, fleur, achieving notable correlations with human judgment. our metric utilizes the latest lmm, llava, to compare only an image and a candidate caption without reference captions. therefore, no additional cost and time such as creating reference captions is needed. directly comparing an image and a candidate caption allows for appropriate scoring and the generation of a reasonable explanation. this explainability not only unveils the reason behind the evaluated score, but also holds the potential to provide insights that can enhance image captioning model performance, or to be utilized as a dataset by employing score-explanation pairs in training other models. note that fleur is the first work to use an lmm in caption evaluation metrics. it may be possible to find more suitable prompts through prompt engineering @xcite in order to obtain better scores and explanations. furthermore, fleur can be extended to other lmms through prompts tailored to the model being used, rather than the prompts we employed. we hope our work can contribute to the explainability of image captioning evaluation metrics.