J a SPICE : Automatic Evaluation Metric Using Predicate-Argument Structures for Image Captioning Models (2023)

Abstract: Image captioning studies heavily rely on automatic evaluation metrics such as BLEU and METEOR. However, such n-gram-based metrics have been shown to correlate poorly with human evaluation, leading to the proposal of alternative metrics such as SPICE for English; however, no equivalent metrics have been established for other languages. Therefore, in this study, we propose an automatic evaluation metric called JaSPICE, which evaluates Japanese captions based on scene graphs. The proposed method generates a scene graph from dependencies and the predicate-argument structure, and extends the graph using synonyms. We conducted experiments employing 10 image captioning models trained on STAIR Captions and PFN-PIC and constructed the Shichimi dataset, which contains 103,170 human evaluations. The results showed that our metric outperformed the baseline metrics for the correlation coefficient with the human evaluation.

## introduction image captioning has been extensively studied and applied to various applications in society, such as generating fetching instructions for robots, assisting blind people, and answering questions from images @xcite @xcite @xcite . in this field, it is important that the quality of the generated captions is evaluated appropriately. however, researchers have reported that automatic evaluation metrics based on n-grams do not correlate well with human evaluation @xcite . alternative metrics that do not rely on n-grams have been proposed for english (e.g., spice @xcite ); however, they are not fully applicable to all languages. therefore, developing an automatic evaluation metric that correlates well with human evaluation for image captioning models in languages other than english would be beneficial. : "akai kasa o sashita hito" a person with a red umbrella. figure 1 : overview of jaspice foot_0 . given a candidate caption and reference captions, our method parses the scene graph from the pas and dependencies, and then computes a score that represents the similarity between the candidate and the references by matching both graphs. spice is a standard metric for image captioning in english and evaluates captions based on scene graphs. spice uses universal dependency (ud) @xcite to generate scene graphs; however, ud can only extract basic dependencies and cannot handle complex relationships. in the case of japanese, the phrase "a no b" @xcite , which is composed of the nouns a and b has multiple semantic relations, which makes the semantic analysis of such phrases a challenging problem. for example, in the noun phrase "kinpatsu no dansei" ("a blond man"), "blond" (a) is an attribute of "a man" (b) and "a man" (b) is an object, whereas the noun phrase "dansei no kuruma" ("man's car") represents the relation of a "car" (b) being owned by a "man" (a), and dependency parsing using ud cannot accurately extract the relationship between a and b. given that ud cannot handle complex relationships and is therefore not suitable for constructing scene graphs, directly applying spice to the evaluation of japanese captions poses challenges. furthermore, problem settings exist that are difficult to evaluate using spice simply computed from an english translation (e.g., textcaps @xcite ). to address these issues, we propose jaspice, which is an automatic evaluation metric for image captioning models in japanese. jaspice is computed from scene graphs generated from dependencies and the predicate-argument structure (pas) and can therefore take complex relationships into account. fig. 1 illustrates our jaspice approach, where the main idea is that we first parse the scene graph from the pas and dependencies, and then computes a score that represents the similarity between the candidate caption and the reference captions by matching both graphs. for example, given the candidate caption "akai kasa o sashita hito" ("a person with a red umbrella") and the reference caption "kasa o sashita dansei" ("a man with an umbrella"), our method parses the scene graph and computes a score by matching both graphs. our method differs from existing methods because it generates scene graphs based on dependencies and the pas and uses synonym sets for the evaluation so that it can evaluate image captioning models in japanese. it is expected that appropriate scene graphs can be generated by reflecting dependencies and pas in scene graphs. it is also expected that the use of synonym sets will improve the correlation of metrics with human evaluation because it considers the matching of synonyms that do not match on the surface. the main contributions are as follows: ## problem statement in this study, we focus on the automatic evaluation of image captioning models in japanese. the terminology used in this study is defined as follows: in this study, we assume that we deal with the automatic evaluation of japanese image captions. however, some of the discussion in this study can be applied to other languages. ## proposed method for the evaluation of image captioning models, semantic structure is expected to be more effective than n-gram because, unlike machine translation, image captioning requires grounding based on the scene and relationships between objects in the image. therefore, utilizing the scene graph, which abstracts the lexical and syntactic aspects of natural language, can be beneficial for the evaluation of image captioning models. in this study, we propose jaspice, which is an automatic evaluation metric for image captioning models in japanese. jaspice is an extension of spice @xcite and can evaluate image captioning models in japanese based on scene graphs. although the proposed metric is an extension of spice, it also takes into account factors not handled by spice, that is, subject completion and the addition of synonymous nodes. therefore, we believe that the novelty of the proposed metric can be applied to other automatic evaluation metrics. the main differences between the proposed metric and spice are as follows: fig. 2 shows the process diagram of our method. the proposed method consists of two main modules: pas-based scene graph parser (pas-sgp) and graph analyzer (ga). ## conclusions in this study, we proposed jaspice, which is an automatic evaluation metric for image captioning models in japanese. the following contributions of this study can be emphasized: in future studies, we will extend our method by taking into account hypernyms and hyponyms.