PR - MCS : Perturbation Robust Metric for M ulti L ingual Image Captioning (2023)

Abstract: Vulnerability to lexical perturbation is a critical weakness of automatic evaluation metrics for image captioning. This paper proposes Perturbation Robust Multi-Lingual CLIPScore(PR-MCS), which exhibits robustness to such perturbations, as a novel reference-free image captioning metric applicable to multiple languages. To achieve perturbation robustness, we fine-tune the text encoder of CLIP with our language-agnostic method to distinguish the perturbed text from the original text. To verify the robustness of PR-MCS, we introduce a new fine-grained evaluation dataset consisting of detailed captions, critical objects, and the relationships between the objects for 3,000 images in five languages. In our experiments, PR-MCS significantly outperforms baseline metrics in capturing lexical noise of all various perturbation types in all five languages, while maintaining a strong correlation with human judgments.

## introduction image captioning @xcite @xcite is a multimodal task that automatically generates captions that describe the visual content of an image and integrates multiple disciplines of visual and textual modality. image captioning is a natural language generation (nlg) task @xcite , but the evaluation metric has different characteristics from other nlg metrics @xcite . image captioning metrics should be able to evaluate not only linguistic fluency and syntactic thoroughness but also semantic correspondence to visual content @xcite . evaluation criteria for image captioning have evolved from n-gram-based metrics (papineni † corresponding author. 1 all the datasets and code are available in https:// github.com/yong1-kim/pr-mcs . our dataset is distributed under the cc-by-nc 4.0 license. ## related works image captioning metrics as with other natural language generation tasks, image captioning can be evaluated using various proposed metrics. bleu @xcite , rouge @xcite , and meteor @xcite are representative image captioning metrics based on n-gram similarity with reference captions. other widely used reference-based metrics include cider @xcite , which weights n-gram similarity @xcite through tf-idf @xcite , and spice @xcite , which evaluates captioning based on scene graphs. recently, reference-based metrics using embedding similarity with reference captions based on a model, such as bertscore @xcite , bert-tbr @xcite , and vilbertscore @xcite , have been introduced. clipscore researchers have also proposed unreferenced image captioning metrics that evaluate generated captions by comparing them with original images that does not require groundtruth caption @xcite @xcite . clipscore @xcite , which is also a reference-free metric, relies heavily on the clip @xcite model, trained with 400 million image caption pairs using a contrastive objective function that distinguishes original image-caption pairs from unmatched captions. the calculated clipscore is the weighted value of cosine similarity between image embedding and text embedding encoded by the clip model. despite the fact that clipscore exhibits a high correlation with human evaluation, it is limited in that it is an image captioning metric that applies only to english. in this study, we propose a new multilingual image captioning metric developed by extending clipscore to a multilingual setting. perturbation robustness in a recent study, @xcite selected various criteria for use in assessing how various nlg evaluation metrics perform. in addition, perturbation was applied to multiple image captioning factors to assess the perturbation robustness of the image captioning metrics. @xcite provided a perturbation checklist of metrics for nlg tasks; we go further and present a novel metric that overcomes the limitations of other metrics. we select some perturbation criteria from among those suggested by sai et al. (2021), designate them as target perturbations, and show that the clipscore cannot detect these perturbations in multiple languages. even if the generated captions are corrupted, clipscore outputs similar results for the original and corrupted sentences. this study proposes a novel metric with perturbation robustness based on clipscore to address its weaknesses in multiple languages. ## m-finecap3k dataset to evaluate the performance of pr-mcs, we introduce a new image captioning dataset, m-finecap3k. most existing image captioning datasets are limited to english @xcite . therefore, a model for machine translation (mt) from english to other languages is essential to evaluate image captioning in various languages. however, translated evaluation set is highly dependent on the performance of the mt model and is highly likely to have an english-language bias. in addition, translation is inarticulate in the target language because it is difficult to reflect the unique characteristics of each language, such as word order and lexical choice @xcite . therefore, the results obtained using a translated evaluation set achieve poorer agreement with human evaluation than those obtained using the english evaluation set. for these reasons, a human-annotated image captioning evaluation set with a wide variety of languages is needed. ## experiments our framework seeks to identify whether a given metric can detect lexical noise in a generated caption. through exhaustive experiments, we evaluate whether the pr-mcs developed as described in this paper successfully distinguishes the perturbed caption from the original caption. ## conclusion in this paper, we propose pr-mcs, a perturbationrobust metric for multilingual image captioning using language-agnostic fine-tuning. pr-mcs, developed by fine-tuning the text encoder of clip, can distinguish lexically perturbed text from original text. we also propose a new fine-grained multilingual image captioning evaluation set, m-finecap3k, for use in perturbation robustness evaluation. experimental results for existing datasets and our new dataset show that pr-mcs detects perturbation well and is robust to perturbation in multiple languages. furthermore, we verify that pr-mcs is a useful metric with strong correlation with human judgment, using our m-capeval1k.