CIDE r- R : Robust Consensus-based Image Description Evaluation (2021)

Abstract: This paper shows that CIDEr-D, a traditional evaluation metric for image description, does not work properly on datasets where the number of words in the sentence is significantly greater than those in the MS COCO Captions dataset. We also show that CIDEr-D has performance hampered by the lack of multiple reference sentences and high variance of sentence length. To bypass this problem, we introduce CIDEr-R, which improves CIDEr-D, making it more flexible in dealing with datasets with high sentence length variance. We demonstrate that CIDEr-R is more accurate and closer to human judgment than CIDEr-D; CIDEr-R is more robust regarding the number of available references. Our results reveal that using Self-Critical Sequence Training to optimize CIDEr-R generates descriptive captions. In contrast, when CIDEr-D is optimized, the generated captions’ length tends to be similar to the reference length. However, the models also repeat several times the same word to increase the sentence length.

## introduction automatically describing image content using natural sentences is an important task to help include people with visual impairments on the internet, making it more inclusive and democratic. this task is known as image captioning. it is still a big challenge that requires understanding the objects, their attributes, and the actions they are involved in. thus, linguistic models are also needed to verbalize the semantic relations in addition to visual interpretation methods. in the last few years, there has been significant progress in image captioning thanks to the availability of a large amount of annotated data in relevant datasets @xcite @xcite @xcite . these datasets have in common multiple reference captions, which comprehend short sentences with a low length variance. also, most of the datasets contain only english captions, whereas datasets with captions described in other languages are scarce. thus, we proposed the #pracegover dataset @xcite for the image captioning task with descriptions foot_0 in portuguese. our dataset consists of images and captions collected from posts on instagram tagging the hashtag #pracegover. pracegover is a movement (web para todos, 2018) that aims to spread the accessibility culture on social media. it has inspired many local laws in brazil that establish that all posts made by public agencies must tag #prace-gover and contain a short description of the image. also, many business and personal profiles have joined this initiative. hence, the amount of data produced by those users has great growth potential. in contrast to the other datasets in this literature, in #pracegover, each image has only one reference description, whose length is about four times longer than the average length, in terms of the number of words, in datasets such as ms coco @xcite . also, the variance of length in our dataset is around ten times higher than in the literature. thus, #pracegover is more challenging than others. we demonstrated that training models using self-critical sequence training @xcite (scst) to optimize cider-d is not a good approach for datasets with the same characteristics as ours, in particular with single reference and high variance of length. cider-d @xcite has a severe penalty concerning the length that punishes at the same intensity any difference in the number of words between the generated and the reference captions. hence, it forces the model to generate descriptions similar in length to the ground truth even though it has to repeat words to reach the reference length. in the end, the model learns to repeat words impairing the lexical cohesion of the generated description. we tackle the problem of penalizing at the same intensity any difference in length by modeling the penalty proportionally to the reference length. also, we introduced a penalty regarding lexical cohesion so that the more a word is repeated, relative to the ground truth, the higher is the penalty. these two penalties combined aim to give the models the opportunity of generating more succinct captions when the ground truth comprehends longer descriptions. we name this new metric cider-r, a robust alternative to cider-d for datasets with high length variance. also, we demonstrated that cider-r is preferred to cider-d when only one reference is available. finally, our results reveal that cider-r is more suitable for the context of datasets created from data collected from social media. our key contributions are fourfold: (1) we investigated the impact of evaluation metrics in the scenario of datasets with high variance of length (e.g., #pracegover; (2) we showed that optimizing cider-d on datasets with long references and high variance of length can make the model learn to repeat words to increase the sentence length (figure 1 ); (3) we proposed cider-r that better measures the lexical cohesion and has a flexible penalty to length variance; (4) we demonstrated that optimizing cider-r in context of long sentences and high variance of length produce more descriptive captions. ## related work the image captioning task has been accelerated thanks to the availability of a large amount of annotated data in relevant datasets, for instance, flickr8k @xcite , and ms coco @xcite . these data made it possible to train complex models such as deep neural networks @xcite . in the last few years, a broad collection of architectures has been proposed in image captioning @xcite @xcite @xcite @xcite @xcite @xcite . notable achievements have been made with self-critical sequence training @xcite , which is a formulation of reinforcement learning that enables the use of nondifferentiable functions as optimization objectives, such as caption metrics. evaluating the generated captions is a challenging task itself because it is based on the evaluator's subjectivity. many metrics were proposed to measure the generated text's quality to encourage progress in this field despite the subjectivity intrinsic to this process. typically, these metrics compare a candidate sentence against a set of reference sentences (ground truth). most of the metrics are rule-based that relies on n-gram to assign a score to the candidate sentence, e.g. bleu @xcite , rouge @xcite and me-teor @xcite . however, ngram based metrics cannot capture the semantic relations and negatively correlate with human judgment. thus, @xcite , a semantically sensitive score, which computes the term frequency-inverse document frequency (tf-idf) weights to n-grams in the candidate and reference sentences and then compare them using cosine similarity. spice @xcite uses the similarity of scene graphs parsed from the reference and candidate captions to evaluate the generated descriptions. although spice obtains a significantly higher correlation with human judgments compared to n-gram based metrics, it does not check whether the grammar is correct, and it ignores syntactic quality, as pointed out in @xcite . also, spice uses stanford scene graph parser @xcite which is only available for a limited number of languages. recently, word embeddings has also been employed for evaluation in bertscore @xcite , moverscore @xcite and bleurt @xcite . ## #pracegover dataset recently, a social movement called pracegover (web para todos, 2018) arose on the internet, standing for people with visual impairments besides having an educational purpose. the initiative aims to call attention to the accessibility question by stimulating users to post images accompanied by a short description of their content. this project has inspired many local laws that establish that public agencies' posts on social media must contain a description of the image. inspired by this movement, we developed a crawler to collect public posts from instagram associated with the tag used by the followers. we leverage these data, which comprise images and text captions, to create #pracegover (santos et al., 2021) dataset, the first large dataset for image captioning in portuguese. similar to vizwiz-captions @xcite , our dataset's captions are addressed to visually impaired people. we are continuously collecting more data, and, thus, our dataset increases over time. currently, it comprehends 551,000 instances with images labeled with a single caption. ## model architecture in our experiments, we employ aoanet @xcite since it is one of the state-of-art models for image captioning. this model is based on the attention on attention (aoa) module, which determines the importance of attention results given queries. aoa module is an extension of multi-head attention @xcite that introduces an "attention gate" and an "information vector" to address the issue of the attention mechanism returning irrelevant results to the query. the model is an encoder-decoder architecture where the encoder module comprises the feature extractor and a refining module to refine feature vectors' representation. in contrast, the decoder network comprehends lstm, multi-head attention, attention on attention, and word prediction layers. the encoder takes the image features as input and creates an embedding representation. the decoder, in turn, uses the embedding and the previous state to model the context vector that represents the newly acquired information. the context vector is used to predict the next word. we train our models using the method self-critical sequence training @xcite , which consists of two phases. the first phase optimizes the cross-entropy loss, and then an evaluation metric is optimized through a reinforcement learning approach. our experiments trained the models during two epochs, minimizing the cross-entropy loss and 15 epochs maximizing the considered evaluation metric. finally, the cider-d score is defined as: although vedantam et al. @xcite have introduced the gaussian penalty exp{- in light of this problem, we propose cider-d modified, which we called cider-r, where we replace the gaussian penalty with two other penalties: length and repetition. we relax the gaussian penalty to be more flexible in the context of varied sentence length. therefore, we developed the length penalty that changes according to the reference length so that the longer the reference sentence, the more permitted is the difference |l(c i ) -l(s ij )|. also, to avoid the model to predict repeated words, we introduce the repetition penalty that deals specifically with this problem. this penalty considers the number of occurrences of a word in the candidate and reference sentences, assigning an intensified penalty score as that word frequency in the candidate sentence differs from the frequency in the reference. formally, given the generated description c i and the reference s ij , the length penalty p en l (c i , s ij ) is defined as follows: repetition penalty p en r (c i , s ij ) is formally defined as: note that the length penalty (equation 5 ) is similar to the gaussian penalty in equation 3, except for the denominator, which we replace by the term l(s ij ) 2 . this change makes the metric more flexible concerning the difference in lengths generated and reference sentences. finally, we define cider-r as follows: our formulation of cider-r has replaced the gaussian penalty with the weighted geometric average of length and repetition penalties, where k r and 1 -k r is the weight assigned to repetition and length penalty, respectively. we have executed a random search @xcite to tune the weights assigned to each penalty, and we find the best configuration is k @xmath0.8. this configuration forces the model to avoid repetition. on the other hand, it generates shorter sentences than when k @xmath1.0, which is expected since we are decreasing the length penalty's weight. ## experiments 6.1 caption-level human correlation in this experiment, we evaluate how much the automated metrics match human judgment. we use the pascal-50s proposed by vedantam et al. @xcite . this dataset consists of triplets of sentences a, b, and c, where a is a reference caption and b and c are candidates. each triplet has an annotation with the majority vote for the question, "which of the two sentences, b or c, is more similar to a?". the dataset comprehends four different combinations of sentences b and c: human-correct (hc), human-incorrect (hi), human-model (hm), and model-model (mm). to evaluate the metrics, we compute the score for each triplet, using different metrics, for b and c having a set of references, and then we calculate the metric's accuracy. the accuracy is defined as the number of times the metric assigns the highest score to the candidate pair most commonly preferred by human evaluators divided by the total number of pairs (b, c). to assess the impact of the number of available references on metrics performance, we compute the accuracy for each metric, varying the number of available references from 1 to 48. we show the results in figure 3 foot_1 . we can see that cider-r has a similar performance to cider-d, considering the four types of pairs. however, cider-r outperforms the other metrics at distinguishing between two machinegenerated captions (model-model pairs). especially when only one reference caption is available, which is the most common scenario of large datasets such as ours, cider-r outperforms cider-d by 1.2% of accuracy. model-model pairs is a most interesting subtask from a practical point of view because distinguishing better-performing algorithms is the primary motivation of the automated metrics. ## conclusion this work demonstrated that cider-d's performance is hampered by the lack of multiple reference sentences and sentence lengths variance. thus, we proposed cider-r as an alternative to cider-d in the context of a single reference and where the sentence length variance matters, such as large datasets based on social media data. we modified the gaussian penalty to make cider-d invariant to the difference in length between the reference caption and the generated description. also, we introduced a penalty for word repetition to better compare captions with a length similar to the reference but with words repeated several times. further, we showed that cider-r is more accurate and closer to human judgment than cider-d, and it is also more robust regarding the number of references. still, we showed that cider-r is a preferred evaluation metric to cider-d for datasets with single reference and high variance of sentence length, such as #pracegover. additionally, we demonstrated that using scst to optimize cider-d is not a good approach regarding datasets with a high length variance because the models are more likely to repeat words to increase the length of the generated caption. finally, we demonstrated that training models to optimize cider-r instead of cider-d produces more descriptive captions. for future work, we intend to create a dataset similar to pascal-50s with captions in portuguea young girl is about to blow out her candle. a young dog is preparing to blow out her candle. a mottled brown dog and cow with two little birds outdoors. a bird sitting on the on on on cat and a dog and a dog and and sitting on next to. cat laying on couch looking into distance with remote control by paw. se with the same characteristics of #pracegover to validate the accuracy of the metrics in the context of long sentences and high variance. also, we intend to remove proper names and process synonyms to reduce the vocabulary size and replace words with low-frequency with synonyms more common in subsets of #pracegover to evaluate how this impacts the models' performance.