SMURF : S e M antic and linguistic U nde R standing Fusion for Caption Evaluation via Typicality Analysis (2021)

Abstract: The open-ended nature of visual captioning makes it a challenging area for evaluation. The majority of proposed models rely on specialized training to improve human-correlation, resulting in limited adoption, generalizability, and explainabilty. We introduce “typicality”, a new formulation of evaluation rooted in information theory, which is uniquely suited for problems lacking a definite ground truth. Typicality serves as our framework to develop a novel semantic comparison, SPARCS, as well as referenceless fluency evaluation metrics. Over the course of our analysis, two separate dimensions of fluency naturally emerge: style, captured by metric SPURTS, and grammar, captured in the form of grammatical outlier penalties. Through extensive experiments and ablation studies on benchmark datasets, we show how these decomposed dimensions of semantics and fluency provide greater system-level insight into captioner differences. Our proposed metrics along with their combination, SMURF, achieve state-of-the-art correlation with human judgment when compared with other rule-based evaluation metrics.

## introduction visual captioning serves as a foundation for image/video understanding tools and relies on caption evaluation for identifying promising research directions. rule-based caption evaluation approaches like the n-gram based cider @xcite and parsed semantic proposal based spice @xcite specifically are able to provide researchers with meaningful feedback on what their algorithm is lacking. however, ngram based methods are sensitive to stop words and sentence parsers are often inconsistent, leading to @xcite showing that neither method 1 smurf source codes and data will be released at https: //github.com/joshuafeinglass/smurf . figure 1 : scatter plot utilizing standardizations of sparcs and spurts. the ground truth captions are sourced from the karpathy test split of the coco dataset @xcite with one used as a baseline for automatic captioners @xcite @xcite . for each captioner, a 75% confidence ellipse (1.15 standard deviations from the mean) is generated. a caption near the centroid of each captioner is shown as an example along with the caption scores from 100 randomly sampled images. the normalized ellipse overlap between an automatic captioner and human captions, h∩m area m area , gives an overall evaluation of typical performance at a system-level on a scale of 0 to 1, with 1 being human-caption level. the primary difficulty of caption evaluation is its cross-modal nature introducing ambiguity into the expected output, resulting in a ground truth that is no longer a single outcome, but a large set of potential outcomes of varying levels of quality. from this problem setting, the novel concept of "typicality" arises naturally. a desirable caption is one that is atypical enough linguistically that it uniquely describes the scene, follows typical natural language protocols, and matches a typical semantic description of a scene. linguistically, the number of typical sequences is characterized by the entropy rate @xcite . current work estimates the english language as having an entropy rate of only 1.44 bits/letter @xcite , implying that the typical set of english is only a tiny fraction of the full space of potential text. self-attention transformers are language models that are able to identify the distinguishing contextual features of this typical set and as a result have now become the staple of natural language understanding tasks. here we define typicality based on the distance of a candidate text's features from expected features of the typical set. we call this linguistic typicality estimation method model-integrated meta-analysis (mima) and use the function, f mima , to create referenceless fluency metrics attune to captioning needs. rather than assuming a predefined evaluation task and introducing bias by fine-tuning the self-attention transformer, our method extracts the inherent properties of language learned by transformers @xcite by treating self-attention layers as probability distributions as demonstrated in @xcite . our approach represents the first integration of a fluency specific metric that demonstrably improves correlation with human judgment for caption evaluation. by removing stop words from the candidate text, f mima is able to create a metric that assesses a relatively new fluency criteria in captioning: style. we refer to this metric as stochastic process understanding rating using typical sets (spurts). style can be thought of as the instantiation of diction and is necessary for generating human-level quality captions. stylized captions describe a much smaller set of media, leading to machines instead generating the most typical caption that is still semantically correct. this results in a significant gap between machine and human captioners that can be seen in diction-based examples such as the use of the common words like "dog" and "food" instead of more descriptive words like "schnauzer" and "lasagna". the other aspect of fluency assessed by f mima is grammar. unlike style, grammar is not essential for caption quality, however, highly atypical syntax can potentially lead to awkward captions, so we develop a separate grammatical outlier penalty. we then define a lightweight and reliable typicality based semantic similarity measure, semantic proposal alikeness rating using concept similarity (sparcs), which complements our referenceless metrics and grounds them to the reference captions. by matching word sequences, current methods limit the scope of their evaluation. instead, we take non-stopword unigrams and further coalesce them into concepts through stemming, then combine the reference texts, like in @xcite , using a novel semantic typicality measure of the reference text's concepts to evaluate the semantic similarity of a candidate and reference text. spurts and sparcs can be used to assess system-level differences between captioners as shown in figure 1 . based on this analysis, the m 2 transformer lags behind 2015 models in terms of similarity to human captions, even though both 2020 captioners achieved state-of-the-art results based on cider standards. this difference becomes even more significant when you consider that the use of style makes it more difficult for a caption to be semantically correct. human captions, m 2 transformer @xcite , x-transformer @xcite , and google @xcite incur a total grammar outlier penalty of -44.93, -7.47, -7.56, and -4.46, respectively. in order to provide caption-level insight as well, we combine spurts, sparcs, and our grammar outlier penalty into one metric -semantic and linguistic understanding fusion (smurf) -which rewards captions based on semantics and fluency. contributions: our key contributions are: 1. a novel and widely-applicable model metaanalysis technique, mima, which estimates the typicality of candidate text and which provides a means of assessing transformer robustness. 2. three novel evaluation metrics useful for both caption-level and system-level evaluation: stylefocused spurts, semantic-focused sparcs, and their combination which incorporates grammatical outliers as well, smurf. 3. experiments showing that sparcs and smurf achieve sota performance in their respective areas of semantic evaluation and humanmachine evaluation at both a system and caption-level. 4. evidence showing that the performance of automatic evaluation metrics has been underestimated relative to voting-based human evaluation metrics. ## related work originally, popular rule-based metrics from machine translation that were mostly n-gram based, namely meteor @xcite , bleu @xcite , and rouge @xcite , were used for caption evaluation. @xcite introduced the more semantically sensitive cider which uses tf-idf to identify distinguishing n-grams and then compares them using cosine similarity. spice @xcite greatly improved upon n-gram based approaches by using a sentence parser to generate semantic propositions. word moving distance scores @xcite have also been used for semantic evaluation with limited success. bertscore @xcite used cosine similarity of embeddings from the self-attention transformer, bert, and achieved state-of-the-art results on coco but provided little interpretation of their approach. domain specific training approaches have also been introduced with limited adoption. @xcite al. (2018); jiang et al. (2019); sharif et al. (2019) present a training approach for caption evaluation where an image grounding and/or caption based turing test is learned based on training data from human and machine captioners. an adjusted bertscore @xcite , bleurt @xcite , and nubia @xcite utilize transformer embeddings for comparison between reference and candidate text, then perform caption dataset specific fine-tuning of the model downstream. the importance of fluency in captioning has been widely recognized. @xcite attempted to integrate cider and spice to create a cost function attune to both lexicographical and semantic qualities for captioning optimization. @xcite identified the presence of less frequent, distinguishing words within human-generated text in the coco dataset. @xcite recognized the importance of style in captions and integrated it into their model without sacrificing semantics. referenceless evaluation, first proposed in @xcite as a referenceless grammar error correction (gec) evaluation metric, has been recognized as an effective avenue for fluency evaluation as a whole @xcite , along with combined approaches @xcite . more recently, perception score @xcite outlined a general paradigm for training referenceless quality evaluation. ## conclusion and future work in this paper, we use information theory based typicality analysis to capture a new perspective on the problem of caption evaluation. our analysis leads us to two caption evaluation metrics that capture separate dimensions of caption quality and a fused metric. we have performed experiments demonstrating their correlation with human judgment, showed how these methods could be used to perform multi-aspect system-level analysis of algorithm performance, and performed caption-level studies explaining why combining these two algorithms leads to more robust and generalizable evaluations. the underlying mechanism, mima, opens many new avenues for the analysis of selfattention transformers and potentially other models. future work could also focus on optimal weighting between semantics and style.