Belief Revision Based Caption Re-ranker with Visual Semantic Information (2022)

Abstract: In this work, we focus on improving the captions generated by image-caption generation systems. We propose a novel re-ranking approach that leverages visual-semantic measures to identify the ideal caption that maximally captures the visual information in the image. Our re-ranker utilizes the Belief Revision framework (Blok et al., 2003) to calibrate the original likelihood of the top-n captions by explicitly exploiting semantic relatedness between the depicted caption and the visual context. Our experiments demonstrate the utility of our approach, where we observe that our re-ranker can enhance the performance of a typical image-captioning system without necessity of any additional training or fine-tuning.

## introduction image caption generation is a task that predominantly lies at the intersection of the areas of computer vision and natural language processing. the task is primarily aimed at generating a natural language description for a given image. caption generation systems usually consist of an image encoder that encodes a given image (usually by using a cnn) whose encoding is fed to a decoder (usually by using a generative model such as rnn) to generate a natural language sentence which describes the image succinctly. the most widely used approaches include a cnn-rnn end-to-end system @xcite , end-to-end systems with attention that attend to specific regions of the image for generation @xcite and systems with reinforcement learning based methods @xcite . furthermore, recent advances have resulted in end-to-end systems that use transformer based architecture for language generation and have become the current state-of-the-art 1 https://github.com/ahmedssabir/ belief-revision-score we use the visual context from the image to revise and re-rank the most closely related caption to its visual context. these semantic relatedness measures are learned at the word-to-sentence level. in this example, we showcase our visual re-ranker (visual beam), a post-processing approach, which is able to re-rank the most 'descriptive caption' from the 5-best beam @xcite . @xcite @xcite . while the state-of-the-art models generate captions that are comparable to human level captions, they are known to lack lexical diversity, are often not very distinct, and sound synthetic. we here highlight a few recent approaches that have focused on this problem, these include @xcite that uses generative adversarial networks towards generating diverse and human like captions. @xcite use a beam search with a distractor image to force the model to produce diverse captions by encouraging the models to be discriminative. other recent works use a beam search directly to produce diverse captions by forcing richer lexical word choices @xcite @xcite . in this work, we follow a similar line of research and focus on the problem of improving diversity and making captions natural and human like and propose a novel re-ranking approach. in this approach, we use n-best reranking with a given beam that explicitly uses the semantic correlation between the caption and the visual context through belief revision (an approach inspired by human logic). we refer the reader to figure 1 , where the approach results in a caption that is a) visually relevant and b) the most natural and human like. our primary contributions in this paper are: ## belief revision with simprob model in this section, we briefly introduce simprob, which is based on the philosophical intuitions of belief revision, an idea that helps to convert similarity measures to probability estimates. @xcite introduce a conditional probability model that assumes that the preliminary probability result is updated or revised to the degree that the hypothesis proof warrants. the range of revision is based on the informativeness of the argument and its degree of similarity. that is, the similarity to probability conversion can be defined in terms of belief revision. belief revision is a process of forming a belief by taking into account a new piece of information. let us consider the following statements: in the first case, the statement seems logical because it matches our prior belief i.e. jaguars are similar to tigers, so we expect them to be able to do similar things. we hence consider that the statement is consistent with our previous belief, and there is no need to revise it. in the second case, the statement is surprising because our prior belief is that kittens are not as similar to jaguars, and thus, not so strong. but if we assume the veracity of the statement, then we need to revise and update our prior belief about the strength of kittens. this work formalizes belief as probabilities and revised belief as conditional probabilities and provides a framework to compute them based on the similarities of the involved objects. according to the authors, belief revision should be proportional to the similarity of the involved objects (i.e. in the example, the statement about kittens and jaguars would cause a stronger belief revision than e.g. the same statement involving pigeons and jaguars because they are less similar). in our case, we use the same rationale and the same formulas to convert similarity (or relatedness) scores into probabilities suitable for reranking. simprob model to obtain the likelihood revisions based on similarity scores, we need three parameters: (1) hypothesis: prior probabilities, (2) informativeness: conclusion events and (3) similarities: measuring the relatedness between involved categories.the goal is to predict a conditional probability of statements, given one or more other statements. in order to predict the conditional probability of the argument's conclusion, given its premise or hypothesis, we will need only the prior probabilities of the statements, as well as the similarities between the involved categories (e.g. kittens and tigers). ## evaluation of diversity we follow the standard diversity evaluation metrics @xcite : (1) div-1 the ratio of unique unigram to the number of words in caption (2) div-2 the ratio of unique bi-gram to the number of words in the caption, (3) mbleu is the bleu score between the candidate caption against all human captions (lower value indicate diversity). however, since even though we obtained the top-20 candidates from the base systems, many of them are the same or have very small differences (beam search drawback), which will reflect in small performance differences before and after re-ranking. therefore, some of the standard metrics are not able to capture these small changes, as shown in for semantic diversity, we use the standard metric (wang and chan, 2019) self-cider. also, inspired by bertscore and following @xcite that introduce sentence semantic semantic (sss) for machine translation, we use sbertsts (fine-tuned on sts task) with a cosine score to measure the sentence level semantic correlation against all human references. we observe that the sbert-sts capture the semantic content better than self-cider. ## human evaluation we conducted a human study to investigate human preferences over the visual re-ranked caption. we randomly select 26 test images and give 12 reliable human subjects the option to choose between two captions: (1) best-beam (beams) 5 and (2) visual r-ranker. we obtain mixed results, as some reranked captions are grammatically incorrect, such as singulars instead of plurals and sitting on for objects instead of subjects. overall, we can observe that 46% of native speakers agreed with our visual re-ranker. meanwhile, the result for non-native speakers is 61%. in some details, we observe that our model and the non-native human subjects chose those re-ranked captions because they correlated more closely with the visual information regarding 5 the best result by the baseline in standard metrics. the grammatical error in the sentence and unlike the native speaker. ## ablation study belief revision relies on a different block (i.e. lm, similarity and visual context) to make the final revision. in this study, we perform an ablation study over a random 100 samples from the test set to investigate the effectiveness of the proposed setup. ## discussion: limitations object classifier failure cases: as the belief revision approach relies heavily on the object in the image for the likelihood revision, the quality of the object classifier is critical for the final decision. here, we show some failure cases when the visual classifier struggle with complex background (i.e. wrong visual, object hallucination, etc.) as shown in figure 5 . note that, if no related visual is present in the image the belief revision score will back off to 1 (no revision needed). object-to-caption similarity score: another limitation is the low/high cosine similarity score, which unbalances the likelihood revision. for example, a visual context paddle and the caption: a man riding a surfboard on a wave have low cosine scores when using a pre-trained model that is finetuned on sentence-to-sentence semantic similarity tasks (i.e. sts-b). note that, we tackle this problem by adding multiple visual contexts as shown in figure 2 .