Enhancing Descriptive Image Captioning with Natural Language Inference (2021)

Abstract: Generating descriptive sentences that convey non-trivial, detailed, and salient information about images is an important goal of image captioning. In this paper we propose a novel approach to encourage captioning models to produce more detailed captions using natural language inference, based on the motivation that, among different captions of an image, descriptive captions are more likely to entail less descriptive captions. Specifically, we construct directed inference graphs for reference captions based on natural language inference. A PageRank algorithm is then employed to estimate the descriptiveness score of each node. Built on that, we use reference sampling and weighted designated rewards to guide captioning to generate descriptive captions. The results on MSCOCO show that the proposed method outperforms the baselines significantly on a wide range of conventional and descriptiveness-related evaluation metrics.

## introduction automatically generating visually grounded descriptions for given images, a problem known as image captioning @xcite , has drawn extensive attention recently. in spite of the significant improvement of image captioning performance @xcite @xcite , existing models tend to play safe and generate generic captions. however, generating descriptive captions that carry detailed and salient information is an important goal of image captioning. for example, recent work @xcite @xcite ) leveraged crossmodal retrieval @xcite to solve this problem, based on the observation that more descriptive captions often result in better discriminativity in retrieval. in the paper, we explore to develop better descriptive image captioning models from a novel perspective-considering that among different captions of an image, descriptive captions are more likely to entail less descriptive ones, we develop descriptive image captioning models that leverage natural language inference (nli, or also known as recognizing textual entailment) @xcite @xcite , which can utilize multiple references of captions @xcite to guide the models to produce more descriptive captions. specifically, the proposed model first predicts nli relations for all pairs of references, i.e., entailment or neutral foot_1 . built on that, we construct inference graphs and employ a pagerank algorithm to estimate descriptiveness scores for individual captions. we use reference sampling and weighted designated rewards to incorporate the descriptiveness signal into the maximum likelihood estimation and reinforcement learning phase, respectively, to guide captioning models to produce descriptive captions. extensive experiments were conducted on the mscoco dataset using different benchmark baseline methods @xcite @xcite . we demonstrate that the proposed method outperforms the baselines, achieving better performances on various evaluation metrics. in summary, the major contributions of the paper are three-fold: (1) to the best of our knowledge, this is the first attempt to connect natural language inference to image captioning, which helps generate more descriptive captions; (2) we propose a reference sampling distribution and weighted designated rewards to guide captioning model to produce more descriptive captions; (3) the proposed method attains better performance on various evaluation metrics over the state-of-the-art baselines. the work of @xcite ) is most related to ours, which uses retrieval loss as a rewarding signal to encourage descriptive captioning. different from the above approaches, our method explicitly explore the different descriptiveness in references using nli models and incorporate the information into the training objectives to guide the model to generate more informative sentences. we build our method on top of the existing methods to verify the effectiveness. applications of nli there are basically three major application types for nli, (1) direct application of trained nli models. trained nli models are directly used in fact extraction and verification @xcite to decide whether a piece of evidence supports a claim @xcite and generation of longer sentences as a discriminator @xcite to prevent a text decoder from contradicting itself; (2) nli as a research and evaluation task for new methods. it is widely used as a major evaluation when developing novel language model pretraining @xcite @xcite ; (3) nli as a pre-training task in transfer learning. training neural network models on nli corpora and then fine-tuning them on target tasks often yields substantial improvements in performance @xcite . ## our method the goal of image captioning is to train conditional generation model p θ (c | x) based on training instances (x i , c i ) m @xmath0 the typical models leverage a two-phase learning process to estimate p θ (c | x): the first uses mle objective, which minimizes a cross-entropy loss with regard to the ground truth captions: rl is then used to optimize models by maximizing the expected reward for generating captions. in this work, we enhance these two basic learning objectives by considering the descriptiveness of references ## conclusions we explore a novel approach to encourage image captioning models to produce more descriptive sentences using natural language inference. we construct inference graphs and descriptiveness scores are assigned to nodes using the pagerank algorithm. built on that, we use reference sampling and weighted designated rewards to guide captioning to generate descriptive captions. we demonstrate the effectiveness of the model on various evaluation metrics and perform detailed analyses.