Validity-Based Sampling and Smoothing Methods for Multiple Reference Image Captioning (2021)

Abstract: In image captioning, multiple captions are often provided as ground truths, since a valid caption is not always uniquely determined. Conventional methods randomly select a single caption and treat it as correct, but there have been few effective training methods that utilize multiple given captions. In this paper, we proposed two training technique for making effective use of multiple reference captions: 1) validity-based caption sampling (VBCS), which prioritizes the use of captions that are estimated to be highly valid during training, and 2) weighted caption smoothing (WCS), which applies smoothing only to the relevant words the reference caption to reflect multiple reference captions simultaneously. Experiments show that our proposed methods improve CIDEr by 2.6 points and BLEU4 by 0.9 points from baseline on the MSCOCO dataset.

## introduction image captioning is a very challenging task that requires recognizing and understanding the objects in the image and then verbalizing the recognition results using natural language. this task is expected to have a wide range of practical applications, including use in text-based image retrieval systems and providing assistance for the visually impaired @xcite . with the development of the field of deep learning, research in the area has primarily focused on the end-to-end method, which consists of an encoder that extracts information from images and a decoder that generates a description from the extracted information @xcite @xcite . for example, some of the recent models use pretrained object detection models @xcite @xcite and selfattention mechanisms @xcite for encoders or decoders. image captioning is often a multi-reference task where multiple reference captions are used for training. mscoco @xcite , one of the most famous datasets of image captions, has about five reference captions for each image. some of these reference captions are subject to uncertainty due to speculation, and may differ in subject matter and wording. such label variance may affect the training of the model and the evaluation of the generated captions. in typical training for conventional models, one caption is randomly selected by uniform sampling at each training epoch, which means the validity and variance of reference captions are not considered. in addition, reference captions that were not selected in the training epoch are treated as incorrect. to address this problem, @xcite proposed a new metric that correlates well with human evaluation by taking into account the variance of captions. however, to the best of our knowledge, appropriate training methods that consider such variation in captions have not yet been sufficiently studied. in this paper, we propose a simple and effective training method that uses multiple reference captions to generate appropriate captions. the proposed training method consists of two techniques: validity-based caption sampling (vbcs), which selects highly valid reference captions, and weighted caption smoothing (wcs), which reflects multiple reference captions simultaneously in training. we define that a highly valid caption has common phrases among reference captions. in vbcs, the validity score for each reference caption is estimated based on similarities among the reference captions. when training the model, the training captions to be used in each epoch are sampled, one per image, according to this score. in addition, wcs improves the generality of the model by applying soft labels only for highly relevant words based on their validity scores. by effectively utilizing multiple captions, the proposed method improves cider by 2.6 points and bleu4 by 0.9 points in the evaluation experiments using the mscoco dataset. main contributions of this paper include: ## conclusion and future works in this paper, we proposed two novel techniques called vbcs and wcs that effectively utilize multiple references in image captioning tasks, and demonstrated their advantages. the former determines a sampling probability (i.e., validity score), for each caption based on similarities among the reference captions. the latter simultaneously reflects multiple reference captions in the training. in the future, we would like to consider the grammar in wcs and, extend the proposed method to be adaptable to reinforcement learning.