Paper:
We hypothesize that end-to-end neural image captioning systems work seemingly well because they exploit and learn 'distributional similarity' in a multimodal feature space, by mapping a test image to similar training images in this space and generating a caption from the same space. To validate our hypothesis, we focus on the 'image' side of image captioning, and vary the input image representation but keep the RNN text generation model of a CNN-RNN constant. We propose a sparse bag-of-objects vector as an interpretable representation to investigate our distributional similarity hypothesis. We found that image captioning models (i) are capable of separating structure from noisy input representations; (ii) experience virtually no significant performance loss when a high dimensional representation is compressed to a lower dimensional space; (iii) cluster images with similar visual and linguistic information together; (iv) are heavily reliant on test sets with a similar distribution as the training set; (v) repeatedly generate the same captions by matching images and 'retrieving' a caption in the joint visual-textual space. Our experiments all point to one fact: that our distributional similarity hypothesis holds. We conclude that, regardless of the image representation, image captioning systems seem to match images and generate captions in a learned joint image-text semantic subspace.

Review:
This paper analyzes the effect of image features on image captioning. The authors propose to use a model similar to that of Vinyals et al., 2015 and change the image features it is conditioned on. The MSCOCO captioning and Flickr30K datasets are used for evaluation.
Introduction
- The introduction to the paper could be made clearer - the authors talk about the language of captioning datasets being repetitive, but that fact is neither used or discussed later.
- The introduction also states that the authors will propose ways to improve image captioning. This is never discussed.
Captioning Model and Table 1
- The authors use greedy (argmax) decoding which is known to result in repetitive captions. In fact, Vinyals et al. note this very point in their paper. I understand this design choice was made to focus more on the image side, rather than the decoding (language) side, but I find it to be very limiting. In this regime of greedy decoding it is hard to see any difference between the different ConvNet features used for captioning - Table 1 shows meteor scores within 0.19 - 0.22 for all methods.
- Another effect (possibly due to greedy decoding + choice of model), is that the numbers in Table 1 are rather low compared to the COCO leaderboard. The top 50 entries have METEOR scores >= 0.25, while the maximum METEOR score reported by the authors is 0.22. Similar trend holds for other metrics like BLEU-4.
- The results of Table 5 need to be presented and interpreted in the light of this caveat of greedy decoding.
Experimental Setup and Training Details
- How was the model optimized? No training details are provided. Did you use dropout? Were hyperparamters fixed for training across different feature sizes of VGG19 and ResNet-152? What is the variance in the numbers for Table 1?
Main claim of the paper
Devlin et al., 2015 show a simple nearest neighbor baseline which in my opinion shows this more convincingly. Two more papers from the same group which use also make similar observations - tweaking the image representation makes image captioning better: (1) Fang et al., 2015: Multiple-instance Learning using bag-of-objects helps captioning (2) Misra et al. 2016 (not cited): label noise can be modeled which helps captioning. This claim has been both made and empirically demonstrated earlier.
Metrics for evaluation
- Anderson et al., 2016 (not cited) proposed the SPICE metric and also showed how current metrics including CiDER may not be suitable for evaluating image captions. The COCO leaderboard also uses this metric as one of its evaluation metrics. If the authors are evaluating on the test set and reporting numbers, then it is odd that they `skipped' reporting SPICE numbers.
Choice of Datasets
- If we are thoroughly evaluating the effect of image features, doing so on other datasets is very important. Visual Genome (Krishnan et al., not cited) and SIND (Huang et al., not cited) are two datasets which are both larger than Flickr30k and have different image distributions from MSCOCO. These datasets should show whether using more general features (YOLO-9k) helps.
The authors should evaluate on these datasets to make their findings stronger and more valuable.
Minor comments
- Figure 1 is hard to read on paper. Please improve it.
- Figure 2 is hard to read even on screen. It is really interesting, so improving the quality of this figure will really help.