Original: W HAT ARE IMAGE CAPTIONS MADE OF ?
Revised: W HAT IS IMAGE CAPTIONING MADE OF ?

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We hypothesize that end-to-end neural image captioning systems work seemingly well because they exploit and learn ‘distributional similarity’ in a multimodal feature space, by mapping a test image to similar training images in this space and generating a caption from the same space.
Revised: We hypothesize that end-to-end neural image captioning systems work seemingly well because they exploit and learn ‘distributional similarity’ in a multimodal feature space, by mapping a test image to similar training images in this space and generating a caption from the same space.

Original: To validate our hypothesis, we focus on the ‘image’ side of image captioning, and vary the input image representation but keep the RNN text generation model of a CNN-RNN constant.
Revised: To validate our hypothesis, we focus on the ‘image’ side of image captioning, and vary the input image representation but keep the RNN text generation model of a CNN-RNN constant.

Original: We propose a sparse bag-of-objects vector as an interpretable representation to investigate our distributional similarity hypothesis.
Revised: We propose a sparse bag-of-objects vector as an interpretable representation to investigate our distributional similarity hypothesis.

Original: We found that image captioning models (i) are capable of separating structure from noisy input representations; (ii) experience virtually no signiﬁcant performance loss when a high dimensional representation is compressed to a lower dimensional space; (iii) cluster images with similar visual and linguistic information together; (iv) are heavily reliant on test sets with a similar distribution as the training set; (v) repeatedly generate the same captions by matching images and ‘retrieving’ a caption in the joint visual-textual space.
Revised: We found that image captioning models (i) are capable of separating structure from noisy input representations; (ii) experience virtually no signiﬁcant performance loss when a high dimensional representation is compressed to a lower dimensional space; (iii) cluster images with similar visual and linguistic information together; (iv) are heavily reliant on test sets with a similar distribution as the training set; (v) repeatedly generate the same captions by matching images and ‘retrieving’ a caption in the joint visual-textual space.

Original: Our experiments all point to one fact: that our distributional similarity hypothesis holds.
Revised: Our experiments all point to one fact: that our distributional similarity hypothesis holds.

Original: We conclude that, regardless of the image representation, image captioning systems seem to match images and generate captions in a learned joint image-text semantic subspace.
Revised: We conclude that, regardless of the image representation, image captioning systems seem to match images and generate captions in a learned joint image-text semantic subspace.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Image description generation, or image captioning (IC), is the task of automatically generating a textual description for a given image.
Revised: Image description generation, or image captioning (IC), is the task of automatically generating a textual description for a given image.

Original: The generated text is expected to describe, in a single sentence, what is visually depicted in the image, for example the entities/objects present in the image, their attributes, the actions/activities performed, entity/object interactions (including quantiﬁcation), the location/scene, etc.
Revised: The generated text is expected to describe, in a single sentence, what is visually depicted in the image, for example the entities/objects present in the image, their attributes, the actions/activities performed, entity/object interactions (including quantiﬁcation), the location/scene, etc.

Original: (e.g. “ a man riding a bike on the street ”).
Revised: (e.g. “ a man riding a bike on the street ”).

Original: Signiﬁcant progress has been made with end-to-end approaches to tackling this problem, where large-scale parallel image–description datasets such as Flickr30k (Young et al., 2014) and
Revised: Signiﬁcant progress has been made with end-to-end approaches to tackling this problem, where large-scale parallel image–description datasets such as Flickr30k (Young et al., 2014) and

Original: MSCOCO (Chen et al., 2015b) are used to train a CNN-RNN based neural network IC system (Vinyals et al., 2016; Karpathy & Fei-Fei, 2015; Xu et al., 2015).
Revised: MSCOCO (Chen et al., 2015b) are used to train a CNN-RNN based neural network IC system (Vinyals et al., 2016; Karpathy & Fei-Fei, 2015; Xu et al., 2015).

Original: Such systems have demonstrated impressive performance in the COCO captioning challenge 1 according to automatic metrics, seemingly even surpassing human performance in many instances (e.g. CIDEr score > 1 . 0 vs. human’s 0.85)
Revised: Such systems have demonstrated impressive performance in the COCO captioning challenge 1 according to automatic metrics, seemingly even surpassing human performance in many instances (e.g. CIDEr score > 1 . 0 vs. human’s 0.85)

Original: (Chen et al., 2015a).
Revised: (Chen et al., 2015a).

Original: However, in reality, the performance of end-to-end systems is still far from satisfactory according to metrics based on human judgement 2 .
Revised: However, in reality, the performance of end-to-end systems is still far from satisfactory according to metrics based on human judgement 2 .

Original: Thus, despite the progress, this task is currently far from being a solved problem.
Revised: Thus, despite the progress, this task is currently far from being a solved problem.

Original: In this paper, we challenge the common assumption that end-to-end IC systems are able to achieve strong performance because they have learned to ‘understand’ and infer semantic information from visual representations, i.e. they can for example deduce that “ a boy is playing football ” purely by learning directly from mid-level image features and the corresponding textual descriptions in an implicit manner, without explicitly modeling the presence of boy , ball , green ﬁeld , etc.
Revised: In this paper, we challenge the common assumption that end-to-end IC systems are able to achieve strong performance because they have learned to ‘understand’ and infer semantic information from visual representations, i.e. they can for example deduce that “ a boy is playing football ” purely by learning directly from mid-level image features and the corresponding textual descriptions in an implicit manner, without explicitly modeling the presence of boy , ball , green ﬁeld , etc.

Original: in the image.
Revised: in the image.

Original: It is believed that the IC system has managed to infer that the phrase green ﬁeld is associated with some ‘green-like’ area in the image and is thus generated in the output description, or that the word boy is generated because of some CNN activations corresponding to a young person.
Revised: It is believed that the IC system has managed to infer that the phrase green ﬁeld is associated with some ‘green-like’ area in the image and is thus generated in the output description, or that the word boy is generated because of some CNN activations corresponding to a young person.

Original: However, there seems to be no concrete evidence that this is the case.
Revised: However, there seems to be no concrete evidence that this is the case.

Original: Instead, we hypothesize that the apparently strong performance of end-to-end systems is attributed to the fact that they are exploiting the distributional similarity in the multimodal feature space.
Revised: Instead, we hypothesize that the apparently strong performance of end-to-end systems is attributed to the fact that they are exploiting the distributional similarity in the multimodal feature space.

Original: To our best knowledge, our paper gives the ﬁrst empirical analysis on visual representations for the task of image captioning.
Revised: To our best knowledge, our paper gives the ﬁrst empirical analysis on visual representations for the task of image captioning.

Original: What we mean by ‘distributional similarity’ is that IC systems essentially attempt to match images from the training set that is most similar to a test image, and generate a caption from the most similar training instances (or generate a ‘novel’ description from a combination of training instances, for example by ‘averaging’ the descriptions).
Revised: What we mean by ‘distributional similarity’ is that IC systems essentially attempt to match images from the training set that is most similar to a test image, and generate a caption from the most similar training instances (or generate a ‘novel’ description from a combination of training instances, for example by ‘averaging’ the descriptions).

Original: Previous work has alluded to this observation (Karpathy, 2016; Vinyals et al., 2016), but it has not been thoroughly investigated.
Revised: Previous work has alluded to this observation (Karpathy, 2016; Vinyals et al., 2016), but it has not been thoroughly investigated.

Original: This phenomena could also be in part attributed to the fact that the datasets are repetitive and simplistic, with an almost constant and predictable linguistic structure (Lebret et al., 2015; Devlin et al., 2015; Vinyals et al., 2016).
Revised: This phenomena could also be in part attributed to the fact that the datasets are repetitive and simplistic, with an almost constant and predictable linguistic structure (Lebret et al., 2015; Devlin et al., 2015; Vinyals et al., 2016).

Original: In this paper we investigate the hypothesis of distributional similarity in IC by focusing on the image side of image captioning.
Revised: In this paper we investigate the hypothesis of distributional similarity in IC by focusing on the image side of image captioning.

Original: Most previous work has concentrated on the text side of image captioning, e.g. by optimizing the language modelling capabilities of the RNN (Rennie et al., 2016; Liu et al., 2017) to improve its performance on automatic metrics.
Revised: Most previous work has concentrated on the text side of image captioning, e.g. by optimizing the language modelling capabilities of the RNN (Rennie et al., 2016; Liu et al., 2017) to improve its performance on automatic metrics.

Original: While there are efforts at improving image captioning by utilizing or modeling images more effectively, for example by using attention over mid-level image features (Xu et al., 2015) and high-level object proposals (Anderson et al., 2017),however, in this work we are speciﬁcally interested in interpretability and we focus on using a simpler (and faster) model for empirical evaluation.
Revised: While there have been efforts on improving IC by utilizing or modeling images more effectively, for example by using attention over mid-level image features (Xu et al., 2015) and high-level object proposals (Anderson et al., 2017), in this work we are speciﬁcally interested in interpretability and we focus on using a simpler (and faster) model for empirical evaluation.

Original: We explore the basic yet effective CNN-RNN model (Karpathy & Fei-Fei, 2015), and investigate the representational contributions while keeping the RNN generator constant.
Revised: We explore the basic yet effective CNN-RNN model (Karpathy & Fei-Fei, 2015), and investigate the representational contributions while keeping the RNN generator constant.

Original: More advanced models can be considered speciﬁc variants of Karpathy & Fei-Fei (2015).
Revised: More advanced models can be considered speciﬁc variants of Karpathy & Fei-Fei (2015).

Original: It is worth noting that we are interested in demonstrating the phenomenon of distributional similarity in IC, rather than achieving or improving state-of-the-art performance, As such, we do not resort to ﬁne-tuning or extensive hyperparameter optimization or ensembles.
Revised: It is worth noting that we are interested in demonstrating the phenomenon of distributional similarity in IC, rather than achieving or improving state-of-the-art performance, As such, we do not resort to ﬁne-tuning or extensive hyperparameter optimization or ensembles.

Original: Therefore, our model is not comparable to state-of-the-art models such as Vinyals et al.
Revised: Therefore, our model is not comparable to state-of-the-art models such as Vinyals et al.

Original: (2016), which optimizes IC by ﬁne-tuning the image representations, exploring beam size, scheduled sampling, and using ensemble models.
Revised: (2016), which optimize IC by ﬁne-tuning the image representations, exploring beam size, scheduled sampling, and using ensemble models.

Original: Instead, we vary only the image representation to demonstrate that end-to-end IC systems utilize distributional similarity on the image side to generate captions, regardless of the image representation used.
Revised: Instead, we vary only the image representation to demonstrate that end-to-end IC systems utilize distributional similarity on the image side to generate captions, regardless of the image representation used.

Original: Our main contributions are
Revised: Our main contributions are:

Original: : • an IC experiment where we vary the input image representation but keep the RNN text generation model constant (Section 3).
Revised: • An IC experiment where we vary the input image representation but keep the RNN text generation model constant (Section 3).

Original: This experiment demonstrates that regardless of  image representation (a continuous image embedding or a sparse, low-dimensional vector), end-to-end IC systems seem to utilize a visual-semantic subspace for IC.
Revised: This experiment demonstrates that regardless of the image representation (a continuous image embedding or a sparse, low-dimensional vector), end-to-end IC systems seem to utilize a visual-semantic subspace for IC.

Original: • the introduction of a simple, sparse bag-of-objects representation that contains information about the presence of objects in the images.
Revised: • The introduction of a simple, sparse bag-of-objects representation that contains information about the presence of objects in the images.

Original: We use thisspeciﬁcally as a tool to investigate the contribution of images in the image captioning framework.
Revised: We use this as a tool to investigate the contribution of images in the image captioning framework.

Original: • the introduction of pseudo-random vectors derived from object-level representations as a means to evaluate IC systems.
Revised: • The introduction of pseudo-random vectors derived from object-level representations as a means to evaluate IC systems.

Original: Our results show that end-to-end models in this framework are remarkably capable of separating structure from noisy input representations.
Revised: Our results show that end-to-end models in this framework are remarkably capable of separating structure from noisy input representations.

Original: • an experiment where IC models are conditioned on image representations factorized and compresssed to a lower dimensional space (Section 4.1).
Revised: • An experiment where IC models are conditioned on image representations factorized and compresssed to a lower dimensional space (Section 4.1).

Original: We show that high dimensional image embeddings that are factorized to a lower dimensional representation and used as input to an IC model result in virtually no signiﬁcant loss in performance, further strengthening our claim that IC models perform similarity matching rather than image understanding.
Revised: We show that high dimensional image embeddings that are factorized to a lower dimensional representation and used as input to an IC model result in virtually no signiﬁcant loss in performance, further strengthening our claim that IC models perform similarity matching rather than image understanding.

Original: • an analysis of different image representations and their transformed representations (Sections 4.2 and 4.3).
Revised: • An analysis of different image representations and their transformed representations (Sections 4.2 and 4.3).

Original: We visualize the initial visual subspace and the learned joint visual semantic subspace and observe that the visual semantic subspace has learned to cluster images with similar visual and linguistic information together, further validating our claims of distributional similarity.
Revised: We visualize the initial visual subspace and the learned joint visual semantic subspace and observe that the visual semantic subspace has learned to cluster images with similar visual and linguistic information together, further validating our claims of distributional similarity.

Original: • an experiment where the IC model is tested on an out-of-domain dataset (Section 4.4), which has a slightly different image distribution.
Revised: • An experiment where the IC model is tested on an out-of-domain dataset (Section 4.4), which has a slightly different image distribution.

Original: We observe that models, including the state-of-the-art models, show a better performance on test sets that have a similar distribution as the training.
Revised: We observe that models, including the state-of-the-art models, show a better performance on test sets that have a similar distribution as the training.

Original: However, their performance deteriorates when the distributions are slightly different.
Revised: However, their performance deteriorates when the distributions are slightly different.

Original: • an analysis on the uniqueness of captions generated by IC models using different image representations (Section 4.5).
Revised: • An analysis on the uniqueness of captions generated by IC models using different image representations (Section 4.5).

Original: We hypothesize that the captions are often repeated as they are usually generated by matching images in the joint space and retrieving a relevant caption.
Revised: We hypothesize that the captions are often repeated as they are usually generated by matching images in the joint space and retrieving a relevant caption.

Original: We perform experimentsto validate our claim.
Revised: Our experiments validate this claim.

Original: Overall, the study suggests that regardless of the representation used, end-to-end IC models implicitly learn and exploit multimodal similarity spaces rather than performing actual image understanding.
Revised: Overall, the study suggests that regardless of the representation used, end-to-end IC models implicitly learn and exploit multimodal similarity spaces rather than performing actual image understanding.

Original: This study is in line with the recent work that explore understanding of deep learning models and the representational interpretations (Papernot et al., 2016; Szegedy et al., 2013; Sturm, 2014) and works that have tried to delve into the image captioning task (Devlin et al., 2015; Vinyals et al., 2016).
Revised: This study is in line with the recent work that explore understanding of deep learning models and the representational interpretations (Papernot et al., 2016; Szegedy et al., 2013; Sturm, 2014) and works that have tried to delve into the image captioning task (Devlin et al., 2015; Vinyals et al., 2016).

Original: To the best of our knowledge, this work isamong the ﬁrst works that tries to investigate the image captioning model that is explicitly focused on image representations and their effects.
Revised: To the best of our knowledge, ours is the ﬁrst work that investigates IC focusing speciﬁcally on image representations and their effects.

Original:  M ODEL S ETTING
Revised:  M ODEL S ETTING

Original: For the experiments in Section 3, we base our implementation on a simple end-to-end approach by Karpathy & Fei-Fei (2015).
Revised: For the experiments in Section 3, we base our implementation on a simple end-to-end approach by Karpathy & Fei-Fei (2015).

Original: We use the LSTM (Hochreiter & Schmidhuber, 1997) based language model as described in Zaremba et al.
Revised: We use the LSTM (Hochreiter & Schmidhuber, 1997) based language model as described in Zaremba et al.

Original: To condition the image information, we ﬁrst perform a linear projection of the image representation followed by a non-linearity:
Revised: To condition the image information, we ﬁrst perform a linear projection of the image representation followed by a non-linearity:

Original: [Equation]
Revised: [Equation]

Original: Here, I m ∈ R
Revised: Here, I m ∈ R

Original: d is the d -dimensional initial image representation, W ∈ R n × d is the linear transformation matrix, σ is the non-linearity.
Revised: d is the d -dimensional initial image representation, W ∈ R n × d is the linear transformation matrix, σ is the non-linearity.

Original: We use Exponential Linear Units (Clevert et al., 2015) as the non-linear activation in all our experiments.
Revised: We use Exponential Linear Units (Clevert et al., 2015) as the non-linear activation in all our experiments.

Original: Following Vinyals et al.
Revised: Following Vinyals et al.

Original: (2015), we initialize the LSTM based caption generator with the projected image feature.
Revised: (2015), we initialize the LSTM based caption generator with the projected image feature.

Original: Training and Inference
Revised: Training and Inference

Original: The caption generator is trained to generate sentences conditioned on the image representation.
Revised: The caption generator is trained to generate sentences conditioned on the image representation.

Original: We train the model by minimizing the cross-entropy, i.e., the sentence-level loss corresponds to the sum of the negative log likelihood of the correct word at each time step:
Revised: We train the model by minimizing the cross-entropy, i.e., the sentence-level loss corresponds to the sum of the negative log likelihood of the correct word at each time step:

Original: [Equation]
Revised: [Equation]

Original: where Pr ( S | Im feat ; θ ) is the sentence-level loss conditioned on the image feature Im feat
Revised: where Pr ( S | Im feat ; θ ) is the sentence-level loss conditioned on the image feature Im feat

Original: and
Revised: and

Original: Pr( w t ) is the probability of the word at time step t .
Revised: Pr( w t ) is the probability of the word at time step t .

Original: This is trained with standard teacher forcing as described in Sutskever et al.
Revised: This is trained with standard teacher forcing as described in Sutskever et al.

Original: (2014) where the correct word information is fed to the next state in the
Revised: (2014) where the correct word information is fed to the next state in the

Original: LSTM.
Revised: LSTM.

Original: Inference is typically performed with approximation techniques like beam search or sampling (Karpathy & Fei-Fei, 2015; Vinyals et al., 2015).
Revised: Inference is typically performed with approximation techniques like beam search or sampling (Karpathy & Fei-Fei, 2015; Vinyals et al., 2015).

Original: In this paper, as we are mainly interested in the studying effect of different image representations, we focus on the language output that the models can most conﬁdently produce.
Revised: In this paper, as we are mainly interested in the studying effect of different image representations, we focus on the language output that the models can most conﬁdently produce.

Original: Therefore, in order to isolate any other variables from the experiments, we generate captions using a greedy argmax based approach for consistency (unless stated otherwise, we always use greedy decoding).
Revised: Therefore, in order to isolate any other variables from the experiments, we generate captions using a greedy argmax based approach for consistency (unless stated otherwise, we always use greedy decoding).

Original:  I MAGE CAPTIONING WITH DIFFERENT IMAGE REPRESENTATIONS
Revised:  I MAGE CAPTIONING WITH DIFFERENT IMAGE REPRESENTATIONS

Original: In this section, we verify our hypothesis that a ‘distributional similarity’ space exist in end-to-end
Revised: In this section, we verify our hypothesis that a ‘distributional similarity’ space exist in end-to-end

Original: IC systems.
Revised: IC systems.

Original: Such systems attempt to match image representations in order to condition the RNN decoder to generate captions that are similar to the closest images, rather than actually understanding the image in order to describe the image.
Revised: Such systems attempt to match image representations in order to condition the RNN decoder to generate captions that are similar to the closest images, rather than actually understanding the image in order to describe the image.

Original: We keep the IC model constant (Section 2) across experiments, and vary only the image representation used.
Revised: We keep the IC model constant (Section 2) across experiments, and vary only the image representation used.

Original: 3. I MAGE R EPRESENTATIONS 3.1. L OWER - BOUND REPRESENTATION
Revised: 3. I MAGE R EPRESENTATIONS 3.1. L OWER - BOUND REPRESENTATION

Original: Random: We condition the LSTM with a 300-dimensional vector comprising random values sampled uniformly between [0 , 1) 3 .
Revised: Random: We condition the LSTM with a 300-dimensional vector comprising random values sampled uniformly between [0 , 1) 3 .

Original: This feature essentially gives us a worst case image feature and provides an artiﬁcial lower bound.
Revised: This feature essentially gives us a worst case image feature and provides an artiﬁcial lower bound.

Original: 3.1. R EPRESENTATIONS FROM IMAGE - LEVEL CLASSIFICATION
Revised: 3.1. R EPRESENTATIONS FROM IMAGE - LEVEL CLASSIFICATION

Original: The following pre-trained CNNs are used:
Revised: The following pre-trained CNNs are used:

Original: • VGG19 (Simonyan & Zisserman, 2015) pre-trained on ILSVRC (
Revised: • VGG19 (Simonyan & Zisserman, 2015) pre-trained on ILSVRC (

Original: Russakovsky
Revised: Russakovsky

Original: et al.
Revised: et al.

Original: • ResNet

Revised: • ResNet

Original: (He et al., 2016) also pre-trained on ILSVRC.
Revised: (He et al., 2016) also pre-trained on ILSVRC.

Original: • Places365-ResNet152 (Zhou et al., 2014), a variant of ResNet152 pre-trained on the Places dataset (Zhou et al., 2017).
Revised: • Places365-ResNet152 (Zhou et al., 2014), a variant of ResNet152 pre-trained on the Places dataset (Zhou et al., 2017).

Original: We investigate whether scene-speciﬁc categories are useful for IC without the network being trained to classify object-speciﬁc categories.
Revised: We investigate whether scene-speciﬁc categories are useful for IC without the network being trained to classify object-speciﬁc categories.

Original: • Hybrid1365-ResNet

Revised: • Hybrid1365-ResNet

Original: (Zhou et al., 2014), a ResNet152 variant trained on the concatenation of the ILSVRC and Places2 datasets and predicts both object and scene classes.
Revised: (Zhou et al., 2014), a ResNet152 variant trained on the concatenation of the ILSVRC and Places2 datasets and predicts both object and scene classes.

Original: We explore various representations derived from the CNNs above:
Revised: We explore various representations derived from the CNNs above:

Original: Penultimate layer ( Penultimate ): Most previous attempts for IC use the output of the penultimate layer of a CNN pre-trained on ILSVRC.
Revised: Penultimate layer ( Penultimate ): Most previous attempts for IC use the output of the penultimate layer of a CNN pre-trained on ILSVRC.

Original: Previous work motivates using ‘off-the-shelf’ feature extractors in the framework of transfer learning (Razavian et al., 2014; Donahue et al., 2014).
Revised: Previous work motivates using ‘off-the-shelf’ feature extractors in the framework of transfer learning (Razavian et al., 2014; Donahue et al., 2014).

Original: Such features have often been applied to image captioning (Mao et al., 2014; Karpathy & Fei-Fei, 2015;
Revised: Such features have often been applied to image captioning (Mao et al., 2014; Karpathy & Fei-Fei, 2015;

Original: Xu et al., 2015; Gao et al., 2015; Vinyals et al., 2015; Donahue et al., 2015) and have been shown to produce state-of-the-art results.
Revised: Xu et al., 2015; Gao et al., 2015; Vinyals et al., 2015; Donahue et al., 2015) and have been shown to produce state-of-the-art results.

Original: Therefore, for each image, we extract the fc7 layer of VGG ( 4096 D ) and the pool5 layer for the ResNet152 variants ( 2048 D ) .
Revised: Therefore, for each image, we extract the fc7 layer of VGG ( 4096 D ) and the pool5 layer for the ResNet152 variants ( 2048 D ) .

Original: Class prediction vector ( Softmax ): We also investigate higher-level image representations, where each element in the vector is an estimated posterior probability of object categories.
Revised: Class prediction vector ( Softmax ): We also investigate higher-level image representations, where each element in the vector is an estimated posterior probability of object categories.

Original: Note that the categories may not directly correspond to the captions in the dataset.
Revised: Note that the categories may not directly correspond to the captions in the dataset.

Original: While there are alternative methods that ﬁne-tune the image network on a new set of object classes extracted in ways that are directly relevant to the captions (Fang et al., 2015; Wu et al., 2016), we study the impact of off-the-shelf prediction vectors on the IC task.
Revised: While there are alternative methods that ﬁne-tune the image network on a new set of object classes extracted in ways that are directly relevant to the captions (Fang et al., 2015; Wu et al., 2016), we study the impact of off-the-shelf prediction vectors on the IC task.

Original: The intuition is that category predictions from pretrained CNN classiﬁers may also be beneﬁcial for IC, alongside the standard approach of using midlevel features from the penultimate layer.
Revised: The intuition is that category predictions from pretrained CNN classiﬁers may also be beneﬁcial for IC, alongside the standard approach of using midlevel features from the penultimate layer.

Original: Therefore, for each image, we use the predicted category posterior distributions of VGG19 and ResNet152 for 1000 object categories), Places365-ResNet ( 365 scene categories), and Hybrid-ResNet152 ( 1365 object and scene categories).
Revised: Therefore, for each image, we use the predicted category posterior distributions of VGG19 and ResNet152 for 1000 object categories), Places365-ResNet ( 365 scene categories), and Hybrid-ResNet152 ( 1365 object and scene categories).

Original: Object class word embeddings ( Top- k ):
Revised: Object class word embeddings ( Top- k ):

Original: Here we experiment with a method that utilizes the averaged word representations of top- k predicted object classes.
Revised: Here we experiment with a method that utilizes the averaged word representations of top- k predicted object classes.

Original: We ﬁrst obtain Softmax predictions using ResNet152 for 1000 object categories (synsets) per image.
Revised: We ﬁrst obtain Softmax predictions using ResNet152 for 1000 object categories (synsets) per image.

Original: We then select the objects that have a posterior probability score > 5% and use the 300 -dimensional pre-trained word2vec (Mikolov et al., 2013) representations 4 to obtain the averaged vector over all retained object categories.
Revised: We then select the objects that have a posterior probability score > 5% and use the 300 -dimensional pre-trained word2vec (Mikolov et al., 2013) representations 4 to obtain the averaged vector over all retained object categories.

Original: This is motivated by the central observation that averaged word embeddings can represent semantic-level properties and are useful for classiﬁcation tasks (Arora et al., 2016).
Revised: This is motivated by the central observation that averaged word embeddings can represent semantic-level properties and are useful for classiﬁcation tasks (Arora et al., 2016).

Original: 3.1. R EPRESENTATIONS FROM OBJECT - LEVEL DETECTIONS
Revised: 3.1. R EPRESENTATIONS FROM OBJECT - LEVEL DETECTIONS

Original: We also explore representing images using information from object detectors that identiﬁes instances of object categories present in an image, rather than a global, image-level classiﬁcation.
Revised: We also explore representing images using information from object detectors that identiﬁes instances of object categories present in an image, rather than a global, image-level classiﬁcation.

Original: This can potentially provide for a richer and more informative image representation.
Revised: This can potentially provide for a richer and more informative image representation.

Original: For this we use:
Revised: For this we use:

Original: • ground truth ( Gold ) region annotations for instances of 80 pre-deﬁned categories provided with MSCOCO.
Revised: • ground truth ( Gold ) region annotations for instances of 80 pre-deﬁned categories provided with MSCOCO.

Original: It is worth noting that these were annotated independently of the image
Revised: It is worth noting that these were annotated independently of the image

Original: captions,
Revised: captions,

Original: i
Revised: i

Original: e. people writing the captions had no knowledge of the 80 categories and the annotations.
Revised: e. people writing the captions had no knowledge of the 80 categories and the annotations.

Original: As such, there is no direct correspondence between the region annotations and image captions.
Revised: As such, there is no direct correspondence between the region annotations and image captions.

Original: • a state-to-the-art object detector YOLO (Redmon & Farhadi, 2016), pre-trained on MSCOCO for 80 categories ( YOLO-Coco ), and on MSCOCO and ILSVRC for overcategories ( YOLO-9k ).
Revised: • a state-to-the-art object detector YOLO (Redmon & Farhadi, 2016), pre-trained on MSCOCO for 80 categories ( YOLO-Coco ), and on MSCOCO and ILSVRC for overcategories ( YOLO-9k ).

Original: We explore several representations derived from instance-level object class annotations/detectors above:
Revised: We explore several representations derived from instance-level object class annotations/detectors above:

Original: Bag of objects ( BOO ): We represent each image as a sparse ‘bag of objects’ vector, where each element represents the frequency of occurrence for each object category in the image ( Counts ).
Revised: Bag of objects ( BOO ): We represent each image as a sparse ‘bag of objects’ vector, where each element represents the frequency of occurrence for each object category in the image ( Counts ).

Original: We also explore an alternative representation where we only encode the presence or absence of the object category regardless of its frequency ( Binary ), to determine whether it is important to encode object counts in the image.
Revised: We also explore an alternative representation where we only encode the presence or absence of the object category regardless of its frequency ( Binary ), to determine whether it is important to encode object counts in the image.

Original: These representations help us examine the importance of explicit object categories and in a sense interactions between object categories ( dog and ball ) in the image representation.
Revised: These representations help us examine the importance of explicit object categories and in a sense interactions between object categories ( dog and ball ) in the image representation.

Original: We investigate whether such a sparse and high-level BOO representation is helpful for IC.
Revised: We investigate whether such a sparse and high-level BOO representation is helpful for IC.

Original: It is also worth noting that BOO is different from the Softmax representation above as it encodes the number of object occurrences, not the conﬁdence of class predictions at image level.
Revised: It is also worth noting that BOO is different from the Softmax representation above as it encodes the number of object occurrences, not the conﬁdence of class predictions at image level.

Original: We compare BOO representations derived from the Gold annotations ( Gold-Binary and Gold-Counts ) and both YOLO-Coco and YOLO-9k detectors ( Counts only).
Revised: We compare BOO representations derived from the Gold annotations ( Gold-Binary and Gold-Counts ) and both YOLO-Coco and YOLO-9k detectors ( Counts only).

Original: Pseudo-random vectors: To further probe the capacity of the model to discern image representations in an image distributional similarity space, we propose a novel experiment where we examine a type of representation where similar images are represented using similar random vectors , which we term as pseudo-random vectors .
Revised: Pseudo-random vectors: To further probe the capacity of the model to discern image representations in an image distributional similarity space, we propose a novel experiment where we examine a type of representation where similar images are represented using similar random vectors , which we term as pseudo-random vectors .

Original: We form this representation from BOO Gold-Counts and BOO
Revised: We form this representation from BOO Gold-Counts and BOO

Original: Gold-Binary .
Revised: Gold-Binary .

Original: Formally, Im feat =
Revised: Formally, Im feat =

Original: (cid:80) o ∈ Objects
Revised: (cid:80) o ∈ Objects

Original: f
Revised: f

Original: × φ o , where φ o ∈ R d is an object-speciﬁc random vector and f is a scalar representing counts of the object category.
Revised: × φ o , where φ o ∈ R d is an object-speciﬁc random vector and f is a scalar representing counts of the object category.

Original: In the case of PseudorandomCounts , f is the frequency counts from Gold-Counts .
Revised: In the case of PseudorandomCounts , f is the frequency counts from Gold-Counts .

Original: In the case of Pseudorandom-Binary , f is either 0 or 1 based on Gold-Binary .
Revised: In the case of Pseudorandom-Binary , f is either 0 or 1 based on Gold-Binary .

Original: We use d = 120 for these experiments.
Revised: We use d = 120 for these experiments.

Original: 3. D ATASETS AND EXPERIMENTAL SETUP
Revised: 3. D ATASETS AND EXPERIMENTAL SETUP

Original: Dataset We evaluate image captioning conditioned on different representations on the most widely used dataset for IC, MSCOCO (Chen et al., 2015b).
Revised: Dataset We evaluate image captioning conditioned on different representations on the most widely used dataset for IC, MSCOCO (Chen et al., 2015b).

Original: The dataset consists of 82 , 783 images for training, with at least ﬁve captions per image, totaling to 413 , 915 captions.
Revised: The dataset consists of 82 , 783 images for training, with at least ﬁve captions per image, totaling to 413 , 915 captions.

Original: We perform model selection on a 5000 -image development set and report the results on a 5000 -image test set using standard, publicly available splits 5 of the MSCOCO validation dataset as in previous work (Karpathy & Fei-Fei, 2015).
Revised: We perform model selection on a 5000 -image development set and report the results on a 5000 -image test set using standard, publicly available splits 5 of the MSCOCO validation dataset as in previous work (Karpathy & Fei-Fei, 2015).

Original: Evaluation Metrics We evaluated system outputs using the standard evaluation metrics for image captioning using the most common metrics: BLEU (Papineni et al., 2002) which is computed from 1 -gram to 4 -gram precision scores (B- 1 · · · B- 4 ), Meteor (Denkowski & Lavie, 2014) (M) and CIDEr (Vedantam et al., 2015) (C) and SPICE (Anderson et al., 2016) (S).
Revised: Evaluation Metrics We evaluated system outputs using the standard evaluation metrics for image captioning using the most common metrics: BLEU (Papineni et al., 2002) which is computed from 1 -gram to 4 -gram precision scores (B- 1 · · · B- 4 ), Meteor (Denkowski & Lavie, 2014) (M) and CIDEr (Vedantam et al., 2015) (C) and SPICE (Anderson et al., 2016) (S).

Original: All these metrics are based on some form of n -gram overlap between the system output and the reference captions (i.e. no image information is used).
Revised: All these metrics are based on some form of n -gram overlap between the system output and the reference captions (i.e. no image information is used).

Original: For each system-generated caption, we compare against ﬁve references.
Revised: For each system-generated caption, we compare against ﬁve references.

Original: We used the publicly available cocoeval script for evaluation.
Revised: We used the publicly available cocoeval script for evaluation.

Original: Model Settings and Hyperparameters
Revised: Model Settings and Hyperparameters

Original: We use a single hidden layer LSTM with 128dimensional word embeddings and 256 -dimensional hidden dimensions.
Revised: We use a single hidden layer LSTM with 128dimensional word embeddings and 256 -dimensional hidden dimensions.

Original: As training vocabulary we retain only words that appear at least twice.
Revised: As training vocabulary we retain only words that appear at least twice.

Original: 3. I MAGE C APTIONING R ESULTS
Revised: 3. I MAGE C APTIONING R ESULTS

Original: We report results of IC on MSCOCO in Table 1, where the IC model (Section 2) is conditioned on the various image representations described in Section 3.1.
Revised: We report results of IC on MSCOCO in Table 1, where the IC model (Section 2) is conditioned on the various image representations described in Section 3.1.

Original: As expected, using random image
Revised: As expected, using random image

Original: [Table]
Revised: [Table]

Original: the MSCOCO test split, where we vary only the image representation and keep other parameters constant.
Revised: the MSCOCO test split, where we vary only the image representation and keep other parameters constant.

Original: The captions are generated with beam = 1 .
Revised: The captions are generated with beam = 1 .

Original: We report B LEU, M eteor,
Revised: We report B LEU (1-4),

Original:  C IDEr and S PICE scores.
Revised: M eteor, C IDEr and S PICE scores.

Original: embeddings clearly does not provide any useful information and performs poorly.
Revised: embeddings clearly does not provide any useful information and performs poorly.

Original: The Softmax representations with similar sets of object classes ( VGG19 , ResNet152 , and Hybrid1365-ResNet152 ) have very similar performance.
Revised: The Softmax representations with similar sets of object classes ( VGG19 , ResNet152 , and Hybrid1365-ResNet152 ) have very similar performance.

Original: However, the Places365-ResNet representations perform poorly.
Revised: However, the Places365-ResNet representations perform poorly.

Original: We note that the posterior distribution may not directly correspond to captions as there are many words and concepts that are not contained in the set of object classes.
Revised: We note that the posterior distribution may not directly correspond to captions as there are many words and concepts that are not contained in the set of object classes.

Original: Our results differ from those by
Revised: Our results differ from those by

Original: Wu et al.
Revised: Wu et al.

Original: (2016) and Yao et al.
Revised: (2016) and Yao et al.

Original: (2016) where the object classes have been ﬁne-tuned to correspond directly to the caption vocabulary.
Revised: (2016) where the object classes have been ﬁne-tuned to correspond directly to the caption vocabulary.

Original: We posit that the degradation in performance is due to spurious probability distributions over object classes for similar looking images.
Revised: We posit that the degradation in performance is due to spurious probability distributions over object classes for similar looking images.

Original: The performance of the Pool5 image representations shows a similar trend for VGG19 , ResNet152 , and Hybrid1365-ResNet152 .
Revised: The performance of the Pool5 image representations shows a similar trend for VGG19 , ResNet152 , and Hybrid1365-ResNet152 .

Original: ResNet152 is slightly better in performance.
Revised: ResNet152 is slightly better in performance.

Original: The Places365-ResNet representation performs poorly.
Revised: The Places365-ResNet representation performs poorly.

Original: We posit that the representations from the image network trained on object classes rather than scene classes are able to capture more ﬁne-grained image details from the images, whereas the image network trained with scene-based classes captures more coarse-grained information.
Revised: We posit that the representations from the image network trained on object classes rather than scene classes are able to capture more ﬁne-grained image details from the images, whereas the image network trained with scene-based classes captures more coarse-grained information.

Original: The performance of the averaged top- k word embeddings is similar to that of the Softmax representation.
Revised: The performance of the averaged top- k word embeddings is similar to that of the Softmax representation.

Original: This is interesting, since the averaged word representational information is mostly noisy: we combine top-
Revised: This is interesting, since the averaged word representational information is mostly noisy: we combine top-

Original: k synset-level information into one single vector, however, it still performs competitively.
Revised: k synset-level information into one single vector, however, it still performs competitively.

Original: We observe that the performance of the Bag of Objects (BOO) sparse 80 -dimensional annotation vector is better than all other image representations judging by the CIDEr score.
Revised: We observe that the performance of the Bag of Objects (BOO) sparse 80 -dimensional annotation vector is better than all other image representations judging by the CIDEr score.

Original: We remark here again, that this is despite the fact that the annotations may not directly correspond to the semantic information in the image or the captions.
Revised: We remark here again, that this is despite the fact that the annotations may not directly correspond to the semantic information in the image or the captions.

Original: The sparse representational information is indicative of the presence of only a subset of potentially useful objects.
Revised: The sparse representational information is indicative of the presence of only a subset of potentially useful objects.

Original: We notice two distinct patterns, a marked difference with Binary and Count based representations.
Revised: We notice two distinct patterns, a marked difference with Binary and Count based representations.

Original: This takes us back to the motivation that image captioning ideally requires information about objects, interaction between the objects with attribute information.
Revised: This takes us back to the motivation that image captioning ideally requires information about objects, interaction between the objects with attribute information.

Original: Although our representation is really sparse on the object interactions, it captures the basic concept of the presence of more than one object of the same kind, and thus provides some kind of extra information.
Revised: Although our representation is really sparse on the object interactions, it captures the basic concept of the presence of more than one object of the same kind, and thus provides some kind of extra information.

Original: A similar trend is observed by Yin & Ordonez (2017), although in their models they further try to learn interactions using a speciﬁed object RNN.
Revised: A similar trend is observed by Yin & Ordonez (2017), although in their models they further try to learn interactions using a speciﬁed object RNN.

Original: We also notice that predicted objects using YOLOCoco performs better than the YOLO9k .
Revised: We also notice that predicted objects using YOLOCoco performs better than the YOLO9k .

Original: This is probably expected as the YOLOCoco was trained on the same dataset hence obtaining better object proposals.
Revised: This is probably expected as the YOLOCoco was trained on the same dataset hence obtaining better object proposals.

Original: We also observed that with YOLO9k , we had a signiﬁcant number of objects being predicted for the test images that were not seen with the training set (around 20%).
Revised: We also observed that with YOLO9k , we had a signiﬁcant number of objects being predicted for the test images that were not seen with the training set (around 20%).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: The most surprising result is the performance of the pseudo-random vectors.
Revised: The most surprising result is the performance of the pseudo-random vectors.

Original: We notice that both the pseudo-random binary and the pseudo-random count based vectors perform almost as good as the
Revised: We notice that both the pseudo-random binary and the pseudo-random count based vectors perform almost as good as the

Original: Gold objects.
Revised: Gold objects.

Original: This suggests that the conditioned RNN is able to remove noise and learn some sort of a common ‘visual-linguistic’ semantic subspace.
Revised: This suggests that the conditioned RNN is able to remove noise and learn some sort of a common ‘visual-linguistic’ semantic subspace.

Original:  A NALYSIS ON DISTRIBUTIONAL SIMILARITY IN IC
Revised:  A NALYSIS ON DISTRIBUTIONAL SIMILARITY IN IC

Original: We perform further analysis on the different image representations to gain a further understanding of the representations and demonstrate our distributional similarity hypothesis.
Revised: We perform further analysis on the different image representations to gain a further understanding of the representations and demonstrate our distributional similarity hypothesis.

Original: 4. F ACTORIZING THE R EPRESENTATIONS
Revised: 4. F ACTORIZING THE R EPRESENTATIONS

Original: In Section 3.3, we observed encouraging results from the bag of objects based representations despite being sparse, low-dimensional, and only partially relevant to captions.
Revised: In Section 3.3, we observed encouraging results from the bag of objects based representations despite being sparse, low-dimensional, and only partially relevant to captions.

Original: Interestingly, using pseudo-random vectors derived from bag of objects also gave excellent performance despite the added noise.
Revised: Interestingly, using pseudo-random vectors derived from bag of objects also gave excellent performance despite the added noise.

Original: This leads to the question: are high-dimensional vectors necessary or relevant?
Revised: This leads to the question: are high-dimensional vectors necessary or relevant?

Original: To answer this, we evaluate whether the performance of the model signiﬁcantly degrades if we reduce the dimensionality of a high dimensional representation.
Revised: To answer this, we evaluate whether the performance of the model signiﬁcantly degrades if we reduce the dimensionality of a high dimensional representation.

Original: We experiment with three exploratory factor analysis based methods – Principal Component Analysis (PCA) (Halko et al., 2011), Probabilistic Principal Component Analysis (PPCA) (Tipping & Bishop, 1999) and Independent Component
Revised: We experiment with three exploratory factor analysis based methods – Principal Component Analysis (PCA) (Halko et al., 2011), Probabilistic Principal Component Analysis (PPCA) (Tipping & Bishop, 1999) and Independent Component

Original: Analysis (ICA)
Revised: Analysis (ICA)

Original: (Hyv ¨ arinen et al., 2004).
Revised: (Hyv ¨ arinen et al., 2004).

Original: In all cases, we obtain 80 -dimensional factorized representations on ResNet152 pool5 ( 2048 D ) that is commonly used in IC.
Revised: In all cases, we obtain 80 -dimensional factorized representations on ResNet152 pool5 ( 2048 D ) that is commonly used in IC.

Original: We summarize our experiment in Table 2.
Revised: We summarize our experiment in Table 2.

Original: We observe that, the representations obtained by all the factor models seem to retain the necessary representational power to produce appropriate captions equivalent to the original representation.
Revised: We observe that, the representations obtained by all the factor models seem to retain the necessary representational power to produce appropriate captions equivalent to the original representation.

Original: This seems contradictory as we expected a loss in the information content when compressing to arbitrary 80 -dimensions.
Revised: This seems contradictory as we expected a loss in the information content when compressing to arbitrary 80 -dimensions.

Original: This experiment indicates that the model is not explicitly utilizing the full expressiveness of the full 2048 -dimensional representations.
Revised: This experiment indicates that the model is not explicitly utilizing the full expressiveness of the full 2048 -dimensional representations.

Original: We conclude that the model is able to learn from a seemingly weak, structured information and is able to result in a performance that is close to one that uses the full representation.
Revised: We conclude that the model is able to learn from a seemingly weak, structured information and is able to result in a performance that is close to one that uses the full representation.

Original: 4. A NALYZING I MAGE R EPRESENTATIONS
Revised: 4. A NALYZING I MAGE R EPRESENTATIONS

Original: In this section, we investigate the distributional similarity hypothesis by inspecting the regularities in the initial representation state for several representations from Section 3.1, using the interpretable bag-of-objects representation.
Revised: In this section, we investigate the distributional similarity hypothesis by inspecting the regularities in the initial representation state for several representations from Section 3.1, using the interpretable bag-of-objects representation.

Original: If the representation is informative for IC, then the representations should ideally semantically related images together, and in turn allow for relevant captions to be generated.
Revised: If the representation is informative for IC, then the representations should ideally semantically related images together, and in turn allow for relevant captions to be generated.

Original: We compare different image representations with respect to their ability to group and distinguish between semantically related images.
Revised: We compare different image representations with respect to their ability to group and distinguish between semantically related images.

Original: For this, we selected three categories from MSCOCO (“dog”, “person”, “toilet”) and also pairwise combinations of these (“dog+person”, “dog+toilet”, “person+toilet”).
Revised: For this, we selected three categories from MSCOCO (“dog”, “person”, “toilet”) and also pairwise combinations of these (“dog+person”, “dog+toilet”, “person+toilet”).

Original: Up to 25 images were randomly selected for each of these six groups (single category or pair) such that the images are annotated with only the associated categories.
Revised: Up to 25 images were randomly selected for each of these six groups (single category or pair) such that the images are annotated with only the associated categories.

Original: Each group is represented by the average image feature of these images.
Revised: Each group is represented by the average image feature of these images.

Original: Figure 1 shows the cosine distances between each group, for each of our image representations.
Revised: Figure 1 shows the cosine distances between each group, for each of our image representations.

Original: The Bag of Objects model clusters these groups the best, as expected (e.g. the average image representation of “dog” correlates with images containing “dog” as a pair like “dog+person” and “dog+toilet”).
Revised: The Bag of Objects model clusters these groups the best, as expected (e.g. the average image representation of “dog” correlates with images containing “dog” as a pair like “dog+person” and “dog+toilet”).

Original: The Softmax models seem to also to exhibit semantic clusters, although to a lesser extent.
Revised: The Softmax models seem to also to exhibit semantic clusters, although to a lesser extent.

Original: This can be observed with “person”, where the features are not semantically similar to any other groups.
Revised: This can be observed with “person”, where the features are not semantically similar to any other groups.

Original: The most likely reason is that there is no “person” category in ILSVRC.
Revised: The most likely reason is that there is no “person” category in ILSVRC.

Original: Also, Place365 and Hybrid1365 Softmax (Figure 1c) also showed
Revised: Also, Place365 and Hybrid1365 Softmax (Figure 1c) also showed

Original: [Figure]
Revised: [Figure]

Original: very strong similarity for images containing “toilet”, where or not they contain “dog” or “person”, possibly because they capture scene features.
Revised: very strong similarity for images containing “toilet”, where or not they contain “dog” or “person”, possibly because they capture scene features.

Original: On the other hand, Pool5 features seem to result in images that are more similar to each other than Softmax overall.
Revised: On the other hand, Pool5 features seem to result in images that are more similar to each other than Softmax overall.

Original: 4. A NALYZING T RANSFORMED I MAGE R EPRESENTATIONS
Revised: 4. A NALYZING T RANSFORMED I MAGE R EPRESENTATIONS

Original: Considering our earlier hypothesis as proposed in Section 3.3 that the conditioned RNN is learning some sort of a common ‘visual-linguistic’ semantic space, we explore the difference in representations in the initial representational space and the transformed representational space.
Revised: Considering our earlier hypothesis as proposed in Section 3.3 that the conditioned RNN is learning some sort of a common ‘visual-linguistic’ semantic space, we explore the difference in representations in the initial representational space and the transformed representational space.

Original: The transformation is learned jointly as a subtask of the image captioning.
Revised: The transformation is learned jointly as a subtask of the image captioning.

Original: We posit that image representations in the transformed space will be more semantically coherent with respect to both images and captions.
Revised: We posit that image representations in the transformed space will be more semantically coherent with respect to both images and captions.

Original: To visualize the two representational spaces, we use Barnes-Hut t-SNE (Maaten & Hinton, 2008) to compute a 2 -dimensional embedding over the test split.
Revised: To visualize the two representational spaces, we use Barnes-Hut t-SNE (Maaten & Hinton, 2008) to compute a 2 -dimensional embedding over the test split.

Original: In general, we found that images are initially clustered by visual similarity ( Pool5 ) and semantic similarity ( Softmax , Bag of Objects ).
Revised: In general, we found that images are initially clustered by visual similarity ( Pool5 ) and semantic similarity ( Softmax , Bag of Objects ).

Original: After transformation, we observe that some linguistic information from the captions has resulted in different types of clusters.
Revised: After transformation, we observe that some linguistic information from the captions has resulted in different types of clusters.

Original: Figure 2 highlights some interesting observations of the changes in clustering across three different representations.
Revised: Figure 2 highlights some interesting observations of the changes in clustering across three different representations.

Original: For Pool5 , images seem to be clustered by their visual appearance, for example snow scenes in Figure 2a, regardless of the subjects in the images (people or dogs).
Revised: For Pool5 , images seem to be clustered by their visual appearance, for example snow scenes in Figure 2a, regardless of the subjects in the images (people or dogs).

Original: After transformation, separate clusters seem to be formed for snow scenes involving a single person, groups of people, and dogs.
Revised: After transformation, separate clusters seem to be formed for snow scenes involving a single person, groups of people, and dogs.

Original: Interestingly, images of dogs in ﬁelds and snow scenes are also drawn closer together.
Revised: Interestingly, images of dogs in ﬁelds and snow scenes are also drawn closer together.

Original: Softmax (Figure 2b) shows many small, isolated clusters before transformation.
Revised: Softmax (Figure 2b) shows many small, isolated clusters before transformation.

Original: After transformation, bigger clusters seem to be formed – suggesting that the captions have again drawn related images together despite being different in the Softmax space.
Revised: After transformation, bigger clusters seem to be formed – suggesting that the captions have again drawn related images together despite being different in the Softmax space.

Original: [Figure]
Revised: [Figure]

Original: For bag of objects (Figure 2c), objects seem to be clustered by co-occurrence of object categories, for example toilets and kitchens are clustered since they share sinks.
Revised: For bag of objects (Figure 2c), objects seem to be clustered by co-occurrence of object categories, for example toilets and kitchens are clustered since they share sinks.

Original: Toilets and kitchens seem to be further apart in the transformed space.
Revised: Toilets and kitchens seem to be further apart in the transformed space.

Original: A similar observation was made by Vinyals et al.
Revised: A similar observation was made by Vinyals et al.

Original: (2016) in which the authors observe that endto-end based image captioning models are capable of performing retrieval tasks with comparable performance to the task speciﬁc models that are trained with the ranking loss.
Revised: (2016) in which the authors observe that endto-end based image captioning models are capable of performing retrieval tasks with comparable performance to the task speciﬁc models that are trained with the ranking loss.

Original: We further perform a similar analysis on the pseudorandom representations (Figure 2d).
Revised: We further perform a similar analysis on the pseudorandom representations (Figure 2d).

Original: We observe that the initial representations have very little explicit information and do not cluster.
Revised: We observe that the initial representations have very little explicit information and do not cluster.

Original: The projected representations however form clusters that mimic the projected space of the bag-of-object cluster.
Revised: The projected representations however form clusters that mimic the projected space of the bag-of-object cluster.

Original: Full sized version of images in Figure 2 are presented anonymously in: https://github.com/ anonymousiclr/HJNGGmZ0Z
Revised: Full sized version of images in Figure 2 are presented anonymously in: https://github.com/ anonymousiclr/HJNGGmZ0Z

Original: 4. D OMAIN D EPENDENCY
Revised: 4. D OMAIN D EPENDENCY

Original: We now demonstrate that end-to-end models are heavily reliant on datasets that have a similar training and test distribution.
Revised: We now demonstrate that end-to-end models are heavily reliant on datasets that have a similar training and test distribution.

Original: We posit that an IC system that performs similarity matching will not perform well on a slightly different domain for the same task.
Revised: We posit that an IC system that performs similarity matching will not perform well on a slightly different domain for the same task.

Original: Demonstrating this will further validate our hypothesis that IC systems perform image matching to generate image captions.
Revised: Demonstrating this will further validate our hypothesis that IC systems perform image matching to generate image captions.

Original: Thus, we evaluate several models trained on MSCOCO on 1000 test image samples from the
Revised: Thus, we evaluate several models trained on MSCOCO on 1000 test image samples from the

Original: Flickr30k (Young et al., 2014) dataset 7 .
Revised: Flickr30k (Young et al., 2014) dataset 7 .

Original: Like MSCOCO, Flickr30k is an image description dataset; however, unlike MSCOCO, the images have a different distribution and the descriptions are slightly longer and more descriptive.
Revised: Like MSCOCO, Flickr30k is an image description dataset; however, unlike MSCOCO, the images have a different distribution and the descriptions are slightly longer and more descriptive.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: We evaluate the captions generated by our model with Resnet152 pool5 representation and by two other state-of-the-art models pretrained on MSCOCO: a) Self-Critical (SC) (Rennie et al., 2016) based on self critical sequence training that uses reinforcement learning using metrics, and b) Bottom
Revised: We evaluate the captions generated by our model with Resnet152 pool5 representation and by two other state-of-the-art models pretrained on MSCOCO: a) Self-Critical (SC) (Rennie et al., 2016) based on self critical sequence training that uses reinforcement learning using metrics, and b) Bottom

Original: Up and Top Down (TDBU) (Anderson et al., 2017) based on top-down and bottom-up attention using object region proposals.
Revised: Up and Top Down (TDBU) (Anderson et al., 2017) based on top-down and bottom-up attention using object region proposals.

Original: Both the state-of-the-art models are much more complex than the image conditioned RNN based language model.
Revised: Both the state-of-the-art models are much more complex than the image conditioned RNN based language model.

Original: The results are summarized in Table 3.
Revised: The results are summarized in Table 3.

Original: We observe that the scores drop by a large margin.
Revised: We observe that the scores drop by a large margin.

Original: A similar observation was made by Vinyals et al.
Revised: A similar observation was made by Vinyals et al.

Original: and they alluded the drop in scores to the linguistic mismatch between the datasets.
Revised: and they alluded the drop in scores to the linguistic mismatch between the datasets.

Original: However, we probed the out of training vocabulary words in the Flickr30k test set and observed that it was around 8.6% which seems to be the usual unseen rate.
Revised: However, we probed the out of training vocabulary words in the Flickr30k test set and observed that it was around 8.6% which seems to be the usual unseen rate.

Original: This suggests that there is more to the issue than mere vocabulary mismatch.
Revised: This suggests that there is more to the issue than mere vocabulary mismatch.

Original: We observe that while typical sentences on Flickr30k are structurally different and are generally longer, the model is still unable to generate good bigrams or even unigrams as is evident from B-1 and B-2 scores in Table 3.
Revised: We observe that while typical sentences on Flickr30k are structurally different and are generally longer, the model is still unable to generate good bigrams or even unigrams as is evident from B-1 and B-2 scores in Table 3.

Original: We further investigated the distributions of objects in the images by using YOLO object detector (trained on MSCOCO).
Revised: We further investigated the distributions of objects in the images by using YOLO object detector (trained on MSCOCO).

Original: We ﬁrst detect objects on the MSCOCO training set and our MSCOCO test set followed by objects on the Flickr30k test set.
Revised: We ﬁrst detect objects on the MSCOCO training set and our MSCOCO test set followed by objects on the Flickr30k test set.

Original: It Figure 

Revised: It Figure

Original: we show the normalized frequency versus the distribution of objects over the MSCOCO train and test sets.
Revised: we show the normalized frequency versus the distribution of objects over the MSCOCO train and test sets.

Original: We notice that the distributions are very similar and mostly overlap.
Revised: We notice that the distributions are very similar and mostly overlap.

Original: In Figure 4 we show the normalized frequency versus distribution of objects detected over MSCOCO train and Flickr30k test sets.
Revised: In Figure 4 we show the normalized frequency versus distribution of objects detected over MSCOCO train and Flickr30k test sets.

Original: We observe that the two distributions are slightly different, they don’t overlap as closely as we see in Figure 3.
Revised: We observe that the two distributions are slightly different, they don’t overlap as closely as we see in Figure 3.

Original: We hypothesize that the difference in distribution is one of the difference that reﬂects in the lower performance of a model that is trained on the MSCOCO dataset performing poorly on the Flickr30k test set.
Revised: We hypothesize that the difference in distribution is one of the difference that reﬂects in the lower performance of a model that is trained on the MSCOCO dataset performing poorly on the Flickr30k test set.

Original: 4. U NIQUENESS OF C APTIONS
Revised: 4. U NIQUENESS OF C APTIONS

Original: We posulate that image captions are often repeated because they are generated by ‘retrieving’  images in the joint image-text semantic space and generating the relevant caption at this point in the space.
Revised: We postulate that image captions are often repeated because they are generated by ‘retrieving’ similar images in the joint image-text semantic space and generating the relevant caption at this point in the space.

Original: To investigate this, we ﬁrst show that, regardless of the representation, end-to-end IC systems do not generate unique captions for every distinct image.
Revised: To investigate this, we ﬁrst show that, regardless of the representation, end-to-end IC systems do not generate unique captions for every distinct image.

Original: We use the full validation portion of the
Revised: We use the full validation portion of the

Original: MSCOCO dataset (40,504 images) and produce captions with four types of distinct image representations.
Revised: MSCOCO dataset (40,504 images) and produce captions with four types of distinct image representations.

Original: We report the results in Table 4.
Revised: We report the results in Table 4.

Original: We observe that in almost all cases, the produced representations are far from unique.
Revised: We observe that in almost all cases, the produced representations are far from unique.

Original: In most cases, there is a signiﬁcant portion of the captions that are repeated.
Revised: In most cases, there is a signiﬁcant portion of the captions that are repeated.

Original: This is also observed by Devlin et al.
Revised: This is also observed by Devlin et al.

Original: (2015) on different test splits, but using retrieval-based and pipeline based methods for IC.
Revised: (2015) on different test splits, but using retrieval-based and pipeline based methods for IC.

Original: Intrigued by the results that almost all the representations end up with similar proportion of unique captions, we further investigate the model using a k -nearest neighbor.
Revised: Intrigued by the results that almost all the representations end up with similar proportion of unique captions, we further investigate the models using a k -nearest neighbor approach.

Original: The key idea is that if the IC systems perform some form of image matching and a complex text retrieval from the training set, then the nearest neighbour (from training) of a test image should have a similar caption to the one generated by the model.
Revised: The key idea is that if the IC systems perform some form of image matching and a complex text retrieval from the training set, then the nearest neighbour (from training) of a test image should have a similar caption to the one generated by the model.

Original: We note that the model is clearly not performing text retrieval as the LSTM does generate novel captions, possibly by aggregating or ‘averaging’ the captions of similar images and performing some factorization.
Revised: We note that the model is clearly not performing text retrieval as the LSTM does generate novel captions, possibly by aggregating or ‘averaging’ the captions of similar images and performing some factorization.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: To perform this experiment, weﬁrst begin by generating captions for every training image using the bag of objects model (with frequency counts).
Revised: To perform this experiment, we begin by generating captions for every training image using the bag of objects model (with frequency counts).

Original: We then compute the k -nearest training images for each given test image using both the bag of objects representation and its projection (Eq. 2).
Revised: We then compute the k -nearest training images for each given test image using both the bag of objects representation and its projection (Eq. 2).

Original: Finally, we compute the similarity score between the generated caption of the test image against all k -nearest captions.
Revised: Finally, we compute the similarity score between the generated caption of the test image against all k -nearest captions.

Original: The similarity score measures how well a generated caption matches its nearest neighbour’s captions.
Revised: The similarity score measures how well a generated caption matches its nearest neighbour’s captions.

Original: We expect the score to be high if the IC system generates an image similar to something ‘summarized’ from the training set.
Revised: We expect the score to be high if the IC system generates an image similar to something ‘summarized’ from the training set.

Original: We report our results in Table 5.
Revised: We report our results in Table 5.

Original: We observe that overall the captions seem to closely match the captions of 5 nearest training images.
Revised: We observe that overall the captions seem to closely match the captions of 5 nearest training images.

Original: Further analysis showed that 2301 captions had nearest images at a zero distance, i.e., the same exact representation was seen during at least 5 times in training (note that CIDEr gives a score of 10 only if the test caption and all references are the same).
Revised: Further analysis showed that 2301 captions had nearest images at a zero distance, i.e., the same exact representation was seen during at least 5 times in training (note that CIDEr gives a score of 10 only if the test caption and all references are the same).

Original: We found that among the non-exact image matches, the projected image representation better captures candidates in the training set than the bag of objects.
Revised: We found that among the non-exact image matches, the projected image representation better captures candidates in the training set than the bag of objects.

Original: We further analyze the captions and provide details in the appendix.
Revised: We further analyze the captions and provide details in the appendix.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We hypothesized that IC systems essentially exploit a distributional similarity space to ‘generate’ image captions, by attempting to match a test image to similar training image(s) and generate an image caption from these similar images.
Revised: We hypothesized that IC systems essentially exploit a distributional similarity space to ‘generate’ image captions, by attempting to match a test image to similar training image(s) and generate an image caption from these similar images.

Original: We focus on the image side of image captioning, and vary the image representations while keeping the text generation component of an end-to-end CNN-RNN model constant.
Revised: Our study focused on the image side of image captioning: We varied the image representations while keeping the text generation component of an end-toend CNN-RNN model constant.

Original: We found that regardless of the image representation, end-to-end IC systems seem to match images and generate captions in a visual-semantic subspace for IC.
Revised: We found that regardless of the image representation, end-to-end IC systems seem to match images and generate captions in a visual-semantic subspace for IC.

Original: We conclude that:
Revised: We conclude that:

Original: • a sparse, low-dimensional bags-of-objects representation can be used as a tool to investigate the contribution of images in IC; we demonstrated that such a vector is sufﬁcient for generating good image captions; • end-to-end IC models are remarkably capable of separating structure from noisy input representations, as demonstrated by pseudo-random vectors ; • end-to-end IC models experience virtually no signiﬁcant loss in performance when a high dimensional representation is factorized to a lower dimensional space; • end-to-end IC models have learned a joint visual-textual semantic subspace by clustering images with similar visual and linguistic information together; • end-to-end IC models rely on test sets with a similar distribution as the training set for generating good captions; • end-to-end IC models repeatedly generate the same captions by matching images in the joint visual-textual space and ‘retrieving’ a caption in the learned joint space.
Revised: • A sparse, low-dimensional bags-of-objects representation can be used as a tool to investigate the contribution of images in IC; we demonstrated that such a vector is sufﬁcient for generating good image captions; • End-to-end IC models are remarkably capable of separating structure from noisy input representations, as demonstrated by pseudo-random vectors ; • End-to-end IC models suffer virtually no signiﬁcant loss in performance when a high dimensional representation is factorized to a lower dimensional space; • End-to-end IC models have learned a joint visual-textual semantic subspace by clustering images with similar visual and linguistic information together; • End-to-end IC models rely on test sets with a similar distribution as the training set for generating good captions; • End-to-end IC models repeatedly generate the same captions by matching images in the joint visual-textual space and ‘retrieving’ a caption in the learned joint space.

Original: All the observations above strengthen our distributional similarity hypothesis – that end-to-end IC performs image matching and generates captions for a test image from similar image(s) from the training set – rather than performing actual image understanding.
Revised: All the observations above strengthen our distributional similarity hypothesis – that end-to-end IC performs image matching and generates captions for a test image from similar image(s) from the training set – rather than performing actual image understanding.

Original: Our ﬁndings provide novel insights into what end-to-end IC systems are actually doing, that previous work only suggests or hints at without concretely demonstrate the distributional similarity hypothesis.
Revised: Our ﬁndings provide novel insights into what end-to-end IC systems are actually doing, which previous work only suggests or hints at without concretely demonstrating the distributional similarity hypothesis.

Original: We are able to show this through interpretable representations.

Original: We believe our ﬁndings is important to the  community to further advance image captioning in a more informed manner, rather than by ‘guessing’.
Revised: We believe our ﬁndings are important for the IC community to further advance image captioning in a more informed manner.

Original: A A NALYSIS ON G ENERATED C APTIONS
Revised: A A NALYSIS ON G ENERATED C APTIONS

Original: Here, we provide a qualitative analysis of different image representations presented and gain some insights into how they contribute to the the IC task.
Revised: Here, we provide a qualitative analysis of different image representations presented and gain some insights into how they contribute to the the IC task.

Original: The Bag of Objects representation led to a strong performance in IC despite being extremely sparse and low-dimensional (80 dimensions).
Revised: The Bag of Objects representation led to a strong performance in IC despite being extremely sparse and low-dimensional (80 dimensions).

Original: Analyzing the test split, we found that each vector consists of only 2.86 non-zero entries on average (standard deviation 1.8, median 2).
Revised: Analyzing the test split, we found that each vector consists of only 2.86 non-zero entries on average (standard deviation 1.8, median 2).

Original: Thus, with the minimal information being provided to the generator RNN, we ﬁnd it surprising that it is able to perform so well.
Revised: Thus, with the minimal information being provided to the generator RNN, we ﬁnd it surprising that it is able to perform so well.

Original: We compare the output of the remaining models against the Bag of Objects representation by investigating what each representation adds to or subtracts from this simple, yet strong model.
Revised: We compare the output of the remaining models against the Bag of Objects representation by investigating what each representation adds to or subtracts from this simple, yet strong model.

Original: We start by selecting images (from the test split) annotated with the exact same Bag of Objects representation – which should result in the same caption.
Revised: We start by selecting images (from the test split) annotated with the exact same Bag of Objects representation – which should result in the same caption.

Original: For our qualitative analysis, several sets of one to three
Revised: For our qualitative analysis, several sets of one to three

Original: MSCOCO categories were manually chosen.
Revised: MSCOCO categories were manually chosen.

Original: For each set, images were selected such that there is exactly one instance of each category in the set and zero for others.
Revised: For each set, images were selected such that there is exactly one instance of each category in the set and zero for others.

Original: We then shortlisted images where the captions generated by the Bag of Objects model produced the ﬁve highest and ﬁve lowest
Revised: We then shortlisted images where the captions generated by the Bag of Objects model produced the ﬁve highest and ﬁve lowest

Original: CIDEr scores (ten images per set).
Revised: CIDEr scores (ten images per set).

Original: We then compare the captions sampled for each of the other representations.
Revised: We then compare the captions sampled for each of the other representations.

Original: Figure 5 shows some example outputs from this analysis.
Revised: Figure 5 shows some example outputs from this analysis.

Original: In Figure 5a, Bag of Objects achieved a high CIDEr score despite only being given “bird” as input, mainly by ‘guessing’ that the bird will be perching/sitting on a branch.
Revised: In Figure 5a, Bag of Objects achieved a high CIDEr score despite only being given “bird” as input, mainly by ‘guessing’ that the bird will be perching/sitting on a branch.

Original: The object-based Softmax (VGG and ResNet) models gave an even more accurate description as “owl” is the top-1 prediction of both representations (96% conﬁdence for VGG, 77% for ResNet).
Revised: The object-based Softmax (VGG and ResNet) models gave an even more accurate description as “owl” is the top-1 prediction of both representations (96% conﬁdence for VGG, 77% for ResNet).

Original: Places365 predicted “swamp” and “forest”.
Revised: Places365 predicted “swamp” and “forest”.

Original: The Penultimate features on the other hand struggled with representing the images correctly.
Revised: The Penultimate features on the other hand struggled with representing the images correctly.

Original: In Figure 5b, Bag of Objects struggled with lack of information (only “airplane” is given), the Softmax features mainly predicted “chainlink fence”, Places365 predicted “kennel” (hence the dog description), and it most likely that
Revised: In Figure 5b, Bag of Objects struggled with lack of information (only “airplane” is given), the Softmax features mainly predicted “chainlink fence”, Places365 predicted “kennel” (hence the dog description), and it most likely that

Original: Penultimate has captured the fence-like features in the image rather than the plane.
Revised: Penultimate has captured the fence-like features in the image rather than the plane.

Original: In Figure 5c, the
Revised: In Figure 5c, the

Original: Softmax features generally managed to generate a caption describing a woman despite not explicitly containing the ‘woman’ category.
Revised: Softmax features generally managed to generate a caption describing a woman despite not explicitly containing the ‘woman’ category.

Original: This is because other correlated categories were predicted, such as “mask”, “wig”, “perfume”, “hairspray” and in the case of Places365 “beauty salon” and “dressing room”.
Revised: This is because other correlated categories were predicted, such as “mask”, “wig”, “perfume”, “hairspray” and in the case of Places365 “beauty salon” and “dressing room”.

Original: B H YPERPARAMETER S ETTINGS
Revised: B H YPERPARAMETER S ETTINGS

Original: Our model settings were:
Revised: Our model settings were:

Original: • LSTM with 128 dimensional word embeddings and 256 dimensional hidden representations Dropout over LSTM of 0.
Revised: • LSTM with 128 dimensional word embeddings and 256 dimensional hidden representations Dropout over LSTM of 0.

Original: • We used Adam for optimization.
Revised: • We used Adam for optimization.

Original: • We ﬁxed the learning rate to 4e-
Revised: • We ﬁxed the learning rate to 4e-

Original: We report our results by keeping the above settings constant.
Revised: We report our results by keeping the above settings constant.