Paper:
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
We introduce a model for bidirectional retrieval of images and sentences through a deep, multi-modal embedding of visual and natural language data. Unlike previous models that directly map images or sentences into a common embedding space, our model works on a finer level and embeds fragments of images (objects) and fragments of sentences (typed dependency tree relations) into a common space. We then introduce a structured max-margin objective that allows our model to explicitly associate these fragments across modalities. Extensive experimental evaluation shows that reasoning on both the global level of images and sentences and the finer level of their respective fragments improves performance on image-sentence retrieval tasks. Additionally, our model provides interpretable predictions for the image-sentence retrieval task since the inferred inter-modal alignment of fragments is explicit.

Review:
This paper proposes a novel approach to learning joint image-text embeddings for the tasks of image to text retrieval and vice versa. The authors propose the use of image and sentence fragments, which come from convolutional net features of detected objects and dependency tree relations. A ranking objective is proposed that combines both a fragment and global objective, where latent dependency-fragment alignments are inferred with a MIL extension. Extensive experimentation is performed on the ranking tasks proposed by Hodosh et al, including results on the new Flickr30k dataset.This is a nice paper. The authors obtain strong performance on each of the datasets and the experimentation is well done. Model choices are experimentally verified with convincing results. I have some minor questions / comments below that I hope the authors can respond to:- lines 018-021: While this work is the first to go beyond a global ranking objective, conceivably even previous methods that work globally should be able to reason locally. For example, each intermediate node of an RNN would live within the same embedding space (as well as words, if they are fine-tuned). This would allow, for example, the retrieval of images from a single word or short phrase, even through training was only done globally. Of course, I would expect better results from your approach since such fragment-level embeddings are being optimized for directly.- line 137: Wouldn't W_e be d x 400,000, since each of w_1, w_2 is 400,000 x 1?- It would be helpful to summarize the space of hyperparameters that you tune (there seems to be a lot of them). Do the best validation hyperparameter choices vary greatly across datasets? Any intuition about the hyperparameter space (are some choices consistently better than others, etc) would be helpful for the reader to gain some intuition. How sensitive are the validation results to \beta (the weight on the global objective)?- On a related note, even though you were not able to cross-validate all the hyperparameters of Flickr30k with SDT-RNN, perhaps you could report results on Flickr30k using the best selected hyperparameters from Flickr8k (unless there is overlap between the two datasets)- section 4.1 baselines: Are you also including the CNN features for the full image, or just the detected objects? - You mention that the class predictions from the convolutional net are discarded. Wouldn't these have been useful to inferring the fragment alignments?- One of the challenges of this task is obtaining large enough datasets with detailed, accurate image captions. The experimental results are convincing in this case, since the three datasets used went through an extensive annotation process. How do you think your method would do on larger, but less accurate captioned datasets? For example, the SBU captioned photo dataset has 1 million images but many of these captions do not necessarily depect visual scenes (e.g. 'fido' referring to a dog). On such datasets, I would suspect that the use of fragment embeddings would be less beneficial.