Paper:
Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images
In this paper, we focus on training and evaluating effective word embeddings with both text and visual information. More specifically, we introduce a large-scale dataset with 300 million sentences describing over 40 million images crawled and downloaded from publicly available Pins (i.e. an image with sentence descriptions uploaded by users) on Pinterest [2]. This dataset is more than 200 times larger than MS COCO [22], the standard large-scale image dataset with sentence descriptions. In addition, we construct an evaluation dataset to directly assess the effectiveness of word embeddings in terms of finding semantically similar or related words and phrases. The word/phrase pairs in this evaluation dataset are collected from the click data with millions of users in an image search system, thus contain rich semantic relationships. Based on these datasets, we propose and compare several Recurrent Neural Networks (RNNs) based multimodal (text and image) models. Experiments show that our model benefits from incorporating the visual information into the word embeddings, and a weight sharing strategy is crucial for learning such multimodal embeddings. The project page is: http://www.stat. ucla.edu/~junhua.mao/multimodal_embedding.html 1 .

Review:
This paper presents a new multi-million image-caption style dataset and use it for training multimodal embedding models. They also present a new user click based similarity dataset for evaluation. Finally, they try 2-3 CNN-RNN style models for multimodal phrase embedding similarity. The datasets are decent contributions (but with some issues). However, the evaluation has multiple issues in its current form and the models are also borrowed from previous work. 
After author response: Thanks for answering some of my questions -- I updated some of my scores accordingly. I still encourage the authors to answer the rest of the questions, esp. eval on better downstream tasks like captioning and visual zero-shot learning.  --------------------- Evaluation issues:   -- No existing datasets have been used to evaluate, e.g., WordSim353, SimLex-999 or the new visual similarity datasets VisSim and SemSim from Lazaridou et al., 2015. The authors have not compared to any existing paper/model on these datasets to show the advantage of either their larger/better training dataset or their models, hence leaving no takeaways.  -- Why only work on word similarity? Why not show other image-language tasks in addition, e.g., captioning, visual zero-shot learning? This will better demonstrate the advantages of the training dataset.  -- For word embedding baseline, please use stronger latest models such as GloVe and Skip-thought and paragram.  Evaluation dataset issues:  -- The dataset mixes multiple types of similarity, e.g., related versus paraphrase/synonym, and this has been discussed to be a big issue for evaluating embedding models.  -- the data uses triplets instead of the standard ranking+correlation method, and then they also choose random negative phrases for these triplets (which will be easy to detect), which makes the task much easier.  -- In the CrowdFlower experiment, the turkers will assume they need to choose at least one of the two phrases most of the time, and even if they choose the correct phrase, it might be only because it is more related to the query phrase as compared to the other random negative phrase, but overall the positive phrase might still be only very slightly related to the query phrase (because of the initial recommendation system based retrieval). To verify this, the authors should get turkers to also rate how much these 'positive' phrases are related to the query phrase in absolute terms/ratings.  -- they use 3-5 annotators and then choose phrases where >50% of the annotators agree, but this will mean agreement between just two people in many cases. For such a noisy initial dataset, the filtering should be more strict than 50%, or the #annotators should be higher.  Training dataset:  -- Since the data is from Pinterest and the 'captions' are just user comments, I am worried that some of these captions might not be standard description style sentences but instead might contain some non-image/visual story or information -- the authors should investigate and present this, and also verify if this corpus can be used for training captioning systems.  Model:  -- main model figure is very unclear and should be expanded and maybe separated.  -- all models are from previous work; would have been good to also suggest some new model variants for this task/datasets.  -- why use average pooling at test time for a phrase representation and not run the trained RNN model?  Other:   -- Lots of typo's throughout the paper, e.g.,  Line15: "crutial" --> "crucial" Line34: "commemted" --> "commented" Table 3: "weigh" --> "weight" Line 232: "to Model A, We" --> "to Model A, we"