Paper:
Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
In this paper, we present the mQA model, which is able to answer questions about the content of an image. The answer can be a sentence, a phrase or a single word. Our model contains four components: a Long Short-Term Memory (LSTM) to extract the question representation, a Convolutional Neural Network (CNN) to extract the visual representation, an LSTM for storing the linguistic context in an answer, and a fusing component to combine the information from the first three components and generate the answer. We construct a Freestyle Multilingual Image Question Answering (FM-IQA) dataset to train and evaluate our mQA model. It contains over 150,000 images and 310,000 freestyle Chinese question-answer pairs and their English translations. The quality of the generated answers of our mQA model on this dataset is evaluated by human judges through a Turing Test. Specifically, we mix the answers provided by humans and our model. The human judges need to distinguish our model from the human. They will also provide a score (i.e. 0, 1, 2, the larger the better) indicating the quality of the answer. We propose strategies to monitor the quality of this evaluation process. The experiments show that in 64.7% of cases, the human judges cannot distinguish our model from humans. The average score is 1.454 (1.918 for human). The details of this work, including the FM-IQA dataset, can be found on the project page: http://idl.baidu.com/FM-IQA.html.
function. To lower down the risk of overfitting, we allow the weight sharing of the word embedding layer between the LSTMs in the first and third components. We also adopt the transposed weight sharing scheme as proposed in [25], which allows the weight sharing between word embedding layer and the fully connected Softmax layer.
To train our method, we construct a large-scale Freestyle Multilingual Image Question Answering dataset 1 (FM-IQA, see details in Section 4) based on the MS COCO dataset [21]. The current version of the dataset contains 158,392 images with 316,193 Chinese question-answer pairs and their corresponding English translations. 2 To diversify the annotations, the annotators are allowed to raise any question related to the content of the image. We propose strategies to monitor the quality of the annotations. This dataset contains a wide range of AI related questions, such as action recognition (e.g., "Is the man trying to buy vegetables?"), object recognition (e.g., "What is there in yellow?"), positions and interactions among objects in the image (e.g. "Where is the kitty?") and reasoning based on commonsense and visual content (e.g. "Why does the bus park here?", see last column of Figure 3).
Because of the variability of the freestyle question-answer pairs, it is hard to accurately evaluate the method with automatic metrics. We conduct a Visual Turing Test [38] using human judges. Specifically, we mix the question-answer pairs generated by our model with the same set of questionanswer pairs labeled by annotators. The human judges need to determine whether the answer is given by a model or a human. In addition, we also ask them to give a score of 0 (i.e. wrong), 1 (i.e. partially correct), or 2 (i.e. correct). The results show that our mQA model passes 64.7% of this test (treated as answers of a human) and the average score is 1.454. In the discussion, we analyze the failure cases of our model and show that combined with the m-RNN [24] model, our model can automatically ask a question about an image and answer that question.

Review:
The paper was fairly clear overall. As stated above, my main concern is that the VTT evaluation might produce very different results from one set of evaluators to another, as is a danger for any such manual evaluation. The scale of resources needed to reproduce it makes it somewhat inaccessible to academic groups. A more detailed experimental protocol, including instructions given to examiners, should be provided.Aside from that, there are a few few questions/comments for the authors to consider:* The first part of the title is a little weird. The main/best contribution of the paper is the model. Something more standard might be better.* Section 3.1 explains the model hyperparameters and architecture, but the selection mechanism was not specified? Why did you use 400 memory cells? Why were the weight matrices of LSTM(Q) and LSTM(A) not shared in the first components?* 'where "+" denotes element-wise addition" <- as opposed to what??* Are the top layers of the ConvNet fixed after separate training, or only the convolutional layers?* Section 3.2 <- in neural language modelling, a common strategy is to (counter-intuitively, perhaps) decouple input and output embeddings, as representations for the purpose of predicting the next symbol may need to encode different information as those modelling the context of the prediction. If tying these matrices (which is also done in many models) provided better validation errors, this should be clearly stated. Otherwise, I'd recommend dropping some of the hand-wavvy fluffiness from this section.* Section 3.3 <- The initial learning rate is 1 and we decrease it by a factor of 10 for every epoch of the data" <- this sounds massively hacky. How was this procedure selected? Surely not against test results, right?* "We stop training when the loss does not decrease within three epochs" <- also hacky. How was this procedure selected? Why not use loss on a validation set and some form of patience/early stopping?* For VTT in section 5.1, what is variance in annotator scoring of models? Is there some form of inter-annotator agreement you could report?Overall, the idea for the model is not bad, but resembles many LSTM/RNN based QA or sequence to sequence models. My main concern with this paper is that many aspects of the model design and training procedure seem ill-motivated, and make the reported results somewhat hard to judge. The Visual Turing Test is also a problematic means of evaluation, as discussed above. I'd like to see these issues substantially addressed before recommending acceptance.