Paper:
Dialog-based Language Learning
A long-term goal of machine learning research is to build an intelligent dialog agent. Most research in natural language understanding has focused on learning from fixed training sets of labeled data, with supervision either at the word level (tagging, parsing tasks) or sentence level (question answering, machine translation). This kind of supervision is not realistic of how humans learn, where language is both learned by, and used for, communication. In this work, we study dialog-based language learning, where supervision is given naturally and implicitly in the response of the dialog partner during the conversation. We study this setup in two domains: the bAbI dataset of [23] and large-scale question answering from [3]. We evaluate a set of baseline learning strategies on these tasks, and show that a novel model incorporating predictive lookahead is a promising approach for learning from a teacher's response. In particular, a surprising result is that it can learn to answer questions correctly without any reward-based supervision at all.

Review:
This paper proposes to learn question answering models using feedback in the form of short textual exchanges. This can be seen as learning from a very primitive form of dialog between a teacher (who provides the questions and some feedback) and another student (who provides answers that are correct with some fixed probability).   The paper proposes alterations to two existing datasets: the artificial bAbI dataset, and a movie QA dataset, in order to incorporate 10 different types of supervision from the teacher and student. In some settings, an explicit reward is given when the student answers correctly. An augmentation to the memory network architecture is proposed that achieves strong performance on the bAbI dataset, and moderate performance on the real-world MovieQA dataset. 
The proposed research direction of exploring models that learn from different types of conversational feedback is interesting and, to my knowledge, novel. The presentation of the task is well-motivated, and could lead to some interesting future research directions. The paper is fairly well-written.   When proposing a new task, it is often difficult to determine how challenging to make the task. By far the most interesting task proposed is task 7, where no reward-based feedback is given. However, it would be very nice to have more variations of this task. For example, one could have a varying number of script templates for correct and incorrect examples, vary the length of the responses, or have more realistic ‘noisier’ responses (ie. where the feedback is hidden in a larger response). This would isolate the language component of the feedback for learning QA policies – it is important to determine what kind of language feedback can be learned from, and this paper only explores one such aspect (whether additional information is included from the teacher in the response), and this only in the setting with reward-based feedback.  In my view such tasks would be more useful than some of the other proposed tasks (for example tasks 9 and 10, detailed below).  I am slightly concerned that the remainder of the tasks proposed, those with reward-based feedback, are too easy, and that the variations between tasks are too small. However, in some ways this is good for a new task, as one can isolate the cases where a proposed model under-performs (which is hard to do with complex tasks). It is impossible to know with certainty whether the tasks are too easy until more diverse models (other than just memory networks) are tried.  A crucial point of the paper is that the learner (ie. the memory network model being learned) is not the one actually interacting with the teacher. Instead, the model simply observes the conversation between the teacher and some other student, and learns to answer the questions from varying forms of feedback. It should be made more explicit in the abstract and introduction that the model being learned in this paper is not a dialog agent, but is a question answering system. This becomes evident much later in the paper, however for clarity this should be addressed earlier.  The formulation of tasks 9 and 10, where the student asks for help, is also unclear. In the description, it is stated that ‘the learner must ask for help in the dialog’. This seems to imply that the learner must *learn* to ask for the help of the teacher. However, the learned model is not interacting with the teacher at all – it is simply observing the other student answering this question. Thus, this is indeed very similar to tasks 3 and 5 (which is reinforced by the similarity of the results in Tables 1 and 2). This can be inferred from later sections in the paper, but is not clear enough at the outset.  The proposed memory network extension incorporating forward prediction is interesting, and its success is well-justified in section 5. The experiments are well-conducted – in particular, the results on the MovieQA dataset are crucial to demonstrate that this supervision approach can apply to real-world tasks, and provides room for improvement for future models. The fact that all of the models and baselines are memory network-variants is a slight negative. Also, it would be good to have some kind of error estimates – the models that should achieve random performance (FP model on task 1, RBI model on task 7) exhibit significant variance on the bAbI dataset (fluctuating from 20-29%), which implies that there could be some variance in the remainder of the results. This appears to be less of a problem on the MovieQA dataset.  Overall, this is a fairly good paper that makes an interesting contribution in terms of new tasks and models. There is some concern that these tasks are not difficult enough, as well as some issues with clarity in the paper (detailed above), which should be addressed by the authors. However, I could see the proposed tasks becoming a bAbI-style toy task used for model justification, so that these models can later be used for more difficult versions of the tasks.  Small comments: Line 57: put a period after [14] in related work