Paper:
Published as a conference paper at ICLR 2018 LEARNING TO TEACH
Teaching plays a very important role in our society, by spreading human knowledge and educating our next generations. A good teacher will select appropriate teaching materials, impact suitable methodologies, and set up targeted examinations, according to the learning behaviors of the students. In the field of artificial intelligence, however, one has not fully explored the role of teaching, and pays most attention to machine learning. In this paper, we argue that equal attention, if not more, should be paid to teaching, and furthermore, an optimization framework (instead of heuristics) should be used to obtain good teaching strategies. We call this approach "learning to teach". In the approach, two intelligent agents interact with each other: a student model (which corresponds to the learner in traditional machine learning algorithms), and a teacher model (which determines the appropriate data, loss function, and hypothesis space to facilitate the training of the student model). The teacher model leverages the feedback from the student model to optimize its own teaching strategies by means of reinforcement learning, so as to achieve teacher-student co-evolution. To demonstrate the practical value of our proposed approach, we take the training of deep neural networks (DNN) as an example, and show that by using the learning to teach techniques, we are able to use much less training data and fewer iterations to achieve almost the same accuracy for different kinds of DNN models (e.g., multi-layer perceptron, convolutional neural networks and recurrent neural networks) under various machine learning tasks (e.g., image classification and text understanding).

Review:
This paper focuses on the problem of "machine teaching", i.e., how to select a good strategy to select training data points to pass to a machine learning algorithm, for faster learning. The proposed approach leverages reinforcement learning by defining the reward as how fast the learner learns, and use policy gradient to update the teacher parameters. I find the definition of the "state" in this case very interesting. The experimental results seem to show that such a learned teacher strategy makes machine learning algorithms learn faster. 
Overall I think that this paper is decent. The angle the authors took is interesting (essentially replacing one level of the bi-level optimization problem in machine teaching works with a reinforcement learning setup). The problem formulation is mostly reasonable, and the evaluation seems quite convincing. The paper is well-written: I enjoyed the mathematical formulation (Section 3). The authors did a good job of using different experiments (filtration number analysis, and teaching both the same architecture and a different architecture) to intuitively explain what their method actually does. 
At the same time, though, I see several important issues that need to be addressed if this paper is to be accepted. Details below. 
1. As much as I enjoyed reading Section 3, it is very redundant. In some cases it is good to outline a powerful and generic framework (like the authors did here with defining "teaching" in a very broad sense, including selecting good loss functions and hypothesis spaces) and then explain that the current work focuses on one aspect (selecting training data points). However, I do not see it being the case here. In my opinion, selecting good loss functions and hypothesis spaces are much harder problems than data teaching - except maybe when one use a pre-defined set of possible loss functions and select from it. But that is not very interesting (if you can propose new loss functions, that would be way cooler). I also do not see how to define an intuitive set of "states" in that case. Therefore, I think this section should be shortened. I also think that the authors should not discuss the general framework and rather focus on "data teaching", which is the only focus of the current paper. The abstract and introduction should also be modified accordingly to more honestly reflect the current contributions. 
2. The authors should do a better job at explaining the details of the state definition, especially the student model features and the combination of data and current learner model. 
3. There is only one definition of the reward - related to batch number when the accuracy first exceeds a threshold. Is accuracy stable, can it drop back down below the threshold in the next epoch? The accuracy on a held-out test set is not guaranteed to be monotonically increasing, right? Is this a problem in practice (it seems to happen on your curves)? What about other potential reward definitions? And what would they potentially lead to? 
4. Experimental results are averaged over 5 repeated runs - a bit too small in my opinion. 
5. Can the authors show convergence of the teacher parameter \theta? I think it is important to see how fast the teacher model converges, too. 
6. In some of your experiments, every training method converges to the same accuracy after enough training (Fig.2b), while in others, not quite (Fig. 2a and 2c). Why is this the case? Does it mean that you have not run enough iterations for the baseline methods? My intuition is that if the learner algorithm is convex, then ultimately they will all get to the same accuracy level, so the task is just to get there quicker. I understand that since the learner algorithm is an NN, this is not the case - but more explanation is necessary here - does your method also reduces the empirical possibility to get stuck in local minima? 
7. More explanation is needed towards Fig.4c. In this case, using a teacher model trained on a harder task (CIFAR10) leads to much improved student training on a simpler task (MNIST). Why?
8. Although in terms of "effective training data points" the proposed method outperforms the other methods, in terms of time (Fig.5) the difference between it and say, NoTeach, is not that significant (especially at very high desired accuracy). More explanation needed here. 
Read the rebuttal and revision and slightly increased my rating.