Paper:
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Recurrent Neural Networks can be trained to produce sequences of tokens given some input, as exemplified by recent results in machine translation and image captioning. The current approach to training them consists of maximizing the likelihood of each token in the sequence given the current (recurrent) state and the previous token. At inference, the unknown previous token is then replaced by a token generated by the model itself. This discrepancy between training and inference can yield errors that can accumulate quickly along the generated sequence. We propose a curriculum learning strategy to gently change the training process from a fully guided scheme using the true previous token, towards a less guided scheme which mostly uses the generated token instead. Experiments on several sequence prediction tasks show that this approach yields significant improvements. Moreover, it was used succesfully in our winning entry to the MSCOCO image captioning challenge, 2015.

Review:
Recent work on neural machine translation and other text generation tasks has trained models directly to minimize perplexity/negative log-likelihood of observed sequences. While this has shown very promising results, the setup ignores the fact that in practice the model is conditioning on generated symbols as opposed to gold symbols, and may therefore be conditioning on contexts that are quite different from the contexts seen in the gold data. This paper attempts to remedy this problem with by utilizing generated sequences at training time. Instead of conditioning on the gold context it utilizes the generated context. Unfortunately at early rounds of the algorithm this produces junk, so they introduce a "scheduled sampling" approach that alternates between the two training methods based on a predefined decay schedule inspired by curriculum learning.The strength of this paper is in its simplicity and the comprehensive empirical testing. The important model and inference assumptions are defined clearly, and the details about the internal architecture of the model are appropriately elided. It seems like it would be very straightforward to re-implement this approach on LSTM's or any other non-Markov model.  Empirically, the method seems to work quite well. There is a relatively large gain across several quite different tasks, and the schedule part seems to have a significant effect as always sampled does quite poorly.The parsing results still pretty far behind state-of-the-art, but they use a very reduced input representation (no features). The speech results also seem to be using a somewhat unique setup, but the improvement here is quite large. - I would be to know how performance changes based on the footnote 1. It seems like flipping on a token level is very different than flipping on an example level, since the worst-case distance between gold tokens is much lower.The main weakness is the lack of comparison to other methods that attempt a similar goal.For one, the authors are too quick to dismiss early-update perceptron (Collins and Roark, 2004) with beam search as being not applicable, "as the state sequence can not easily be factored". While the factoring is utilized in parsing, nothing about beam search requires this assumption to work.(This connection between beam search and DP is also made on l.130. Beam search is rarely used for HMMs, at least in NLP, and when it is, it is often exactly when it is not possible to use dynamic programming.) The continuous nature of the state shouldn't effect the use of this algorithm, and in fact there is a paper at ACL this year "Structured Training for Neural Network Transition-Based Parsing" that that using this method on neural-net model that makes similar assumptions.Secondly, I did not feel like an appropriate distinction was made with SEARN and reinforcement learning type algorithms. The related work talks about these as as "batch" approaches. While the SEARN paper itself may have chosen to use a batch multi-class classifier (since they are fast), that does not mean it couldn't be applied in the SGD case. It seems like the key idea of SEARN is to interpolate the current prediction from the model with gold based to produce a sampled trajectory. The major difference is that they may learn the policy versus using a schedule.