Paper:
Decoding with Value Networks for Neural Machine Translation
Neural Machine Translation (NMT) has become a popular technology in recent years, and beam search is its de facto decoding method due to the shrunk search space and reduced computational complexity. However, since it only searches for local optima at each time step through one-step forward looking, it usually cannot output the best target sentence. Inspired by the success and methodology of AlphaGo, in this paper we propose using a prediction network to improve beam search, which takes the source sentence x, the currently available decoding output y 1 , · · · , y t−1 and a candidate word w at step t as inputs and predicts the long-term value (e.g., BLEU score) of the partial target sentence if it is completed by the NMT model. Following the practice in reinforcement learning, we call this prediction network value network. Specifically, we propose a recurrent structure for the value network, and train its parameters from bilingual data. During the test time, when choosing a word w for decoding, we consider both its conditional probability given by the NMT model and its long-term value predicted by the value network. Experiments show that such an approach can significantly improve the translation accuracy on several translation tasks.

Review:
This paper addresses one of the limitation of NMT, the so-called
exposure bias, that results from the fact that each word is chosen
greedily. For this, the authors build on standard technique of
reinforcement learning and try to predict, for each outgoing
transition of a given state, the expected reward that will be achieved
if the system take this transition.
The article is overall very clear and the proposed ideas quite
appealing, even if many of the decisions seem quite ad hoc (e.g. why
doing a linear combination between the (predicted) expected reward and
the prediction of the NMT system and not decoding directly with the
former) and it is not clear whether the proposed approach can be
applied to other NLP tasks.
More importantly, several implementation "details" are not specified.
For instance, in Equation (6), the BLEU function is defined at the
sentence level while in the actual BLEU metric is defined at the
corpus level. The authors should have specified which approximation of
BLEU that have used.
The connection between structure prediction (especially in NLP) and
reinforcement learning has been made since 2005 (see for instance [1],
[2] or, more recently, [3]). It is, today, at the heart of most
dependency parsers (see, among others, the work of Goldberg) that have
deal with similar issues for a long time. The related work section
should not (and most not) be only about neural network works.
The MT system used in the experiments is far behind the performance
achieved by the best systems of the WMT campaign: the best system (a
"simple" phrase-base system as I remember) achieves a BLEU score of
~20 for En -> De and 35 for En -> Fr while the scores reported by the
authors are respectively ~15 and ~30. I wonder if the authors would
have observed similar gains had they considered a better baseline
system.
More generally, most of the reported experimental results are quite
disturbing. Figure 3.a  shows that increasing the beam actually
degrades translation quality. This counter-intuitive results should
have been analyzed. Figure 3.b indicates that taking into account the
predicted value of the expected BLEU actually hurts translation
performance: the less weight the predicted score gets in the linear
combination, the better the translation.
[1] Learning as Search Optimization: Approximate Large Margin Methods for Structured
Prediction, Hal Daumé III and Daniel Marcu, ICML'05
[2] Searched-based structured prediction, Hal Daumé III, John Langford
and Daniel Marcu, ML'09
[3] Learning to Search Better than Your Teacher, Kai-Wei Chang, Akshay
Krishnamurthy, Alekh Agarwal, Hal Daumé III and John Langford