Paper:
Sequence to Sequence Learning with Neural Networks
Deep Neural Networks (DNNs) are powerful models that have achieved excellent performance on difficult learning tasks. Although DNNs work well whenever large labeled training sets are available, they cannot be used to map sequences to sequences. In this paper, we present a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure. Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector. Our main result is that on an English to French translation task from the WMT-14 dataset, the translations produced by the LSTM achieve a BLEU score of 34.8 on the entire test set, where the LSTM's BLEU score was penalized on out-of-vocabulary words. Additionally, the LSTM did not have difficulty on long sentences. For comparison, a phrase-based SMT system achieves a BLEU score of 33.3 on the same dataset. When we used the LSTM to rerank the 1000 hypotheses produced by the aforementioned SMT system, its BLEU score increases to 36.5, which is close to the previous state of the art. The LSTM also learned sensible phrase and sentence representations that are sensitive to word order and are relatively invariant to the active and the passive voice. Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier.

Review:
SUMMARY:This paper introduces an LSTM-based approach to map arbitrary-length sequences to other arbitrary-length sequences, which is a required property in fully neural machine translation. The LSTM is trained to produce a fixed-size representation of a sentence, which then serves as the initial state of a second LSTM which generates a translated sentence. The authors evaluate their architecture on two machine translation datasets.GENERAL OPINION:The ideas represented in this work are extremely interesting, and I love the elegance and simplicity of the proposed RNN architecture. I am happy with the changes made in the second version in terms of performance.REMARKS:Figure 1: If I'm correct, the symbols W, X, Y, and Z are samples, and used as input for the next time step, right? If so, it might help to put arrows indicating this (output to input of next time step).line 88: What is BLUE point and BLUE score? A short explanation or reference would help.line 92: "a BLUE score of 28.w"lines 112 to 118: I'm somewhat sceptical that common RNNs would find it hard to model "long-term dependencies", as word sequences are not that long at all. Most sentences are not much longer than a dozen or so words, which falls well into the dependency length that can be modelled by RNNs. I do believe RNNs would be slower and harder to train though, so please rephrase that sentence.lines 138-139: Please refer to Alex Graves work on multi-layered LSTMs: A. Graves, A. Mohamed, G. Hinton. Speech Recognition with Deep Recurrent Neural Networks. ICASSP 2013, Vancouver, Canada.line 154: Please remain consistent with notations. "DE-> EN" is not much shorter than "German -> English". line 183: What is a beam search decoder? Reference? Explanation?lines 191-193: "any translation that has an UNK cannot possibly correct". Why not? Either I'm misunderstanding something here, or this sentence is wrong. If there is an UNK in the input sentence, it makes sense that there would be one in the output, no? Conversely, if there are no UNKs in the input dataset at all, there would be no reason to include UNKs at all. Please clarify.line 197: "Le Mans University"line 249: "more careful tuning"Figure 2: It would be more convincing to have some more examples. The ones presented now could be cherry-picked.