Paper:
Neural machine translation (NMT) systems have reached state of the art performance in translating text and are in wide deployment. Yet little is understood about how these systems function or break. Here we show that NMT systems are susceptible to producing highly pathological translations that are completely untethered from the source material, which we term hallucinations. Such pathological translations are problematic because they are are deeply disturbing of user trust and easy to find with a simple search. We describe a method to generate hallucinations and show that many common variations of the NMT architecture are susceptible to them. We study a variety of approaches to reduce the frequency of hallucinations, including data augmentation, dynamical systems and regularization techniques, showing that data augmentation significantly reduces hallucination frequency. Finally, we analyze networks that produce hallucinations and show that there are signatures in the attention matrix as well as in the hidden states of the decoder.

Review:
My major concern about the work is that the studied model is quite weak. 
	"All models we present are well trained with a BLEU score of at least 20.0 on the test set, a reasonable score for 2-layer models with 256 hidden units." 
	"We then used the WMT De!En 2016 test set (2,999 examples) to compute the hallucination percentage for each model."
	I checked the WMT official website http://matrix.statmt.org/matrix. It shows that the best result was a BLEU score of 40.2, which was obtained at 2016. The models used in this work are about 20.0, which are much less than the WMT results reported two years ago. Note that neural machine translation has made remarkable progress in recent two years, not to mention that production systems like Google translator perform much better than research systems. Therefore, the discoveries reported in this work are questionable. I strongly suggest the authors to conduct the studies base on the latest NMT architecture, i.e., Transformer.
	
	Furthermore, I checked the examples given in  introduction in Google translator and found no hallucination. So I'm not sure whether such hallucinations are really critical to today's NMT systems. I'd like to see that the study on some production translation systems, e.g., applying Algo 1 to Google translator and check its outputs, which can better motivate this work.
	
	For the analysis in Section 6.1, if attention is the root cause of hallucinations, some existing methods should have already address this issue. Can you check whether the model trained by the following work still suffers from hallucinations?
Modeling Coverage for Neural Machine Translation, ACL 16.