  Autoregressive sequence Generation models have achieved state-of-the-art
performance in areas like machine translation and image captioning. These
models are autoregressive in that they generate each word by conditioning on
previously generated words, which leads to heavy latency during inference.
Recently, non-autoregressive decoding has been proposed in machine translation
to speed up the inference time by generating all words in parallel. Typically,
these models use the word-level cross-entropy loss to optimize each word
independently. However, such a learning process fails to consider the
sentence-level consistency, thus resulting in inferior generation quality of
these non-autoregressive models. In this paper, we propose a simple and
efficient model for Non-Autoregressive sequence Generation (NAG) with a novel
training paradigm: Counterfactuals-critical Multi-Agent Learning (CMAL). CMAL
formulates NAG as a multi-agent reinforcement learning system where element
positions in the target sequence are viewed as agents that learn to
cooperatively maximize a sentence-level reward. On MSCOCO image captioning
benchmark, our NAG method achieves a performance comparable to state-of-the-art
autoregressive models, while brings 13.9x decoding speedup. On WMT14 EN-DE
machine translation dataset, our method outperforms cross-entropy trained
baseline by 6.0 BLEU points while achieves the greatest decoding speedup of
17.46x.
