Helping the Weak Makes You Strong: Simple Multi-Task Learning Improves Non-Autoregressive Translators (2022)

Abstract: Recently, non-autoregressive (NAR) neural machine translation models have received increasing attention due to their efficient parallel decoding.However, the probabilistic framework of NAR models necessitates conditional independence assumption on target sequences, falling short of characterizing human language data.This drawback results in less informative learning signals for NAR models under conventional MLE training, thereby yielding unsatisfactory accuracy compared to their autoregressive (AR) counterparts.In this paper, we propose a simple and model-agnostic multi-task learning framework to provide more informative learning signals.During training stage, we introduce a set of sufficiently weak AR decoders that solely rely on the information provided by NAR decoder to make prediction, forcing the NAR decoder to become stronger or else it will be unable to support its weak AR partners.Experiments on WMT and IWSLT datasets show that our approach can consistently improve accuracy of multiple NAR baselines without adding any additional decoding overhead.

## introduction state-of-the-art neural machine translation (nmt) systems are mainly autoregressive (ar) models @xcite , which decompose the joint probability of a sequence of tokens in a left-to-right order, modeling dependencies of each token with its preceding ones. despite having strong performance, such sequential decoding causes considerable latency, thereby unsatisfactory efficiency. in contrast, non-autoregressive (nar) translation models @xcite permit potentially more efficient parallel decoding. to do so, nar 1 code will be released at https://github.com/ wxy-nlp/multitasknat . figure 1 : illustration of our approach, where we introduce a set of auxiliary weak ar decoders, each of which must make its predictions solely relying on the information contained in the nar decoder hidden states. thus, the information provided by the nar decoder must be sufficiently useful for the ar decoders to be capable of predicting the target sequence because the ar decoders are parameterized as weakly as possible, which will in turn let the nar decoder learn to get stronger. a natural question may arise: can we encourage the nar decoder to learn from sources of signals that are more informative than that of the conditional independence assumption, in order to better capture target dependencies? it would be more advantageous if it is also modification-free regarding model architectures and could also used with all current nar systems. in this paper, we propose a simple multi-task learning framework that introduces auxiliary weak ar decoders to make nar models stronger. the key idea is that we parameterize the auxiliary ar decoders as weakly as possible and force them to predict target sequences solely based on the information from nar decoder's hidden representations, such that they can no longer model the target sequence on their own unless the knowledge provided by the nar decoder is sufficiently useful. as a result, the nar decoder has no choice but to become stronger so as to support the ar partners that are poorly parameterized. additionally, our approach is plug-and-play and model-agnostic, and the weak ar decoders that we introduce are discarded during the inference stage, resulting in no additional decoding overhead. we empirically evaluate its applications to several classes of nar model, including vanilla nar transformer @xcite and its ctcbased variant @xcite . experiments on widely-used wmt14 english-to-german, wmt16 englishto-romanian, and iwslt14 german-to-english benchmarks show that our approach consistently helps build more accurate nar models over strong baselines. ## preliminary neural machine translation (nmt) is formally defined as a conditional probability model p(y|x; θ) parameterized by deep neural networks θ. given an input sequence @xmath0, x 2 , • • • , x m ) , a neural autoregressive model @xcite predicts the target sequence @xmath1, y 2 , • • • , y n ) sequentially based on the conditional distribution, which decomposes p(y|x; θ) by the autoregressive factorization: ## methodology in this section, we will dive deep into our simple yet effective multi-task learning framework, including model architecture and training scheme. model architecture. the overall illustration of our approach is depicted in figure 1 . specifically, for every nar decoder layer, we introduce an auxiliary weak ar decoder, where each ar decoder is parameterized by one transformer layer, being as weak as possible. in this case, these ar decoders will no longer capture the underlying structure of target sequences on their own, unless their nar decoder layers can provide useful neural representations. as a result, the nar decoder layers can additionally learn from such informative task signals and become stronger to support the weak ar partners, being forced to capture sufficient context and dependency information. training objective. our training objective composes two parts for the nar model of interest and the auxiliary weak ar decoders, respectively. for the nar part, we keep the original model-specific training objective unaltered. for instance, we apply ctc loss for ctc-based nar models @xcite . as for the ar decoders, we apply the cross-entropy loss for training. the final loss is a weighted sum of the two components: ## conclusion in this paper, we propose a multi-task learning framework for nar. along with the training of the weak ar decoder, the nar hidden state will contain more contextual information, resulting in performance improvement. experiments on wmt and iwslt benchmarks show that our method can significantly and consistently improve the translation quality. when using beam search decoding, our ctc-based variant outperforms strong transformer on all of the benchmarks, while introducing no additional decoding overhead.