Integrated Training for Sequence-to-Sequence Models Using Non-Autoregressive Transformer (2021)

Abstract: Complex natural language applications such as speech translation or pivot translation traditionally rely on cascaded models. However,cascaded models are known to be prone to error propagation and model discrepancy problems. Furthermore, there is no possibility of using end-to-end training data in conventional cascaded systems, meaning that the training data most suited for the task cannot be used. Previous studies suggested several approaches for integrated end-to-end training to overcome those problems, however they mostly rely on(synthetic or natural) three-way data. We propose a cascaded model based on the non-autoregressive Transformer that enables end-to-end training without the need for an explicit intermediate representation. This new architecture (i) avoids unnecessary early decisions that can cause errors which are then propagated throughout the cascaded models and (ii) utilizes the end-to-end training data directly. We conduct an evaluation on two pivot-based machine translation tasks, namely French→German and German→Czech. Our experimental results show that the proposed architecture yields an improvement of more than 2 BLEU for French→German over the cascaded baseline.

## introduction many complex natural language applications such as speech translation @xcite or pivot translation @xcite traditionally rely on cascaded models. the technique of model cascading is commonly used to solve problems that can be divided into a sequence of sub-problems where the solution to the first problem is used as an input to the second and so on. typically cascaded systems include several consecutive and independently trained models, each of which aims to solve a particular sub-task. for example in a cascaded speech translation system an automatic speech recognition model receives the audio signal as an input and generates a transcription as an output of the first sub-task. this output could be passed to a system that adds punctuation and capitalization to the sequence, before, as a final step, a machine translation system is applied. cascaded models are appealing if there is more training data for each of the sub-tasks than for the full task. examples for such scenarios include automatic speech translation (ast), image captioning in non-english languages, and non-english machine translation. however, cascaded models are prone to error propagation, meaning that decision errors in the first model are forwarded to and possibly amplified by the second model. usually, there is also a loss of information when passing information between models since the interface between models traditionally requires each model to output a discrete decision. this means that the deeper knowledge that the model may encode in its representation of the output is reduced to a 'surface form' of a particular prediction, which is passed on to the following model. lastly, in conventional cascaded system there is no possibility to make use of end-to-end training data, meaning that the training data most suited for the task cannot be used. to tackle these problems, several approaches for integrated end-to-end training of cascaded models have been proposed and applied to different nlp tasks @xcite @xcite . integrated end-to-end training is usually achieved by merging the consecutive models and fine-tuning the resulting system on the endto-end training data. although the idea of this approach is simple, it remains an open challenge how to choose the interface between the models in such a way that they can be trained, e.g. by gradient propagation. furthermore, most of these approaches rely on synthetic or natural multi-way training data, i.e. data that does not only provide an (input, output) pair but also the correct label for all sub-tasks involved. for a detailed discussion of the literature, we refer to section 2. in this work we focus on the task of pivot-based machine translation, i.e. the translation from a source (src) language via a pivot (piv) language to the desired target (trg) language, as an example for a two-stage task that is traditionally solved by model cascading. we propose a cascaded model based on the nonautoregressive transformer (nat) that enables endto-end training without the need for an explicit intermediate representation, that is inevitable in autoregressive models. this new architecture (i) avoids unnecessary early decisions that can cause errors which are then propagated throughout the cascaded models (ii) utilizes the @xmath0, @xmath1. ## related work several approaches were proposed in recent years to address the weaknesses of the traditional cascaded models. early works investigated the applications of the n-best list decoding both in speech translation and pivot-based translation @xcite @xcite . the n-best list decoding allows to pass multiple intermediate hypotheses and avoid unnecessary early decisions. an efficient alternative to the n-best list is lattices, which replaced the n-best list for the speech translation models @xcite @xcite . however, the usage of the discrete decisions does not allow to train cascaded model jointly on @xmath2. most recent works are focusing instead on the joint or integrated training for sequence-tosequence cascaded models. thus, @xcite suggested a joint training approach for the pivot-based neural machine translation. in their work, two attention-based rnn models @xcite are trained jointly with different connection terms in the objective function and the @xmath3. another approach is to apply the transfer-learning technique for pivotbased nmt @xcite , meaning that the direct @xmath4, and fine-tuned on @xmath5. pivot-based nmt is typically used in a low-resource @xmath6, and multilingual nmt systems proved to be successful in this scenario @xcite @xcite . to tackle a low-resource nmt problem, @xcite also explore different ways to extend the back-translation idea @xcite for @xmath7. however, since this work aims to provide the general framework for the integrated training of cascaded sequence-to-sequence models, we do not aim for comprehensive comparisons with multilingual nmt systems and various data augmentation strategies. we refer to @xcite for in-depth comparison studies. in speech translation, the tight model integration for the cascaded models also attracted attention from the community. @xcite @xcite discussed either use of attention or hidden state vectors as a connection interface for the tight model integration in cascaded systems. recently, @xcite proposed to use posterior distribution as an input to the encoder of the second model. ## model integration starting from the conventional cascaded model, as described in section 3.3, we propose to connect the two consecutive encoder-decoder models through an end-to-end trainable interface. the @xmath8, similarly the @xmath9. we introduce an interface which connects decoder s2p to the encoder p2t . the main requirement for this connection interface is to be differentiable to make the gradient propagation possible. in order to fulfill this requirement, we follow the previous work (see more in section 2) and choose to focus on two possible interfaces: hence, the decoder s2p and encoder p2t are connected through the softmax layer, as shown shown in figure 1b . note that the decoder posteriors interface requires the @xmath10. two autoregressive encoder-decoder models can be connected through these interfaces as shown in figure 2a. however, at training time the decoder s2p requires a pivot sequence as an input. if there is no access to the three-way @xmath11, the pivot sequence has to be obtained by doing a search in training, which is computationally very prohibitive in a real world task, or via forward or backward translation beforehand (synthetic data). the disadvantage of using synthetic data is that the pivot sequences remain static throughout the training, this means that the cascaded @xmath12. to avoid a sub-optimal, discrete intermediate representation while still benefit from the model integration, we propose to replace @xmath13. the usage of nat allows to replace the pivot sequence with a sequence of unknowns during the training on @xmath14. since the decoder states interface do not use the embeddings of the encoder p2t , similar to other works, the encoder p2t can be safely omitted in the integrated model (figure 2c ). training such a cascaded model can be done with the following steps: this yields a @xmath15, with the exception of one linear layer and an embedding matrix in the decoder states interface. please note that although we are focusing on pivot-based nmt as our target task, we argue that the proposed integration method can be easily adapted to any transformer-based cascaded model. ## experimental results to test and verify the proposed cascaded model, we conduct experiments on @xmath16. ## analysis 6.1 error propagation error propagation is a well-known problem of cascaded models. in the following we investigate how significantly errors in one model influence the following models. to this end, we monitor both the individual model performance and the end-to-end cascaded performance by running experiments on a three-way test set that consists of (source, pivot, target) triples. for that purpose, we extract 3000 overlapping sentences from newscommentary v14 for wmt @xmath17. we train a 6-layer 'base' transformer for @xmath18. in order to analyse the impact of disturbances and simulate errors in the @xmath19, we generate a weaker hypothesis by: @xmath20,2012} as dev and test respectively. results marked with † are taken from @xcite . the sentence with the best bleu score among ten candidates is then passed to the @xmath21. this cheating experiment results in an improvement of 6.2% absolute bleu on the @xmath22, which in turn however only results in 1.4% absolute bleu improvement on the cascaded @xmath23. we conclude that (i) the @xmath24.0 bleu. ## conclusion in this work, we propose a novel architecture for the integrated training of cascaded models based on a non-autoregressive transformer. we train the model on @xmath25, @xmath26, and @xmath27. moreover, it provides a natural inter-face between two transformer-based models and avoids unnecessary early decisions for intermediate representations. our experimental results on the task of pivot-based machine translations show that the nat-based integrated model outperforms the pivot baseline by up to 2.1% bleu on wmt @xmath28. we analyze the integrated model and conclude that the @xmath29. further work is required to apply established nat improvements to this new architecture, such as iterative decoding in the cascaded training and further experiments on knowledge distillation in the @xmath30, both of which show significant improvements in standalone systems @xcite @xcite . additionally, more sophisticated techniques for length modeling, such as an external length model or multiple length candidates, can be applied in the future to improve the quality of the pivot hypotheses. even though we test our cascaded architecture on the task for pivot-based machine translation, we can use the architecture in any application, where a combination of sequential models is beneficial.