One Source, Two Targets: C hallenges and Rewards of Dual Decoding (2021)

Abstract: Machine translation is generally understood as generating one target text from an input source document. In this paper, we consider a stronger requirement: to jointly generate two texts so that each output side effectively depends on the other. As we discuss, such a device serves several practical purposes, from multi-target machine translation to the generation of controlled variations of the target text. We present an analysis of possible implementations of dual decoding, and experiment with four applications. Viewing the problem from multiple angles allows us to better highlight the challenges of dual decoding and to also thoroughly analyze the benefits of generating matched, rather than independent, translations.

## introduction neural machine translation (nmt) is progressing at a rapid pace. since the introduction of the first encoder-decoder architecture @xcite , then completed with an attention mechanism @xcite , the performance of nmt systems is now good enough for a growing number of services, both for the general public and the translation industry. not only are neural translation systems better, they are also more versatile and have been extended in many ways to meet new application demands. this is notably the case with multilingual extensions @xcite @xcite , which aim to develop systems capable of processing multiple translation directions with one single model. another common situation for mt applications is the multi-source / multi-target scenario, where source documents in language s l need to be published in several target languages t 1 l , t 2 l , . . . . this is, for instance, what happens in multilingual institutions, or with crowdsourced translations of tv shows. the multi-source way @xcite f i could do that again if you want . e1 只要 你 愿意 我 可以 重复 一遍 。 e2 もう 一 回 やり ましょ う か e1 je peux le refaire si vous le voulez . e2 . voulez le vous si refaire le peux je e1 ich kann das noch mal machen , wenn sie wollen . e2 ich kann das noch mal machen , wenn du willst . an alternative, that we thoroughly explore here, is to simultaneously generate translations in t 1 l and t 2 l , an approach termed multi-target translation by @xcite . while the same goal is achieved with a multilingual system translating independently in t 1 l and t 2 l , several pay-offs are expected from a joint decoding: (a) improved disambiguation capacities (as for multi-source systems); (b) a better collaboration between the stronger and the weaker decoders; (c) more consistent translations in t 1 l and t 2 l than if they were performed independently. as it turns out, a dual decoder computing joint translations can be used for several other purposes, that we also consider: to simultaneously decode in two directions, providing a new implementation of the idea of @xcite ; @xcite ; to disentangle mixed language (code-switched) texts into their two languages @xcite ; finally, to generate coherent translation alternatives, an idea we use to compute polite and impolite variants of the same input @xcite . considering multiple applications allows us to assess the challenges and rewards of dual decoding under various angles and to better evaluate the actual agreement between the two decoders' outputs. our main contributions are the following: (i) a comparative study of architectures for dual decoding ( § 2); (ii) four short experimental studies where we use these architectures to simultaneously generate several outputs from one input ( § 3- § 6); (iii) practical remedies to the shortage of multi-parallel corpora that are necessary to implement multi-target decoding; (iv) concrete solutions to mitigate exposure bias between two decoders; (v) quantitative evaluations of the increased consistency incurred by a tight interaction between decoders. an additional empirical finding that is of practical value is the benefits of exploiting multi-parallel corpora to finetune multilingual systems. ## bi-directional mt bi-directional mt @xcite @xcite aims to integrate future information in the decoder by jointly translating in the forward (left to right, l2r) and in the backward (right to left, r2l) directions. another expectation is that the two decoders, having different views of the source, will deliver complementary translations. dual decoding readily applies in this setting, with one decoder for each direction, with the added benefit of generating more coherent outputs than independent decoders. we evaluate this added consistency by reusing the experimental setting (data, implementation and hyperparameters) of section 3, and by training 4 bi-directional systems, from english into german, french, chinese and japanese. similar to @xcite , we output the translation with the highest probability, inverting the translation if the r2l output is picked. we first train models on tri-parallel corpora obtained by adding an inverted version of the target sentence to each training sample. in this setting, the dual model again suffers a clear drop of bleu scores as compared to indep model ( by computing the bleu score between the two output translations, we can also evaluate the increment of consistency incurred in dual decoding. these scores are reported in ## mt for code-switched inputs in this section, we turn to a novel task, consisting in translating a code-switched (csw) sentence (containing fragments from two languages) simultaneously into its two components. an example in code-switching is an important phenomenon in informal communications between bilingual speakers. it generally consists of short inserts of a secondary language which are embedded within larger fragments in the primary language. when simultaneously translating into these two languages, we expect the following "copy" constraint to be satisfied: every word in the source text should appear in at least one of the two outputs. our main interest in this experiment is to assess how much dual decoding actually enforces this constraint. as tri-parallel corpora for this task are scarce @xcite , we mostly follow @xcite and automatically generate artificial csw sentences from regular parallel data. working with the en-fr pair, we use the wmt14 en-fr data to generate training data as well as a csw version of the newstest2014 test set. approximately half of the test sentences are mostly english with inserts in french, and mostly french with inserts in english for the other half. we use the same pretraining procedure as in section 3.2 and evaluate with csw-newstest2014 data. we also measure how well the constraint expressed above is satisfied. it stipulates that every model csw-en csw-fr second primary second primary base 67.8 67.4 35.5 97.4 37.5 95.3 indep 67.7 67.3 35.1 97.4 37.1 95.5 dual 67.7 67.5 35.1 97.5 37.5 95. 4 ## generating translation variants as a last application of dual decoding, we study the generation of pairs of consistent translation alternatives, using variation in "politeness" as our test bed. we borrow the experimental setting and data of @xcite . 17 the training set contains 5.58m sentences pairs, out of which 0.48m are annotated as polite and 1.06m as impolite. the rest is deemed neutral. 18 using this data, we generate triparallel data as follows. we first train a tag-based nmt with politeness control as in @xcite and use it to predict the polite counterpart of each impolite sentence, and vice-versa. we also include an equivalent number of randomly chosen neutral sentences: for these, the polite and impolite versions are identical. the resulting 3-way corpus contains 3.07m sentences. similar to the multitarget task (section 3), we fine-tune a pre-trained model with this data until convergence. we use the test data of @xcite as development set and test our model on the testyou set, which contains 2k sentences with a second-person pronoun you(r(s(elf))) in the english source. the annotation tool distributed with the data is used to assess the politeness of the output translations. as discussed in section 2.4.2, our dual decoder model can delay one decoder until the other is finished. we redo the same sequential decoding procedure as in section 3.4. results in ## related work the variety of applications considered here makes it difficult to give a thorough analysis of all the related work, and we only mention the most significant landmarks. multi-source / multi-target machine translation multi-source mt was studied in the framework of smt, considering with a tight integration (in the decoder), or a late integration (by combining multiple hypotheses obtained with different sources). this idea was revisited in the neural framework @xcite . setting multilingual mt aside @xcite , studies of the multi-target case are comparatively rarer @xcite . notable references are @xcite , which introduces a multi-task framework; @xcite , which studies ways to strengthen a basic multilingual decoder; while closer to our work, @xcite consider a dual decoder relying on dual self-attention mechanism. related techniques have also been used to simultaneously generate a transcript and a translation for a spoken input @xcite and to generate consistent caption and subtitle for an audio source @xcite . bi-directional decoding is an old idea from the statistical mt era @xcite . instantiations of these techniques for nmt are in @xcite , where asynchronous search techniques are considered; and in @xcite @xcite where, similar to our work, various ways to enforce a tighter interaction between directions are considered in synchronous search, while @xcite also study ways to increase the agreement between l2r and r2l directions. more recently, @xcite combines multi-target and bi-directional decoding within a single architecture, where, in each layer and block, all cross-attentions are combined with a single hidden state; four softmax layers are used for the output symbols in a proposal that creates an even stronger dependency between decoders than what we consider here. code-switching is an important linguistic phenomenon in bilingual communities that is getting momentum within the natural language processing communities @xcite . several tasks have been considered: token-level language identification @xcite , language modeling @xcite , named entity recognition @xcite , part-of-speech tagging @xcite and sentiment analysis @xcite . machine translation for csw texts is considered in @xcite . ## conclusion and future work in this paper, we have explored various possible implementations of dual decoding, as a way to generate pairs of consistent translation. dual decoding can be viewed as a tight form of multi-task learning, and, as we have seen, can be effectively trained using actual or partly artificial data; it can also directly benefit from pre-trained models. considering four applications of mt, we have observed that dual decoding was prone to exposure bias in the two decoders, and we have proposed practical remedies. using these, we have achieved bleu scores that match those of a simple multi-task learners, and display an increased level of consistency. in our future work, we plan to consider other strategies, such as scheduled sampling @xcite , to mitigate the exposure bias. another area where we seek to improve is the relaxation of strict synchronicity in decoding. we finally wish to study more applications of this technique, notably to generate controlled variation: controlling gender variation @xcite or more complex form of formality levels, as in @xcite , are obvious candidates.