HI - CMLM : Improve CMLM with Hybrid Decoder Input (2021)

Abstract: Mask-predict CMLM (Ghazvininejad et al.,2019) has achieved stunning performance among non-autoregressive NMT models, but we find that the mechanism of predicting all of the target words only depending on the hidden state of [MASK] is not effective and efficient in initial iterations of refinement, resulting in ungrammatical repetitions and slow convergence. In this work, we mitigate this problem by combining copied source with embeddings of [MASK] in decoder. Notably. it’s not a straightforward copying that is shown to be useless, but a novel heuristic hybrid strategy — fence-mask. Experimental results show that it gains consistent boosts on both WMT14 En<->De and WMT16 En<->Ro corpus by 0.5 BLEU on average, and 1 BLEU for less-informative short sentences. This reveals that incorporating additional information by proper strategies is beneficial to improve CMLM, particularly translation quality of short texts and speeding up early-stage convergence.

## introduction in neural machine translation (nmt), autoregressive models decode tokens one-by-one: p(y |x) = t i p(y i |@xmath0, which ensures the robustness of intrinsic language models but slows down the inference. non-autoregressive models break the dependency between adjacent tokens: p(y |x) = t i p(y i |x), enabling to generate all outputs in parallel. recent years have witnessed impressive advances in non-autoregressive models, such as fully-nat and its variants @xcite @xcite , insertion-based models @xcite and iterative refinement models @xcite . mask-predict cmlm (cmlm) stands out of them owing to both significantly-fast inference and remarkable performance @xcite . it extends the masked language model @xcite and enables it to solving generation tasks with iterative refinement. in each step, the model decodes target conditioned on m well-predicted tokens with high confidence and (l -m)×[mask], where l is the length of target (see section 2 for details). this mechanism leads to the issue that it's liable to generate repeated tokens and slow down the convergence in early-stage iterations. we speculate this is because the proportion of useful tokens, i.e. @xmath1, is too small to provide enough information for the next step prediction. intuitively, the model tends to predict similar or even identical tokens when observing [mask] only and constantly. to alleviate this problem, we ameliorate cmlm by incorporating additional information from source embedding into the decoder input (hi-cmlm in short). experimental results show that it gains consistent boosts on both wmt14 en↔de and wmt16 en↔ro corpus by 0.5 bleu on average, and 1 bleu for less-informative short sentences. this reveals that incorporating additional information by proper strategies is beneficial to improving cmlm, particularly translation quality of short texts and speeding up the convergence of the first four iterations, compared with cmlm. ## method in this section, we propose three hybrid approaches to incorporate source embeddings and describe modifications on training strategy accordingly. ## conclusion we present hi-cmlm, an extension of cmlm by mixing source embedding with a hybrid strategy -fence replace, which can appropriately balance the information applied to the model. it achieves consistent improvements on two benchmarks in both directions, particularly short sentences.