Original: I MPROVING N ON -A UTOREGRESSIVE T RANSLATION M ODELS W ITHOUT D ISTILLATION
Revised: I MPROVING N ON -A UTOREGRESSIVE T RANSLATION M ODELS W ITHOUT D ISTILLATION

Original: Anonymous authors Paper under double-blind review
Revised: Huang Layer 6 AI {gary,felipe,maks}@layer6.ai

Original: A BSTRACT
Revised: A BSTRACT

Original: Transformer-based autoregressive (AR) machine translation models have achieved signiﬁcant performance improvements, nearing human-level accuracy on some languages.
Revised: Transformer-based autoregressive (AR) machine translation models have achieved signiﬁcant performance improvements, nearing human-level accuracy on some languages.

Original: The AR framework translates one token at a time which can be time consuming, especially for long sequences.
Revised: The AR framework translates one token at a time which can be time consuming, especially for long sequences.

Original: To accelerate inference, recent work has been exploring non-autoregressive (NAR) approaches that translate blocks of tokens in parallel.
Revised: To accelerate inference, recent work has been exploring non-autoregressive (NAR) approaches that translate blocks of tokens in parallel.

Original: Despite signiﬁcant progress, leading NAR models still lag behind their AR counterparts, and only become competitive when trained with distillation.
Revised: Despite signiﬁcant progress, leading NAR models still lag behind their AR counterparts, and only become competitive when trained with distillation.

Original: In this paper we investigate possible reasons behind this performance gap, namely, the indistinguishability of tokens, and mismatch between training and inference.
Revised: In this paper we investigate possible reasons behind this performance gap, namely, the indistinguishability of tokens, and mismatch between training and inference.

Original: We then propose the Conditional Masked Language Model with Correction (CMLMC) that addresses these problems.
Revised: We then propose the Conditional Masked Language Model with Correction (CMLMC) that addresses these problems.

Original: Empirically, we show that CMLMC achieves state-of-the-art NAR performance when trained on raw data without distillation and approaches AR performance on multiple datasets, a ﬁrst for NAR models.
Revised: Empirically, we show that CMLMC achieves stateof-the-art NAR performance when trained on raw data without distillation and approaches AR performance on multiple datasets.

Original: Full code will be released at the time of publication.
Revised: Code for this work is available here: https://github.com/layer6ai-labs/CMLMC .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural machine translation (NMT) models based on the Transformer architecture have achieved leading performance (Vaswani et al., 2017;
Revised: Neural machine translation (NMT) models based on the Transformer architecture have achieved leading performance (Vaswani et al., 2017;

Original: Barrault et al., 2019; Huang et al., 2020).
Revised: Barrault et al., 2019; Huang et al., 2020).

Original: Majority of the proposed approaches are based on the autoregressive (AR) principle, where translation is done one token at a time conditioning on already translated tokens.
Revised: Majority of the proposed approaches are based on the autoregressive (AR) principle, where translation is done one token at a time conditioning on already translated tokens.

Original: AR inference scales linearly with the number of tokens and full forward pass through the decoder is required for each translated token.
Revised: AR inference scales linearly with the number of tokens and full forward pass through the decoder is required for each translated token.

Original: This can be prohibitively expensive for long sequences, particularly as leading models are becoming increasingly larger in size.
Revised: This can be prohibitively expensive for long sequences, particularly as leading models are becoming increasingly larger in size.

Original: To mitigate this problem, recent works have explored the non-autoregressive (NAR) approach where subsets of tokens are translated in parallel (Gu et al., 2018;
Revised: To mitigate this problem, recent works have explored the non-autoregressive (NAR) approach where subsets of tokens are translated in parallel (Gu et al., 2018;

Original: Ghazvininejad et al., 2019; Kasai et al., 2020).
Revised: Ghazvininejad et al., 2019; Kasai et al., 2020).

Original: NAR models achieve signiﬁcantly faster inference speed that no longer depends on sequence length.
Revised: NAR models achieve signiﬁcantly faster inference speed that no longer depends on sequence length.

Original: However, despite considerable progress, leading
Revised: However, despite considerable progress, leading

Original: NAR models still require sequence-level knowledge distillation (Kim & Rush, 2016) to achieve competitive accuracy.
Revised: NAR models still require sequence-level knowledge distillation (Kim & Rush, 2016) to achieve competitive accuracy.

Original: In practice, a large AR Transformer model trained on the raw data is used as the teacher for distillation (Ghazvininejad et al., 2019).
Revised: In practice, a large AR Transformer model trained on the raw data is used as the teacher for distillation (Ghazvininejad et al., 2019).

Original: This process is expensive, as every new language pair requires training a new teacher.
Revised: This process is expensive, as every new language pair requires training a new teacher.

Original: It is also non-standard, and raises questions to the necessity and the underlying problems solved by distillation (Zhou et al., 2020; Ding et al., 2021).
Revised: It is also non-standard, and raises questions to the necessity and the underlying problems solved by distillation (Zhou et al., 2020; Ding et al., 2021).

Original: In this work we focus on one of the leading NAR approaches, the Conditional Masked Language
Revised: In this work we focus on one of the leading NAR approaches, the Conditional Masked Language

Original: Model (CMLM) (Ghazvininejad et al., 2019).
Revised: Model (CMLM) (Ghazvininejad et al., 2019).

Original: CMLM achieved state-of-the-art NAR performance on multiple machine translation benchmarks, especially when combined with semi-autoregressive training (Ghazvininejad et al., 2020b).
Revised: CMLM achieved leading NAR performance on multiple NMT datasets - especially when combined with semi-autoregressive training (Ghazvininejad et al., 2020b) -

Original: However, leading performance is onlyobtained when the model is trained on distilled data, without distillation CMLM’s performance drops signiﬁcantly below AR benchmarks.
Revised: but only when the model is trained on distilled data. Without distillation, CMLM performance drops signiﬁcantly below AR benchmarks.

Original: The need for distillation indicates that CMLM alone is unable to fully leverage the information available in the raw training data (Ding et al., 2021).
Revised: The need for distillation indicates that CMLM alone is unable to fully leverage the information available in the raw training data (Ding et al., 2021).

Original: Here, we identify two shortcomings of CMLM that, when addressed, signiﬁcantly improve NAR translation quality and narrow the gap between raw and distilled performance.
Revised: Here, we identify two shortcomings of CMLM that, when addressed, signiﬁcantly improve NAR translation quality and narrow the gap between raw and distilled performance.

Original: First, input token representations in CMLM can become nearly indistinguishable, especially for adjacent positions.
Revised: First, input token representations in CMLM can become nearly indistinguishable, especially for adjacent positions.

Original: In AR models this problem is avoided by a combination of causal masked attention, sequential inference, and learned positional encodings (PEs).
Revised: In AR models this problem is avoided by a combination of causal masked attention, sequential inference, and learned positional encodings (PEs).

Original: However,in CMLM the use of unmasked attention and simultaneous translation of token blocks  loses most of the information that distinguishes tokens.
Revised: However, unmasked attention and simultaneous translation of token blocks in CMLM loses most of the information that distinguishes tokens.

Original: This problem is particularly severe during the ﬁrst inference step, where translation isdone from a fully masked sentence.
Revised: This problem is particularly severe during the ﬁrst inference step, where the input is fully masked.

Original: The model  only relies on learned PEs to distinguish tokens, which is not sufﬁcient.
Revised: The model thus only relies on learned PEs to distinguish tokens, which is not sufﬁcient.

Original: Poor token separation can cause signiﬁcant translation errors, including the identiﬁed phenomenon of token repetition stemming from the related multi-modality problem (Zhou et al., 2020).
Revised: Poor token separation can cause signiﬁcant translation errors, including the identiﬁed phenomenon of token repetition stemming from the related multi-modality problem (Zhou et al., 2020).

Original: Second, there is a misalignment between CMLM’s training and inference procedures.
Revised: Second, there is a misalignment between CMLM’s training and inference procedures.

Original: During training
Revised: During training

Original: CMLM is optimized with a masked loss analogous to language model training in popular models such as BERT (Devlin et al., 2019).
Revised: CMLM is optimized with a masked loss analogous to language model training in popular models such as BERT (Devlin et al., 2019).

Original: However, CMLM inference always starts with a fully masked sentence and translates all tokens simultaneously.
Revised: However, CMLM inference always starts with a fully masked sentence and translates all tokens simultaneously.

Original: Iterative reﬁnement is then applied where subsets of low conﬁdence tokens are masked and re-translated at each iteration.
Revised: Iterative reﬁnement is then applied where subsets of low conﬁdence tokens are masked and re-translated at each iteration.

Original: During training the model rarely sees a fully masked sentence, and is not trained to self-correct from the initial fully masked translation that can contain signiﬁcant errors.
Revised: During training the model rarely sees a fully masked sentence, and is not trained to self-correct from the initial fully masked translation that can contain signiﬁcant errors.

Original: The misalignment between the two procedures can cause a disconnect, where optimization of the training loss does not transfer to improvements in translation quality.
Revised: The misalignment between the two procedures can cause a disconnect, where optimization of the training loss does not transfer to improvements in translation quality.

Original: In this work we propose the C onditional M asked L anguage M odel with C orrection (CMLMC).
Revised: In this work we propose the C onditional M asked L anguage M odel with C orrection (CMLMC).

Original: Our model builds on the CMLM architecture and addresses the aforementioned problems.
Revised: Our model builds on the CMLM architecture and addresses the aforementioned problems.

Original: We modify the decoder structure by exposing the positional encodings and incorporating causal attention layers to differentiate adjacent tokens.
Revised: We modify the decoder structure by exposing the positional encodings and incorporating causal attention layers to differentiate adjacent tokens.

Original: We also propose a novel correction loss that teaches the model how to correct translation mistakes made in early decoding iterations from the fully masked sentence.
Revised: We also propose a novel correction loss that teaches the model how to correct translation mistakes made in early decoding iterations from the fully masked sentence.

Original: With these improvements, CMLMC achieves new state-of-the-art undistilled NAR results and approaches
Revised: With these improvements, CMLMC achieves new state-of-the-art undistilled NAR results and approaches

Original: AR performance on multiple NMT benchmarks.
Revised: AR performance on multiple NMT benchmarks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Neural machine translation is a sequence to sequence prediction problem where a source sentence
Revised: Neural machine translation is a sequence to sequence prediction problem where a source sentence

Original: X = ( x 1 , . . .
Revised: X = ( x 1 , . . .

Original: , x m ) in one language is transformed into a target sentence Y =
Revised: , x m ) in one language is transformed into a target sentence Y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y n ) in another language.
Revised: , y n ) in another language.

Original: In AR setting this problem is modelled as: arg max Y (cid:81) ni
Revised: In AR setting this problem is modelled as: arg max Y (cid:81) ni

Original: =1 P ( y i |
Revised: =1 P ( y i |

Original: X, Y <i ) , where
Revised: X, Y <i ) , where

Original: Y <
Revised: Y <

Original: i =
Revised: i =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y i − 1 ) so every token is conditioned on all translated tokens before it (Cho et al., 2014).
Revised: , y i − 1 ) so every token is conditioned on all translated tokens before it (Cho et al., 2014).

Original: Greedy inference approach is typically taken where tokens are translated left-to-right sequentially, so a sequence of length n requires n forward passes through the model.
Revised: Greedy inference approach is typically taken where tokens are translated left-to-right sequentially, so a sequence of length n requires n forward passes through the model.

Original: Transformer-based AR models currently achieve leading accuracy on most NMT benchmarks (Barrault et al., 2019), but as models become larger linear inference can become prohibitively expensive for long sequences.
Revised: Transformer-based AR models currently achieve leading accuracy on most NMT benchmarks (Barrault et al., 2019), but as models become larger linear inference can become prohibitively expensive for long sequences.

Original: The NAR approach is proposed to mitigate this problem.
Revised: The NAR approach is proposed to mitigate this problem.

Original: Early work NAT (Gu et al., 2018) models the problem as: arg max Y P ( n | X ) (cid:81) ni
Revised: Early work NAT (Gu et al., 2018) models the problem as: arg max Y P ( n | X ) (cid:81) ni

Original: =1 P ( y i | X ) , where P ( n | X ) is the probability over sequence length.
Revised: =1 P ( y i | X ) , where P ( n | X ) is the probability over sequence length.

Original: Since P ( y i | X ) is independent of Y , inference can be parallelized and all tokens (or their subsets) can be translated simultaneously.
Revised: Since P ( y i | X ) is independent of Y , inference can be parallelized and all tokens (or their subsets) can be translated simultaneously.

Original: The conditional independence assumption can be overly restrictive and typically does not hold in natural language.
Revised: The conditional independence assumption can be overly restrictive and typically does not hold in natural language.

Original: To relax it, NAT adds positional attention and full self-attention between all tokens.
Revised: To relax it, NAT adds positional attention and full self-attention between all tokens.

Original: However, the performance of this model is still considerably worse than the AR benchmarks, which authors attribute to the multi-modality problem (Gu et al., 2018).
Revised: However, the performance of this model is still considerably worse than the AR benchmarks, which authors attribute to the multi-modality problem (Gu et al., 2018).

Original: NAT tackle this problem by applying knowledge distillation (Kim & Rush, 2016) on the training data, where a large AR model is used as a teacher.
Revised: NAT tackle this problem by applying knowledge distillation (Kim & Rush, 2016) on the training data, where a large AR model is used as a teacher.

Original: Distillation signiﬁcantly improves performance, and has since been widely adopted to reduce the performance gap to AR benchmarks.
Revised: Distillation signiﬁcantly improves performance, and has since been widely adopted to reduce the performance gap to AR benchmarks.

Original: Recently, multiple NAR approaches have been proposed that build on the work of NAT (Gu et al., 2018).
Revised: Recently, multiple NAR approaches have been proposed that build on the work of NAT (Gu et al., 2018).

Original: Flowseq (Ma et al., 2019) introduces latent variables and generative ﬂow to decouple token dependencies in the output.
Revised: Flowseq (Ma et al., 2019) introduces latent variables and generative ﬂow to decouple token dependencies in the output.

Original: CMLM (Ghazvininejad et al., 2019), the focus of our work, extends masked language loss training (Devlin et al., 2019) to NMT to facilitate more robust handling of target language input initialization and iterative reﬁnement.
Revised: CMLM (Ghazvininejad et al., 2019), the focus of our work, extends masked language loss training (Devlin et al., 2019) to NMT to facilitate more robust handling of target language input initialization and iterative reﬁnement.

Original: Several works build on CMLM to improve performance such as incorporating the energy of teacher optimization (Tu et al., 2020), adding raw data prior (Ding et al., 2021), and improving the encoder-decoder architecture (Kasai et al., 2021).
Revised: Several works build on CMLM to improve performance such as incorporating the energy of teacher optimization (Tu et al., 2020), adding raw data prior (Ding et al., 2021), and improving the encoder-decoder architecture (Kasai et al., 2021).

Original: DisCo (Kasai et al., 2020) utilizes attention masking instead of token masking to improve both training and inference efﬁciency.
Revised: DisCo (Kasai et al., 2020) utilizes attention masking instead of token masking to improve both training and inference efﬁciency.

Original: ReorderNAT (Ran et al., 2021) uses an explicit module to re-order the input sequence.
Revised: ReorderNAT (Ran et al., 2021) uses an explicit module to re-order the input sequence.

Original: GLAT (Haviv et al., 2021a) compares intermediate predictions with ground truth to dynamically adjust the masking percentage according to translation difﬁculty.
Revised: GLAT (Qian et al., 2021) compares intermediate predictions with ground truth to dynamically adjust the masking percentage according to translation difﬁculty.

Original: Recent work, including AXE (Ghazvininejad et al., 2020a), Imputer (Saharia et al., 2020), FullyNAT (Gu & Kong, 2021), and OAXE (Du et al., 2021) achieved improved performance with a CTC-like alignment loss (Libovický & Helcl, 2018) that has been found to be more suitable for  NAR tasks (Haviv et al., 2021b).
Revised: Recent work, including AXE (Ghazvininejad et al., 2020a), Imputer (Saharia et al., 2020), FullyNAT (Gu & Kong, 2021), and OAXE (Du et al., 2021) achieve improved performance with the alignment loss (Libovický & Helcl, 2018) that has been found to be more suitable for one-step NAR tasks (Haviv et al., 2021).

Original: This  direction is orthogonal to our focus in this paper, and can be readily incorporated in future work.
Revised: This important direction is orthogonal to our focus in this paper, and can be readily incorporated in future work.

Original: The closest to our work is the SMART model (Ghazvininejad et al., 2020b) which also focuses on the training-inference gap in NAR training; we discuss the key differences between our approach and SMART in Section 3.2.
Revised: The closest to our work is the SMART model (Ghazvininejad et al., 2020b) which also focuses on the training-inference gap in NAR training; we discuss the key differences between our approach and SMART in Section 3.2.

Original: Finally, it is important to note that while signiﬁcant progress has been made, the majority of work in NAR has focused on maximizing the distilled performance.
Revised: Finally, it is important to note that while signiﬁcant progress has been made, the majority of work in NAR has focused on maximizing the distilled performance.

Original: Despite its importance, undistilled performance  remains signiﬁcantly below leading AR baselines and is the focus on this work.
Revised: Despite its importance, NAR performance on raw data remains signiﬁcantly below leading AR baselines and is the focus on this work.

Original:  CMLMC: C ONDITIONAL M ASKED L ANGUAGE M ODEL WITH C ORRECTION
Revised:  CMLMC: C ONDITIONAL M ASKED L ANGUAGE M ODEL WITH C ORRECTION

Original: In this section, we analyze the CMLM architectures and identify several weaknesses in its design, namely, lack of token distinguishability and training-inference mismatch.
Revised: In this section, we analyze the CMLM architecture and identify several weaknesses in its design, namely, lack of token distinguishability and training-inference mismatch.

Original: We then propose the
Revised: We then propose the

Original: CMLMC model, which addresses these problems leading to signiﬁcant improvements in accuracy.
Revised: CMLMC model, which addresses these problems leading to signiﬁcant improvements in accuracy.

Original: 3. T OKEN D ISTINGUISHABILITY
Revised: 3. T OKEN D ISTINGUISHABILITY

Original: To generate accurate translations the model must be able to align the source and translated sentences.
Revised: To generate accurate translations the model must be able to align the source and translated sentences.

Original: This is particularly important in Transformer-based architectures where source sentence is typically encoded as memory, and decoder attends over memory during translation.
Revised: This is particularly important in Transformer-based architectures where source sentence is typically encoded as memory, and decoder attends over memory during translation.

Original: To infer which part(s) of the memory to attend for a given translated token, the decoder needs to know exactly where the token is in the translated sequence and its relative position to other translated tokens.
Revised: To infer which part(s) of the memory to attend for a given translated token, the decoder needs to know exactly where the token is in the translated sequence and its relative position to other translated tokens.

Original: Transformer attention doesn’t preserve positional information so in AR models this is achieved by a combination of causal masked attention, positional encodings (PEs), and sequential inference.
Revised: Transformer attention doesn’t preserve positional information so in AR models this is achieved by a combination of causal masked attention, positional encodings (PEs), and sequential inference.

Original: [Figure]
Revised: [Figure]

Original: Causal masked attention matrix has a lowertriangularshape so only attends before  learned) addedand enable the model to infer the relative and absolute distance between tokens (Yan et al., 2019).
Revised: Causal masked attention matrix has a lowertriangular attends before or learned) added model to infer the relative and absolute distance between tokens (Yan et al., 2019).

Original: Finally, sequential inference translates one token at a time conditioning on all previously translated tokens, so the model can identify exactly which token is being translated at each step.
Revised: Finally, sequential inference translates one token at a time conditioning on all previously translated tokens, so the model can identify exactly which token is being translated at each step.

Original: Jointly these design choices lead to adequate separation between tokens and provide strong positional information.
Revised: Jointly these design choices lead to adequate separation between tokens and provide strong positional information.

Original: In NAR models, parallel translation signiﬁcantly complicates sentence alignment.
Revised: In NAR models, parallel translation signiﬁcantly complicates sentence alignment.

Original: In particular,
Revised: In particular,

Original: CMLM replaces causal masked attention with a full attention over all tokens.
Revised: CMLM replaces causal masked attention with a full attention over all tokens.

Original: Unmasked attention combined with parallel inference over all tokens implies that the model has to rely solely on PEs for position information.
Revised: Unmasked attention combined with parallel inference over all tokens implies that the model has to rely solely on PEs for position information.

Original: If learned PEs for different positions are very similar to each other, CMLM would not be able to distinguish them.
Revised: If learned PEs for different positions are very similar to each other, CMLM would not be able to distinguish them.

Original: Token indistinguishability is especially problematic during the ﬁrst inference step since CMLM inference always starts with a fully masked sentence.
Revised: Token indistinguishability is especially problematic during the ﬁrst inference step since CMLM inference always starts with a fully masked sentence.

Original: During this step, all tokens are replaced with the same mask embedding < M > summed into PEs.
Revised: During this step, all tokens are replaced with the same mask embedding < M > summed into PEs.

Original: The model thus has to translate the entire sentence using only PEs to distinguish the input tokens.
Revised: The model thus has to translate the entire sentence using only PEs to distinguish the input tokens.

Original: Figure 1(a) shows binned cosine similarity between unique pairs of well-trained PE encodings learned by the CMLM model on IWSLT’14 De-En dataset.
Revised: Figure 1(a) shows binned cosine similarity between unique pairs of trained PE encodings learned by the CMLM model on IWSLT’14 De-En dataset.

Original: We see that the similarity distribution has a long tail on the positive side.
Revised: We see that the similarity distribution has a long tail on the positive side.

Original: Signiﬁcant number of PE pairs have cosine similarity above 0.5, and over 200 pairs have similarity greater than 0.7.
Revised: Signiﬁcant number of PE pairs have cosine similarity above 0.5, and over 200 pairs have similarity greater than 0.7.

Original: These results indicate that during inference with a fully masked sentence, some token pairs can be nearly indistinguishable for CMLM.
Revised: These results indicate that during inference with a fully masked sentence, some token pairs can be nearly indistinguishable for CMLM.

Original: This can lead to severe translation errors, the most direct of which is token repetition where indistinguishable tokens get translated to the same word.
Revised: This can lead to severe translation errors, the most direct of which is token repetition where indistinguishable tokens get translated to the same word.

Original: Figure 1(b) shows the distribution of consecutive token repetitions across test sentences on the same dataset.
Revised: Figure 1(b) shows the distribution of consecutive token repetitions across test sentences on the same dataset.

Original: We see that CMLM can produce over 5 repetitions in a translated sentence, and hundreds of sentences have 3 or more repetitions.
Revised: We see that CMLM can produce over 5 repetitions in a translated sentence, and hundreds of sentences have 3 or more repetitions.

Original: It is evident that CMLM has difﬁculty translating from fully masked sentence, and bad initial translation can be difﬁcult to correct even with multiple steps of iterative reﬁnement.
Revised: It is evident that CMLM has difﬁculty translating from fully masked sentence, and bad initial translation can be difﬁcult to correct even with multiple steps of iterative reﬁnement.

Original: We address the problem of token distinguishability with a dual strategy.
Revised: We address the problem of token distinguishability with a dual strategy.

Original: First, each decoder block in CMLM is augmented with a causal masked attention layer, inserted between the unmasked self attention and encoder attention layers, leading to the following block structure: FULL-ATT →
Revised: First, each decoder block in CMLM is augmented with a causal masked attention layer, inserted between the unmasked self attention and encoder attention layers, leading to the following block structure: FULL-ATT →

Original: MASKED-ATT → ENCODER-ATT → FFN .
Revised: MASKED-ATT → ENCODER-ATT → FFN .

Original: Re-introducing masked attention brings back the left-to-right sequential hierarchy and breaks symmetry between positions with similar PEs.
Revised: Re-introducing masked attention brings back the left-to-right sequential hierarchy and breaks symmetry between positions with similar PEs.

Original: We deliberately insert this layer before the encoder attentions, as they align source and translated sentences, so properly separating input tokens is particularly important here.
Revised: We deliberately insert this layer before the encoder attention, as it aligns source and translated sentences, so properly separating input tokens is particularly important there.

Original: We use the standard masked attention layer where the upper triangular portion of the SoftMax matrix is set to 0.
Revised: We use the standard masked attention layer where the upper triangular portion of the SoftMax matrix is set to 0.

Original: Second, we modify how token embeddings and PEs are combined.
Revised: Second, we modify how token embeddings and PEs are combined.

Original: The majority of transformer-based models simply sum the two embeddings.
Revised: The majority of transformer-based models simply sum the two embeddings.

Original: However, this can be insufﬁcient to propagate the positional information to the upper layers since the model has to balance the scales of both embeddings.
Revised: However, this can be insufﬁcient to propagate the positional information to the upper layers since the model has to balance the scales of both embeddings.

Original: This can again be particularly problematic during the fully masked inference step when learned mask embedding < M > is added to PEs.
Revised: This can again be particularly problematic during the fully masked inference step when learned mask embedding < M > is added to PEs.

Original: < M > is shared across all masked positions, and in training receives gradient updates that are typically larger in magnitude than individual PEs.
Revised: < M > is shared across all masked positions, and in training receives gradient updates that are typically larger in magnitude than individual PEs.

Original: Empirically we found that gradient magnitude for < M > can be up to 50x larger than the average gradient magnitude for ﬁrst 100 most commonly updated tokens throughout training.
Revised: Empirically we found that gradient magnitude for < M > can be up to 50x larger than the average gradient magnitude for ﬁrst 100 most commonly updated tokens throughout training.

Original: This can in turn make the magnitude of < M > much larger than PE, making it difﬁcult for the model to distinguish PEs at different positions once they are summed with < M > .
Revised: This can in turn make the magnitude of < M > much larger than PE, making it difﬁcult for the model to distinguish PEs at different positions once they are summed with < M > .

Original: To deal with this problem we instead propose to combine token embeddings and PEs with a feed-forward layer (FFN).
Revised: To deal with this problem we instead propose to combine token embeddings and PEs with a feed-forward layer (FFN).

Original: Formally, given an input token embedding y i and positional encoding pe i at position i , we ﬁrst expand by concatenating them and then shrink back down with an FFN: y (cid:48)
Revised: Formally, given an input token embedding y i and positional encoding pe i at position i , we ﬁrst expand by concatenating them and then shrink back down with an FFN: y (cid:48)

Original: i = FFN ([ y i , pe i ]) .
Revised: i = FFN ([ y i , pe i ]) .

Original: Here, [ · , · ] is the concatenation operation and FFN :
Revised: Here, [ · , · ] is the concatenation operation and FFN :

Original: R 2 d → R d where d is the original embedding dimension.
Revised: R 2 d → R d where d is the original embedding dimension.

Original: FFN enables the model to appropriately adjust the embedding scales and (de-)emphasize speciﬁc dimensions.
Revised: FFN enables the model to appropriately adjust the embedding scales and (de-)emphasize speciﬁc dimensions.

Original: The joint effect of these architecture modiﬁcations is shown in Figure 1.
Revised: The joint effect of these architecture modiﬁcations is shown in Figure 1.

Original: We refer to this method as RevealPosition.
Revised: We refer to this method as RevealPosition.

Original: From Figure 1(a)
Revised: From Figure 1(a)

Original: we see that PE cosine similarity distribution is now mainly clustered around 0, and no pair has similarity above 0.5.
Revised: we see that PE cosine similarity distribution is now mainly clustered around 0, and no pair has similarity above 0.5.

Original: This is a signiﬁcant improvement from
Revised: This is a signiﬁcant improvement from

Original: CMLM where hundreds of PE pairs have cosine similarity above 0.7.
Revised: CMLM where hundreds of PE pairs have cosine similarity above 0.7.

Original: Furthermore, from Figure 1(b) we also see that RevealPosition consistently reduces the number of sentences that have repetitions across all repetition counts.
Revised: Furthermore, from Figure 1(b) we also see that RevealPosition consistently reduces the number of sentences that have repetitions across all repetition counts.

Original: In particular, the most frequent one and two token repetitions are reduced by over 30% and 35% respectively.
Revised: In particular, the most frequent one and two token repetitions are reduced by over 30% and 35% respectively.

Original: In the experiments section we further show that these relatively simple modiﬁcations lead to considerable improvements in BLEU of over 1 point.
Revised: In the experiments section we further show that these relatively simple modiﬁcations lead to considerable improvements in BLEU of over 1 point.

Original: However, despite the reduction in repetitions, RevealPosition can still makeconsiderable mistakes, so in the next section we proposed a new loss function that aims to teach the model how to correct them.
Revised: However, despite the reduction in repetitions, RevealPosition can still make mistakes, so in the next section we proposed a new loss function that aims to teach the model how to correct them.

Original: 3. T RAINING /I NFERENCE M ISMATCH
Revised: 3. T RAINING /I NFERENCE M ISMATCH

Original: It is generally accepted that training and inference procedures should match as closely as possible (Ranzato et al., 2015; Mihaylova & Martins, 2019).
Revised: It is generally accepted that training and inference procedures should match as closely as possible (Ranzato et al., 2015; Mihaylova & Martins, 2019).

Original: When this is the case, improving training loss during optimization typically translates to better inference performance.
Revised: When this is the case, improving training loss during optimization typically translates to better inference performance.

Original: In AR models, training and inference are well aligned, in both cases the model translates one token at a time conditioned on all previously translated tokens.
Revised: In AR models, training and inference are well aligned, in both cases the model translates one token at a time conditioned on all previously translated tokens.

Original: The only major difference is that during training previously translated tokens are ﬁxed to ground truth, but during inference they are set to model predictions.
Revised: The only major difference is that during training previously translated tokens are ﬁxed to ground truth, but during inference they are set to model predictions.

Original: This is not the case in many NAR models, including CMLM.
Revised: This is not the case in many NAR models, including CMLM.

Original: The training strategy in CMLM follows a masked language model approach similar to BERT (Devlin et al., 2019) pretraining: a random subset of tokens from the target sentence Y get masked, effectively splitting Y into masked tokens
Revised: The training strategy in CMLM follows a masked language model approach similar to BERT (Devlin et al., 2019) pretraining: a random subset of tokens from the target sentence Y get masked, effectively splitting Y into masked tokens

Original: Y mask and observed ground truth tokens Y obs ; masked tokens are replaced with the mask embedding < M > , and the sentence is passed through the decoder to get predictions for Y mask .
Revised: Y mask and observed ground truth tokens Y obs ; masked tokens are replaced with the mask embedding < M > , and the sentence is passed through the decoder to get predictions for Y mask .

Original: The loss function then aims to maximize the probability of masked tokens for reconstruction accuracy.
Revised: The loss function then aims to maximize the probability of masked tokens for reconstruction accuracy.

Original: Unlike the ﬁxed 15% masking typically used in BERT, CMLM uses a more aggressive strategy, sampling the number of masked tokens uniformly between 1 and sentence length.
Revised: Unlike the ﬁxed 15% masking typically used in BERT, CMLM uses a more aggressive strategy, sampling the number of masked tokens uniformly between 1 and sentence length.

Original: During inference CMLM always starts with a fully masked sentence and translates all tokens.
Revised: During inference CMLM always starts with a fully masked sentence and translates all tokens.

Original: Iterative reﬁnement are then applied, where a subset of tokens with the lowest probability is re-masked and re-translated at every iteration.
Revised: Iterative reﬁnement are then applied, where a subset of tokens with the lowest probability is re-masked and re-translated at every iteration.

Original: Comparing these training and inference procedures we clearly see a mismatch.
Revised: Comparing these training and inference procedures we can see a mismatch.

Original: During training CMLM is not trained to correct its predictions.
Revised: During training CMLM is not trained to correct its predictions.

Original: In particular, it is not trained to recognize the errors made in the crucial ﬁrst inference step.
Revised: In particular, it is not trained to recognize the errors made in the crucial ﬁrst inference step.

Original: As we discussed in the previous section, translations from this ﬁrst step can have signiﬁcant errors (repeated tokens etc.), so learning to correct these errors can be critical to model performance.
Revised: As we discussed in the previous section, translations from this ﬁrst step can have signiﬁcant errors (repeated tokens etc.), so learning to correct these errors can be critical to model performance.

Original: To better align training and inference, we build an error correction mechanism into our model by adding a correction loss term during training.
Revised: To better align training and inference, we build an error correction mechanism into our model by adding a correction loss term during training.

Original: This term focuses on correcting mistakes after inference with fully masked input.
Revised: This term focuses on correcting mistakes after inference with fully masked input.

Original: Formally, given a source sentence X and a target sentence Y , we ﬁrst split Y into Y obs and Y mask as in CMLM training.
Revised: Formally, given a source sentence X and a target sentence Y , we ﬁrst split Y into Y obs and Y mask as in CMLM training.

Original: We then apply the decoder to the fully masked sentence Y ∅ , where every token is replaced with < M > , obtaining
Revised: We then apply the decoder to the fully masked sentence Y ∅ , where every token is replaced with < M > , obtaining

Original: TransformerEncoder + memory +
Revised: TransformerEncoder + memory +

Original: CMLMCDecoder <
Revised: CMLMCDecoder <

Original: M> <M> <M> <M> +
Revised: M> <M> <M> <M> +

Original: CMLMCDecoder +
Revised: CMLMCDecoder +

Original: CMLMCDecoder we work <M> NLP -- -- on -- work work <M> NLP work work on on --
Revised: CMLMCDecoder we work <M> NLP -- -- on -- work work <M> NLP work work on on --

Original: Ground truth
Revised: Ground truth

Original: Model Prediction
Revised: Model Prediction

Original: Substitution Token wir arbeiten an NLP -- -- we
Revised: Substitution Token wir arbeiten an NLP -- -- we

Original: ˆ Y ∅ =
Revised: ˆ Y ∅ =

Original: Decoder ( X, Y ∅ ) .
Revised: Decoder ( X, Y ∅ ) .

Original: Here, ˆ Y ∅ is identical to the output of the ﬁrst inference step, where all tokens are predicted at once and can contain signiﬁcant errors.
Revised: Here, ˆ Y ∅ is identical to the output of the ﬁrst inference step, where all tokens are predicted at once and can contain signiﬁcant errors.

Original: With probability p we replace each token y ∈ Y obs with a corresponding predicted token ˆ y ∈
Revised: With probability p we replace each token y ∈ Y obs with a corresponding predicted token ˆ y ∈

Original: ˆ Y ∅ .
Revised: ˆ Y ∅ .

Original: This splits Y obs into two sets: Y pred , where tokens are substituted with predictions from ˆ Y ∅ , and the remaining portion Y obs \ Y pred .
Revised: This splits Y obs into two sets: Y pred , where tokens are substituted with predictions from ˆ Y ∅ , and the remaining portion Y obs \ Y pred .

Original: Our correction loss then aims to predict the tokens in Y pred :
Revised: Our correction loss then aims to predict the tokens in Y pred :

Original: [Equation]
Revised: [Equation]

Original: To compute this loss we make a forward pass through the decoder with a sentence that now contains mask Y mask , predictions Y pred , and unmasked ground truth tokens Y obs \ Y pred .
Revised: To compute this loss we make a forward pass through the decoder with a sentence that now contains mask Y mask , predictions Y pred , and unmasked ground truth tokens Y obs \ Y pred .

Original: The model generates predictions for every substituted token in Y pred , and we maximize the probability of the corresponding ground truth tokens.
Revised: The model generates predictions for every substituted token in Y pred , and we maximize the probability of the corresponding ground truth tokens.

Original: This process approximates the correction procedure, where model re-translates a subset of unmasked tokens by conditioning on current translation.
Revised: This process approximates the correction procedure, where model re-translates a subset of unmasked tokens by conditioning on current translation.

Original: Early self-correction steps primarily condition on tokens from ˆ Y ∅ obtained during the initial pass.
Revised: Early self-correction steps primarily condition on tokens from ˆ Y ∅ obtained during the initial pass.

Original: As shown in (Ghazvininejad et al., 2019), these early steps are very important and lead to large improvements in BLEU.
Revised: As shown in (Ghazvininejad et al., 2019), these early steps are very important and lead to large improvements in BLEU.

Original: So the correction loss L corr aims to gradually optimize the model for this process during training.
Revised: So the correction loss L corr aims to gradually optimize the model for this process during training.

Original: In addition to L corr , we keep the original mask prediction loss from CMLM:
Revised: In addition to L corr , we keep the original mask prediction loss from CMLM:

Original: [Equation]
Revised: [Equation]

Original: Note that Y pred is dropped from this loss and conditioning is done on ground truth tokens in Y obs .
Revised: Note that Y pred is dropped from this loss and conditioning is done on ground truth tokens in Y obs .

Original: The ﬁnal loss that we use in our model is a combination of correction and mask prediction losses:
Revised: The ﬁnal loss that we use in our model is a combination of correction and mask prediction losses:

Original: [Equation]
Revised: [Equation]

Original: Figure 2 illustrates how the joint loss is computed for an example German to English translation.
Revised: Figure 2 illustrates how the joint loss is computed for an example German to English translation.

Original: The mismatch between NAR training and inference procedures is also recognized by SMART (Ghazvininejad et al., 2020b).
Revised: The mismatch between NAR training and inference procedures is also recognized by SMART (Ghazvininejad et al., 2020b).

Original: Similarly to our approach, SMART applies the decoder during training to generate predictions for a subset of tokens and then learns to self-correct these predictions while simultaneously reconstructing masked tokens.
Revised: Similarly to our approach, SMART applies the decoder during training to generate predictions for a subset of tokens and then learns to self-correct these predictions while simultaneously reconstructing masked tokens.

Original: Our approach however has two key differences.
Revised: Our approach however has two key differences.

Original: First, SMART generates token predictions from a partially masked sentence { Y mask , Y obs } .
Revised: First, SMART generates token predictions from a partially masked sentence { Y mask , Y obs } .

Original: While this does teach the model to self-correct, the decoder always sees some ground truth tokens Y obs as input.
Revised: While this does teach the model to self-correct, the decoder always sees some ground truth tokens Y obs as input.

Original: This procedure thus does not address the key ﬁrst inference step where tokens are predicted from the fully masked sentence with no ground truth input.
Revised: This procedure thus does not address the key ﬁrst inference step where tokens are predicted from the fully masked sentence with no ground truth input.

Original: We discussed that the most signiﬁcant mistakes are made during this step (also empirically shown in CMLM) so learning to correct them is particularly important.
Revised: We discussed that the most signiﬁcant mistakes are made during this step (also empirically shown in CMLM) so learning to correct them is particularly important.

Original: We consequently argue that our approach to instead apply the decoder to fully masked sentence Y ∅ for self-correction loss better addresses the training/inference gap.
Revised: We consequently argue that our approach to instead apply the decoder to fully masked sentence Y ∅ for self-correction loss better addresses the training/inference gap.

Original: Second, SMART uses the same input with masked tokens Y mask and decoder predicted tokens Y pred for both mask reconstruction and self-correction tasks.
Revised: Second, SMART uses the same input with masked tokens Y mask and decoder predicted tokens Y pred for both mask reconstruction and self-correction tasks.

Original: This can potentially hamper training since to predict masked tokens decoder also conditions on Y pred which can contain signiﬁcant errors especially early in optimization.
Revised: This can potentially hamper training since to predict masked tokens decoder also conditions on Y pred which can contain signiﬁcant errors especially early in optimization.

Original: To preserve clean signal in masked training we only condition on ground truth tokens
Revised: To preserve clean signal in masked training we only condition on ground truth tokens

Original: Y obs in L mask .
Revised: Y obs in L mask .

Original: Empirically, we demonstrate that our approach leads to signiﬁcant improvements in accuracy particularly on undistilled data.
Revised: Empirically, we demonstrate that our approach leads to signiﬁcant improvements in accuracy particularly on undistilled data.

Original: 3. CMLMC T RAINING AND I NFERENCE
Revised: 3. CMLMC T RAINING AND I NFERENCE

Original: [Table]
Revised: [Table]

Original: In previous sections we introduced modiﬁcations to CMLMarchitecture that improve token distinguishability and better align training with inference.
Revised: In previous sections we introduced modiﬁcations to CMLM that improve token distinguishability and better align training with inference.

Original: Combined they form our CMLMC approach.
Revised: Combined they form our CMLMC approach.

Original: Algorithm 1 outlines the training procedure for CMLMC.
Revised: Algorithm 1 outlines the training procedure for CMLMC.

Original: Given a training corpus S
Revised: Given a training corpus S

Original: = {
Revised: = {

Original: ( X i , Y i ) } i of source and target sentence pairs, for each pair ( X, Y )
Revised: ( X i , Y i ) } i of source and target sentence pairs, for each pair ( X, Y )

Original: we ﬁrst sample mask Y mask uniformly from the interval [1 , | Y | ] .
Revised: we ﬁrst sample mask Y mask uniformly from the interval [1 , | Y | ] .

Original: We then compute correction and mask losses L corr and L mask as outlined in Section 3.2, and update model parameters using gradients.
Revised: We then compute correction and mask losses L corr and L mask as outlined in Section 3.2, and update model parameters using gradients.

Original: During inference we follow the same iterative approach used in CMLM.
Revised: During inference we follow the same iterative approach used in CMLM.

Original: First, a fully masked sentence Y ∅ is passed through the decoder that translates all tokens at once to get ˆ Y (1) ∅ .
Revised: First, a fully masked sentence Y ∅ is passed through the decoder that translates all tokens at once to get ˆ Y (1) ∅ .

Original: In subsequent iterations t ≥ 1 , a subset of tokens with the lowest likelihood in ˆ Y ( t ) ∅ is masked and re-translated by making a pass through the decoder to get ˆ Y ( t +1) ∅ .
Revised: In subsequent iterations t ≥ 1 , a subset of tokens with the lowest likelihood in ˆ Y ( t ) ∅ is masked and re-translated by making a pass through the decoder to get ˆ Y ( t +1) ∅ .

Original: The length of
Revised: The length of

Original: Y ∅ is predicted by the special LENGTH token that is appended to the source sentence in the encoder (Ghazvininejad et al., 2019).
Revised: Y ∅ is predicted by the special LENGTH token that is appended to the source sentence in the encoder (Ghazvininejad et al., 2019).

Original: We repeat the inference procedure for top- k predicted lengths and select the translation with the highest average token probability, analogous to using beam search in
Revised: We repeat the inference procedure for top- k predicted lengths and select the translation with the highest average token probability, analogous to using beam search in

Original: AR models.
Revised: AR models.

Original: [Equation]

Original:   XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our approach on multiple public NMT datasets: IWSLT’14 De-En/En-De, WMT’14 DeEn/En-De, and WMT’16 Ro-En/En-Ro.
Revised: We evaluate our approach on multiple public NMT datasets: IWSLT’14 De-En/En-De, WMT’14 DeEn/En-De, and WMT’16 Ro-En/En-Ro.

Original: We use the same training/validation/test sets as in previous work (Ghazvininejad et al., 2019) and report test set performance in BLEU for direct comparison.
Revised: We use the same training/validation/test sets as in previous work (Ghazvininejad et al., 2019) and report test set performance in BLEU for direct comparison.

Original: For each dataset we compute performance on both raw and distilled settings, resulting in 12 dataset in total.
Revised: For each dataset we compute performance on both raw and distilled settings, resulting in 12 dataset in total.

Original: Distillation is done using the Transformer small/base/large AR model (Vaswani et al., 2017) for the IWSLT’14/WMT’16/WMT’14 datasets respectively, accounting for model capacity and dataset size.
Revised: Distillation is done using the Transformer small/base/large AR model (Vaswani et al., 2017) for the IWSLT’14/WMT’16/WMT’14 datasets respectively, accounting for model capacity and dataset size.

Original: We compare CMLMC against leading NAR baselines including, Flowseq (Ma et al., 2019),
Revised: We compare CMLMC against leading NAR baselines including, Flowseq (Ma et al., 2019),

Original: CMLM and its variants, CMLM+SMART and CMLM+RawPrior (Ghazvininejad et al., 2019; 2020b;
Revised: CMLM and its variants, CMLM+SMART and CMLM+RawPrior (Ghazvininejad et al., 2019; 2020b;

Original: Ding et al., 2021), LevenshteinNAR (Gu et al., 2019), DisCo (Kasai et al., 2020), ENGINE (Tu et al., 2020), ReorderNAT (Ran et al., 2021), GLAT (Haviv et al., 2021a), as well as the models using the alignment losses.
Revised: Ding et al., 2021), LevenshteinNAR (Gu et al., 2019), DisCo (Kasai et al., 2020), ENGINE (Tu et al., 2020), ReorderNAT (Ran et al., 2021), GLAT (Qian et al., 2021), as well as the models using the alignment losses.

Original: Descriptions for all baselines are in the related work section.
Revised: Descriptions for all baselines are in the related work section.

Original: All experiments are done using the Fairseq library (Gehring et al., 2017).
Revised: All experiments are done using the Fairseq library (Gehring et al., 2017).

Original: To stay consistent with previous work, on the IWSLT’14 dataset we use the Transformer small conﬁguration 512-1024-4, while on the WMT datasets we use the Transformer base conﬁguration 512-2048-8 for encoder and decoder in CMLMC.
Revised: To stay consistent with previous work, on the IWSLT’14 dataset we use the Transformer small conﬁguration 512-1024-4, while on the WMT datasets we use the Transformer base conﬁguration 512-2048-8 for encoder and decoder in CMLMC.

Original: The numbers correspond to embedding dimension, FFN layer size, and number of attention heads respectively.
Revised: The numbers correspond to embedding dimension, FFN layer size, and number of attention heads respectively.

Original: Hyper-parameters for each dataset are selected through grid search and are listed in Table B.1 in Appendix.
Revised: Hyper-parameters for each dataset are selected through grid search and are listed in Table B.1 in Appendix.

Original: We apply our modiﬁcations to CMLM using the code released by the authors 1 .
Revised: We apply our modiﬁcations to CMLM using the code released by the authors 1 .

Original: In all experiments we use a linear FFN to combine the token and PE embeddings.
Revised: In all experiments we use a linear FFN to combine the token and PE embeddings.

Original: For datasets where CMLM performance is not reported, we train the model ourselves using the original code, and choose the hyper-parameters by applying the same grid search as in CMLMC.
Revised: For datasets where CMLM performance is not reported, we train the model ourselves using the original code, and choose the hyper-parameters by applying the same grid search as in CMLMC.

Original: We also implement SMART to match the performance on the reported datasets since code is not publicly available, this implementation is then used to evaluate SMART on datasets not reported in the original paper.
Revised: We also implement SMART to match the performance on the reported datasets since code is not publicly available, this implementation is then used to evaluate SMART on datasets not reported in the original paper.

Original: For CMLMC training we compute the correction loss L corr by randomly substituting 30% ( p = 0 . 3 ) of tokens in Y obs with predictions from ˆ Y ∅ ; the substitution percentage is chosen by grid search from [0.1, 0.2, 0.3, 0.5].
Revised: For CMLMC training we compute the correction loss L corr by randomly substituting 30% ( p = 0 . 3 ) of tokens in Y obs with predictions from ˆ Y ∅ ; the substitution percentage is chosen by grid search from [0.1, 0.2, 0.3, 0.5].

Original: At inference time we follow the procedure outlined in Section 3.3.
Revised: At inference time we follow the procedure outlined in Section 3.3.

Original: As in CMLM, we use 10 iterative reﬁnement steps, but lower the beam search length to 3.
Revised: As in CMLM, we use 10 iterative reﬁnement steps, but lower the beam search length to 3.

Original: Adam optimizer (Kingma & Ba, 2015) with default settings is used for all experiments, and we train the models on the IBM servers with 160 POWER9 CPUs, 600GB RAM and 4 Tesla V100 GPUs (32G).
Revised: Adam optimizer (Kingma & Ba, 2015) with default settings is used for all experiments, and we train the models on the IBM servers with 160 POWER9 CPUs, 600GB RAM and 4 Tesla V100 GPUs (32G).

Original: [Table]
Revised: [Table]

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: Results on the WMT datasets are shown in Table 1.
Revised: Results on the WMT datasets are shown in Table 1.

Original: We see that CMLMC outperforms all NAR benchmarks using CE loss, in many cases by a wide margin.
Revised: We see that CMLMC outperforms all NAR benchmarks using CE loss, in many cases by a wide margin.

Original: On the distilled setting, with the exception of WMT’14 De-En, CMLMC improves over CMLM by more than 0.8 BLEU points, and achieves new state-of-the-art results for Cross-Entropy-based NAR translation on all datasets.
Revised: On the distilled setting, with the exception of WMT’14 De-En, CMLMC improves over CMLM by more than 0.8 BLEU points, and achieves new state-of-the-art results for Cross-Entropy-based NAR translation on all datasets.

Original: Even when compared to models using alignment loss, CMLMC only falls short on WMT’14 De-En when compared to Imputer (Saharia et al., 2020), but is still superior to all other baselines despite using the standard CE loss.
Revised: Even when compared to models using alignment loss, CMLMC only falls short on WMT’14 De-En when compared to Imputer (Saharia et al., 2020), but is still superior to all other baselines despite using the standard CE loss.

Original: Similarly, on the raw setting CMLMC improves over CMLM by 1.5 to over
Revised: Similarly, on the raw setting CMLMC improves over CMLM by 1.5 to over

Original: BLEU points on all datasets.
Revised: BLEU points on all datasets.

Original: Improvements of this magnitude lead to CMLMC largely closing the gap to AR performance on undistilled data.
Revised: Improvements of this magnitude lead to CMLMC largely closing the gap to AR performance on undistilled data.

Original: On all datasets except En-De the difference between raw
Revised: On all datasets except En-De the difference between raw

Original: CMLMC and AR is less than 0.5 BLEU which to the best of our knowledge is the ﬁrst time when such small performance difference is achieved without distillation.
Revised: CMLMC and AR is less than 0.5 BLEU which to the best of our knowledge is the ﬁrst time when such small performance difference is achieved without distillation.

Original: This implies that it is possible to train NAR models without distillation and achieve comparable translation quality to AR counterparts.
Revised: This implies that it is possible to train NAR models without distillation and achieve comparable translation quality to AR counterparts.

Original: So we believe that with further research in this area, the performance difference between the two frameworks can be eliminated.
Revised: So we believe that with further research in this area, the performance difference between the two frameworks can be eliminated.

Original: Jointly, these results, and in particular the improvement over CMLM, indicate that the identiﬁed problems do cause a performance bottleneck.
Revised: Jointly, these results, and in particular the improvement over CMLM, indicate that the identiﬁed problems do cause a performance bottleneck.

Original: They also indicate that our proposed solutions are effective and can lead to signiﬁcant improvements, while being conceptually simple and easy to implement.
Revised: They also indicate that our proposed solutions are effective and can lead to signiﬁcant improvements, while being conceptually simple and easy to implement.

Original: Results on the IWSLT’14 datasets are shown in Table A.1 in the
Revised: Results on the IWSLT’14 datasets are shown in Table A.1 in the

Original: Appendix.
Revised: Appendix.

Original: CMLMC also shows considerable improvements of over 0.7 BLEU on distilled datasets and over 1.8 BLEU on raw datasets compared to the leading baselines.
Revised: CMLMC also shows considerable improvements of over 0.7 BLEU on distilled datasets and over 1.8 BLEU on raw datasets compared to the leading baselines.

Original: Analysis of iterative reﬁnement is shown in Table 3(a).
Revised: Analysis of iterative reﬁnement is shown in Table 3(a).

Original: For both CMLM and CMLMC we compute
Revised: For both CMLM and CMLMC we compute

Original: BLEU accuracy at different iterations.
Revised: BLEU accuracy at different iterations.

Original: We also compute total inference time to translate the entire
Revised: We also compute total inference time to translate the entire

Original: IWSLT’14 De-En test set and fraction of token repetitions.
Revised: IWSLT’14 De-En test set and fraction of token repetitions.

Original: Fraction of repetitions is computed by counting all consecutively repeating tokens and dividing by the total number of translated tokens.
Revised: Fraction of repetitions is computed by counting all consecutively repeating tokens and dividing by the total number of translated tokens.

Original: This number can thus be interpreted as the repetition rate that we expect to see from each model, for example at 5% we expect to see 50 repetitions in every 1000 translated tokens.
Revised: This number can thus be interpreted as the repetition rate that we expect to see from each model, for example at 5% we expect to see 50 repetitions in every 1000 translated tokens.

Original: From Table 3(a) we see that the repetition rate is consistently lower in CMLMC.
Revised: From Table 3(a) we see that the repetition rate is consistently lower in CMLMC.

Original: This can partially explain the overpoints gain in BLEU after the initial translation from fully masked sentences (iteration 1).
Revised: This can partially explain the overpoints gain in BLEU after the initial translation from fully masked sentences (iteration 1).

Original: This gain is preserved throughout the reﬁnement iterations, and after the last iteration CMLMC has repetition rate that is almost 50% lower than CMLM.
Revised: This gain is preserved throughout the reﬁnement iterations, and after the last iteration CMLMC has repetition rate that is almost 50% lower than CMLM.

Original: The architectural changes to distinguish tokens together with correction training, enable CMLMC to start with fewer mistakes and more effectively correct them during iterative reﬁnement.
Revised: The architectural changes to distinguish tokens together with correction training, enable CMLMC to start with fewer mistakes and more effectively correct them during iterative reﬁnement.

Original: We also see that additional masked attention layers in the decoder blocks add an overhead during inference forward passes.
Revised: We also see that additional masked attention layers in the decoder blocks add an overhead during inference forward passes.

Original: However, after the full 10 iterations this overhead is around 20% which we do not believe to be signiﬁcant for a model of this size.
Revised: However, after the full 10 iterations this overhead is around 20% which we do not believe to be signiﬁcant for a model of this size.

Original: Contrasting CMLMC with SMART, we note that CMLMC outperforms SMART on all datasets, with an average BLEU gain of 0.54 in the distilled datasets and 1.15 in raw, further supporting our arguments in section 3.2.
Revised: Contrasting CMLMC with SMART, we note that CMLMC outperforms SMART on all datasets, with an average BLEU gain of 0.54 in the distilled datasets and 1.15 in raw, further supporting our arguments in section 3.2.

Original: Auto-Regressive
Revised: [Figure]

Original: CMLMC
Revised: CMLMC

Original: I t e r
Revised: I t e r

Original: I t e rde de de de de fr fr fr fr fr es es es es es
Revised: I t e rde de de de de fr fr fr fr fr es es es es es

Original: CMLM
Revised: CMLM

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: Ablation results on the WMT datasets are shown in Table 1; IWSLT’14 ablations are shown in
Revised: Ablation results on the WMT datasets are shown in Table 1; IWSLT’14 ablations are shown in

Original: Table A.1 in the Appendix.
Revised: Table A.1 in the Appendix.

Original: Here, CMLM+RevPos incorporates the architectural changes in the decoder to better distinguish tokens (Section 3.1) but is trained with the original CMLM loss.
Revised: Here, CMLM+RevPos incorporates the architectural changes in the decoder to better distinguish tokens (Section 3.1) but is trained with the original CMLM loss.

Original: On the other hand, CMLM+Corr keeps the original CMLM architecture but incorporates the correction loss during training (Section 3.2).
Revised: On the other hand, CMLM+Corr keeps the original CMLM architecture but incorporates the correction loss during training (Section 3.2).

Original: We observe that both CMLM+RevPos and CMLM+Corr improve performance over CMLM on all datasets in both raw and distilled settings.
Revised: We observe that both CMLM+RevPos and CMLM+Corr improve performance over CMLM on all datasets in both raw and distilled settings.

Original: The improvements are signiﬁcant and range from 0.2 to over 2 points gain in BLEU.
Revised: The improvements are signiﬁcant and range from 0.2 to over 2 points gain in BLEU.

Original: CMLM+Corr generally leads to larger improvement, demonstrating the importance of training the model to correct mistakes.
Revised: CMLM+Corr generally leads to larger improvement, demonstrating the importance of training the model to correct mistakes.

Original: Additional ablation study on the effect of Masked Attention layers and concatenation of PEs are shown in
Revised: Additional ablation study on the effect of Masked Attention layers and concatenation of PEs are shown in

Original: Table A.1, indicating the necessity of both in the RevealPosition mechanism.
Revised: Table A.1, indicating the necessity of both in the RevealPosition mechanism.

Original: The gains over CMLM are more pronounced on raw setting than distilled.
Revised: The gains over CMLM are more pronounced on raw setting than distilled.

Original: We hypothesize that a smaller impact on distilled setting is due to the simpliﬁcation of the underlying training data structure.
Revised: We hypothesize that a smaller impact on distilled setting is due to the simpliﬁcation of the underlying training data structure.

Original: Knowledge distillation compresses multiple modes in the raw data into a single mode learned by the AR model (Zhou et al., 2020).
Revised: Knowledge distillation compresses multiple modes in the raw data into a single mode learned by the AR model (Zhou et al., 2020).

Original: NAR models trained with this data are less likely to make translation mistakes related to multi-modality so correction is not as important in this case.
Revised: NAR models trained with this data are less likely to make translation mistakes related to multi-modality so correction is not as important in this case.

Original: We still see however, that even on distilled data CMLM+Corr produces signiﬁcant improvements of up to 1.3 BLEU points.
Revised: We still see however, that even on distilled data CMLM+Corr produces signiﬁcant improvements of up to 1.3 BLEU points.

Original: Finally, combining both approaches in CMLMC gives additional improvement, and in all cases CMLMC outperforms both CMLM+RevPos and CMLM+Corr.
Revised: Finally, combining both approaches in CMLMC gives additional improvement, and in all cases CMLMC outperforms both CMLM+RevPos and CMLM+Corr.

Original: This indicates that distinguishing tokens and error correction are complementary and jointly lead to better translations.
Revised: This indicates that distinguishing tokens and error correction are complementary and jointly lead to better translations.

Original: Architectural modiﬁcations to the decoder outlined in Section 3.1 introduce additional parameters through weights in masked attention layers and input FFN.
Revised: Architectural modiﬁcations to the decoder outlined in Section 3.1 introduce additional parameters through weights in masked attention layers and input FFN.

Original: From Table 1 we see that CMLMC has around 9% more parameters than CMLM.
Revised: From Table 1 we see that CMLMC has around 9% more parameters than CMLM.

Original: To remove the performance effect of extra parameters, we also train a smaller model CMLMC 456 where input embedding dimension is reduced fromto 456; this model has 63M parameters which is 6% smaller than CMLM.
Revised: To remove the performance effect of extra parameters, we also train a smaller model CMLMC 456 where input embedding dimension is reduced fromto 456; this model has 63M parameters which is 6% smaller than CMLM.

Original: As seen in Table 1, the reduction in parameters does impact the performance, however, CMLMC 456 still outperforms nearly all baselines especially on raw data.
Revised: As seen in Table 1, the reduction in parameters does impact the performance, however, CMLMC 456 still outperforms nearly all baselines especially on raw data.

Original: This demonstrates that most performance gains in CMLMC come from our modiﬁcations and not from additional parameters.
Revised: This demonstrates that most performance gains in CMLMC come from our modiﬁcations and not from additional parameters.

Original: 4. M ULTI -M ODALITY
Revised: 4. M ULTI -M ODALITY

Original: We discussed that knowledge distillation is a time-consuming and non-standard step, yet all leading
Revised: We discussed that knowledge distillation is a time-consuming and non-standard step, yet all leading

Original: NAR models rely on it since their raw scores signiﬁcantly underperform the distilled counterparts.
Revised: NAR models rely on it since their raw scores signiﬁcantly underperform the distilled counterparts.

Original: Previous work (Zhou et al., 2020) argues that distillation primarily helps to resolve the multi-modality problem in the raw dataset, which NAR models cannot handle directly.
Revised: Previous work (Zhou et al., 2020) argues that distillation primarily helps to resolve the multi-modality problem in the raw dataset, which NAR models cannot handle directly.

Original: Following that work, we use the aligned sentences from the Europarl corpus to create a multi-target English to German, Spanish and French (En-De/Es/Fr) corpus.
Revised: Following that work, we use the aligned sentences from the Europarl corpus to create a multi-target English to German, Spanish and French (En-De/Es/Fr) corpus.

Original: For every En source sentence, we have three target sentences in
Revised: For every En source sentence, we have three target sentences in

Original: De, Es, and Fr, effectively creating a tri-modal data.
Revised: De, Es, and Fr, effectively creating a tri-modal data.

Original: For each architecture we train a single model on raw data to translate in all three languages by sharing the encoder/decoder layers but learning token representations for each language.
Revised: For each architecture we train a single model on raw data to translate in all three languages by sharing the encoder/decoder layers but learning token representations for each language.

Original: During inference the model is applied to En test sentences, and we estimate the probability of translation belonging to each of the three target languages by computing relative count of translated tokens from each language.
Revised: During inference the model is applied to En test sentences, and we estimate the probability of translation belonging to each of the three target languages by computing relative count of translated tokens from each language.

Original: Figure 
Revised: Figure 3(

Original: [Table]
Revised: [Table]

Original: We see that at iteration 1 CMLM outputs a highly mixed distribution where translations have tokens from multiple languages, which clearly shows the multi-modality problem.
Revised: We see that at iteration 1 CMLM outputs a highly mixed distribution where translations have tokens from multiple languages, which clearly shows the multi-modality problem.

Original: After 10 iterations of self-correction CMLM is able to separate the languages and mostly produces translations where the majority of tokens are from one target language.
Revised: After 10 iterations of self-correction CMLM is able to separate the languages and mostly produces translations where the majority of tokens are from one target language.

Original: This demonstrates that iterative reﬁnement is an important component of NAR translation that helps the models settle into one mode.
Revised: This demonstrates that iterative reﬁnement is an important component of NAR translation that helps the models settle into one mode.

Original: We also see that CMLMC has a much better language separation at iteration 1 than CMLM.
Revised: We also see that CMLMC has a much better language separation at iteration 1 than CMLM.

Original: So our proposed modiﬁcations improve the multi-modality problem even when translation is done from fully masked sentence.
Revised: So our proposed modiﬁcations improve the multi-modality problem even when translation is done from fully masked sentence.

Original: By comparing the graphs from CMLM and CMLMC at iteration 10 vs autoregressive baseline, we observe that they all exhibit a similar degree of separation.
Revised: By comparing the graphs from CMLM and CMLMC at iteration 10 vs autoregressive baseline, we observe that they all exhibit a similar degree of separation.

Original: However, as we have seen from other experiments their BLEU scores are typically quite different.
Revised: However, as we have seen from other experiments their BLEU scores are typically quite different.

Original: This suggests that the multi-modality problem might not be the only issue NAR models face when trained on raw datasets.
Revised: This suggests that the multi-modality problem might not be the only issue NAR models face when trained on raw datasets.

Original: We believe that an additional investigation is necessary here and leave it for future work.
Revised: We believe that an additional investigation is necessary here and leave it for future work.

Original: 4. Q UALITATIVE A NALYSIS
Revised: 4. Q UALITATIVE A NALYSIS

Original: IWSLT’14 De-En translation examples are shown in Table 2.
Revised: IWSLT’14 De-En translation examples are shown in Table 2.

Original: We focus on the common NAR problem of token repetition and show two example sentences that are representative of the repetition errors that
Revised: We focus on the common NAR problem of token repetition and show two example sentences that are representative of the repetition errors that

Original: CMLM typically makes.
Revised: CMLM typically makes.

Original: We also show the ﬁnal translation after 10 iterative reﬁnements and compare against the CMLMC translation.
Revised: We also show the ﬁnal translation after 10 iterative reﬁnements and compare against the CMLMC translation.

Original: The ﬁrst sentence has a sequence of numbers “one, two, three, four, ﬁve".
Revised: The ﬁrst sentence has a sequence of numbers “one, two, three, four, ﬁve".

Original: Embeddings for number tokens tend to be close together in the learned space, and CMLM has difﬁculty removing these repetitions.
Revised: Embeddings for number tokens tend to be close together in the learned space, and CMLM has difﬁculty removing these repetitions.

Original: After the ﬁrst iteration “one" and “three" tokens repeat, and while CMLM is able to correct these, it can’t fully remove repetitions and the ﬁnal translation still has repeating “two" token.
Revised: After the ﬁrst iteration “one" and “three" tokens repeat, and while CMLM is able to correct these, it can’t fully remove repetitions and the ﬁnal translation still has repeating “two" token.

Original: CMLMC on the other hand has no difﬁculty with the number tokens and translates them correctly after the ﬁrst iteration.
Revised: CMLMC on the other hand has no difﬁculty with the number tokens and translates them correctly after the ﬁrst iteration.

Original: The second sentence shows a long repetition that is difﬁcult to correct.
Revised: The second sentence shows a long repetition that is difﬁcult to correct.

Original: Here, “power" is repeated four times by CMLM after the ﬁrst iteration.
Revised: Here, “power" is repeated four times by CMLM after the ﬁrst iteration.

Original: After 10 iterations CMLM is able to correct the “nuclear" repetition and remove one “power" token but the other three remain.
Revised: After 10 iterations CMLM is able to correct the “nuclear" repetition and remove one “power" token but the other three remain.

Original: Since each reﬁnement iteration re-translates a subset of tokens with the lowest probability, longer repetitions are increasingly harder to correct as all repeating tokens need to eventually end up in the lowest probability set.
Revised: Since each reﬁnement iteration re-translates a subset of tokens with the lowest probability, longer repetitions are increasingly harder to correct as all repeating tokens need to eventually end up in the lowest probability set.

Original: We see that
Revised: We see that

Original: CMLMC also repeats “power", but only once, allowing the correction mechanism to come into play and ﬁx the error.
Revised: CMLMC also repeats “power", but only once, allowing the correction mechanism to come into play and ﬁx the error.

Original: These examples demonstrate that improved token distinguishability can lead to initial translations with fewer repetition mistakes.
Revised: These examples demonstrate that improved token distinguishability can lead to initial translations with fewer repetition mistakes.

Original: Our model is then the able to more effectively correct them, resulting in the better end translations.
Revised: Our model is then the able to more effectively correct them, resulting in the better end translations.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduce the C onditional M asked L anguage M odel with C orrection (CMLMC), an NAR translation model that addresses the design shortcomings in leading NAR approaches.
Revised: We introduce the C onditional M asked L anguage M odel with C orrection (CMLMC), an NAR translation model that addresses the design shortcomings in leading NAR approaches.

Original: Through a dual strategy of revealing positional information and adding error correction mechanism, we signiﬁcantly improve NAR translation performance on both raw and distilled datasets.
Revised: Through a dual strategy of revealing positional information and adding error correction mechanism, we signiﬁcantly improve NAR translation performance on both raw and distilled datasets.

Original: In particular, when trained on raw data, CMLMC approaches the performances of leading AR models which to the best of our knowledge is the ﬁrst such result in NAR.
Revised: In particular, when trained on raw data, CMLMC approaches the performances of leading AR models which to the best of our knowledge is the ﬁrst such result in NAR.

Original: Future work involves expanding our approach to other
Revised: Future work involves expanding our approach to other

Original: NAR models and further investigation into the relationship between multi-modality and distillation.
Revised: NAR models and further investigation into the relationship between multi-modality and distillation.

Original: A CMLMC P ERFORMANCE ON IWSLT’14 E N -D E D ATASET
Revised: A CMLMC P ERFORMANCE ON IWSLT’14 E N -D E D ATASET

Original: [Table]
Revised: [Table]

Original: B CMLMC H YPERPARAMETERS
Revised: H YPERPARAMETERS

Original: [Table]
Revised: [Table]