Original: D IFFU S EQ : S EQUENCE TO S EQUENCE T EXT G ENERATION WITH D IFFUSION M ODELS
Revised: D IFFU S EQ : S EQUENCE TO S EQUENCE T EXT G ENERATION WITH D IFFUSION M ODELS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently, diffusion models have emerged as a new paradigm for generative models.
Revised: Recently, diffusion models have emerged as a new paradigm for generative models.

Original: Despite the success in domains using continuous signals such as vision and audio, adapting diffusion models to natural language is difficult due to the discrete nature of text.
Revised: Despite the success in domains using continuous signals such as vision and audio, adapting diffusion models to natural language is under-explored due to the discrete nature of texts, especially for conditional generation.

Original: We tackle this challenge by proposing D IFFU S EQ : a diffusion model designed for sequence-to-sequence (S EQ 2S EQ ) text generation tasks.
Revised: We tackle this challenge by proposing D IFFU S EQ : a diffusion model designed for sequenceto-sequence (S EQ 2S EQ ) text generation tasks.

Original: Upon extensive evaluation over a wide range of S EQ 2S EQ tasks, we find D IFFU S EQ achieving comparable or even better performance than six established baselines, including a state-of-the-art model that is based on pre-trained language models.
Revised: Upon extensive evaluation over a wide range of S EQ 2S EQ tasks, we find D IFFU S EQ achieving comparable or even better performance than six established baselines, including a state-of-theart model that is based on pre-trained language models.

Original: Apart from quality, an intriguing property of D IFFU S EQ is its high diversity during generation, which is desired in many S EQ 2S EQ tasks.
Revised: Apart from quality, an intriguing property of D IFFU S EQ is its high diversity during generation, which is desired in many S EQ 2S EQ tasks.

Original: We further include a theoretical analysis revealing the connection between D IFFU S EQ and autoregressive/non-autoregressive models.
Revised: We further include a theoretical analysis revealing the connection between D IFFU S EQ and autoregressive/non-autoregressive models.

Original: Bringing together theoretical analysis and empirical evidence, we demonstrate the great potential of diffusion models in complex conditional language generation tasks.
Revised: Bringing together theoretical analysis and empirical evidence, we demonstrate the great potential of diffusion models in complex conditional language generation tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Existing generative models such as GAN (Goodfellow et al.,2014), VAE (Kingma & Welling, 2014) and Flow-based models (Dinh et al., 2017) suffer from the instability issue (Salimans et al., 2016), subjecting to mode collapse (Metz et al., 2017), and have to rely on surrogate objectives to approximate maximum likelihood training.
Revised: Among existing generative models, GAN (Goodfellow et al., 2014) suffers from the instability issue (Salimans et al., 2016), subjecting to mode collapse (Metz et al., 2017); VAE (Kingma & Welling, 2014) has to rely on surrogate objectives to approximate maximum likelihood training and Flow-based models (Dinh et al., 2017) has to use specialized architectures to construct reversible transform.

Original: Diffusion models (Ho et al., 2020; Nichol & Dhariwal, 2021) have circumvented several of these limitations and emerged as a new paradigm for generative models, theoretically underpinned by non-equilibrium thermodynamics (Sohl-Dickstein et al., 2015) and score-matching network (Song & Ermon, 2019).
Revised: Diffusion models (Ho et al., 2020; Nichol & Dhariwal, 2021) have circumvented several of these limitations and emerged as a new paradigm for generative models, theoretically underpinned by non-equilibrium thermodynamics (Sohl-Dickstein et al., 2015) and score-matching network (Song & Ermon, 2019).

Original: To date, the major breakthroughs are in domains using continuous signals, such as vision (Saharia et al., 2022a;b; Ramesh et al., 2022) and audio (Kong et al., 2020).
Revised: To date, the major breakthroughs are in domains using continuous signals, such as vision (Saharia et al., 2022a;b; Ramesh et al., 2022) and audio (Kong et al., 2020).

Original: However, extending continuous diffusion models to natural language remains an open challenge due to the inherently discrete nature of texts.
Revised: However, extending continuous diffusion models to natural language remains an open challenge due to the inherently discrete nature of texts.

Original: On the basis of unconditional generation in continuous space (illustrated in Figure 1(a)), existing efforts (Hoogeboom et al., 2021; Austin et al., 2021) start customizing diffusion models to text in discrete space on unconditional language modeling (i.e., free text generation).
Revised: On the basis of unconditional generation in continuous space which is illustrated in Figure 1(a), existing efforts (Hoogeboom et al., 2021; Austin et al., 2021) start customizing diffusion models to text in discrete space on unconditional language modeling (i.e., free text generation).

Original: Diffusion-LM (Li et al., 2022) (as in Figure 1(b)) models texts in continuous space and proposes to use a extra-trained classifier asa guidance (i.e., the condition signal x ) to impose subtle changes (usually complex, finegrained constraints) on generated sentences.
Revised: Diffusion-LM (Li et al., 2022), as in Figure 1(b), models texts in continuous space and proposes to use an extra-trained classifier as guidance (i.e., the condition signal x ) to impose subtle changes (usually complex, finegrained constraints) on generated sentences.

Original: Nonetheless, these models do not naturally generalized to conditional language modeling (i.e., the model assigns probabilities p ( w | x ) to sequences of words w given x ).
Revised: Nonetheless, these models do not naturally generalize to conditional language modeling (i.e., the model assigns probabilities p ( w | x ) to sequences of words w given x ).

Original: In the more general sequence-to-sequence (S EQ 2S EQ ) setting where the condition x is also a sequence of words, applying Diffusion-LM can be difficult.
Revised: In the more general sequence-to-sequence (S EQ 2S EQ ) setting where the condition x is also a sequence of words, applying Diffusion-LM can be difficult.

Original: The reason is that classifiers are attributes-oriented, and we can not train hundreds-of-thousands classifiers to model the semantic meaning between conditions and generated sentences.
Revised: The reason is that classifiers are attributes-oriented, and we can not train hundreds-of-thousands classifiers to model the semantic meaning between conditions and generated sentences.

Original: S EQ 2S EQ is an essential setting in NLP that covers a wide range of important tasks such as openended sentence generation, dialogue, paraphrasing, and text style transfer.
Revised: S EQ 2S EQ is an essential setting in NLP that covers a wide range of important tasks such as openended sentence generation, dialogue, paraphrasing, and text style transfer.

Original: In this paper, we propose
Revised: In this paper, we propose

Original:  D IFFU S EQ (in Figure 1 (c)), a classifier-free diffusion model that supports S EQ 2S EQ text generation tasks.
Revised: [Figure] D IFFU S EQ , depicted in Figure 1(c), a classifier-free diffusion model that supports S EQ 2S EQ text generation tasks.

Original: By modeling the conditional probability of the target sentence w given context x using one single model, one advantage of D IFFU S EQ is that this paradigm allows a complete model to fit data distribution and utilize conditional guidance, rather than depending on a separate classifier.
Revised: By modeling the conditional probability of the target sentence w given context x using one single model, one advantage of D IFFU S EQ is that this paradigm allows a complete model to fit data distribution and utilize conditional guidance, rather than depending on a separate classifier.

Original: [Figure]
Revised: Different from canonical generation approaches in an autoregressive (AR) left-to-right manner (Radford et al., 2019), D IFFU S EQ generates text tokens parallelly in the non-autoregressive (NAR) way.

Original: To corroborate the effectiveness of our D IFFU S EQ , we conduct experiments on four S EQ 2S EQ tasks.
Revised: To corroborate the effectiveness of our D IFFU S EQ , we conduct experiments on four S EQ 2S EQ tasks.

Original: Compared to autoregressive (AR) and non-autoregressive (NAR) models, which suffer from the “degeneration” problem (Holtzman et al., 2019) and rely on decoding strategies, D IFFU S EQ can achieve considerable sentence-level diversity without sacrificing the quality (see § 4.2).
Revised: Compared to AR and NAR models, which suffer from the “degeneration” problem (Holtzman et al., 2019) and rely on decoding strategies, D IFFU S EQ can achieve considerable sentence-level diversity without sacrificing the quality (see § 4.2).

Original: To sum up, we make a series of technical and conceptual contributions: (a) we are the first to deploy the diffusion model on S EQ 2S EQ text generation, and our proposed D IFFU S EQ as a conditional language model is trained end-to-end in a classifier-free manner; (b) we establish a theoretical connection among AR, NAR and D IFFU S EQ models, and justify D IFFU S EQ as an extension of iterative-NAR models; (c) with strong empirical evidence, we demonstrate the great potential of diffusion models in complex conditional language generation tasks.
Revised: To sum up, we make a series of technical and conceptual contributions: (a) we are the first to deploy the diffusion model on S EQ 2S EQ text generation, and our proposed D IFFU S EQ as a conditional language model is trained end-to-end in a classifier-free manner; (b) we establish a theoretical connection among AR, NAR and D IFFU S EQ models, and justify D IFFU S EQ as an extension of iterative-NAR models; (c) with strong empirical evidence, we demonstrate the great potential of diffusion models in complex conditional language generation tasks.

Original:  P RELIMINARY AND P ROBLEM S TATEMENT
Revised:  P RELIMINARY AND P ROBLEM S TATEMENT

Original: Preliminary.
Revised: Preliminary.

Original: A diffusion model typically contains forward and reverse processes.
Revised: A diffusion model typically contains forward and reverse processes.

Original: Given a data point sampled from a real-world data distribution x 0 ∼ q ( x ) , the forward process gradually corrupts x 0 into a standard Gaussian noise x T ∼ N (0 , I ) .
Revised: Given a data point sampled from a real-world data distribution z 0 ∼ q ( z ) , the forward process gradually corrupts z 0 into a standard Gaussian noise z T ∼ N (0 , I ) .

Original: For each forward step t ∈ [1 , 2 , ..., T ] , the perturbation is controlled by q ( x t | x t − 1 ) =
Revised: For each forward step t ∈ [1 , 2 , ..., T ] , the perturbation is controlled by q ( z t | z t − 1 ) =

Original: N ( x t ; √ 1 − β t x t − 1 , β t I ) , with β t ∈ (0 , 1) as different variance scales.
Revised: N ( z t ; √ 1 − β t z t − 1 , β t I ) , with β t ∈ (0 , 1) as different variance scales.

Original: Once the forward process is completed, the reverse denoising process tries to gradually reconstruct the original data x 0 via sampling from x T by learning a diffusion model f θ .
Revised: Once the forward process is completed, the reverse denoising process tries to gradually reconstruct the original data z 0 via sampling from z T by learning a diffusion model f θ .

Original: Problem Statement.
Revised: Problem Statement.

Original: Many recent efforts have been devoted to adapting diffusion models to discrete texts (See §5).
Revised: Many recent efforts have been devoted to adapting diffusion models to discrete texts (See § 5).

Original: However, they all focus on unconditional sequence modeling.
Revised: However, they all focus on unconditional sequence modeling.

Original: In this paper, we target the sequence-to-sequence text generation tasks.
Revised: In this paper, we target the sequence-to-sequence text generation tasks.

Original: In particular, given a  source sequence w x = { w x 1 , ..., w xm } , we aim to learn a diffusion model that can produce a  target sequence w y =
Revised: In particular, given a m -length source sequence w x = { w x 1 , ..., w xm } , we aim to learn a diffusion model that can produce a n -length target sequence w y =

Original: { w y 1 , ..., w yn } conditioning on the source sequence.
Revised: { w y 1 , ..., w yn } conditioning on the source sequence.

Original:  D IFFU S EQ
Revised:  D IFFU S EQ

Original: We propose D IFFU S EQ to extend vanilla diffusion models to learn conditional text generation (as shown in Figure 2), concerning the model architecture and the training objective.
Revised: We propose D IFFU S EQ to extend vanilla diffusion models to learn conditional text generation (as shown in Figure 2), concerning the model architecture and the training objective.

Original: Forward Process with Partial Noising.
Revised: Forward Process with Partial Noising.

Original: In the beginning of forward process, we follow DiffusionLM (Li et al., 2022) to design an embedding function E MB ( w ) to map the discrete text w into a continuous space.
Revised: In the beginning of forward process, we follow DiffusionLM (Li et al., 2022) to design an embedding function E MB ( w ) to map the discrete text w into a continuous space.

Original: In particular, given a pair of sequence w x and w y , D IFFU S EQ learns a unifiedfeature space of w x and w y by embedding transformation and concatenation as E MB ( w x ⊕ y ) = [ E MB ( w x 1 ) , ..., E MB ( w xm ) , E MB ( w y 1 ) , ..., E MB ( w yn )] ∈ R ( m + n ) × d . The transformation allows us to adapt discrete textual input into the standard forward process, by extending the original forward chain to a new Markov transition q ϕ ( z 0 | w x ⊕ y )
Revised: In particular, given a pair of sequence w x and w y , D IFFU S EQ learns a unified

Original: =

Original: N ( E MB ( w x ⊕ y ) , β 0 I ) .

Original: [Figure]
Revised: [Figure]

Original:  z 𝑇 z 𝑡 z 𝑡− z… … 𝑝 𝜃 (z 𝑡−1 |z 𝑡 ) 𝑞(z 𝑡 |z 𝑡−1 )
Revised: Gaussian Noise z 𝑇 z 𝑡 z 𝑡− z… … 𝑝 𝜃 (z 𝑡−1 |z 𝑡 ) 𝑞(z 𝑡 |z 𝑡−1 )

Original: Word Embeddings
Revised: Word Embeddings

Original: Text w 𝑥 w 𝑦
Revised: Text w 𝑥 w 𝑦

Original: Embedding map 𝑝 𝜃 (w|z 0 )
Revised: Embedding map 𝑝 𝜃 (w|z 0 )

Original: 𝑞 𝜙 (z 0 |w)
Revised: 𝑞 𝜙 (z 0 |w)

Original: How long was the trip?
Revised: How long was the trip?

Original: It was a year.
Revised: It was a year.

Original: Sequence to Sequence
Revised: Sequence to Sequence

Original: E.g. Open-Domain Dialogue
Revised: E.g. Open-Domain Dialogue

Original: Forward process [Figure]
Revised: Forward process Reverse process

Original: Gaussian Noise
Revised: Gaussian Noise

Original: Rounding 
Revised: Rounding feature space of w x and w y by embedding transformation and concatenation as E MB ( w x ⊕ y ) =

Original: For each forward step q ( z t | z t − 1 ) , we gradually inject noise into last step’s hidden state z t − 1  to obtain z t .
Revised: [ E MB ( w x 1 ) , ..., E MB ( w xm ) , E MB ( w y 1 ) , ..., E MB ( w yn )] ∈ R ( m + n ) × d . The transformation allows us to adapt discrete textual input into the standard forward process, by extending the original forward chain to a new Markov transition q ϕ ( z 0 | w x ⊕ y ) =

Original: By abuse of notation, we denote z t = x t ⊕ y t to simplify the wordings, where x t and y t represent parts of z t that belong to w x and w y , respectively.
Revised: N ( E MB ( w x ⊕ y ) , β 0 I ) . We denote z t = x t ⊕ y t to simplify the wordings, where x t and y t represent parts of z t that belong to w x and w y , respectively.

Revised: For each forward step q ( z t | z t − 1 ) , we gradually inject noise into last step’s hidden state z t − 1 to obtain z t .

Original: However, unlike conventional diffusion models that corrupt the whole z t (both x t and y t ) without distinction, we only impose noising on y t .
Revised: Unlike conventional diffusion models that corrupt the whole z t (both x t and y t ) without distinction, we only impose noising on y t .

Original: This modification (termed partial noising ) allows us to adapt diffusion models for conditional language modeling.
Revised: This modification (termed partial noising ) allows us to adapt diffusion models for conditional language modeling.

Original: In terms of implementation, we execute an anchoring function that replaces the corrupted x t with the original x 0 .

Original: Reverse Process with Conditional Denoising.
Revised: Reverse Process with Conditional Denoising.

Original: The ultimate goal of the reverse process is to recover the original z 0 by denoising z t : p θ ( z 0: T ) :
Revised: The ultimate goal of the reverse process is to recover the original z 0 by denoising z t : p θ ( z 0: T ) :

Original: = p ( z T ) (cid:81) Tt =1 p θ ( z t − 1 | z t ) .
Revised: = p ( z T ) (cid:81) Tt =1 p θ ( z t − 1 | z t ) .

Original: We model the learning process p θ ( z t − 1 | z t ) =
Revised: We model the learning process p θ ( z t − 1 | z t ) =

Original: N ( z t − 1 ; µ θ ( z t , t ) , σ θ ( z t , t )) using the proposed diffusion model
Revised: N ( z t − 1 ; µ θ ( z t , t ) , σ θ ( z t , t )) using the proposed diffusion model

Original: D IFFU S EQ : f θ ( z t , t ) , where the µ θ ( · ) and σ θ ( · ) is the parameterization of the predicted mean and standard deviation of q ( z t − 1 | z t )(derived using Bayes’ rule) in forward process.
Revised: D IFFU S EQ : f θ ( z t , t ) , where the µ θ ( · ) and σ θ ( · ) is the parameterization of the predicted mean and standard deviation of q ( z t − 1 | z t ) in forward process, derived using Bayes’ rule.

Original: The detailed derivations are in Appendix A. With the partial nosing strategy adopted in the forward process, we can impose the input as the condition when denoising as shown in Figure 1.
Revised: The detailed derivations are in Appendix A. With the partial nosing strategy adopted in the forward process, we can impose the input as the condition when denoising as shown in Figure 1.

Original: The proposed conditional denoising is classifier-free by nature: we do not require extra-trained classifiers to control the denoising process.
Revised: The proposed conditional denoising is classifier-free by nature: we do not require extra-trained classifiers to control the denoising process.

Original: Specifically, we use a transformer architecture to model f θ , which spontaneously models the semantic relation between x t and y t .
Revised: Specifically, we use a transformer architecture to model f θ , which spontaneously models the semantic relation between x t and y t .

Original: We compute the variational lower bound ( L VLB ) following the original diffusion process.
Revised: We compute the variational lower bound ( L VLB ) following the original diffusion process.

Original: L round corresponds to rounding operation in Figure 2.
Revised: L round corresponds to rounding operation in Figure 2.

Original: [Equation]
Revised: [Equation]

Original: We further simplify the training objective as follows (details in Appendix A):
Revised: We further simplify the training objective as follows (details in Appendix A):

Original: [Equation]
Revised: [Equation]

Revised: θ

Revised: L VLB =

Revised: min θ (cid:34) T (cid:88) t = ||

Revised: z 0 − f θ ( z t , t ) ||

Revised: 2 + || E MB

Revised: w x ⊕ y )

Revised: − f θ ( z 1 , 1) ||

Revised: 2 − log p θ ( w x ⊕ y | z 0 ) (cid:35) → min θ (cid:34) T (cid:88) t = ||

Revised: y 0 − ˜ f

Revised: θ ( z t , t )

Revised: ||

Original:  here we use ˜ f
Revised: 2 + || E MB ( w y ) − ˜ f θ ( z 1 , 1) || 2 + R ( || z 0 || 2 ) (cid:35) , (2) here we use ˜ f

Original: θ ( z t , t ) to denote the fractions of recovered z 0 corresponding to y 0 .
Revised: θ ( z t , t ) to denote the fractions of recovered z 0 corresponding to y 0 .

Original: Note that although in the first term we only compute the loss w.r.t y 0 , due to the attention mechanism in the transformer, the reconstruction of y 0 also takes x 0 into account, thus the gradients from the first term will also affect the learning of x 0 .
Revised: Note that although in the first term, we only compute the loss w.r.t y 0 , due to the attention mechanism in the transformer, the reconstruction of y 0 also takes x 0 into account, thus the gradients from the first term will also affect the learning of x 0 .

Original: The mathematically equivalent regularization term R ( || z 0 || 2 ) ) regularize the embedding learning.
Revised: The mathematically equivalent regularization term R ( || z 0 || 2 ) ) regularize the embedding learning.

Original: We further share the embedding function between source and target sequences, enabling the training of two different feature spaces jointly.
Revised: We further share the embedding function between source and target sequences, enabling the training of two different feature spaces jointly.

Original: This sets D IFFU S EQ away from existing solutions in vision such as GLIDE (Nichol et al., 2022).
Revised: This sets D IFFU S EQ away from existing solutions in vision such as GLIDE (Nichol et al., 2022).

Revised: Training and Inference Methods.

Revised: In our preliminary experiments, we find that the high diversity in NLP datasets and long diffusion steps often result in insufficient training.

Revised: We hypothesize the reason is that sampling step t uniformly causes unnecessary noise in the L VLB objective.

Revised: We hence employ importance sampling (Nichol & Dhariwal, 2021) to address this problem.

Revised: [Equation]

Original: Connections to AR, Iter-NAR, and Fully-NAR Models.
Revised: Intuitively, the importance-weighted sampling algorithm will spend more steps on diffusion steps with larger L t , and vice versa. To conduct S EQ 2S EQ generation given the condition E MB ( w x ) , we randomly sample y T ∼ N (0 , I ) and concatenate y T with E MB ( w x ) to obtain z T . We can now repeat the reverse process until we arrive at z 0 .

Revised: At each sampling step, an anchoring function is executed towards reparameterized z t .

Revised: Specifically, the anchoring function: (a) operates rounding on z t to map it back to word embedding space following Li et al.

Revised: (2022); (b) replaces the part of recovered z t − 1 that belongs to w x with the original x 0 , considering that this part is recovered from corrupted z t via f θ and not strictly equals to x 0 .

Revised: Note that (b) is designed for D IFFU S EQ .

Revised: To improve the quality of generation, we apply the widely used Minimum Bayes Risk (MBR) decoding strategy (Koehn, 2004).

Revised: We first generate a set of candidate samples S from different random seeds of D IFFU S EQ and select the best output sequence that achieves the minimum expected risk under a meaningful loss function (e.g. BLEU or other cheaper metrics like precision).

Revised: In practice, we use the negative BLEU score in our implementation.

Revised: Connections to AR, Iter-NAR, and Fully-NAR Models.

Original: To better understand the behavior of
Revised: To better understand the behavior of

Original: D IFFU S EQ , we give the theoretical connection to autoregressive (AR), iterative non-autoregressive (iter-NAR), and fully non-autoregressive (fully-NAR) models.
Revised: D IFFU S EQ , we give the theoretical connection to autoregressive (AR), iterative non-autoregressive (iter-NAR), and fully non-autoregressive (fully-NAR) models.

Original: We argue that D IFFU S EQ can be seen as an extension of iter-NAR model.
Revised: We argue that D IFFU S EQ can be seen as an extension of iter-NAR model.

Original: Detailed graphical learning discrepancies of these four cases are discussed in Appendix B for reference.
Revised: Detailed graphical learning discrepancies of these four cases are discussed in Appendix B for reference.

Original: We start from the conditional sequence generation problem, which aims to learn a conditional probability p ( w y 1: n | w x ) with w x and w y .

Original: AR models learn p ( w y 1: n | w x ) by autoregressive decomposition based on left-context:
Revised: AR models learn p ( w y 1: n | w x ) by autoregressive decomposition based on left-context:

Original: [Equation]
Revised: [Equation]

Original: while fully-NAR models (Gu et al., 2018; Qian et al., 2021) learn the conditional probability given independent assumption for fast inference:
Revised: while fully-NAR models (Gu et al., 2018; Qian et al., 2021) learn the conditional probability given independent assumption for fast inference:

Original: [Equation]
Revised: [Equation]

Original: To make a better analogy to AR and NAR models, we use a lossless way to formulate iterative
Revised: To make a better analogy to AR and NAR models, we use a lossless way to formulate iterative

Original: NAR models (Gu et al., 2019; Ghazvininejad et al., 2019) by introducing a series of intermediate sequences w y 1: K − 1 , w yK 
Revised: NAR models (Gu et al., 2019; Ghazvininejad et al., 2019) by introducing a series of intermediate sequences w y 1: K − 1 , w yK = w y with K editable iterations:

Original: [Equation]
Revised: [Equation]

Revised: w y 1 ,..., w yK −(cid:89)

Original: Previous study (Huang et al., 2022) shows that there is a gap called conditional total correlation between AR (Eq. 3) and fully-NAR (Eq. 4) learning paradigms, because of lossy decomposition of
Revised: i =1 ...n p ( w y 1 ,i | w x ) (cid:124) (cid:123)(cid:122) (cid:125) initial prediction (cid:89) k =1 ..K −(cid:89) i =1 ...n p (

Original: NAR models.
Revised: [Equation] +1 ,i | [Equation] , 1: n ,

Original: This gap is mainly responsible for the performance drop from AR to NAR models.
Revised: [Equation] x ) (cid:124) (cid:123)(cid:122) (cid:125) progressive full-context prediction . Previous study (Huang et al., 2022) shows that there is a gap called conditional total correlation between AR Eq. (4) and fully-NAR Eq.

Revised: (5) learning paradigms, because of lossy decomposition of

Original:  However, when comparing iter-NAR (Eq. 5) with AR (Eq. 3) models, they both can be factorized into an initial prediction term and a progressive prediction process based on different context (i.e. left-context in AR and full-context in iter-NAR).
Revised: NAR models. However, when comparing iter-NAR Eq. (6) with AR Eq. (4) models, they both can be factorized into an initial prediction term and a progressive prediction process based on different context (i.e. left-context in AR and full-context in iter-NAR), and the discrepancy pointed out by

Original: The discrepancy as pointed out by Huang et al.
Revised: Huang et al.

Original: (2022) is therefore closed in iter-NAR assuming sufficient steps.
Revised: (2022) is therefore closed in iter-NAR assuming sufficient steps.

Original: By showing D IFFU S EQ is an extension of the iter-NAR model, we offer a justification that it will not suffer from the conditional total correlation for the same reason.
Revised: By showing D IFFU S EQ is an extension of the iter-NAR model, we offer a justification that it will not suffer from the conditional total correlation for the same reason.

Original: A straight-forward way to formulate  diffusion models is to introduce a series of Gaussian noisecorrupted features  y 1: T − 1 , y 

Revised: A straight-forward way to formulate pure continuous diffusion models is to introduce a series of Gaussian noise-corrupted features along with diffusion steps: y 1: T − 1 , y

Original: =
Revised: =

Original: y , y T ∼ N (0 , I ) on continuous space as: [Equation] [Table]
Revised: y , y T ∼ N (0 , I ) .

Original: [Equation]
Revised: [Equation]

Original: where p ( y t − 1 | y t , w x ) describes the diffusion step on continuous representations y .
Revised: where p ( y t − 1 | y t , w x ) describes the diffusion step on continuous representations y .

Original: The total number of diffusion steps is denoted as T . Thereafter we omit the independent decomposition on w y and y  t . To apply diffusion models on discrete space
Revised: The rounding operation in D IFFU S EQ maps the continuous vectors y to discrete w y for each time step t , we in addition introduce this into Eq.

Original: [Equation]
Revised: [Equation]

Original: we in addition introduce variables w y to represent the discrete text into Eq. 6: [Equation]
Revised: y t , w x ) .

Original: By rearranging Eq.
Revised: By rearranging Eq.

Original: 7 into Eq.
Revised: (8) into Eq.

Original: [Equation] we can see D IFFU S EQ can be seen as a more generalized form of iter-NAR (Eq. 5) before marginalizing out { y T , . . .
Revised: (9), we can see D IFFU S EQ can be seen as a more generalized form of iter-NAR Eq. (6) before marginalizing out { y T , . . .

Original: , y 0 } , despite the different initialization of y T  .
Revised: , y 0 } , despite the different initialization of y T 2 .

Original: A more detailed derivation is shown in Appendix C.
Revised: A more detailed derivation is shown in Appendix C.

Original: Training and Inference Methods.

Original: In our preliminary experiments, we find that the high diversity in NLP datasets and long diffusion steps often result in insufficient training.

Original: We hypothesize the reason is that sampling step t uniformly causes unnecessary noise in the L VLB objective.

Original: We hence employ importance sampling (Nichol & Dhariwal, 2021) to address this problem.

Original: [Equation]

Original: Intuitively, the importance-weighted sampling algorithm will spend more steps on diffusion steps with larger L t , and vice versa.

Original: To conduct S EQ 2S EQ generation given the condition E MB ( w x ) , we randomly sample y T ∼

Original: N (0 , I ) and concatenate y T with E MB ( w x ) to obtain z T .

Original: We can now repeat the reverse process until we arrive at z 0 .

Original: At each sampling step, the anchoring function is executed towards reparameterized z t in the same manner as training.

Original: To improve the quality of generation, we apply the widely used Minimum Bayes Risk (MBR) decoding strategy (Koehn, 2004).

Original: We first generate a set of candidate samples S from different random seeds of D IFFU S EQ and select the best output sequence that achieves the minimum expected risk under a meaningful loss function (e.g. BLEU or other cheaper metrics like precision).

Original: In practice, we use the negative BLEU score in our implementation.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conduct experiments to validate the effectiveness of D IFFU S EQ on four different tasks, against six strong AR/NAR baselines.
Revised: We conduct experiments to validate the effectiveness of D IFFU S EQ on four different tasks, against six strong AR/NAR baselines.

Original: 4. E XPERIMENTAL S ETUP 
Revised: 4. E XPERIMENTAL S ETUP Tasks and Datasets.

Original: Tasks and Datasets.

Original: S EQ 2S EQ generation covers a wide range of tasks, among which we choose four typical and popular tasks, listed as follows.
Revised: S EQ 2S EQ generation covers a wide range of tasks, among which we choose four typical and popular tasks.

Original: Open domain dialogue requires models to generate informative responses given a dialogue context.
Revised: Open domain dialogue requires models to generate informative responses given a dialogue context.

Original: We use Commonsense Conversation Dataset (Zhou et al., 2018), which is extracted from Reddit single-round dialogs, with over 3 million conversational pairs.
Revised: We use Commonsense Conversation Dataset (Zhou et al., 2018), which is extracted from Reddit single-round dialogs, with over 3 million conversational pairs.

Original: Question generation (QG) aims to generate questions given a context as input.
Revised: Question generation (QG) aims to generate questions given a context as input.

Original: To obtain sufficient training samples, we use the dataset Quasar-T (Dhingra et al., 2017) preprocessed by Lin et al.
Revised: To obtain sufficient training samples, we use the dataset Quasar-T (Dhingra et al., 2017) preprocessed by Lin et al.

Original: (2018), and then generate document-question pairs to obtain 119K training samplesin the end (details in Appendix D.1).
Revised: (2018), and then generate document-question pairs to obtain 119K training samples (details in Appendix D.1).

Original: Text simplification aims to revise the complex text into sequences with simplified grammar and word choice.
Revised: Text simplification aims to revise the complex text into sequences with simplified grammar and word choice.

Original: Jiang et al.
Revised: Jiang et al.

Original: (2020) constructs a corpus consisting of 666K complex-simple sentences with revision alignment.
Revised: (2020) constructs a corpus consisting of 677K complexsimple sentences with revision alignment.

Original: Paraphrase task generates an alternative surface form in the same language expressing the same semantic content.
Revised: Paraphrase task generates an alternative surface form in the same language expressing the same semantic content.

Original: We adopt widely used Quora Question Pairs 3(QQP) sourced from the community question answering forum Quora, with 147K positive pairs.
Revised: We adopt widely used QQP 3 sourced from the community question answering forum Quora, with 147K positive pairs.

Original: Baselines.
Revised: Baselines.

Original: We consider three groups of models as baselines, covering both AR and NAR architectures.
Revised: We consider three groups of models as baselines, covering both AR and NAR architectures.

Original: The first group of methods adopts encoder-decoder architecture (Cho et al., 2014) which is well-studied for S EQ 2S EQ tasks, and we conduct experiments on two popular models: GRU with attention and Transformer (Vaswani et al., 2017).
Revised: The first group of methods adopts encoder-decoder architecture (Cho et al., 2014) which is well-studied for S EQ 2S EQ tasks, and we conduct experiments on two popular models: GRU with attention and Transformer (Vaswani et al., 2017).

Original: The second group is the finetuned large pre-trained language model (PLM), among which GPT2 (Radford et al., 2019) has demonstrated great success in almost all S EQ 2S EQ tasks.
Revised: The second group is the finetuned large pre-trained language model (PLM), among which GPT2 (Radford et al., 2019) has demonstrated great success in almost all S EQ 2S EQ tasks.

Original: We further compare to GPVAE (Du et al., 2022), which augments a pre-trained T5 (Raffel et al., 2020) with VAE to improve the generation diversity.
Revised: We further compare to GPVAE (Du et al., 2022), which augments a pre-trained T5 (Raffel et al., 2020) with VAE to improve the generation diversity.

Original: For the last group of baselines, we consider LevT (Gu et al., 2019), a widely used, strong iterative NAR model.
Revised: For the last group of baselines, we consider LevT (Gu et al., 2019), a widely used, strong iterative NAR model.

Original: All baselines are trained following instructions in their papers, and details can be found in Appendix D.2.
Revised: All baselines are trained following instructions in their papers, and details can be found in Appendix D.2.

Original: Evaluation.
Revised: [Table]

Original: We evaluate the generated sequences from two aspects: quality and diversity.
Revised: [Table]

Original: To evaluate the quality, we use the standard metric BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) score.
Revised: [Table] [Table] [Table]

Original: Since string-similarity-based metrics can be unsatisfactory for [Table] [Table] [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: Evaluation.

Original: [Table] [Table] [Table] [Table]
Revised: We evaluate the generated sequences from two aspects: quality and diversity.

Original: [Table] [Table] [Table] [Table]
Revised: To evaluate the quality

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: we use the standard metric BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) score.

Original: [Table] [Table] [Table] we also report BERTScore (Zhang et al., 2019) that assesses the semantic similarity between generated sentences and references.
Revised: Since string-similarity-based metrics can be unsatisfactory for open-ended generation, we also report BERTScore (Zhang et al., 2019) that assesses the semantic similarity between generated sentences and references.

Revised: Details are in Appendix D.4.

Original: Higher scores of BLEU, ROUGE and BERTScore reflect better performance.
Revised: Higher scores of BLEU, ROUGE and BERTScore reflect better performance.

Original: As for diversity, we use distinct unigram (dist-1) to measure intra-diversity within each generated sentence, where the lower dist-1 indicates that the generated sentence contains more repeated words.
Revised: As for diversity, we use distinct unigram (dist-1) to measure intra-diversity within each generated sentence [Table] where the lower dist-1 indicates that the generated sentence contains more repeated words.

Original: For sentence-level diversity evaluation, we consider sentence-level self-BLEU
Revised: For sentence-level diversity evaluation, we consider sentence-level self-BLEU

Original: (Zhu et al., 2018) to measure the n-gram overlap between the set of outputs w.r.t one source sentence [Table] and we additionally use diverse 4-gram (div-4) (Deshpande et al., 2019) to measure the ratio of distinct 4-grams in the set of outputs per source sentence.
Revised: (Zhu et al., 2018) to measure the n-gram overlap between the set of outputs w.r.t one source sentence, and we additionally use diverse 4-gram (div-4) (Deshpande et al., 2019) to measure the ratio of distinct 4-grams in the set of outputs per source sentence.

Original: The lower self-BLEU and higher div-4 suggest higher diversity of generation.
Revised: The lower self-BLEU and higher div-4 suggest higher diversity of generation.

Original: For each method including D IFFU S EQ , we generate 3 samples for each source sentence to compute the diversity metrics.
Revised: For each method including D IFFU S EQ , we generate 3 samples for each source sentence to compute the diversity metrics.

Original: Implementation Details.
Revised: Implementation Details.

Original: Our D IFFU S EQ is based on the 12 layers of Transformer with 12 attention heads, where the time step embedding is plugged akin to the position embedding.
Revised: Our D IFFU S EQ is based on the 12 layers of Transformer with 12 attention heads, where the time step embedding is plugged akin to the position embedding.

Original: The maximum sequence length is 128 , with embedding dimension d = 128 , diffusion steps T = 2 ,and a square-root noise schedule.
Revised: The maximum sequence length is 128 , with embedding dimension d = 128 , diffusion steps T = 2 ,and a square-root noise schedule.

Original: To reduce the out-of-vocabulary generation, we apply Byte Pair
Revised: To reduce the out-of-vocabulary generation, we apply Byte Pair

Original: Encoding (Sennrich et al., 2016) to construct the vocabulary.
Revised: Encoding (Sennrich et al., 2016) to construct the vocabulary.

Original: We compute the accuracy metrics of
Revised: After conducting the diversity beam [Table] search (DBS) (Vijayakumar et al., 2016) for the Transformer-base model and GPT model, we find that DBS does not always promote diversity over temperature sampling and therefore we list the best diversity results.

Original:  D IFFU S EQ using MBR with the size of candidate samples |S| = 10 .
Revised: We compute the accuracy metrics of D IFFU S EQ using MBR with the size of candidate samples |S| = 10 .

Original: The experiment is deployed on NVIDIA A100 Tensor CoreGPUs, and we use 4 GPUson training and single GPU on sampling.
Revised: The experiment is deployed on NVIDIA A100 Tensor Core GPUs

Original: [Table]
Revised: [Table]

Revised: and we use 4 GPUs on training and single GPU on sampling.

Original: 4. M AIN R ESULTS
Revised: 4. M AIN R ESULTS

Original: As shown in Table
Revised: As shown in Table

Original: [Table]
Revised: [Table]

Original: we conclude that D IFFU S EQ achieves comparable or even higher generation quality compared with strong baselines.
Revised: we conclude that D IFFU S EQ achieves comparable or even higher generation quality compared with strong baselines.

Original: At the same time, D IFFU S EQ consistently demonstrates its superiority in generating diverse outputs given the same input sequence.
Revised: At the same time [Table] D IFFU S EQ consistently demonstrates its superiority in generating diverse outputs given the same input sequence [Table]

Original: As we can see from Table
Revised: As we can see from Table

Original: [Table]
Revised: [Table]

Original: D IFFU S EQ wins23 out of 24 competitions (2 baselines × 4 tasks × 3 quality metrics) against the first group of AR baselines.
Revised: D IFFU S EQ wins competitions over at least one quality metric againstbaselines × 4 tasks.

Original: Although NAR models such as LevT can also outperform AR baselines sometimes, they still lag well behind D IFFU S EQ by large margins (i
Revised: Although NAR models such as LevT can also outperform AR baselines sometimes, they still lag well behind D IFFU S EQ by large margins (i

Original: [Table]
Revised: [Table]

Original: e., relative improvements over 50 % for BLEU in QG task and R-L in Dialogue task).
Revised: e., relative improvements over 50 % for BLEU in QG task and R-L in Dialogue task).

Original: Even compared with pre-trained then finetunedGPT2 models [Table]
Revised: Even compared with pre-trained then finetuned

Original:  D IFFU S EQ still delivers superior performance than the base variant [Table] and is comparable with the large variant, which has 8 .
Revised: GPT2 models, D IFFU S EQ still delivers superior performance than the base variant, and is comparable with the large variant, which has 8 .

Original: 2 times more parameters than D IFFU S EQ .
Revised: 2 times more parameters than D IFFU S EQ .

Original: This empirical results amply support our findings in § [Table] where we theoretically analyse the potential of diffusion models in modeling text sequences compared with AR models given sufficient diffusion steps.
Revised: These empirical results amply support our findings in § 3, where we theoretically analyze the potential of diffusion models in modeling text sequences compared with AR models given sufficient diffusion steps.

Original: D IFFU S EQ , as a member of the deep generative model family, also exhibit the capacity to generate highly diverse sequences.
Revised: D IFFU S EQ , as a member of the deep generative model family, also exhibit the capacity to generate highly diverse sequences.

Original: As suggested by self-BLEU (lower is better) and div-4 (higher is better), in almost all cases, D IFFU S EQ significantly outperforms 4 AR baselines in terms of sentence-level diversity (i.e., producing diverse outputs given the same input).
Revised: As suggested by self-BLEU (lower is better) and div-4 (higher is better), in almost all cases, D IFFU S EQ significantly outperforms 4 AR baselines in terms of sentence-level diversity (i.e., producing diverse outputs given the same input).

Original: For diversity in word choice within one sentence, we consider dist-1: a higher dist-1 indicates less repetition within a sentence.
Revised: For diversity in word choice within one sentence, we consider dist-1: a higher dist-1 indicates less repetition within a sentence.

Original: As we can see from Table 1, D IFFU S EQ has less repetition compared with encoder-decoder methods, but still fall behind the pre-trained GPT2 models (the same situation with BERTScore).
Revised: As we can see from Table 1, D IFFU S EQ has less repetition compared with encoder-decoder methods, but still fall behind the pre-trained GPT2 models (the same situation with BERTScore).

Original: These results suggest there is still room for improvement (e.g., use pre-training techniques) in diffusion models’ token-level choice.
Revised: These results suggest there is still room for improvement (e.g., use pre-training techniques) in diffusion models’ token-level choice.

Original: Different from NAR-LevT, D IFFU S EQ does not rely on an extra length prediction module, and is able to generate longer output sentences.
Revised: Different from NAR-LevT, D IFFU S EQ does not rely on an extra length prediction module but automatically decides by the padding token instead and is able to generate longer output sentences, indicated by the last column for average generation length.

Original: In Table 2, we provide examples to showcase D IFFU S EQ ’s ability to generate diverse samples.
Revised: In Table 2, we provide examples to showcase D IFFU S EQ ’s ability to generate diverse samples.

Original: More examples can be found in Appendix D.5. 
Revised: More examples can be found in Appendix D.5. [Figure] [Figure] [Figure]

Revised: [Figure]

Original: 4. A NALYSIS
Revised: 4. A NALYSIS

Original: We conduct a series of analysis to investigate the effectiveness of different aspects in D IFFU S EQ .
Revised: We conduct a series of analysis to investigate the effectiveness of different aspects in D IFFU S EQ .

Original: Diversity Ensures Quality.
Revised: Diversity Ensures Quality.

Original: Generating high-quality texts with high diversity is an important requirement for many text generation applications and the trade-off between quality and diversity is [Figure] [Figure] off between quality and diversity (details in Appendix D.3).
Revised: Generating high-quality texts with high diversity is an important requirement for many text generation applications and the trade-off between quality and diversity is always a critical concern in open-ended NLG tasks (Zhang et al., 2021).

Original: [Figure]

Original: [Figure]

Original: always a critical concern in open-ended NLG tasks (Zhang et al., 2021).

Original: Different from AR models relying on the decoding strategy like temperature and nucleus sampling (Holtzman et al., 2019) and
Revised: Different from AR models relying on the decoding strategy like temperature and nucleus sampling (Holtzman et al., 2019) and

Original: VAE models sampling latent variable from Gaussian Prior, the natural advantage of D IFFU S EQ is to generate different sentences along with a series of random Gaussian noise.
Revised: VAE models sampling latent variable from Gaussian Prior, the natural advantage of D IFFU S EQ is to generate different sentences along with a series of random Gaussian noise.

Original: In Figure [Figure] we elucidate that D IFFU S EQ have better trade-off between generation quality (BLEU) and sentence-level diversity (div-4).
Revised: In Figure 4, we elucidate that D IFFU S EQ have better trade-off between generation quality (BLEU) and sentence-level diversity (div-4).

Original: Here we further demonstrate that the high diversity provided by D IFFU S EQ can be turned into better quality.
Revised: Here we further demonstrate that the high diversity provided by D IFFU S EQ can be turned into better quality.

Original: MBR is a common strategy to improve generation quality by aggregating and ranking candidate sequences, and we find that the upper bound of MBR is decided by a diversified candidate set.
Revised: MBR is a common strategy to improve generation quality by aggregating and ranking candidate sequences [Figure] and we find that the upper bound of MBR is decided by a diversified candidate set.

Original: To valid this, we simultaneously apply MBR on both D IFFU S EQ and GPT2 with various candidate sizes |S| .
Revised: To valid this, we simultaneously apply MBR on both D IFFU S EQ and GPT2 with various candidate sizes |S| .

Original: The results are shown in Figure 3.
Revised: The results are shown in Figure 3.

Original: As we can see, D IFFU S EQ lags behind GPT2 without using
Revised: As we can see, D IFFU S EQ lags behind GPT2 without using

Original: MBR ( |S| = 1 ) or with a small candidate set ( |S| = 3 ).
Revised: MBR ( |S| = 1 ) or with a small candidate set ( |S| = 3 ).

Original: However, as |S| increases, D IFFU S EQ starts to outperform GPT2 by an increasing margin.
Revised: However, as |S| increases, D IFFU S EQ starts to outperform GPT2 by an increasing margin.

Original: The reason is that autoregressive models like GPT tend to generate highly similar candidates (as discussed in § 4.2), which impedes the effectiveness of MBR.
Revised: The reason is that autoregressive models like GPT tend to generate highly similar candidates (as discussed in § 4.2), which impedes the effectiveness of MBR.

Original: As |S| increases to 20 , D IFFU S EQ still shows better rising trends than GPT2.
Revised: As |S| increases to 20 , D IFFU S EQ still shows better rising trends than GPT2.

Original: Our findings also stress the importance of better ranking methods in diffusion research.
Revised: Our findings also stress the importance of better ranking methods in diffusion research.

Original: Step-wise Analysis against Iterative NAR.
Revised: Step-wise Analysis against Iterative NAR.

Original: Given the underlying theoretical connection between iterative NAR and D IFFU S EQ discussed in § 3, we empirically investigate the behavior of LevT and D IFFU S EQ by analyzing their step-wise quality (i.e. BLEU) and diversity (i.e. div-4) curves.
Revised: Given the underlying theoretical connection between iterative NAR and D IFFU S EQ discussed in § 3, we empirically investigate the behavior of LevT and D IFFU S EQ by analyzing their step-wise quality (i.e. BLEU) and diversity (i.e. div-4) curves.

Original: As is suggested in Figure 5, LevT grows fiercely in quality at the very beginning of generation, and quickly slows down in the successive refinement process.
Revised: As is suggested in Figure 5, LevT grows fiercely in quality at the very beginning of generation, and quickly slows down in the successive refinement process.

Original: But D IFFU S EQ behaves differently, with
Revised: But D IFFU S EQ behaves differently, with

Original: BLEU score growing slowly at first, increasing rapidly as the diffusion process progresses and finally surpassing LevT.
Revised: BLEU score growing slowly at first, increasing rapidly as the diffusion process progresses and finally surpassing LevT.

Original: It is also observed that the diversity of both LevT and D IFFU S EQ is determined at the very early stage regardless of future refinement or diffusion, where D IFFU S EQ consistently outperforms LevT on diversity at any stage of generation.
Revised: It is also observed that the diversity of both LevT and D IFFU S EQ is determined at the very early stage regardless of future refinement or diffusion, where D IFFU S EQ consistently outperforms LevT on diversity at any stage of generation.

Original: We conjecture that D IFFU S EQ explores more possible results at the first half of generation process, and soon converges to several potential candidates when it is closed to the end of steps.
Revised: We conjecture that D IFFU S EQ explores more possible results at the first half of generation process, and soon converges to several potential candidates when it is closed to the end of steps.

Original: In this case, D IFFU S EQ shows its capacity to take both generation quality and diversity into consideration, and this is the capacity that iterative-NAR and even AR models can not obtain, due to the different learning paradigms.
Revised: In this case, D IFFU S EQ shows its capacity to take both generation quality and diversity into consideration, and this is the capacity that iterative-NAR and even AR models can not obtain, due to the different learning paradigms.

Original: Inference Speed.
Revised: Inference Speed.

Original: The slow sampling speed is one of the major concerns about diffusion models.
Revised: The slow sampling speed is one of the major concerns about diffusion models.

Original: Here we fix the number of diffusion steps during training for D IFFU S EQ while shrinking the inference steps following DDIM (Song et al., 2020).
Revised: Here we fix the number of diffusion steps during training for D IFFU S EQ while shrinking the inference steps following DDIM (Song et al., 2020).

Original: As we can see from Figure 6, when reducing the inference to 1,000 diffusion steps on single GPU, D IFFU S EQ achieves a higher BLEU score than
Revised: As we can see from Figure 6, when reducing the inference to 1,000 diffusion steps on single GPU, D IFFU S EQ achieves a higher BLEU score than

Original: GPT2-large yet registers a closer inference speed to GPT2-large.
Revised: GPT2-large yet registers a closer inference speed to GPT2-large.

Original: Effectiveness of Joint Training.
Revised: Effectiveness of Joint Training.

Original: In D IFFU S EQ , the representations of w x and w y are jointly trained using the same embedding function E MB ( · ) (stated in § 3).
Revised: In D IFFU S EQ , the representations of w x and w y are jointly trained using the same embedding function E MB ( · ) (stated in § 3).

Original: To validate the effectivenessof this joint training strategy, we compared it with the training strategy commonly used in text-toimage diffusion models (Nichol et al., 2022; Ramesh et al., 2022). In particular, we decouple the training of E MB ( w x ) and E MB ( w y ) by replacing E MB ( w x ) with representations extracted from a pre-trained BERT-tiny model (Turc et al., 2019). From Table 3, we find that the decoupled training strategy results in poor performance.
Revised: To validate the effectiveness

Original: [Table]
Revised: [Table]

Original:  E MB ( w x ) as pre-trained
Revised: Fix E MB ( w x ) as pre-trained

Original: [Table]
Revised: [Table]

Revised: of this joint training strategy, we compared it with the training strategy commonly used in text-toimage diffusion models (Nichol et al., 2022; Ramesh et al., 2022).

Revised: In particular, we decouple the training of E MB ( w x ) and E MB ( w y ) by replacing E MB ( w x ) with representations extracted from a pre-trained BERT-tiny model (Turc et al., 2019).

Revised: From Table 3, we find that the decoupled training strategy results in poor performance.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Diffusion Models for Text Modeling.
Revised: Diffusion Models for Text Modeling.

Original: Text-to-Image generation using diffusion models has developed many potential applications.
Revised: Text-to-Image generation using diffusion models has developed many potential applications.

Original: Models such as Imagen (Saharia et al., 2022b) and DALLE (Ramesh et al., 2022) are usually two-staged relying on the pre-trained models, requiring the alignment between the embedding vectors from two sources.
Revised: Models such as Imagen (Saharia et al., 2022b) and DALLE (Ramesh et al., 2022) are usually two-staged relying on the pre-trained models, requiring the alignment between the embedding vectors from two sources.

Original: GLIDE (Nichol et al., 2022) explores diffusion model with classifier-free (Ho & Salimans, 2022) guidance by setting guidance scale during training.
Revised: GLIDE (Nichol et al., 2022) explores diffusion model with classifier-free (Ho & Salimans, 2022) guidance by setting guidance scale during training.

Original: The target space of these models is not discrete text space but stable vectors of pixel values.
Revised: The target space of these models is not discrete text space but stable vectors of pixel values.

Original: There are other works of diffusion on text generation, but they stick to the original encoderdecoder architecture and the diffusion process is interspersed on the decoder (Savinov et al., 2021), or the latent space (Yu et al., 2022).
Revised: There are other works of diffusion on text generation, but they stick to the original encoderdecoder architecture and the diffusion process is interspersed on the decoder (Savinov et al., 2021), or the latent space (Yu et al., 2022).

Original: For text generation using the diffusion models, Hoogeboom et al.
Revised: For text generation using the diffusion models, Hoogeboom et al.

Original: (2021) introduce the multinomial diffusion for character-level text generation, the forward categorical noise is applied through the
Revised: (2021) introduce the multinomial diffusion for character-level text generation, the forward categorical noise is applied through the

Original: Markov transition matrix.
Revised: Markov transition matrix.

Original: Austin et al.
Revised: Austin et al.

Original: (2021) generalize discrete text diffusion models by introducing the absorbing state ( [MASK] ).
Revised: (2021) generalize discrete text diffusion models by introducing the absorbing state ( [MASK] ).

Original: However, discrete diffusion models may suffer from the scaling of the one-hot row vectors, and they only generate text samples unconditionally in discrete space.
Revised: However, discrete diffusion models may suffer from the scaling of the one-hot row vectors, and they only generate text samples unconditionally in discrete space.

Original: Diffusion-LM (Li et al., 2022) and Analog Bits (Chen et al., 2022) propose a new language model diffused on the continuous latent representations, with different mapping functions that connect the discrete and continuous space of texts.
Revised: Diffusion-LM (Li et al., 2022) and Analog Bits (Chen et al., 2022) propose a new language model diffused on the continuous latent representations, with different mapping functions that connect the discrete and continuous space of texts.

Original: Compared with our work, we focus on the S EQ 2S EQ diffusion models for text generation in the continuous space and our work is the first to explore this setting to the best of our knowledge.
Revised: Compared with our work, we focus on the S EQ 2S EQ diffusion models for text generation in the continuous space and our work is the first to explore this setting to the best of our knowledge.

Original: Diffusion Models for Conditional Generation.
Revised: Diffusion Models for Conditional Generation.

Original: Related to conditional-VAE (Zhao et al., 2017), we can consider the latent encoded input x as a condition.
Revised: Related to conditional-VAE (Zhao et al., 2017), we can consider the latent encoded input x as a condition.

Original: Diffusion-LM (Li et al., 2022) adopts the plug-and-play approaches (Dathathri et al., 2020) to compose fine-grained constraints on the generated sentences, but it fails to condition on the whole source sentence in S EQ 2S EQ tasks.
Revised: Diffusion-LM (Li et al., 2022) adopts the plug-and-play approaches (Dathathri et al., 2020) to compose fine-grained constraints on the generated sentences, but it fails to condition on the whole source sentence in S EQ 2S EQ tasks.

Original: Noted that this controllable generation method is orthogonal to our D IFF S EQ , in other words, we can further add classifier-guided constraints on the S EQ 2S EQ output to further control the text generation.
Revised: Noted that this controllable generation method is orthogonal to our D IFF S EQ , in other words, we can further add classifier-guided constraints on the S EQ 2S EQ output to further control the text generation.

Original: There are other conditional diffusion models on the time series prediction like CSDI (Tashiro et al., 2021) or audio generation like WaveGrad (Chen et al., 2021), but their class conditions are usually attributes that are easy to model, while the contextual texts as conditions are much more complex.
Revised: There are other conditional diffusion models on the time series prediction like CSDI (Tashiro et al., 2021) or audio generation like WaveGrad (Chen et al., 2021), but their class conditions are usually attributes that are easy to model, while the contextual texts as conditions are much more complex.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We propose D IFFU S EQ to tackle S EQ 2S EQ tasks in a diffusion way, which contains the strong potential to achieve better generation quality and diversity trade-off.
Revised: We propose D IFFU S EQ to tackle S EQ 2S EQ tasks in a diffusion way, which contains the strong potential to achieve better generation quality and diversity trade-off.

Original: In other words, D IFFU S EQ is capable of producing highly diverse sentences without sacrificing much of the quality.

Original: The capability enables favorable characteristics of D IFFU S EQ to further enhance the quality of final results, by leveraging a minimum Bayes risk decoding algorithm.
Revised: The capability enables favorable characteristics of D IFFU S EQ to further enhance the quality of final results, by leveraging a minimum Bayes risk decoding algorithm.

Original: Besides, we theoretically connect the AR and NAR models to D IFFU S EQ , and show that D IFFU S EQ is a powerful extension of iterativeNAR model.
Revised: Besides, we theoretically connect the AR and NAR models to D IFFU S EQ , and show that D IFFU S EQ is a powerful extension of iterative-NAR model.

Original: The empirical results demonstrate that D IFFU S EQ is also a powerful model for text generation, matching or even surpassing competitive autoregressive, iterative non-autoregressive, and large-scale pre-trained models on quality and diversity.
Revised: The empirical results demonstrate that D IFFU S EQ is also a powerful model for text generation, matching or even surpassing competitive AR, iterative NAR, and large-scale pre-trained models on quality and diversity.

Original: Given the limited progress of current diffusion models on text generation, our study addresses promising achievements by such a new sequence-to-sequence learning paradigm
Revised: Given the limited progress of current diffusion models on text generation, our study addresses promising achievements by such a new sequence-to-sequence learning paradigm. A CKNOWLEDGMENTS

Revised: We would like to thank the anonymous reviewers and other peers for their valuable advice and we also acknowledge Chenxin An’s efforts to update the generation results for the Transformer-base model on QG and Paraphrasing tasks.

Revised: This work is partially supported by the Shanghai Committee of Science and Technology (Grant No. 21DZ1100100) and the joint research scheme of the National

Revised: Natural Science Foundation of China (NSFC) and the Research Grants Council (RGC) under grant number N HKU714/

Original: A O BJECTIVE D ERIVATIONS O F D IFFU S EQ
Revised: A O BJECTIVE D ERIVATIONS O F D IFFU S EQ

Original: The diffusion model is well-known as its ability to achieve the trade-off between flexibility and tractability of the models’ probability distributions, compared with GAN, VAE and Flow-based models.
Revised: The diffusion model is well-known as its ability to achieve the trade-off between flexibility and tractability of the models’ probability distributions, compared with GAN, VAE and Flow-based models.

Original: Following Ho et al.
Revised: Following Ho et al.

Original: (2020); Nichol & Dhariwal (2021);
Revised: (2020); Nichol & Dhariwal (2021);

Original: Song et al.
Revised: Song et al.

Original: (2020), we systematically define the forward noising process and reverse denoising process on latent continuous space z .
Revised: (2020), we systematically define the forward noising process and reverse denoising process on latent continuous space z .

Original: The forward noising is to perturb the structure of data z 0 .
Revised: The forward noising is to perturb the structure of data z 0 .

Original: z 0 is finally changed into the partial
Revised: z 0 is finally changed into the partial

Original: Gaussian noise with y T ∼ N (0 , I ) through T -step forward random disturbance
Revised: Gaussian noise with y T ∼ N (0 , I ) through T -step forward random disturbance

Original: [Equation]
Revised: [Equation]

Original: with t = 1 , 2 , ..., T and { β t ∈ (0 , 1) } Tt =1 are the variance schedule.
Revised: with t = 1 , 2 , ..., T and { β t ∈ (0 , 1) } Tt =1 are the variance schedule.

Original: Let α t = 1 − β t
Revised: Let α t = 1 − β t

Original: and ¯ α t = (cid:81) ti =
Revised: and ¯ α t = (cid:81) ti =

Original: α i , we have:
Revised: α i , we have:

Original: [Equation]
Revised: [Equation]

Original: (11) where ϵ stands for Gaussian noises.
Revised: (11) where ϵ stands for Gaussian noises.

Original: In the end, q ( z t | z 0 )
Revised: In the end, q ( z t | z 0 )

Original: =
Revised: =

Original: N ( z t ; √ ¯ α t z 0 , (1 − ¯ α t ) I ) .
Revised: N ( z t ; √ ¯ α t z [Equation] , (1 − ¯ α t ) I ) .

Original: We use a sqrt noise schedule in Diffusion-LM (Li et al., 2022), that is, ¯ α t = 1 − (cid:112) t/T + s with s as a small constant at the start of noise level.
Revised: We use a sqrt noise schedule in Diffusion-LM (Li et al., 2022), that is, ¯ α t = 1 − [Equation] t/T + s with s as a small constant at the start of noise level.

Original: The reverse process then denoises z t , aiming to recover original z 0 , and is defined as:
Revised: The reverse process then denoises z t , aiming to recover original z 0 , and is defined as:

Original: [Equation]
Revised: [Equation]

Original: The learning of p θ is based on our diffusion model D IFFU S EQ :
Revised: The learning of p θ is based on our diffusion model D IFFU S EQ :

Original: f θ ( z t , t ) , where the µ θ ( · ) and σ θ ( · ) is the predicted parameterization of the mean and standard variation of q ( z t | z t − 1 ) in forward process.
Revised: f θ ( z t , t ) , where the µ θ ( · ) and σ θ ( · ) is the predicted parameterization of the mean and standard variation of q ( z t | z t − 1 ) in forward process.

Original: Using Bayes’ rule:
Revised: Using Bayes’ rule:

Original: [Equation]
Revised: [Equation]

Original: Substitute Eq.
Revised: Substitute Eq.

Original: 11 to it
Revised: (11) to it

Original: and we can get the parameterized mean of q ( z t − 1 | z t , z 0 ) :
Revised: and we can get the parameterized mean of q ( z t − 1 | z t , z 0 ) :

Original: [Equation]
Revised: [Equation]

Original: and for brevity, we short the coefficient of z t and z 0 as U and E respectively.
Revised: and for brevity, we short the coefficient of z t and z 0 as U and E respectively.

Original: We can use the variational lower bound to optimize the negative log-likelihood E [ − log p θ ( x 0 )] ≤
Revised: We can use the variational lower bound to optimize the negative log-likelihood E [ − log p θ ( x 0 )] ≤

Original: L VLB .
Revised: L VLB .

Original: The objective can be further rewritten to be a combination of several KL-divergence and entropy terms following Sohl-Dickstein et al.
Revised: The objective can be further rewritten to be a combination of several KL-divergence and entropy terms following Sohl-Dickstein et al.

Original: [Equation]
Revised: [Equation]

Original: For 1 ≤ t ≤ T − 1 , we compute the parameterization of L t by substituting Eq.
Revised: For 1 ≤ t ≤ T − 1 , we compute the parameterization of L t by substituting Eq.

Original: 14 to minimize the difference from µ t and µ θ following Ho et al.
Revised: (14) to minimize the difference from µ t and µ θ following Ho et al.

Original: [Equation]
Revised: [Equation]

Original: where C = 2 ||
Revised: where C = 2 ||

Original: σ θ
Revised: σ θ

Original: || 2 is a loss independent constant.
Revised: || 2 is a loss independent constant.

Original: Then the optimization of training loss min θ L VLB can be further simplified as:
Revised: Then the optimization of training loss min θ L VLB can be further simplified as:

Original: [Equation]
Revised: [Equation]

Original: B G RAPHICAL M ODELS OF AR, F ULLY NAR, ITERATIVE NAR AND D IFFU S EQ MODELS
Revised: B G RAPHICAL M ODELS OF AR, F ULLY NAR, ITERATIVE NAR AND D IFFU S EQ MODELS

Revised: We start from the conditional sequence generation problem

Original: We present a lossless way to formulate iterative NAR models by introducing a series of intermediate sequences w y 1: K − 1 , w yK =
Revised: [Equation] which aims to learn a conditional probability p ( w y 1: n | w x ) with w x and w y . AR models learn p ( w y 1: n | w x ) by autoregressive decomposition based on left-context: [Equation]

Revised: [Equation]

Revised: consisting of an initial prediction and an autoregressive left-context prediction process, while fullyNAR models (Gu et al., 2018; Qian et al., 2021) learn the conditional probability given independent assumption for fast inference:

Revised: [Equation]

Revised: To make a better analogy to AR and NAR models

Revised: [Equation]

Revised: we use a lossless way to formulate iterative

Revised: NAR models (Gu et al., 2019; Ghazvininejad et al., 2019) by introducing a series of intermediate sequences w y 1: K − 1 , w yK =

Original: w y as:
Revised: w y as:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: Previous study (Huang et al., 2022) shows that there is a gap called conditional total correlation between AR and fully-NAR learning paradigms, because of the lossy decomposition of NAR models.

Revised: This gap is mainly responsible for the performance drop from AR to NAR models.

Revised: However, when comparing iter-NAR, Eq.

Revised: (20), with AR models, they both can be factorized into an initial prediction term and a progressive prediction process based on different context (i.e. left-context in

Revised: AR and full-context in iter-NAR).

Revised: The discrepancy as pointed out by Huang et al.

Revised: (2022) is therefore closed in iter-NAR assuming sufficient steps.

Revised: By showing D IFFU S EQ is an extension of the iter-NAR model, we offer a justification that it will not suffer from the conditional total correlation for the same reason.

Original: Similarly, naive diffusion models learn with a series of Gaussian noise-fused intermediate features y 1: T − 1 , y 

Revised: A straight-forward way to formulate naive diffusion models is to introduce a series of Gaussian noise-corrupted features y 1: T − 1 , y

Original: =
Revised: =

Original: y , y T ∼ N (0 , I ) on contiguous space as:
Revised: y , y T ∼ N (0 , I ) on continuous space as:

Original: [Equation]
Revised: [Equation]

Original: where p ( y t − 1 | y t , w x ) describes the diffusion process on contiguous representations y .
Revised: where p ( y t − 1 | y t , w x ) describes the diffusion process on contiguous representations y .

Original:  To connect Eq. 5 and  Eq. 
Revised: The total number of diffusion steps is denoted as T . Thereafter we omit the independent decomposition on w y and y t . To apply diffusion models on discrete space, the rounding operation in D IFFU S EQ maps the continuous vectors y to discrete w y for each time step t , we hence in addition introduce both contiguous feature y and the discrete text w y to represent the discrete text into Eq. [Equation]

Original: 19, D IFFU S EQ introduces both of contiguous feature y and the discrete text w y by modeling:
Revised: By rearranging Eq. (23) and Eq. (24), we can see that D IFFU S EQ can be seen as a more generalized form of iter-NAR before marginalizing out { y T , . . . , y 0 } , where Eq.

Original: [Equation]
Revised: (23) and Eq.

Original: [Figure]
Revised: (24) are equivalent with different computation order, despite the different initialization of y T . For NAR models, y T is uniform copied from the source sentence or unk ’s token embedding (Gu et al., 2018); for diffusion models, y T is sampled from normal distribution N (0 , I ) . It is notable that unlike AR and fully NAR models generating text all at once, iterative NAR and diffusion models feature a self-corrected text generation process. The graphical comparison is shown in Figure 7.

Original: C F ROM D IFFU S EQ TO I TERATIVE NAR AND D IFFUSION M ODELS
Revised: C F ROM D IFFU S EQ TO I TERATIVE NAR AND D IFFUSION M ODELS

Original: From D IFFU S EQ to Iterative NAR We show how to derive D IFFU S EQ to iterative nonautoregressive model on discrete space. 
Revised: From D IFFU S EQ to Iterative NAR We show how to derive D IFFU S EQ to iterative nonautoregressive model on discrete space. [Figure]

Original: p D IFFU S EQ ( w y | w x ) =
Revised: p [Equation]

Original:  (cid:88) w yT ,..., w y(cid:90)
Revised: = (cid:88) w yT ,..., w y(cid:90)

Original: y T ,..., yp ( w y | y 0 , w x ) (cid:89) t
Revised: y T ,..., yp ( w [Equation] | [Equation]

Original: = T,...
Revised: [Equation]

Original:  ,p ( y t − 1 | w yt ) p ( w yt | y t , w x ) = (cid:88) w yT ,..., w y(cid:90)
Revised: x ) (cid:89) t = T,... ,p ( y t − 1 | w yt ) p ( w yt | y t , w x ) = (cid:88) w yT ,..., w y(cid:90)

Original: y T ,..., yp ( w yT |
Revised: y T ,..., yp ( w y

Revised: [Equation]

Original:  y T , w x ) (cid:89) t =
Revised: | y T , w x ) (cid:89) t =

Original: T − 1 ,...,p ( w yt | y t , w x ) p ( y t | w yt +1 ) reorder computation = 
Revised: T − 1 ,...,p ( w y t | y t , w x ) p ( y t | w y t +1 ) reorder computation = (cid:88) [Equation] [Equation]

Original: (cid:88) w yT ,..., w yp ( w yT | y T , w x ) (cid:89) t = T − 1 ,...,(cid:90) 
Revised: (cid:89) t = T − 1 ,...,(cid:90) [Equation] [Equation] = (cid:88)

Revised: [Equation]

Revised: yT

Revised: [Equation]

Revised: [Equation]

Original: y t p 
Revised: [Equation] p ( [Equation] ) (cid:89) t =

Original: [Equation]
Revised: [Equation]

Original: marginalize over y
Revised: marginalize over y

Original: [Equation]
Revised: [Equation]

Original: align t and k reversely.
Revised: align t and k reversely.

Original: [Equation]
Revised: [Equation]

Original: From D IFFU S EQ to diffusion model We show how to derive D IFFU S EQ to the straight-forward diffusion model on continuous space. 
Revised: From D IFFU S EQ to diffusion model We show how to derive D IFFU S EQ to the straight-forward diffusion model on continuous space. p D IFFU S EQ ( w y | w x ) =

Original: [Equation]
Revised: (cid:88) w yT ,..., w y(cid:90) y T ,..., yp ( w y | y 0 , w x ) (cid:89) t = T,... ,p ( y t − 1 | w yt ) p ( w yt | y t , w x ) =

Original:  w 
Revised: (cid:90) y T ,..., y 0 p ( w y | y 0 , w x ) (cid:89) t = T,...,(cid:88) w yt p ( y t − 1 | w yt ) p ( w yt | y t , w x ) = (cid:90) y T ,..., yp ( w y | y 0 , w x ) (cid:89) t =

Original: [Equation]
Revised: T,...

Original:  marginalize over w y [Equation]
Revised: ,p ( y t − 1 | y t , w x ) marginalize over w y = p diffusion ( w y | w x )

Original: D D ETAILS OF E XPERIMENTS D. P ROCESSING OF Q UESTION G ENERATION D ATASET
Revised: D D ETAILS OF E XPERIMENTS D. P ROCESSING OF Q UESTION G ENERATION D ATASET

Original: To construct high-quality document-question pairs from the Quasar-T dataset, which consists of ⟨ document i , question, answer ⟩ triplets, we extract ⟨ document i , question ⟩ pairs if answer exactly matches document i . 
Revised: To construct high-quality document-question pairs from the Quasar-T dataset, which consists of ⟨ document i , question, answer ⟩ triplets, we extract ⟨ document i , question ⟩ pairs if answer exactly matches document i . After pre-processing, we obtain 119K document-question training pairs.

Original: D. S ETTINGS OF B ASELINES
Revised: D. S ETTINGS OF B ASELINES

Original: We compare the settings of different models, including the number of parameters and how to sample the different output sentences, as shown in Table 4.
Revised: We compare the settings of different models, including the number of parameters and how to sample the different output sentences, as shown in Table 4.

Original: For plain GRU-based encoder-decoder methods, we do not implement diversity search algorithms on it, thus its sentence-level diversity could be very poor.
Revised: For plain GRU-based encoder-decoder methods, we do not implement diversity search algorithms on it, thus its sentence-level diversity could be very poor.

Original: For NAR-LevT, we set max iteration to 9 and follow the termination condition mentioned in the original paper.
Revised: For NAR-LevT, we set max iteration to 9 and follow the termination condition mentioned in the original paper.

Original: For GPVAE-T5, we tune the scalar to find the best trade-off between quality and diversity on the dev set.
Revised: For GPVAE-T5, we tune the scalar to find the best trade-off between quality and diversity on the dev set.

Original: The scalars of all four tasks are set to 
Revised: The scalars of all four tasks are set to 2.

Original: [Table]
Revised: We implement GPT2 baselines using HuggingFace Transformers and for the baseline Transformer-base, we use Fairseq .

Original: [Table]
Revised: [Table]

Original: D. D IVERSITY AND QUALITY T RADE - OFF S ETTINGS
Revised: D. D IVERSITY AND QUALITY T RADE - OFF S ETTINGS

Original: We list the details to obtain Figure 4.
Revised: We list the details to obtain Figure 4.

Original: For GPVAE-T5, we set different scalars as 1 , 2 , 3 , 4 .
Revised: For GPVAE-T5, we set different scalars as 1 , 2 , 3 , 4 .

Original: For
Revised: For

Original: D IFFU S EQ , we choose trained models at different training steps to achieve different trade-off points
Revised: D IFFU S EQ , we choose trained models at different training steps to achieve different trade-off points

Original: [Table]
Revised: [Table]

Original: For other baselines, there is no explicit factor to control the diversity generation, so we leave them as single points in the figure.[Table]
Revised: For other baselines, there is no explicit factor to control the diversity generation, so we leave them as single points in the figure.

Original: D. M ETRICS
Revised: D. M ETRICS

Original: The used BLEU score is sentence-level smoothed from BLEU-1 to 4, and used ROUGE-L score is longest common subsequence based statistics.
Revised: The used BLEU score is sentence-level smoothed from BLEU-1 to 4, and used ROUGE-L score is longest common subsequence based statistics.

Original: The implementation is based on NLTK  and torchmetrics .
Revised: The implementation is based on NLTK 5 and torchmetrics .

Original: The n-gram based metrics may fail to capture the semantic meaning of sentences, so we consider using BERTScore 5 .
Revised: The n-gram based metrics may fail to capture the semantic meaning of sentences, so we consider using BERTScore 6 .

Original: Specifically, we use microsoft/deberta-xlarge-mnli to help BERTScore correlate better with human scores.
Revised: Specifically, we use microsoft/deberta-xlarge-mnli to help BERTScore correlate better with human scores.

Original: D. G ENERATION R ESULTS
Revised: D. G ENERATION R ESULTS

Original: For different tasks we list some generation examples.
Revised: For different tasks we list some generation examples.

Original: As we can see in Table 7, Table 6 and Table 5,
Revised: As we can see in Table 5, Table 6 and Table 7,

Original: D IFFU S EQ tends to generate diverse outputs, but sometimes the sentence is not as fluent as finetuned
Revised: D IFFU S EQ tends to generate diverse outputs, but sometimes the sentence is not as fluent as finetuned

Original: GPT2.
Revised: GPT2.

Revised: [Table]

Revised: dye

Original: [Table]
Revised: [Table]

Original: Response : Just did this two 
Revised: Response : Just did this two days ago,

Original: [Table]
Revised: [Table]

Original: ago
Revised: it’ll fade yet

Original: [Table]
Revised: [Table]

Original: * I’m not sure, I’m not sure.
Revised: * I’m not sure, I’m not sure.

Original: I’ve tested it a few times, but I don’t know for sure.
Revised: I’ve tested it a few times, but I don’t know for sure.

Original: I’ve
Revised: I’ve

Original: [Table]
Revised: [Table]

Original: half 
Revised: half .

Original: [Table]

Original: * I’m not sure.
Revised: * I’m not sure.

Original: I’m not sure how long it lasts, I’m sure it ’ll get better.
Revised: I’m not sure how long it lasts, I’m sure it ’ll get better.

Original: It’s been a while since [Table] half .
Revised: It’s been a while since * half .

Original: * I’ve been using it for about a year and a half.
Revised: * I’ve been using it for about a year and a half.

Original: I’ve been using it for about a year and a half.
Revised: I’ve been using it for about a year and a half.

Original: [Table]
Revised: [Table]

Original: half
Revised: half

Original: [Table]
Revised: [Table]

Original: GPT2-large finetune
Revised: GPT2-large finetune

Original: [Table]
Revised: [Table]

Original: * Two weeks in my case.
Revised: * Two weeks in my case.

Original: * About an hour, 5 days or so.
Revised: * About an hour, 5 days or so.

Original: * I’ve had it for about a year.
Revised: * I’ve had it for about a year.

Original: [Table]
Revised: [Table]

Original: * The dye can sit around for a month then you can wash it.
Revised: * The dye can sit around for a month then you can wash it.

Original: * I’m not sure about this, about the same kind of time.
Revised: * I’m not sure about this, about the same kind of time.

Original: Statement : The Japanese yen is the official and only currency recognized in Japan.
Revised: Statement : The Japanese yen is the official and only currency recognized in Japan.

Original: Question :
Revised: Question :

Original: What is the Japanese currency?
Revised: What is the Japanese currency?

Original: [Table]
Revised: [Table]

Revised: -T

Original: [Table]
Revised: [Table]

Original: * What is the japanese currency *
Revised: * What is the japanese currency *

Original: What is the basic unit of currency for Japan ?
Revised: What is the basic unit of currency for Japan ?

Original: * What is the japanese currency *
Revised: * What is the japanese currency *

Original: What is the basic unit of currency for Japan ?
Revised: What is the basic unit of currency for Japan ?

Original: * What is the japanese currency *
Revised: * What is the japanese currency *

Original: What is the basic unit of currency for Japan ?
Revised: What is the basic unit of currency for Japan ?

Original: GPT2-large finetune
Revised: GPT2-large finetune

Original: [Table]
Revised: [Table]

Original: * What is the basic unit of currency for Japan?
Revised: * What is the basic unit of currency for Japan?

Original: * What is the Japanese currency *
Revised: * What is the Japanese currency *

Original: What is the Japanese currency * Which country uses the “yen yen” in currency *
Revised: What is the Japanese currency * Which country uses the “yen yen” in currency *

Original: What is the basic unit of currency for Japan?
Revised: What is the basic unit of currency for Japan?

Original: * What is the basic unit of currency?
Revised: * What is the basic unit of currency?

Original: Complex sentence : People can experience loneliness for many reasons, and many life events may cause it, such as a lack of friendship relations during childhood and adolescence, or the physical absence of meaningful people around a person.
Revised: Complex sentence : People can experience loneliness for many reasons, and many life events may cause it, such as a lack of friendship relations during childhood and adolescence, or the physical absence of meaningful people around a person.

Original: Simplified : One cause of loneliness is a lack of friends during childhood and teenage years.
Revised: Simplified : One cause of loneliness is a lack of friends during childhood and teenage years.

Original: [Table]
Revised: [Table]

Original: People can experience loneliness for many reasons, and many life events may cause it, such as a lack of friendship relations during childhood and adolescence, or the physical absence of meaningful people around a person 
Revised: People can experience loneliness for many reasons, and many life events may cause it, such as a lack of friendship relations during childhood and adolescence, or the physical absence of meaningful people around a person *

Original: [Table] [Table]
Revised: [Table] [Table]

Original: People can experience loneliness for many reasons, and many lifeevents may cause it, such as a lack of friendship relations during childhood and adolescence, or the physical absence of meaningful people around a person
Revised: life

Original: [Table] [Table]
Revised: [Table] [Table]

Original: People can experience loneliness for many reasons, and many life events may cause it, such as a lack of friendship relations during childhood and adolescence, or the physical absence of meaningful people around a person *
Revised: People can experience loneliness for many reasons, and many life events may cause it, such as a lack of friendship relations during childhood and adolescence, or the physical absence of meaningful people around a person *

Original: [Table]
Revised: [Table]

Original: experience 
Revised: experience reashapphap- phapphapphapphappabout life

Original: [Table] 
Revised: [Table] [Table]

Revised: People can experience loneliness for many reasons, and many life events may cause it, such as a lack of friendship relations during childhood and adolescence, or the physical absence of meaningful people around a person * People may experience reashapphap- phapphapphapphappabout life reasit.

Original: [Table]
Revised: [Table]

Original: Loneliness can be caused by many things.
Revised: Loneliness can be caused by many things.

Original: * Many life events may cause of loneliness * Loneliness can affect people in many ways.
Revised: * Many life events may cause of loneliness * Loneliness can affect people in many ways.

Original: * People can also be very experience loneliness for many reasons.
Revised: * People can also be very experience loneliness for many reasons.

Original: * Loneliness can be caused by many things.
Revised: * Loneliness can be caused by many things.

Original: * People can experience loneliness for many reasons, and many life events may, cause it.
Revised: * People can experience loneliness for many reasons, and many life events may, cause it.