Original: D IFFUS ER: D IFFUSION VIA E DIT - BASED R ECONSTRUCTION
Revised: D IFFUS ER: D IFFUSION VIA E DIT - BASED R ECONSTRUCTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In text generation, models that generate text from scratch one token at a time are currently the dominant paradigm.
Revised: In text generation, models that generate text from scratch one token at a time are currently the dominant paradigm.

Original: Despite being performant, these models lack the ability to revise existing text , which limits their usability in many practical scenarios.
Revised: Despite being performant, these models lack the ability to revise existing text , which limits their usability in many practical scenarios.

Original: We look to address this, with D IFFUS ER ( Diffus ion via E dit-based
Revised: We look to address this, with D IFFUS ER ( Diffus ion via E dit-based

Original: R econstruction), a new edit-based generative model for text based on denoising diffusion models – a class of models that use a Markov chain of denoising steps to incrementally generate data.
Revised: R econstruction), a new edit-based generative model for text based on denoising diffusion models – a class of models that use a Markov chain of denoising steps to incrementally generate data.

Original: D IFFUS ER is not only a strong generative model in general, rivalling autoregressive models on several tasks spanning machine translation, summarization, and style transfer; it can also perform other varieties of generation that standard autoregressive models are not well-suited for.
Revised: D IFFUS ER is not only a strong generative model in general, rivalling autoregressive models on several tasks spanning machine translation, summarization, and style transfer; it can also perform other varieties of generation that standard autoregressive models are not well-suited for.

Original: For instance, we demonstrate that D IFFUS ER makes it possible for a user to condition generation on a prototype, or an incomplete sequence, and continue revising based on previous edit steps.
Revised: For instance, we demonstrate that D IFFUS ER makes it possible for a user to condition generation on a prototype, or an incomplete sequence, and continue revising based on previous edit steps.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Revision and editing are central to how humans produce content; we write and revise emails and papers, gradually produce works of art, and iterate on plans for a project.
Revised: Revision and editing are central to how humans produce content; we write and revise emails and papers, gradually produce works of art, and iterate on plans for a project.

Original: Despite this, the most dominant paradigm in text generation is purely autoregressive, producing text left-to-right in a single pass (Bengio et al., 2003).
Revised: Despite this, the most dominant paradigm in text generation is purely autoregressive, producing text left-to-right in a single pass (Bengio et al., 2003).

Original: Although models employing this single-pass form of generation are highly performant, they are limited by the inability to refine existing text.
Revised: Although models employing this single-pass form of generation are highly performant, they are limited by the inability to refine existing text.

Original: To address this, we propose D IFFUS ER: Diffus ion via E dit-based R econstruction, a flexible method to apply edit-based generative processes to arbitrary text generation tasks.
Revised: To address this, we propose D IFFUS ER: Diffus ion via E dit-based R econstruction, a flexible method to apply edit-based generative processes to arbitrary text generation tasks.

Original: Specifically, we take inspiration from diffusion models (Sohl-Dickstein et al., 2015;
Revised: Specifically, we take inspiration from diffusion models (Sohl-Dickstein et al., 2015;

Original: Ho et al., 2020), generative models that generate by way of incremental denoising steps, and adapt this approach to the text generation paradigm with a formulation similar to natural editing processes.
Revised: Ho et al., 2020), generative models that generate by way of incremental denoising steps, and adapt this approach to the text generation paradigm with a formulation similar to natural editing processes.

Original: Prior work on text generation either focuses on improving the performance of standard autoregressive (AR) models through larger models and datasets (Vaswani et al., 2017;
Revised: Prior work on text generation either focuses on improving the performance of standard autoregressive (AR) models through larger models and datasets (Vaswani et al., 2017;

Original: Sutskever et al., 2014;
Revised: Sutskever et al., 2014;

Original: Radford et al.
Revised: Radford et al.

Original: ; Brown et al., 2020)
Revised: ; Brown et al., 2020)

Original: or on proposing new, non-autoregressive approaches (Gu et al., 2017;
Revised: or on proposing new, non-autoregressive approaches (Gu et al., 2017;

Original: Ghazvininejad et al., 2019; Gu et al., 2019) to improve general modes of text generation.
Revised: Ghazvininejad et al., 2019; Gu et al., 2019) to improve general modes of text generation.

Original: A thus far separate line of models has taken the perspective of modeling text edits for specific tasks: e.g. style transfer (Reid & Zhong, 2021; Malmi et al., 2020), sentence fusion (Malmi et al., 2019), and grammatical error correction (Dale & Kilgarriff, 2011).
Revised: A thus far separate line of models has taken the perspective of modeling text edits for specific tasks: e.g. style transfer (Reid & Zhong, 2021; Malmi et al., 2020), sentence fusion (Malmi et al., 2019), and grammatical error correction (Dale & Kilgarriff, 2011).

Original: D IFFUS ER unifies these two perspectives by enabling edit processes to be applied to general purpose text generation without compromising performance or requiring external supervised data (Guu et al., 2018).
Revised: D IFFUS ER unifies these two perspectives by enabling edit processes to be applied to general purpose text generation without compromising performance or requiring external supervised data (Guu et al., 2018).

Original: This design enables it to both generate and edit text, including externally produced content, a natural extension of the text generation paradigm.
Revised: This design enables it to both generate and edit text, including externally produced content, a natural extension of the text generation paradigm.

Original: D IFFUS ER models text generation as a series of diffusion steps at the token level.
Revised: D IFFUS ER models text generation as a series of diffusion steps at the token level.

Original: This form of generation allows us to develop a synthetic formulation of natural editing processes (Reid & Neubig, 2022) using edit-based corruption and reconstruction.
Revised: This form of generation allows us to develop a synthetic formulation of natural editing processes (Reid & Neubig, 2022) using edit-based corruption and reconstruction.

Original: Our method starts from an arbitrary sequence (either a prototype generation, randomly sampled tokens, or a null sequence) and progressively edits it into the final sequence guided by the Levenshtein edit operations of I NSERT , D ELETE , K EEP , and R EPLACE as shown in Figure 1.
Revised: Our method starts from an arbitrary sequence (either a prototype generation, randomly sampled tokens, or a null sequence) and progressively edits it into the final sequence guided by the Levenshtein edit operations of I NSERT , D ELETE , K EEP , and R EPLACE as shown in Figure 1.

Original: This enables flexible editing in a range of contexts, including
Revised: This enables flexible editing in a range of contexts, including

Original: These model guilty named
Revised: These model guilty named

Original: DiffusER
Revised: DiffusER

Original: These model named
Revised: These model named

Original: DiffusER uses editing procedures
Revised: DiffusER uses editing procedures

Original: This model , named
Revised: This model , named

Original: DiffusER, uses editing processes
Revised: DiffusER, uses editing processes

Original: This model , named DiffusER, uses editing processes for flexible generation filter
Revised: This model , named DiffusER, uses editing processes for flexible generation filter

Original: Toronto guilty trough feel
Revised: Toronto guilty trough feel

Original: C o rr u p t i o n P r o c e ss
Revised: C o rr u p t i o n P r o c e ss

Original: R e c o n s t r u c t
Revised: R e c o n s t r u c t

Original: i o n
Revised: i o n

Original: P r o c e ss machine translation, summarization, style transfer, while also allowing for the possibility of taking outside input to guide and constrain generation.
Revised: P r o c e ss machine translation, summarization, style transfer, while also allowing for the possibility of taking outside input to guide and constrain generation.

Original: Learning these edit-based diffusion processes required several innovations over standard autoregressive and MLM-style iterative generation approaches (Ghazvininejad et al., 2019;
Revised: Learning these edit-based diffusion processes required several innovations over standard autoregressive and MLM-style iterative generation approaches (Ghazvininejad et al., 2019;

Original: Austin et al., 2021;
Revised: Austin et al., 2021;

Original: Savinov et al., 2022), including forming edit-based corruption and reconstruction processes for training (Sec 3), as well as techniques to improve the quality of decoding sequences across both timesteps and token-level generations (including 2D beam search; Sec 3.6, Sec 3.5).
Revised: Savinov et al., 2022), including forming edit-based corruption and reconstruction processes for training (Sec 3), as well as techniques to improve the quality of decoding sequences across both timesteps and token-level generations (including 2D beam search; Sec 3.6, Sec 3.5).

Original: To demonstrate the effectiveness of D IFFUS ER, we test our method on three text generation tasks: machine translation, abstractive summarization, and text style transfer, and show on-par or improved performance compared to purely autoregressive, single-pass and non-autoregressive methods.
Revised: To demonstrate the effectiveness of D IFFUS ER, we test our method on three text generation tasks: machine translation, abstractive summarization, and text style transfer, and show on-par or improved performance compared to purely autoregressive, single-pass and non-autoregressive methods.

Original: We also provide qualitative samples of the edit processes learned by the models in different settings and analyses on training and inference speeds, as well as the relationship between edit steps and performance.
Revised: We also provide qualitative samples of the edit processes learned by the models in different settings and analyses on training and inference speeds, as well as the relationship between edit steps and performance.

Original: Overall, we demonstrate the potential of edit-based generative models to offer 1) more performant generation, 2) greater interactivity between different models (as we can now perform edits in the discrete space on model generated output), and 3) more flexible/controllable generation.
Revised: Overall, we demonstrate the potential of edit-based generative models to offer 1) more performant generation, 2) greater interactivity between different models (as we can now perform edits in the discrete space on model generated output), and 3) more flexible/controllable generation.

Original:  B ACKGROUND D IFFUS
Revised:  B ACKGROUND D IFFUS

Original: ER operates at the intersection of text generation, editing processes, and diffusion models.
Revised: ER operates at the intersection of text generation, editing processes, and diffusion models.

Original: We first provide the background and intuition of these three techniques.
Revised: We first provide the background and intuition of these three techniques.

Original: 2. T EXT G ENERATION
Revised: 2. T EXT G ENERATION

Original: Most text generation models used in NLP today are autoregressive in nature.
Revised: Most text generation models used in NLP today are autoregressive in nature.

Original: In this paradigm, given a sequence s =
Revised: In this paradigm, given a sequence s =

Original: [ s 0 , s 1 , . .
Revised: [ s 0 , s 1 , . .

Original: , s N ] , one can model the likelihood of the entire sequence P ( s ) by modeling the probability of predicting each token in an autoregressive, often left-to-right, manner.
Revised: , s N ] , one can model the likelihood of the entire sequence P ( s ) by modeling the probability of predicting each token in an autoregressive, often left-to-right, manner.

Original: This formulation, where the likelihood of a token p ( s t ) is conditioned on its predecessors s <t , is shown below (Bengio et al., 2003):
Revised: This formulation, where the likelihood of a token p ( s t ) is conditioned on its predecessors s <t , is shown below (Bengio et al., 2003):

Original: [Equation]
Revised: [Equation]

Original: Models trained with this objective can then be sampled from, or searched over (e.g. using beam search), to provide generations in downstream tasks such as machine translation or summarization.
Revised: Models trained with this objective can then be sampled from, or searched over (e.g. using beam search), to provide generations in downstream tasks such as machine translation or summarization.

Original: Non-autoregressive models (Gu et al., 2017) are a different variety of generative models, in which a sequence is generated in a single pass (removing the autoregressive conditioning on previously generated tokens) with multiple revision-level passes, often in the name of efficiency.
Revised: Non-autoregressive models (Gu et al., 2017) are a different variety of generative models, in which a sequence is generated in a single pass (removing the autoregressive conditioning on previously generated tokens) with multiple revision-level passes, often in the name of efficiency.

Original: 2. E DITING P ROCESSES
Revised: 2. E DITING P ROCESSES

Original: Editing processes (Reid & Neubig, 2022) are a paradigm for modeling text by way of incremental revisions, taking inspiration from the the way humans generate text.
Revised: Editing processes (Reid & Neubig, 2022) are a paradigm for modeling text by way of incremental revisions, taking inspiration from the the way humans generate text.

Original: Specifically, let
Revised: Specifically, let

Original: X = { x 0 , x 1 , . . .
Revised: X = { x 0 , x 1 , . . .

Original: , x R } be a series of R versions of a document, where x 0 , x i , x R represents the initial, intermediate (at timestep t ), and final/current state of a document, respectively.
Revised: , x R } be a series of R versions of a document, where x 0 , x i , x R represents the initial, intermediate (at timestep t ), and final/current state of a document, respectively.

Original: Using editing processes, we can model the probability of this series of documents versions occurring consecutively as follows:
Revised: Using editing processes, we can model the probability of this series of documents versions occurring consecutively as follows:

Original: [Equation]
Revised: [Equation]

Original: With this formulation, editing processes can also be used to calculate the probability of only the final document while taking into account previous revisions, which is not possible in the traditional text generation setup as intermediate revisions are not explicitly known.
Revised: With this formulation, editing processes can also be used to calculate the probability of only the final document while taking into account previous revisions, which is not possible in the traditional text generation setup as intermediate revisions are not explicitly known, using the equation below (Reid & Neubig, 2022).

Original: [Equation]
Revised: [Equation]

Original: 2. D IFFUSION M ODELS
Revised: 2. D IFFUSION M ODELS

Original: We now make the connection between editing processes and diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020).
Revised: We now make the connection between editing processes and diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020).

Original: Continuous diffusion processes are commonly applied in computer vision tasks to iteratively convert a sample of noise into an image.
Revised: Continuous diffusion processes are commonly applied in computer vision tasks to iteratively convert a sample of noise into an image.

Original: This can be seen as an edit process in which the model iteratively edits a noisy image to bring it closer to a final, complete image.
Revised: This can be seen as an edit process in which the model iteratively edits a noisy image to bring it closer to a final, complete image.

Original: These continuous diffusion models are often trained by modeling a Markov chain x T . . .
Revised: These continuous diffusion models are often trained by modeling a Markov chain x T . . .

Original: x t . . .
Revised: x t . . .

Original: x 0 , where x 0 represents the original image and x T represents Gaussian noise.
Revised: x 0 , where x 0 represents the original image and x T represents Gaussian noise.

Original: This chain is typically produced by incrementally adding Gaussian noise to x t to form x t +1 (known as the forward or corruption process), wherein a model parameterised by p θ is trained to reverse (or “ denoise ”)
Revised: This chain is typically produced by incrementally adding Gaussian noise to x t to form x t +1 (known as the forward or corruption process), wherein a model parameterised by p θ is trained to reverse (or “ denoise ”)

Original: this process to form the chain (cid:80) Ti =1 p θ ( x t − 1 | x t ) .
Revised: this process to form the chain (cid:80) Ti =1 p θ ( x t − 1 | x t ) .

Original: Analogized to text, this allows us to formulate natural edit processes as a discrete diffusion process in which a null string or a prototype is iteratively edited into free form text.
Revised: Analogized to text, this allows us to formulate natural edit processes as a discrete diffusion process in which a null string or a prototype is iteratively edited into free form text.

Original: Our D IFFUS ER method (Figure 1) takes inspiration from this process, but parameterises the corruption process by way of sampled discrete edit operations applied over a discrete sequence of tokens.
Revised: Our D IFFUS ER method (Figure 1) takes inspiration from this process, but parameterises the corruption process by way of sampled discrete edit operations applied over a discrete sequence of tokens.

Original: The success of our method supports the findings in the vision domain Bansal et al.
Revised: The success of our method supports the findings in the vision domain Bansal et al.

Original: (2022), where it is found that diffusion models can learn to invert arbirtary transformations.
Revised: (2022), where it is found that diffusion models can learn to invert arbirtary transformations.

Original: Previous work in diffusion models has largely focused on computer vision (Ho et al., 2020;
Revised: Previous work in diffusion models has largely focused on computer vision (Ho et al., 2020;

Original: Austin et al., 2021), in which the diffusion process is applied to raw image values.
Revised: Austin et al., 2021), in which the diffusion process is applied to raw image values.

Original: Within the context of natural language, both discrete diffusion models using only replacement operations (either applied to random tokens or masked tokens)
Revised: Within the context of natural language, both discrete diffusion models using only replacement operations (either applied to random tokens or masked tokens)

Original: (Savinov et al., 2022; Austin et al., 2021), and continuous diffusion over word embeddings (Li et al., 2022) have been proposed.
Revised: (Savinov et al., 2022; Austin et al., 2021), and continuous diffusion over word embeddings (Li et al., 2022) have been proposed.

Original: Our model is a more flexible approach, using all four edit operations, towards diffusion models when compared with this work owing to its edit process formulation, and is also more compatible with current models (e.g. AR bootstrapping).
Revised: Our model is a more flexible approach, using all four edit operations, towards diffusion models when compared with this work owing to its edit process formulation, and is also more compatible with current models (e.g. AR bootstrapping).

Original:  D IFFUS ER
Revised:  D IFFUS ER

Original: D IFFUS ER, being a diffusion-based method, has two main procedures: corruption and denoising.
Revised: D IFFUS ER, being a diffusion-based method, has two main procedures: corruption and denoising.

Original: Unlike previous work (Ghazvininejad et al., 2019; Savinov et al., 2022; Gu et al., 2019) in which this procedure is relatively inflexible (e.g., due to length restrictions and/or using continuous representations for the basis of the diffusion process), both our corruption process and denoising process are based on Levenshtein operations, allowing our model to learn to take advantage of the flexibility of text editing when generating.
Revised: Unlike previous work (Ghazvininejad et al., 2019; Savinov et al., 2022; Gu et al., 2019) in which this procedure is relatively inflexible (e.g., due to length restrictions and/or using continuous representations for the basis of the diffusion process), both our corruption process and denoising process are based on Levenshtein operations, allowing our model to learn to take advantage of the flexibility of text editing when generating.

Original: 3. E DIT O PERATIONS
Revised: 3. E DIT O PERATIONS

Original: Given the central role of the Levenshtein edit operations in our models, we provide a brief overview of each operation and its role in the editing process.
Revised: Given the central role of the Levenshtein edit operations in our models, we provide a brief overview of each operation and its role in the editing process.

Original: We use Figure 1 as a guide when explaining each operation.
Revised: We use Figure 1 as a guide when explaining each operation.

Original: I NSERT :
Revised: I NSERT :

Original: The insertion operation is used to add new text to a sequence.
Revised: The insertion operation is used to add new text to a sequence.

Original: For example in Figure 1, “ uses editing processes ” is added by DiffusER at timestep x T − 2 .
Revised: For example in Figure 1, “ uses editing processes ” is added by DiffusER at timestep x T − 2 .

Original: D ELETE : The deletion operation erases existing text.
Revised: D ELETE : The deletion operation erases existing text.

Original: In Figure 1, this is shown when “ These ” gets deleted at timestep x T − 2 → x T − 3 .
Revised: In Figure 1, this is shown when “ These ” gets deleted at timestep x T − 2 → x T − 3 .

Original: R EPLACE :
Revised: R EPLACE :

Original: The replacement operation works overwriting existing text with new text.
Revised: The replacement operation works overwriting existing text with new text.

Original: This is shown in Figure 1 at step x T → x T − 1 where “ filter Toronto guilty trough feel ” is replaced by “ These model guilty named DiffusER ”.
Revised: This is shown in Figure 1 at step x T → x T − 1 where “ filter Toronto guilty trough feel ” is replaced by “ These model guilty named DiffusER ”.

Original: K EEP : The keep operation ensures that a portion of the text remains unchanged into the next iteration.
Revised: K EEP : The keep operation ensures that a portion of the text remains unchanged into the next iteration.

Original: This is illustrated in timestep x T − 2 → x T − 3 where “ model named DiffusER ” is kept.
Revised: This is illustrated in timestep x T − 2 → x T − 3 where “ model named DiffusER ” is kept.

Original: 3. E DIT - BASED C ORRUPTION
Revised: 3. E DIT - BASED C ORRUPTION

Original: The four Levenshtein edit operations described above allow us to transform any arbitrary sequence of tokens into another.
Revised: The four Levenshtein edit operations described above allow us to transform any arbitrary sequence of tokens into another.

Original: This is in contrast to iterative mask replacement, which can only introduce new tokens (Ghazvininejad et al., 2019;
Revised: This is in contrast to iterative mask replacement, which can only introduce new tokens (Ghazvininejad et al., 2019;

Original: Austin et al., 2021; Savinov et al., 2022).
Revised: Austin et al., 2021; Savinov et al., 2022).

Original: For every timestep i , corruption process q ( x i | x i − 1 ; E t , E l ) is parameterized by two distributions: the distribution over edit types E t (e.g. 60% keep, 20% replace, 10% delete, 10% insert), and the distribution over edit length E l .
Revised: For every timestep i , corruption process q ( x i | x i − 1 ; E t , E l ) is parameterized by two distributions: the distribution over edit types E t (e.g. 60% keep, 20% replace, 10% delete, 10% insert), and the distribution over edit length E l .

Original: The latter can be parameterized by any distribution over non-negative integers, such as a uniform distribution or a Poisson distribution.
Revised: The latter can be parameterized by any distribution over non-negative integers, such as a uniform distribution or a Poisson distribution.

Original: For instance, to learn a deletion operation in the reconstruction process, we insert randomly sampled distractor tokens, whereas, to learn an insertion operation we delete a subset of tokens contained in the sequence.
Revised: For instance, to learn a deletion operation in the reconstruction process, we insert randomly sampled distractor tokens, whereas, to learn an insertion operation we delete a subset of tokens contained in the sequence.

Original: 3. E DIT - BASED R ECONSTRUCTION
Revised: 3. E DIT - BASED R ECONSTRUCTION

Original: Our generative process is trained via the E dit-based R econstruction (ER) process.
Revised: Our generative process is trained via the E dit-based R econstruction (ER) process.

Original: ER can be thought of as the opposite of our corruption process, in which we need to find the appropriate edit operations to transform x T to x 0 , by way of x T − 1 , . . .
Revised: ER can be thought of as the opposite of our corruption process, in which we need to find the appropriate edit operations to transform x T to x 0 , by way of x T − 1 , . . .

Original: , x 1 .
Revised: , x 1 .

Original: That is, given a corrupted sequence x T , we aim to learn the process by which we can reverse the corruption in the following form.
Revised: That is, given a corrupted sequence x T , we aim to learn the process by which we can reverse the corruption in the following form.

Original: [Equation]
Revised: [Equation]

Original: Given that, we model the likelihood of each timestep x t , this can also be referred to as an edit process (Reid & Neubig, 2022).
Revised: Given that, we model the likelihood of each timestep x t , this can also be referred to as an edit process (Reid & Neubig, 2022).

Original: As we include an edit process in our model and use Levenshtein tags for editing, one can think of ER as two distinct steps: identify which edits should take place (tagging process) and deciding which tokens should go in these positions (generative process).
Revised: As we include an edit process in our model and use Levenshtein tags for editing, one can think of ER as two distinct steps: identify which edits should take place (tagging process) and deciding which tokens should go in these positions (generative process).

Original: This decomposition is shown here:
Revised: This decomposition is shown here:

Original: [Equation]
Revised: [Equation]

Original: where p tag θ parameterises the tagging model to estimate the likelihood of producing a given set of
Revised: where p tag θ parameterises the tagging model to estimate the likelihood of producing a given set of

Original: Levenshtein edit operations { I NSERT ,D ELETE ,K EEP ,R EPLACE } given x t , and p gen θ parametersies the generator model given sequence x t and edit operations e t .
Revised: Levenshtein edit operations { I NSERT ,D ELETE ,K EEP ,R EPLACE } given x t , and p gen θ parametersies the generator model given sequence x t and edit operations e t .

Original: This decomposition via editoperations allows the generation process to be more controllable and more flexible as it allows up to explicitly specify edit types associated with tokens to be edited, rather than leaving both processes to be implicit.
Revised: This decomposition via editoperations allows the generation process to be more controllable and more flexible as it allows up to explicitly specify edit types associated with tokens to be edited, rather than leaving both processes to be implicit.

Original: 3. I MPLEMENTING D IFFUS ER WITH T RANSFORMERS
Revised: 3. I MPLEMENTING D IFFUS ER WITH T RANSFORMERS

Original: When implemented with Transformers (Vaswani et al., 2017), D IFFUS ER consists of two components: a tagger and generator.
Revised: When implemented with Transformers (Vaswani et al., 2017), D IFFUS ER consists of two components: a tagger and generator.

Original: The tagger, a transformer network, is trained using cross-entropy loss over the ground-truth tag types to predict the edit operations that should be applied to the sequence, in preparation for the next generation step.
Revised: The tagger, a transformer network, is trained using cross-entropy loss over the ground-truth tag types to predict the edit operations that should be applied to the sequence, in preparation for the next generation step.

Original: Then, in the generation step, after removing tokens selected for deletion, we sum a learned embedding to insert and replace types and generate the inserted and replaced sequences autoregressively.
Revised: Then, in the generation step, after removing tokens selected for deletion, we sum a learned embedding to insert and replace types and generate the inserted and replaced sequences autoregressively.

Original: Following this, we feed the output of this diffusion step into the tagger and perform another diffusion step.
Revised: Following this, we feed the output of this diffusion step into the tagger and perform another diffusion step.

Original: One step of this process can be compared to the reconstruction process used in Aghajanyan et al.
Revised: One step of this process can be compared to the reconstruction process used in Aghajanyan et al.

Original: 3. D ECODING M ETHODS
Revised: 3. D ECODING M ETHODS

Original: D IFFUS ER has an inherently different generation process from a standard autoregressive language generation model—in addition to operating on a sequence/token level (in which generation is composed of generating individual tokens in a single-revision; intra-revision ), we also operate on a revision level (in which the text is expanded across diffusion steps, inter-revision ).
Revised: D IFFUS ER has an inherently different generation process from a standard autoregressive language generation model—in addition to operating on a sequence/token level (in which generation is composed of generating individual tokens in a single-revision; intra-revision ), we also operate on a revision level (in which the text is expanded across diffusion steps, inter-revision ).

Original: This allows us to experiment with different methods for decoding on both the intra-revision (single sequence level) and inter-revision levels (multiple version level), which we explain below.
Revised: This allows us to experiment with different methods for decoding on both the intra-revision (single sequence level) and inter-revision levels (multiple version level), which we explain below.

Original: Beam Search One method for decoding is to perform beam search over b hypotheses at every step on the output of our autoregressive generator (intra-revision level), while performing greedy decoding at the inter-revision level.
Revised: Beam Search One method for decoding is to perform beam search over b hypotheses at every step on the output of our autoregressive generator (intra-revision level), while performing greedy decoding at the inter-revision level.

Original: Although being conceptually straightforward, this method has the limitation of not searching over the inter-revision space (despite revisions being a key component of our approach).
Revised: Although being conceptually straightforward, this method has the limitation of not searching over the inter-revision space (despite revisions being a key component of our approach).

Original: 2D Beam Search We propose 2D beam search, in which we extend beam search as it is applied to token-level autoregressive generative models, and perform beam search using both an intra-revision width of b and an inter-revision beam width of r .
Revised: 2D Beam Search We propose 2D beam search, in which we extend beam search as it is applied to token-level autoregressive generative models, and perform beam search using both an intra-revision width of b and an inter-revision beam width of r .

Original: This allows us to perform search on the interrevision level, which we find results in better downstream performance, but increases the beam count to r × b beams.
Revised: This allows us to perform search on the interrevision level, which we find results in better downstream performance, but increases the beam count to r × b beams.

Original: Assuming a fixed sequence length and maximum number of diffusion steps, we would decode as follows: We first use beam search with width b at the token level and take the r most likely candidates (measured with log-likelihood).
Revised: Assuming a fixed sequence length and maximum number of diffusion steps, we would decode as follows: We first use beam search with width b at the token level and take the r most likely candidates (measured with log-likelihood).

Original: These r candidates are then fed to the next step of the diffusion model, wherein for each of r hypotheses the next diffusion step is performed with the token-level generator decoding with beam width of b .
Revised: These r candidates are then fed to the next step of the diffusion model, wherein for each of r hypotheses the next diffusion step is performed with the token-level generator decoding with beam width of b .

Original: This leads us to have r × b candidate hypotheses, of which we take the top r .
Revised: This leads us to have r × b candidate hypotheses, of which we take the top r .

Original: This process repeats for each diffusion step thereafter.
Revised: This process repeats for each diffusion step thereafter.

Original: Nucleus Sampling To improve the diversity of generations, we also consider a nucleus sampling based approach, where at every timestep x t , we use nucleus sampling (Holtzman et al., 2019) with p = 0 .
Revised: Nucleus Sampling To improve the diversity of generations, we also consider a nucleus sampling based approach, where at every timestep x t , we use nucleus sampling (Holtzman et al., 2019) with p = 0 .

Original: 6 to sample each token autoregressively at the intra-revision level, and greedily decode at the inter-revision level (i.e. no search or sampling is performed over multiple diffusion steps).
Revised: 6 to sample each token autoregressively at the intra-revision level, and greedily decode at the inter-revision level (i.e. no search or sampling is performed over multiple diffusion steps).

Original: 3. D ECODER I NITIALIZATION T ECHNIQUES
Revised: 3. D ECODER I NITIALIZATION T ECHNIQUES

Original: These model guilty named DiffusER filter
Revised: These model guilty named DiffusER filter

Original: Toronto guilty trough feel
Revised: Toronto guilty trough feel

Original: This model named DiffuSER is edit
Revised: This model named DiffuSER is edit

Original: This model named DiffusER uses editing processes
Revised: This model named DiffusER uses editing processes

Original: Revision and editing ... generation task
Revised: Revision and editing ... generation task

Original: This model named DiffusER uses editing processes
Revised: This model named DiffusER uses editing processes

Original: Random Tokens
Revised: Random Tokens

Original: These model guilty DiffusER ∅
Revised: These model guilty DiffusER ∅

Original: Null Sequence
Revised: Null Sequence

Original: AR Bootstrap
Revised: AR Bootstrap

Original: Source Bootstrap
Revised: Source Bootstrap

Original: Since our model is based on edit processes, it offers flexibility in terms of the discrete sequence from which to initialize the text generation.
Revised: Since our model is based on edit processes, it offers flexibility in terms of the discrete sequence from which to initialize the text generation.

Original: Previous work on non-autoregressive translation often starts with [MASK] tokens (Ghazvininejad et al., 2019), a null string (Gu et al., 2019) or random tokens (Savinov et al., 2022).
Revised: Previous work on non-autoregressive translation often starts with [MASK] tokens (Ghazvininejad et al., 2019), a null string (Gu et al., 2019) or random tokens (Savinov et al., 2022).

Original: We include the latter two methods in our experiments, in addition to (1) experimenting with an AR Bootstrap, in which we learn to bootstrap from text generated by a purely autoregressive model, and (2) proposing to use the source-side text as an initial state for the
Revised: We include the latter two methods in our experiments, in addition to (1) experimenting with an AR Bootstrap, in which we learn to bootstrap from text generated by a purely autoregressive model, and (2) proposing to use the source-side text as an initial state for the

Original: D IFFUS ER decoder.
Revised: D IFFUS ER decoder.

Original: Null Sequence
Revised: Null Sequence

Original: In this setting, we simply initialize D IFFUS ER with a null string, in which the first edit is constrained to be insertion.
Revised: In this setting, we simply initialize D IFFUS ER with a null string, in which the first edit is constrained to be insertion.

Original: Random Tokens
Revised: Random Tokens

Original: In this setting, we initialize D IFFUS ER with a series of random tokens, following (Savinov et al., 2022).
Revised: In this setting, we initialize D IFFUS ER with a series of random tokens, following (Savinov et al., 2022).

Original: The model then learns to edit this random sequence.
Revised: The model then learns to edit this random sequence.

Original: AR Bootstrap We bootstrap the reverse diffusion process by taking the output of D IFFUS ER constrained to generate autoregressively (essentially mimicking a standard autoregressive generator).
Revised: AR Bootstrap We bootstrap the reverse diffusion process by taking the output of D IFFUS ER constrained to generate autoregressively (essentially mimicking a standard autoregressive generator).

Original: We then use D IFFUS ER to further edit the output of this operation.
Revised: We then use D IFFUS ER to further edit the output of this operation.

Original: Source Bootstrap
Revised: Source Bootstrap

Original: In a sequence-to-sequence setting, we can also generate by bootstrapping using the source text, by setting x T to be equivalent to s .
Revised: In a sequence-to-sequence setting, we can also generate by bootstrapping using the source text, by setting x T to be equivalent to s .

Original: As we show in later sections, this is particularly useful in tasks such as summarization in which the output can be easily formulated as an editing version of the input.
Revised: As we show in later sections, this is particularly useful in tasks such as summarization in which the output can be easily formulated as an editing version of the input.

Original:  E XPERIMENTS 4. M ODELS D IFFUS ER
Revised:  E XPERIMENTS 4. M ODELS D IFFUS ER

Original: We instantiate D IFFUS ER with two separate Transformer models for the tagger and generator.
Revised: We instantiate D IFFUS ER with two separate Transformer models for the tagger and generator.

Original: We use the Transformer-base  (Vaswani et al., 2017) architecture, with 6 layers, for the a hidden dimension of 512, feedforward dimension of 2048, 8 attention heads, and dropout p = 0 .
Revised: We use the Transformer-base encoder-decoder (Vaswani et al., 2017) architecture, with 6 layers, for the a hidden dimension of 512, feedforward dimension of 2048, 8 attention heads, and dropout p = 0 .

Original: Baselines (MT & Summ)
Revised: Baselines (MT & Summ)

Original: We use several Transformer baselines from previous literature for our various tasks.
Revised: We use several Transformer baselines from previous literature for our various tasks.

Original: We include a conventional 6-layer encoder-decoder Transformer model from Vaswani et al.
Revised: We include a conventional 6-layer encoder-decoder Transformer model from Vaswani et al.

Original: (2017), as well as models proposed in related work from the non-autoregressive generation literature: Levensthein Transformer (Gu et al., 2019), CMLM (Ghazvininejad et al., 2019), DisCo (Kasai et al., 2020b), Imputer (Saharia et al., 2020), and SUNDAE (Savinov et al., 2022).
Revised: (2017), as well as models proposed in related work from the non-autoregressive generation literature: Levensthein Transformer (Gu et al., 2019), CMLM (Ghazvininejad et al., 2019), DisCo (Kasai et al., 2020a), Imputer (Saharia et al., 2020), and SUNDAE (Savinov et al., 2022).

Original: 4. T ASKS
Revised: 4. T ASKS

Original: Machine Translation We use the WMT’14 English-German dataset for our machine translation experiments.
Revised: Machine Translation We use the WMT’14 English-German dataset for our machine translation experiments.

Original: We use the same preprocessing and post-processing steps as Ghazvininejad et al.
Revised: We use the same preprocessing and post-processing steps as Ghazvininejad et al.

Original: Unlike the standard in non-autoregressive translation work (Zhou et al., 2019), we focus on using the gold machine translation data instead of distilled data.
Revised: Unlike the standard in non-autoregressive translation work (Zhou et al., 2019), we focus on using the gold machine translation data instead of distilled data.

Original: We use a Poisson distribution E l ( λ = 3) over edit operation lengths in our corruption process.
Revised: We use a Poisson distribution E l ( λ = 3) over edit operation lengths in our corruption process.

Original: Note that we compute the edit operations over words rather than tokens.
Revised: Note that we compute the edit operations over words rather than tokens.

Original: For this task, as well as the following ones, we use 12 diffusion steps, b = 5 , and r = 3 for beam search, and
Revised: For this task, as well as the following ones, we use 12 diffusion steps, b = 5 , and r = 3 for beam search, and

Original: E t (60% K EEP , 20% R EPLACE , 10% I NSERT , 10% D ELETE ) based on numbers from preliminary experiments.
Revised: E t (60% K EEP , 20% R EPLACE , 10% I NSERT , 10% D ELETE ) based on numbers from preliminary experiments.

Original: Summarization We also benchmark on the CNN/DailyMail dataset for summarization (Nallapati et al., 2016).
Revised: Summarization We also benchmark on the CNN/DailyMail dataset for summarization (Nallapati et al., 2016).

Original: Summarization is different in nature from machine translation in that it can be described as more conducive to edits as a good summary tends to preserve many parts of the input.
Revised: Summarization is different in nature from machine translation in that it can be described as more conducive to edits as a good summary tends to preserve many parts of the input.

Original: We use the same post-processing steps as See et al.
Revised: We use the same post-processing steps as See et al.

Original: We use a Poisson distribution E l ( λ = 8) over edit operation lengths in our corruption process (to roughly model sentence boundaries).
Revised: We use a Poisson distribution E l ( λ = 8) over edit operation lengths in our corruption process (to roughly model sentence boundaries).

Original: Text Style Transfer We perform experiments using the Yelp (Shen et al., 2017) dataset for the unsupervised text-style transfer task.
Revised: Text Style Transfer We perform experiments using the Yelp (Shen et al., 2017) dataset for the unsupervised text-style transfer task.

Original: We compare against methods such as Tag-and-Generate (Madaan et al., 2020), Masker (Malmi et al., 2020), and LEWIS (Reid & Zhong, 2021).
Revised: We compare against methods such as Tag-and-Generate (Madaan et al., 2020), Masker (Malmi et al., 2020), and LEWIS (Reid & Zhong, 2021).

Original: In contrast with machine translation and summarization, text style transfer datasets are often unaligned (i.e. without source-target pairs) leading to the prominence of unsupervised text style transfer methods.
Revised: In contrast with machine translation and summarization, text style transfer datasets are often unaligned (i.e. without source-target pairs) leading to the prominence of unsupervised text style transfer methods.

Original: We propose a method of performing unsupervised text style transfer using D IFFUS ER, following the synthetic generation method in Reid & Zhong (2021).
Revised: We propose a method of performing unsupervised text style transfer using D IFFUS ER, following the synthetic generation method in Reid & Zhong (2021).

Original: We train two separate, style-specific (e.g. positive and negative)
Revised: We train two separate, style-specific (e.g. positive and negative)

Original: D IFFUS ER models on the style-specific data.
Revised: D IFFUS ER models on the style-specific data.

Original: We then perform transfer at test time, feeding text from each style into the model trained to edit in the opposite style (e.g. positive text → negative D IFFUS ER model; negative text → positive D IFFUS ER model). 
Revised: We then perform transfer at test time, feeding text from each style into the model trained to edit in the opposite style (e.g. positive text → negative D IFFUS ER model; negative text → positive D IFFUS ER model). Following stan-

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: dard practice, we measure performance with BLEU, Self-BLEU and Accuracy (based on a classifier trained to disambiguate between different styles of text; we use the classifier from Reid & Zhong (2021)).

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: Main Results We summarize our main results on both machine translation and summarization in Table 1.
Revised: Main Results We summarize our main results on both machine translation and summarization in Table 1.

Original: As can be seen, for both machine translation and summarization tasks, D IFFUS ER, using 12 diffusion steps, outperforms all non-autoregressive baselines 2 and rivals or outperforms the fully autoregressive model.
Revised: As can be seen, for both machine translation and summarization tasks, D IFFUS ER, using 12 diffusion steps, outperforms all non-autoregressive baselines 2 and rivals or outperforms the fully autoregressive model.

Original: Particularly interesting is how the various methods of initializing our model (i.e. AR Bootstrap and Source Bootstrap) can further improve performance well beyond the autoregressive baseline, depending on the task.
Revised: Particularly interesting is how the various methods of initializing our model (i.e. AR Bootstrap and Source Bootstrap) can further improve performance well beyond the autoregressive baseline, depending on the task.

Original: We can see that for summarization, bootstrapping from the source input is more effective than bootstrapping from an abstractive autoregressive model.
Revised: We can see that for summarization, bootstrapping from the source input is more effective than bootstrapping from an abstractive autoregressive model.

Original: However, for both tasks, unlike many non-autoregressive methods, we show that D IFFUS ER is complementary with token-level autoregressive methods and can be used naturally in conjunction with them.
Revised: However, for both tasks, unlike many non-autoregressive methods, we show that D IFFUS ER is complementary with token-level autoregressive methods and can be used naturally in conjunction with them.

Original: Style Transfer Results We also perform unsupervised text style transfer using our D IFFUS ER models using the Yelp (Shen et al., 2017) dataset.
Revised: Style Transfer Results We also perform unsupervised text style transfer using our D IFFUS ER models using the Yelp (Shen et al., 2017) dataset.

Original: The results can be seen in Table 2.
Revised: The results can be seen in Table 2.

Original: We show that even without task-specific techniques (such as synthetic data generation and classifier based stylespecific token identification), we still have competitive performance with state of the art methods.
Revised: We show that even without task-specific techniques (such as synthetic data generation and classifier based stylespecific token identification), we still have competitive performance with state of the art methods.

Original: 4. A NALYSIS
Revised: 4. A NALYSIS

Original: [Table]
Revised: [Table]

Original: We perform additional analyses on D IFFUS ER, specifically focusing on the decoding method, the number of iterations versus the final BLEU score, and also a qualitative analysis of how text changes at every step.
Revised: We perform additional analyses on D IFFUS ER, specifically focusing on the decoding method, the number of iterations versus the final BLEU score, and also a qualitative analysis of how text changes at every step.

Original: Decoding Method Ablation We perform an ablation of the decoding method, using D IFFUS ER for 12 steps (as used in our main results)

Original: [Figure]
Revised: [Figure]

Original:  and showing results when comparing greedy decoding, (1D) beam search, nucleus decoding, and 2D beam search.
Revised: Decoding Method Ablation We perform an ablation of the decoding method, using D IFFUS ER for 12 steps (as used in our main results) and showing results when comparing greedy decoding, (1D) beam search, nucleus decoding and 2D beam search.

Original: We show that 2D-beam search tends to perform the best, likely because it searches over multiple diffusion steps, while other methods (greedy, beam, nucleus) are still competitive.
Revised: We show that 2D-beam search tends to perform the best, likely because it searches over multiple diffusion steps, while other methods (greedy, beam, nucleus) are still competitive.

Original: Number of Edit Steps versus Performance We perform an analysis where we compare the number of timesteps in our denoising diffusion process and the final BLEU score on WMT’14 En-De when using 2D-Beam Search and random token initialization in Figure 4.
Revised: Number of Edit Steps versus Performance We perform an analysis where we compare the number of timesteps in our denoising diffusion process and the final BLEU score on WMT’14 En-De when using 2D-Beam Search and random token initialization in Figure 4.

Original: Here it can be seen that most performance gains are in the initial diffusion timesteps (0-10), with diminishing gains (for machine translation) or gradual losses (for summarization) between 10 and 30, after which performance marginally decreases towards 60 steps.
Revised: Here it can be seen that most performance gains are in the initial diffusion timesteps (0-10), with diminishing gains (for machine translation) or gradual losses (for summarization) between 10 and 30, after which performance marginally decreases towards 60 steps.

Original: How does text change every step?
Revised: How does text change every step?

Original: We include a qualitative sample from our D IFFUS ER summarization model (Table 4).
Revised: We include a qualitative sample from our D IFFUS ER summarization model (Table 4).

Original: We find that D IFFUS ER learns edit processes intuitive to the task at hand: namely largely deleting portions and making minor edits to the remaining text (similar to how a human may perform summarization given a news article).
Revised: We find that D IFFUS ER learns edit processes intuitive to the task at hand: namely largely deleting portions and making minor edits to the remaining text (similar to how a human may perform summarization given a news article).

Original: Time comparsion between decoding methods We also measure the impact of the various decoding algorithms we used with results shown in Figure 3.
Revised: Time comparsion between decoding methods We also measure the impact of the various decoding algorithms we used with results shown in Figure 3.

Original: Beam search and 2D-Beam Search performs significantly slower than greedy and nucleus sampling, demonstrating the potential for improved decoding algorithms tailored for improving the trade-off between efficiency and accuracy in diffusion models.
Revised: Beam search and 2D-Beam Search performs significantly slower than greedy and nucleus sampling, demonstrating the potential for improved decoding algorithms tailored for improving the trade-off between efficiency and accuracy in diffusion models.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Non-Autoregressive Generation Work in machine translation has explored non/semiautoregressive generation (Gu et al., 2017; Lee et al., 2018), which often includes an iterative refinement step (Lee et al., 2018;
Revised: Non-Autoregressive Generation Work in machine translation has explored non/semiautoregressive generation (Gu et al., 2017; Lee et al., 2018), which often includes an iterative refinement step (Lee et al., 2018;

Original: Ghazvininejad et al., 2019; Kasai et al., 2020a; Gu et al., 2019).
Revised: Ghazvininejad et al., 2019; Kasai et al., 2020a; Gu et al., 2019).

Original: Previous methods in this space are often highly specialized underperform non-autoregressive methods due to the constraints imposed on generation for efficiency.
Revised: Previous methods in this space are often highly specialized underperform non-autoregressive methods due to the constraints imposed on generation for efficiency.

Original: This being said, Kasai et al.
Revised: This being said, Kasai et al.

Original: (2020c) demonstrated that non-autoregressive models are actually comparable in speed when using a larger batch size instead of 1 .
Revised: (2020b) demonstrated that non-autoregressive models are actually comparable in speed when using a larger batch size instead of 1 .

Original: Our method allows us to hone in on the notion of iterative refinement by way of editing processes, and is also relatively general, allowing us to combine D IFFUS ER with standard autoregressive models.
Revised: Our method allows us to hone in on the notion of iterative refinement by way of editing processes, and is also relatively general, allowing us to combine D IFFUS ER with standard autoregressive models.

Original: Learning Properties of Edits Previous work has also looked at studying or exploiting the properties of edits.
Revised: Learning Properties of Edits Previous work has also looked at studying or exploiting the properties of edits.

Original: This was initially worked on in the context of vector representation learning of edits(Yin et al., 2019; Marrese-Taylor et al., 2021).
Revised: This was initially worked on in the context of vector representation learning of edits

Original: Concurrently, a line of work has used edits for specific tasks such as sentence fusion, style transfer and grammatical error correction (Malmi et al.,

Original: [Table]
Revised: [Table]

Original:  2019; 2020; Reid
Revised: (Yin et al., 2019; Marrese-

Original:  Zhong, 
Revised: Concurrently, a line of work has used edits for specific tasks such as sentence fusion, style transfer and grammatical error correction (Malmi et al., 2019; 2020; Reid & Zhong, 2021; Omelianchuk et al., 2020).

Original: Recent work has proposed editing processes (Reid & Neubig, 2022), in which document generation is looked at through the lens of its revision history, rather than just at a token level.
Revised: Recent work has proposed editing processes (Reid & Neubig, 2022), in which document generation is looked at through the lens of its revision history, rather than just at a token level.

Original: We take inspiration from this work and devise a process by which arbitrary text generation tasks can be fitted into this framework.
Revised: We take inspiration from this work and devise a process by which arbitrary text generation tasks can be fitted into this framework.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We proposed D IFFUS ER, an diffusion-based generative model for text using edits.
Revised: We proposed D IFFUS ER, an diffusion-based generative model for text using edits.

Original: D IFFUS ER shows improvements across the tasks considered (machine translation, summarization, style transfer), with improved generative flexibility via incremental text improvement, and compatibility with standard autoregressive models.
Revised: D IFFUS ER shows improvements across the tasks considered (machine translation, summarization, style transfer), with improved generative flexibility via incremental text improvement, and compatibility with standard autoregressive models.

Original: We hope that D IFFUS ER with spur research on edit-based generative models, with further potentials including how we can leverage edits to ensemble models (regardless of parameter count) in the discrete space.
Revised: We hope that D IFFUS ER with spur research on edit-based generative models, with further potentials including how we can leverage edits to ensemble models (regardless of parameter count) in the discrete space.