L EARNING TO M ODEL E DITING P ROCESSES
Anonymous authors
Paper under double-blind review
A BSTRACT
Most existing sequence generation models produce outputs in one pass, usually left-to-right.
However, this is in contrast with a more natural approach that humans use in generating content; iterative reﬁnement and editing.
Recent work has introduced edit-based models for various tasks (such as neural machine translation and text style transfer), but these generally model a single edit.
In this work, we propose modeling editing processes , modeling the whole process of iteratively generating sequences.
We form a conceptual framework to describe the likelihood of multi-step edits, and describe neural models that can learn a generative model of sequences based on these multi-step edits.
We introduce baseline results and metrics on this task, ﬁnding that modeling editing processes improves performance on a variety of axes on both our proposed task and related downstream tasks compared to previous single-step models of edits.
I NTRODUCTION
Revising and editing are a central part of the the human creative worﬂow, with most original content (e.g. art, books, articles, source code) being developed not in a single iteration, but in many iterations with each more reﬁned than the last.
How can we model these editing processes from inception to completion?
In this paper, we attempt to provide a ﬁrst answer to this question, speciﬁcally focusing on generation of sequential data such as natural language documents or source code.
Most current work on language generation tasks such as machine translation (Vaswani et al., 2017), language modeling (Baevski & Auli, 2018), or summarization (See et al., 2017) generates the target sentence or document in a single pass (usually from left to right).
There has been a reasonable amount of work that can generate edits to existing sequences for the purposes of post-editing, grammatical error correction (Omelianchuk et al., 2020), text style transfer (Mallinson et al., 2020; Malmi et al., 2020; Reid & Zhong, 2021), sentence fusion (Malmi et al., 2019), or machine translation (Gu et al., 2019).
However, these works all 1) model only a single editing step and 2) do not fully deﬁne a model of incrementally editing a document from a blank slate to the ﬁnal text, and thus do not stand in for the one-pass generative models of sequences described above.
In this context, we propose the task of modeling editing processes in which we look to explicitly model the likelihood of the entire process of revising a document to a polished form.
In particular, and in contrast to previous works on modeling edits, we hypothesize that in order to edit more accurately, instead of simply learning to predict the next revision given the current revision, we should have context of multiple previous revisions when deciding when and how to edit the document next.
Given the novelty of this task, this paper simultaneously proposes 1) both baseline and novel models for the task, 2) evaluation methodology, and 3) evaluation datasets that can be used to compare model efﬁcacy.
The proposed multi-step editing model predicts discrete edit operations (Levenshtein, 1966) to enable progressive reﬁnement as shown in Figure 1, rather than framing sequence editing as a sequence to sequence task (Reid & Zhong, 2021; Faltings et al., 2021).
In the ﬁgure, for each step of the editing process discrete operations (insert, replace, delete, keep) are predicted and then actions (such as generating a replaced span) are performed based on this.
This has two beneﬁts: 1) it allows the model to scale well with respect to input sequence length, and 2) allows us to make substantial changes with fewer actions (Grangier & Auli, 2018).
We use these edit operations to condition a semi-autoregressive model that is able to insert and replace multiple spans at once.
Combined with
The animal cute a is dog
The a is dog domestic domesticated desccendant wolf the of
The animal cute a is dog and is friendly
The animal an is dog and is descended from wolf the
INSERT
DELETE
REPLACE
KEEP an encoder that is able to quickly specify which spans of text need to be changed and how , this allows for considerable changes to be made to the text (including insertion, deletion, re-ordering, and replacement) in a relatively simple and cheap manner.
Furthermore, this allows us to disentangle how likely the model is to operate (replace, delete, etc...) on a given span, and how likely the model thinks the generated text for a given span is.
As we are modeling editing processes , and hypothesize that context from edits applied to the sequence are helpful, we propose a method for edit-aware sequence compression which can compress sequences into their edit operations and use relative edit positional embeddings to specify the position of edits relative to each other.
To evaluate the efﬁcacy of proposed and baseline models, we propose an evaluation metric, edit perplexity (ePPL), which combines both of these ideas for a uniﬁed metric to measure edit likelihood.
However we can easily disentangle this into operation perplexity (oPPL) and generative perplexity (gPPL) which measures the likelihood of performing a given operation on a span, versus the likelihood of newly generated text for the next revision.
Given that the task of modeling natural editing processes in itself is novel, we collect new datasets to study this behavior; W IKI R EVISIONS and C ODE R EVISIONS .
These datasets, in the code and natural language domains respectively, cover over 2.5M and 2.3M natural sequential revisions.
We train and evaluate our proposed models on these datasets.
Our experimental results demonstrate that edit perplexity, generative perplexity, and operation perplexity are all reduced when one models edits with previous changes/edits as context.
We also qualitatively demonstrate the ability of the model to generate natural edit sequences, and demonstrate the utility of the learned representations on downstream tasks of commit message generation (Loyola et al., 2017) and edit intention classiﬁcation (Yang et al., 2017).
P ROBLEM D EFINITION
Let X = { x 0 , x 1 , . . .
, x N } be a series of N versions of a document, where the i th revised document is denoted by x i .
x 0 represents an initial state (generally the null string), and x N represents the current state of the edited document.
The probability of this series of document versions occurring can be decomposed as
[Equation]
where x i − 1:= x 0 , . .
, x i − 1 (similarly below).
The right hand side is the likelihood of the transformation of the previous document version x i − 1 to the current document version x i given the previous revision history x <i .
We refer to the likelihood of the whole revision process as the edit likelihood , and judge learned models based on their ability to achieve high edit likelihood on held-out data.
Note that standard generative models (speciﬁcally language models; LMs) calculate the probability of only the ﬁnal version p ( x N ) , whereas the proposed formulation calculates the probability of the entire sequence of document edits.
It nonetheless could be used to calculate the ﬁnal version’s likelihood by treating the editing process as latent and marginalizing over it
[Equation]
Thus our formulation, in contrast to previous single-step models of edits (Yin et al., 2019b; Malmi et al., 2019; Reid & Zhong, 2021), can also be used to deﬁne a generative model over single documents.
It is also worth noting that the ﬁnal document likelihood is lower-bounded by the edit likelihood; i.e. p ( x N ) ≥ p ( X ) .
M ODELING E DITING P ROCESSES
In this section, we now describe our approach to actually modeling these sequences of edits through (1) a decomposition of the modeling process into a sequential process of modeling edit operations then actual edits, and (2) neural model of modeling these operations and edits.
3. M ODELING O PERATIONS AND O PERATION - CONDITIONED E DITS
While the probability p ( x i | x i − 1 0 ) of the next document given all previous document versions could theoretically be modeled with a single neural sequence model, this is infeasible computationally (and likely infeasible from learning perspective as well).
To simplify this problem, we employ the n -th order Markov assumption, assuming that the probability of the next document is conditioned only on the previous n documents p ( x i | x i − 1 i − n ) .
This probability could be modeled directly, and in fact in the case of n = 1 this becomes analogous to the single-step editing problem tackled by previous work (Yin et al., 2019b;
Malmi et al., 2019; Reid & Zhong, 2021;
Faltings et al., 2021).
To our knowledge, no previous work has modeled natural editing processes with n > 1 .
However, in the interest of both efﬁciency and efﬁcacy, we take an alternative approach where we ﬁrst predict a set of edit operations e i , and then predict the next document version based on the previous documents and these edit operations:
[Equation]
The ﬁrst approximation becomes an equality when the edit operations can be deterministically derived from x i and x i − 1 , i.e. p ( e i | x i , x i − 1 )
= 1 , as is the case described below.
Edit Operations.
We base the edit operations in e on those calculated by the Levenshtein algorithm (Levenshtein, 1966), including token-level insertions, deletions, and substitutions.
These are expressed as four operations insert, delete, keep, and replace denoted by { INSERT , DELETE , KEEP ,
REPLACE } .
For multi-word insertions and replacements, e.g. a replacement of a contiguous span of words, we apply the the same REPLACE label to all tokens in this span.
An example of each operation is shown in Figure 1.
Decomposed Edit Likelihood.
We can then re-deﬁne our previous formulation of edit likelihood:
[Equation]
and analogously deﬁne edit log-likelihood
[Equation]
We can further decompose this into only the components corresponding to the edit operations L e := (cid:80)
[Equation]
) , or the operation-conditioned edits L x | e := (cid:80)
Ni =1 log p ( x i | e i , x i −
i − n ) , both of which we will utilize for devising evaluation metrics in Section 5.2 below.
The animal an is dog and is descended from wolf the
Encoder
The animal an is dog and is descended from wolf the
Decoder Decoder Decoder domestic a domesticated descendant of
Autoregressive Predictor edit compressed sequences
INSERT
DEL REPL KEEP KEEP
DEL
DEL
REPL
REPL
KEEP KEEP
3. E DIT P RO
In this section, we propose a model of multi-step editing processes, E DIT P RO , which is based on a semi-autoregressive edit-conditioned encoder-decoder model with a Transformer (Vaswani et al., 2017).
The model (depicted in Figure 2) contains three main components: (1) an edit encoder, (2) an operation classiﬁer and (3) an insertion-replacement decoder.
Edit Encoder.
The encoder f enc takes in a document version x i − 1 and feeds it through multiple self-attention and feedforward layers (Vaswani et al., 2017) to produce contextual representations for each token.
In the case that we perform variable-order edit modeling, we use cross-attention to feed in representations of previous edit steps.
For models where n > 1 , we feed in n − 1 additional edit sequences – we describe this process after describing our methods for edit sequence prediction.
Edit Operation Prediction.
We use an autoregressive tagger, using a single Transformer layer with a causal attention mask, that models the probability of each edit in edit operation sequence e =
e M 1 from left to right, p ( e j | e j − 1) .
Notably, we also performed preliminary experiments with a tagger that predicts operations independently, but found it was heavily biased towards the KEEP operation as most words are kept in any single document revision, and thus did not produce coherent multi-word edit sequences when sampling sequences of edits.
Generating Replacements and Insertions.
When editing, given our four Levenshtein operations ( INSERT , REPLACE , KEEP , DELETE ), two of them — INSERT and REPLACE — entail generation of new content conditioned on the current revision of the document.
Given our predicted edit operations e , we propose a semi-autoregressive model with a Transformer decoder that can decode multiple spans in parallel for efﬁciency purposes.
Each edit span contains the following properties: it has a start index (denoted by s start ), end index (denoted by s end ), and an operation type (denoted by s type ) .
Note that these can be simply be extracted by looking at contiguous spans of a certain type in an edit (e.g. REPLACE for descended from → domesticated descendant of in Figure 1).
We use a mean pooling operation to aggregate the contextual vectors produced by f enc ( x ) into span representation ˆ x s :
[Equation]
We then update the span representation ˆ x s by taking the sum of the appropriate operation embedding for the span type and the current span representation and feed it to a multi-layer perceptron with an intermediate non-linearity: ˆ x s ← MLP( W op ( e ) s +ˆ x s ) , where W op denotes an embedding matrix for each operation.
ˆ x s is then used to initialize the 
 token for the decoder span to further condition the generative process.
Encoding Edit History.
As we look to investigate variable order edit modeling over long sequences of text, we need a way to be able to represent edits in a way useful for predicting the next editing steps.
Previous work (Yin et al., 2019a; Marrese-Taylor et al., 2021; Yao et al., 2021) has focused largely on learning a single-vector representation for edits which is compressed but limited in expres-
[Table]
siveness.
One the other hand, a perhaps more intuitive way taken from common Transformer-based (Vaswani et al., 2017) models would be to use cross-attention between all n previous documents, which is more expressive but prohibitively expensive when n is scaled upwards.
Instead, we make a compromise between the above approaches, leveraging predicted edits e i − 1 i − n to compress the sequence and their derived spans (as discussed above).
Given each of these spans, we compute the edit-compressed sequence, composed of a sequence of vector representations with each vector representing a different span.
For each span in each of the previous revisions in x i − 1 i − n ) , we mean pool the encoder (pre-edit) and the decoder (post-edit) representations for that span.
We then sum this representation with the operation representing its edit operation and feed it into an
MLP.
Once we have done this for each span, we sum a learned relative edit positional embedding , where we learn an embedding matrix where each index in the matrix represents each order.
This is to specify the order of the previous edits .
Finally, we compose these into a sequence and treat that as the “edit-compressed” sequence representation for that edit.
Turning Pre-trained Encoder-Decoder Models into Editors.
Despite the fact that our model introduces both an edit prediction and a semi-autoregressive component, it is easy to ﬁnetune a pretrained language model into an editor with our method as it uses vanilla Transformer layers as a backbone.
We perform this by batching various spans and their conditioning variables together and training the model to adapt to decode these in parallel.
D ATA
While some datasets of edits exist (Faruqui et al., 2018; Marrese-Taylor et al., 2021), to our knowledge they only consider a single isolated edit, i.e. dealing with a document X
= { x 0 , x 1 } , N = 1 .
As we propose learning to model multi-step edits, we develop new datasets in both the code and natural language domains.
In addition, previous datasets have only concerned themselves with atomic edits (Faruqui et al., 2018) which only occur at a small scale (usually sentence-level), and we instead look to model larger-scale edits as document level changes, which are more representative of the natural editing process.
4. W IKI R EVISIONS
For the natural language domain of Wikipedia, we collect data for each revision using dumps from
English Wikipedia.
Given that the dumps are provided in the XML format, we extract the text with beautifulsoup and remove wikitext (custom Wikipedia markup) with wikiextractor .
With this sanitized data, we gather revision of each document in chronological order removing any metadata-based edits which were stripped as a result of the sanitization process.
Now, with our sets of revisions we tokenize all text with the sentencepiece model used by Radford et al.
Liu et al.
(2019) for congruence with pre-trained models (see Section 3.2).
We pre-compute Levenshtein operations using python-Levenshtein 2 for use during training.
In the case that an article exceeds 2000 tokens, we split the articles into its subsections and treat each subsection as an article (for the purpose of modeling editing processes).
Dataset statistics are shown in Table 1.
We note that there is a signiﬁcant imbalance for the INSERT operation, this is because we deﬁne insertions to be applied to the token preceding the insertion (as shown in Figure 1), rather than applied to an entire span (as we do for the deletion, replacement, and keep operations).
Edit Summaries.
When extracting each edit we keep the edit summary (akin to a commit message) supplied by the editor at time of editing.
We then curate these comments and develop a dataset
4. C ODE R EVISIONS
When buildling C ODE R EVISIONS , we scrape a total of 700 Python GitHub repositories using the
MIT License with at least 1000 commits and 500 stars.
We extract line-level patches from each repository’s commit history when forming our code-based corpus and progressively apply each patch and compute the token-level Levenshtien operations between each revision.
Note that we also keep commit messages for each commit.
For this dataset we operate on the ﬁle level.
For each series of revisions, we precompute Levenshtein operations based on tokens derived from a sentencepiece (Kudo & Richardson, 2018) model with a 10k vocabulary.
We also curate a dataset of revisions with commit messages as described in the previous subsection.
E XPERIMENTAL S ETUP 5. B ASELINES
We use the following baseline for our edit modeling task: (1) LEWIS (Reid & Zhong, 2021), a state-of-the-art single-step editing model, which uses a separate encoder-only tagger and sequenceto-sequence generator setup during training.
5. M ETRICS
Many previous works on editing (Malmi et al., 2019; Gu et al., 2019; Reid & Zhong, 2021) have used non-likelihood based metrics such as BLEU or F1, representing the downstream tasks which they were trained for, however as we look to model the intrinstic editing process (i.e. applying and generating a sequence of edits) and the likelihood of performing certain edits, we propose the following metrics.
Note that | x | refers to the token count for the newly generated/inserted spans, and | e | refers to the number of edit operations:
Edit Perplexity ePPL) is the exponent of the negative log likelihood for both the edit operations and generated outputs, divided by the length of both sequences, exp( −L xe | x | + | e | ) .
Generation Perplexity (gPPL) measures the likelihood of generating replaced or inserted spans when compared with the ground truth edit sequence as follows exp( −L x | e | x | ) .
Operation Perplexity (oPPL) exp( −L e | e | ) is the likelihood of predicting a set of edit operations.
5. T RAINING S ETUP
We train our models using the Transformer implementation in the HuggingFace (Wolf et al., 2020) library.
We tokenize data using SentencePiece (Kudo & Richardson, 2018), using the same vocabulary employed in Liu et al.
; Lewis et al.
for natural language, and using a custom 10k subword vocabulary for code.
We use the Transformer architecture of a hidden dimension of 768, feed- forward size of 3072, and 6 layers for both the encoder and decoder.
We initialize all natural language models with BART (Lewis et al., 2019), and code models with random initialization.
We set the maximum sequence length for all models to be 2048, using a batch size of 65K tokens distributed over 8 A100 GPUs.
5. D OWNSTREAM TASKS
In addition to assessing our proposed model’s generative capacity, we assess the quality of the learned representations on downstream tasks:
Conditional Editing We also continue training using the commit messages gathered during the cleaning process as a conditioning variable, essentially reformulating our p ( x i | x i − 1 i − n ) to p ( x i | x i − 1 i − n , c ) to add the additional conditional variable c , which we set to be the edit summary or commit message in this setting.
With our model, we append the comment to each document , delimiting with a separator token 
 as follows:
DOCUMENT 
 COMMENT .
Edit-conditioned Generation.
We deﬁne edit-conditioned generation to be tasks which rely on intermediate edit representations of documents to generate text describing the changes in text, similar to that proposed by Loyola et al.
(2017) for source-code commit message generation.
As we aim to determine whether the information contained about the edit itself is more informative as we add additional context, we condition the generation solely on the edit-compressed representations of the last edit step.
To accomplish this, we use a randomly initialized Transformer decoder with cross-attention on these edit-compressed representations.
Edit-conditioned Classiﬁcation.
In the natural language domain, we also test our representations on an edit-oriented classiﬁcation task, namely semantic intent classiﬁcation (Yang et al., 2017).
In Yang et al.
, they classify 5,777 Wikipedia revisions into 10 intention classes, such as “Clariﬁcation”, “Vandalism”, and others with each representing a different intention.
We form splits of 4,601 train examples, 588 valid examples, and 588 test examples.
3 Similarly to our setup for edit-conditioned generation, we also test our classiﬁer (consisting of a self-attentive span extractor (Lee et al., 2017) and a multi-layer perceptron) on this task.
R ESULTS 6. E DIT M ODELING
Results on edit modeling for both C ODE R EVISIONS and W IKI R EVISIONS can be seen in Table 2, where we measure edit perplexity, operation perplexity, and generative perplexity.
We ﬁrst note that our model signiﬁcantly outperforms LEWIS (Reid & Zhong, 2021) on W IKI R EVISIONS , by 8.6 ePPL, suggesting that our model formulation is superior at this task.
We believe that this stems from the fact that our model is trained to explicitly generate the newly added spans, and because it directly connects the operation prediction and generation processes.
We also take note that ePPL decreases when the order of context increases.
In particular, we take note of the signiﬁcant gain when introducing the notion of editing processes to the model in our 2-order setting (in contrast to a single edit step), with a reduction of 3.4 ePPL on natural language and 4.4 ePPL on source code.
We also note that while the gPPL consistently decreases as the number of orders increases, operation perplexity does not perform as consistently.
Interestingly, we ﬁnd that single-order models tend to be more conﬁdent with respect to keeping tokens (the overwhemingly dominant edit operation), while other operations (deletions, replacements and insertions) are not predicted as well.
In contrast, learning higher-order editing processes almost universally decreases the oPPL for non-KEEP operations, indicating the necessity to model longer context to capture these rarer edit operations.
Likely and Unlikely Edits.
We perform a qualitative analysis on a subsample 4,000 natural language edits, 4 examining which edits are judged to be likely (or unlikely) and with respect to which metrics.
We do this by identifying outlier values for each metric (signiﬁcantly above or below the average) and further analysing these for unique properties.
As a result, we found that many of the edits with higher oPPL were spam and vandalism-type edits, as many of the edit operations have more of a random nature.
However we notice that generative perplexity was much lower as these edits tend to be repetitive in nature with the same ngrams often being repeated for long spans.
However, we notice that, irrespective of the number of orders, when editing reverted spam-like content, the oPPL for the REPLACE and DELETE operations are extremely low (on average 1.07 and 4.4 respectively).
The importance of variable-order modeling was particularly evident these revisions where the gPPL in the single-order setting averages at 123.90 gPPL, however when using 2-orders we are able to attain 67.83 gPPL indicating that the edit-compressed sequences provide useful context about the previous revisions.
We also notice that models are able to predict insertions (2.25 INSERT oPPL) signiﬁcantly better when they come after the end of a sentence, representative of many insertions in Wikipedia.
We also notice that outside of the above settings, models with extra context generally predict more likely edits supporting the notion of modeling edit processes compared to modeling changes individually.
[Table] [Table]
with micro-F1), and
Conditional Edit Generation (measured Edit Perplexity = ePPL).
Note that the ∆ symbol refers to the change between the model’s non-message conditioned version in Table 2.
6. D OWNSTREAM P ERFORMANCE
Results on conditional edit generation, edit classiﬁcation and edit-conditioned generation can be seen in Table 3 .
We note that ﬁndings generally follow the edit modeling results, with additional context improving performance further giving supporting evidence to modeling editing processes.
Speciﬁcally, increasing context from single-order to 3-order improves commit message generation performance by 1.9 and 0.7 BLEU for both natural language and source code respectively.
We also note that ePPL decreases similarly when we add natural language conditioning to the editing process, which indicates that multi-order editing encodes ﬁne-grained information not found in the commit message.
Edit Modeling In particular, when performing editing using an editor pre-trained on edit modeling, we note that when sampling from the autoregressive tagger it almost always predicts KEEP with extremely high conﬁdence , given the overwhelming class majority.
We instead perform a variety of posterior regularisation (Ganchev et al.), reducing the probability of the KEEP class until the sampled edit labels to grow closer in proportion to the true distribution of edit operations (Table 1).
Combined with this technique, we are able to generate more diverse edits, which we show in Table 4.
Semantic Coherence
In looking at the example generations in Table 4 we note that the generated text is not perfectly semantically coherent, despite showing topical coherence and some degree of discourse coherence.
We believe this is largely due to the size of the language model we use, being trained solely on Wikipedia data (which contains a variety of minor edits including spam/vandalism).
Given this, we expect improved semantic coherence upon scaling up data, data diversity and model scale.
However, we note the improved context-awareness of the edit path shown by the 2-order model over the 1-order model, providing qualitative evidence for modeling editing processes and looking at different forms of document construction.
R ELATED W ORK
Learning Edit Representations.
Previous work on learning properties inherent to editing has largely focused on learning distributed representations of edits.
Yin et al.
(2019a) proposed this task, using an attentional sequence-to-sequence model to learn representations.
Marrese-Taylor et al.
(2021) expands upon this approach, introducing an auxiliary loss and a variational setup.
More recently, Yao et al.
(2021) worked on using this approach for editing tree-structured data.
However, in
[Table]
contrast with this work, these approaches only consider modeling single editing steps instead of the more general multi-step setting tackled here.
Semi-Autoregressive Sequence Generation.
Work in machine translation has explored nonautoregressive methods that use an iterative generation process.
This was ﬁrst proposed by Lee et al.
(2018) using latent variables and denoising autoencoding.
Iterative reﬁnement was then explored with conditional masked language models (CMLM; Ghazvininejad et al., 2019), simplifying previously proposed methods, by iteratively replacing predicted tokens with low conﬁdence.
Gu et al.
(2019), introduced the Levenshtein Transformer, making this approach more ﬂexible by introducing insertion and deletion operations.
However, these methods have not yet considered modeling natural editing processes, instead using either random sampling or heuristically determined orders.
Other Editing-based Work.
Other work on editing has included editing for sentence fusion (Malmi et al., 2019), in which one can perform minimal operations to join two sentences together grammatically.
Furthermore, with text style transfer in which the difference between sentences in different styles (e.g. positive vs negative) can be relatively minimal (Reid & Zhong, 2021; Mallinson et al., 2020).
Furthermore, Faltings et al.
(2021) explored natural language conditioned editing as a means for controllable text generation using a T5-based (Raffel et al., 2020) sequence to sequence model.
Also related to our work is text morphing (Huang et al., 2018), in which they look at an edit-based model to interpolate between two sentences.
We also note that diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) can be formulated as a ﬂavor of editing models, where the model learns to iteratively edit some representation of information in order to construct a ﬁnal version of said representation.
C ONCLUSIONS
In this work, we proposed the novel task of modeling editing processes, in which we model the likelihood of documents by way of their natural editing processes.
We develop new datasets and curate existing datasets for downstream tasks.
We ﬁnd that modeling editing processes is beneﬁcial to this end, in contrast to modeling single-order edits, as has been done in much of previous literature.
More broadly, we believe that tackling iterative reﬁnement tasks in broader NLP may be beneﬁcial given its resemblance to the natural generative and creative process.
In future work, we look to investigate methods for transferring pre-trained edit models to a wider range of NLP tasks.
E THICS S TATEMENT
This work has impact in the ﬁelds of natural language processing and machine learning, and proposes a new task of modeling editing processes, while proposing new datasets and models to this end.
There are several potential ethical concerns.
First, we initialize our models with a pre-trained model (trained on webtext), which may capture and amplify biases found in the pre-training data.
We further ﬁne tune on natural data which, in addition to benign edits, also contains a portion of human-generated spam and potentially offensive edits (especially in the natural language case), which may also amplify and capture biases in that regard.
On the other hand, we have also demonstrated the ability of the proposed model to identify and revert adverse edits performed by spammers or malicious editors, a potentially useful tool for content moderation, or even detecting automatically generated spam edits, as has been done in the detection of fake news (Zellers et al., 2019).
R EPRODUCIBILITY S TATEMENT
As noted on the ﬁrst page, all code and datasets necessary to reproduce the experiments in this paper will be released upon conclusion of paper review.
Furthermore, all data processing steps are discussed in Section 4 and Appendix A.1 for both code and natural language.
A A PPENDIX A. F URTHER TRAINING DETAILS
When training we employ data sharding to enable cheaper, on the ﬂy data processing.
We shard each documents’ into 10 shards and form splits based on these shards.
Our train-valid-test splits are split 90%,5%,5% for commit message generation, commit-conditioned edit modeling and edit modeling.
We use a dropout value of 0 . 3 and use the GELU activation for all MLPs.
We use a learning rate of 1e-4 warmed up for 1000 iterations.
We also note that we extended the positional embedding matrix for BART to handle longer sequences.