Original: G ENERATING W IKIPEDIA BY S UMMARIZING L ONG S EQUENCES
Revised: G ENERATING W IKIPEDIA BY S UMMARIZING L ONG S EQUENCES

Original: Anonymous authors Paper under double-blind review ABSTRACT
Revised: A

Original: We show that generating English Wikipedia articles can be approached as a multidocument summarization of source documents.
Revised: We show that generating English Wikipedia articles can be approached as a multidocument summarization of source documents.

Original: We use extractive summarization to coarsely identify salient information and a neural abstractive model to generate the article.
Revised: We use extractive summarization to coarsely identify salient information and a neural abstractive model to generate the article.

Original: For the abstractive model, we introduce a decoder-only architecture that can scalably attend to very long sequences, much longer than typical encoderdecoder architectures used in sequence transduction.
Revised: For the abstractive model, we introduce a decoder-only architecture that can scalably attend to very long sequences, much longer than typical encoderdecoder architectures used in sequence transduction.

Original: We show that this model can generate ﬂuent, coherent multi-sentence paragraphs and even whole Wikipedia articles.
Revised: We show that this model can generate ﬂuent, coherent multi-sentence paragraphs and even whole Wikipedia articles.

Original: When given reference documents, we show it can extract relevant factual information as reﬂected in perplexity, ROUGE scores and human evaluations
Revised: When given reference documents, we show it can extract relevant factual information as reﬂected in perplexity, ROUGE scores and human evaluations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The sequence-to-sequence framework has demonstrated success in natural-language sequence transduction tasks such as machine translation.
Revised: The sequence-to-sequence framework has demonstrated success in natural-language sequence transduction tasks such as machine translation.

Original: More recently, neural techniques have been applied to do single-document, abstractive (paraphrasing) text summarization of news articles (Rush et al.
Revised: More recently, neural techniques have been applied to do single-document, abstractive (paraphrasing) text summarization of news articles (Rush et al.

Original: Nallapati et al.
Revised: Nallapati et al.

Original: In this prior work, the input to supervised models ranged from the ﬁrst sentence to the entire text of an article, and they are trained end-to-end to predict reference summaries.
Revised: In this prior work, the input to supervised models ranged from the ﬁrst sentence to the entire text of an article, and they are trained end-to-end to predict reference summaries.

Original: Doing this end-to-end requires a signiﬁcant number of parallel article-summary pairs since language understanding is a pre-requisite to generate ﬂuent summaries.
Revised: Doing this end-to-end requires a signiﬁcant number of parallel article-summary pairs since language understanding is a pre-requisite to generate ﬂuent summaries.

Original: In contrast, we consider the task of multi-document summarization, where the input is a collection of related documents from which a summary is distilled.
Revised: In contrast, we consider the task of multi-document summarization, where the input is a collection of related documents from which a summary is distilled.

Original: Prior work has focused on extractive summarization, which select sentences or phrases from the input to form the summaries, rather than generating new text.
Revised: Prior work has focused on extractive summarization, which select sentences or phrases from the input to form the summaries, rather than generating new text.

Original: There has been limited application of abstractive neural methods and one possible reason is the paucity of large, labeled datasets.
Revised: There has been limited application of abstractive neural methods and one possible reason is the paucity of large, labeled datasets.

Original: In this work, we consider English Wikipedia as a supervised machine learning task for multidocument summarization where the input is comprised of a Wikipedia topic (title of article) and a collection of non-Wikipedia reference documents, and the target is the Wikipedia article text.
Revised: In this work, we consider English Wikipedia as a supervised machine learning task for multidocument summarization where the input is comprised of a Wikipedia topic (title of article) and a collection of non-Wikipedia reference documents, and the target is the Wikipedia article text.

Original: We describe the ﬁrst attempt to abstractively generate the ﬁrst section, or lead , of Wikipedia articles conditioned on reference text.
Revised: We describe the ﬁrst attempt to abstractively generate the ﬁrst section, or lead , of Wikipedia articles conditioned on reference text.

Original: In addition to running strong baseline models on the task, we modify the
Revised: In addition to running strong baseline models on the task, we modify the

Original: Transformer architecture (Vaswani et al., 2017) to only consist of a decoder, which performs better in the case of longer input sequences compared to recurrent neural network (RNN) and Transformer encoder-decoder models.
Revised: Transformer architecture (Vaswani et al., 2017) to only consist of a decoder, which performs better in the case of longer input sequences compared to recurrent neural network (RNN) and Transformer encoder-decoder models.

Original: Finally we show our modeling improvements allow us to generate
Revised: Finally we show our modeling improvements allow us to generate

Original: entire
Revised: entire

Original: Wikipedia articles.
Revised: Wikipedia articles.

Original:  R ELATED W ORK 2. O THER DATASETS USED IN NEURAL ABSTRACTIVE SUMMARIZATION
Revised:  R ELATED W ORK 2. O THER DATASETS USED IN NEURAL ABSTRACTIVE SUMMARIZATION

Original: Neural abstractive summarization was pioneered in Rush et al.
Revised: Neural abstractive summarization was pioneered in Rush et al.

Original: (2015), where they train headline generation models using the English Gigaword corpus (Graff & Cieri, 2003), consisting of news articles from number of publishers.
Revised: (2015), where they train headline generation models using the English Gigaword corpus (Graff & Cieri, 2003), consisting of news articles from number of publishers.

Original: However, the task is more akin to sentence paraphrasing than summarization as only the ﬁrst sentence of an article is used to predict the headline, another sentence.
Revised: However, the task is more akin to sentence paraphrasing than summarization as only the ﬁrst sentence of an article is used to predict the headline, another sentence.

Original: RNN-based encoder-decoder models with attention (seq2seq) perform very well on this task in both ROUGE (Lin, 2004), an automatic metric often used in summarization, and human evaluation (Chopra et al., 2016).[Table]
Revised: RNN-based encoder-decoder models with attention (seq2seq) perform very well on this task in both ROUGE (Lin, 2004), an automatic metric often used in summarization, and human evaluation (Chopra et al., 2016).

Original: In Nallapati et al.
Revised: In Nallapati et al.

Original: (2016), an abstractive summarization dataset is proposed by modifying a questionanswering dataset of news articles paired with story highlights from Daily Mail and CNN.
Revised: (2016), an abstractive summarization dataset is proposed by modifying a questionanswering dataset of news articles paired with story highlights from Daily Mail and CNN.

Original: This task is more difﬁcult than headline-generation because the information used in the highlights may come from many parts of the article and not only the ﬁrst sentence.
Revised: This task is more difﬁcult than headline-generation because the information used in the highlights may come from many parts of the article and not only the ﬁrst sentence.

Original: One downside of the dataset is that it has an order-of-magnitude fewer parallel examples (310k vs. 3.8M) to learn from.
Revised: One downside of the dataset is that it has an order-of-magnitude fewer parallel examples (310k vs. 3.8M) to learn from.

Original: Standard seq2seq models with attention do less well, and a number of techniques are used to augment performance.
Revised: Standard seq2seq models with attention do less well, and a number of techniques are used to augment performance.

Original: Another downside is that it is unclear what the guidelines are for creating story highlights and it is obvious that there are signiﬁcant stylistic differences between the two news publishers.
Revised: Another downside is that it is unclear what the guidelines are for creating story highlights and it is obvious that there are signiﬁcant stylistic differences between the two news publishers.

Original: In our work we also train neural abstractive models, but in the multi-document regime with
Revised: In our work we also train neural abstractive models, but in the multi-document regime with

Original: Wikipedia.
Revised: Wikipedia.

Original: As can be seen in Table 1, the input and output text are generally much larger, with signiﬁcant variance depending on the article.
Revised: As can be seen in Table 1, the input and output text are generally much larger, with signiﬁcant variance depending on the article.

Original: The summaries (Wikipedia lead) are multiple sentences and sometimes multiple paragraphs, written in a fairly uniform style as encouraged by the
Revised: The summaries (Wikipedia lead) are multiple sentences and sometimes multiple paragraphs, written in a fairly uniform style as encouraged by the

Original: Wikipedia Manual of Style 1 .
Revised: Wikipedia Manual of Style 1 .

Original: However, the input documents may consist of documents of arbitrary style originating from arbitrary sources.
Revised: However, the input documents may consist of documents of arbitrary style originating from arbitrary sources.

Original: We also show in Table 1 the ROUGE-1 recall scores of the output given the input, which is the proportion of unigrams/words in the output co-occuring in the input.
Revised: We also show in Table 1 the ROUGE-1 recall scores of the output given the input, which is the proportion of unigrams/words in the output co-occuring in the input.

Original: A higher score corresponds to a dataset more amenable to extractive summarization.
Revised: A higher score corresponds to a dataset more amenable to extractive summarization.

Original: In particular, if the output is completely embedded somewhere in the input (e.g. a wiki-clone), the score would be 100.
Revised: In particular, if the output is completely embedded somewhere in the input (e.g. a wiki-clone), the score would be 100.

Original: Given a score of only 59.2 compared to 76.1 and 78.7 for other summarization datasets shows that ours is the least amenable to purely extractive methods.
Revised: Given a score of only 59.2 compared to 76.1 and 78.7 for other summarization datasets shows that ours is the least amenable to purely extractive methods.

Original: 2. T ASKS INVOLVING W IKIPEDIA
Revised: 2. T ASKS INVOLVING W IKIPEDIA

Original: There is a rich body of work incorporating Wikipedia for machine learning tasks, including questionanswering (Hewlett et al.
Revised: There is a rich body of work incorporating Wikipedia for machine learning tasks, including questionanswering (Hewlett et al.

Original: (2016), Rajpurkar et al.
Revised: (2016), Rajpurkar et al.

Original: and information extraction (Lehmann et al., 2015), and text generation from structured data (Lebret et al., 2016).
Revised: and information extraction (Lehmann et al., 2015), and text generation from structured data (Lebret et al., 2016).

Original: The closest work to ours involving generating Wikipedia is Sauper & Barzilay (2009), where articles are generated extractively (instead of abstractively in our case) from reference documents using learned templates.
Revised: The closest work to ours involving generating Wikipedia is Sauper & Barzilay (2009), where articles are generated extractively (instead of abstractively in our case) from reference documents using learned templates.

Original: The Wikipedia articles are restricted to two categories, whereas we use all article types.
Revised: The Wikipedia articles are restricted to two categories, whereas we use all article types.

Original: The reference documents are obtained from a search engine, with the Wikipedia topic used as query similar to our search engine references.
Revised: The reference documents are obtained from a search engine, with the Wikipedia topic used as query similar to our search engine references.

Original: However we also show results with documents only found in the References section of the Wikipedia articles.
Revised: However we also show results with documents only found in the References section of the Wikipedia articles.

Original: 2. T RANSFORMER MODELS
Revised: 2. T RANSFORMER MODELS

Original: Previous work on neural abstractive summarization relies on RNNs as fundamental modules, mirroring techniques successful in machine translation (MT).
Revised: Previous work on neural abstractive summarization relies on RNNs as fundamental modules, mirroring techniques successful in machine translation (MT).

Original: Recently, state-of-the-art MT results were obtained using a non-recurrent architecture, called the Transformer (Vaswani et al., 2017).
Revised: Recently, state-of-the-art MT results were obtained using a non-recurrent architecture, called the Transformer (Vaswani et al., 2017).

Original: The lack of recurrence enables greater within-training-example parallelization, at the cost of quadratic complexity in the input sequence length.
Revised: The lack of recurrence enables greater within-training-example parallelization, at the cost of quadratic complexity in the input sequence length.

Original: We ﬁnd the Transformer transfers well to medium length, input sequence summarization and describe modiﬁcations to better handle longer sequences.
Revised: We ﬁnd the Transformer transfers well to medium length, input sequence summarization and describe modiﬁcations to better handle longer sequences.

Revised: [Table]

Original: [Table]
Revised: [Table]

Original:  E NGLISH W IKIPEDIA AS A MULTI - DOCUMENT SUMMARIZATION DATASET
Revised:  E NGLISH W IKIPEDIA AS A MULTI - DOCUMENT SUMMARIZATION DATASET

Original: Wikipedia, being an encyclopedia, can be viewed as a collection of summaries on various topics given by their title, e.g. ”Canada” or ”Machine Learning”.
Revised: Wikipedia, being an encyclopedia, can be viewed as a collection of summaries on various topics given by their title, e [Table] g. ”Canada” or ”Machine Learning”.

Original: The source material to be summarized can be viewed as all reputable documents on the Web or books; however, to make the problem more tractable we consider the following subsets of all documents, D :
Revised: The source material to be summarized can be viewed as all reputable documents on the Web or books; however, to make the problem more tractable we consider the following subsets of all documents, D :

Original: Cited sources: A Wikipedia article that conforms to the style guidelines should be wellsupported by citations found in the References section of Wikipedia articles.
Revised: Cited sources: A Wikipedia article that conforms to the style guidelines should be wellsupported by citations found in the References section of Wikipedia articles.

Original: For each article, a i , we extract all text without markup from crawlable citation documents, C i ⊂ D , to use as input to our method.
Revised: For each article, a i , we extract all text without markup from crawlable citation documents, C i ⊂ D , to use as input to our method.

Original: Web Search results: To expand the collection of reference documents, we crawl the search results from the Google search engine, using the article section titles as queries.
Revised: Web Search results: To expand the collection of reference documents, we crawl the search results from the Google search engine, using the article section titles as queries.

Original: For each query, we collect 10 result pages.
Revised: For each query, we collect 10 result pages.

Original: From this collection we remove the Wikipedia article itself, which is often among the top results.
Revised: From this collection we remove the Wikipedia article itself, which is often among the top results.

Original: We also remove ”clones”, which are detected when there is a high-level of unigram overlap with the article (details provided in A.2.1).
Revised: We also remove ”clones”, which are detected when there is a high-level of unigram overlap with the article (details provided in A.2.1).

Original: We denote these reﬁned search results for an article, a i , as S i ⊂ D .
Revised: We denote these reﬁned search results for an article, a i , as S i ⊂ D .

Original: Similar to C i , we extract only the text to use as input.
Revised: Similar to C i , we extract only the text to use as input.

Original: Table 2 describes overall properties of our WikiSum dataset.
Revised: Table 2 describes overall properties of our WikiSum dataset.

Original: Many articles have few citations, motivating our supplementation of the source documents with web search results.
Revised: Many articles have few citations, motivating our supplementation of the source documents with web search results.

Original: On the other hand, citations when available, tend to be of higher-quality.
Revised: On the other hand, citations when available, tend to be of higher-quality.

Original: When counting the total words in the entire dataset, it is orders-of-magnitude larger than previous summarization datasets.
Revised: When counting the total words in the entire dataset, it is orders-of-magnitude larger than previous summarization datasets.

Original: To have consistent train/development/test data across corpus-comparison experiments, we restrict the articles to those with at least one crawlable citation.
Revised: To have consistent train/development/test data across corpus-comparison experiments, we restrict the articles to those with at least one crawlable citation.

Original: We divide the articles roughly into 80/10/ for train/development/test subsets, resulting in 1865750, 233252, and 232998 examples respectively.
Revised: We divide the articles roughly into 80/10/ for train/development/test subsets, resulting in 1865750, 233252, and 232998 examples respectively.

Original:  M ETHODS AND M ODELS
Revised:  M ETHODS AND M ODELS

Original: Because the amount of text in input reference documents ( C i , S i ) can be very large (see Table 2) it is infeasible to train an end-to-end abstractive model given the memory constraints of current hardware.
Revised: Because the amount of text in input reference documents ( C i , S i ) can be very large (see Table 2) it is infeasible to train an end-to-end abstractive model given the memory constraints of current hardware.

Original: Hence, we ﬁrst coarsely select a subset of the input using extractive summarization.
Revised: Hence, we ﬁrst coarsely select a subset of the input using extractive summarization.

Original: The second stage involves training an abstractive model that generates the Wikipedia text while conditioning on this extraction.
Revised: The second stage involves training an abstractive model that generates the Wikipedia text while conditioning on this extraction.

Original: This two-stage process is inspired by by how humans might summarize multiple long documents: First highlight pertinent information, then conditionally generate the summary based on the highlights.
Revised: This two-stage process is inspired by by how humans might summarize multiple long documents: First highlight pertinent information, then conditionally generate the summary based on the highlights.

Original: 4. E XTRACTIVE STAGE
Revised: 4. E XTRACTIVE STAGE

Original: We investigate three extractive methods from the summarization literature, along with a trivial and cheating method, to assess the importance of this stage.
Revised: We investigate three extractive methods from the summarization literature, along with a trivial and cheating method, to assess the importance of this stage.

Original: For each article, a i we create a ranked list of paragraphs, { p iR i ( j ) } , occurring in ( C i , S i ) where R i ( j ) is the rank of the j th paragraph p ij of ( C i , S i ) .
Revised: For each article, a i we create a ranked list of paragraphs, { p iR i ( j ) } , occurring in ( C i , S i ) where R i ( j ) is the rank of the j th paragraph p ij of ( C i , S i ) .

Original: From this we select the ﬁrst L tokens as input to the second abstractive stage.
Revised: From this we select the ﬁrst L tokens as input to the second abstractive stage.

Original: Identity :
Revised: Identity :

Original: As a trivial baseline extractor, we simply use the ﬁrst L tokens of the input.
Revised: As a trivial baseline extractor, we simply use the ﬁrst L tokens of the input.

Original: tf-idf : A non-trivial ranking is to consider ranking paragraphs as documents in a queryretrieval problem, where the query is the title of the article, T ( a i ) .
Revised: tf-idf : A non-trivial ranking is to consider ranking paragraphs as documents in a queryretrieval problem, where the query is the title of the article, T ( a i ) .

Original: We compute tf-idf (Ramos et al., 2003) for the query, with respect to the documents, { p ij } .
Revised: We compute tf-idf (Ramos et al., 2003) for the query, with respect to the documents, { p ij } .

Original: That is, we summate for each word in the query
Revised: That is, we summate for each word in the query

Original: [Equation]
Revised: [Equation]

Original: where N w , N d , and N dw are the count of the word in the document, total number of documents, and total number of documents containing the word, respectively.
Revised: where N w , N d , and N dw are the count of the word in the document, total number of documents, and total number of documents containing the word, respectively.

Original: TextRank (Mihalcea & Tarau, 2004):
Revised: TextRank ( ? ):

Original: A weighted graph is deﬁned where text units are nodes and edges are deﬁned by a similarity measure based on word overlap.
Revised: A weighted graph is deﬁned where text units are nodes and edges are deﬁned by a similarity measure based on word overlap.

Original: An algorithm similar to PageRank (Page et al., 1999) is then used to compute the ranking of text units.
Revised: An algorithm similar to PageRank ( ? ) is then used to compute the ranking of text units.

Original: We used paragraphs for the text units.
Revised: We used paragraphs for the text units.

Original: SumBasic (Nenkova & Vanderwende, 2005): Word frequencies in the input text are used to assign scores to words, which are in turn used to score sentences.
Revised: SumBasic ( ? ): Word frequencies in the input text are used to assign scores to words, which are in turn used to score sentences.

Original: After selecting the best scoring sentence, words in it have their scores reduced, and the process is repeated until the desired summary length is reached.
Revised: After selecting the best scoring sentence, words in it have their scores reduced, and the process is repeated until the desired summary length is reached.

Original: Cheating To further demonstrate the quality of extraction on the ﬁnal performance, we implement a cheating extractor that ranks { p ij } using recall of bigrams in the ground truth text:
Revised: Cheating To further demonstrate the quality of extraction on the ﬁnal performance, we implement a cheating extractor that ranks { p ij } using recall of bigrams in the ground truth text:

Original: [Equation]
Revised: [Equation]

Original: 4. A BSTRACTIVE STAGE 4.2. D ATA REPRESENTATION
Revised: 4. A BSTRACTIVE STAGE 4.2. D ATA REPRESENTATION

Original: Given the ordered paragraphs { p iR i ( j ) } , we derive the raw text input simply as the concatenation of the paragraphs in order, the most relevant at the beginning, and preﬁxed with the title.
Revised: Given the ordered paragraphs { p iR i ( j ) } , we derive the raw text input simply as the concatenation of the paragraphs in order, the most relevant at the beginning, and preﬁxed with the title.

Original: We then encode the text using sub-word tokenization similar to Wu et al.
Revised: We then encode the text using sub-word tokenization similar to Wu et al.

Original: with a vocabulary size of 32,000 yielding tokenized input, x i :
Revised: with a vocabulary size of 32,000 yielding tokenized input, x i :

Original: [Equation]
Revised: [Equation]

Revised: tokenize

Revised: [Equation]

Original: For various values of L in experiments, we truncate the tokens to form the input sequence:
Revised: For various values of L in experiments, we truncate the tokens to form the input sequence:

Original: [Equation]
Revised: [Equation]

Original: For the output, we use the same vocabulary and tokenization for the Wikipedia lead text but do not do any truncation across experiments.
Revised: For the output, we use the same vocabulary and tokenization for the Wikipedia lead text but do not do any truncation across experiments.

Original: Next we describe the abstractive models, W , that learn to write articles, a i = W ( m Li ) , which we treat as a sequence transduction problem from very long input sequences (up to L = 11000 ) to medium output sequences (typically less than 500).
Revised: Next we describe the abstractive models, W , that learn to write articles, a i = W ( m Li ) , which we treat as a sequence transduction problem from very long input sequences (up to L = 11000 ) to medium output sequences (typically less than 500).

Original: 4.2. B ASELINE M ODELS
Revised: 4.2. B ASELINE M ODELS

Original: As a baseline we apply the standard LSTM encoder-decoder with attention (seq2seq-att) as in Bahdanau et al.
Revised: As a baseline we apply the standard LSTM encoder-decoder with attention (seq2seq-att) as in Bahdanau et al.

Original: (2014) to this task.
Revised: (2014) to this task.

Original: As is typical we train to optimize the maximum-likelihood objective:
Revised: As is typical we train to optimize the maximum-likelihood objective:

Original: [Equation]
Revised: [Equation]

Original: A stronger, more recent baseline that we use is the non-recurrent Transformer model described in 2.3, which also has symmetric encoder and decoder modules (T-ED).
Revised: A stronger, more recent baseline that we use is the non-recurrent Transformer model described in 2.3, which also has symmetric encoder and decoder modules (T-ED).

Original: 4.2. T RANSFORMER D ECODER (T-D)
Revised: 4.2. T RANSFORMER D ECODER (T-D)

Original: We introduce a simple but effective modiﬁcation to T-ED for long sequences that drops the encoder module (almost reducing model parameters by half for a given hyper-parameter set), combines the input and output sequences into a single ”sentence” and is trained as a standard language model.
Revised: We introduce a simple but effective modiﬁcation to T-ED for long sequences that drops the encoder module (almost reducing model parameters by half for a given hyper-parameter set), combines the input and output sequences into a single ”sentence” and is trained as a standard language model.

Original: That is, we convert a sequence-transduction example ( m 1 , ..., m n ) (cid:55)→ ( y 1 , ..., y η ) into the sentence ( w 1 , ..., w n + η +1 ) 
Revised: That is, we convert a sequence-transduction example ( m 1 , ..., m n ) (cid:55)→ ( y 1 , ..., y η ) into the sentence ( w 1 , ..., w n + η +1 ) [Equation]

Original: = ( m 1 , ..., m n , δ, y 1 , ..., y η ) , where δ is a special separator token and train a model to predict the next word given the previous ones:
Revised: [Equation] [Equation] , δ, y 1 , ..., y η ) , where δ is a special separator token and train a model to predict the next word given the previous ones:

Original: [Equation]
Revised: [Equation]

Original: Since the model is forced to predict the next token in the input, m , as well as y , error signals are propagated from both input and output time-steps during training.
Revised: Since the model is forced to predict the next token in the input, m , as well as y , error signals are propagated from both input and output time-steps during training.

Original: We also suspect that for monolingual text-to-text tasks redundant information is re-learned about language in the encoder and decoder.
Revised: We also suspect that for monolingual text-to-text tasks redundant information is re-learned about language in the encoder and decoder.

Original: We believe this allows for easier optimization and empirically observe this with longer sequences (see Section 5.3).
Revised: We believe this allows for easier optimization and empirically observe this with longer sequences (see Section 5.3).

Original: Note that because of the self-attention of the Transformer, when generating the next token, attention from both m and y are considered.
Revised: Note that because of the self-attention of the Transformer, when generating the next token, attention from both m and y are considered.

Original: At inference we provide the input sequence, m i , initially, and auto-regressively generate the output, y i , as normal.
Revised: At inference we provide the input sequence, m i , initially, and auto-regressively generate the output, y i , as normal.

Original: 4.2. T RANSFORMER D ECODER WITH MEMORY - COMPRESSED ATTENTION (T-DMCA)
Revised: 4.2. T RANSFORMER D ECODER WITH MEMORY - COMPRESSED ATTENTION (T-DMCA)

Original: To re-use the terminology used to describe the Transformer, the attention is a function of a query ( Q ) and set of key ( K ) and value ( V ) pairs.
Revised: To re-use the terminology used to describe the Transformer, the attention is a function of a query ( Q ) and set of key ( K ) and value ( V ) pairs.

Original: To handle longer sequences, we modify the multi-head self-attention of the Transformer to reduce memory usage by limiting the dot products between Q and K in:Attention
Revised: To handle longer sequences, we modify the multi-head self-attention of the Transformer to reduce memory usage by limiting the dot products between Q and K in:

Original: [Equation]
Revised: [Equation]

Original: Local attention : Sequence tokens are divided into blocks of similar length and attention is performed in each block independently.
Revised: Local attention : Sequence tokens are divided into blocks of similar length and attention is performed in each block independently.

Original: As the attention memory cost per block becomes constant, this modiﬁcation allow us to keep the number of activations linear with respect to the sequence length.
Revised: As the attention memory cost per block becomes constant, this modiﬁcation allow us to keep the number of activations linear with respect to the sequence length.

Original: In our experiments, we choose to have blocks of 256 tokens.
Revised: In our experiments, we choose to have blocks of 256 tokens.

Original: Memory-compressed attention : After projecting the tokens into the query, key, and value embeddings, we reduce the number of keys and values by using a strided convolution.
Revised: Memory-compressed attention : After projecting the tokens into the query, key, and value embeddings, we reduce the number of keys and values by using a strided convolution.

Original: The number of queries remains unchanged.
Revised: The number of queries remains unchanged.

Original: This modiﬁcation allows us to divide the number of activations by a compression factor.
Revised: This modiﬁcation allows us to divide the number of activations by a compression factor.

Original: In our experiments we use convolution kernels of size 3 with stride 3.
Revised: In our experiments we use convolution kernels of size 3 with stride 3.

Original: In contrast to local attention layers, which only capture the local information within a block, the memorycompressed attention layers are able to exchange information globally on the entire sequence.
Revised: In contrast to local attention layers, which only capture the local information within a block, the memorycompressed attention layers are able to exchange information globally on the entire sequence.

Original: These modiﬁcations (see Figure 1) allow us in practice to process sequences 3x in length over the T-D model.
Revised: These modiﬁcations (see Figure 1) allow us in practice to process sequences 3x in length over the T-D model.

Original: For both local and memory-compressed attention, masking is added to prevent the queries from attending to future keys and values.
Revised: For both local and memory-compressed attention, masking is added to prevent the queries from attending to future keys and values.

Original: Our ﬁnal architecture is a 5-layer network (LMLML) alternating between local-attention (L) layers and memory-compressed attention (M) layers (in Vaswani et al.
Revised: Our ﬁnal architecture is a 5-layer network (LMLML) alternating between local-attention (L) layers and memory-compressed attention (M) layers (in Vaswani et al.

Original: (2017) it is 6 identical layers).
Revised: (2017) it is 6 identical layers).

Original: We also added in some experiments one mixture of experts (MoE) layer (Shazeer et al., 2017) to increase thenetwork’s capacity.
Revised: We also added in some experiments one mixture of experts (MoE) layer (Shazeer et al., 2017) to increase the capacity.

Original:  E XPERIMENTS 5. E VALUATION
Revised:  E XPERIMENTS 5. E VALUATION

Original: In experiments we evaluate based on perplexity (per-wordpiece), a common language modeling metric, and ROUGE-L F1 (version ROUGE-1.5.5 ), a common metric used in comparing candidate and reference summaries.
Revised: In experiments we evaluate based on perplexity (per-wordpiece), a common language modeling metric, and ROUGE-L F1 (version ROUGE-1.5.5 ), a common metric used in comparing candidate and reference summaries.

Original: Note the F1 ﬂavor of ROUGE is more appropriate in this setting as we do not explicitly constrain the output length in abstractive models; it is the harmonic mean of ROUGERecall (which favors long summaries) and ROUGE-Precision (which favors short summaries).
Revised: Note the F1 ﬂavor of ROUGE is more appropriate in this setting as we do not explicitly constrain the output length in abstractive models; it is the harmonic mean of ROUGERecall (which favors long summaries) and ROUGE-Precision (which favors short summaries).

Original: Although optimizing ROUGE directly has been shown to not always yield the best summaries as evaluated by human judgment (Paulus et al., 2017), we found that for our task optimizing for perplexity correlates with increased ROUGE and human judgment.
Revised: Although optimizing ROUGE directly has been shown to not always yield the best summaries as evaluated by human judgment (Paulus et al., 2017), we found that for our task optimizing for perplexity correlates with increased ROUGE and human judgment.

Original: We suspect that the relatively uniform style of Wikipedia articles makes ROUGE more appropriate here than in general abstractive summarization tasks.
Revised: We suspect that the relatively uniform style of Wikipedia articles makes ROUGE more appropriate here than in general abstractive summarization tasks.

Original: 5. M ODEL T RAINING D ETAILS AND D ECODING
Revised: 5. M ODEL T RAINING D ETAILS AND D ECODING

Original: For all abstractive model training, we use the open-source tensor2tensor 2 library.
Revised: For all abstractive model training, we use the open-source tensor2tensor 2 library.

Original: The seq2seq baseline had a hidden size of 128 with 2 layers (we use the hyper-parameter set deﬁned in the library as lstm attention ).
Revised: The seq2seq baseline had a hidden size of 128 with 2 layers (we use the hyper-parameter set deﬁned in the library as lstm attention ).

Original: For the
Revised: For the

Original: Transformer encoder-decoder (T-ED), we use the hyper-parameter set transfomer base v1 and train for 1 million steps.
Revised: Transformer encoder-decoder (T-ED), we use the hyper-parameter set transfomer base v1 and train for 1 million steps.

Original: Models exhibited very little overﬁtting and did not require early-stopping.
Revised: Models exhibited very little overﬁtting and did not require early-stopping.

Original: The Transformer Decoder (T-D) was identical to the decoder part of T-ED.
Revised: The Transformer Decoder (T-D) was identical to the decoder part of T-ED.

Original: The T-DMCA model is similar to T-D, but with the enhancements described in section 4.2.4.
Revised: The T-DMCA model is similar to T-D, but with the enhancements described in section 4.2.4.

Original: Unless otherwise stated, during decoding we use a beam search of size 4 and length penalty α = 0 .(Wu et al., 2016) and decode until an end-of-sequence token is reached. 
Revised: Unless otherwise stated, during decoding we use a beam search of size 4 and length penalty α = 0 .(Wu et al., 2016) and decode until an end-of-sequence token is reached. [Table] [Table]

Original: 5. R ESULTS AND D ISCUSSION
Revised: R ESULTS AND D ISCUSSION

Original: There are four main dimensions we vary in experiments in generating Wikipedia lead sections:
Revised: There are four main dimensions we vary in experiments in generating Wikipedia lead sections:

Original: Extractive method: SumBasic , TextRank , tf-idf , identity , cheating extractor 2.
Revised: Extractive method: SumBasic , TextRank , tf-idf , identity , cheating extractor 2.

Original: Input corpus: citations , search results , combined 3.
Revised: Input corpus: citations , search results , combined 3.

Original: Abstractive model input length,
Revised: Abstractive model input length,

Original: L : We try values between 100 and 11000. 4.
Revised: L : We try values between 100 and 11000. 4.

Original: Abstractive model architecture: seq2seq-att , T-ED , T-D , T-DMCA
Revised: Abstractive model architecture: seq2seq-att , T-ED , T-D , T-DMCA

Original: Extractive-only is not enough: We investigate performance of extractive methods without the abstractive model by looking at the ROUGE-L F1 scores after running tf-idf , SumBasic , and TextRank[Table] in Figure 2, without any abstractive model.
Revised: Extractive-only is not enough: We investigate performance of extractive methods without the abstractive model by looking at the ROUGE-L F1 scores after running tf-idf , SumBasic , and TextRank in Figure 2, without any abstractive model.

Original: In the case of TextRank and SumBasic we matched the output length to the target length and observe the extractive methods perform roughly in-line with each other in terms of ROUGE-L F1.
Revised: In the case of TextRank and SumBasic we matched the output length to the target length and observe the extractive methods perform roughly in-line with each other in terms of ROUGE-L F1.

Original: Our best abstractive model more than doubled this metric.
Revised: Our best abstractive model more than doubled this metric.

Original: Further, this model yields large improvements in perceived linguistic quality (elaborated below).
Revised: Further, this model yields large improvements in perceived linguistic quality (elaborated below).

Original: Extractive method: From Table 3 we observe that smart extraction is critical for ﬁnal abstractive performance.
Revised: Extractive method: From Table 3 we observe that smart extraction is critical for ﬁnal abstractive performance.

Original: There is a signiﬁcant gap between doing nothing, identity , and extractive summarization, tf-idf .
Revised: There is a signiﬁcant gap between doing nothing, identity , and extractive summarization, tf-idf .

Original: Further, there is a signiﬁcant gap between tf-idf and the cheating extractor, suggesting future work in improving the extraction step could result in signiﬁcant improvements.
Revised: Further, there is a signiﬁcant gap between tf-idf and the cheating extractor, suggesting future work in improving the extraction step could result in signiﬁcant improvements.

Original: One possibility is to train a supervised model to predict relevance (Eq. 1), which we leave as future work.
Revised: One possibility is to train a supervised model to predict relevance (Eq. 1), which we leave as future work.

Original: For subsequent experiments we ﬁx the extractive method to tf-idf .
Revised: For subsequent experiments we ﬁx the extractive method to tf-idf .

Original: Input Corpus: From table 3 we also observe that, unsurprisingly, the combined dataset performs best, but the gaps between it and using only one of citations or search results are both signiﬁcant and their contributions are complementary.
Revised: Input Corpus: From table 3 we also observe that, unsurprisingly, the combined dataset performs best, but the gaps between it and using only one of citations or search results are both signiﬁcant and their contributions are complementary.

Original: In subsequent experiments, we report only the combined results.
Revised: In subsequent experiments, we report only the combined results.

Revised: [Table]

Original: Abstractive model architecture and input length:
Revised: Abstractive model architecture and input length:

Original: As we see from Table 4, seq2seq-attention as a baseline does quite poorly on this task compared to the Transformer architectures.
Revised: As we see from Table 4, seq2seq-attention as a baseline does quite poorly on this task compared to the Transformer architectures.

Original: As seen in Figure 3, we observe that the Transformer encoder-decoder, T-ED, architecture consistently improves in performance until a best of around L = 500 − 1000 and is unable to learn at L = 2000 .
Revised: As seen in Figure 3, we observe that the Transformer encoder-decoder, T-ED, architecture consistently improves in performance until a best of around L = 500 − 1000 and is unable to learn at L = 2000 .

Original: This motivated the Transformer-Decoder, which we found could learn and improve up to L = 4000 , before running out of memory on our machines equipped with 16GB of GPU RAM (NVIDIA P100).
Revised: This motivated the Transformer-Decoder, which we found could learn and improve up to L = 4000 , before running out of memory on our machines equipped with 16GB of GPU RAM (NVIDIA P100).

Original: By using the T-DMCA modiﬁcations, we were able to train up to L = 11000 and continued to see improvements in performance.
Revised: By using the T-DMCA modiﬁcations, we were able to train up to L = 11000 and continued to see improvements in performance.

Original: We also found the MoE-layer helped performance by adding model capacity at high L , for example dropping log-perplexity from 2.05 to 1.93 at L = 11000 withexperts.
Revised: We also found the MoE-layer helped performance by adding model capacity at high L , for example dropping log-perplexity from 2.05 to 1.93 at L = 11000 withexperts.

Original: Our best model attempted uses 256 experts at L = 7500 (we were unable to use 256 experts with L = 11000 due to memory constraints) and achieves a perplexity of 1.90,[Table]
Revised: Our best model attempted uses 256 experts at L = 7500 (we were unable to use 256 experts with L = 11000 due to memory constraints) and achieves a perplexity of 1.90,

Original: Human Evaluation - Linguistic quality We conducted a DUC-style human evaluation of linguistic quality 3 of samples from a baseline abstractive (seq2seq), the best extractive ( tf-idf ), and our best TDMCA models.
Revised: Human Evaluation - Linguistic quality We conducted a DUC-style human evaluation of linguistic quality 3 of samples from a baseline abstractive (seq2seq), the best extractive ( tf-idf ), and our best TDMCA models.

Original: Five different dimensions are assessed: grammaticality, non-redundancy, referential clarity, focus, and structure/coherence.
Revised: Five different dimensions are assessed: grammaticality, non-redundancy, referential clarity, focus, and structure/coherence.

Original: As seen in Table 5, the T-DMCA model does statistically signiﬁcantly better on all dimensions, except on non-redundancy where tf-idf does about as well.
Revised: As seen in Table 5, the T-DMCA model does statistically signiﬁcantly better on all dimensions, except on non-redundancy where tf-idf does about as well.

Original: Overall, we observed high ﬂuency and coherence from our best abstractive model.
Revised: Overall, we observed high ﬂuency and coherence from our best abstractive model.

Original: Occasionally we observed some repetition of phrases which hurt the non-redundancy and structure, but it was much rarer compared with the other abstractive method, seq2seq .
Revised: Occasionally we observed some repetition of phrases which hurt the non-redundancy and structure, but it was much rarer compared with the other abstractive method, seq2seq .

Original: The biggest weakness of the extractive  method compared with our best abstractive model was the lack of structure and coherence in the summaries.
Revised: The biggest weakness of the extractive [Table] [Table] method compared with our best abstractive model was the lack of structure and coherence in the summaries.

Original: Human Evaluation - side-by-side preference We validated our chosen metrics correlate with human preference by conducting two side-by-side human evaluation experiments, comparing models with large gaps in perplexity/ROUGE.
Revised: Human Evaluation - side-by-side preference We validated our chosen metrics correlate with human preference by conducting two side-by-side human evaluation experiments, comparing models with large gaps in perplexity/ROUGE.

Original: We observe in 6 that human judgment correlates with our automatic metrics, but it becomes more difﬁcult to distinguish at the higher-end of model performance.
Revised: We observe in 6 that human judgment correlates with our automatic metrics, but it becomes more difﬁcult to distinguish at the higher-end of model performance.

Original: Details of the human evaluation experimental designs can be found in Appendix A.3.
Revised: Details of the human evaluation experimental designs can be found in Appendix A.

Original: To summarize the quantitative results, we believe the highest impact future work will be from improving the extractive stage and extending the decoder-only architectures to learn from larger L while maintaining sufﬁcient model capacity.

Original: [Table]
Revised: [Table]

Revised: To summarize the quantitative results

Original: [Table]
Revised: [Table]

Revised: we believe the highest impact future work will be from improving the extractive stage and extending the decoder-only architectures to learn from larger L while maintaining sufﬁcient model capacity.

Original: Comparison with Sauper & Barzilay (2009): A direct comparison with Sauper & Barzilay (2009) is difﬁcult for three reasons: (a) they report results only for two small subsets of Wikipedia, Diseases and American Actors; (b) we report on lead generation instead of full-articles; (c) we were unable to obtain the exact articles they used as input and output (in particular they make no claim of Wikiclone detection).
Revised: Comparison with Sauper & Barzilay (2009): A direct comparison with Sauper & Barzilay (2009) is difﬁcult for three reasons: (a) they report results only for two small subsets of Wikipedia, Diseases and American Actors; (b) we report on lead generation instead of full-articles; (c) we were unable to obtain the exact articles they used as input and output (in particular they make no claim of Wikiclone detection).

Original: However, we make a best-effort comparison by ﬁnding the subset of articles of our test set that correspond to Diseases and American Actors, the two categories reported on by Sauper & Barzilay and reporting our ROUGE-1 scores (Table 7).
Revised: However, we make a best-effort comparison by ﬁnding the subset of articles of our test set that correspond to Diseases and American Actors, the two categories reported on by Sauper & Barzilay and reporting our ROUGE-1 scores (Table 7).

Original: We observe that we perform better on
Revised: We observe that we perform better on

Original: American Actors than Diseases, probably because of the prevalence of the former (and biographies) in Wikipedia compared to the latter in our training set for our single, global model, whereas Sauper & Barzilay likely beneﬁt from the category-speciﬁc templates.
Revised: American Actors than Diseases, probably because of the prevalence of the former (and biographies) in Wikipedia compared to the latter in our training set for our single, global model, whereas Sauper & Barzilay likely beneﬁt from the category-speciﬁc templates.

Original: On average our ROUGE-1 scores are higher but do worse on the less common and somewhat speciﬁc disease category.
Revised: On average our ROUGE-1 scores are higher but do worse on the less common and somewhat speciﬁc disease category.

Original: 5. Q UALITATIVE D ISCUSSION
Revised: 5. Q UALITATIVE D ISCUSSION

Original: In Figure 4, we show the predictions from three different models (using tf-idf extraction, and the combined corpus) along with the Wikipedia ground truth.
Revised: In Figure 4, we show the predictions from three different models (using tf-idf extraction, and the combined corpus) along with the Wikipedia ground truth.

Original: As the perplexity decreases we see improvements in the model outputs, in terms of ﬂuency, factual accuracy, and narrative complexity.
Revised: As the perplexity decreases we see improvements in the model outputs, in terms of ﬂuency, factual accuracy, and narrative complexity.

Original: In particular, the T-DMCA model offers a respectable alternative to the Wikipedia version and is more succinct, while mentioning key facts, such as where the law ﬁrm was located, when and how it was formed, and the rise and fall of the ﬁrm.
Revised: In particular, the T-DMCA model offers a respectable alternative to the Wikipedia version and is more succinct, while mentioning key facts, such as where the law ﬁrm was located, when and how it was formed, and the rise and fall of the ﬁrm.

Original: In manual inspection of model outputs, we noticed an unexpected side-effect: models learn to translate names from English into multiple languages, e.g. Rohit Viswanath into Hindi (see Figure 5).
Revised: In manual inspection of model outputs, we noticed an unexpected side-effect: models learn to translate names from English into multiple languages, e.g. Rohit Viswanath into Hindi (see Figure 5).

Original: Although we did not do a systematic evaluation of the translations, we found they are often correct, and often they are not found in the Wikipedia article itself.
Revised: Although we did not do a systematic evaluation of the translations, we found they are often correct, and often they are not found in the Wikipedia article itself.

Original: We also veriﬁed that in general the  translation is not merely copied from the source, such as example cases where the target language is the incorrect one (e.g. translation of an English name into Ukrainian).
Revised: We also veriﬁed that in general the [Table] translation is not merely copied from the source, such as example cases where the target language is the incorrect one (e.g. translation of an English name into Ukrainian)

Original:  5. G ENERATING FULL -W IKIPEDIA ARTICLES
Revised: . 5. G ENERATING FULL -W IKIPEDIA ARTICLES

Original: Given that we have shown it is possible to learn sequence transduction models on combined inputoutput sequence lengths of approximately 12000 using the T-D architecture, we show that it is possible to train a model to generate entire Wikipedia articles.
Revised: Given that we have shown it is possible to learn sequence transduction models on combined inputoutput sequence lengths of approximately 12000 using the T-D architecture, we show that it is possible to train a model to generate entire Wikipedia articles.

Original: As a preliminary result, we trained two T-DMCA models: One is trained to use L = 6000 reference tokens to predict at most 2192 article tokens (longer examples are ignored) and another is conditioned only on the title and generates articles up to 4000 tokens long.[Table] Figure Appendix
Revised: As a preliminary result, we trained two T-DMCA models: One is trained to use L = 6000 reference tokens to predict at most 2192 article tokens (longer examples are ignored) and another is conditioned only on the title and generates articles up to 4000 tokens long.

Original: We show samples from both models in Appendix A.1.
Revised: We show samples from both models in Appendix A.1.

Original: Although the generated articles are not as good as the real Wikipedia or our lead section samples, the models can be seen to organize the article into plausible sections and exhibit global coherence over multi-paragraph text.
Revised: Although the generated articles are not as good as the real Wikipedia or our lead section samples, the models can be seen to organize the article into plausible sections and exhibit global coherence over multi-paragraph text.

Original: The model with access to reference documents inserts factual information in the generated article.
Revised: The model with access to reference documents inserts factual information in the generated article.

Original: Although we did not focus or tune on the full-article task we see this as an interesting future work for abstractive summarization.
Revised: Although we did not focus or tune on the full-article task we see this as an interesting future work for abstractive summarization.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have shown that generating Wikipedia can be approached as a multi-document summarization problem with a large, parallel dataset, and demonstrated a two-stage extractive-abstractive framework for carrying it out.
Revised: We have shown that generating Wikipedia can be approached as a multi-document summarization problem with a large, parallel dataset, and demonstrated a two-stage extractive-abstractive framework for carrying it out.

Original: The coarse extraction method used in the ﬁrst stage appears to have a signiﬁcant effect on ﬁnal performance, suggesting further research on improving it would be fruitful.
Revised: The coarse extraction method used in the ﬁrst stage appears to have a signiﬁcant effect on ﬁnal performance, suggesting further research on improving it would be fruitful.

Original: We introduce a new, decoder-only sequence transduction model for the abstractive stage, capable of handling very long input-output examples.
Revised: We introduce a new, decoder-only sequence transduction model for the abstractive stage, capable of handling very long input-output examples.

Original: This model signiﬁcantly outperforms traditional encoderdecoder architectures on long sequences, allowing us to condition on many reference documents and to generate coherent and informative Wikipedia articles.
Revised: This model signiﬁcantly outperforms traditional encoderdecoder architectures on long sequences, allowing us to condition on many reference documents and to generate coherent and informative Wikipedia articles.

Original:  P UBLIC RELEASE OF D ATASET AND C ODE
Revised:  P UBLIC RELEASE OF D ATASET AND C ODE

Original: To encourage further research on large-scale summarization we will release the URLs used in our experiments (the Wikipedia URL as well as the URLs of its references) that are available as part of the CommonCrawl dataset 4 , which is freely available for download.
Revised: To encourage further research on large-scale summarization, we will release the URLs used in our experiments (the Wikipedia URL as well as the URLs of its references) that are available as part of the CommonCrawl dataset 4 , which is freely available for download.

Revised: We use the open-source tensor2tensor 5 library for training abstractive models and will be releasing our abstractive modeling code extensions.

Original: Further details are available at github.com/RedactedForReview . 
Revised: Further details are available at https:// goo.gl/wSuuS9 . A CKNOWLEDGMENTS

Original: We use the open-source tensor2tensor 5 library for training abstractive models and will be releasing our abstractive modeling code to github.com/RedactedForReview .
Revised: We thank Samy Bengio, Jeff Dean, Claire Cui, Fred Bertsch, Chad Whipkey, Anurag Rana, Ashish Vaswani, Llion Jones, and the tensorflow/tensor2tensor contributors for help with the project.

Original: A. I MPLEMENTATION D ETAILS A.2. W IKIPEDIA CLONE DETECTION
Revised: A. I MPLEMENTATION D ETAILS A.2. W IKIPEDIA CLONE DETECTION

Original: For detecting whether a reference document, d , is a Wikipedia article clone we compute the maximum recall of unigrams for each section of the Wikipedia article, a : r ( d, a )
Revised: For detecting whether a reference document, d , is a Wikipedia article clone we compute the maximum recall of unigrams for each section of the Wikipedia article, a : r ( d, a )

Original: = max s ∈ sections ( a ) | unigrams ( d ) ∩ unigrams ( s ) | | unigrams ( s ) | and detect a clone if r > 0 .
Revised: = max s ∈ sections ( a ) | unigrams ( d ) ∩ unigrams ( s ) | | unigrams ( s ) | and detect a clone if r > 0 .

Original: A
Revised: A.

Original: H UMAN E VALUATION EXPERIMENT
Revised: H UMAN E VALUATION EXPERIMENT

Original: To assess linguistic quality, we randomly selected samples generated by models from the test set and ask raters to choose a score from 1 to 5 (higher is better) for ﬁve dimensions: Grammaticality,
Revised: To assess linguistic quality, we randomly selected samples generated by models from the test set and ask raters to choose a score from 1 to 5 (higher is better) for ﬁve dimensions: Grammaticality,

Original: Non-
Revised: Non-

Original: redundancy, Referential clarity, Focus, and Structure and Coherence.
Revised: redundancy, Referential clarity, Focus, and Structure and Coherence.

Original: These were used in the past at DUC for evaluating summaries (Dang, 2005).
Revised: These were used in the past at DUC for evaluating summaries ( ? ).

Original: For each model we selected 25 examples and averaged the scores for each question across 3 raters (out of pool of 7).
Revised: For each model we selected 25 examples and averaged the scores for each question across 3 raters (out of pool of 7).

Original: To compare two models by human evaluation, we randomly select examples from the test set and show model outputs side-by-side in the interface shown in Figure 9.
Revised: To compare two models by human evaluation, we randomly select examples from the test set and show model outputs side-by-side in the interface shown in Figure 9.

Original: Which side a model appears on is randomized per example and rater.
Revised: Which side a model appears on is randomized per example and rater.

Original: For the experiments in Table 6 we had 3 raters scoreexamples each and computed the ratio of ratings preferring one model over the other.
Revised: For the experiments in Table 6 we had 3 raters scoreexamples each and computed the ratio of ratings preferring one model over the other.

Original: A. E XAMPLE ABSTRACTIVE MODEL INPUT
Revised: A. E XAMPLE ABSTRACTIVE MODEL INPUT