Original: L EARNING TO W RITE BY L EARNING THE O BJECTIVE
Revised: L EARNING TO W RITE BY L EARNING THE O BJECTIVE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recurrent Neural Networks (RNNs) are powerful autoregressive sequence models for learning prevalent patterns in natural language.
Revised: Recurrent Neural Networks (RNNs) are powerful autoregressive sequence models for learning prevalent patterns in natural language.

Original: Yet language generated by RNNs often shows several degenerate characteristics that are uncommon in human language; while ﬂuent, RNN language production can be overly generic, repetitive, and even self-contradictory.
Revised: Yet language generated by RNNs often shows several degenerate characteristics that are uncommon in human language; while ﬂuent, RNN language production can be overly generic, repetitive, and even self-contradictory.

Original: We postulate that the objective function optimized by RNN language models, which amounts to the overall perplexity of a text, is not expressive enough to capture the abstract qualities of good generation such as Grice’s Maxims.
Revised: We postulate that the objective function optimized by RNN language models, which amounts to the overall perplexity of a text, is not expressive enough to capture the abstract qualities of good generation such as Grice’s Maxims.

Original: In this paper, we introduce a general learning framework that can construct a decoding objective better suited for generation.
Revised: In this paper, we introduce a general learning framework that can construct a decoding objective better suited for generation.

Original: Starting with a generatively trained RNN language model, our framework learns to construct a substantially stronger generator by combining several discriminatively trained models that can collectively address the limitations of RNN generation.
Revised: Starting with a generatively trained RNN language model, our framework learns to construct a substantially stronger generator by combining several discriminatively trained models that can collectively address the limitations of RNN generation.

Original: Human evaluation demonstrates that text generated by the resulting generator is preferred over that of baselines by a large margin and signiﬁcantly enhances the overall coherence, style, and information content of the generated text.
Revised: Human evaluation demonstrates that text generated by the resulting generator is preferred over that of baselines by a large margin and signiﬁcantly enhances the overall coherence, style, and information content of the generated text.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recurrent Neural Network (RNN) based language models such as Long Short-Term Memory Networks (LSTMs) (Hochreiter & Schmidhuber, 1997) and Gated Recurrent Units (GRUs) (Cho et al., 2014) have achieved enormous success across a variety of language tasks due to their ability to learn ﬂuency patterns in natural language (Jozefowicz et al., 2016;
Revised: Recurrent Neural Network (RNN) based language models such as Long Short-Term Memory Networks (LSTMs) (Hochreiter & Schmidhuber, 1997) and Gated Recurrent Units (GRUs) (Cho et al., 2014) have achieved enormous success across a variety of language tasks due to their ability to learn ﬂuency patterns in natural language (Jozefowicz et al., 2016;

Original: Kim et al., 2016; Mikolov et al., 2010).
Revised: Kim et al., 2016; Mikolov et al., 2010).

Original: When used as a generator, however, the quality of language generated from RNNs deviates drastically from that of human language.
Revised: When used as a generator, however, the quality of language generated from RNNs deviates drastically from that of human language.

Original: While ﬂuent, RNN-produced language displays several degenerative characteristics, favoring generic and contentless output and being easily repetitive and self-contradictory.
Revised: While ﬂuent, RNN-produced language displays several degenerative characteristics, favoring generic and contentless output and being easily repetitive and self-contradictory.

Original: These issues are especially prominent when RNNs are used for open-ended text generation dealing with long-term context, as illustrated in Figure 1.
Revised: These issues are especially prominent when RNNs are used for open-ended text generation dealing with long-term context, as illustrated in Figure 1.

Original: RNNs model the conditional probability P ( x t | x 1 ,...,t − 1 ) of generating the next word x t given all previous words observed or generated.
Revised: RNNs model the conditional probability P ( x t | x 1 ,...,t − 1 ) of generating the next word x t given all previous words observed or generated.

Original: In theory, this conditional model should be able to learn all crucial aspects of human language production, for example, that we don’t normally repeat the same content over and over.
Revised: In theory, this conditional model should be able to learn all crucial aspects of human language production, for example, that we don’t normally repeat the same content over and over.

Original: In practice, however, the learned conditional probability model often assigns higher probability to a repetitive, overly generic sentence than to higher quality sentences, as shown in Figure 1.
Revised: In practice, however, the learned conditional probability model often assigns higher probability to a repetitive, overly generic sentence than to higher quality sentences, as shown in Figure 1.

Original: We postulate that this is in part because the network architectures of RNN variants do not provide a strong enough inductive bias for the model to learn the complex communication goals pursued in human writing.
Revised: We postulate that this is in part because the network architectures of RNN variants do not provide a strong enough inductive bias for the model to learn the complex communication goals pursued in human writing.

Original: In addition, long-term context easily gets lost as it is explained away in the presence of more immediately relevant short-term context (Yu et al., 2017), and as gradients diminish over a long sequence (Pascanu et al., 2013).
Revised: In addition, long-term context easily gets lost as it is explained away in the presence of more immediately relevant short-term context (Yu et al., 2017), and as gradients diminish over a long sequence (Pascanu et al., 2013).

Original: Consequently, RNNs acquire relatively shallow and myopic patterns, and the corresponding probability score P ( x t | x 1 ,...,t − 1 ) is also insufﬁcient for generating language that matches the complexity and coherence of human generated text.
Revised: Consequently, RNNs acquire relatively shallow and myopic patterns, and the corresponding probability score P ( x t | x 1 ,...,t − 1 ) is also insufﬁcient for generating language that matches the complexity and coherence of human generated text.

Original: Several methods in the literature attempt to mitigate these issues.
Revised: Several methods in the literature attempt to mitigate these issues.

Original: Overly simple and generic generation can be improved by and using a diversity-boosting objective function (Shao et al., 2017;
Revised: Overly simple and generic generation can be improved by and using a diversity-boosting objective function (Shao et al., 2017;

Original: Vijayakumar et al., 2016).
Revised: Vijayakumar et al., 2016).

Original: Repetitive generation can be reduced by prohibiting recurrence of the same trigrams as hard rules (Paulus et al., 2017), although such constraints are only a partial solution as they cannot stop RNNs from repeating through paraphrasing.
Revised: Repetitive generation can be reduced by prohibiting recurrence of the same trigrams as hard rules (Paulus et al., 2017), although such constraints are only a partial solution as they cannot stop RNNs from repeating through paraphrasing.

Original: All in all, I would highly recommend this hotel to anyone who wants to be in the heart of the action, and want to be in the heart of the action.
Revised: All in all, I would highly recommend this hotel to anyone who wants to be in the heart of the action, and want to be in the heart of the action.

Original: If you want to be in the heart of the action, this is not the place for you.
Revised: If you want to be in the heart of the action, this is not the place for you.

Original: However, If you want to be in the middle of the action, this is the place to be.
Revised: However, If you want to be in the middle of the action, this is the place to be.

Original: A common thread across all these methods is to seek a new objective function that is better suited for generation.
Revised: A common thread across all these methods is to seek a new objective function that is better suited for generation.

Original: Since it is not straightforward to write the perfect objective function that can handle multiple abstract properties of good generation, we instead propose a general learning framework to construct a better decoding objective by learning.
Revised: Since it is not straightforward to write the perfect objective function that can handle multiple abstract properties of good generation, we instead propose a general learning framework to construct a better decoding objective by learning.

Original: Starting with a generatively trained RNN language model, our framework learns to construct a substantially stronger generator by combining several discriminatively trained models that can collectively address limitations of the base RNN generator.
Revised: Starting with a generatively trained RNN language model, our framework learns to construct a substantially stronger generator by combining several discriminatively trained models that can collectively address limitations of the base RNN generator.

Original: Our learning framework therefore generalizes over various existing modiﬁcations to the decoding objective.
Revised: Our learning framework therefore generalizes over various existing modiﬁcations to the decoding objective.

Original: In addition, our approach learns to overcome the particular limitations of the
Revised: In addition, our approach learns to overcome the particular limitations of the

Original: RNN generator directly by incorporating language generated from RNNs as negative samples to discriminatively train several companion models, each specializing different aspects of Grice’s Maxims of communication.
Revised: RNN generator directly by incorporating language generated from RNNs as negative samples to discriminatively train several companion models, each specializing different aspects of Grice’s Maxims of communication.

Original: Empirical results demonstrate that our learning framework is highly effective in converting a generic
Revised: Empirical results demonstrate that our learning framework is highly effective in converting a generic

Original: RNN language model into a substantially stronger generator.
Revised: RNN language model into a substantially stronger generator.

Original: Human evaluation conﬁrms that language generated by our model is preferred over that of competitive baselines by a large margin and signiﬁcantly enhances the overall coherence, style, and information content of the generated text.
Revised: Human evaluation conﬁrms that language generated by our model is preferred over that of competitive baselines by a large margin and signiﬁcantly enhances the overall coherence, style, and information content of the generated text.

Original:  B ACKGROUND : G RICE ’ S M AXIMS
Revised:  B ACKGROUND : G RICE ’ S M AXIMS

Original: We motivate our learning framework using Grice’s Maxims of communication (Grice et al., 1975):
Revised: We motivate our learning framework using Grice’s Maxims of communication (Grice et al., 1975):

Original: Quantity:
Revised: Quantity:

Original: Make your contribution as informative as required, and no more than necessary.
Revised: Make your contribution as informative as required, and no more than necessary.

Original: RNN generation tends to violate this maxim as it is often overly short and generic, therefore is less informative than desired.
Revised: RNN generation tends to violate this maxim as it is often overly short and generic, therefore is less informative than desired.

Original: When encouraged to generate longer text, RNNs easily repeat themselves, which also works against conveying the right amount of information.
Revised: When encouraged to generate longer text, RNNs easily repeat themselves, which also works against conveying the right amount of information.

Original: This observation motivates the length and the repetition models in §
Revised: This observation motivates the length and the repetition models in §

Original: .2.1 and § 3.2.2.
Revised: .2.1 and § 3.2.2.

Original: Quality: Do not say what you believe to be false.
Revised: Quality: Do not say what you believe to be false.

Original: When used for generation with long term context, RNNs can generate text that is self-contradictory.
Revised: When used for generation with long term context, RNNs can generate text that is self-contradictory.

Original: We propose the entailment model to address this problem in § 3.2.3.
Revised: We propose the entailment model to address this problem in § 3.2.3.

Original: Relation: Be relevant.
Revised: Relation: Be relevant.

Original: RNNs used for long generation can start digressing as the long-term context gets easily washed out.
Revised: RNNs used for long generation can start digressing as the long-term context gets easily washed out.

Original: We address this problem by proposing two different relevance models in § 3.2.4 and § 3.2.5.
Revised: We address this problem by proposing two different relevance models in § 3.2.4 and § 3.2.5.

Original: Manner
Revised: Manner

Original: Avoid obscurity of expression.
Revised: Avoid obscurity of expression.

Original: Avoid ambiguity.
Revised: Avoid ambiguity.

Original: Be brief.
Revised: Be brief.

Original: Be orderly.
Revised: Be orderly.

Original: As RNN generation favors highly probable but generic words and phrases, it lacks adequate style and speciﬁcity.
Revised: As RNN generation favors highly probable but generic words and phrases, it lacks adequate style and speciﬁcity.

Original: We address this issue with the lexical style model in § 3.2.6.
Revised: We address this issue with the lexical style model in § 3.2.6.

Original:  T HE L EARNING F RAMEWORK
Revised:  T HE L EARNING F RAMEWORK

Original: We propose a general learning framework for conditional language generation of sequence x given context y .
Revised: We propose a general learning framework for conditional language generation of sequence x given context y .

Original: The decoding objective for generation takes the general form:
Revised: The decoding objective for generation takes the general form:

Original: [Equation]
Revised: [Equation]

Original: This objective combines the RNN probability ( § 3.1) with a set of additional scores s k ( x , y ) taken from discriminatively trained communication models ( § 3.2).
Revised: This objective combines the RNN probability ( § 3.1) with a set of additional scores s k ( x , y ) taken from discriminatively trained communication models ( § 3.2).

Original: This corresponds to a Product of Experts (PoE) model (Hinton, 2006) when the scores s k are log probabilities.
Revised: This corresponds to a Product of Experts (PoE) model (Hinton, 2006) when the scores s k are log probabilities.

Original: The language model decomposes into per-word probabilities using the chain rule applied over RNNs.
Revised: The language model decomposes into per-word probabilities using the chain rule applied over RNNs.

Original: However, in order to allow for more expressive learning over long range context we do not require the discriminative model scores to factorize over the elements of x , thereby addressing a key limitation of RNNs.
Revised: However, in order to allow for more expressive learning over long range context we do not require the discriminative model scores to factorize over the elements of x , thereby addressing a key limitation of RNNs.

Original: More speciﬁcally, we use an estimated score s (cid:48) k
Revised: More speciﬁcally, we use an estimated score s (cid:48) k

Original: ( x 1: i , y ) that can be computed for any preﬁx of x = x 1: n to approximate the objective during beam search, such that s (cid:48) k ( x 1: n , y ) =
Revised: ( x 1: i , y ) that can be computed for any preﬁx of x = x 1: n to approximate the objective during beam search, such that s (cid:48) k ( x 1: n , y ) =

Original: s k
Revised: s k

Original: ( x , y ) .
Revised: ( x , y ) .

Original: The scoring functions are typically trained on preﬁxes of x to simulate their application to rank partial continuations during beam search.
Revised: The scoring functions are typically trained on preﬁxes of x to simulate their application to rank partial continuations during beam search.

Original: Further model and hyperparameter details are given in appendix B.
Revised: Further model and hyperparameter details are given in appendix B.

Original: 3. B ASE L ANGUAGE MODEL
Revised: 3. B ASE L ANGUAGE MODEL

Original: We train a RNN language model to estimate
Revised: We train a RNN language model to estimate

Original: [Equation]
Revised: [Equation]

Original: The language model does not distinguish between the context y and the continuation, but treats everything as a single sequence x .
Revised: The language model does not distinguish between the context y and the continuation, but treats everything as a single sequence x .

Original: 3. C OMPOSITE C OMMUNICATION M ODELS
Revised: 3. C OMPOSITE C OMMUNICATION M ODELS

Original: Next we introduce a set of models that are motivated by Grice’s Maxims of communication.
Revised: Next we introduce a set of models that are motivated by Grice’s Maxims of communication.

Original: Each model is trained to discriminate between good and bad generation using a ranking objective, so that its model scores have the effect of re-ranking in the decoder.
Revised: Each model is trained to discriminate between good and bad generation using a ranking objective, so that its model scores have the effect of re-ranking in the decoder.

Original: We vary the model parameterization and training examples to guide each model to focus on different aspects of Grice’s Maxims.
Revised: We vary the model parameterization and training examples to guide each model to focus on different aspects of Grice’s Maxims.

Original: The classiﬁer scores are interpreted as classiﬁcation probabilities and added to the objective function as log probabilities.
Revised: The classiﬁer scores are interpreted as classiﬁcation probabilities and added to the objective function as log probabilities.

Original: Let D =
Revised: Let D =

Original: ( x 1 , y 1 ) , . . .
Revised: ( x 1 , y 1 ) , . . .

Original: ( x n , y n ) } be the set of conditional generation training examples.
Revised: ( x n , y n ) } be the set of conditional generation training examples.

Original: In all models the ﬁrst layer embeds words into 300-dimensional vectors initialized with GloVe (Pennington et al., 2014) embeddings, which are ﬁne-tuned during training unless stated otherwise.
Revised: In all models the ﬁrst layer embeds words into 300-dimensional vectors initialized with GloVe (Pennington et al., 2014) embeddings, which are ﬁne-tuned during training unless stated otherwise.

Original: The dimensions of hidden layers are also 300 .
Revised: The dimensions of hidden layers are also 300 .

Original: Let e ( x i ) be the word embedding of word x i .
Revised: Let e ( x i ) be the word embedding of word x i .

Original: 3.2. L ENGTH M ODEL
Revised: 3.2. L ENGTH M ODEL

Original: RNNs tend to bias toward shorter generation even with a highly expressive network structure with attention, thus length normalization is still a common practice (Wu et al., 2016).
Revised: RNNs tend to bias toward shorter generation even with a highly expressive network structure with attention, thus length normalization is still a common practice (Wu et al., 2016).

Original: We use a geometrically decaying length reward.
Revised: We use a geometrically decaying length reward.

Original: We tune the initial value and decaying rate on the ﬁnal systems, and use the same value for all systems with the initial value and decay rate being 1 and 0 .
Revised: We tune the initial value and decaying rate on the ﬁnal systems, and use the same value for all systems with the initial value and decay rate being 1 and 0 .

Original: respectively.
Revised: respectively.

Original: 3.2. R EPETITION M ODEL
Revised: 3.2. R EPETITION M ODEL

Original: The goal of the repetition model is to distinguish between RNN generated continuation and gold continuations by detecting repetitions in the generated continuation.
Revised: The goal of the repetition model is to distinguish between RNN generated continuation and gold continuations by detecting repetitions in the generated continuation.

Original: First a distance score d i is computed for each token in the continuation x , based on pairwise cosine similarity, as
Revised: First a distance score d i is computed for each token in the continuation x , based on pairwise cosine similarity, as

Original: [Equation]
Revised: [Equation]

Original: Therefore d i = 1 for all positions at which words are repeated.
Revised: Therefore d i = 1 for all positions at which words are repeated.

Original: The score is then computed as
Revised: The score is then computed as

Original: [Equation]
Revised: [Equation]

Original: where RNN ( d ) is the ﬁnal state of a unidirectional RNN ran over the similarity scores.
Revised: where RNN ( d ) is the ﬁnal state of a unidirectional RNN ran over the similarity scores.

Original: The model is trained to maximize binary cross-entropy, with gold continuations as positive examples and samples from the base RNNs, which often contain some repetition, as negative examples:
Revised: The model is trained to maximize binary cross-entropy, with gold continuations as positive examples and samples from the base RNNs, which often contain some repetition, as negative examples:

Original: [Equation]
Revised: [Equation]

Original: The word embeddings are kept ﬁxed during training for this model.
Revised: The word embeddings are kept ﬁxed during training for this model.

Original: 3.2. E NTAILMENT M ODEL
Revised: 3.2. E NTAILMENT M ODEL

Original: The goal of the entailment model is to guide the generator to neither contradict its own past generation (the maxim of Quality) nor state an obvious entailment that readily follows from the context (the maxim of Quantity).
Revised: The goal of the entailment model is to guide the generator to neither contradict its own past generation (the maxim of Quality) nor state an obvious entailment that readily follows from the context (the maxim of Quantity).

Original: The unwanted entailment cases include repetitions and paraphrasing.
Revised: The unwanted entailment cases include repetitions and paraphrasing.

Original: We train a classiﬁer using the MultiSNLI dataset (Williams et al., 2017) that takes two sentences a and b as input and predicts the relation between them as either contradiction , entailment or neutral .
Revised: We train a classiﬁer using the MultiSNLI dataset (Williams et al., 2017) that takes two sentences a and b as input and predicts the relation between them as either contradiction , entailment or neutral .

Original: We use the score of the neutral class probability of the sentence pair to discourage both contradiction and entailment.
Revised: We use the score of the neutral class probability of the sentence pair to discourage both contradiction and entailment.

Original: We use a continuous bag-of-words model similar to previously reported NLI baselines (Mou et al., 2016).
Revised: We use a continuous bag-of-words model similar to previously reported NLI baselines (Mou et al., 2016).

Original: Sentence representations are obtained by summing word embeddings (which are ﬁne-tuned during training), such that r ( a ) =
Revised: Sentence representations are obtained by summing word embeddings (which are ﬁne-tuned during training), such that r ( a ) =

Original: (cid:80) i e ( a i ) and r ( b ) =
Revised: (cid:80) i e ( a i ) and r ( b ) =

Original: (cid:80) i e ( b i ) .
Revised: (cid:80) i e ( b i ) .

Original: An MLP classiﬁer with a single tanh hidden layer takes as input concatenations of the two sentence embeddings, along with their element-wise difference and multiplication; the output is a three-way softmax.
Revised: An MLP classiﬁer with a single tanh hidden layer takes as input concatenations of the two sentence embeddings, along with their element-wise difference and multiplication; the output is a three-way softmax.

Original: The classiﬁer obtains 63% validation accuracy.
Revised: The classiﬁer obtains 63% validation accuracy.

Original: In contrast to other communication models, this classiﬁer cannot be applied directly to the full context and continuation it is scoring.
Revised: In contrast to other communication models, this classiﬁer cannot be applied directly to the full context and continuation it is scoring.

Original: Instead every completed sentence in the continuation should be scored against all preceding sentences in both the context and continuation.
Revised: Instead every completed sentence in the continuation should be scored against all preceding sentences in both the context and continuation.

Original: Let S ( x ) be the list of complete sentences in x , and t ( a , b ) the log probability of the neutral class.
Revised: Let S ( x ) be the list of complete sentences in x , and t ( a , b ) the log probability of the neutral class.

Original: The entailment score is computed as
Revised: The entailment score is computed as

Original: [Equation]
Revised: [Equation]

Original: that is, the score of the sentence-pair for which we have the least conﬁdence that entailment is neutral.
Revised: that is, the score of the sentence-pair for which we have the least conﬁdence that entailment is neutral.

Original: To enable fair comparison across beam hypotheses with different sentence lengths, s (cid:48) entail does not accumulate scores across sentences; rather assume that the effect of the last score will already be reﬂected in the content of the beam when the next sentence is scored.
Revised: To enable fair comparison across beam hypotheses with different sentence lengths, s (cid:48) entail does not accumulate scores across sentences; rather assume that the effect of the last score will already be reﬂected in the content of the beam when the next sentence is scored.

Original: While the ﬁrst sentence is generated, s (cid:48) entail takes an initial constant value to avoid bias towards incomplete sentences.
Revised: While the ﬁrst sentence is generated, s (cid:48) entail takes an initial constant value to avoid bias towards incomplete sentences.

Original: 3.2. R ELEVANCE M ODEL
Revised: 3.2. R ELEVANCE M ODEL

Original: The purpose of the relevance model is to predict whether the content of a candidate continuation is relevant to the given context.
Revised: The purpose of the relevance model is to predict whether the content of a candidate continuation is relevant to the given context.

Original: We train the model to distinguish between true continuations and random continuations sampled from other (human-written) endings in the corpus, conditioned on the given context.
Revised: We train the model to distinguish between true continuations and random continuations sampled from other (human-written) endings in the corpus, conditioned on the given context.

Original: A single convolutional layer is applied to both the context and candidate embedding sequences.
Revised: A single convolutional layer is applied to both the context and candidate embedding sequences.

Original: The scoring function is deﬁned as
Revised: The scoring function is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where 1D maxpooling is performed over each sequence to obtain a vector representing its most important semantic dimensions.
Revised: where 1D maxpooling is performed over each sequence to obtain a vector representing its most important semantic dimensions.

Original: Element-wise multiplication of the context and continuation vectors will amplify similarities between the two.
Revised: Element-wise multiplication of the context and continuation vectors will amplify similarities between the two.

Original: We optimize the following ranking log likelihood:
Revised: We optimize the following ranking log likelihood:

Original: [Equation]
Revised: [Equation]

Original: where x t and x r are true and random endings, respectively.
Revised: where x t and x r are true and random endings, respectively.

Original: For each context and true continuation we extract 5 randomly selected endings as training examples.
Revised: For each context and true continuation we extract 5 randomly selected endings as training examples.

Original: The model ranks true endings higher than randomly selected ones with 85% accuracy on the validation set.
Revised: The model ranks true endings higher than randomly selected ones with 85% accuracy on the validation set.

Original: The trained scores do not correspond to probabilities but we scale with the logistic (sigmoid) function for compatibility.
Revised: The trained scores do not correspond to probabilities but we scale with the logistic (sigmoid) function for compatibility.

Original: 3.2. W ORKING V OCABULARY M ODEL
Revised: 3.2. W ORKING V OCABULARY M ODEL

Original: Given even a fragment of context (e.g. “We were excited about our beach getaway...”)
Revised: Given even a fragment of context (e.g. “We were excited about our beach getaway...”)

Original: certain topics become more relevant (e.g. “swimming”, “relax”, “romantic”).
Revised: certain topics become more relevant (e.g. “swimming”, “relax”, “romantic”).

Original: To capture this intuition, we predict a centroid in the embedding space from the context, which describes a neighborhood of the
Revised: To capture this intuition, we predict a centroid in the embedding space from the context, which describes a neighborhood of the

Original: Data: context y
Revised: Data: context y

Original: =
Revised: =

Original: y 1 ··· y n , beam size k , meta-weights λ Result: best continuation centroid
Revised: y 1 ··· y n , beam size k , meta-weights λ Result: best continuation centroid

Original: = working vocab( y ) (cid:46) used to compute topicality scores best =
Revised: = working vocab( y ) (cid:46) used to compute topicality scores best =

Original: None beam =
Revised: None beam =

Original: [ y ] for step = 0; step < max steps; step = step +1 do next beam =
Revised: [ y ] for step = 0; step < max steps; step = step +1 do next beam =

Original: [] for candidate in beam do expand(next beam, next k(candidate, centroid, λ ))
Revised: [] for candidate in beam do expand(next beam, next k(candidate, centroid, λ ))

Original: (cid:46) mix LM and topicality scores if lookahead score(candidate.append(termination token))
Revised: (cid:46) mix LM and topicality scores if lookahead score(candidate.append(termination token))

Original: > best.score then best = candidate.append(term) end endfor candidate in next beam do candidate.score += f λ ( candidate ) (cid:46) score with additional models end beam =
Revised: > best.score then best = candidate.append(term) end endfor candidate in next beam do candidate.score += f λ ( candidate ) (cid:46) score with additional models end beam =

Original: sort(next beam, key=score)[:k] (cid:46) select top k candidates endif learning then update lambda with gradient descent by comparing beam against the gold end return best embedding space we expect the continuation to intersect.
Revised: sort(next beam, key=score)[:k] (cid:46) select top k candidates endif learning then update lambda with gradient descent by comparing beam against the gold end return best embedding space we expect the continuation to intersect.

Original: The score is computed as
Revised: The score is computed as

Original: [Equation]
Revised: [Equation]

Original: where RNN ( x ) is the ﬁnal state of the RNN.
Revised: where RNN ( x ) is the ﬁnal state of the RNN.

Original: The model is trained with a discriminative loss:
Revised: The model is trained with a discriminative loss:

Original: [Equation]
Revised: [Equation]

Original: 3.2. L EXICAL S TYLE M ODEL
Revised: 3.2. L EXICAL S TYLE M ODEL

Original: The goal of the lexical style model is to learn the stylistic aspects of the desirable writing based on the lexical distributions.
Revised: The goal of the lexical style model is to learn the stylistic aspects of the desirable writing based on the lexical distributions.

Original: The scoring function is deﬁned as
Revised: The scoring function is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The model is trained with a similar ranking criteria as the relevance model, except that the negative examples are sampled from the language model:
Revised: The model is trained with a similar ranking criteria as the relevance model, except that the negative examples are sampled from the language model:

Original: [Equation]
Revised: [Equation]

Original: 3. B EAM SEARCH AND META - WEIGHT LEARNING
Revised: 3. B EAM SEARCH AND META - WEIGHT LEARNING

Original: Once all the sub-objectives have been trained, we need to learn the combined objective.
Revised: Once all the sub-objectives have been trained, we need to learn the combined objective.

Original: We learn the weight coefﬁcients λ k in equation 1 to linearly combine the scoring functions, using a discriminative objective to rank true continuations above continuations found during beam-search:
Revised: We learn the weight coefﬁcients λ k in equation 1 to linearly combine the scoring functions, using a discriminative objective to rank true continuations above continuations found during beam-search:

Original: [Equation]
Revised: [Equation]

Original: where A is our inference algorithm, shown in Algorithm 1.
Revised: where A is our inference algorithm, shown in Algorithm 1.

Original: For each training iteration beam-search is performed based on the current values of λ , meaning the the objective function changes dynamically, leading to a form of meta-weight learning.
Revised: For each training iteration beam-search is performed based on the current values of λ , meaning the the objective function changes dynamically, leading to a form of meta-weight learning.

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTS 4. T HE TROUBLE WITH AUTOMATIC EVALUATION
Revised:  E XPERIMENTS 4. T HE TROUBLE WITH AUTOMATIC EVALUATION

Original: Previous work has reported that automatic measures such as BLEU, ROUGE, and Meteor, do not lead to meaningful evaluation when used for long or creative text generation where there can be high variance among correct generation output (Wiseman et al., 2017b; Vedantam et al., 2015).
Revised: Previous work has reported that automatic measures such as BLEU, ROUGE, and Meteor, do not lead to meaningful evaluation when used for long or creative text generation where there can be high variance among correct generation output (Wiseman et al., 2017b; Vedantam et al., 2015).

Original: For open-ended generation tasks such as our own, human evaluation is the only reliable measure (Li et al., 2016b; Wiseman et al., 2017b).
Revised: For open-ended generation tasks such as our own, human evaluation is the only reliable measure (Li et al., 2016b; Wiseman et al., 2017b).

Original: However, we also report the automatic measures to echo the previous observation regarding the mismatch between automatic and human evaluation
Revised: However, we also report the automatic measures to echo the previous observation regarding the mismatch between automatic and human evaluation

Original: [Table]
Revised: [Table]

Original: and provide multiple generation samples that give insights into the characteristics of different models.
Revised: and provide multiple generation samples that give insights into the characteristics of different models.

Original: 4. E XPERIMENTAL SETUP
Revised: 4. E XPERIMENTAL SETUP

Original: We pose the evaluation of our model as a conditional natural language generation task.
Revised: We pose the evaluation of our model as a conditional natural language generation task.

Original: Given an initial context of n sentences, the task is to generate an appropriate ending.
Revised: Given an initial context of n sentences, the task is to generate an appropriate ending.

Original: For the purposes of automatic evaluation, the ending is compared against a gold reference ending that was written by a human (drawn from the original corpus).
Revised: For the purposes of automatic evaluation, the ending is compared against a gold reference ending that was written by a human (drawn from the original corpus).

Original: For human evaluation, the ending is presented to a human, who assesses the text according to several criteria.
Revised: For human evaluation, the ending is presented to a human, who assesses the text according to several criteria.

Original: The criteria given to human judges are four metrics closely inspired by Grice’s Maxims:
Revised: The criteria given to human judges are four metrics closely inspired by Grice’s Maxims:

Original: Quantity : Does the generation avoid repetition?
Revised: Quantity : Does the generation avoid repetition?

Original: Quality : Does the generation contradict itself (or the initial context)?
Revised: Quality : Does the generation contradict itself (or the initial context)?

Original: Relation : Does the generation appropriately continue the initial context?
Revised: Relation : Does the generation appropriately continue the initial context?

Original: Manner : Does the generation use the same style as the initial context?
Revised: Manner : Does the generation use the same style as the initial context?

Original: Finally, a Turing test question is presented to each judge: was the ending written by a human?
Revised: Finally, a Turing test question is presented to each judge: was the ending written by a human?

Original: In all of our experiments, we split each passage in our corpus into n = 4 initial context sentences and leave the remaining sentences as the generation target.
Revised: In all of our experiments, we split each passage in our corpus into n = 4 initial context sentences and leave the remaining sentences as the generation target.

Original: For both automatic and human evaluation, we select 1000 passages from the test set of each corpus and generate endings using all models.
Revised: For both automatic and human evaluation, we select 1000 passages from the test set of each corpus and generate endings using all models.

Original: Automatic evaluation simply scores each generated ending against its reference ending.
Revised: Automatic evaluation simply scores each generated ending against its reference ending.

Original: To conduct human evaluation, we present the initial context with the generation from each of the models individually to workers on Mechanical Turk, who score them using the above criteria.
Revised: To conduct human evaluation, we present the initial context with the generation from each of the models individually to workers on Mechanical Turk, who score them using the above criteria.

Original: As a control, we also ask workers to score the original (human-written) reference endings for each selected passage.
Revised: As a control, we also ask workers to score the original (human-written) reference endings for each selected passage.

Original: [Table]
Revised: [Table]

Original: Automated metrics are from 0–100.
Revised: Automated metrics are from 0–100.

Original: Human metrics are from 1– except for the Turing test, which is from 0–100.
Revised: Human metrics are from 1– except for the Turing test, which is from 0–100.

Original: Each cell is the micro-averaged across 1000 datums from a held-out test set.
Revised: Each cell is the micro-averaged across 1000 datums from a held-out test set.

Original: 4. C ORPORA
Revised: 4. C ORPORA

Original: We use two corpora for evaluation.
Revised: We use two corpora for evaluation.

Original: The ﬁrst is TripAdvisor reviews.
Revised: The ﬁrst is TripAdvisor reviews.

Original: 1 We use only reviews that have at least 5 sentences, using the ﬁrst 4 as the context and the remainder as the reference ending.
Revised: 1 We use only reviews that have at least 5 sentences, using the ﬁrst 4 as the context and the remainder as the reference ending.

Original: There are on average 11 sentences per review.
Revised: There are on average 11 sentences per review.

Original: We use the ﬁrst 1M reviews for training.
Revised: We use the ﬁrst 1M reviews for training.

Original: The second is the
Revised: The second is the

Original: ROCStory corpus (Mostafazadeh et al., 2016), which is a collection of crowdsourced commonsense stories in short ﬁve sentences (98k stories in the training set).
Revised: ROCStory corpus (Mostafazadeh et al., 2016), which is a collection of crowdsourced commonsense stories in short ﬁve sentences (98k stories in the training set).

Original: 4. M ODELS
Revised: 4. M ODELS

Original: Language model We use the language model from Section 3.1 (2-layer RNN with 1024 GRU cells per layer) as our reference baseline.
Revised: Language model We use the language model from Section 3.1 (2-layer RNN with 1024 GRU cells per layer) as our reference baseline.

Original: For the (larger) TripAdvisor corpus, we train the language model on that corpus alone.
Revised: For the (larger) TripAdvisor corpus, we train the language model on that corpus alone.

Original: For the ROCStory corpus, we pretrained the model for 420M tokens on the Toronto Books corpus 2 before training on the ROCStory text.
Revised: For the ROCStory corpus, we pretrained the model for 420M tokens on the Toronto Books corpus 2 before training on the ROCStory text.

Original: We generate text using beam search with a beam size of 8.
Revised: We generate text using beam search with a beam size of 8.

Original: We include three variants of our model: L2W ( NO META - LEARN ) uses a uniform mixture of scores for the learned sub-objectives (no meta-weight learning); L2W ( NO WK .
Revised: We include three variants of our model: L2W ( NO META - LEARN ) uses a uniform mixture of scores for the learned sub-objectives (no meta-weight learning); L2W ( NO WK .

Original: VOCAB ) omits the working vocabulary model (Section 3.2.5) but uses meta-weight learning for the other components; and L2W ( FULL ) is our full model.
Revised: VOCAB ) omits the working vocabulary model (Section 3.2.5) but uses meta-weight learning for the other components; and L2W ( FULL ) is our full model.

Original:  R ESULTS AND A NALYSIS 5. Q UANTITATIVE A NALYSIS
Revised:  R ESULTS AND A NALYSIS 5. Q UANTITATIVE A NALYSIS

Original: Results for both automatic and human evaluation metrics are presented for both corpora in Table 2.
Revised: Results for both automatic and human evaluation metrics are presented for both corpora in Table 2.

Original: For the TripAdvisor corpus, the language model baseline does best on all of the automated metrics.
Revised: For the TripAdvisor corpus, the language model baseline does best on all of the automated metrics.

Original: However on all human metrics, the L2W ( NO WK .
Revised: However on all human metrics, the L2W ( NO WK .

Original: VOCAB ) model scores higher, and achieves nearly twice the percentage of passed Turing tests (75% compared to 39%).
Revised: VOCAB ) model scores higher, and achieves nearly twice the percentage of passed Turing tests (75% compared to 39%).

Original: It scores even better than the L2W ( FULL ) model.
Revised: It scores even better than the L2W ( FULL ) model.

Original: Manual inspection reveals that for the TripAdvisor task, the topical focus and lexical speciﬁcity given by vocabulary scoring actually lead to slightly worse model behavior.
Revised: Manual inspection reveals that for the TripAdvisor task, the topical focus and lexical speciﬁcity given by vocabulary scoring actually lead to slightly worse model behavior.

Original: In TripAdvisor reviews, many possible sentences could logically follow a reviews context.
Revised: In TripAdvisor reviews, many possible sentences could logically follow a reviews context.

Original: A well-behaving but generic language generator is better able to leave judge’s unsurprised, if also uninformed.
Revised: A well-behaving but generic language generator is better able to leave judge’s unsurprised, if also uninformed.

Original: The full model brings additional focus to the generations, but at the cost of a reduction in overall coherence.
Revised: The full model brings additional focus to the generations, but at the cost of a reduction in overall coherence.

Original: In the ROCStory generation task the language model is more competitive with L2W ( FULL ).
Revised: In the ROCStory generation task the language model is more competitive with L2W ( FULL ).

Original: The raw language model is able to take advantage of the simpler data inherent in this task: ROCStories are shorter (the endings are always a single sentence) and have a smaller vocabulary.
Revised: The raw language model is able to take advantage of the simpler data inherent in this task: ROCStories are shorter (the endings are always a single sentence) and have a smaller vocabulary.

Original: However, the L2W full model still performs the best on all human metrics.
Revised: However, the L2W full model still performs the best on all human metrics.

Original: The ROCStory corpus presents speciﬁc contexts that require a narrower set of responses to maintain coherency.
Revised: The ROCStory corpus presents speciﬁc contexts that require a narrower set of responses to maintain coherency.

Original: The L2W ( FULL ) takes advantage of reranking by matching the topic of the context, yielding a higher ‘Relation’ score and more Turing test passes overall.
Revised: The L2W ( FULL ) takes advantage of reranking by matching the topic of the context, yielding a higher ‘Relation’ score and more Turing test passes overall.

Original: 5. Q UALITATIVE A NALYSIS
Revised: 5. Q UALITATIVE A NALYSIS

Original: Learning to Write (L2W) generations are more topical and coherently mold with the context.
Revised: Learning to Write (L2W) generations are more topical and coherently mold with the context.

Original: Tableshows that both the L2W system as well as the classic RNN start similarly, commenting on the hotel staff and the room.
Revised: Tableshows that both the L2W system as well as the classic RNN start similarly, commenting on the hotel staff and the room.

Original: L2W is able to condense the same content into fewer words, following the context’s style.
Revised: L2W is able to condense the same content into fewer words, following the context’s style.

Original: Beginning with the third sentence, L2W and the LM diverge more signiﬁcantly.
Revised: Beginning with the third sentence, L2W and the LM diverge more signiﬁcantly.

Original: The
Revised: The

Original: LM makes a generic comment: “The location is perfect,” while L2W goes on to list speciﬁc items it enjoyed at breakfast, which was mentioned in the context.
Revised: LM makes a generic comment: “The location is perfect,” while L2W goes on to list speciﬁc items it enjoyed at breakfast, which was mentioned in the context.

Original: Furthermore, the LM becomes “stuck” repeating itself—a common problem for RNNs in general—whereas the L2W system ﬁnds a natural ending.
Revised: Furthermore, the LM becomes “stuck” repeating itself—a common problem for RNNs in general—whereas the L2W system ﬁnds a natural ending.

Original: 5. E RROR A NALYSIS
Revised: 5. E RROR A NALYSIS

Original: The L2W models do not ﬁx every degenerate characteristic of RNNs.
Revised: The L2W models do not ﬁx every degenerate characteristic of RNNs.

Original: The TripAdvisor L2W generation in Table 1, while coherent and diverse, leaves room for improvement.
Revised: The TripAdvisor L2W generation in Table 1, while coherent and diverse, leaves room for improvement.

Original: The need to relate to topical phrases can override ﬂuency, as with the long list of “food, fresh fruit, juices, coffee, tea, hot chocolate.”
Revised: The need to relate to topical phrases can override ﬂuency, as with the long list of “food, fresh fruit, juices, coffee, tea, hot chocolate.”

Original: Furthermore phrases such as “friendly and helpful” and “clean and comfortable” occur in the majority of generations.
Revised: Furthermore phrases such as “friendly and helpful” and “clean and comfortable” occur in the majority of generations.

Original: These phrases, while common, tend to have semantics that are expressed in many different surface forms in real writing (e.g., “the staff were kind and quick to respond” ).
Revised: These phrases, while common, tend to have semantics that are expressed in many different surface forms in real writing (e.g., “the staff were kind and quick to respond” ).

Original: Appendix A presents sample generations from all models on both corpora for further inspection.
Revised: Appendix A presents sample generations from all models on both corpora for further inspection.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Generation with Long-term Context While relatively less studied, there have been several attempts at RNN-based paragraph generation for multiple domains including image captions (Krause et al., 2017), product reviews (Lipton et al., 2015; Dong et al., 2017), sport reports (Wiseman et al., 2017a), and recipes (Kiddon et al., 2016).
Revised: Generation with Long-term Context While relatively less studied, there have been several attempts at RNN-based paragraph generation for multiple domains including image captions (Krause et al., 2017), product reviews (Lipton et al., 2015; Dong et al., 2017), sport reports (Wiseman et al., 2017a), and recipes (Kiddon et al., 2016).

Original: Most of these approaches are based on sequence-tosequence (seq-to-seq) type architectures.
Revised: Most of these approaches are based on sequence-tosequence (seq-to-seq) type architectures.

Original: One effective way to avoid the degenerative characteristics of RNNsunder seq-to-seq is to augment the input with sufﬁcient details and structure that can guide the output generation through attention and a copying mechanism.
Revised: One effective way to avoid the degenerative characteristics of RNNsunder seq-to-seq is to augment the input with sufﬁcient details and structure that can guide the output generation through attention and a copying mechanism.

Original: However, for many NLP generation tasks, it is not easy to obtain such the large-scale training data required to support a robust sequence-to-sequence model.
Revised: However, for many NLP generation tasks, it is not easy to obtain such the large-scale training data required to support a robust sequence-to-sequence model.

Original: For example, a recent dataset for image-to-paragraph generation contains 14,575 training pairs Krause et al.
Revised: For example, a recent dataset for image-to-paragraph generation contains 14,575 training pairs Krause et al.

Original: (2017), and state-of-the-art hierarchical models, while improving over strong baselines, still lead to generation that repeats, contradicts, and is generic.
Revised: (2017), and state-of-the-art hierarchical models, while improving over strong baselines, still lead to generation that repeats, contradicts, and is generic.

Original: Alternative Decoding Objectives
Revised: Alternative Decoding Objectives

Original: The tendency of RNN generation to be short, blend, repetitive and contradictory has been noted multiple times in prior literature.
Revised: The tendency of RNN generation to be short, blend, repetitive and contradictory has been noted multiple times in prior literature.

Original: A number of papers propose approaches that can be categorized as alternative decoding objectives (Shao et al., 2017).
Revised: A number of papers propose approaches that can be categorized as alternative decoding objectives (Shao et al., 2017).

Original: For example,
Revised: For example,

Original: Li et al.
Revised: Li et al.

Original: (2016a) propose a diversity-promoting objective that interpolates the conditional probability score with negative marginal or reverse conditional probabilities.
Revised: (2016a) propose a diversity-promoting objective that interpolates the conditional probability score with negative marginal or reverse conditional probabilities.

Original: Unlike the negative marginal term that blindly penalizes generic generation, all our communication models are contextual in that they compare the context with continuation candidates.
Revised: Unlike the negative marginal term that blindly penalizes generic generation, all our communication models are contextual in that they compare the context with continuation candidates.

Original: Incorporating the reverse conditional probability has been also proposed through the noisy channel models of Yu et al.
Revised: Incorporating the reverse conditional probability has been also proposed through the noisy channel models of Yu et al.

Original: The clear beneﬁt of the reverse conditional probability model is that it can prevent the explaining away problem of the long-term context.
Revised: The clear beneﬁt of the reverse conditional probability model is that it can prevent the explaining away problem of the long-term context.

Original: However, decoding with the reverse conditional probability is signiﬁcantly more expensive, making it impractical for paragraph generation.
Revised: However, decoding with the reverse conditional probability is signiﬁcantly more expensive, making it impractical for paragraph generation.

Original: In addition, both above approaches do presented in our work can be easily integrated into the existing beam search procedure , and each model is lightweight to compute.
Revised: In addition, both above approaches do presented in our work can be easily integrated into the existing beam search procedure, and each model is lightweight to compute.

Original: The modiﬁed decoding objective has long been a common practice in statistical machine translation literature (Koehn et al., 2003;
Revised: The modiﬁed decoding objective has long been a common practice in statistical machine translation literature (Koehn et al., 2003;

Original: Och, 2003; Watanabe et al., 2007; Chiang et al., 2009).
Revised: Och, 2003; Watanabe et al., 2007; Chiang et al., 2009).

Original: Notably, it still remains a common practice with neural machine translation, even with a highly expressive network structure trained on an extremely large amount of data (Wu et al., 2016).
Revised: Notably, it still remains a common practice with neural machine translation, even with a highly expressive network structure trained on an extremely large amount of data (Wu et al., 2016).

Original: Inspired by all above approaches, our work presents a general learning framework together with a more comprehensive set of composite communication models.
Revised: Inspired by all above approaches, our work presents a general learning framework together with a more comprehensive set of composite communication models.

Original: Unlike previous approaches, our composite models are trained to directly address the particular limitations of the base RNN models, by integrating RNN generation into the discriminative learning.
Revised: Unlike previous approaches, our composite models are trained to directly address the particular limitations of the base RNN models, by integrating RNN generation into the discriminative learning.

Original: This allows for customization of the decoding objective to better cope with the undesired behavior of the base language models.
Revised: This allows for customization of the decoding objective to better cope with the undesired behavior of the base language models.

Original: Meta Learning There has been a broad range of literature on meta learning, where the learning goal broadens the search space of learning by targeting to learn model architectures, hyperparameters, and learning algorithms that are typically hand-designed or selected.
Revised: Meta Learning There has been a broad range of literature on meta learning, where the learning goal broadens the search space of learning by targeting to learn model architectures, hyperparameters, and learning algorithms that are typically hand-designed or selected.

Original: Andrychowicz et al.
Revised: Andrychowicz et al.

Original: (2016), for example, learn the learning rate, while Zoph & Le (2016) train a neural network to generate neural network architectures.
Revised: (2016), for example, learn the learning rate, while Zoph & Le (2016) train a neural network to generate neural network architectures.

Original: while Snoek et al.
Revised: while Snoek et al.

Original: (2012) proposed Bayesian optimization to learn hyperparameters that cannot be optimized with gradient descent.
Revised: (2012) proposed Bayesian optimization to learn hyperparameters that cannot be optimized with gradient descent.

Original: The learning framework we presented here can be interpreted as a type of meta learning in a broad sense in that there are multiple layers of learning: learning the base RNN language model, then learning a collection of composite communication models that are customized to the particular behavior of the base model, and ﬁnally the generator that learns to combine all sub-components.
Revised: The learning framework we presented here can be interpreted as a type of meta learning in a broad sense in that there are multiple layers of learning: learning the base RNN language model, then learning a collection of composite communication models that are customized to the particular behavior of the base model, and ﬁnally the generator that learns to combine all sub-components.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Our work presents a uniﬁed learning framework that can learn to generate long text overcoming the limitations of RNNs as text generation models.
Revised: Our work presents a uniﬁed learning framework that can learn to generate long, coherent text overcoming the limitations of RNNs as text generation models.

Original: Our framework learns a decoding objective suitable for generation through a combination of sub-models that capture linguistically-motivated qualities of good writing.
Revised: Our framework learns a decoding objective suitable for generation through a combination of sub-models that capture linguisticallymotivated qualities of good writing.

Original: Our work makes a unique contribution that complements existing literature on long text generation that is predominantly based on seq-to-seq models with a large amount of indomain training data; we demonstrate that the ﬂuency of general RNN language models can be successfully guided to generate more semantically coherent text, which can be useful for domains where in-domain data is not sufﬁcient to support seq-to-seq type training.
Revised: Our work makes a unique contribution that complements existing literature on long text generation that is predominantly based on seq-to-seq models with a large amount of in-domain training data; we demonstrate that the ﬂuency of general RNN language models can be successfully guided to generate more lengthy and sensical text, which can be useful for domains where in-domain data is not sufﬁcient to support seq-to-seq type training.

Original: We propose a general framework for learning a decoding objective in two parts: learning component models to rank candidate generations, which are motivated by different aspects of Grice’s Maxims, and learning a weighing scheme that balances the inﬂuence of each of these scoring functions.
Revised: We propose a general framework for learning a decoding objective in two parts: learning component models to rank candidate generations, which are motivated by different aspects of Grice’s Maxims, and learning a weighing scheme that balances the inﬂuence of each of these scoring functions.

Original: This framework is amenable to any number and kind of sub-objectives, allowing for multiple qualities of good writing to be modeled across diverse domains.
Revised: This framework is amenable to any number and kind of sub-objectives, allowing for multiple qualities of good writing to be modeled across diverse domains.

Original: Human evaluation shows that the quality of the text produced by our model exceeds that of RNN baselines by a large margin and the generations score signiﬁcantly higher on a Turing test evaluation.
Revised: Human evaluation shows that the quality of the text produced by our model exceeds that of RNN baselines by a large margin and the generations score signiﬁcantly higher on a Turing test evaluation.

Original:  E XAMPLE G ENERATIONS
Revised: A E XAMPLE G ENERATIONS

Original: ROCStory Context LanguageModel NoMeta-Learning
Revised: [Table]

Original: NoWorkingVocab Learn2Write Human Jennyisverythirsty.
Revised: gave

Original: Shesuspectssheisdehydrated. Finally, shewasabletoﬁndawaterbottle. She drankthewholething. Thethethethethe
Revised: [Table]

Original: Bottledwaterbottledwaterbottledwaterbottledwaterdrinkingwaterdrinkingsodadrinkingsodabottledwater .Bottledwater bottledwaterbottledwaterdrinkingwater. Bottledwaterbottledwaterbottledwater. Drink Sheendeduphavingacupofcoffeefortherestoftheday.
Revised: She

Original: Thenextday,she wenttothedoctorandgotaﬂushotinthehospital. The doctortoldhershewasallergictopeanutsandwasallergictopeanuts . Shefeltmuchbetterafterdrinkingthewater. Jennywouldalwaysremembertopackawaterbottle.
Revised: [Table]

Original: Ginawasboredindetention. She decidedtotextherfriendmaryacrosstheroom. Sheknew themeanteacherwouldn’tapprove. Shekeptherphonein herpursetohideitfromtheteacher.
Revised: shed

Original: Shewassoexcitedthatshecouldn’tﬁndherphone. Maryaskedhermotherifshecouldaskedherifshewasstudents .Teacher Marywassoupsetthatshecouldn’tﬁndherphoneintheparkinglotoftheschoolbus .Her friendhadtogotothestoretobuyheranewoneforherbirthday .Shewasso
Revised: [Table]

Original: happythatshecouldn’twaitto Marywasrelievedwhenherfriendcameover. Marytextedback. Jackhasabiggarden.
Revised: it

Original: Heweedsiteveryday. Onceaweekhe takesveggiestothemarket. Jackwaters hisgardeninthemorning. Hewassoexcitedthathecouldn’twaittogobacktothestore .
Revised: [Table]

Original: Treesgrowtreestreesandfruittrees [Table] [Table] NoMeta
Revised: We

Original: [Table]
Revised: [Table]

Original: NoWorkingVocab Learn2Write Human
Revised: She

Original: [Table]
Revised: [Table]

Original: :) thehotelprovidedagoodbreakfastandthereweresomegreatrestaurantsinthearea [Table] [Table] eveningwewereabletowatchtheﬁreworksfromourbalconyandwatchthesunsetfromourwindow
Revised: The hotel staff were very kind and helpful.

Original: [Table] familyroomthatdidn’thavea
Revised: Our room was large but a little sparse.

Original: [Table] needed [Table]
Revised: The garden is really nice in the courtyard.

Original: walkfromthebusstationor1minwalkfromthetrainstationandlessthana100mfromthecanalgrande
Revised: The location is perfect, next door to the station

Original: [Table]
Revised: [Table]

Original: factthatit’satthestartofthecitysothatyoubeginyourveniceexperienceslowly .Irecommend walkingtothesanmarcopiazzaonyourﬁrsttime(makesure youwearcomfortableshoes). Thehotelisinagreatlocation. Theroom wascleanandcomfortable.
Revised: If you want to be in the middle of the action, this is not the place for you.

Original: Thestaffwasvery friendlyandhelpful.

Original: Thebreakfastwasverygood.
Revised: If you want to be in the middle of the action, this is not the place for you. If you want [Table] But

Revised: [Table]

Revised: is a great place to

Revised: [Table]

Revised: [Table]

Revised: There are no stairs to the hotel but quite a few in the hotel.

Revised: Restaurants right outside the door.

Revised: The only problem would be for people who are not early risers, the church bells go early ( about 7am ) are loud and go on for a while ( a few minutes ).

Revised: We used it as an alarm, which was ﬁne for us but some may not like it.

Original: Thestaff wasveryfriendlyandhelpful.
Revised: The breakfast was a little adverage but no un-eatable [Table] [Table] multiple

Original: Iwould recommendthishoteltoanyonewhoislookingforaniceplacetostayinvenice .Iwould
Revised: [Table] liki tiki village i was a bit nervous about stay- ing there. We loved every minute. I wish that we had more time to enjoy the

Original: recommendthishoteltoanyonewhoislookingforaplacetostayinvenice .I wouldrecommendthishoteltoanyonewhoislookingforaplacetostaywhilein
Revised: [Table] , but we were visiting the parks most of the time. The rooms were clean and the kitchen well stocked

Original: Thestaffwereveryfriendlyandhelpfulandthebreakfastbuffet
Revised: [Table] staff was very friendly and helpful. I would recommend this hotel to anyone who is looking for a nice place to stay in the heart of the city. I would highly recommend this hotel to anyone who is looking for a place to stay in san diego.

Original: Theroomwascleanandthebedwascomfortable.
Revised: I would highly recommend this hotel to anyone who is looking for a place to stay in san diego. I would highly recommend this hotel to anyone who is looking for a place to stay in san diego. If you [Table]

Original: Theroomsare spaciousandclean. Breakfastisverygood.
Revised: Our [Table] The rooms were clean and the kitchen well stocked.

Original: Thenyouexitthealleyandfacethemagniﬁcentsanmarcopiazzaand trustme,youwillbepleasantlyshocked.
Revised: The staff were very friendly and helpfu! I would deﬁnately recommend this hotel to anyone who is looking for a place to stay in the heart of san diego.

Original: Whatanexperience!!!:))the abbaziaroomsaredecentandveryclean.
Revised: Thank you to all of the staff for making our stay so enjoyable! Human

Original: Breakfastis poorbutadequateandwi-ﬁisfree.
Revised: We did go to the time share presentation but we left almost immediately since we are owners elsewhere. I guess they ﬁgured out that they would be wasting their time. Great to have a washer and dryer. Also great to have a seperate room for the kids.

Original: Thegardenisvery peacefulandofferssomeveryrelaxingmoments.
Revised: The grounds were clean and we felt safe.

Original: Iwas worriedaboutnoisesfromthetrainstationnextdoorbutyoucan’thearathingsonoproblemthere .
Revised: Checkin was a little confusing since they gave us a map of the property and then wrote all over it in black pen so that you could not see the map.

Original: Theguysatthereceptionareamazing.
Revised: Overall it was a great experience.

Original: Veryfriendlyandvery helpful:))whatyou wantfromahotelinveniceisadecentplacetosleep,havearelaxingbathand somebreakfastinthemorning.
Revised: Close to the grocery store and to target. A little difﬁcult to ﬁnd but the gps got us there. Great for the price and amenities. [Table]

Original: Fromthen onyouwillbespendingallyour timeintownanywaysofometheabbaziahotelwasanexcellentchoiceandiwillgobackforsure .Priceisnotcheap,butnothingischeap inveniceanyway.
Revised: [Table] ﬁ [Table] [Table]

Original: B M ODEL DETAILS AND HYPERPARAMETERS
Revised: will [Table] fo [Table]

Revised: was

Revised: [Table]

Revised: will

Revised: [Table]

Revised: sure

Revised: [Table]

Revised: B M ODEL DETAILS AND HYPERPARAMETERS

Original: Base language model We use a 2-layer RNN language model with 1024 GRU cells per layer.
Revised: Base language model We use a 2-layer RNN language model with 1024 GRU cells per layer.

Original: Embeddings vectors are of length 1024.
Revised: Embeddings vectors are of length 1024.

Original: Following Inan et al.
Revised: Following Inan et al.

Original: (2017) we tie the input and output embedding layers’ parameters.
Revised: (2017) we tie the input and output embedding layers’ parameters.

Original: To regularize we dropout (Srivastava et al., 2014) cells in the output layer of the ﬁrst layer with probability 0.2.
Revised: To regularize we dropout (Srivastava et al., 2014) cells in the output layer of the ﬁrst layer with probability 0.2.

Original: We use mini-batch stochastic gradient descent (SGD) and anneal the learning rate regularly when the validation set performance fails to improve.
Revised: We use mini-batch stochastic gradient descent (SGD) and anneal the learning rate regularly when the validation set performance fails to improve.

Original: Learning rate, annealing rate, and batch size were tuned on the validation set for each dataset, with details in section 4.4.
Revised: Learning rate, annealing rate, and batch size were tuned on the validation set for each dataset, with details in section 4.4.

Original: Gradients are backpropagated 35 time steps and clipped to a maximum value of 0.25.
Revised: Gradients are backpropagated 35 time steps and clipped to a maximum value of 0.25.

Original: Entailment model Dropout is performed on both the input word embeddings and the MLP hidden layer with rate 0 .
Revised: Entailment model Dropout is performed on both the input word embeddings and the MLP hidden layer with rate 0 .

Original: Training is performed with Adam (Kingma & Ba, 2015) with learning rate 0 .
Revised: Training is performed with Adam (Kingma & Ba, 2015) with learning rate 0 .

Original: 0005 , batch size 128 .
Revised: 0005 , batch size 128 .

Original: Relevance model
Revised: Relevance model

Original: The convolutional layer is a 1D convolution with ﬁlter size 3 and stride 1 ; the sequences are padded so that the sequence output is the same length as the input.
Revised: The convolutional layer is a 1D convolution with ﬁlter size 3 and stride 1 ; the sequences are padded so that the sequence output is the same length as the input.

Original: The model is trained with Adam, learning rate 0 .
Revised: The model is trained with Adam, learning rate 0 .

Original: 001 and dropout is applied before the ﬁnal linear layer with rate 0 .
Revised: 001 and dropout is applied before the ﬁnal linear layer with rate 0 .

Original: Meta-weight learning Training is performed with SGD with a learning rate of 1.
Revised: Meta-weight learning Training is performed with SGD with a learning rate of 1.