Original: L EARNING TO W RITE BY L EARNING THE O BJECTIVE
Revised: L EARNING TO W RITE BY L EARNING THE O BJECTIVE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recurrent Neural Networks (RNNs) are powerful autoregressive sequence models for learning prevalent patterns in natural language.
Revised: Recurrent Neural Networks (RNNs) are powerful autoregressive sequence models for learning prevalent patterns in natural language.

Original: Yet language generated by RNNs often shows several degenerate characteristics that are uncommon in human language; while ﬂuent, RNN language production can be overly generic, repetitive, and even self-contradictory.
Revised: Yet language generated by RNNs often shows several degenerate characteristics that are uncommon in human language; while ﬂuent, RNN language production can be overly generic, repetitive, and even self-contradictory.

Original: We postulate that the objective function optimized by RNN language models, which amounts to the overall perplexity of a text, is not expressive enough to capture the abstract qualities of good generation such as Grice’s Maxims.
Revised: We postulate that the objective function optimized by RNN language models, which amounts to the overall perplexity of a text, is not expressive enough to capture the abstract qualities of good generation such as Grice’s Maxims.

Original: In this paper, we introduce a general learning framework that can construct a decoding objective better suited for generation.
Revised: In this paper, we introduce a general learning framework that can construct a decoding objective better suited for generation.

Original: Starting with a generatively trained RNN language model, our framework learns to construct a substantially stronger generator by combining several discriminatively trained models that can collectively address the limitations of RNN generation.
Revised: Starting with a generatively trained RNN language model, our framework learns to construct a substantially stronger generator by combining several discriminatively trained models that can collectively address the limitations of RNN generation.

Original: Human evaluation demonstrates that text generated by the resulting generator is preferred over that of baselines by a large margin and signiﬁcantly enhances the overall coherence, style, and information content of the generated text.
Revised: Human evaluation demonstrates that text generated by the resulting system is preferred over that of baselines by a large margin and signiﬁcantly enhances the overall coherence, style, and information content of the generated text.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recurrent Neural Network (RNN) based language models such as Long Short-Term Memory Networks (LSTMs) (Hochreiter & Schmidhuber, 1997) and Gated Recurrent Units (GRUs) (Cho et al., 2014) have achieved enormous success across a variety of language tasks due to their ability to learn ﬂuency patterns in natural language (Jozefowicz et al., 2016;
Revised: Recurrent Neural Network (RNN) based language models such as Long Short-Term Memory Networks (LSTMs) (Hochreiter & Schmidhuber, 1997) and Gated Recurrent Units (GRUs) (Cho et al., 2014) have achieved enormous success across a variety of language tasks due to their ability to learn ﬂuency patterns in natural language (Jozefowicz et al., 2016;

Original: Kim et al., 2016; Mikolov et al., 2010).
Revised: Kim et al., 2016; Mikolov et al., 2010).

Original: When used as a generator, however, the quality of language generated from RNNs deviates drastically from that of human language.
Revised: When used as a generator, however, the quality of language generated from RNNs deviates drastically from that of human language.

Original: While ﬂuent, RNN-produced language displays several degenerative characteristics, favoring generic and contentless output and being easily repetitive and self-contradictory.
Revised: While ﬂuent, RNN-produced language displays several degenerate characteristics, favoring generic and contentless output that tends to be repetitive and self-contradictory.

Original: These issues are especially prominent when RNNs are used for open-ended text generation dealing with long-term context, as illustrated in Figure 1.
Revised: These issues are especially prominent when RNNs are used for open-ended, long-form text generation, as illustrated in Figure 1.

Original: RNNs model the conditional probability P ( x t | x 1 ,...,t − 1 ) of generating the next word x t given all previous words observed or generated.
Revised: RNNs model the conditional probability P ( x t | x 1 , ..., x t − 1 ) of generating the next word x t given all previous words observed or generated.

Original: In theory, this conditional model should be able to learn all crucial aspects of human language production, for example, that we don’t normally repeat the same content over and over.
Revised: In theory, this conditional model should be able to learn all crucial aspects of human language production, for example, that we don’t normally repeat the same content over and over.

Original: In practice, however, the learned conditional probability model often assigns higher probability to a repetitive, overly generic sentence than to higher quality sentences, as shown in Figure 1.
Revised: In practice, however, the learned conditional probability model often assigns higher probability to a repetitive, overly generic sentence than to higher quality sentences, as shown in Figure 1.

Original: We postulate that this is in part because the network architectures of RNN variants do not provide a strong enough inductive bias for the model to learn the complex communication goals pursued in human writing.
Revised: We postulate that this is in part because the network architectures of RNN variants do not provide a strong enough inductive bias for the model to learn the complex communication goals pursued in human writing.

Original: In addition, long-term context easily gets lost as it is explained away in the presence of more immediately relevant short-term context (Yu et al., 2017), and as gradients diminish over a long sequence (Pascanu et al., 2013).
Revised: In addition, long-term context easily gets lost as it is explained away in the presence of more immediately relevant short-term context (Yu et al., 2017), and as gradients diminish over a long sequence (Pascanu et al., 2013).

Original: Consequently, RNNs acquire relatively shallow and myopic patterns, and the corresponding probability score P ( x t | x 1 ,...,t − 1 ) is also insufﬁcient for generating language that matches the complexity and coherence of human generated text.
Revised: Consequently, RNNs acquire relatively shallow and myopic patterns, which tend to only take advantage of a small fraction of the training set vocabulary Kiddon et al. RNNs are thus unable to generate language that matches the complexity and coherence of human generated text.

Original: Several methods in the literature attempt to mitigate these issues.
Revised: Several methods in the literature attempt to mitigate these issues.

Original: Overly simple and generic generation can be improved byand using a diversity-boosting objective function (Shao et al., 2017; Vijayakumar et al., 2016).
Revised: Overly simple and generic generation can be improved by using a diversity-boosting objective function (Shao et al., 2017; Vijayakumar et al., 2016).

Original: Repetitive generation can be reduced by prohibiting recurrence of the same trigrams as  hard rules (Paulus et al., 2017), although such constraints are only a partial solution as theycannot stop RNNs from repeating through paraphrasing.
Revised: Repetitive generation can be reduced by prohibiting recurrence of the same trigrams as a hard rule (Paulus et al., 2017). Although such constraints form a partial solution, they

Original: All in all, I would highly recommend this hotel to anyone who wants to be in the heart of the action, and want to be in the heart of the action.
Revised: All in all, I would highly recommend this hotel to anyone who wants to be in the heart of the action, and want to be in the heart of the action.

Original: If you want to be in the heart of the action, this is not the place for you.
Revised: If you want to be in the heart of the action, this is not the place for you.

Original: However, If you want to be in the middle of the action, this is the place to be.
Revised: However, If you want to be in the middle of the action, this is the place to be.

Original: A common thread across all these methods is to seek a new objective function that is better suited for generation. Since it is not straightforward to write the perfect objective function that can handle multiple abstract properties of good generation, we instead propose a general learning framework to construct a better decoding objective by learning.
Revised: Hand tailoring rules is both time consuming and unstable across different generative scenarios, so we instead propose a general learning framework to construct a better decoding objective.

Original: Starting with a generatively trained RNN language model, our framework learns to construct a substantially stronger generator by combining several discriminatively trained models that can collectively address limitations of the base RNN generator.
Revised: Starting with a generatively trained RNN language model, our framework learns to construct a substantially stronger generator by combining several discriminatively trained models that can collectively address limitations of the base RNN generator.

Original: Our learning framework therefore generalizes over various existing modiﬁcations to the decoding objective.
Revised: Our learning framework therefore generalizes over various existing modiﬁcations to the decoding objective.

Original: In addition, our approach learns to overcome the particular limitations of the RNN generator directly by incorporating language generated from RNNs as negative samples to discriminatively train several companion models, each specializing  different aspects of Grice’s Maxims of communication.
Revised: Our approach learns to overcome the particular limitations of the RNN generator directly by incorporating language generated from RNNs as negative samples to discriminatively train several companion models, each specializing in a different aspect of Grice’s Maxims of communication (Grice et al.

Original: Empirical results demonstrate that our learning framework is highly effective in converting a generic
Revised: Empirical results demonstrate that our learning framework is highly effective in converting a generic

Original: RNN language model into a substantially stronger generator.
Revised: RNN language model into a substantially stronger generator.

Original: Human evaluation conﬁrms that language generated by our model is preferred over that of competitive baselines by a large margin and signiﬁcantly enhances the overall coherence, style, and information content of the generated text.
Revised: Human evaluation conﬁrms that language generated by our model is preferred over that of competitive baselines by a large margin and signiﬁcantly enhances the overall coherence, style, and information content of the generated text.

Original:  B ACKGROUND : G RICE ’ S M AXIMS
Revised:  B ACKGROUND : G RICE ’ S M AXIMS

Original: We motivate our learning framework using Grice’s Maxims of communication (Grice et al., 1975): 
Revised: We motivate our learning framework using Grice’s Maxims of communication (Grice et al., 1975): 1.

Original: Quantity: Make your contribution as informative as required, and no more than necessary.
Revised: Quantity: Make your contribution as informative as required, and no more than necessary.

Original: RNN generation tends to violate this maxim as it is often overly short and generic,  thereforeis less informative than desired.
Revised: RNN generations tend to violate this maxim as they are often overly short and generic, and therefore less informative than desired.

Original: When encouraged to generate longer text, RNNs easily repeat themselves, which also works against conveying the right amount of information.
Revised: When encouraged to generate longer text, RNNs easily repeat themselves, which also works against conveying the right amount of information.

Original: This observation motivates the length andthe repetition models in § .2.1 and § 3.2.2.
Revised: This observation motivates the length and repetition models in § 3.2.1 and § 3.2.2.

Original: Quality: Do not say what you believe to be false.
Revised: Quality: Do not say what you believe to be false.

Original: When used for generation with long term context, RNNs can generate text that is self-contradictory.
Revised: When used for generation with long term context, RNNs often generate text that is self-contradictory.

Original: We propose the entailment model to address this problem in § 3.2.3.
Revised: We propose an entailment model to address this problem in § 3.2.3.

Original: Relation: Be relevant.
Revised: Relation: Be relevant.

Original: RNNs used for long generation can start digressing as the long-term context gets easily washed out.
Revised: RNNs used for long generation can start digressing as the long-term context gets easily washed out.

Original: We address this problem by proposing two different relevance models in § 3.2.4 and § 3.2.5.
Revised: We address this problem by proposing two different relevance models in § 3.2.4 and § 3.2.5.

Original: Manner Avoid obscurity of expression.
Revised: Manner: Avoid obscurity of expression.

Original: Avoid ambiguity.
Revised: Avoid ambiguity.

Original: Be brief.
Revised: Be brief.

Original: Be orderly.
Revised: Be orderly.

Original: As RNN generation favors highly probable but generic words and phrases, it lacks adequate style and speciﬁcity.
Revised: As RNN generation favors highly probable but generic words and phrases, it lacks adequate style and speciﬁcity.

Original: We address this issue with the lexical style model in § 3.2.6.
Revised: We address this issue with the lexical style model in § 3.2.6.

Original:  T HE L EARNING F RAMEWORK
Revised:  T HE L EARNING F RAMEWORK

Original: We propose a general learning framework for conditional language generation of sequence x given context y .
Revised: We propose a general learning framework for conditional language generation of sequence y given context x .

Original: The decoding objective for generation takes the general form:
Revised: The decoding objective for generation takes the general form:

Original: [Equation]
Revised: [Equation]

Original: This objective combines the RNN  probability  ( § 3.1) with a set of additional scores s k ( x , y ) taken from discriminatively trained communication models ( § 3.2).
Revised: This objective combines the RNN language model probability P lm ( § 3.1) with a set of additional scores s k ( x , y ) produced by discriminatively trained communication models ( § 3.2) and weighted with learned mixture coefﬁcients λ k ( § 3.3).

Original: This corresponds to a Product of Experts (PoE) model (Hinton, 2006) when the scores s k are log probabilities.
Revised: This corresponds to a Product of Experts (PoE) model (Hinton, 2006) when the scores s k are log probabilities.

Revised: Further model and hyperparameter details are given in appendix B.

Revised: Generation is performed using beam search ( § 3.4), scoring partially generated candidate generations y 1: i at each time step i .

Original: The  language model decomposes into per-word probabilities using the chain rule applied over RNNs.
Revised: The RNN language model decomposes into per-word probabilities using the chain rule.

Original: However, in order to allow for more expressive learning over long range context we do not require the discriminative model scores to factorize over the elements of x , thereby addressing a key limitation of RNNs.
Revised: However, in order to allow for more expressivity over long range context we do not require the discriminative model scores to factorize over the elements of y , thereby addressing a key limitation of RNNs.

Original: More speciﬁcally, we use an estimated score s (cid:48) k
Revised: More speciﬁcally, we use an estimated score s (cid:48) k

Original: ( x1: i , y  ) that can be computed for any preﬁx of x = x 1: n to approximate the objective during beam search, such that s (cid:48) k ( x1: n , y  ) =
Revised: ( x , y 1: i ) that can be computed for any preﬁx of y = y 1: n to approximate the objective during beam search, such that s (cid:48) k ( x , y 1: n ) =

Original: s k
Revised: s k

Original: ( x , y ) .
Revised: ( x , y ) .

Original: The scoring functions aretypically trained on preﬁxes of x to simulate their application to rank partial continuations during beam search.
Revised: The scoring functions are trained on preﬁxes of y (except where stated otherwise) to simulate their application to rank partial continuations during beam search.

Original: Further model and hyperparameter details are given in appendix B.

Original: 3. B ASE L ANGUAGE MODEL
Revised: 3. B ASE L ANGUAGE MODEL

Original: We train a RNN language model to estimate
Revised: We train an RNN language model to estimate

Original: [Equation]
Revised: [Equation]

Original: The language model does not distinguish between the context y and the continuation, but treats everything as a single sequence x .
Revised: The language model treats the context x and the continuation y as a single sequence s , in contrast to sequence to sequence models which distinguish between them.

Original: 3. C OMPOSITE C OMMUNICATION M ODELS
Revised: 3. C OMPOSITE C OMMUNICATION M ODELS

Original: Next we introduce a set of modelsthat are motivated by Grice’s Maxims of communication.
Revised: Next we introduce a set of models motivated by Grice’s Maxims of communication.

Original: Each model is trained to discriminate between good and bad generation using a ranking objective, so that its model scores have the effect of re-ranking in the decoder.
Revised: Each model is trained to discriminate between good and bad generation using a ranking objective, so that its model scores have the effect of re-ranking in the decoder.

Original: We vary the model parameterization and training examples to guide each model to focus on different aspects of Grice’s Maxims.
Revised: We vary the model parameterization and training examples to guide each model to focus on different aspects of Grice’s Maxims.

Original: The classiﬁer scores are interpreted as classiﬁcation probabilities and added to the objective function as log probabilities.
Revised: The classiﬁer scores are interpreted as classiﬁcation probabilities and added to the objective function as log probabilities.

Original: Let D =
Revised: Let D =

Original: ( x 1 , y 1 ) , . . .
Revised: ( x 1 , y 1 ) , . . .

Original: ( x n , y n ) } be the set ofconditional generation training examples.
Revised: ( x n , y n ) } be the set of training examples for conditional generation.

Revised: D x denote all contexts and D y all continuations.

Original: In all models the ﬁrst layer embeds words into 300-dimensional vectors initialized with GloVe (Pennington et al., 2014) embeddings, which are ﬁne-tuned during training unless stated otherwise.
Revised: In all models the ﬁrst layer embeds words into 300-dimensional vectors initialized with GloVe (Pennington et al., 2014) embeddings, which are ﬁne-tuned during training unless stated otherwise.

Original: The dimensions of hidden layers are also 300 .
Revised: The dimensions of hidden layers are also 300 .

Original: Let e ( x i ) be the word embedding of word x i .
Revised: Let e ( w ) be the word embedding of word w .

Original: 3.2. L ENGTH M ODEL
Revised: 3.2. L ENGTH M ODEL

Original: RNNs tend to bias toward shorter generation even with a highly expressive network structure with attention, thus length normalization is still a common practice (Wu et al., 2016).
Revised: RNNs tend to bias toward shorter generation even with a highly expressive network structure with attention, thus length normalization is still a common practice (Wu et al., 2016).

Original: We use a geometrically decaying length reward.
Revised: We use a geometrically decaying length reward.

Original: We tune the initial value and decaying rate on the ﬁnal systems, and use the same value for all systems with the initial value and decay rate being 1 and 0 .
Revised: We tune the initial value and decaying rate on the ﬁnal systems, and use the same value for all systems with the initial value and decay rate being 1 and 0 .

Original: respectively.
Revised: respectively.

Revised: The score per a time step (for a partially generated completion) is thus

Revised: [Equation]

Original: 3.2. R EPETITION M ODEL
Revised: [Equation] R EPETITION M ODEL

Original: The goal of the repetition model is to  distinguish between RNN generated continuation and gold continuations by detecting repetitions  in the generated continuation.
Revised: The goal of this model is to learn to distinguish between RNN-generated and gold continuations by exploiting our empirical observation that repetitions are more common in completions generated by the RNN language model.

Revised: We quantify the claim that samples from the base RNN is an appropriate source of negative examples for repetition detection on our datasets in § 4.1.

Revised: However, we do not want to completely discourage repetition, as words sometimes do recur in English.

Revised: Thus we model natural levels of repetition as follows.

Original: First adistance score d i is computed for each token in the continuation x , based on pairwise cosine similarity, as
Revised: First a score d i is computed for each token in the continuation y based on pairwise cosine similarity of word embeddings within a ﬁxed window of the previous k words:

Original: [Equation]
Revised: [Equation]

Original: Therefore d i = 1 for all positions at which  words are repeated.
Revised: This score can be interpreted as the repetition “strength” of the sequence at position i , where d i =for all positions at which a word is repeated within k + 1 words of its previous use.

Original: The score  is then computed as
Revised: The score of the entire continuation is then deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where RNN ( d ) is the ﬁnal state of a unidirectional RNN ran over the similarity scores.
Revised: where RNN ( d ) is the ﬁnal state of a unidirectional RNN ran over the similarity scores and w r is a learned vector.

Original: The model is trained to maximize binary cross-entropy, with gold continuations as positive examples and samples from the base RNNs, which often contain some repetition, as negative examples:
Revised: The model is trained to maximize binary cross-entropy, with gold continuations as positive examples and samples from the base RNN as negative examples:

Original: [Equation]
Revised: [Equation]

Original: The word embeddings are kept ﬁxed during training for this model.
Revised: Word embeddings are kept ﬁxed during training for this model.

Original: 3.2. E NTAILMENT M ODEL
Revised: 3.2. E NTAILMENT M ODEL

Original: The goal of the entailment model is to guide the generator to neither contradict its own past generation (the maxim of Quality) nor state an obvious entailment that readily follows from the context (the maxim of Quantity).
Revised: The goal of the entailment model is to guide the generator to neither contradict its own past generation (the maxim of Quality) nor state something that readily follows from the context (the maxim of Quantity).

Original: The unwanted entailment cases include repetitions and paraphrasing.
Revised: The latter case is driven by the RNNs habit of paraphrasing itself (alongside its tendency for more direct repetitions, which is captured by the repetition model).

Original: We train a classiﬁer using the MultiSNLI dataset (Williams et al., 2017) that takes two sentences a and b as input and predicts the relation between them as either contradiction , entailment or neutral .
Revised: We train a classiﬁer using the MultiSNLI dataset (Williams et al., 2017) that takes two sentences a and b as input and predicts the relation between them as either contradiction , entailment or neutral .

Original: We use the score of the neutral class probability of the sentence pair  to discourage both contradiction and entailment.
Revised: We use the score of the neutral class probability of the sentence pair in order to discourage both contradiction and entailment.

Original: We use a continuous bag-of-words model similar to previously reported NLI baselines (Mou et al., 2016).
Revised: We use a continuous bag-of-words model similar to previously reported NLI baselines (Mou et al., 2016).

Original: Sentence representations are obtained by summing word embeddings (which are ﬁne-tuned during training), such that r ( a ) =
Revised: Sentence representations are obtained by summing word embeddings (which are ﬁne-tuned during training), such that r ( a ) =

Original: (cid:80) i e ( a i ) and r ( b ) =
Revised: (cid:80) i e ( a i ) and r ( b ) =

Original: (cid:80) i e ( b i ) .
Revised: (cid:80) i e ( b i ) .

Original: An MLP classiﬁer with a single tanh hidden layer takes as input concatenations of the two sentence embeddings, along with their element-wise difference and multiplication; the output is a three-way softmax.
Revised: An MLP classiﬁer with a single tanh hidden layer takes as input concatenations of the two sentence embeddings, along with their element-wise difference and multiplication; the output is a three-way softmax.

Revised: The log probability of the neural class is t ( a , b ) .

Original: The classiﬁer obtains 63% validation accuracy.
Revised: The classiﬁer obtains 63% validation accuracy.

Original: In contrast to  other communication models, this classiﬁer cannot be applied directly to the full context and continuation  it is scoring.
Revised: In contrast to our other communication models, this classiﬁer cannot be applied directly to the full context and continuation sequences it is scoring.

Original: Instead every completed sentence in the continuation should be scored against all preceding sentences in both the context and continuation.
Revised: Instead every completed sentence in the continuation should be scored against all preceding sentences in both the context and continuation.

Original: Let S ( x ) be the list of complete sentences in x , and t ( a , b ) the log probability of the neutral class.
Revised: Let S ( y 1: i ) be the set of complete sentences in y 1: i , and S − 1 ( y 1: i ) the last complete sentence.

Original: The entailment score  is computed as
Revised: We compute the entailment score of S − 1 ( y 1: i ) against all preceding sentences in x and y , and use the score of the sentence-pair for which we have the least conﬁdence that entailment is neutral:

Original: [Equation]
Revised: [Equation]

Original: that is, the score  of the sentence-pair for which we have the least conﬁdence that entailment is neutral. To enable fair comparison across beam hypotheses with different sentence lengths, s (cid:48) entail does not accumulate scores across sentences; rather assume that the effect of the last score will already be reﬂected in the content of the beam when the next sentence is scored. While the ﬁrst sentence is generated, s (cid:48) entail takes an initial constant value to avoid bias towards incomplete sentences.
Revised: In contrast to our other models, the score this model returns for any given continuation only corresponds to a subsequence of the continuation, but as we will see below ( § 3.4) due to the way generation is performed with beam search this score will not be accumulated across sentences.

Original: 3.2. R ELEVANCE M ODEL
Revised: 3.2. R ELEVANCE M ODEL

Original: The purpose of the relevance model is to predict whether the content of a candidate continuation is relevant to the given context.
Revised: The purpose of the relevance model is to predict whether the content of a candidate continuation is relevant to the given context.

Original: We train the model to distinguish between true continuations and random continuations sampled from other (human-written) endings in the corpus, conditioned on the given context.
Revised: We train the model to distinguish between true continuations and random continuations sampled from other (human-written) endings in the corpus, conditioned on the given context.

Original: A single convolutional layer is applied to both the context and candidate embedding sequences.
Revised: A single convolutional layer is applied to both the context and candidate embedding sequences.

Original: The scoring function is deﬁned as
Revised: The scoring function is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where 1D maxpooling is performed over each sequence to obtain a vector representing its most important semantic dimensions.
Revised: where 1D maxpooling is performed over each sequence to obtain a vector representing its most important semantic dimensions.

Original: Element-wise multiplication of the context and continuation vectors will amplify similarities between the two.
Revised: Element-wise multiplication of the context and continuation vectors will amplify similarities between the two.

Original: We optimize the following ranking log likelihood:
Revised: We optimize the following ranking log likelihood:

Original: [Equation]
Revised: [Equation]

Original: where x t and x r are true and random endings, respectively.
Revised: i.e., the probability of the true ending y t receiving a higher score than the random ending y r .

Original: For each context and true continuation we extract  5 randomly selected endings as  training examples.
Revised: For each context and true continuation we extract k = 5 randomly selected endings as negative training examples.

Original: The model ranks true endings higher than randomly selected ones with 85% accuracy on the validation set.
Revised: The model ranks true endings higher than randomly selected ones with 85% accuracy on the validation set.

Original: The trained scores do not correspond to probabilities but we scale with the logistic (sigmoid) function for compatibility.
Revised: The trained scores do not correspond to probabilities but we scale with the logistic (sigmoid) function for compatibility with other scoring modules.

Original: 3.2. W ORKING V OCABULARY M ODEL
Revised: 3.2. W ORKING V OCABULARY M ODEL

Original: Given even a fragment of context (e.g. “We were excited about our beach getaway...”)
Revised: Given even a fragment of context (e.g. “We were excited about our beach getaway...”)

Original: certain topics become more relevant (e.g. “swimming”, “relax”, “romantic”).
Revised: certain topics become more relevant (e.g. “swimming”, “relax”, “romantic”).

Original: To capture this intuition, we predict a centroid in the embedding space from the context, which describes a neighborhood of the Data: context y =
Revised: To capture this intuition, we predict a centroid in the embedding space from the context, which describes a neighborhood of the embedding space we expect the continuation to intersect.

Original: y 1 ··· y n , beam size k , meta-weights λ Result: best continuation centroid

Original: = working vocab( y ) (cid:46) used to compute topicality scores best =

Original: None beam =

Original: [ y ] for step = 0; step < max steps; step = step +1 do next beam =

Original: [] for candidate in beam do expand(next beam, next k(candidate, centroid, λ ))

Original: (cid:46) mix LM and topicality scores if lookahead score(candidate.append(termination token))

Original: > best.score then best = candidate.append(term) end endfor candidate in next beam do candidate.score += f λ ( candidate ) (cid:46) score with additional models end beam =

Original: sort(next beam, key=score)[:k] (cid:46) select top k candidates endif learning then update lambda with gradient descent by comparing beam against the gold end return best embedding space we expect the continuation to intersect.

Original: The score is computed as
Revised: The score is computed as

Original: [Equation]
Revised: [Equation]

Original: where RNN ( x ) is the ﬁnal state of the RNN. The model is trained  with  a discriminative loss:
Revised: where RNN ( x ) is the ﬁnal state of an RNN trained end-to-end with s voc using a discriminative loss:

Original: [Equation]
Revised: [Equation]

Revised: During decoding, this score is combined with the language model before sampling next word candidates, while the other models are used to rescore candidates (see § 3.4).

Revised: The reason for this is that this model improves diversity in the next word distribution, while encouraging continuation of topics in the context.

Original: 3.2. L EXICAL S TYLE M ODEL
Revised: 3.2. L EXICAL S TYLE M ODEL

Original: The goal of the lexical style model is to learnthe stylistic aspects ofthe desirable writing based on the lexical distributions.
Revised: The goal of the lexical style model is to learn stylistic aspects of desirable writing based on observed lexical distributions.

Original: The scoring function is deﬁned as
Revised: The scoring function is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The model is trained with a similar ranking criteria as the relevance model, except that the negative examples are sampled from the language model:
Revised: The model is trained with a similar ranking criteria as the relevance model, except that the negative examples are sampled from the language model:

Original: [Equation]
Revised: [Equation]

Original: 3. B EAM SEARCH AND META - WEIGHT LEARNING
Revised: 3. M IXTURE W EIGHT L EARNING

Original: Once all the sub-objectives have been trained, weneed to learn the combined objective.
Revised: Once all the communication models have been trained, we learn the combined objective.

Original: We learn the weight coefﬁcients λ k in equation 1 to linearly combine the scoring functions, using a discriminative objectiveto rank true continuations above continuations found during beam-search:
Revised: In particular we learn the weight coefﬁcients λ k in equation 1 to linearly combine the scoring functions, using a discriminative objective

Original: [Equation]
Revised: [Equation]

Original: where A is our inference algorithm, shown in Algorithm 1.
Revised: where A is the inference algorithm for beam search decoding.

Revised: The objective learns to rank the gold continuation above the continuation predicted by the current model.

Revised: The full model is therefore trained discriminatively to classify sequences, and the same model is used to generate the sequences.

Original: For each training iteration beam-search is performed based on the current values of λ , meaning the  the objective function changes dynamically, leading to a form of meta-weight learning.
Revised: For each training iteration inference is performed based on the current values of λ . This has the effect that the objective function changes dynamically during training: As the current samples from the model are used to update the mixture weights, it creates its own learning signal by applying the generative model discriminatively.

Original: [Table]
Revised: [Table]

Original: E XPERIMENTS 4. T HE TROUBLE WITH AUTOMATIC EVALUATION
Revised: 3. B EAM S EARCH

Revised: Due to the limitations of greedy decoding and the fact that our scoring functions do not decompose across time steps, we perform generation with a beam search procedure, shown in Algorithm 1. The naive approach would be to perform beam search based only on the language model, and then rescore the k best candidate completions with our full model. We found that this approach leads to limited diversity in the beam and therefore cannot exploit the strengths of the full model.

Revised: Therefore we rescore the current hypotheses in the beam with the full (partial) objective function and keep the k best candidate sequences after expanding to the next word.

Revised: To further increase diversity we sample k candidate next words from the distribution when expanding a hypothesis, instead of obtaining the k highest scoring next words.

Original: Previous work has reported that automatic measures such as BLEU, ROUGE, and Meteor, do not lead to meaningful evaluation when used for long or creative text generation where there can be high variance among correct generation output (Wiseman et al., 2017b; Vedantam et al., 2015).
Revised: In our experiments we generate text using a beam size of 8. Two modules are handled differently during beam search: First, the working vocabulary score is integrated before sampling next word candidates, i.e. inside the next k call. Second, the entailment score of a candidate is recomputed only when sentence-terminating punctuation is generated. Otherwise the current entailment score is re-used.

Revised: Due to the nature of beam-search the entailment score is not accumulated across sentences; rather we assume that the effect of the last sentence score will already be reﬂected in the content of the beam when the next sentence is scored.

Revised: While the ﬁrst sentence is generated, s (cid:48) entail takes an initial constant value to avoid bias towards incomplete sentences.

Revised:  E XPERIMENTS 4. E XPERIMENTAL S ETUP

Revised: Corpora We use two corpora for evaluation.

Revised: The ﬁrst is TripAdvisor reviews.

Revised: 1 We use only reviews that have at least 5 sentences, using the ﬁrst 4 as the context and the remainder as the ending to be generated.

Revised: There are on average 11 sentences per review.

Revised: We use the ﬁrst 1M reviews for training.

Revised: The second is the ROCStory corpus (Mostafazadeh et al., 2016), which is a collection of crowdsourced commonsense short stories, consisting of ﬁve sentences each (98k stories in the training set).

Revised: We use this corpus to train our model to predict a coherent ﬁnal sentence given the ﬁrst four.

Revised: For the (larger) TripAdvisor corpus, we train the language model on that corpus alone.

Revised: For the

Revised: ROCStory corpus, we pretrain the model for 420M tokens on the Toronto Books corpus 2 before training on the ROCStory text.

Revised: [Table]

Revised: Models We use the language model from Section 3.1 (a 2-layer RNN with 1024 GRU cells per layer) as our reference baseline.

Revised: We include three variants of our model: L2W ( NO META - LEARN ) uses a uniform mixture of scores for the learned sub-objectives (no meta-weight learning); L2W ( NO WK .

Revised: VOCAB ) omits the working vocabulary model (Section 3.2.5) but uses mixture weight learning for the other components; and L2W ( FULL ) is our full model.

Revised: To analyze the suitability of the reference endings and base language model samples to train the repetition model, we deﬁne a repetition ratio as the ratio of total words to unique words.

Revised: We found that ROCStory endings generated by the language model have a higher repetition ratio than the reference endings (1.12 vs. 1.02).

Revised: On TripAdvisor the difference is more pronounced, with a ratio of 1.48 for the language model against 1.17 for the references.

Revised: 4. E VALUATION S ETUP

Revised: Previous work has reported that automatic measures such as BLEU, ROUGE, and Meteor, do not lead to meaningful evaluation when used for long or creative text generation where there can be high variance among correct generation output (Wiseman et al., 2017b; Vedantam et al., 2015).

Original: For open-ended generation tasks such as our own, human evaluation is the only reliable measure (Li et al., 2016b; Wiseman et al., 2017b).
Revised: For open-ended generation tasks such as our own, human evaluation is the only reliable measure (Li et al., 2016b; Wiseman et al., 2017b).

Original: However, we also report the automatic measures to echo the previous observation regarding the mismatch between automatic and human evaluation
Revised: However, we also report the automatic measures to echo the previous observation regarding the mismatch between automatic and human evaluation.

Original: [Table]

Original: and provide multiple generation samples that give insights into the characteristics of different models.
Revised: Additionally we provide multiple generation samples that give insights into the characteristics of different models.

Original: 4. E XPERIMENTAL SETUP

Original: We pose the evaluation of our model as a conditional natural language generation task. Given an initial context of n sentences, the task is to generate an appropriate ending.
Revised: We pose the evaluation of our model as the task of generating an appropriate ending given an initial context of n sentences.

Original: For the purposes of automatic evaluation, the ending is compared against a gold reference ending that was written by a human (drawn from the original corpus).
Revised: For the purposes of automatic evaluation, the ending is compared against a gold reference ending that was written by a human (drawn from the original corpus).

Original: For human evaluation, the ending is presented to a human, who assesses the text according to several criteria. The criteria given to human judges arefour metrics closely inspired by Grice’s Maxims:
Revised: For human evaluation, the ending is presented to a human, who assesses the text according to several criteria, which are closely inspired by Grice’s Maxims:

Original: Quantity : Does the generation avoid repetition?
Revised: Quantity : Does the generation avoid repetition?

Original: Quality : Does the generation contradict itself (or the initial context)?
Revised: Quality : Does the generation contradict itself (or the initial context)?

Original: Relation : Does the generation appropriately continue the initial context?
Revised: Relation : Does the generation appropriately continue the initial context?

Original: Manner : Does the generation use the same style as the initial context?
Revised: Manner : Does the generation use the same style as the initial context?

Original: Finally, a Turing test question is presented to each judge: was the ending written by a human?
Revised: Finally, a Turing test question is presented to each judge: was the ending written by a human?

Original: In all of our experiments, we split each passage in our corpus into n = 4 initial context sentences and leave the remaining sentences as the generation target. For both automatic and human evaluation, we select 1000 passages from the test set of each corpus and generate endings using all models.
Revised: For both automatic and human evaluation, we select 1000 passages from the test set of each corpus and generate endings using all models, using the initial n = 4 sentences as context.

Original: Automaticevaluation simply scores each generated ending against its reference ending.
Revised: Automatic

Original: To conduct human evaluation, we present the initial context with the generation from each of the models individually to workers on Mechanical Turk, who score them using the above criteria. As a control, we also ask workers to score the original (human-written) reference endings for each selected passage.
Revised: [Table]

Original: [Table] Automated metrics are from 0–100.
Revised: Automated metrics are from 0–100.

Original: Human metrics are from 1– except for the Turing test, which is from 0–100.
Revised: Human metrics are from 1– except for the Turing test, which is from 0–100.

Original: Each cell is the micro-averaged across 1000 datums from a held-out test set.
Revised: Each cell is the micro-averaged across 1000 datums from a held-out test set.

Original: 4. C ORPORA

Original: We use two corpora for evaluation. The ﬁrst is TripAdvisor reviews. 1 We use only reviews that have at least 5 sentences, using the ﬁrst 4 as the context and the remainder as the reference ending.There are on average 11 sentences per review.
Revised: evaluation scores each generated ending against its reference ending.

Original: We use the ﬁrst 1M reviews for training. The second is the ROCStory corpus (Mostafazadeh et al., 2016), which is a collection ofcrowdsourced commonsense stories in short ﬁve sentences (98k stories in the training set). 4. M ODELS
Revised: To conduct human evaluation, we present the initial context with the generation from each of the models individually to workers on Mechanical Turk, who score them using the above criteria.

Original: Language model We use the language model from Section 3.1 (2-layer RNN with 1024 GRU cells per layer) as our reference baseline. For the (larger) TripAdvisor corpus, we train the language model on that corpus alone. For the ROCStory corpus, we pretrained the model for 420M tokens on the Toronto Books corpus 2 before training on the ROCStory text. We generate text using beam search with a beam size of 8.
Revised: As a control, we also ask workers to score the original (human-written) reference endings for each selected passage.

Original: We include three variants of our model: L2W ( NO META - LEARN ) uses a uniform mixture of scores for the learned sub-objectives (no meta-weight learning); L2W ( NO WK .

Original: VOCAB ) omits the working vocabulary model (Section 3.2.5) but uses meta-weight learning for the other components; and L2W ( FULL ) is our full model.

Original:  R ESULTS AND A NALYSIS 5. Q UANTITATIVE A NALYSIS
Revised:  R ESULTS AND A NALYSIS 5. Q UANTITATIVE A NALYSIS

Original: Results for both automatic and human evaluation metrics are presented for both corpora in Table 2.
Revised: Results for both automatic and human evaluation metrics are presented for both corpora in Table 2.

Original: For the TripAdvisor corpus, the language model baseline does best on all of the automated metrics.
Revised: For the TripAdvisor corpus, the language model baseline does best on all of the automated metrics.

Original: However on all human metrics, the L2W ( NO WK .
Revised: However on all human metrics, the L2W ( NO WK .

Original: VOCAB ) model scores higher, and achieves nearly twice the percentage of passed Turing tests (75% compared to 39%).
Revised: VOCAB ) model scores higher, and achieves nearly twice the percentage of passed Turing tests (75% compared to 39%).

Original: It scores even better than the L2W ( FULL ) model.
Revised: It scores even better than the L2W ( FULL ) model.

Original: Manual inspection reveals that for the TripAdvisor task, the topical focus and lexical speciﬁcity given by vocabulary scoring actually lead to slightly worse model behavior.
Revised: Manual inspection reveals that for the TripAdvisor task, the topical focus and lexical speciﬁcity given by vocabulary scoring actually lead to slightly worse model behavior.

Original: In TripAdvisor reviews, many possible sentences could logically follow a reviews context.
Revised: In TripAdvisor reviews, many possible sentences could logically follow a reviews context.

Original: A well-behaving but generic language generator is better able to leave judge’s unsurprised, if also uninformed.
Revised: A well-behaving but generic language generator is better able to leave judge’s unsurprised, if also uninformed.

Original: The full model brings additional focus to the generations, but at the cost of a reduction in overall coherence.
Revised: The full model brings additional focus to the generations, but at the cost of a reduction in overall coherence.

Original: In the ROCStory generation task the language model is more competitive with L2W ( FULL ).
Revised: In the ROCStory generation task the language model is more competitive with L2W ( FULL ).

Original: The raw language model is able to take advantage of the simpler data inherent in this task: ROCStories are shorter (the endings are always a single sentence) and have a smaller vocabulary.
Revised: The raw language model is able to take advantage of the simpler data inherent in this task: ROCStories are shorter (the endings are always a single sentence) and have a smaller vocabulary.

Original: However, the L2W full model still performs the best on all human metrics.
Revised: However, the L2W full model still performs the best on all human metrics.

Original: The ROCStory corpus presents speciﬁc contexts that require a narrower set of responses to maintain coherency.
Revised: The ROCStory corpus presents speciﬁc contexts that require a narrower set of responses to maintain coherency.

Original: The L2W ( FULL ) takes advantage of reranking by matching the topic of the context, yielding a higher ‘Relation’ score and more Turing test passes overall.
Revised: The L2W ( FULL ) takes advantage of reranking by matching the topic of the context, yielding a higher ‘Relation’ score and more Turing test passes overall.

Revised: The very low BLEU scores observed in our results in the TripAdvisor domain are an artifact of the

Revised: BLEU metrics length penalty.

Revised: The average length of reference completions is 12 sentences, which is much longer than the average length of endings generated by our Learning to Write models.

Revised: This forces the BLEU score’s length penalty to drive down the scores, despite our observation that there is still a signiﬁcant amount of word and phrase overlap.

Revised: The completions generated by the base language model are longer on average (as it tends to repeat itself over and over) and therefore to not suffer from this problem.

Original: 5. Q UALITATIVE A NALYSIS
Revised: 5. Q UALITATIVE A NALYSIS

Original: Learning to Write (L2W) generations are more topical and coherently mold with the context.
Revised: Learning to Write (L2W) generations are more topical and coherently mold with the context.

Original: Tableshows that both the L2W system as well as the classic RNN start similarly, commenting on the hotel staff and the room.
Revised: Tableshows that both the L2W system as well as the classic RNN start similarly, commenting on the hotel staff and the room.

Original: L2W is able to condense the same content into fewer words, following the context’s style.
Revised: L2W is able to condense the same content into fewer words, following the context’s style.

Original: Beginning with the third sentence, L2W and the LM diverge more signiﬁcantly.
Revised: Beginning with the third sentence, L2W and the LM diverge more signiﬁcantly.

Original: The
Revised: The

Original: LM makes a generic comment: “The location is perfect,” while L2W goes on to list speciﬁc items it enjoyed at breakfast, which was mentioned in the context.
Revised: LM makes a generic comment: “The location is perfect,” while L2W goes on to list speciﬁc items it enjoyed at breakfast, which was mentioned in the context.

Original: Furthermore, the LM becomes “stuck” repeating itself—a common problem for RNNs in general—whereas the L2W system ﬁnds a natural ending.
Revised: Furthermore, the LM becomes “stuck” repeating itself—a common problem for RNNs in general—whereas the L2W system ﬁnds a natural ending.

Original: 5. E RROR A NALYSIS

Original: The L2W models do not ﬁx every degenerate characteristic of RNNs.
Revised: The L2W models do not ﬁx every degenerate characteristic of RNNs.

Original: The TripAdvisor L2W generation in Table 1, while coherent and diverse, leaves room for improvement.
Revised: The TripAdvisor L2W generation in Table 1, while coherent and diverse, leaves room for improvement.

Original: The need to relate to topical phrases can override ﬂuency, as with the long list of “food, fresh fruit, juices, coffee, tea, hot chocolate.”
Revised: The need to relate to topical phrases can override ﬂuency, as with the long list of “food, fresh fruit, juices, coffee, tea, hot chocolate.”

Original: Furthermore phrases such as “friendly and helpful” and “clean and comfortable” occur in the majority of generations.
Revised: Furthermore phrases such as “friendly and helpful” and “clean and comfortable” occur in the majority of generations.

Original: These phrases, while common, tend to have semantics that are expressed in many different surface forms in real writing (e.g., “the staff were kind and quick to respond” ).
Revised: These phrases, while common, tend to have semantics that are expressed in many different surface forms in real writing (e.g., “the staff were kind and quick to respond” ).

Original: Appendix A presents sample generations from all models on both corpora for further inspection.
Revised: Appendix A presents sample generations from all models on both corpora for further inspection.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Generation with Long-term Context While relatively less studied, there have been several attempts at RNN-based paragraph generation for multiple domains including image captions (Krause et al., 2017), product reviews (Lipton et al., 2015; Dong et al., 2017), sport reports (Wiseman et al., 2017a), and recipes (Kiddon et al., 2016).
Revised: Generation with Long-term Context While relatively less studied, there have been several attempts at RNN-based paragraph generation for multiple domains including image captions (Krause et al., 2017), product reviews (Lipton et al., 2015; Dong et al., 2017), sport reports (Wiseman et al., 2017a), and recipes (Kiddon et al., 2016).

Original: Most of these approaches are based on sequence-tosequence (seq-to-seq) type architectures.
Revised: Most of these approaches are based on sequence-tosequence (seq-to-seq) type architectures.

Original: One effective way to avoid the degenerative characteristics of RNNsunder seq-to-seq is to augment the input with sufﬁcient details and structure that can guide the output generation through attention and a copying mechanism.
Revised: One effective way to avoid the degenerative characteristics of RNNsunder seq-to-seq is to augment the input with sufﬁcient details and structure that can guide the output generation through attention and a copying mechanism.

Original: However, for many NLP generation tasks, it is not easy to obtain such the large-scale training data required to support a robust sequence-to-sequence model.
Revised: However, for many NLP generation tasks, it is not easy to obtain such the large-scale training data required to support a robust sequence-to-sequence model.

Original: For example, a recent dataset for image-to-paragraph generation contains 14,575 training pairs Krause et al.
Revised: For example, a recent dataset for image-to-paragraph generation contains 14,575 training pairs Krause et al.

Original: (2017), and state-of-the-art hierarchical models, while improving over strong baselines, still lead to generation that repeats, contradicts, and is generic.
Revised: (2017), and state-of-the-art hierarchical models, while improving over strong baselines, still lead to generation that repeats, contradicts, and is generic.

Original: Alternative Decoding Objectives
Revised: Alternative Decoding Objectives

Original: The tendency of RNN generation to be short, blend, repetitive and contradictory has been noted multiple times in prior literature.
Revised: The tendency of RNN generation to be short, blend, repetitive and contradictory has been noted multiple times in prior literature.

Original: A number of papers propose approaches that can be categorized as alternative decoding objectives (Shao et al., 2017).
Revised: A number of papers propose approaches that can be categorized as alternative decoding objectives (Shao et al., 2017).

Original: For example,
Revised: For example,

Original: Li et al.
Revised: Li et al.

Original: (2016a) propose a diversity-promoting objective that interpolates the conditional probability score with negative marginal or reverse conditional probabilities.
Revised: (2016a) propose a diversity-promoting objective that interpolates the conditional probability score with negative marginal or reverse conditional probabilities.

Original: Unlike the negative marginal term that blindly penalizes generic generation, all our communication models are contextual in that they compare the context with continuation candidates.
Revised: Unlike the negative marginal term that blindly penalizes generic generation, all our communication models are contextual in that they compare the context with continuation candidates.

Original: Incorporating the reverse conditional probability has been also proposed through the noisy channel models of Yu et al.
Revised: Incorporating the reverse conditional probability has been also proposed through the noisy channel models of Yu et al.

Original: The clear beneﬁt of the reverse conditional probability model is that it can prevent the explaining away problem of the long-term context.
Revised: The clear beneﬁt of the reverse conditional probability model is that it can prevent the explaining away problem of the long-term context.

Original: However, decoding with the reverse conditional probability is signiﬁcantly more expensive, making it impractical for paragraph generation.
Revised: However, decoding with the reverse conditional probability is signiﬁcantly more expensive, making it impractical for paragraph generation.

Original: In addition, both above approaches do  presented in our work can be easily integrated into the existing beam search procedure, and each model is lightweight to compute.
Revised: In addition, both above approaches do not allow for learning more expressive models than our work presents. The communication models presented in our work can be easily integrated into the existing beam search procedure, and each model is lightweight to compute.

Original: The modiﬁed decoding objective has long been a common practice in statistical machine translation literature (Koehn et al., 2003;
Revised: The modiﬁed decoding objective has long been a common practice in statistical machine translation literature (Koehn et al., 2003;

Original: Och, 2003; Watanabe et al., 2007; Chiang et al., 2009).
Revised: Och, 2003; Watanabe et al., 2007; Chiang et al., 2009).

Original: Notably, it still remains a common practice with neural machine translation, even with a highly expressive network structure trained on an extremely large amount of data (Wu et al., 2016).
Revised: Notably, it still remains a common practice with neural machine translation, even with a highly expressive network structure trained on an extremely large amount of data (Wu et al., 2016).

Original: Inspired by all above approaches, our work presents a general learning framework together with a more comprehensive set of composite communication models.
Revised: Inspired by all above approaches, our work presents a general learning framework together with a more comprehensive set of composite communication models.

Original: Unlike previous approaches, our composite models are trained to directly address the particular limitations of the base RNN models, by integrating RNN generation into the discriminative learning.
Revised: Unlike previous approaches, our composite models are trained to directly address the particular limitations of the base RNN models, by integrating RNN generation into the discriminative learning.

Original: This allows for customization of the decoding objective to better cope with the undesired behavior of the base language models.
Revised: This allows for customization of the decoding objective to better cope with the undesired behavior of the base language models.

Original: Meta Learning There has been a broad range of literature on meta learning, where the learning goal broadens the search space of learning by targeting to learn model architectures, hyperparameters, and learning algorithms that are typically hand-designed or selected.
Revised: Meta Learning There has been a broad range of literature on meta learning, where the learning goal broadens the search space of learning by targeting to learn model architectures, hyperparameters, and learning algorithms that are typically hand-designed or selected.

Original: Andrychowicz et al.
Revised: Andrychowicz et al.

Original: (2016), for example, learn the learning rate, while Zoph & Le (2016) train a neural network to generate neural network architectures.
Revised: (2016), for example, learn the learning rate, while Zoph & Le (2016) train a neural network to gen- erate neural network architectures.

Original: while Snoek et al.
Revised: while Snoek et al.

Original: (2012) proposed Bayesian optimization to learn hyperparameters that cannot be optimized with gradient descent.
Revised: (2012) proposed Bayesian optimization to learn hyperparameters that cannot be optimized with gradient descent.

Original: The learning framework we presented here can be interpreted as a type of meta learning in a broad sense in that there are multiple layers of learning: learning the base RNN language model, then learning a collection of composite communication models that are customized to the particular behavior of the base model, and ﬁnally the generator that learns to combine all sub-components.
Revised: The learning framework we presented here can be interpreted as a type of meta learning in a broad sense in that there are multiple layers of learning: learning the base RNN language model, then learning a collection of composite communication models that are customized to the particular behavior of the base model, and ﬁnally the generator that learns to combine all sub-components.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Our work presents a uniﬁed learning framework that can learn to generate long, coherent text overcoming the limitations of RNNs as text generation models.
Revised: Our work presents a uniﬁed learning framework that can learn to generate long, coherent text overcoming the limitations of RNNs as text generation models.

Original: Our framework learns a decoding objective suitable for generation through a combination of sub-models that capture linguisticallymotivated qualities of good writing.
Revised: Our framework learns a decoding objective suitable for generation through a combination of sub-models that capture linguisticallymotivated qualities of good writing.

Original: Our work makes a unique contribution that complements existing literature on long text generation that is predominantly based on seq-to-seq models with a large amount of in-domain training data; we demonstrate that the ﬂuency of general RNN language models can be successfully guided to generate more lengthy and sensical text, which can be useful for domains where in-domain data is not sufﬁcient to support seq-to-seq type training.
Revised: Our work makes a unique contribution that complements existing literature on long text generation that is predominantly based on seq-to-seq models with a large amount of in-domain training data; we demonstrate that the ﬂuency of general RNN language models can be successfully guided to generate more lengthy and sensical text, which can be useful for domains where in-domain data is not sufﬁcient to support seq-to-seq type training.

Original: We propose a general framework for learning a decoding objective in two parts: learning component models to rank candidate generations, which are motivated by different aspects of Grice’s Maxims, and learning a weighing scheme that balances the inﬂuence of each of these scoring functions.
Revised: We propose a general framework for learning a decoding objective in two parts: learning component models to rank candidate generations, which are motivated by different aspects of Grice’s Maxims, and learning a weighing scheme that balances the inﬂuence of each of these scoring functions.

Original: This framework is amenable to any number and kind of sub-objectives, allowing for multiple qualities of good writing to be modeled across diverse domains.
Revised: This framework is amenable to any number and kind of sub-objectives, allowing for multiple qualities of good writing to be modeled across diverse domains.

Original: Human evaluation shows that the quality of the text produced by our model exceeds that of RNN baselines by a large margin and the generations score signiﬁcantly higher on a Turing test evaluation.
Revised: Human evaluation shows that the quality of the text produced by our model exceeds that of RNN baselines by a large margin and the generations score signiﬁcantly higher on a Turing test evaluation.

Original: A E XAMPLE G ENERATIONS
Revised: A E XAMPLE G ENERATIONS

Original: [Table]
Revised: [Table]

Original: gave
Revised: gave

Original: [Table]
Revised: [Table]

Original: She
Revised: She

Original: [Table]
Revised: [Table]

Original: shed
Revised: shed

Original: [Table]
Revised: [Table]

Original: it
Revised: it

Original: [Table]
Revised: [Table]

Original: We
Revised: We

Original: [Table]
Revised: [Table]

Original: She
Revised: She

Original: [Table]
Revised: [Table]

Original: The hotel staff were very kind and helpful.
Revised: The hotel staff were very kind and helpful.

Original: Our room was large but a little sparse.
Revised: Our room was large but a little sparse.

Original: The garden is really nice in the courtyard.
Revised: The garden is really nice in the courtyard.

Original: The location is perfect, next door to the station
Revised: The location is perfect, next door to the station

Original: [Table]
Revised: [Table]

Original: If you want to be in the middle of the action, this is not the place for you. If you want to be in the middle of the action, this is not the place for you.
Revised: If you want to be in the middle of the action, this is not the place for you. If you want to be in the middle of the action, this is not the place for you.

Original: If you want
Revised: If you want

Original: [Table]
Revised: [Table]

Original: But
Revised: only

Original: [Table]
Revised: [Table]

Original: is a great place to
Revised: be

Original: [Table]
Revised: [Table]

Revised: go

Original: [Table]
Revised: [Table]

Original: There are no stairs to the hotel but quite a few in the hotel. Restaurants right outside the door. The only problem would be for people who are not early risers, the church bells go early ( about 7am ) are loud and go on for a while( a few minutes ).
Revised: while

Original: We used it as an alarm, which was ﬁne for us but some may not like it.
Revised: [Table] [Table] was ﬁne for us but some may not [Table]

Original: The breakfast was a little adverage but no un-eatable
Revised: it

Original: [Table] [Table]
Revised: [Table] [Table]

Original: multiple
Revised: multiple

Original: [Table]
Revised: [Table]

Original: liki tiki village i was a bit nervous about stay- ing there.
Revised: liki tiki village i was a bit nervous about stay- ing there.

Original: We loved every minute.
Revised: We loved every minute.

Original: I wish that we had more time to enjoy the
Revised: I wish that we had more time to enjoy the

Original: [Table]
Revised: [Table]

Original: , but we were visiting the parks most of the time.
Revised: , but we were visiting the parks most of the time.

Original: The rooms were clean and the kitchen well stocked
Revised: The rooms were clean and the kitchen well stocked

Original: [Table]
Revised: [Table]

Original: staff was very friendly and helpful.
Revised: staff was very friendly and helpful.

Original: I would recommend this hotel to anyone who is looking for a nice place to stay in the heart of the city.
Revised: I would recommend this hotel to anyone who is looking for a nice place to stay in the heart of the city.

Original: I would highly recommend this hotel to anyone who is looking for a place to stay in san diego.
Revised: I would highly recommend this hotel to anyone who is looking for a place to stay in san diego.

Original: I would highly recommend this hotel to anyone who is looking for a place to stay in san diego. I would highly recommend this hotel to anyone who is looking for a place to stay in san diego.
Revised: I would highly recommend this hotel to anyone who is looking for a place to stay in san diego. I would highly recommend this hotel to anyone who is looking for a place to stay in san diego.

Original: If you
Revised: If you

Original: [Table]
Revised: [Table]

Original: Our
Revised: Our

Original: [Table]
Revised: [Table]

Original: The rooms were clean and the kitchen well stocked.
Revised: The rooms were clean and the kitchen well stocked.

Original: The staff were very friendly and helpfu!
Revised: The staff were very friendly and helpfu!

Original: I would deﬁnately recommend this hotel to anyone who is looking for a place to stay in the heart of san diego.
Revised: I would deﬁnately recommend this hotel to anyone who is looking for a place to stay in the heart of san diego.

Original: Thank you to all of the staff for making our stay so enjoyable!
Revised: Thank you to all of the staff for making our stay so enjoyable!

Original: Human
Revised: Human

Original: We did go to the time share presentation but we left almost immediately since we are owners elsewhere.
Revised: We did go to the time share presentation but we left almost immediately since we are owners elsewhere.

Original: I guess they ﬁgured out that they would be wasting their time.
Revised: I guess they ﬁgured out that they would be wasting their time.

Original: Great to have a washer and dryer.
Revised: Great to have a washer and dryer.

Original: Also great to have a seperate room for the kids.
Revised: Also great to have a seperate room for the kids.

Original: The grounds were clean and we felt safe.
Revised: The grounds were clean and we felt safe.

Original: Checkin was a little confusing since they gave us a map of the property and then wrote all over it in black pen so that you could not see the map.
Revised: Checkin was a little confusing since they gave us a map of the property and then wrote all over it in black pen so that you could not see the map.

Original: Overall it was a great experience.
Revised: Overall it was a great experience.

Original: Close to the grocery store and to target.
Revised: Close to the grocery store and to target.

Original: A little difﬁcult to ﬁnd but the gps got us there.
Revised: A little difﬁcult to ﬁnd but the gps got us there.

Original: Great for the price and amenities.
Revised: Great for the price and amenities.

Original: [Table]

Original: [Table]

Original: ﬁ

Original: [Table]

Original: [Table]

Original: will

Original: [Table]
Revised: [Table]

Original: fo

Original: [Table]

Original: was

Original: [Table]

Original: will

Original: [Table]
Revised: [Table]

Original: sure

Original: [Table]
Revised: [Table]

Original: B M ODEL DETAILS AND HYPERPARAMETERS
Revised: B M ODEL DETAILS AND HYPERPARAMETERS

Original: Base language model We use a 2-layer RNN language model with 1024 GRU cells per layer.
Revised: Base language model We use a 2-layer RNN language model with 1024 GRU cells per layer.

Original: Embeddings vectors are of length 1024.
Revised: Embeddings vectors are of length 1024.

Original: Following Inan et al.
Revised: Following Inan et al.

Original: (2017) we tie the input and output embedding layers’ parameters.
Revised: (2017) we tie the input and output embedding layers’ parameters.

Original: To regularize we dropout (Srivastava et al., 2014) cells in the output layer of the ﬁrst layer with probability 0.2.
Revised: To regularize we dropout (Srivastava et al., 2014) cells in the output layer of the ﬁrst layer with probability 0.2.

Original: We use mini-batch stochastic gradient descent (SGD) and anneal the learning rate regularly when the validation set performance fails to improve.
Revised: We use mini-batch stochastic gradient descent (SGD) and anneal the learning rate regularly when the validation set performance fails to improve.

Original: Learning rate, annealing rate, and batch size were tuned on the validation set for each dataset, with details in section 4.4.
Revised: Learning rate, annealing rate, and batch size were tuned on the validation set for each dataset, with details in section ?? .

Original: Gradients are backpropagated 35 time steps and clipped to a maximum value of 0.25.
Revised: Gradients are backpropagated 35 time steps and clipped to a maximum value of 0.25.

Original: Entailment model Dropout is performed on both the input word embeddings and the MLP hidden layer with rate 0 .
Revised: Entailment model Dropout is performed on both the input word embeddings and the MLP hidden layer with rate 0 .

Original: Training is performed with Adam (Kingma & Ba, 2015) with learning rate 0 .
Revised: Training is performed with Adam (Kingma & Ba, 2015) with learning rate 0 .

Original: 0005 , batch size 128 .
Revised: 0005 , batch size 128 .

Original: Relevance model
Revised: Relevance model

Original: The convolutional layer is a 1D convolution with ﬁlter size 3 and stride 1 ; the sequences are padded so that the sequence output is the same length as the input.
Revised: The convolutional layer is a 1D convolution with ﬁlter size 3 and stride 1 ; the sequences are padded so that the sequence output is the same length as the input.

Original: The model is trained with Adam, learning rate 0 .
Revised: The model is trained with Adam, learning rate 0 .

Original: 001 and dropout is applied before the ﬁnal linear layer with rate 0 .
Revised: 001 and dropout is applied before the ﬁnal linear layer with rate 0 .

Original: Meta-weight learning Training is performed with SGD with a learning rate of 1.
Revised: Meta-weight learning Training is performed with SGD with a learning rate of 1.