A BIRD ’ S EYE VIEW ON COHERENCE , AND A WORM ’ S EYE VIEW ON COHESION
Anonymous authors
Paper under double-blind review
A BSTRACT
Generating coherent and cohesive long-form texts is a challenging problem in natural language generation.
Previous works relied on a large amount of humangenerated texts to train neural language models, however, few attempted to explicitly model the desired linguistic properties of natural language text, such as coherence and cohesion using neural networks.
In this work, we train two expert discriminators for coherence and cohesion to provide hierarchical feedback for text generation.
We also propose a simple variant of policy gradient, called negative-critical sequence training in which the reward baseline is constructed from randomly generated negative samples.
We demonstrate the effectiveness of our approach through empirical studies, showing improvements over the strong baseline – attention-based bidirectional MLE-trained neural language model – in a number of automated metrics.
The proposed model can serve as baseline architectures to promote further research in modeling additional linguistic properties for downstream NLP tasks.
I NTRODUCTION
The terms coherence and cohesion in linguistics are commonly deﬁned as follows (Williams &
Colomb, 1995).
Cohesion : sentence pairs ﬁtting together the way two pieces of a jigsaw puzzle do.
Coherence : what all the sentences in a piece of writing add up to, the way all the pieces in a puzzle add up to the picture on the box.
In layman’s terms, cohesion indicates that two consecutive sentences are locally well-connected, and coherence indicates that multiple sentences globally hold together.
Generating cohesive and coherent natural language texts that span multiple sentences is a challenging task mainly due to two reasons.
First, there is no formal speciﬁcation of cross-sentence linguistic properties, such as coherence and cohesion of a text.
Second, there is no widely accepted surrogate neural model to evaluate the two properties.
Most state-of-the-art neural approaches to natural language generation (NLG) relied on a large amount of human-generated texts to train language models (Cho et al., 2014; Graves, 2013; Sutskever et al., 2014).
Although these models can generate sentences that, if judged individually, are similar to human-generated ones, they often fail to capture the local and global dependencies among sentences, resulting in a neither coherent nor cohesive text.
For example, neural language models based on Recurrent Neural Networks (RNNs) are widely applied to response generation for dialogue (Vinyals & Le, 2015; Shang et al., 2015;
Sordoni et al., 2015; Li et al., 2015).
Although the responses by themselves look reasonable, they are either bland such as “I don’t know”, or incoherent with the whole dialogue session.
See Gao et al.
(2018) for a comprehensive survey.
In this paper, we strive to address the challenge in a principled manner.
We propose a pair of discriminators to score whether and to what extent a text is coherent or cohesive.
The coherence discriminator measures the compatibility among all sentences in a text using sentence-level features, thus providing a macro-level view on the multi-sentence text.
The cohesion discriminator, on the other hand, measures the compatibility of each pair of consecutive sentences using only word-level features, thus providing a micro-level view on neighboring sentences.
These models, given a conditional input text and multiple candidate output texts, are learned to score the candidates with respect to the criterion by optimizing pairwise ranking losses.
These scores are then used as reward signals to train an RNN-based language model to generate (more) coherent and cohesive texts.
Our main contributions are three-fold: (1) we propose two linguistic discriminators for modeling coherence and cohesion of a text; (2) we present a simple yet effective training mechanism to encode these linguistic properties; and (3) we propose negative-critical sequence training , a variant of policy gradient method, which uses negative samples to construct its reward baseline .
This paper proposes a neural approach to explicitly model cross-sentence linguistic properties, i.e., coherence and cohesion, for long text generation.
Despite the encouraging initial results, we only scratched the surface of the problem.
The proposed method is yet to be signiﬁcantly improved to meet the ultimate goal of generating meaningful and logical long-form texts.
We cast the text generation as an RL problem and review recent work in Section 2, and detail our approach in Section 3.
R ELATED WORK
Coherence and cohesion.
Coherence and cohesion have been extensively studied in the computational linguistics community, particularly in the ‘pre-deep-learning’ era.
Lack of formal speciﬁcations for coherence and cohesion (Mani et al., 1998), resulted in many different formalism, such as Rhetorical Structure Theory (Mann & Thompson, 1988), and other forms of coherence and cohesion relations and their quantiﬁcation (Mani et al., 1998; Hobbs, 1985; Hovy, 1988; McKeown, 1985; Cohen & Levesque, 1985; Hovy, 1991; Cristea et al., 1998; Halliday & Hasan, 1996; Liddy, 1991; Van Dijk, 2013; Edmundson, 1969; Barzilay & Lapata, 2008).
In fact, the list is not exhaustive. However, modeling coherence and cohesion of a text using models parametrized by neural networks have not been previously explored. Word sequence generation in a reinforcement learning framework. A word sequence generation task can be framed as a reinforcement learning (RL) problem, in which the generator G is acting as a policy π , with parameters θ π , and each generated word at time t , w t , can be viewed as an action to be chosen by the policy from a large discrete space, or vocabulary, conditioned on state s t − 1 = w ≤ t − 1 , which encodes the previously generated text sequence.
Let r t be the reward for a partially generated text sequence w ≤ t .
We deﬁne the long-term expected reward J ( π ) =
E s 0 ∼ q,π
[ (cid:80) ∞ t =1 γ t − 1 r t ] , where q is the initial distribution of conditional input texts.
Following Sutton et al.
(1999), the gradient of J with respect to θ π is
[Equation]
where ρ π is the stationary distribution and Q π ( s, a ) is the expected return from state s and taking action a , both following policy π .
For brevity, we omit the derivation.
In our work, we formulate text generation as an episodic RL problem with episode length L , rewards r L being available only at the end of episode and γ = 1 .
There are many works on training neural language models using reward signals, such as Ranzato et al.
and Paulus et al.
These works directly optimize for speciﬁc metrics, such as BLEU (Papineni et al., 2002) or ROUGE (Lin & Hovy, 2003), using REINFORCE (Williams, 1992; Sutton et al., 1999).
However, it is well-known that these metrics do not give a complete picture on the quality of the generated text.
Only recently have there been efforts to provide more relevant quality objectives for which to optimize (Li et al., 2015; 2016a; Holtzman et al., 2018) the quality of interest such as consistency, repetition of text. But these works use the objective function to re-rank candidate outputs, not to reward or penalize outputs when they are generated in the ﬁrst place.
Li et al.
(2016b) constructed a set of reward models, such as information ﬂow and semantic coherence, to tune the generator, yet they do not provide an ablation study to elaborate the relative contribution of these reward models individually.
GANs for text generation. Another line of research is to use Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) to incorporate feedback signals for text generation (Yu et al., 2017; Lin et al., 2017; Zhang et al., 2017c; Guo et al., 2017; Fedus et al., 2018;
Zhang et al., 2018).
However, the discriminator in these works are trained to distinguish real texts from the generated ones, operating as a black-box rather than providing ﬁne-grained feedback on particular linguistic aspects of the texts.
In fact, Yang et al.
(2018) has partially addressed this issue by using a trained language model as the discriminator.
Although the discriminator provides a ﬁne-grained feedback at the word-level, it does not critique on many important linguistic properties of generated texts, such as cohesion and coherence.
These text generators, when facing a long-form text generation task that span multiple sentences, are by no means perfect and often exhibit some critical errors, such as a breakdown of local connections between consecutive sentences (cohesion), let alone globally solid intention (coherence).
As a result, readers can easily take these cues and discriminate such generated texts against real texts.
In this paper, we argue that the primary reason is the lack of an effective mechanism of measuring and controlling the text quality in the generation process. The method we propose in the next section is intended to address the problem.
M ODEL
We assume that global coherence of a text depends to a large degree upon how its individual sentences with different meanings are organized.
So we focus our evaluation of coherence solely on the sentence-level.
If the sentences are not organized properly, the intention of the paragraph as a whole is obscure, regardless of seamless local connectivity between consecutive sentences.
This is not to say that local connections between any two neighboring sentences should be overlooked.
One can easily distinguish a model-generated sentence from a real one, simply by looking at whether the sentence followed by another sentence relates , besides grammar.
We instill these two different yet important concepts in two discriminators, each operating on the sentence level and word level.
Our models closely resemble successful models for computer vision, such as StackGAN (Zhang et al., 2017a;b) and PatchGAN (Isola et al., 2017) in that they all provide hierarchical signals to their corresponding generators, where the signals are derived from raw lowlevel data.
We call the sentence-level discriminator the coherence discriminator D coherence , and the word-level discriminator the cohesion discriminator D cohesion .
3. C OHERENCE DISCRIMINATOR : D COHERENCE
This discriminator measures how likely two text chunks form a coherent paragraph.
Let S := [ s 1 , s 2 , ..., s n ] be the source text chunk that consists of n sentences, T :=
[ t 1 , t 2 , ..., t m ] be the real target text chunk that consists of m sentences, and (cid:101) T :
= (cid:2)(cid:101) t 1 , (cid:101) t 2 , ..., (cid:101) t (cid:101) m (cid:3) be the model-generated target text chunk that consists of (cid:101) m sentences. D coherence is designed to distinguish a real pair ( S, T ) from a synthetic pair ( S, (cid:101) T ) by assigning them with different scores, i.e., Score coherence ( S, T ) > Score coherence ( S, (cid:101) T ) .
Design .
Our design of D coherence is inspired by the Deep Structured Semantic Model (DSSM)
(Huang et al., 2013; Gao et al., 2014; Xu et al., 2017).
Given source text chunk S and target text chunk T , D coherence computes their coherence score in two steps, as illustrated in Figure 1. First, a pair of convolutional networks (CNNs) 1 or RNNs are applied to encode both S and T into two lowdimensional continuous vectors.
Second, the coherence score is computed as the cosine similarity of the two vectors. The score is a real value between − 1 and 1, where 1 indicates the maximal coherence score, and − 1 the minimal coherence score.
D coherence measures how likely S and T add up to a single coherent passage.
The score depends on the parameters of the two CNNs/RNNs, or in other words, how S and T are encoded. Since we focus solely on the sentence-level features, we view a text chunk as a sequence of sentences, and view each sentence as a bag-of-words.
Therefore, we represent each word using its pre-trained word embedding vector (Pennington et al., 2014) and represent each sentence using a vector which takes the average of its word embedding vectors. A text chunk is then represented as a sequence of sentence vectors which are fed to the CNN/RNN (either f ( · ) or g ( · ) in Figure 1).
Notice that f ( · ) and g ( · ) do not share parameters. The parameters of f ( · ) and g ( · ) are optimized in such a way that a real pair scores higher than a synthetic pair: [Equation]
Formally, the task of optimizing D coherence can be stated as follows. Given a set of training samples of the form (( S, T ) , ( S, (cid:101) T ))
( i ) , i = 1 ... M , we optimize parameters ( θ f , θ g ) by minimizing the pairwise rank loss on training data deﬁned as [Equation] where L ( · ) is a loss function, differentiable w.r.t.
( θ f , θ g ) .
In the following subsection, we will describe in turn how we construct the pairwise training samples and the form of the loss function.
Since D coherence is used as a pairwise ranker, we employ the metrics commonly used in information retrieval for evaluation, such as recall at K (R @ K ), which is deﬁned as the fraction of correctly identifying an item in the TOP- K retrieved list (Baeza-Yates & Ribeiro-Neto, 1999).
We present the retrieval results on test data in Table 2.
Training mechanism .
How do we construct these list of candidate target sentences T , given the source sentences S ?
We assume the T that follows an S in the data is a positive target sample, or the correct item to retrieve.
Negative samples are constructed using three different methods within a batch while iterating through the training data, motivated by Wieting et al.
• Rotate T with S ﬁxed (mismatch S and T ) across a batch.
For a single S , this method yields B − 1 negative samples, where B is the batch size.
• Shufﬂe the sentence order once, different from its original order, known as a derangement, in each T to break coherence, and this yields one negative sample.
• Combine the previous two methods: rotate T across a batch and shufﬂe sentences within T , yielding B − 1 negative samples.
These 2 B − 1 negative samples and a single positive sample, in sum, pose a signiﬁcant challenge in learning.
To ﬁt this training task into a ranking framework, we optimize over [Equation] where the arithmetic mean parametrized by λ : AM λ ( x
[Equation]
= N [Equation] In our experiments, we ﬁx λ = 2 and this assigns more weight to a more challenging negative sample 2 . Notice that AM λ is the mean if λ = 0 , and approaches the max as λ → ∞ .
Empirically, training the models using the weighted mean resulted in faster convergence, as opposed to using the single most challenging negative sample score ( max ) or the mean of all negative sample batch.
3. C OHESION DISCRIMINATOR : D COHESION
Our second discriminator pays attention only to low-level features to capture a local connection between arbitrary two consecutive sentences because the pairwise connectivity inﬂuence readability.
For simplicity, D cohesion is similar to D coherence , except that its architecture, input, and negative sample construction are modiﬁed to encode cohesion between any pair of sentences on the word-level.
A single input sample to D cohesion is a pair of two consecutive sentences: [ s i, 1 , s i, 2 , ..., s i,n ] and [ s i +1 , 1 ,
s i +1 , 2 , ..., s i +1 ,m ] , where s i,k denotes the k -th word in sentence i .
We construct the negative samples using the three methods for training D coherence , where shufﬂing occurs on the word level within each sentence, rather than shufﬂing multiple sentences on the sentence level.
3. G ENERATOR : G
The two pre-trained discriminators, D coherence and D cohesion , are used to modify the text generation behavior of G . G is an attention-based bidirectional sequence-to-sequence model. It is initially pretrained via maximizing the word-level likelihood given the training data, and we denote as G MLE .
However, the model-generated texts from G MLE often do not hold to the standards of two discriminators. Therefore, we need to change the text generation behavior of G with respect to the criteria. To this end, the scores from the discriminators that model the criteria are used as direct reward or penalty signals to modify the parameters of G . Given these signals, we use our proposed variant of the policy gradient, negative-critical sequence training, to apply parameter updates in G .
We discuss the details in the next section.  N EGATIVE -C RITICAL S EQUENCE T RAINING Actor-critic methods (Barto et al., 1983; Witten, 1977) parameterized by neural networks typically require learning a separate critic network to estimate the expected future reward as a baseline , which in many cases is a difﬁcult task by itself.
In NLP, we have observed similar practices and challenges by Ranzato et al.
, Bahdanau et al.
(2016), and Nguyen et al.
However, recently Rennie et al.
(2017) proposed an effective self-critical sequence training that avoids learning a separate critic network.
Similarly, our method does not require learning a separate critic network, instead we directly use the scores of negative samples assigned by the discriminators as the baseline .
For an arbitrary pair of S and T gen , which is the generator’s output conditioned on S , we compute the coherence and cohesion scores by calling D coherence and D cohesion .
Since each review consists of multiple sentences, the overall cohesion score is computed as the average of scores of all consecutive sentence pairs.
These scalar scores, however, have no interpretation since the discriminators are trained by optimizing a margin ranking loss.
Instead, the differences between positive sample scores and the maximal or average negative sample scores provide insight of how well the models can distinguish between the positives and the negatives.
Therefore, these margins can be considered as rewards with baselines, and thus we deﬁne the reward functions as:
[Equation]
where ˜ · denotes a negative sample for a given source condition, and E (cid:101) T ( and E (cid:101) s i +1 ) are computed by averaging over an ensemble of negative samples.
Notice that this reward resembles the ranking loss we use to train our discriminators, except that our baseline is an average score (instead of the weighted arithmetic mean) over negative samples.
The rational for this difference is that: the maximal or the weighted arithmetic mean score baseline is in fact noisy to be used as rewards, because the best randomly constructed negative samples may be a formidably good sample.
To alleviate such noise, we use the average discriminator scores of negative samples as the baseline , and this turns out to be an empirically better alternative.
Finally, we use policy gradient (Williams, 1992; Sutton et al., 1999) to maximize a weighted combination of the coherence and cohesion rewards.
For illustrative purposes, we equally weigh them for updating our policy, i.e., the generator G .
[Table]
E XPERIMENTS
In this section we show results of training both D coherence and D cohesion and compare RL-tuned generators G MLE+RL(cohesion) , G MLE+RL(coherence) , and G MLE+RL(coherence, cohesion) with the baseline model G
MLE .
We argue that through the use of feedback from our simple discriminators to G MLE , we improve the quality of generated texts.
See Table 1 for a sample comparison.
5. D ATASET AND EVALUATION METRICS
We use the publicly available TripAdvisor’s hotel reviews dataset collected by Wang et al.
and the Yelp review dataset 3 .
We consider only subsets of the two review datasets satisfying the following two conditions: a review must have (1) at least 10 sentences, and (2) each sentence should have more than 5 and less than 30 words.
This yields roughly 60,000 TripAdvisor reviews and 220,000 Yelp reviews, split into [0 . 8 , 0 . 1 , 0 .
1] ratio for train/dev/test.
We merge the source and target vocabularies, and limit it to the top 50,000 frequent words, excluding special tokens.
For each of these reviews, as in Holtzman et al.
(2018), we consider the ﬁrst ﬁve sentences as the source input S to G , and the following ﬁve sentences as the target output T from G .
It is widely known that there is no accurate metric to evaluate the generator.
Nevertheless, we report scores of standard metrics, such as negative log-likelihood (NLL), perplexity (PPL), BLEU and proportion of unique n -grams within a single generation (intra-unique- n ), and across generations (inter-unique- n ), as in Gu et al.
5. I MPLEMENTATION DETAILS
G takes individual words as inputs and embeds into a pre-trained 300-dimensional word vectors from GloVe (Pennington et al., 2014).
This embedding layer is ﬁxed throughout training.
G uses a gated recurrent unit with two layers and a hidden size of 1024 for both bidirectional encoder and attention-based decoder.
During optimization using Adam (Kingma & Ba, 2014), we set the learning rate to 2 e- 4 and clip the gradient’s L2-norm to 1.0.
We initially train G MLE by maximizing the wordlevel likelihood estimation (MLE) from data that consist of positive samples for 60 epochs on the
TripAdvisor data and 30 epochs on the Yelp dataset, separately.
These are our baseline models against which to empirically prove value of our hierarchical discriminators.
D coherence also uses the pre-trained GloVe word vectors 4 , which are ﬁxed.
The source processing network and the target processing network have the same structure, but different parameters.
If the encoder type is a CNN, the convolutional layer has ﬁlters of sizes 2, 3, 4, and 5, each withﬁlters.
Each convolution ﬁlter is followed by a tanh activation.
Then we max-pool in time over the features and append a fully connected layer into a feature embedding of dimension 512, followed by a batch normalization layer and a tanh activation. If the encoder type is a RNN, we use a 1-layered bi-directional GRU, concatenate ﬁnal hidden states from both ends, and append the same network layers as used in the CNN-encoded counterpart.
We use an Adam optimizer with a learning rate of [Table] 1 e- 5 .
D cohesion is the same as D coherence , except the CNN-encoded D cohesion has convolutional ﬁlters of sizes 3, 4, 5, and 6.
We train both discriminators for 50 epochs and choose models with the best R @ 1 validation scores.
Retrieval results are shown in Table 2.
In the tuning stage, we use the negative-critical sequence training as explained in Section 4 up to 5 epochs, with a learning rate of 1 e- 5 .
We also continue with supervised learning to G to limit the policy search within a grammatically correct space, similar to Paulus et al.
; Wu et al.
(2016); Lewis et al.
In practice, sequence-level rewards are only available upon a completed generation, so they are sparse signals for the generator.
Typically, sparse end-of-sequence rewards entail a noisy training, yet would want the learning generalize to the testing data.
We observed that, for our particular task, most noises were caused by exploration [Table] and the learning generalized to the testing data.
Thus, reward shaping was unnecessary, unlike previous works (Li et al., 2017; Yang et al., 2018) that further provided signals for partially generated sequences.
For all generations, we used greedy decoding since we did not see a signiﬁcant difference using beam search.
Results are shown in Table [Table] 5. D ISCUSSION
In Table
[Table]
notice that for RNNs outperform CNNs for coherence models
[Table]
and CNNs outperform RNNs for cohesion models. One explanation is that RNNs are effective in encoding a sequential input yet exhibit drawbacks when encoding into hidden states at both ends of a long input, otherwise well-known as a long-range dependency problem. To aid understanding of the roles of D coherence and D cohesion , we show randomly selected positive and negative samples and corresponding rewards in Table 4.
While reinforcing coherence and cohesion properties in text generation through surrogate models is an important research direction, we consider our results to be preliminary
[Table]
and our experiment results allude to room for improvement, such as recall scores.
This is because our methods to
[Table]
is an example of contradiction and inconsistent sentiment, suggestive of incoherence.
We append more examples with extreme cohesion margin scores.
construct negative samples from an unlabelled dataset are not thorough.
For example, a randomly mismatched sentence that follows a given sentence may actually be a valid continuation.
In this work, we overlook this problem since our proposed schemes are shown to be effective in modeling coherence and cohesion.
C ONCLUSION
In this paper, we propose to model coherence and cohesion through simple training mechanisms via models parametrized by neural networks, and quantify coherence and cohesion into negative-critical margin scores.
The coherence discriminator D coherence provides a macro-level view on structuring a multi-sentence text. It assesses how likely two text chunks form a coherent paragraph, using sentence-level features.
On the other hand, the cohesion discriminator D cohesion provides a microlevel view on local connectivity between sentence pairs. It assesses how cohesive two consecutive sentences are, using word-level features.
The margin scores computed by these discriminators are used as reward signals for training neural language models via policy gradient.
Empirical results on two long-form text generation tasks show that our surrogate coherence and cohesion models, trained through simple yet effective methods, help improve over the strong baseline, an attention-based bidirectional MLE-trained sequence-tosequence model in a number of automatic metrics.
Future work will focus on casting the long-form text generation task using the GANs framework. In this framework, the coherence and cohesion discriminators are modiﬁed against model-generated texts, and in turn, provide signals to learn neural language models.