Analyzing Neural Discourse Coherence Models (2020)

Abstract: In this work, we systematically investigate how well current models of coherence can capture aspects of text implicated in discourse organisation. We devise two datasets of various linguistic alterations that undermine coherence and test model sensitivity to changes in syntax and semantics. We furthermore probe discourse embedding space and examine the knowledge that is encoded in representations of coherence. We hope this study shall provide further insight into how to frame the task and improve models of coherence assessment further. Finally, we make our datasets publicly available as a resource for researchers to use to test discourse coherence models.

## introduction coherence refers to the properties of a text that indicate how meaningful (sub-)sentential constituents are connected to convey document-level meaning. different theories have been proposed to describe the properties that contribute to discourse coherence and some have been integrated with computational models for empirical evaluation. a popular approach is the entity-based model which hypothesizes that coherence can be assessed in terms of the distribution of and transitions between entities in a text -by constructing an entity-grid (egrid) representation @xcite , building on centering theory @xcite . subsequent work has adapted and further extended egrid representations @xcite @xcite . other research has focused on syntactic patterns that co-occur in text @xcite or semantic relatedness between sentences @xcite @xcite as key aspects of coherence modeling. there have also been attempts to model coherence by identifying rhetorical relations that connect textual units @xcite @xcite or capturing topic shifts via hidden markov models (hmm, @xcite . other work has combined approaches to study whether they are complementary @xcite . more recently, neural networks have been used to model coherence. some models utilize structured representations of text (e.g. egrid representations, tien nguyen and @xcite and others operate on unstructured text, taking advantage of neural models' ability to learn useful representations for the task @xcite @xcite @xcite . coherence has typically been assessed by a model's ability to rank a well-organized document higher than its noisy counterparts created by corrupting sentence order in the original document (binary discrimination task), and neural models have achieved remarkable accuracy on this task. recent efforts have targeted additional tasks such as recovering the correct sentence order @xcite , evaluating on realistic data @xcite and focusing on open-domain models of coherence @xcite . however, less attention has been directed to investigating and analyzing the properties of coherence that current models can capture, nor what knowledge is encoded in their representations and how it might relate to aspects of coherence. in this work, we systematically examine what properties of discourse coherence current coherence models can capture. we devise two datasets that exhibit various kinds of incoherence and analyze model ability to capture syntactic and semantic aspects of text implicated in discourse organisation. we furthermore investigate a set of probing tasks to better understand the information that is encoded in their representations and how it might relate to aspects of coherence. we hope this study shall provide further insight into how to frame the task and improve models of coherence assessment further. finally, we release our evaluation datasets as a resource for the community to use to test discourse coherence models. foot_0 ## neural coherence models we experiment with a number of existing and stateof-the-art neural approaches to coherence assessment, that have publicly available implementations, and present details of the models below. across all the bert-based models, we use bert-large-uncased and layer 16 following @xcite and @xcite . multi-task learning (mtl, farag and yannakoudakis, 2019): the model applies a bi-lstm on input glove word embeddings @xcite followed by attention to build sentence representations; then builds a second bi-lstm with attention to compose a document vector. a linear operation followed by a sigmoid function is applied to the document representation to predict an overall coherence score as the main objective. inspired by the egrid approaches, the model is also optimized to predict the grammatical roles of the input words at the bottom layer of the network as an auxiliary task. mtl with bert embeddings (mtl bert ): we replicate the previous mtl model but now use bert embeddings @xcite to initialize the input words. single-task learning (stl, @xcite : this model has the same architecture as mtl but only performs the coherence prediction task, excluding the grammatical role auxiliary objective. stl with bert (stl bert ): this is the same as stl but uses bert embeddings. local coherence discriminator with language modeling (lcd rnnlm , @xcite : the model generates sentence representations via an rnn language model, where word embeddings are initialized using glove. it then generates a representation for two consecutive sentences via concatenating the output of a set of linear transformations applied to the two sentences: concatenation, element-wise dif-ference, element-wise product and absolute value of element-wise difference. this representation is fed to an mlp layer to predict a local coherence score. 2 the overall coherence of a document is the average of its local scores. lcd with bert (lcd bert ): we create a variant of the lcd rnnlm model where instead of using an rnn language model encoder, we encode each sentence as the average bert vectors of the words it contains. everything else remains the same. local coherence (lc, li and jurafsky, 2017): the model generates sentence vectors via an lstm over glove-initialized word embeddings; then a window approach is applied over adjacent sentences to get embeddings of groups of sentences and predict local coherence scores. the final document score is calculated by averaging its local scores. egrid cnn (egrid cnn , tien nguyen and joty, 2017): the model applies a cnn over egrid representations across groups of consecutive sentences; the cnn slides multiple filters of weights to extract feature maps that represent high-level entitytransition features, followed by a max pooling function to focus on the important features. furthermore, additional entity-related features are integrated such as salience, proper mentions and named entity type. ## binary discrimination task binary discrimination is a typical approach to assessing neural coherence models where a wellorganized document should be ranked higher than its permuted counterparts created by corrupting sentence order. following previous work, we train and test foot_2 the coherence models on the wsj foot_3 and evaluate them using pairwise ranking accuracy (pra), which is calculated based on the fraction of correct pairwise rankings between a coherent document and its incoherent counterparts. in ## cloze coherence (cc) dataset we compile a large-scale dataset, to which we refer as cloze coherence (cc), of coherent and incoherent examples, where the former are intact well-written texts while the latter are the result of applying syntactic or semantic perturbations to the coherent ones. ## controlled linguistic alterations (cla) dataset in order to further understand the properties of coherence that current coherence models capture, we manually construct a dataset of controlled sets of linguistic changes. we first identify a set of coherent, well-written texts of two consecutive sentences from business and financial articles in the bbc, the independent and financial times (this allows us to stay in the same domain as the one used for training the models -the wsj). we focus on sentence pairs where the subject of the first sentence is pronom-coherent example incoherent example from cloze swap incoherent example from cloze rand tyrese joined a new gym.the membership allows him to work out for a year. the membership allows him to work out for a year. tyrese joined a new gym. tyrese joined a new gym. as children they hated being dressed alike. jasmine doesn't know how to play the guitar. she asked her dad to take her to guitar class. she asked her dad to take her to guitar class. jasmine doesn't know how to play the guitar. jasmine doesn't know how to play the guitar. may thought her milk was no good. i wanted to play an old game one day. when i looked in the game's case the cd was missing. when i looked in the game's case the cd was missing. i wanted to play an old game one day. i wanted to play an old game one day. jason pressed the buzzer since he knew the answer. ## experiments we investigate domain shift effects and fine-tune the wsj-trained models on each of the cloze swap and cloze rand training sets (section 4) and reevaluate performance on the respective test sets. specifically, we use an mlp layer over the models' pre-prediction representation, followed by sigmoid non-linearity. the models are optimized using the mean squared error between the gold labels (0 or 1) and the predicted scores. 12 in this setup, only the mlp layer is fine-tuned and not the whole coherence model which allows us to create a fast efficient evaluation framework that can be applied as a further examination step after coherence models are developed and tuned on their respective datasets, instead of training the models from scratch. the results of the fine-tuned models are presented in in we furthermore observe that now egrid cnn is the second best model on cla random (pra 71.6). a sparser entity grid where entities in the two sentences are different allows the model to detect such cases (e.g. in the example in performance is overall low on lexical perturbations and corrupt pronouns which suggests that the models are not sensitive to minor lexical changes even if they result in implausible meaning and they also struggle to resolve pronominal references. l2 regularization, and a learnable penalty rate (search space {0.00001, 0.0001, 0.001, 0.01}). we use early stopping and stop training if pra does not improve on the dev set over 5 epochs (max epochs 200). mlp hidden unit size is 100. however, the exception is lcd bert (with pra 80 on lexical perturbations and 76.6 on corrupt pronoun) suggesting a better ability at capturing semantics and resolving references. across all six cla datasets ('all data'; ## probing coherence embedding space inspired by previous work @xcite , and to better understand the information that is encoded in the representations of coherence models, we investigate probing tasks that can capture coherence-related features. we experiment with the following set of sentence-level tasks that are relevant to discourse coherence: 1) the subject number (subjnum) task that detects the number of the subject of the main clause; 2) the object number (objnum) task that detects the number of the direct object of the main clause; 3) the coordination inversion (coordinv) task that contains sentences consisting of two coordinate clauses, where the two clauses are inverted in half of the sentences and kept intact in the other half (the task is to detect whether a sentence is modified or not); 4) the corrupt agreement (cor-ruptagr) task where sentences are corrupted by inverting the verb number (the task is to identify corrupted sentences). tasks 1, 2 and 4 align with centering theory as they probe for subject and object relevant information; the theory suggests that subject and object roles are indicators of entity salience. on the other hand, task 3 tests whether the models can capture intra-sentential coherence. for these tasks, we use the datasets from @xcite (tasks 1,2 and 3) and linzen et al. (2016) (task 4 ). probing model we adopt the senteval framework of @xcite . our probing model consists of an mlp layer over model sentence representations, followed by sigmoid non-linearity. we use the same training parameters as @xcite . 13 results across all tasks, we find that lcd bert achieves the best performance, outperforming all other approaches. we note, however, that lcd bert does not fine-tune its sentence representations during coherence training in the wsj but they are rather fixed and based on the average of bert-based word embeddings (section 2). this means the probing model fine-tunes averaged bert-based word embeddings rather than actual sentence parameters from the lcd coherence model. therefore, the level of performance observed is not representative of the maximum performance coherence models can achieve on these tasks. 15 we surmise that the comparatively lower performance observed with mtl bert and stl bert (whose sentence representations are fine-tuned during coherence training) is due to their coherence training objective. the models are optimized on the binary discrimination task, i.e. learning to rank a well-organized document higher than its permuted counterparts. this is an overly simplistic approach to coherence modeling that may be making models (and their representations) more susceptible to losing useful linguistic information. having said that, though, mtl bert , that has direct training signal with respect to the words' grammatical roles, is able to alleviate this issue to an extent and is the next best performing model on subjnum, objnum and corruptagr. across tasks, lc is the odd one out, and the worst performing model. this can be explained partly by its comparatively lower performance on the simpler binary discrimination task ( ## discussion our evaluation experiments on two coherence datasets reveal that rnn-or egrid-based coherence models are able to detect syntactic alterations that undermine coherence, but are less effecient at detecting semantic ones even after fine-tuning on the latter. we furthermore find that they particularly struggle with recognizing minor lexical changes even if they result in implausible meaning and resolving pronominal references. on the other hand, these models are particularly good at detecting cases where a prefix is inserted or the subject pronoun is substituted with a lexical item, suggesting that they are capable of capturing the relevant syntactic patterns and do not solely rely on positional features. we find that the best performing model overall is lcd bert which does not use an rnn sentence encoder but rather builds sentence representations by averaging bert embeddings then utilizes a number of linear transformations over adjacent sentences to facilitate learning richer representations. our probing experiments reveal that models are better at encoding information regarding subject and object number followed by verb number (cor-ruptagr). these probing tasks align with centering theory as they probe for subject and object relevant information. the task that tests for knowledge on coordination inversion is the lowest performing one overall, suggesting that there is little capacity at capturing information related to intra-sentential coherence. excluding lcd bert , mtl bert is the best performing model; nevertheless, there is still scope for substantial improvement across all probing tasks and particularly on coordinv and corrup-tagr. ## conclusion we systematically studied how well current models of coherence can capture aspects of text implicated in discourse organisation. we devised datasets of various kinds of incoherence and examined model susceptibility to syntactic and semantic alterations. our results demonstrate the models are robust with respect to corrupted syntactic patterns, prefix insertions and lexical substitutions. however, they fall short in capturing rhetorical and semantic corruptions, lexical perturbations and corrupt pronouns. we furthermore find that discourse embedding space encodes subject and object relevant information; however, there is scope for substantial improvement in terms of encoding linguistic properties relevant to discourse coherence. experiments on coordination inversion further suggest that current models have little capacity at encoding information related to intra-sentential coherence. we hope this study shall provide further insight into how to frame the task of coherence modeling and improve model performance further. finally, we make our datasets publicly available for researchers to use to test coherence models.