Evaluating Document Coherence Modeling (2021)

Abstract: While pretrained language models (LMs) have driven impressive gains over morpho-syntactic and semantic tasks, their ability to model discourse and pragmatic phenomena is less clear. As a step towards a better understanding of their discourse modeling capabilities, we propose a sentence intrusion detection task. We examine the performance of a broad range of pretrained LMs on this detection task for English. Lacking a dataset for the task, we introduce INSteD, a novel intruder sentence detection dataset, containing 170,000+ documents constructed from English Wikipedia and CNN news articles. Our experiments show that pretrained LMs perform impressively in in-domain evaluation, but experience a substantial drop in the cross-domain setting, indicating limited generalization capacity. Further results over a novel linguistic probe dataset show that there is substantial room for improvement, especially in the cross- domain setting.

## introduction rhetorical relations refer to the transition of one sentence to the next in a span of text @xcite . they are important as a discourse device that contributes to the overall coherence, understanding, and flow of the text. these relations span a tremendous breadth of types, including contrast, elaboration, narration, and justification. these connections allow us to communicate cooperatively in understanding one another @xcite . the ability to understand such coherence (and conversely detect incoherence) is potentially beneficial for downstream tasks, such as storytelling @xcite , recipe generation @xcite , document-level text generation @xcite , and essay scoring @xcite . however, there is little work on document coherence understanding, especially examining the capacity of pretrained language models (lms) to model the coherence of longer documents. to address this gap, we examine the capacity of pretrained language models to capture document coherence, focused around two research questions: we propose the sentence intrusion detection task: (1) to determine whether a document contains an intruder sentence (coarse-grained level); and (2) to identify the span of any intruder sentence (fine-grained level). we restrict the scope of the intruder text to a single sentence, noting that in practice, the incoherent text could span multiple sentences, or alternatively be sub-sentential. existing datasets in document coherence measurement @xcite @xcite @xcite are unsuitable for our task: they are either prohibitively small, or do not specify the span of incoherent text. for example, in the dataset of @xcite , each document is assigned a coherence score, but the span of incoherent text is not specified. there is thus a need for a large-scale dataset which includes annotation of the position of intruder text. identifying the span of incoherent text can benefit tasks where explainability and immediate feedback are important, such as essay scoring @xcite . in this work, we introduce a dataset consisting of english documents from two domains: wikipedia articles (106k) and cnn news articles (72k). this dataset fills a gap in research pertaining to document coherence: our dataset is large in scale, includes both coherent and incoherent documents, and has mark-up of the position of any intruder sentence. figure 1 is an example document with an intruder sentence. here, the highlighted sentence reads as though it should be an elaboration of the previous sentence, but clearly exhibits an abrupt change of topic and the pronoun it cannot be readily resolved. this paper makes the following contributions: (1) we propose the sentence intrusion detection task, and examine how pretrained lms perform over the task and hence at document coherence understanding; (2) we construct a large-scale dataset from two domains-wikipedia and cnn news articles-that consists of coherent and incoherent documents, and is accompanied with the positions of intruder sentences, to evaluate in both in-domain and cross-domain settings; (3) we examine the behavior of models and humans, to better understand the ability of models to model the intrinsic properties of document coherence; and (4) we further hand-craft adversarial test instances across a variety of linguistic phenomena to better understand the types of incoherence that a given model can detect. ## related work we first review tasks relevant to our proposed task, then describe existing datasets used in coherence measurement, and finally discuss work on dataset artefacts and linguistic probes. ## models we model intruder sentence detection as a binary classification task: each non-opening sentence in a document is concatenated with the document, and a model is asked to predict whether the sentence is an intruder sentence to the document. our focus is on the task, dataset, and how existing models perform at document coherence prediction rather than modeling novelty, and we thus experiment with pre-existing pre-trained models. the models are as follows, each of which is fed into an mlp layer with a softmax output. bow: average the word embeddings for the combined document (sentence + sequence of sentences in the document), based on pretrained 300d glove embeddings trained on a 840b-token corpus @xcite . bi-lstm: feed the sequence of words in the combined document into a single-layer 512d bi-lstm with average-pooling; word embeddings are initialized as with bow. infersent: generate representations for the sentence and document with infersent @xcite , and concatenate the two; infersent is based on a bi-lstm with a max-pooling layer, trained on snli @xcite . skip-thought: generate representations for the sentence and document with skip-thought @xcite , and concatenate the two; skip-thought is an encoder-decoder model where the encoder extracts generic sentence embeddings and the decoder reconstructs surrounding sentences of the encoded sentence. bert: generate representations for the concatenated sentence and document with bert @xcite , which was pretrained on the tasks of masked language modeling and next sentence prediction over wikipedia and bookscorpus @xcite ; we experiment with both bert-large and bert-base (the cased versions). roberta: generate representations for the concatenated sentence and document with roberta @xcite , which was pretrained on the task of masked language modeling (dynamically masking) and each input consisting of continuous sentences from the same document or multiple documents (providing broader context) over cc-news, openwebtextcorpus, and sto-ries (trinh and le, 2018), in addition to the same data bert was pretrained on; we experiment with both roberta-large and roberta-base. albert: generate representations for the concatenated sentence and document with albert @xcite , which was pretrained over the same dataset as bert but replaces the next sentence prediction objective with a sentence-order prediction objective, to model document coherence; we experiment with both albert-large and albert-xxlarge. xlnet: generate representations for the concatenated sentence and document with xlnet @xcite , which was pretrained using a permutation language modeling objective over datasets including wikipedia, bookscorpus, giga5 @xcite , clueweb 2012-b @xcite , and common crawl; we experiment with both xlnet-largeand xlnet-base (the cased versions). although xlnet-large is used in removing data artefacts when selecting the intruder sentences, our experiments suggest that the comparative results across models (with or without artefact filtering) are robust. ## linguistic probes to further examine the models, we constructed a language probe dataset. ## conclusion we propose the new task of detecting whether there is an intruder sentence in a document, generated by replacing an original sentence with a similar sentence from a second document. to benchmark model performance over this task, we construct a large-scale dataset consisting of documents from english wikipedia and cnn news articles. experimental results show that pretrained lms that incorporate larger document contexts in pretraining perform remarkably well in-domain, but experience a substantial drop cross-domain. in follow-up analysis based on human annotations, substantial divergences from human intuitions were observed, pointing to limitations in their ability to model document coherence. further results over a linguistic probe dataset show that pretrained models fail to identify some linguistic characteristics that affect document coherence, suggesting room to improve for them to truly capture document coherence, and motivating the construction of a dataset with intruder text at the intra-sentential level.