Jigsaw Pieces of Meaning: Modeling Discourse Coherence with Informed Negative Sample Synthesis (2024)

Abstract: Coherence in discourse is fundamental for comprehension and perception. Much research on coherence modeling has focused on better model architectures and training setups optimizing on the permuted document task, where random permutations of a coherent document are considered incoherent. However, there’s very limited work on creating “informed” synthetic incoherent samples that better represent or mimic incoherence. We source a diverse positive corpus for local coherence and propose six rule-based methods leveraging information from Constituency trees, Part-of-speech, semantic overlap and more, for “informed” negative sample synthesis for better representation of incoherence. We keep a straightforward training setup for local coherence modeling by fine-tuning popular transformer models, and aggregate local scores for global coherence. We evaluate on a battery of independent downstream tasks to assess the impact of improved negative sample quality. We assert that a step towards optimality for coherence modeling requires better negative sample synthesis in tandem with model improvements.

## introduction and motivation coherence is the bridge between elements of discourse which imposes strong logical connections, semantic relationships, smooth transitions, and thematic progressions. @xcite formally defined coherence as a text's interpretive unity through cohesion, introducing local and global coherence concepts, the former addressing connections between adjacent text units, while the latter looking at the broader discourse organization for a document. van @xcite additionally emphasizes the role of macrostructures and cognitive processes, going beyond mere textual properties. coherence modeling has been a fundamental task in discourse and pragmatics, with applications in text generation, dialogue systems, and reasoning, yet presents formidable challenges in modeling and a veritable lack of quality data. entity-based models @xcite capture patterns of entity distribution in text by focusing on the roles of salient entities @xcite . to this, tien nguyen and joty (2017) apply a neural approach using convnets. rhetorical structure theory (rst) based methods formalize coherence as discourse relations @xcite . @xcite feature recurrent layers to encode individual sentences within 3-sentence windows. @xcite use an encoder-decoder architecture to incorporate global topic information. @xcite model changes in salient semantic information. the transferable neural model @xcite focuses on local coherence, training forward and backward models on adjacent sentences, along with generative pre-training of sentence encoders. the unified coherence model, proposed by @xcite , is highly regarded for its impressive results, employing a siamese framework with a bilinear layer and lightweight convolution pooling. coherence models often learn and evaluate using a pairwise-ranking task on the wall street journal (wsj) corpus documents. an original document serves as a coherent "positive" sample, while its permuted version is the incoherent "negative" sample. the primary goal is to train models to predict a higher coherence score for the original than its random permutations and determine total accuracy. introduced by @xcite , the corpus and task have been prime sets for most research in modeling and evaluating coherence. @xcite assessed state-of-the-art models trained on the wsj permuted data. while the models excelled in the permuted document task, they struggled in downstream evaluations. @xcite note that success on the permuted document task doesn't fully reflect true coherence modeling abilities advocating for broader evaluations of these models. @xcite present the state-ofthe-art for the pairwise wsj task using an extensive contrastive setup that contrasts positive samples with permuted negatives via automatic hard negative mining to harness "harder" samples during training. this approach, leveraging hardmining negative samples during training, achieves improved results. @xcite adopted a different approach from random permutations, focusing on intruder-detection. to formulate incoherent documents using the cnn and wikipedia corpora, they substitute a sentence from a coherent document with a comparable sentence from a different document. through bigram hashing and tf-idf matching, they retrieve 10 similar documents, then choose a random non-opening sentence from these to create 10 potential replacements. they further refine the substitution using filters based on tf-idf similarity, thereby making an "informed" change that turns a positive document into a negative one. their findings indicate that fine-tuned transformer models excel at this task. based on this we propose that relatively straightforward training setups akin to document classification using pre-trained models and aggregation can yield comparative or better scores against prominent models for coherence, achieved by creating more sophisticated "informed" synthetic samples for incoherent data leveraging granular and nuanced syntactic and semantic text information, as opposed to the simpler data curations based on random permutations that many complex models and setups currently rely on. incoherent "negative" samples from six, rulebased-heuristic, "informed" negative data synthesis processes are crafted from a novel 3-sentence locally coherent "positive" text corpora obtained from diverse sources after a curated extraction process. these 3-sentence local windows are used to fine-tune transformer models, from which a simple aggregation method yields a global document coherence estimation system. this system is then evaluated on a battery of downstream evaluations and compared against prominent models. we achieve results comparable to state-of-the-art models trained explicitly on the wsj permutation training set, with fast convergence and significantly better performance on a logical coherence evaluation test. we conduct an ablation analysis examining the incoherent sample synthesis methods, followed by a discussion. our conclusion emphasizes the importance of nuanced incoherent data synthesis that mimics natural incoherence. scripts made available foot_0 (refer ethics statement). ## extracting coherent samples we select a 3-sentence window for our local coherence analysis @xcite . our locally coherent "positive" set is curated after an extraction and filtration process from four diverse sources of text: arxiv abstracts -arx -summaries of academic literature, wikipedia "good" -wki -articles tagged to be "good" on wikipedia foot_1 , roc stories -roc -short commonsense stories @xcite , book plots -bkp -book plot texts foot_2 . for roc we eliminate all samples that may have any overlap with the storycloze test which we evaluate on later (1571 samples). text from all sources is human-written. we iterate over and parse documents from each source into lists of sentences using a parser except for roc where sentences are pre-parsed. from these sentence lists, we extract three-sentence windows. every sentence undergoes cleaning to remove unicode errors and filter urls/tags. moreover, as an additional filtration heuristic, each sentence is evaluated for linguistic acceptability using the 'textattack/roberta-base-cola' model @xcite trained on cola @xcite . if a sentence in a window fails the check, the window is discarded. on average, 5.21% of windows per set are rejected. we ensure significant distance and no overlaps between windows from the same document. the detailed extraction process is explained in algorithm 1. the summary of the positive corpus is presented in algorithm 1 window extraction require: source files f src ensure: all other functions are defined 1: for f in f src do 2: for doc in f do 3: sents ← p arser(doc) 4: if len(sents) < 3 then continue 5: end if 6: split sents to groups (2 < @xmath0: ## negative samples we craft incoherent samples using six methods to perturb samples directly from the positive set, ensuring positive-negative samples remain within the same general space. m1 and m2 incorporate syntactic details from sentences to execute informed substitutions. they primarily focus on modifying the contextual and descriptive elements of the sentences: m1. constituency subtree substitution: subtree substitution has been an explored topic in the nlp predicament especially for data augmentation @xcite . we substitute prepositional phrases (pp), adjective phrases (adjp) and adverb phrases (advp) in positive sample sentences. by replacing the adjp, advp, or pp modifiers, we change the "where," "how," and "why" of a sentence, not the "who" or "what". using a neural constituency parser @xcite , we flatten the positive corpus, extract a subset, iterate over sentences, and form a dictionary of adjps, advps, and pps called bank (b). for a given sentence s and b with keys: adjp, adv p, and p p , if con-stituency parse tree structure s contains subtree with @xmath1, adv p, p p }, it generates a set of 5 candidate replacements s ′ @xmath2, s 2 , . . . , s 5 }, where each s i is a variant of s with the key text substituted from b[key]. the candidate s ′ is chosen such that s ′ = argmax i (acceptability(s i )) (acceptability is modeled similarly to the positive method). this process is applied to a maximum of two sentences in each positive window w , with the number of substitutions constrained by 1 ≤ @xmath3. a visual depiction is provided in figure 1(a) . ## downstream evaluations we test our document coherence modeling (dcm) approach on a battery of downstream taskindependent pairwise test sets similar to @xcite . these include the wsj test set, summeval annotated set @xcite , insted-cnn -insted-wiki sets @xcite and the storycloze test @xcite . we use a pairwise setup where the score of a positive sample is ranked against a negative one, measuring on total accuracy. pairwise comparisons are scale-invariant, they focus on relative score positions thus, despite varied task or dataset scales, the evaluation is consistent. we also test on the gcdc test sets @xcite for pairwise ranking and minority class prediction to compare with benchmarks and assess natural use cases. we compare against state-of-the-art baseline models with previously reported scores on these tasks: local coherence discriminator (lcd) model @xcite : (i) lcd-g with glove representations @xcite , (ii) lcd-i using infersent @xcite , and lcd-l from an rnn-trained language model; (unc) model @xcite and the contrastive and contrastive with hard-mined negatives (hmn) model @xcite . for gcdc we have the lexgraph @xcite , egraph @xcite , clique @xcite and sen-tavg, sentseq/parseq models from lai and tetreault. all these prominent models allow for a good comparison as they have exhibited excellent results on a myriad of downstream sets in the past. summeval: the summeval collection of human judgments of model generated summaries on the cnn dailymail dataset @xcite consists 1600 model generated summaries by 16 generation systems on 100 articles @xcite . each summary has coherence ratings from three expert annotators using a likert-like scale. @xcite adapts this to a pairwise setup pairing summaries for every system and unique source article. the summary with superior coherence becomes the positive document, while its counterpart is the negative one. this yields 16 2 × 100 = 12, 000 pairs for assessment. a constraint to consider is the notably low inter-annotator agreement (krippendorff's alpha -0.492 for workers, 0.413 for experts, improved to 0.712). story cloze test: this is an independent commonsense reasoning set proposed. following on @xcite , we assess models using the storycloze dataset @xcite . this dataset offers short narratives with two endings, one being implausible and logically incoherent. using the validation set (as test labels are private), we pair narratives with correct endings as positive and incorrect ones as negative, yielding 1,571 evaluation pairs. as outlined in section 2, any windows that contained even a single sentence from these test samples were removed from our roc set prior to training. insted: as introduced previously, the task presented by @xcite to assess the coherence abilities of pre-trained language models by detecting intruding sentences is again adapted to a pairwise setting. the pairwise framework pairs the original document with its corrupted incoherent gcdc: lai and tetreault (2018) provide a realworld text corpus to model coherence, the grammarly corpus of discourse coherence (gcdc), incorporating texts from the yahoo answers l6 corpus, clinton & enron mails corpora, and the yelp open dataset, with 200 test samples from each source. our evaluation delves into two primary tests of this dataset: sentence ordering (pairwise setting) and minority class prediction. the former follows a setting similar to the wsj evaluation (20 random permutations), specifically targeting texts with high coherence (gold rating 3). for sets yahoo, clinton, enron and yelp containing 76, 111, 88 and 108 positive samples respectively we get a total of 7660 test samples. the minority class prediction aims to categorize a subset where only 15-20% is labeled as low coherence. texts are designated "low coherence". the f 0.5 score, which favors precision over recall serves as the evaluation metric. echoing the patterns in summeval annotations, there's a discernible low inter-annotator ## conclusion and future work in this paper, we take a parallel approach to coherence modeling as opposed to optimization on the permuted document task by sourcing a diverse positive corpus and synthesizing "informed" incoherent samples from the positive corpus with six methods utilising constituency parse information, pos, semantic similarity and more. we perform local coherence model training using a simple finetuning setup and form a score aggregation method for global document coherence modeling. using this setup we test on multiple independent downstream tasks which capture some form on incoherence in the text. our nuanced approach to forming negative samples and obtaining scores results in getting comparable performance in the tasks (particularly standing out in a few) against many popular models and training setups developed for this task. the efficacy of our models in diverse evaluations, along with our findings, highlights the pivotal role of sophisticated, "informed" negative sample synthesis in advancing the field of coherence modeling. in the future, we plan to expand our scope by training more curated models on this training data such as contrastive models, siamese networks, and more. while these methods are designed to be domain-agnostic, there is an interest in exploring the nuances of incoherence within specific, contextrich discourse domains, such as the medical or legal fields, effectively investigating domain-specific incoherence. we're interested in exploring how generative techniques, such as gans or human-inthe-loop systems, can aid in producing incoherent samples and assist in mining hard negatives during the incoherent text generation phase. a multilingual angle for this can also be explored.