Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling (2022)

Abstract: Given the claims of improved text generation quality across various pre-trained neural models, we consider the coherence evaluation of machine generated text to be one of the principal applications of coherence models that needs to be investigated. Prior work in neural coherence modeling has primarily focused on devising new architectures for solving the permuted document task. We instead use a basic model architecture and show significant improvements over state of the art within the same training regime. We then design a harder self-supervision objective by increasing the ratio of negative samples within a contrastive learning setup, and enhance the model further through automatic hard negative mining coupled with a large global negative queue encoded by a momentum encoder. We show empirically that increasing the density of negative samples improves the basic model, and using a global negative queue further improves and stabilizes the model while training with hard negative samples. We evaluate the coherence model on task-independent test sets that resemble real-world applications and show significant improvements in coherence evaluations of downstream tasks.

## introduction coherence is a property of a well-written text that makes it different from a random set of sentences: sentences in a coherent text are connected in systematic ways such that each sentence follows naturally from previous ones and leads into the following ones @xcite . coherence models @xcite that can distinguish a coherent text from incoherent ones have a wide range of applications in language generation, summarization, and coherence assessment tasks such as essay scoring and sentence ordering. with recent advancements in neural methods, claims of fluency in summarization @xcite , language modeling @xcite , response generation @xcite and human parity in machine translation @xcite have led to calls for finergrained discourse-level evaluations @xcite @xcite , since traditional metrics such as bleu and rouge are unable to measure text quality and readability @xcite . coherence models that can evaluate machine-generated text have become the need of the hour. a majority of coherence models proposed optimize their learning objectives on the permuted document task using the penn treebank (wsj) corpus. an original article is considered a 'positive' sample of a coherent document, while a permutation of its sentences is considered a 'negative' or incoherent sample (see appendix a.1 for an example). models are usually trained in a pairwise ranking fashion to distinguish the two. the basic entity-grid model proposed by @xcite was extended to incorporate entity-specific features @xcite , multiple ranks @xcite , and coherence relations @xcite . their neural extensions have also been proposed @xcite . more recent state-of-the-art models like the transferable neural model @xcite consider coherence at a local level by training a forward and backward model only on adjacent sentences, in addition to generative pre-training of the sentence encoders. the unified coherence model @xcite uses bi-linear layer and lightweight convolution-pooling in a siamese framework to capture discourse relations and topic structures, along with an explicit language model loss to capture syntactic patterns. @xcite recently tested these state-of-the-art models by conducting coherence evaluations on the wsj permuted document task, machine translation, summarization and next utterance ranking tasks. they found that while models performed well on the permuted document task, when tested off-the-shelf, models generalized poorly to downstream evaluation tasks. they call for more comprehensive evaluations of coherence models. @xcite also reached a similar conclusion. they retrained several neural coherence models for tasks analogous to coherence modeling such as detecting connective substitution and topic switching. they found that performance on the permuted document task is only partially indicative of coherence modeling capabilities. in light of these recent findings, our aim is to propose a coherence model that generalizes well to downstream tasks. we train our model purely through self-supervision, without tailoring the model architecture specifically to the permuted document task or any other form of supervision. li and jurafsky (2017) point out that coherence models are exposed to a limited number of incoherent samples in the pairwise setup, since only a small sample of all possible incoherent permutations of a document are used to train models. learning with more negatives can better maximize the mutual information between their representations @xcite . by using a contrastive learning @xcite setup, where each 'positive' document is compared with multiple 'negative' documents, we increase the proportion of negative samples that the model is exposed to, and show that the coherence model shows significant improvements in performance. @xcite show that the difficulty of the negative samples used for contrastive training can strongly influence model success for visual representation learning. guided by this principle, we train the model with automatically mined hard negatives, coupled with a large global negative queue encoded by a momentum encoder @xcite . in summary, our contributions are: ## datasets to ensure that our coherence model is useful for evaluation in downstream applications, we use a selection of task-independent test sets that cover a variety of domains and genres, including machine generated text from summarization systems and language models. following @xcite , we also evaluate the models on a commonsense reasoning narrative dataset. we train (and validate) the coherence models on standard wsj data, while using the rest as "independent" test sets to indicate the generalizability of the trained models. all evaluations on downstream tasks are conducted in a pairwise setting to enable a fair comparison. ## conclusion we show empirically that increasing the ratio and quality of negative samples improves the generalizability of the coherence model. we also test our model on a wide-ranging collection of independent test sets that resemble downstream applications, including machine generated text, on which our model significantly outperforms the previous sota model. our work thus also sets a new evaluation standard for future research in coherence modeling. we open source our code base to encourage research in a new paradigm of coherence modeling.