Can Transformer Models Measure Coherence In Text: Re-Thinking the Shuffle Test (2021)

Abstract: The Shuffle Test is the most common task to evaluate whether NLP models can measure coherence in text. Most recent work uses direct supervision on the task; we show that by simply finetuning a RoBERTa model, we can achieve a near perfect accuracy of 97.8%, a state-of-the-art. We argue that this outstanding performance is unlikely to lead to a good model of text coherence, and suggest that the Shuffle Test should be approached in a Zero-Shot setting: models should be evaluated without being trained on the task itself. We evaluate common models in this setting, such as Generative and Bi-directional Transformers, and find that larger architectures achieve high-performance out-of-the-box. Finally, we suggest the k-Block Shuffle Test, a modification of the original by increasing the size of blocks shuffled. Even though human reader performance remains high (around 95% accuracy), model performance drops from 94% to 78% as block size increases, creating a conceptually simple challenge to benchmark NLP models.

## introduction in recent years, text generation applications, fueled by transformer models pre-trained on large datasets, have achieved dramatic results on a wide range of nlp tasks. these include gpt2 applied to story completion of fan fiction @xcite , the pegasus model @xcite improving state-of-the-art on ten summarization datasets in widely varying domains, and more recently gpt3 @xcite doing well on a diversity of tasks in a zero-shot setting. however, it is not clear how coherent the text generated by these models is. the computational linguistics literature holds many competing definitions of coherence in text; @xcite provide a useful brief summary of key competing theories. this work attempts to identify the absence of coherence, noting that a text might be composed of valid sentences when viewed independently, but when read sequentially, semantic relations are not well-supported. the nlp community has proposed models to measure coherence, as well as repeatable tasks to evaluate these models. in this paper, we outline these common tasks, and describe what we believe is a limitation in the framework of the most common task: the shuffle test. the shuffle test is a conceptually simple and reproducible task, in which a model must differentiate between an original text and a sentence-order shuffled version. because of its simplicity, we make the argument that the shuffle test should be viewed as a probe: a task on which models should be evaluated without directed supervision. prior work @xcite has shown that directly optimizing evaluation metrics such as rouge or bleu leads to inadequate models, exploiting weaknesses in the evaluation metrics. we show that this phenomenon occurs with the current application of the shuffle test in related work. to demonstrate the pitfalls, we finetune a roberta-large model -an architecture several or1059 2008 2010 2012 2014 2016 2018 2020 publication year 50 60 70 80 90 100 accuracy at shuffle test a b c d e f random guess best model performance figure 2: timeline of incremental accuracy improvements on the shuffle test on the wsj corpus. letters are for models described in section 2.2. we first outline prior work on tasks and models to measure textual coherence, then describe the framework for the zero-shot shuffle test, showing how to adapt common models to the setting, and finally propose a variation to the shuffle test that significantly increases the challenge for models, while not affecting human performance at the task. foot_0 2 tasks and models for coherence ## zero-shot shuffle test we now define specifically what factors need to be respected to satisfy the zero-shot setting. in the zero-shot shuffle test, the evaluated model must not be pre-trained, finetuned or modified using shuffled text. more specifically, this restricts the use of the shuffle test as supervision (as a binary classification task), as well as other tasks that involve shuffling text, such as the sentence ordering task. next, we adapt common architectures to the zero-shot shuffle test and assess performance in diverse textual domains. ## the k-block shuffle test results in section 3 can be interpreted to mean that with large enough transformer models, the shuffle test with no supervision is essentially a solved task. we find that a simple modification of the shuffle test can significantly reduce model performance, without affecting human annotator performance. the modification we propose, k-block shuffle test (kbst), is illustrated in figure 1 . in the standard shuffle test, text is divided into sentences and shuffled, with a unit of one sentence. in the k-block shuffle test, sentences are grouped in contiguous blocks of k sentences (resembling paragraphs), and the blocks are shuffled, maintaining sentence order in each block. within a block, sentences remain locally coherent, and as block size increases, the fraction of correct sentence transitions increases, while potentially incoherent transitions decrease. to establish the feasibility of the kbst with differing block-sizes, we performed a human evaluation completed by authors of the paper as well as a third annotator recruited on the upwork foot_1 platform. this annotator is a native english speaker with experience in proofreading, and was remunerated at $15/hour usd. the human evaluation consisted of performing kbst on 500 documents randomly sampled from the wsj dataset, with 100 tests for each block-size from one to five. each shuffle test was performed by at least two annotators. we find that there is high inter-annotator agreement (cohen's kappa κ = 0.86), which does not significantly vary with block-size (ranging from 0.76-0.94). kbst results for human and computational models are shown in timings logged during human annotation show that shuffle tests took on average 40% more time for larger (3-5) than smaller blocks (1-2), showing the task requires more attention from annotators as block size increases. in all three domains, increased block size leads to a decrease in model performance. the magnitude of decrease in performance from block-size 1 to 5 is sensitive to the domain, with a drop of 4% in the legal domain, and 36% for reddit, on which the 5-block performance of 53.8% narrowly outperforms random performance. aggregate model performance drops from 94.5% for block-size 1 to 77.9% for block-size 5, leaving significant room in larger block-size to measure future model improvements. although increasing the block size leads to a more challenging task for current models, we argue models should not be evaluated on a single block size, but on several block sizes, with each block size giving a perspective on the model's performance at a specific point between local and global coherence @xcite . coherence in long text. we limited our analysis to texts with up to 512 words, a common constraint in pre-trained transformers. recent progress in model architectures open the possibility to process longer text, with models such as the reformer @xcite , longformer @xcite and big bird @xcite processing sequences of several thousand words. with longer sequences, one can further increase the block-size of the k-blocked shuffle test (i.e., @xmath0, or main topics and subtopics @xcite . specialized coherence models. in this work, we limit our analysis to popular models out-ofthe-box, establishing baseline performances for the zero-shot kbst. future work should establish whether performance can be further improved, for instance using word-level likelihood signals and surprisal profiles. ## conclusion in this work, we discuss a potential limitation in the framing of the shuffle test, the most commonly used task to evaluate models for textual coherence. we show that a roberta model can be finetuned to achieve near-perfect performance without necessarily measuring coherence, and propose a new framework: the zero-shot shuffle test, in which direct supervision is disallowed. modern nlp architectures can achieve high performance out-ofthe-box in this zero-shot setting on a variety of textual domains. we find however that models struggle when we introduce a simple modification, k-blocking, to the shuffling strategy, with accuracy dropping from around 94% to around 78%. the k-block shuffle test in a zero-shot setting is a straightforward, reproducible task that can be used to benchmark future nlp architectures to measure coherence capabilities. for the human evaluation, we paid the annotator above the minimum wage, and do not release any personal identifiable information. we did not collect payment information and relied on a third party (upwork.com) for remuneration.