Model Criticism for Long-Form Text Generation (2022)

Abstract: Language models have demonstrated the ability to generate highly fluent text; however, it remains unclear whether their output retains coherent high-level structure (e.g., story progression). Here, we propose to apply a statistical tool, model criticism in latent space, to evaluate the high-level structure of the generated text. Model criticism compares the distributions between real and generated data in a latent space obtained according to an assumptive generative process. Different generative processes identify specific failure modes of the underlying model. We perform experiments on three representative aspects of high-level discourse—coherence, coreference, and topicality—and find that transformer-based language models are able to capture topical structures but have a harder time maintaining structural coherence or modeling coreference.

## introduction it is now broadly accepted that neural language models can consistently generate fluent text @xcite @xcite . yet, while large language models make few local word-level errors, human studies have shown that they still often make "high-level" errors such as incoherence, self-contradictions, and off-topic generations @xcite . we hypothesize that researchers have focused on local fluency partly because it is easy to automatically evaluate through metrics such as perplexity and n-gram matching. automatic assessment of high-level text generation quality has received less attention, partially because a single general-purpose metric does not exist. this work takes a step toward the automatic evaluation of the high-level structure of the generated text by applying a tool from statistics, model criticism in latent space @xcite . under this approach, we first project data to a latent space based on an assumptive generative process, and then compare the implied latent distributions between real data and language model samples. this approach unifies past work for evaluating text generation under a single framework, including existing dimensionality reduction techniques such as probabilistic pca @xcite , as well as previous applications of model criticism that were restricted to topic models @xcite . by making different assumptions in the underlying generative process, model criticism in latent space identifies specific failure modes of the generated language. we demonstrate this on three representative high-level properties of the generated discourse-coherence @xcite , coreference @xcite , and topicality @xcite )-as well as on a synthetic dataset for which the true data generating process is known. experiments using our proposed framework enable us to make four observations about modern language models. first, we find that it is possible for a model to get strong word-level perplexity, yet fail to capture longer-term dynamics. second, we find that the transformer language models perform poorly in terms of coherence, in line with previous observations @xcite @xcite , particularly when they do not have access to explicit lexical markers in the context. third, we show that transformer language models do not model coreference structures well. last, we show that transformer language models can capture topical correlations @xcite . all results, data, and code are publicly available at https://github.com/da03/ criticize_text_generation . ## model criticism in latent space model criticism @xcite quantifies the relationship between a data distribution p data (x) and a model p model (x) by comparing statistics over davies, also known by birth as the davies duck, is an american pop music duo consisting of jaleel brown, mike degagne, and john varnado that appeared on the rastafari album, afterburner.. brett butler and will butler, two high schoolers, met in elementary school, and went off-track to go to college. they both performed at local and international jazz clubs... brown and degagne formed the duo after brown's parents found out that their son would play trumpet. the duo released the album, ties that bind us,... on march 26, 2021, brown officially announced that he and degagne's baby were expecting. as of november 2021, the twins are still married. studio albums: ties that bind us (2016) ties that bind us ii (2020) abstract background background personal life discography sample infer criticize x x figure 1 : illustration of applying model criticism in latent space to evaluate discourse coherence. instead of wordlevel errors, we identify improper high-level section transitions (those that are rare in real data), as marked by red crosses. the article shown is generated by gpt-2 finetuned on wiki. see section 4 for more explanations. based on this generative process, the posterior distribution p c (z|x) projects x to the latent space. for a single data point x, we can evaluate the negative log-likelihood of the projected latent variables z ∼ p c (z|x) under the prior p c , given an arbitrary distribution over x, p x , we can take an expected negative log-likelihood, we term t c (p x ) the latent nll. 2 this value is the cross-entropy between the aggregated posterior 1 we discuss the difference between being likely in the latent space versus the observed space in appendix a. 2 when z is the same as x (pc(z|x) = 1[@xmath0, latent nll is the same as the negative log-likelihood of the language model samples under the data distribution @xcite . in practice, we cannot compute t (p x ) analytically due to the existence of the two expectations e x∼px(x) and e z∼pc(z|x) , but can approximate expectations using monte-carlo sampling. when z is a sequence of m discrete states, we define a metric latent ppl analogous to perplexity: with a critic chosen, we can compare p data (x) and p model (x) in the latent space by estimating and comparing t c (p data ) and t c (p model ). similar to a two-sample test @xcite , when p data and p model are the same, the statistics will also stay close. furthermore, with a powerful critic, t c (p model ) is meaningful by itself: a higher value means that model generations are less likely in the latent space, whereas a lower value implies that samples match the critic along the latent projection. 3 the approach can also be applied to individual points, t c (x), to identify outliers. how to select the critic p c choosing the critic p c is obvious only when we know the true latent variables and the generative process of data. in other cases, it depends on the data properties of interest. for example, if we want to criticize the topicality of text, we can use a topic model @xcite to induce a latent space over topics. note that the selected critic p c may underperform p model as a model of x, while still providing useful latent structures. by criticizing strong models, using simpler latent models that are designed to capture a particular aspect of text as p c , we provide a sanity check for the stronger model along a specific target axis. this property motivates the use of this approach with powerful, yet opaque models. the likelihood of z is evaluated using p c (z) to measure how likely the samples are in the latent space. ## a surprising text generation failure as a preliminary experiment, we show a language model with strong word-level perplexity that fails to capture simple long-term dynamics as demonstrated by model criticism. we assume that p data is known and follows a basic pattern. it has a latent high-level sequence of @xmath1, z 2 , . . . , z m where each state can take one of 256 possible values. these states are generated from a transition distribution at the observation level, each latent state z m generates a sub-sequence of words x m 1 , x m 2 , . . . , x m nm conditioned on z m from an emission distribution p (x m 1 , . . . , x m nm |z m ). we also restrict the model so that each sub-sequence can only come from one latent state. the observed sequence is the concatenation of all sub-sequences. the joint distribution of the latent states and the tokens forms: with this generative process, we sample a dataset. 5 we apply a transformer language model as p model and train it on this dataset. given the simplicity of the generative process and the small vocabulary size, we expect this model to do quite well. and in fact we do see that the model achieves a strong perplexity of 2.28, which nearly matches the true data p data perplexity of 1.99. model criticism gives a different method for quantifying model fit. since the true data generating process is known, we can directly use p @xmath2. to project an observation x to the latent space, we need to perform posterior inference p c (z|x). by construction, this mapping is deterministic, since each sub-sequence comes from a unique latent state (see appendix d for details). we then apply model criticism t c by sampling a sequence of transformer outputs, mapping them to a sequence of latent states, counting to compute the aggregated posterior, and then comparing to the known prior. this process is shown in figure 3 . ## critiquing discourse coherence text generation from large language models rarely leads to local fluency errors, but there is evidence of failures like those in the previous section @xcite @xcite . in this section, we apply model criticism to assess discourse coherence @xcite of large lms. we study this through an experiment on generating long-form documents divided into explicit sections. while we do not know the true data generating process, knowing the distribution of section types allows us to assess the latent structure of lm generations. figure 1 illustrates the experiment. here, an lm generates an article. each word transition is fluent, but the system makes two section transition errors: first, it generates two sections of type "background"; second, it generates a section of type "personal life" following the last "background" section, with both transitions being unlikely in the data. 6 we aim to separate the evaluation of these high-level coherence errors from word-level errors. to apply model criticism, we posit a simple critic generative process to capture the section changes. we adapt a hidden semi-markov model (hsmm) which is commonly used to represent segmentations of this form. specifically, the high-level latent variables z 1 , . . . , z m foot_5 model transitions among section types and the bottom level generates text conditioned on the current section type: we can then evaluate on datasets with known (ground truth) section titles and use these section titles as z. we use three english datasets pubmed, arxiv, and wiki @xcite . 8 we compare two language modeling settings, one trained with all section titles removed ("w/o title") and one with section titles before each section ("w/ title"), since we hypothesize that the existence of explicit section type markers might help the model learn the dynamics, inspired by @xcite and @xcite . sections are separated by a special marker, and a special end-of-sequence symbol is used to mark the end of the generation. since all three datasets are relatively small (especially considering that we use them to generate entire articles), we leverage pretrained language models gpt-2 small (lm 1 ) @xcite , and gpt-neo small (lm 2 ) @xcite which is trained on a more diverse dataset @xcite . we finetune these lms for p model . to generate, we sample from the language model until we hit the end-of-sequence symbol. no tempering/truncation @xcite is used during sampling, since we are more interested in the learned distribution rather than its mode here. for the "w/ title" setting, we discard the generated section titles in a postprocessing step. to infer the section types for a generated article, we need to approximate posterior inference to compute t c . we make a simplifying assumption that the posterior section title of each section only depends on its corresponding text: figure 4 : section transition errors on wiki, where each edge is labeled with the difference between p (z m |z m-1 ) of lm 1 (w/o title) and of data, and its width is proportional to the absolute difference. red marks unlikely transitions (p c (z m |z m-1 ) < 0.05). for clarity, we only show the top 20 section titles and remove singletons. ## critiquing coreference chains coreference tracks how multiple mention expressions are used to refer to the same underlying entity @xcite . while coreference represents a ubiquitous and important discourse-level phenomenon @xcite , there is evidence that large neural language models make elementary coreference mistakes @xcite , such as referring to non-existent discourse entities @xcite . in this experiment, we compare the coreference chain @xcite ) distributions be-original text ... [lisa] 0 runs off to find @xcite 1 and [they] 2 kiss passionately. afterwards, [josh] 1 tells [her] 0 the reason why [he] 1 's going to [their] 2 first gig, and that [lisa] 0 is going to do it, too... tween real data and lm generations. a coreference chain consists of a sequence of coreferent mentions. to simplify the representation, we use gender features to replace non-pronominal tokens, as illustrated in figure 5 . 10 presumably these latent chains should be similar in generated text and in real data. for the critic p c , we use a 5-gram language model with kneser-ney smoothing @xcite over chains. to infer z, we use an off-theshelf coreference resolution tool. 11 to avoid data sparsity issues, we relabel entity clusters within each n-gram. we apply model criticism to compare real data and lms trained on wiki (w/ title), after filtering data to only consider articles about films since they contain richer reference structures. results data lm 1 lm 2 6.26 7.22 6.93 lastly, we evaluate whether scaling model size improves coreference modeling. the results are summarized in ## critiquing topic correlations topical structure is another important aspect of long-form document generation @xcite . certain topics are more likely to appear together, for example, a document containing a topic related to "poets" is more likely to also contain one related to "publisher" relative to one related to "football". a text generation model should capture these topical relations. for this experiment, we again sample documents from the trained language model p model . specifically, we utilize the transformer-based lms trained on the datasets in section 4 (w/o title). to explore the topical structure in the generated documents, we need a critic p c . while lda @xcite is the most commonly used generative process for topic modeling, the dirichlet prior does not explicitly model topic correlations in documents. we therefore use the correlated topic model (ctm) specifically designed to model topical structures @xcite . model criticism will then compare the latent space of the real data with the generated texts. for each document, a ctm with m topics first generates a topic coefficient latent variable @xmath3; µ, σ). each coefficient of z can be interpreted as the "strength" of a topic in a document, so the covariance matrix σ captures the correlations among different topics. these weights z are then normalized using a softmax function, the result of which is used to parameterize the distribution over topic t n for the n-th word. each topic t n induces a categorical distribution over word types p (x n |t n ) = φ tn,xn , where φ ij parameterizes the probability of emitting word type j conditioned on topic i. the joint probability of a document with n words is: since we are only interested in criticizing the document-level z, we marginalize out the topic assignments of individual words: since analytical posterior inference is intractable, we use variational inference to estimate p c (z|x). results these results indicate that topic correlation is well represented in text generation systems, and is likely an easier task to model than ordered coherence. ## related work text generation evaluation traditional evaluation metrics include perplexity, and n-gram overlap metrics for translation-type problems such as bleu @xcite , rouge @xcite , meteor @xcite , and nist @xcite . in recent years, with the emergence of neural models that learn contextual representations @xcite , researchers propose to project text to contextual representations and compute distance in this space @xcite @xcite . the closest work to ours is eikema and aziz (2020), which evaluates different decoding strategies in machine translation by comparing the statistics of the produced text. while these past works mainly concern word-level string/meaning representation matching, the goal of our work is to check the high-level aspects of the generated text such as coherence. besides, word-level matching is not suitable for evaluating open-ended generation tasks due to the existence of too many plausible references @xcite , while our work projects text to a more manageable lowerdimensional latent space to make the evaluation of open-ended generation feasible. ## limitations one limitation of the proposed approach is its reliance on choosing a critic generative process p c , which presumes some knowledge of a true data generating process. for an improperly specified critic, it does not expose the latent space that we intend to criticize. however, since we compare statistics between real data and model generations (similar to two-sample tests), for a good model the statistics should be close even with improper critics. another limitation is that not observing any differences does not imply that the model generations conform to the unknown data distribution-it simply means that they are close with regard to the latent aspects that we criticize @xcite . recently, researchers found that certain capabilities such as reasoning under augmented prompts only emerge in large lms beyond tens of billions of parameters @xcite . since the largest lm tested in this paper only has 1.5 billion parameters, future work is required to investigate whether the high-level issues observed in this paper can be solved by further scaling model size. ## conclusions we consider the problem of evaluating long-form text generation for specific discourse properties. we propose a statistical tool, model criticism in latent space, which projects text to a latent space based on an assumptive generative process, and compares the implied latent distribution. different critic generative processes focus on different properties of data. we apply this tool to analyze three representative document properties: coherence, coreference, and topicality, using transformerbased language models. experiments find that while transformer lms can capture topical structures well, they are not currently strong at modeling discourse coherence without explicit markers or at modeling coreference.