Know When To Stop: A Study of Semantic Drift in Text Generation (2024)

Abstract: In this work, we explicitly show that modern LLMs tend to generate correct facts first, then “drift away” and generate incorrect facts later: this was occasionally observed but never properly measured. We develop a semantic drift score that measures the degree of separation between correct and incorrect facts in generated texts and confirm our hypothesis when generating Wikipedia-style biographies. This correct-then-incorrect generation pattern suggests that factual accuracy can be improved by knowing when to stop generation. Therefore, we explore the trade-off between information quantity and factual accuracy for several early stopping methods and manage to improve factuality by a large margin. We further show that reranking with semantic similarity can further improve these results, both compared to the baseline and when combined with early stopping. Finally, we try calling external API to bring the model back to the right generation path, but do not get positive results. Overall, our methods generalize and can be applied to any long-form text generation to produce more reliable information, by balancing trade-offs between factual accuracy, information quantity and computational cost.

## introduction differently from the earlier approaches to generating natural language with explicit content planning @xcite , modern autoregressive language models make predictions token-by-token, without pre-established text structure. one of the consequences of this methodological shift is that newer models lack the capability of maintaining high-level structure throughout generation and overly focus on local coherence. this was noted in the form of repetition @xcite and semantic drift @xcite . the term "semantic drift" emerged to describe the decrease in text generation quality when increasing generation length and has been classified as a sub-type of hallucinations @xcite . before that, semantic drift (or topic shift) was briefly mentioned when talking about question generation @xcite and story generation @xcite . in factual evaluation, recent works also mention a decline in factual accuracy for longer generations @xcite . while quality decrease for longer generations hints at specific order in generation quality (high-quality first, low-quality later), this ordered behavior has not been neither formally defined nor thoroughly studied and measured. in this work, we refer to "semantic drift" as the strength of the order in generation quality and, for the first time, provide tools for understanding this phenomenon. we propose to measure semantic drift by considering the change in truthfulness of a sequence of facts when a model generates a fact-rich text around a topic. intuitively, we measure the degree of separation between correct and incorrect facts in a paragraph: if the model starts by generating correct facts and then switches to systematically generating incorrect ones, we consider this as a semantic drift. to quantify the severity of semantic drift, we use the factscore task which provides correct/incorrect labels for individual facts @xcite . we find that, indeed, several llama2 variants have high semantic drift score: they tend to generate correct facts first, then "drift away" from the topic and generate incorrect facts later. this correct-then-incorrect separation suggests that factual accuracy can be improved by stopping generation early. we show that even a simple method that encourages generating eos leads to large improvements in factuality. we then propose to use resample-then-rerank pipelines where for each sentence, we generate several versions and choose the best based on sentence similarity measures. compared to the baseline, this improves factual accuracy by almost 10% (without shortening texts as with early stopping). this can also be combined with early stopping and allows for different informativeness-vs-factuality trade-offs. finally, we ask: if the model drifts away during generation, could it be brought back onto a correct path by calling an external api? sadly, this does not give noticeable improvements (at least, when working in the previously established settings). overall, we: our methods offer a practical compromise, balancing computation with performance, and build a foundation for further research. importantly, they are directly applicable to any probabilistic autoregressive language models. ## definition of semantic drift since the term "semantic" drift has been used with various meanings, we felt the need for a unifying definition, which we state below. semantic drift describes the phenomenon wherein generated text diverges from the subject matter designated by the prompt, resulting in a growing deterioration in relevance, coherence, or truthfulness. semantic drift results in a loss of three textual characteristics (see examples in appendix a): ## identifying semantic drift in our experiments, we rely on the factscore task @xcite . this task identifies all aspects of semantic drift and scores individual facts from a text as either correct or incorrect. a fact is correct if it is supported by external knowledge and therefore truthful. since two facts that contradict each other cannot be simultaneously correct, correct facts are also coherent. moreover, facts are verified in context, meaning that a fact is correct only if it is relevant to the context. appendix a shows examples of scoring for semantic drift types. ## factual accuracy and uncertainty knowing that the model is largely correct up to some point (to be precise, the semantic drift point) and largely wrong after gives us an opportunity to drift type examples repetition suthida is the queen of thailand. she is the wife of king vajiralongkorn. she was born on 3 june 1978. she is the first wife of king vajiralongkorn. she was born in hat yai, songkhla, thailand. she is the daughter of saman bajrasudha and his wife, nui. she has two brothers and one sister. she is a former flight attendant. she is a former actress. she is a former model. she is a former beauty queen. she is a former miss thailand. she is a former miss universe. she is a former miss world. making up information steve cummins is a professional rugby league footballer of the 1990s and 2000s. a great britain international representative centre, he has played in the super league for the wigan warriors and the warrington wolves, and in the nrl for the brisbane broncos and the cronulla-sutherland sharks. plausible information matthew perry is an american actor and comedian. he is best known for his role as chandler bing on the nbc television sitcom friends, which ran from 1994 to 2004. matthew perry was born in williamstown, massachusetts, on august 19, 1969. his mother, suzanne marie morrison, is a canadian journalist and former press secretary to canadian prime minister pierre trudeau. his father, john bennett perry, is an american actor and former model. perry has two older sisters, caitlin and emily. perry was raised in ottawa, ontario, and attended rockcliffe park public school and ashbury college. he then studied at the university of southern california, where he was a member of the sigma chi fraternity. ## mitigating factual errors as we explained above, the presence of semantic drift suggests that factual accuracy can be improved within the same generated paragraphs simply by shortening them. therefore, we first consider several criteria to stop generation early. next, we try resample-then-rerank pipeline, as well as calling api tools. we compare these methods through the lens of factuality vs informativeness trade-off (figure 4 ). ## beyond biographies the methods we presented can in principle be applied to any type of text generation, not just biographical paragraphs. to showcase these capabili- @xcite . pipeline. the original goal of the pipeline is to measure factuality of a summary with respect to reference text. we re-purpose it to measure factuality of generated text with respect to the original wikipedia article. we retrieve 5000 english wikipedia articles 6 and calculate mean factscore, triplescore, qag and rouge score. evaluation. factscore and triplescore extract triplets (closed-and open-scheme, respectively) and score the overlap of these triplets between the reference and generated texts. for qag score, the module generates question-answer pairs based on the generated text, attempts to answer the questions based on the reference text and notes the number of identical answers. rouge calculates the similarity between the two texts based on ngrams matches. together, all these scores paint a picture of the factuality of the generated text with respect to the wikipedia article. we note that none of these metrics consider recall, but that we provide the average number of tokens generated per paragraph as a measure of information quantity. results: reranking helps again. the fact that the presented methods are robust to various metrics underscores their generality. ## additional related work factual precision. recent surveys @xcite @xcite show that factuality evaluation has mostly been focused on short-form question answering, and improvements have largely been based on learning (pre-training, fine-tuning) or retrieval augmentation. previous work @xcite attempts decoding-time enhancements, but reports these alone achieve factuality on-par with greedy decoding and concludes the need for training enhancements. concurrent work @xcite contrasts various layers' logits. as opposed to sc-bertscore methods, this requires access to model's internals and changes to inference code; futhermore it is not evaluated on long-form generation and restricted to one model class. unlike other factuality enhancements, our methods do not directly fix incorrect facts, but use the semantic drift idea to inform when the model has "ran out of correct facts". they can be combined with any others to generate accurate and relevant text. semantic drift. deng et al. (2022); cho et al. (2019) characterize it linguistically via selfconsistency, not truthfulness. plausible and naturally flowing text would not be identified as drift. ## conclusion by measuring the degree of separation between the correct and incorrect facts in the generated texts, we show that llms largely generate correct facts first and incorrect later. this lead us to methods that improve factual accuracy by stopping generation early. we show that even a simple method that encourages generating eos leads to large improvements. this can further be improved by using a resample-then-rerank pipelines where for each sentence, we generate several versions and choose the best based on sentence similarity measures. overall, our methods offer a practical compromise, balancing computation with performance, and build a foundation for further research. importantly, they are directly applicable to any probabilistic auto-regressive language models. suitable tasks. even though our methods can be applied to any long-form text generation task, they are perhaps most relevant for tasks where factual accuracy is paramount (such as long form question answering or factual text generation). earlystopping methods specifically are more suitable for tasks where generating false information is more harmful than not generating it (for example giving false medical advice). our oracle for earlystopping removes 92% of incorrect facts from the generated text, but this comes with the cost of removing 58% of correct facts. these measurements (as can be seen in appendix f.1) should be used for individual applications to debate trade-offs. textual diversity. as this study is focused on factually-dense text, we did not take into account diversity of generated text, which may be relevant for more creative tasks such as story generation. for early stopping via sentence similarity, we chose to use selfcheck-bertscore which is sensitive to stylistic variations, as well as factual variations. however, there is no reason for which this metric cannot be replaced with other sentence similarity-metrics which account for style, thus retaining the creative factor of text generation.