To Burst or Not to Burst: Generating and Quantifying Improbable Text (2023)

Abstract: While large language models (LLMs) are extremely capable at text generation, their outputs are still distinguishable from human-authored text. We explore this separation across many metrics over text, many sampling techniques, many types of text data, and across two popular LLMs, LLaMA and Vicuna. Along the way, we introduce a new metric, recoverability, to highlight differences between human and machine text; and we propose a new sampling technique, burst sampling, designed to close this gap. We find that LLaMA and Vicuna have distinct distributions under many of the metrics, and that this influences our results: Recoverability separates real from fake text better than any other metric when using LLaMA. When using Vicuna, burst sampling produces text which is distributionally closer to real text compared to other sampling techniques.

## introduction in recent years, large language models (llms) have risen as the top performing category of models for many tasks in natural language processing. pretrained on hundreds of millions of examples of text, these models learn probability distributions over the next token in a sequence, and these probability distributions can be used to generate text. however, while these distributions accurately describe token distributions at the corpus level, they still lead to generations which are distinguishable from human text. in this work, we explore sampling techniques used to generate text and metrics used to evaluate such text, under the lens of differences between human and machine authorship. our contributions in this work are threefold. (1) we introduce a new sampling technique called burst sampling, which is designed to generate text that is statistically more similar to human-authored text than other techniques. a description of this figure 1 : illustration of our contributions. a a probability-ordered token distribution with a long tail, as is commonly seen when sampling from llms. sampling strategies often truncate the majority of this distribution (red line), leading to many possible tokens that cannot be sampled. when analyzing real text with an llm, we define recoverability as the fraction of tokens that occur before this cutoff. b our burst sampling technique first learns a distribution over bins of token ranks. to sample, a bin is first selected. c the probabilities of tokens within the selected bin are renormalized to form a new distribution, which is sampled. these models share a similar formulation. generation of a sequence x from a probabilistic clm with parameters θ can be defined as below: p (x 1:t ; θ) = t @xmath0:t-1 ) where x t is the next token, conditioned on a previous token sequence of length t -1. ## discussion our study over many metrics and sampling strategies, including burst sampling and recoverability, uncovered some interesting data points. a standout trend was the difference between llama and vicuna. our hypothesis is that vicuna sometimes produces probability distributions that are more front-weighted than llama (the first portion of the sorted distribution carries more probability mass than in llama). this is supported by our perplexity results-vicuna has very low perplexity for its own generations, but high perplexity for real text. this is to be expected for a fine-tuned model that is no longer intended for general purpose nlp tasks, and it is nice to find this reflected quantitatively. we found that burst sampling is especially helpful for vicuna, and we believe it acts as a correcting mechanism to vicuna's overconfidence. for llama, which has a less-skewed distribution, our burst sampling is less effective since it is trained to match the distributions of real text (the same objective as llama). thus, for a purely pre-trained network, it would be a good idea to increase (or somehow calibrate) the stochasticity of burst sampling. recoverability is also impacted by model differences. since we test recoverability with @xmath1, we could expect that this encompasses much more probability mass in vicuna than in llama. therefore, more of vicuna's generations are recoverable, and less separable from real text. however, when using llama, more tokens fall beyond this threshold and it is easier to use recoverability as a separating metric for real and generated text. we were surprised that recoverability worked well even for sampling methods other than top-k, and believe it has merit as a general metric. finally, although we initially introduced logistic regression over all metrics to test the effectiveness of burst sampling, we actually found it to be a very reliable synthetic text classifier, especially on vicuna. we would recommend that future synthetic text detection platforms consider this method as part of their system. our analysis also included our burstiness metrics (k, p, and top-p burstiness), which could certainly be further optimized for text detection by considering the entire pattern of rank or probability over a text sequence, and using timeseries classification to detect fluctuations that are more or less natural. this would perhaps motivate stronger implementations of our burst sampling, to consider the time dynamics of when improbable tokens are introduced. ## conclusions and future work we contributed a study of common metrics over many datasets and sampling techniques, using a pretrained model (llama) and a fine-tuned model (vicuna). within this study we tested novel ideas of recoverability and burst sampling, which illuminated many interesting future directions for studying the differences between human-authored and machine generated text. we found recoverability and a logistic regression over all metrics to be promising for detecting synthetic text. future work could further test the applicability of these results to more models and sampling methods, in particular exploring how recoverability could be used for fine-tuned models. we also found burst sampling to somewhat correct for distributional differences in vicuna, but certainly not completely. future work should look to calibrate and amplify burst sampling, as well as look to characterize the probability distributions between pre-trained and fine-tuned models in more detail.