Z ipf’s and Heaps’ Laws for Tokens and LLM -generated Texts (2025)

Abstract: The frequency distribution of words in human-written texts roughly follows a simple mathematical form known as Zipf’s law. Somewhat less well known is the related Heaps’ law, which describes a sublinear power-law growth of vocabulary size with document size. We study the applicability of Zipf’s and Heaps’ laws to texts generated by Large Language Models (LLMs). We empirically show that Heaps’ and Zipf’s laws only hold for LLM-generated texts in a narrow model-dependent temperature range. These temperatures have an optimal value close to t=1 for all the base models except the large Llama models, are higher for instruction-finetuned models and do not depend on the model size or prompting. This independently confirms the recent discovery of sampling temperature dependent phase transitions in LLM-generated texts.

## introduction zipf's law is a simple and long-known dependency between the frequency rank r of a word and its frequency in a natural corpus f (r) @xcite @xcite : similar power laws occur in many other areas of science, often quite distant from language (for an overview, see @xcite and @xcite ). the more intriguing is, as we show in this work, that zipf's and heaps' laws rarely hold for the texts generated by llms. specifically, we study the applicability of zipf's and heaps' laws to llm-generated texts and make the following contributions: in this section, we go on to introduce the key concepts. in section 2 we briefly review the previous research. in section 3 we confirm that the statistics for tokens in natural texts closely follow the statistics for words. in section 4, we study the applicability of zipf's and heaps' laws to llmgenerated texts. ## prior research studies of the zipfian properties of texts generated by language models are not entirely new. in particular, @xcite and @xcite have studied the zipfian properties of texts generated by ngram-and lstmbased models. @xcite studied the dependence of the zipfian coefficient on the gpt-2 decoding strategy, and this dependence was rather weak. these works studied language models that are now obsolete, mostly have architectures different from the now dominant transformers, and with no relation to the sampling temperature. the related phenomena of power-law autocorrelations decay in texts generated by language models have been studied by @xcite tanaka-ishii (2017); shen (2019); takahashi and tanaka-ishii (2019); lippi et al. (2019); mikhaylovskiy and churilov (2023). nakaishi et al. (2024) and bahamondes (2023 ) independently pioneered the application of the correlation-based phase transition apparatus to llm-generated texts and discovered phase transitions dependent on sampling temperature. @xcite have studied the phenomenon in mure detail following the approach of @xcite . our work merges these two lines of research, discovering sampling temperature-dependent phase transitions in zipfian properties of llm-generated texts. 3 zipf's and heaps' laws for tokens in natural texts llms are used to generate diverse sequences in natural and programming languages. at higher temperatures, they generate gibberish that cannot be reasonably interpreted as words. thus, if we want to study statistical properties of llm-generated texts at higher temperatures, we can hardly use words as base units. tokens are a natural choice. zipf's and heaps' laws have been studied primarily for words, and tokenizing text into different units can potentially yield different statistical relationships. thus, we need to verify that zipf's and heaps' laws hold for llm tokenizations of humanwritten texts: while splitting texts into 'words' on arbitrary symbols (for example, 'e') does recover a near-zipfian distribution on these artificial 'words', the exponent α = 0.77 deviates significantly from the usual values @xcite . we verify zipf's and heaps' laws for tokens in natural texts with a dataset from @xcite . it consists of six long literary and philosophical works in 5 languages each. we tokenize each text using naïve splitting into words by separators such as spaces and punctuation, and using mistral ai v3 (tekken) / ministral-8b tokenizer (mistralai, 2024) that uses byte-pair encoding @xcite with tiktoken (openai, 2024). to verify zipf's law for tokens, we calculate a cumulative histogram for each text and fit it with a power law using least squares in log-log coordinates (we agree with @xcite similarly, we build graphs of the number of unique words as a function of position in the text in log-log coordinates to verify heaps' law. a typical example can be seen in figure 2 . although visually we can tell that the lines are not as straight as for zipf's law, they (especially for words) can be well approximated by straight lines, implying decent adherence to the power law. saturation comes earlier for tokens and can be explained by both fundamentally limited number of tokenizer tokens for any language and tokens being shorter than words, so that new lexical items come later in the text if the text is measured in tokens. still, we can say that up to the length of 10 4 the power law behavior of tokens and words is similar. we can conjecture that the relationship between the number of unique words and the text length for large n is best described by w(n) ∝ n/log n rather than w(n) ∝ n β , but this is a topic of future research. in any case, the heaps' law exponent for words averaged over texts and languages is 0.855 with a standard deviation of 0.047, and the similar exponent for tokens is 0.801 with a standard deviation of 0.02. we can infer that the statistical behavior of tokens in human-written texts with respect to heaps' law is also similar to the statistical behavior of words. similarly to zipf's law, from ## conclusion and discussion we have shown empirically that zipf's and heaps' laws hold for tokens in natural (i.e., human-written) texts, but this is rarely the case for tokens of llmgenerated texts. more specifically, for each model, there is the best temperature at which the texts generated have statistical resemblance to humanwritten texts. we tie these extrema of power-law fitness with the phase transition of llm-generated texts from ordered to disordered phase through a critical state that adheres to power laws. thus, we believe to have been the first to discover phase transitions in zipf's and heaps' laws for llmgenerated texts. statistics for different model families behave differently with respect to the instruction-finetuning. for large llama models these extrema come at different temperatures for heaps' and zipf's laws. instruction-finetuned models typically generate human-like texts at higher temperatures than the base models. this shift is more pronounced for the qwen model family. we can conjecture that the finetuning approach followed by the qwen team honors narrower distributions after finetuning. this may also explain a long-standing puzzle in the community: high temperatures enhance the writing quality of llama and mistral, they do not have the same positive effect on qwen. the wide selection of llms used had an explicit goal of showing that pretraining corpus, training algorithm, and model architecture (within transformer architectural paradigm) influence zipfian properties of the llm-generated texts much less than the temperature. we also did not find a significant influence of prompting on the statistical properties studied.