Revisiting the Optimality of Word Lengths (2023)

Abstract: Zipf (1935) posited that wordforms are optimized to minimize utterances’ communicative costs. Under the assumption that cost is given by an utterance’s length, he supported this claim by showing that words’ lengths are inversely correlated with their frequencies. Communicative cost, however, can be operationalized in different ways. Piantadosi et al. (2011) claim that cost should be measured as the distance between an utterance’s information rate and channel capacity, which we dub the channel capacity hypothesis (CCH) here. Following this logic, they then proposed that a word’s length should be proportional to the expected value of its surprisal (negative log-probability in context). In this work, we show that Piantadosi et al.’s derivation does not minimize CCH’s cost, but rather a lower bound, which we term CCH-lower. We propose a novel derivation, suggesting an improved way to minimize CCH’s cost. Under this method, we find that a language’s word lengths should instead be proportional to the surprisal’s expectation plus its variance-to-mean ratio. Experimentally, we compare these three communicative cost functions: Zipf’s, CCH-lower , and CCH. Across 13 languages and several experimental settings, we find that length is better predicted by frequency than either of the other hypotheses. In fact, when surprisal’s expectation, or expectation plus variance-to-mean ratio, is estimated using better language models, it leads to worse word length predictions. We take these results as evidence that Zipf’s longstanding hypothesis holds.

## introduction zipf proposed the idea that languages are optimized to minimize their expected utterance length (zipf, 1935). @xcite under this hypothesis, a word's length should be inversely proportional to its frequency. indeed, this relationship has been attested across a wide variety of the world's languages (grzybek, 2015; bentz and ferrer-i-cancho, 2016, inter alia). @xcite we will refer to this hypothesis as zipf. in subsequent work, piantadosi et al. (2011) offered a complementary account of communicative cost. starting from the hypothesis that information rate should be roughly constant during communication (uid; fenk and fenk, 1980; levy and jaeger, 2007), they argue that word lengths should make information rates as close as possible to a hypothetical channel capacity, where the channel refers to the means by which information is transferred from one person to another. we term this the channel capacity hypothesis (cch). @xcite they conclude that lengths should be proportional to a word's expected surprisal instead. @xcite as the communicative efficiency of language provides important insights into human cognition (gibson et al., 2019), piantadosi et al.'s finding that word lengths are better explained by average surprisal than frequency has been influential. however, there are shortcomings: first, the manner in which piantadosi et al. finds a solution which minimizes the cost associated with cch is not formally specified. and second, piantadosi et al.'s empirical results have been shown to be sensitive to a number of methodological decisions, such as the choice of text-encoding (e.g., ascii vs. unicode), the inclusion of non-conventional wordforms and other orthographic conventions of a language (meylan and griffiths, 2021; levshina, 2022). thus, there remain fundamental open questions about the relationship between communicative efficiency and word length. here, we aim to clarify both theoretical and empirical aspects of this relationship. theoretically, we offer a novel, formal derivation of piantadosi et al.'s claim. we find that piantadosi et al. (2011) optimize not for the objective under the cch, but for a lower bound on it instead; we call this the cch ↓ objective. we then provide a closed-form expression for the function that determines word lengths under cch: word lengths should be proportional to the expected surprisal plus its variance-to-mean ratio. importantly, we derive the solution above by framing the problem of assigning wordforms as the optimization of a cost function. @xcite by instantiating this optimization problem with the objectives posited by each hypothesis (zipf, cch, and cch ↓ ), we can compute their word length predictions within a single, unified framework. empirically, we offer a large-scale comparison of zipf's, cch ↓ 's, and cch's word length predictions across 13 typologically diverse languages. notably, we use neural language models to estimate words' surprisals, which provides more accurate estimates than the n-gram models relied on by prior work on this topic (piantadosi et al., 2011; meylan and griffiths, 2021; levshina, 2022). we find strong evidence (see fig. 1 ) that languages are optimized to minimize their utterance lengths: a word's frequency (zipf's prediction) offers stronger predictive power over word lengths than either the surprisal's expected value (cch ↓ 's prediction) or expected surprisal plus variance-tomean ratio (cch's prediction). we conclude that zipf's longstanding theory stands strong. definition 1. the lexicalization problem is the task of finding an optimal lexicon ϕ * , which minimizes cost [ϕ]. this lexicon can be described formally as the solution to there are many assumptions that one could make about φ ℓ 's characteristics. we make a few explicit in the following remark. remark 1. we take the set φ ℓ to include all lexicons which: 1 only produce phonotactically valid wordforms, 6 2 respect morphological composition, 7 and 3 are uniquely decodable. @xcite another implicit constraint 4 regarding valid ϕwhich comes from our specification of the output @xcite we define this distribution formally in app. a. @xcite phonotactics tells us how phones can be combined to create wordforms in a language. if we denote the set of all possible phonotactically valid wordforms in language ℓ as l ℓ ⊂ σ * , this means that the image of ϕ is contained in l ℓ . in the subsequent sections, we consider relaxations of eq. ( 1 ) to arrive at simple solutions regarding the lengths provided by optimal lexica. specifically, we partially relax constraint 1 and fully relax constraint 2 when deriving a lexicon with minimal utterance length. further, when deriving optimal results for both cch and cch ↓ , we also fully relax constraints 1 , 3 , and 4 . @xcite note that, as in all optimization problems, removing constraints always yields a lower bound on the expected cost we obtain under an optimal lexicon. @xcite 3 revisiting zipf's law of abbreviation zipf (1935, 1949) posited a specific form that the cost function in eq. ( 1 ) should take. concretely, he posited that lexica were optimized with the goal of minimizing speakers' utterance lengths, which can be written as cost[ϕ](w, c) = |ϕ(w)| in our notation. in an attempt to formalize his position, he proposed his eponymous law of abbreviation: over the years, zipf's law of abbreviation has been empirically investigated numerous times (wimmer et al., 1994; sigurd et al., 2004; kanwal et al., 2017; koplenig et al., 2022; levshina, 2022; petrini et al., 2022, 2023). we now present a formal derivation of zipf's law of abbreviation by viewing it as an instantiation of the lexicalization problem. hypothesis 1. zipf's hypothesis predicts that communication is made optimal by the mapping ϕ zipf that satisfies: if we relax constraints 1 and 2 in remark 1, then the optimal solution to eq. ( 3 ) can be achieved by huffman coding (huffman, 1952). @xcite we know that this optimal solution's word lengths respect: fortunately, we can derive a more interesting result where the proportionality in eq. ( 2 ) still holds by only partially relaxing 1 from remark 1. we first assume a very simplistic model of phonotactics. given an alphabet σ of phones, let l ℓ ⊂ σ * be the set of phonotactically valid wordforms in language ℓ. note that this assumes deterministic phonotactics (gorman, 2013; dai and futrell, 2021). @xcite further, define prefixes(l ℓ ) @xmath0, @xmath1. assumption 1. the constant phonotactic assumption assumes there exists a @xmath2, for every string α ∈ prefixes(l ℓ ), there exist exactly k symbols {σ k } k @xmath3. in words, assumption 1 says that there are exactly k valid symbols with which every phonotactically valid prefix can be extended. given this assumption, we can now find a solution to eq. ( 3 ), which only partially relaxes the phonotactic constraint in remark 1. theorem 1. the minimization problem given in hypothesis 1 with constraint 2 relaxed can be solved by huffman coding foot_4 with k symbols. the optimal solution is given by ## results our main results are presented in fig. 1 and 2 . in short, fig. 1 shows that words' frequencies offer stronger predictive power of word lengths (as evinced by smaller mse) than either of the surprisal-dependent metrics. this result provides evidence for zipf's hypothesis over either cch or cch ↓ . this result is particularly surprising since we improve on cch's optimal word length predictions, but zipf's hypothesis still provides the best predictions. @xcite a similar result can be seen in fig. 2 , where frequency offers the strongest correlation with lengths (in terms of both pearson and spearman), in all languages but english. notably, in our results, some languages even have a negative correlation between the two surprisal-based measures and actual word lengths. we now turn to analyzing different methodological choices that could impact our results. ## discussion the answer to what drives the distribution of word lengths in lexica has long been considered important for understanding the evolution and function of language (see gibson et al., 2019 for a review). across multiple languages and various methodological choices, our results support zipf's law of abbreviation over other potential explanations as a driving factor in the development of lexica. these findings deviate from piantadosi et al., who found average surprisal to be a stronger predictor of word lengths. we hypothesize that this is because of methodological choices. specifically, piantadosi et al. derive surprisal estimates from language models that are now outdated (in terms of their quality), and we found that, when cch's predictions were computed using worse surprisal estimates, they had stronger correlations with length than when using better estimates. like prior work on this topic (meylan and griffiths, 2021; levshina, 2022), our analyses suggest the sensitivity of piantadosi et al.'s results to methodological choices. what do these results tell us about the communicative optimization of natural language lexica? in short, our results suggest lexica are optimized to minimize expected utterance lengths. notably, other linguistic properties may be optimized towards other notions of communicative efficiency. while a word's duration is mainly determined by its wordform, speakers can still modulate this duration to a certain extent; such a modulation could target cch. in fact, prior work has shown a correlation between surprisal and duration (bell et al., 2003; aylett and turk, 2004; pimentel et al., 2021a). ## conclusion in this paper, we formalize the problem of assigning wordforms based on different notions of communicative efficiency, which we term the lexicalization problem. under this framework, we describe the optimization problem related to the channel capacity hypothesis, and, in doing so, we show that piantadosi et al.'s predictions optimized for only a lower bound on cch, rather than on the true objective. further, while considering relaxed versions of the lexicalization problem, we derive optimal word length values for zipf's hypothesis and cch. we then empirically evaluate cch's, cch ↓ 's and zipf's predictions in 13 languages. our results strongly support zipf's hypothesis: word lengths are optimized to minimize utterance lengths.