We consider words as a network of interacting letters, and approximate the probability distribution of states taken on by this network. Despite the intuition that the rules of English spelling are highly combinatorial (and arbitrary), we find that maximum entropy models consistent with pairwise correlations among letters provide a surprisingly good approximation to the full statistics of four letter words, capturing ∼ 92% of the multi-information among letters and even 'discovering' real words that were not represented in the data from which the pairwise correlations were estimated. The maximum entropy model defines an energy landscape on the space of possible words, and local minima in this landscape account for nearly two-thirds of words used in written English. Many complex systems convey an impression of order that is not so easily captured by the traditional tools of theoretical physics. Thus, it is not clear what sort of order parameter or correlation function we should compute to detect that natural images are composed of solid objects [1], nor is it obvious what features of the amino acid sequence distinguish foldable proteins from random polymers. Recently, several groups have tried to simplify the problem of characterizing order in biological systems using the classical idea of maximum entropy [2] . Maximum entropy models consistent with pairwise correlations among neurons have proven surprisingly effective in describing the patterns of activity in real networks ranging from the retina [3, 4] to the cortex [5]; these models are identical to the Ising models of statistical mechanics, which have long been explored as abstract models for neural networks [6] . Similar methods have been used to analyze biochemical [7] and genetic [8] networks, and these approaches are connected to an independent stream of work arguing that pairwise correlations among amino acids may be sufficient to define functional proteins [9] . Because of the immediate connection to statistical mechanics, this work also provides a natural path for extrapolating to the collective behavior of large networks, starting with real data [10] . Here we test the limits of these ideas constructing maximum entropy models for the sequence of letters in words. As non-native speakers know well, the rules of English spelling seem arbitrary and almost paradigmatically combinatorial (i before e except after c). In contrast, the whole point of maximum entropy constructions based on pairwise correlations is to ignore such higher order, combinatorial effects [11] . We thus suspect that the statistics of letters in words will provide an interesting test case. There is a long history of statistical approaches in the analysis of language, including applications of maximum entropy ideas [12] , while opposition to such statistical approaches was at the foundation of forty years of linguistic theory [13, 14] ; for a recent view of these debates see Ref [15] . Our goal here is not to enter into these controversies about language in the broad sense, but rather to test the power of pairwise interactions to capture seemingly complex structure. Even with only four letters, there are N = (26) 4 = 456, 976 possible words, but only a tiny fraction of these are real (or even 'legal') words in English. Our problem thus is easy to state: are maximum entropy models powerful enough to capture this restriction of vocabulary? To analyze the interactions among letters, we use two large corpora: a collection of novels from Jane Austen [16] , and a large sampling of American English contained in the American National Corpus (ANC) [17]. To control for potential typographic errors, words were also checked against a large dictionary database [18]. Out of 676,302 total words in our Austen corpus there were 7114 unique words, 763 of which were four letter words; the four letter words occurred in the corpus a total of 135,441 times.
A work of different nature, but also in the statistics field, has recently shown that, by considering words as a network of interacting letters, the maximum entropy models, which are consistent with pairwise correlations among letters, provide a very good approximation to the full statistics of four letter words REF .
Toward a statistical mechanics of four letter words
Biology Computer Science Physics