Phonotactic Complexity and Its Trade-offs (2020)

Abstract: We present methods for calculating a measure of phonotactic complexity—bits per phoneme— that permits a straightforward cross-linguistic comparison. When given a word, represented as a sequence of phonemic segments such as symbols in the international phonetic alphabet, and a statistical model trained on a sample of word types from the language, we can approximately measure bits per phoneme using the negative log-probability of that word under the model. This simple measure allows us to compare the entropy across languages, giving insight into how complex a language’s phonotactics is. Using a collection of 1016 basic concept words across 106 languages, we demonstrate a very strong negative correlation of − 0.74 between bits per phoneme and the average length of words.

## introduction one prevailing view on system wide phonological complexity is that as one aspect increases in complexity (e.g., size of phonemic inventory), another reduces in complexity (e.g., degree of phonotactic interactions). underlying this claimthe so-called compensation hypothesis @xcite )-is the conjecture that languages are, generally speaking, of roughly equivalent complexity, that is, no language is overall inherently more complex than another. this conjecture is widely accepted in the literature and dates back at least to the work of @xcite . because along any one axis, a language may be more complex than another, this conjecture has a corollary that compensatory relationships between different types of complexity must exist. such compensation has been hypothesized to be the result of natural processes of historical change, and is sometimes attributed to a potential linguistic universal of equal communicative capacity @xcite . methods for making hypotheses about linguistic complexity objectively measurable and testable have long been of interest, though existing measures are typically relatively coarse-see, for example, @xcite and §2 below for reviews. briefly, counting-based measures such as inventory sizes (e.g., numbers of vowels, consonants, syllables) typically play a key role in assessing phonological complexity. yet, in addition to their categorical nature, such measures generally do not capture longer-distance (e.g., crosssyllabic) phonological dependencies such as vowel harmony. in this paper, we take an informationtheoretic view of phonotactic complexity, and advocate for a measure that permits straightforward cross-linguistic comparison: bits per phoneme. for each language, a statistical language model over words (represented as phonemic sequences) is trained on a sample of types from the language, and then used to calculate the bits per phoneme for new samples, thus providing an upper bound of the actual entropy @xcite . characterizing phonemes via information theoretic measures goes back to @xcite , who discussed the information content of phonemes in isolation, based on the presence or absence of distinctive features, as well as in groups, (e.g., trigrams or possibly syllables). here we leverage modern recurrent neural language modeling methods to build models over full word forms represented as phoneme strings, thus capturing any dependencies over longer distances (e.g., harmony) in assigning probabilities to phonemes in sequence. by training and evaluating on comparable corpora in each language, consisting of concept-aligned words, we can characterize and compare their phonotactics. probabilistic characterizations of phonotactics have been used extensively in psycholinguistics (see §2.4), but such methods have generally been used to assess single words within a lexicon (e.g., classifying high versus low probability words during stimulus construction), rather than information-theoretic properties of the lexicon as a whole, which our work explores. the empirical portion of our paper exploits this information-theoretic take on complexity to examine multiple aspects of phonotactic complexity: ## the probabilistic lexicon in this work, we are interested in a hypothetical phonotactic distribution p lex : σ * → r + over the lexicon. in the context of phonology, we interpret σ * as all ''universally possible phonological surface forms,'' following @xcite . 7 the distribution p lex , then, assigns a probability to every possible surface form @xmath0. in the special case that p lex is a log-linear model, then we arrive at what is known as a maximum entropy grammar @xcite . a good distribution p lex should assign high probability to phonotactically valid words, including non-existent ones, but little probability to phonotactic impossibilities. for instance, the possible english word blick should receive much higher probability than * bnick, which is not a possible english word. the lexicon of a language, then, is considered to be generated as samples without replacement from p lex . if we accept the existence of the distribution p lex , then a natural manner by which we should measure the phonological complexity of language is through shannon's entropy (cover and thomas, 2012) the units of h(p lex ) are bits as we take log to be base 2. specifically, we will be interested in bits per phoneme, that is, how much information each phoneme in a word conveys, on average. ## conclusion we have presented methods for calculating a wellmotivated measure of phonotactic complexity: bits per phoneme. this measure is derived from information theory and its value is calculated using the probability distribution of a language model. we demonstrate that cross-linguistic comparison is straightforward using such a measure, and find a strong negative correlation with average word length. this trade-off with word length can be seen as an example of complexity compensation or perhaps related to communicative capacity.