On Homophony and Rényi Entropy (2021)

Abstract: Homophony’s widespread presence in natural languages is a controversial topic. Recent theories of language optimality have tried to justify its prevalence, despite its negative effects on cognitive processing time, e.g., Piantadosi et al. (2012) argued homophony enables the reuse of efficient wordforms and is thus beneficial for languages. This hypothesis has recently been challenged by Trott and Bergen (2020), who posit that good wordforms are more often homophonous simply because they are more phonotactically probable. In this paper, we join in on the debate. We first propose a new information-theoretic quantification of a language’s homophony: the sample Rényi entropy. Then, we use this quantification to revisit Trott and Bergen’s claims. While their point is theoretically sound, a specific methodological issue in their experiments raises doubts about their results. After addressing this issue, we find no clear pressure either towards or against homophony—a much more nuanced result than either Piantadosi et al.’s or Trott and Bergen’s findings.

## introduction ambiguity is a hallmark of human language, and is present at all levels of linguistic structure. both the causes and resulting effects of ambiguity are topics which have sparked much debate. for example, while some claim it has a beneficial impact on communication, others have taken it as a sign of inefficiency. in this work, we contribute to the debate surrounding a specific form of lexical ambiguity in which a wordform shares multiple unrelated meanings: homophony. 1 while the quantitative study of homophony dates back to (at least) @xcite , recently, @xcite proposed a new explanation for the higher rate of homophony amongst good, i.e. short or phonotactically well-formed, wordforms: if words were sampled i.i.d. from a phonotactic distribution, then good wordforms would simply be sampled more often. this implies there is no pressure favouring homophony in these words for the sake of efficiency, which directly opposes @xcite hypothesis. in fact, trott and bergen go further, relying on their experimental results to argue that "homophony may even be selected against in real languages." 2 in this work, we join this debate by proposing a novel quantification of a language's homophony: the sample rényi entropy foot_2 -defined as the negative log likelihood (or surprisal) that two instances in an m -sized sample take on the same value. when measured on an observed lexicon, this measure holistically captures the chance that two wordforms coincide, i.e., that they are homophones, providing a new means to test whether lexicons have a pressure in favour or against homophony. further, we revisit trott and bergen's claims. whilst their theoretical arguments are sound, we believe their experimental design could not have provided concrete evidence for or against their hypothesis. specifically, their inadequate modelling of the phonotactic distribution, through the use of weakly regularised n-grams, cause us to question conclusions drawn from their experiments. we take measures to address this issue-relying on more expressive lstm language models-and provide our own analysis of homophony in natural language. experimentally, we arrive at more nuanced results than prior work, finding no pressure either towards or against homophony. we conclude with the warning that biases in our models-and those of other works-need to be deeply considered when relying on them to answer linguistics questions. ## homophony homophony is a widespread phenomenon which has long puzzled linguists. on average, roughly 4% of the words in a language are estimated to be homophones @xcite . this rate, however, has a large variation across languages-in english, for instance, @xcite estimates it to be 7.4%. a number of works hint at the inefficiencies homophony leads to @xcite find that homophonous words are recognised more slowly; @xcite shows homophones are harder for children to learn. yet a large body of work has argued for the efficiency of homophony. @xcite suggest ambiguity is a desirable property in that it increases a language's communicative efficiency. ambiguity would allow a language to reuse good wordforms, which falls in line with zipf's principle of least effort. with this in mind, piantadosi et al. showed that short, frequent and phonotactically probable wordforms have more homophones than their counterparts. in further support of this hypothesis, @xcite found children easily learn to differentiate homophones when pairs have distinct syntactic categories (and that homophony is more likely in these cases); @xcite showed speakers make contexts more informative in the presence of lexical ambiguity. these results suggest people naturally navigate ambiguity. however, trott and bergen recently proposed a new explanation of piantadosi et al.'s findings: they attribute homophony to chance. specifically, they claim that if we model the phonotactic distribution probabilistically, we can see that more homophones amongst good wordforms should be expected-they are simply more probable. yet their methodology to support this claimcomparing natural lexicons with artificial ones sampled from n-gram language models-has an important setback: their use of 5-gram models with only weak laplace smoothing. these models are prone to overfitting. as such, it is not surprising that an artificially generated lexicon would contain more homophony than natural ones; as we will show, overfit distributions will likely produce more collisions. thus, it is not entirely clear what we can conclude from their experiments. ## quantifying homophony as per its definition, homophony should be tightly related to a language's phonotactics-its distribu-tion over wordforms. in this section, we first provide a definition of a language's phonotactic distribution. we then present both the rényi collision entropy and the sample rényi entropy as new measures of homophony. ## experimental methodology the sample rényi entropy, presented in eq. ( 5 ), can be directly computed on an observed lexicon. on the other hand, both the rényi entropy (as depicted in §3.2) and our null hypothesis test are computed over a phonotactic distribution, to which we do not have direct access. an important consideration, thus, is how exactly this distribution can be approximated. recently, trott and bergen (2020) relied on weakly regularised n-gram models for their analysis. as can be seen in our earlier results @xcite , neural language models can capture this phonotactic distribution much more faithfully. in this work, we will compare trott and bergen's n-grams with pimentel et al.'s lstm models, and show how n-grams may give misleading results. n-gram. perhaps the simplest method for estimating distributions of phones in a language is through n-gram, or in this case n-phone, modelling. specifically, we can estimate the probability of observing some phone w t given the previous n -1 phones by computing the proportion of times this phone follows those previous n -1 phones in a corpus. by this definition, sequences not present in the corpus will be assigned 0 probability under the model. this, among other factors, contributes to the often poor generalisation abilities of basic n-gram models. indeed, there exists an entire literature on smoothing and regularisation techniques for n-gram modelling @xcite @xcite . laplace smoothing is a popular choice, being used in a number of recent works in computational linguistics (e.g. @xcite . however, it is perhaps the simplest of such regularisation techniques, and usually leads to much weaker empirical performances than, e.g., kneyser-ney @xcite . it is therefore natural to question whether an n-gram model with simple laplace smoothing can provide a good representation of the true phonotactic distribution of a language. in our experiments, we follow trott and bergen in using a 5-gram model with laplace smoothing with strength 0.01 as p(w t | @xmath0. ## results, discussion and conclusion the rényi collision entropy results must be more carefully analysed. at first glance, we see that the n-gram model has the smallest rényi entropy across all languages, having more than 1 bit difference to the lexicon's sample rényi entropy in both english and german. this may lead one to conclude that homophony is strongly disfavoured in 6 we exclude words with spaces, hyphens, or apostrophes. 7 our code is available at https://github.com/ rycolab/homophony-as-renyi-entropy . when we compare the rényi entropy of the lstm to the lexicon's, we get much more nuanced results. while the english lexicon seems to hinder homophony-homophony is more surprising in real lexicons than expected from their phonotactics-the opposite is true for dutch. meanwhile, german presents no clear trends. we should, however, refrain from making strong claims about these results. while the difference between the lstm's train and test cross-entropy is small, implying that it overfits only to a small degree, its precise quantitative impact on the rényi entropy is hard to quantify. furthermore, expanding our analysis to celex's multi-morphemic words leads to somewhat different results (see app. b). hence, we see no clear pattern across these languages, and find, thus, no pressure either in favour or against homophony. we conclude this section with a warning. when exploring linguistics using language models, one should carefully consider these model's inherent inductive biases and their potential effects on results. while overfit n-grams provide strong evidence towards homophony avoidance in natural lexicons, we arrive at different results using better models.