Estimating the Entropy of Linguistic Distributions (2022)

Abstract: Shannon entropy is often a quantity of interest to linguists studying the communicative capacity of human language. However, entropymust typically be estimated from observed data because researchers do not have access to the underlying probability distribution. While entropy estimation is a well-studied problem in other fields, there is not yet a comprehensive exploration of the efficacy of entropy estimators for use with linguistic data. In this work, we fill this void, studying the empirical effectiveness of different entropy estimators for linguistic distributions. In a replication of two recent information-theoretic linguistic studies, we find evidence that the reported effect size is over-estimated due to over-reliance on poor entropy estimators. We end this paper with a concrete recommendation for the entropy estimators that should be used in future linguistic studies.

## introduction there is a natural connection between information theory, the mathematical study of communication systems, and linguistics, the study of human language-the primary vehicle that humans employ to communicate. researchers have exploited this connection since information theory's inception @xcite @xcite . with the advent of modern computing, the number of information-theoretic linguistic studies has risen, exploring claims about language such as the optimality of the lexicon @xcite , the complexity of morphological systems @xcite @xcite , and the correlation between surprisal and language processing time @xcite @xcite @xcite . in information-theoretic linguistics, a fundamental quantity of research interest is entropy. entropy is both useful to linguists in its own right, and is necessary for estimating other useful quantities, e.g., mutual information. however, the estimation of entropy from raw data can be quite challenging @xcite , e.g., in expectation, the plug-in estimator underestimates entropy @xcite . linguistic distributions often present additional challenges. for instance, many linguistic distributions, such as the unigram distribution, follow a power law @xcite ). 1 linguistics is not the only field with such nuances, and so a large number of entropy estimators have been proposed in other fields @xcite . however, no work to date has attempted a practical comparison of these estimators on natural language data. this work fills this empirical void. our paper offers a large empirical comparison of the performance of 6 different entropy estimators on both synthetic and natural language data, an example of which is shown in figure 1 . we find that @xcite is the best estimator when very few data are available, but @xcite is superior as more data become available. both are significantly better (in terms of meansquared error) than the naïve plug-in estimator. importantly, we also show that two recent studies @xcite show smaller effect sizes when a better estimator is employed; however, we are able to reproduce a significant effect in both replications. we recommend that future studies carefully consider their choice of entropy estimators, taking into account data availability and the nature of the underlying distribution. 2 ## entropy and language shannon entropy is a quantification of the uncertainty in a random variable. given a (discrete) random variable x with probability distribution p over k possible outcomes @xmath0, the shannon entropy of x is defined as entropy has many uses throughout science and engineering; for instance, @xcite originally proposed entropy as a lower bound on the compressibility of a stochastic source. yet the application of information-theoretic techniques to linguistics is not so straightforward: information-theoretic measures are defined over probability distributions and, in the study of natural language, we typically only have access to samples from the distribution of interest, e.g., the phonotactic distribution in english, which permits word we cannot find in a corpus, like blick, rather than the true probabilities required in the computation of eq. ( 1 ). indeed, it is often the case that not all elements of x are even observed in available data-such as words that were coined after the a corpus was collected. rather, p must be approximated in order to estimate h(p). one solution is plug-in estimation: given samples from p, the maximum-likelihood estimate for p is "plugged" into eq. ( 1 ). however, as originally noted by @xcite , this strategy generally yields poor estimates. 3 it is thus necessary 2 our code is available at https://github.com/ aryamanarora/entropy-estimation . ## statistical estimation theory statistical estimation theory provides us with the tools for estimating various quantities of interest based on samples from a distribution. central to this theory is the estimator: a statistic that approximates a property of the distribution our data is drawn from. more formally, let two properties of an estimator are often of interest: bias-the difference between the true value of θ and the expected value of our estimator θ(d) under p-and variance-how much θ(d) fluctuates from sample set to sample set: it is desirable to construct an estimator that has both low bias and low variance. however, the bias-variance trade-off tells us that we often have to pick one, and we should focus on a balance between the two. this trade-off is evinced through mean-squared error (mse), a metric oft-employed for assessing estimator quality: to recognize the trade-oft note that, for any fixed mse, a decrease in bias must be compensated with an increase in variance and vice versa. indeed, it is important to recognize that there is typically no single estimator that is seen as "best." different estimators balance the bias-variance trade-off differently, making their perceived quality specific to one's use-case. importantly, the effectiveness of an estimator also depends on the domain of interest. consequently, an empirical study of various entropy estimators, which this paper provides, is necessary in order to determine which entropy estimators are best suited for linguistic distributions. ## experiments here we provide an evaluation of the entropy estimators presented in §3.2 on linguistic data. ## conclusion this work presents the first empirical study comparing the performance of various entropy estimators for use with natural language distributions. from experiments on synthetic data (appendix) and natural data (celex), and two replication studies of recent papers in information-theoretic linguistics, we find that the oft-employed plug-in estimator of entropy can cause misleading results, e.g., the overestimates of effect sizes seen in both replication studies. the recommendation of our paper is that researchers should carefully consider their choice of entropy estimator based on data availability and the nature of the underlying distribution.