A Cross-Linguistic Pressure for U niform I nformation D ensity in Word Order (2023)

Abstract: While natural languages differ widely in both canonical word order and word order flexibility, their word orders still follow shared cross-linguistic statistical patterns, often attributed to functional pressures. In the effort to identify these pressures, prior work has compared real and counterfactual word orders. Yet one functional pressure has been overlooked in such investigations: The uniform information density (UID) hypothesis, which holds that information should be spread evenly throughout an utterance. Here, we ask whether a pressure for UID may have influenced word order patterns cross-linguistically. To this end, we use computational models to test whether real orders lead to greater information uniformity than counterfactual orders. In our empirical study of 10 typologically diverse languages, we find that: (i) among SVO languages, real word orders consistently have greater uniformity than reverse word orders, and (ii) only linguistically implausible counterfactual orders consistently exceed the uniformity of real orders. These findings are compatible with a pressure for information uniformity in the development and usage of natural languages.1

## introduction human languages differ widely in many respects, yet there are patterns that appear to hold consistently across languages. identifying explanations for these patterns is a fundamental goal of linguistic typology. furthermore, such explanations may shed light on the cognitive pressures underlying and shaping human communication. this work studies the uniform information density (uid) hypothesis as an explanatory principle for word order patterns @xcite @xcite @xcite . the uid hypothesis posits a communicative pressure to avoid spikes in information within an utterance, thereby keeping the information profile of an utterance relatively close to uniform over time. while the uid hypothesis has been proposed as an explanatory principle for a range of linguistic phenomena, e.g., speakers' choices when faced with lexical and syntactic alternations @xcite , its relationship to word order patterns has received limited attention, with the notable exception of @xcite . our work investigates the relationship between uid and word order patterns, differing from prior work in several ways. we (i) use transformer language models (lms) @xcite to estimate information-theoretic operationalizations of information uniformity; (ii) analyze large-scale naturalistic datasets of 10 typologically diverse languages; and (iii) compare a range of theoretically motivated counterfactual grammar variants. experimentally, we find that among svo languages, the real word order has a more uniform information density than nearly all counterfactual word orders; the only orders that consistently exceed real orders in uniformity are generated using an implausibly strong bias for uniformity, at the cost of expressivity. further, we find that counterfactual word orders that place verbs before objects are more uniform than ones that place objects before verbs in nearly every language. our findings suggest that a tendency for uniform information density may exist in human language, with two potential sources: (i) word order rules, with svo order generally being more uniform than sov; and (ii) choices made by speakers, who use the flexibility present in real languages to structure information more uniformly at a global level (and not only in a small number of isolated constructions). ## counterfactual language paradigm following prior work that has used counterfactual languages to study the functional pressures at play in word order patterns, we investigate to what degree a language's word order shows signs of optimization for uid. in this approach, a corpus of natural language is compared against a counterfactual corpus containing minimally changed versions of the same sentences, where the changes target an attribute of interest, e.g., the language's word order. for example, several studies of dlm have compared syntactic dependency lengths in real and counterfactual corpora, generated by permuting the sentences' word order either randomly @xcite or deterministically by applying a counterfactual grammar @xcite @xcite . similarly, we will compare measures of uid in real and counterfactual corpora to investigate whether real languages' word orders exhibit more uniform information density than alternative realizations. ## results estimates of mean per-word surprisal on the test set are in figure 4a . consistent with the results of @xcite , our trained models for nearly all counterfactual variants assign higher per-word surprisal to their respective test sets than the real models assign to theirs. across all 10 languages, reverse has mean surprisal close to, but consistently slightly higher than, that of the real ordering. sort-freq and sort-freq-rev have mean surprisals close to or below those of real. estimates of mean surprisal variance (uid v ) over sentences are shown in figure 4b . notably, there is a dissociation between the rank order of variants according to mean surprisal and according to uid v : variants with similar mean surprisals did not necessarily have similar uid v scores, and vice versa, suggesting that information uniformity and mean surprisal can vary independently of each other. our main observations are as follows: (i) in all languages except turkish and hindi, our estimates of uid v for real are lower than those for reverse, despite the variants' similarities in mean surprisal. (ii) as predicted, the sort-freq baseline has uid v equal to or lower than that of real. (iii) the other counterfactual variants typically exhibit higher uid v than real, with the exception of mixed results for sort-freq-rev. (iv) the efficient-vo variants typically have lower uid v than efficient-ov (with hungarian being a noteworthy exception), which supports findings based on toy grammars showing that svo orders are more uniform than sov orders @xcite . crucially, these results are qualitatively similar using the uid lv metric (figure 6b ). to fairly compare variants using the uid p metric, we first need to account for the fact that, unlike surprisal variance, the metric is sensitive to shifts in mean surprisal. to control for this, we fit a regression model predicting the uid p score based on three variables: the mean surprisal, the grammar variant, and the dataset size (20m, 6.6m, and 2.2m words). we train multiple language models for each language-variant combination (3 dataset sizes and 2 random seeds), resulting in 84 data points per language. we apply treatment coding to the variants, with real as the reference level. figure 5 shows the resulting estimates of the coefficients for each variant, where a coefficient should be positive if that variant is less uniform than real. qualitatively, the regression results match the results given by uid v and uid lv : real is more uniform than reverse in sov languages, sort-freq is the only counterfactual variant that is consistently more uniform than real, and efficient-vo is more uniform than efficient-ov in most languages; the opposite is true in hungarian and the difference is negligible in russian. ## discussion we offer a discussion of the results observed in §6, including their implications for the role of functional pressures in language. ## conclusion in conclusion, we have empirically demonstrated that in many languages, real word orders distribute information more uniformly than a range of counterfactual orders. the fact that this pattern holds in every svo languages but is mixed among sov languages lends support to the view that svo basic word order is preferable to sov order from the perspective of maximizing uid. we posit that there are two potential sources of optimization within a language for greater uid: language evolution favoring word orders that produce less variance in information content, and speaker choices in favor of constructions that smooth the information profile of utterances. our results are consistent with the uid hypothesis, and support the idea that communicative pressures (operationalized in terms of information theory) influence the structure of human language.