Emergent Word Order Universals from Cognitively-Motivated Language Models (2024)

Abstract: The world’s languages exhibit certain so-called typological or implicational universals; for example, Subject-Object-Verb (SOV) languages typically use postpositions. Explaining the source of such biases is a key goal of linguistics.We study word-order universals through a computational simulation with language models (LMs).Our experiments show that typologically-typical word orders tend to have lower perplexity estimated by LMs with cognitively plausible biases: syntactic biases, specific parsing strategies, and memory limitations. This suggests that the interplay of cognitive biases and predictability (perplexity) can explain many aspects of word-order universals.It also showcases the advantage of cognitively-motivated LMs, typically employed in cognitive modeling, in the simulation of language universals.

## introduction there are thousands of attested languages, but they exhibit certain universal tendencies in their design. for example, subject-object-verb (sov) word order often combines with postpositions, while svo order typically employs prepositions @xcite . researchers have argued that such implicational universals are not arbitrary but shaped by their advantage for humans @xcite @xcite . such language universals have been recently studied through neural-based computational simulation to elucidate the mechanisms behind the universals @xcite . the languages which emerge, however, have typically not been humanlike @xcite @xcite . such mismatch arguably stems from the lack of human-like cognitive biases in neural agents @xcite , but injecting cognitive biases into systems and showing their benefits has proved challenging @xcite . in this study, expanding on a study of wordorder biases in language models (lms: white and cotterell (2021)), we demonstrate the advantage of cognitively-motivated lms, which can simulate human cognitive load during sentence processing well @xcite @xcite , and thus predict many implicational word-order universals in terms of their inductive biases (figure 1 ). specifically, we train various types of lms in artificial languages with different word-order configurations ( §3). our experiments show that perplexities estimated by cognitively-motivated lms ( §5) correlate better with frequent word-order configurations in attested languages than standard lms ( §6). this confirms that such biases are a potential source of the word-order universals as well as demonstrate the plausibility of cognitively-motivated lms as models of human language processing. ## preliminary we explain the assumptions behind this research. ## models we examine 23 types of uni-directional lms, which adopt subwords split by byte-pairencoding @xcite and are trained for 10 epochs. for each type of lm, we train/evaluate five models with different random seeds. in each run, 20k sentences (train:dev:test is 8:1:1) are generated from pcfg to train/evaluate the model. see app. c for model details. ## experiments we compare the lm's word-order preferences with attested word-order distributions ( §6). then, we further analyze what kind of word-order combinations lms prefer ( §7). ## conclusions we have shown that cognitively-motivated lms better led to the emergence of word-order universals than standard lms. from the linguistic typology perspective, we provide computational evidence that the universals emerge from cognitive biases, which has been challenging to demonstrate in previous work @xcite . from the cognitive modeling perspective, our results demonstrate that cognitively-motivated lms have human-like biases that are sufficient to replicate some human word-order universals. our results also clarified what typological patterns can still not be modeled by cognitively-motivated lms (agent-first bias), as well as opened the direction to relate lms in the traditional linguistic theory of delineating possible language from the impossible.