Language Models Largely Exhibit Human-like Constituent Ordering Preferences (2025)

Abstract: Though English sentences are typically inflexible vis-à-vis word order, constituents often show far more variability in ordering. One prominent theory presents the notion that constituent ordering is directly correlated with constituent weight: a measure of the constituent’s length or complexity. Such theories are interesting in the context of natural language processing (NLP), because while recent advances in NLP have led to significant gains in the performance of large language models (LLMs), much remains unclear about how these models process language, and how this compares to human language processing. In particular, the question remains whether LLMs display the same patterns with constituent movement, and may provide insights into existing theories on when and how the shift occurs in human language. We compare a variety of LLMs with diverse properties to evaluate broad LLM performance on four types of constituent movement: heavy NP shift, particle movement, dative alternation, and multiple PPs. Despite performing unexpectedly around particle movement, LLMs generally align with human preferences around constituent ordering.

## introduction despite the fact that word order in english is typically strict, constituents in post-verbal positions can be highly flexible in their ordering @xcite . a number of specific phenomena are prime examples of this movement; we show these in figure 1 . the cause of this movement has been attributed to a variety of factors, including lexical bias @xcite @xcite , semantic connectedness (kayne, 1983; 1 all code and data is available at https://github.com/mcgill-nlp/constituent-movement . @xcite , and information structure @xcite @xcite ). however, one of the most prominent factors, as prior research has suggested, is constituent weight and complexity @xcite @xcite . as the following examples show, we tend to prefer sentences where longer and more complex constituents are moved to the end of the sentence: the typical constituent order shown in (5-a), for example, is not readily perturbed to the constituent order shown in (5-b). in sentences (5-c) and (5-d), however, where the np is considerably longer and more complex than the pp, the reverse is true. 2 linguistic theory thus suggests that phrases and constituents are specifically ordered to be presented in increasing complexity, or weight; essentially, the larger the constituent, the further to the end of the sentence we expect it to appear @xcite @xcite . consider the example in figure 3 : other orderings of this sentence, if greatly violating this principle of weight, would likely be considered undesirable. this relationship between the complexity of post-verbal constituents and their ordering raises several questions: psycholinguistic research has provided some insight into these questions for human language processing, but the same cannot yet be said about increasingly powerful non-human language processors @xcite . prior research, however, supports the abilities of modern language models in assigning relative linguistic plausibility scores aligning similar to human preferences @xcite . additionally, we hypothesize that the human-feedback mechanism incorporated in instruction-tuned models will present even more similar judgements. in this work, we study the behavior of llms with constituent movement in english. we model heavy np shift (hnps), dative alternation (da), particle movement (pm), multiple pp shift (mpp)-see figure 1 -as a function of its weight. weight corresponds to a number of selected measures: word length, syllable weight, token length, and modifier weight. we analyze these measures to determine which best explains constituent ordering effects. figure 2 outlines our contributions, which are as follows: ## background at a high level, english follows a subject-verbobject (svo) ordering; beyond this basic structure, other objects, modifiers, constituents, and clauses can be added to form more complex sentences @xcite . the organization and format of how and when each constituent in a sentence is delivered, or its ordering, can be highly flexible @xcite @xcite . constituent shifting is the process of reordering the constituents of a sentence, such that the original meaning of the sentence is maintained, and all semantic truth conditions are unchanged. this work focuses on four specific types of shift, with a prime commonality: each shift involves the movement of constituents from a post-verbal position (i.e. appearing after the verb of a sentence) to another post-verbal position @xcite . 3 in our experiments, we consider the following measures of weight: word length, which corresponds to the number of words in a constituent; syllable weight, which corresponds to the number of syllables in a constituent; token length, which refers to the number of tokens in a constituent, determined by respective model tokenizers; modifier weight, which refers to the number of adjective phrase (adjp) and prepositional phrase (pp) modifying the constituent itself (plus 1 to include the weight of the base constituent itself as normalization). absolute weight itself, however, is not the most effective metric for observing motivations for con-stituent movement; instead, there is reason to believe that it is the relative weight of constituents that determines their ordering @xcite . for instance, for phrases "with her grandmother" and "around the garden", the ratio of word lengths would be 1 (3:3); the ratio of syllables would be 1 (5:5), etc. for phrases "with her grandmother" and "around the decorated entryway garden with the large fountain", the ratio of word length would be 1 3 (3:9); the ratio of syllables would be 5 17 (5:17), etc. as the ratio of any metric increases beyond 1, we know that the weight of the first constituent is larger than the second, and thus, we predict that the motivation to shift will be greater. ## motivating factors of model preference literature regarding human language finds varying levels of importance correlated with different measures of constituent weight, particularly in corpus studies and human judgement tasks (wasow, 1997a; @xcite . whether or not this same trend is seen with language models is unknown. we conduct a regression analysis on the data collected for the first experiment, using a generalized additive mixed model @xcite , with the goal of measuring how significantly each weight measure both impacts the models' judgements and serves to fit regression lines on the data. ## human-model preference correlation 7.1 approach to adequately compare the behaviors of llms with those of humans, a direct study of preferences on identical data points is necessary. we collect human judgements on a subset of data presented to models; though human judgements and model scores are not identical metrics, they can act as proxies when comparing relative trends. we conduct a crowdsourced study through prolific, collecting judgments from 126 native english speakers residing in anglophone countries, on 500 sentence pairs. each participant is presented 25 sentence pairs and asked to judge how natural they sound in relation to each other, assigning a score between 1 and 7; 1 corresponds to the first sentence presented appearing far more natural than the second, and 7 the reverse. we exclude datapoints with low inter-annotator agreement to minimize noise. foot_5 ## discussion in figure 5 and appendix a.2.1, we observe converging effects as weight increases, suggesting that prior theories defining weight as a prime factor in motivating the shift hold with computational models as well. further, we observe, specifically with pm, that weight, beyond a certain threshold, begins to detriment motivation for shifting. we observe similar effects on the scraped corpus data, in appendix a.2.2, though with more noise. given the specificity of the data itself, being rooted in financial reports and news articles, some noise and outliers were expected, and in some cases, observable trends remain. similar to the synthetic data plots, we see convergence on hnps, and some on mpp, as well as an initial drop followed by a rise in m pref erence scores for pm. the syllable weight was the best measure of weight for explaining motivations to shift in llms. this raises an obvious question-why is syllable weight a more effective predictor of model behavior than token weight, which would intuitively be most aligned with a model's processing of weight and complexity? this finding acts as initial evidence that models may induce linguistic information not just at the token level, but also implicitly at the level of syllables. foot_6 broadly speaking, a clear trend is maintained across humans and models, following what was presented by @xcite in their analysis. where human language sees motivation for movement with increasing weight, model behavior follows closely. our experiment, which includes graded data beyond binary weight categorizations, and a wider range of models, yields relatively high correlation effects between preferences of models and humans, as presented in ## conclusions in this work, we present a thorough analysis of the behaviors of llms in response to constituent movement, using both a novel set of nearly 400k minimal pairs of variably ordered sentences, as well as naturally occurring data. we collect human judgements and model preference scores and observe comparable trends between the behaviors of humans and models. such comparisons indicate that humans and llms largely hold similar linguistic preferences around constituent ordering, with the exception of particle movement. our findings-and in particular, the surprising gap we find between instruction-tuned models and their vanilla counterparts-invite further research into when and how linguistic preferences of models and humans align.