Evaluating a Century of Progress on the Cognitive Science of Adjective Ordering (2023)

Abstract: The literature on adjective ordering abounds with proposals meant to account for why certain adjectives appear before others in multi-adjective strings (e.g., the small brown box). However, these proposals have been developed and tested primarily in isolation and based on English; few researchers have looked at the combined performance of multiple factors in the determination of adjective order, and few have evaluated predictors across multiple languages. The current work approaches both of these objectives by using technologies and datasets from natural language processing to look at the combined performance of existing proposals across 32 languages. Comparing this performance with both random and idealized baselines, we show that the literature on adjective ordering has made significant meaningful progress across its many decades, but there remains quite a gap yet to be explained.

## introduction adjective ordering preferences regularly appear across the world's languages: in nominal constructions with multiple adjective modifiers (e.g., the small brown box), speakers often (strongly) prefer one ordering. furthermore, these preferences are often the same across languages for translation-equivalent adjectives. this striking regularity raises the question of what aspects of language or its use in communication yield the observed preferences. after more than a century of research, linguists and cognitive scientists have proposed an array of hypotheses for predicting adjective ordering in terms of cognitive factors affecting language production and linguistic representations. to date, most investigations of these cognitive hypotheses about adjective order have considered single predictors in isolation, or have compared their performance on a single language (i.e., english; for discussion, see @xcite . this situation is not ideal, especially considering the cognitive theories we survey below were developed in the context of predicting adjective order in english only, often leaving their cross-linguistic generality unclear. however, the recent availability of massively cross-linguistic parsed datasets and the development of nlp technologies such as word embeddings have opened up the possibility for large-scale evaluations of a wide variety of cognitive hypotheses against a wide range of data. our goal in this paper is to evaluate the predictive power of cognitive hypotheses for adjective order across 32 languages, and to situate their performance with respect to two baselines: (i) a lower baseline representing random chance accuracy in predicting adjective order, and (ii) a baseline that reflects the best performance that can be achieved in predicting order directly from the distributional and semantic information encoded in modern word embeddings. while this neural distributional baseline provides a strong descriptive account of adjective order, it does not provide an explanation of why adjectives are ordered in the way they are, as the cognitive predictors do. by situating the performance of cognitive predictors between these baselines, our goal is to determine how much progress has been made in the scientific explanation of adjective order over 125 years of research, and how much remains to be explained. the remainder of the paper is structured as follows. section 2 describes the data sources we draw on to operationalize and evaluate predictors of adjective order and how we extract their data. section 3 presents the cognitive predictors and how we implement them. section 4 describes our evaluation method, including the formulation of baselines. section 5 presents the results with some discussion, and section 6 concludes. ## data recent years have seen massive expansions in the availability of crosslinguistic datasets. in particular, the universal dependencies (ud) project @xcite has gathered dependencyparsed corpora of naturalistic text in many languages. it is exactly this dependency-parsed naturalistic data that is useful for a crosslinguistic corpus study of adjective order, because it is possible to easily extract instances of multiple adjectives modifying a single noun, and then to study the ordering patterns found in these instances. our goal is to study the ability of cognitively-motivated theories to predict the attested orders of adjectives as found in these corpora. the primary syntactic configuration that we extract from dependency-parsed corpora is what we call a triple, consisting of a head noun token n with universal part-of-speech noun, modified by exactly two distinct adjective tokens a 1 and a 2 , with universal part-of-speech adj, and with the syntactic relation type amod. given a triple {a 1 , a 2 , n} extracted based on syntactic configuration, our goal is to predict the linear order of the words: whether it is a 1 a 2 n , a 2 a 1 n , a 1 na 2 , etc. we classify triples into three templates: noun-final (aan), noun-medial (ana), and noun-initial (naa), and we predict order within each of these templates. the diversity of lexical types in the triples data is shown in we will refer to these pairs as the training pairs. for our test set of triples, which will be used for the final evaluation of predictors, we use the universal dependencies 2.8 corpora, concatenating non-l2 corpora for each language. in estimating several of our predictors, we make use of word vectors. in all instances, we use the aligned word vectors provided by facebook, foot_0 which were trained on data from wikipedia @xcite . ## predictors we evaluate the performance of eight predictors from the literature on adjective ordering. our choice of predictors is based on the criterion that predictors must have a precise operationalization that can be estimated using the data at hand. the cognitive predictors differ in whether they predict adjectives to come close to the noun, or whether they predict adjectives should come generally earlier in the linear order of an utterance as a whole. when a predictor holds that an adjective should be close to the noun, its effect on linear order should be opposite for pre-and post-nominal adjectives, with varying and often unclear predictions for the ana template. when the predictor holds that an adjective should be generally early, its effect on linear order should have the same sign for pre-and post-nominal adjectives. for predictors that were developed in the monolingual english context, where the only permissible template is aan, the proper polarity of predictions is sometimes unclear for other templates such as naa and ana, as we discuss below. below, we briefly describe each predictor, its history in the linguistics and cognitive science literature, and how it was estimated for our study. frequency several authors have shown adjective frequency to be a reliable predictor of adjective order, with more frequent adjectives appearing earlier @xcite @xcite @xcite . this effect of frequency is consistent with a broader finding that more frequent words appear earlier in sentences. the pattern has been explained in terms of a general preference for more 'accessible' words to go earlier in utterances as a result of a kind of greediness in human sentence production @xcite @xcite . to date, existing studies of frequency effects have focused on english ordering. our frequency predictor consists of the log-transformed raw counts of adjectives appearing as dependents in the training pairs. length another accessibility-based predictor of word order is word length: there is a general tendency for short words and phrases to go before long ones, as evidenced in production experiments and corpus studies @xcite @xcite . applied to adjective order, this predictor has been evaluated successfully only in english @xcite @xcite for a different finding in german). the general short-before-long preference is also considered an accessibility effect @xcite , since short words and phrases are easier to access and produce than long ones. meaning specificity one of the oldest ideas in the literature on adjective ordering holds that adjectives more ''special in meaning'' appear nearer to the noun @xcite . a common way of interpreting meaning specificity concerns the range of nouns an adjective can modify; adjectives applicable to a narrower range of nouns will have a more specific meaning @xcite . here we explore two different operationalizations of meaning specificity. the first is integration complexity (ic), which quantifies the entropy of the probability distribution of a word's heads in dependency trees; adjectives combining with a broader range of nouns as heads will have higher integration complexity and should appear farther form the modified noun @xcite @xcite . the distribution on head nouns given adjectives is estimated from a training corpus to be described below. the second operationalization of meaning specificity is in terms of westbury's (2021) notion of 'likely need'. the intuition for this measure is that adjectives with a multi-purpose meaning will be used across a wider range of contexts, and so, across contexts, speakers will be more likely to need to use those more flexible, more general adjectives and will use them earlier. we adopt @xcite operationalization of this idea: adjectives whose semantic vector is closer to the average adjective vector-the 'category-defining vector', or cdv-will have a more general meaning and will therefore appear earlier. the average adjective meaning is determined according to the token frequency of adjectives in our training pairs. the predictions of the theory for other templates are not clear. meaning closeness another predictor with a century-long history concerns the meaning connection between the adjective and noun. crucially, these predictors depend on the specific noun being modified; the other predictors described so far are only a function of individual adjectives. according to @xcite , the adjective ''most closely connected with [the noun] in meaning'' comes nearest to it. this idea of meaning closeness has resurfaced in various forms in the intervening years (e.g., @xcite @xcite @xcite . for our purposes, we consider two operationalizations. the first, pointwise mutual information, or pmi, quantifies the information that adjectives and nouns have in common on the basis of the extent to which they occur together @xcite @xcite ; adjectives with higher pmi with the modified noun should appear closer to the noun. pmi has a cognitive justification in terms of minimizing processing difficulty under memory limitations @xcite . we calculate pmi using the additively smoothed distribution on head nouns given adjectives in the training data (with smoothing constant α = .001). the second operationalization of meaning closeness is inspired by the distributional hypothesis @xcite , where an adjective operates on a noun by changing its distribution in vector space @xcite . we quantify that change by the vector cosine distance, or vcosd, between the noun and summed nounadjective vectors, which are meant to represent the composition of the adjective and the noun @xcite . the intuition is that some adjectives may drastically change the distribution of the noun, while others do so only negligibly, and this change may relate to adjective order: adjectives with larger vcosd should appear farther from the modified noun. interestingly, vcosd has been related to pmi by @xcite . information gain proposed by @xcite , information gain quantifies the amount of information about a referent provided by the occurrence of an adjective. the cognitive motivation for this predictor is the idea that speakers greedily maximize information gain, resulting in adjectives which offer a greater reduction in uncertainty appearing earlier. in the previous work, information gain was shown to be a strong predictor of adjective order across languages and templates. subjectivity a separate line of research has proposed that adjective subjectivity predicts ordering preferences, with less subjective adjectives appearing closer to the noun @xcite @xcite . the subjectivity hypothesis has been extensively tested in english @xcite @xcite , and in several other languages @xcite @xcite . a number of cognitive justifications for subjectivity as a predictor of adjective order have been offered. for example, @xcite show that orders with more subjective adjectives farther from the noun can maximize communicative success in a setting where semantic composition is noisy. previously, adjective subjectivity has been estimated behaviorally by asking participants how ''subjective'' a given adjective is, or by having them assess its potential for faultless disagreement (i.e., whether two people could both be right while disagreeing about whether some adjective holds of an object; @xcite . this behavioral measure is logistically challenging to collect for a large set of languages and adjectives. therefore, we adopt the method of semantic norm extrapolation @xcite @xcite : we use new and existing experimental datasets to train a neural network to predict subjectivity ratings from word embeddings, and then use this network to deliver estimated subjectivity ratings for adjectives. we use these estimated subjectivity scores in all cases. we use aligned word vectors @xcite ) so that we can transfer this network cross-linguistically to yield extrapolated subjectivity scores across languages. in order to train networks for subjectivity prediction, we adapted existing datasets of experimentally elicited subjectivity ratings for adjectives. for non-english languages, subjectivity ratings were elicited in previous work using the for the subjectivity prediction network's architecture, we used a feedforward network with a single hidden layer of 128 neurons and relu activations, trained using adam (kingma and ba, 2014). we split the above dataset in three ways in an 80/10/10 train/dev/test split within languages. training was performed on the training set until there was no learning on the development set for 10 continuous epochs. evaluations on the resulting test set showed a strong correlation with the empirical data (spearman's ρ = 0.86, pearson's @xmath0.87). ## results tables 3 and 4 present the results of our cognitive predictors, both in terms of the best-performing single predictor (in blue) and the best-performing ensemble of predictors (in red); we also include the lower and distributional baselines. it should be noted up front that, although we present 'best' single predictors and ensembles, the confidence intervals around the predictions are large enough to include nearly all of the alternatives. still, the picture that emerges is clear: ensemble models outperform single predictors, suggesting that no single predictor yet considered will explain all of the ordering regularities; and the distributional baseline exceeds the performance of the ensemble models, suggesting that cognitive science has yet to exhaustively characterize the factors that enter into determining adjective order. however, the progress that has been made over the past century of research is non-negligible: single predictors and ensemble models outperform the lower baseline significantly, at least at the template level, as evidenced by the non-overlapping confidence intervals. in ## conclusion adjective order is an important object of study because it appears to be in many ways universal across languages, and thus offers a test bed for understanding how universal properties of human cognition have shaped language. our results reveal that cognitive theories have made real progress in explaining adjective order across languages: despite these theories being formulated primarily based on the analysis of english, they do yield predictors with fairly consistent crosslinguistic validity. nevertheless, considerable variance remains unexplained. our approach also shows that a massively cross-linguistic approach to comparing and combining cognitive theories is now possible, and we believe this style of approach offers a meaningful way forward for the development and evaluation of future theories of how cognition shapes language.