Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Models (2023)

Abstract: Abstract grammatical knowledge—of parts of speech and grammatical patterns—is key to the capacity for linguistic generalization in humans. But how abstract is grammatical knowledge in large language models? In the human literature, compelling evidence for grammatical abstraction comes from structural priming. A sentence that shares the same grammatical structure as a preceding sentence is processed and produced more readily. Because confounds exist when using stimuli in a single language, evidence of abstraction is even more compelling from crosslingual structural priming, where use of a syntactic structure in one language primes an analogous structure in another language. We measure crosslingual structural priming in large language models, comparing model behavior to human experimental results from eight crosslingual experiments covering six languages, and four monolingual structural priming experiments in three non-English languages. We find evidence for abstract monolingual and crosslingual grammatical representations in the models that function similarly to those found in humans. These results demonstrate that grammatical representations in multilingual language models are not only similar across languages, but they can causally influence text produced in different languages.

## introduction what do language models learn about the structure of the languages they are trained on? under both more traditional generative @xcite and cognitively-inspired usage-based theories of language @xcite @xcite , the key to generalizable natural language comprehension and production is the acquisition of grammatical structures that are sufficiently abstract to account for the full range of possible sentences in a language. in fact, both theoretical and experimental accounts of language suggest that grammatical * equal contribution. the strongest evidence for grammatical abstraction in humans comes from structural priming, a widely used and robust experimental paradigm. structural priming is based on the hypothesis that grammatical structures may be activated during language processing. priming then increases the likelihood of production or increased ease of processing of future sentences sharing the same grammatical structures @xcite @xcite @xcite . for example, @xcite finds that people are more likely to produce an active sentence (e.g. one of the fans punched the referee) than a passive sentence (e.g. the referee was punched by one of the fans) after another active sentence. this has been argued @xcite @xcite @xcite to demonstrate common abstractions generalized across all sentences with the same structure, regardless of content. researchers have found evidence that structural priming for sentences with the same structure occurs even when the two sentences are in different languages @xcite @xcite @xcite @xcite . this crosslingual structural priming takes abstraction one step further. first, it avoids any possible confounding effects of lexical repetition and lexical priming of individual wordswithin a given language, sentences with the same structure often share function words (for discussion, see @xcite . more fundamentally, crosslingual structural priming represents an extra degree of grammatical abstraction not just within a language, but across languages. we apply this same logic to language models in the present study. while several previous studies have explored structural priming in language models @xcite @xcite @xcite , to the best of our knowledge, this is the first to look at crosslingual structural priming in transformer language models. we replicate eight human psycholinguistic studies, investigating structural priming in english, dutch @xcite , spanish @xcite , german @xcite , greek @xcite , polish @xcite , and mandarin @xcite . we find priming effects in the majority of the crosslingual studies and all of the monolingual studies, which we argue supports the claim that multilingual models have shared grammatical representations across languages that play a functional role in language generation. ## background structural priming effects have been observed in humans both within a given language @xcite @xcite @xcite and crosslingually @xcite @xcite @xcite @xcite . in language models, previous work has demonstrated structural priming effects in english @xcite @xcite , and initial results have found priming effects between english and dutch in lstm language models @xcite . as these studies argue, the structural priming approach avoids several possible assumptions and confounds found in previous work investigating abstraction in grammatical learning. for example, differences in language model probabilities for individual grammatical vs. ungrammatical sentences may not imply that the models have formed abstract grammatical representations that generalize across sentences @xcite ; other approaches involving probing (e.g. @xcite often do not test whether the internal model states are causally involved in the text predicted or generated by the model @xcite . the structural priming paradigm allows researchers to evaluate whether grammatical representations generalize across sentences in language models, and whether these representations causally influence model-generated text. furthermore, structural priming is agnostic to the specific language model architecture and does not rely on direct access to internal model states. however, the structural priming paradigm has not been applied to modern multilingual language models. previous work has demonstrated that multilingual language models encode grammatical features in shared subspaces across languages @xcite @xcite , largely relying on probing methods that do not establish causal effects on model predictions. crosslingual structural priming would provide evidence that the abstract grammatical representations shared across languages in the models have causal effects on model-generated text. it would also afford a comparison between grammatical representations in multilingual language models and human bilinguals. these shared grammatical representations may help explain crosslingual transfer abilities in multilingual models, where tasks learned in one language can be transferred to another @xcite @xcite @xcite @xcite @xcite @xcite @xcite . thus, this study presents what is to our knowledge the first experiment testing for crosslingual structural priming in transformer language models. the findings broadly replicate human structural priming results: higher probabilities for sentences that share grammatical structure with prime sentences both within and across languages. ## method we test multilingual language models for structural priming using the stimuli from eight crosslingual and four monolingual priming studies in humans. individual studies are described in §4. ## results in reporting whether the structural priming effects from human experiments replicate in xglm language models, we primarily consider the direction of each effect in the language models (e.g. whether po constructions are more likely after po vs. do primes) rather than effect sizes or raw probabilities. the mean of the relative probabilities assigned by language models to the different constructions in each condition may not be directly comparable to human probabilities of production. humans are sensitive to contextual cues that may not be available to language models; notably, in these tasks, humans are presented with pictures corresponding to events in the structural priming paradigm. furthermore, construction probabilities in language models may be biased by the frequency of related constructions in any of the many languages on which the models are trained. thus, we focus only on whether the language models replicate the direction of the principal effect in each human study. ## discussion we find structural priming effects in at least one language model on each set of stimuli (correcting for multiple comparisons). moreover, we observe a significant effect in all models with the monolingual stimuli, and in the majority of the models for 8 of the 12 crosslingual stimuli. in line with previous work @xcite , this supports the claim that language models learn generalized, abstract, and multilingual representations of grammatical structure. our results further suggest that these shared grammatical representations are causally linked to model output. ## conclusion using structural priming, we measure changes in probability for target sentences that do or do not share structure with a prime sentence. analogously to humans, models predict that a similar target structure is generally more likely than a different one, whether within or across languages. we observe several exceptions, which may reveal features of the languages in question, limitations of the models themselves, or interactions between the two. based on our results, we argue that multilingual autoregressive transformer language models display evidence of abstract grammatical knowledge both within and across languages. our results provide evidence that these shared representations are not only latent in multilingual models' representation spaces, but also causally impact their outputs.