Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations (2022)

Abstract: We investigate the extent to which modern neural language models are susceptible to structural priming, the phenomenon whereby the structure of a sentence makes the same structure more probable in a follow-up sentence. We explore how priming can be used to study the potential of these models to learn abstract structural information, which is a prerequisite for good performance on tasks that require natural language understanding skills. We introduce a novel metric and release Prime-LM, a large corpus where we control for various linguistic factors that interact with priming strength. We find that Transformer models indeed show evidence of structural priming, but also that the generalizations they learned are to some extent modulated by semantic information. Our experiments also show that the representations acquired by the models may not only encode abstract sequential structure but involve certain level of hierarchical syntactic information. More generally, our study shows that the priming paradigm is a useful, additional tool for gaining insights into the capacities of language models and opens the door to future priming-based investigations that probe the model’s internal states.1

## introduction it has become increasingly clear that modern, neural language models (lms) are capable of representing and learning a broad range of linguistic phenomena @xcite @xcite @xcite . however, many open questions remain about the extent to which specific lms have indeed acquired specific linguistic constructions, about whether these models encode an abstract notion of structure in their representations, and about the best ways to even assess the syntactic abilities of these models. a rich literature has emerged in the last few years addressing these questions, often taking inspiration from methodologies developed in theoretical linguistics, psycholinguistics, neurolinguistics, and language acquisition research @xcite @xcite @xcite , where the same questions have been asked about the human mind/brain for centuries. building on this tradition, this paper turns to structural priming to investigate the degree to which lms encode abstract structural information independent from the concrete words that make up sentences. this phenomenon refers to the fact that humans are more likely to produce-or to more easily comprehend-a sentence of a certain structure x (the target) when they have been exposed before to a sentence of a similar structure x (the prime), than if they had been prompted with a sentence of a different structure y . for example, a native speaker of english will be more inclined to produce the target sentence with a prepositional object in (2-a) after having read sentence (1-a) instead of (1-a), and, vice versa, be more inclined to produce the double-object target sentence (2-b) after having read (1-b) instead of @xcite . similar effects are also observed in language comprehension. we use the term 'structural priming' @xcite rather than 'syntactic priming' (first described in katryn @xcite because it comprises priming of abstract structural information that is not restricted to syntactic hierarchical rules, such as the linear positions of semantic roles or the sequential order of parts of speech. in this paper, we focus mostly on the latter and touch upon syntactic rules in section 7.4. in section 3, we define an efficient novel metric for measuring the effect of priming. for our experiments, we create prime-lm, a large-scale corpus for examining structural priming consisting of ∼1.3m prime-target sentence pairs, as we describe in section 4. earlier work on priming in lms by @xcite operationalized priming as adaptation or implicit learning and thus fine-tuned the model weights in between prime and target. while our priming effect metric is compatible with priming as adaptation, our experiments in this paper concentrate on priming after recent exposure to linguistic context without updating the model weights. this allows us to assess the structural representational abilities acquired by the models during training and investigate to what extent such structural information remains active at inference time. in section 6 and 7 we use our corpus and priming paradigm to answer three main research questions: (1) are modern neural language models susceptible to structural priming? (2) which factors influence the strength of the priming effect? (3) what is the nature of the structural representations acquired by those models? our results show that transformer language models do exhibit structural priming. this finding provides evidence that abstract structural information is encoded by the models to some degree and persists as a model makes predictions about upcoming sentences. the strength of the priming effect is influenced by several factors, including the semantic similarity and the proximity between prime and target, as well as the amount of exposure to a given structure during prompting. our final experiment moreover reveals that the structural representations encoded by the model may not only be sequential but involve a certain level of hierarchical syntactic structure. ## measuring priming we capture the effects of priming by measuring the difference in log probability of a target sentence t x given a prime sentence p x of the same syntactic structure x, vs. t x given p y , a sentence of the exact same semantic and lexical content as p x but differing in syntactic structure y. we call this metric the priming effect (pe): by measuring priming based on a fixed prime-target pair our method is akin to structural priming in comprehension. we condition a target sentence on a prime sentence by concatenating them, separated by a period. the log probability is computed as the sum of token log probabilities of the lm: for example, the pe of the example in the introduction would be computed as follows: ## the prime-lm corpus we create a large-scale set of corpora designed to examine the priming behavior of lms. ## language models we focus our experiments on the class of auto-regressive lms, foot_7 which are trained to predict the next token, in line with human incremental language processing. our methodology can be applied to masked lms as well; we briefly reflect on this in the discussion ( §8). the main focus of our analysis is directed on transformer models @xcite , which constitute the current state of the art in language modeling, and have been shown to produce representations that correlate strongly with human brain signals @xcite . this is the set of models we consider: all transformer lms are imported with the transformers library @xcite . the extraction of the model probabilities is done using the diagnnose library (jumelet, 2020), which provides support for efficient activation extraction. our implementation allows our priming procedure to be efficiently tested on any kind of language model and to be easily reproducible. all our code and corpora are available at https:// github.com/dmg-illc/prime-lm . ## core priming results across lms we initially test all lms described in the previous section on our core corpus, designed to control for lexical overlap and semantic similarity. this provides a clean experimental setup, where the only element shared between prime and target is the abstract sequential structure. the results are reported in figure 2 , split by the structure type of the target sentence. it can be seen that across many models a positive pe is present. we will now discuss these results in more detail. there are two models that exhibit symmetrical priming for both transitive and dative alternations: gpt2-large and gpt2-xl. the other gpt2 models exhibit symmetrical priming for transitive as well, but exhibit moderate asymmetrical priming ## impact of specific factors the next battery of experiments isolates various factors that have been shown to be of influence to priming in human language processing. for each experimental condition, we present a specialized corpus followed by an analysis of the priming effects exhibited by gpt2-large on this data, comparing them to the model's behavior on the core corpus. examples from the core and specialized conditions can be found in ## discussion and conclusions in this paper, we investigated three main questions: (1) are modern neural lms susceptible to structural priming? (2) which factors influence the strength of the priming effect? ( 3 ) what is the nature of the structural representations acquired by those models? to answer these questions, we designed a series of carefully curated large-scale corpora, proposed a metric to measure the degree to which a model is susceptible to priming, and ran a series of experiments on several transformer lms. this methodology constitutes a new way of assessing the representational abilities of lms via examining their behavior in controlled setups, which complements tools like targeted syntactic evaluations and the adaptation-based priming measure by @xcite . our results in section 6 showed that on our core corpus, where we control for lexical overlap and semantic similarity between prime and target, most of the language models we test exhibit some degree of priming for most of the constructions we study. this is important, as it opens up the possibility of using priming to investigate what influences the learned representations of these language models. in section 7, we focused on gpt2-large to conduct a series of subsequent experiments to dig deeper into the impact of different factors on the model's susceptibility to priming. in line with psycholinguistic accounts of residual activation, we found that the effects of priming decrease with the distance between prime and target and increase with the amount of exposure to a given structure. our results indicate that the structural information being encoded is not fully autonomous from semantics: the priming effect is highly boosted by semantic similarity and lexical overlap between the words used in prime and target. such boosting effects are well known to be present in human language processing as well. furthermore, the priming effect partly disappears with semantically implausible prime sentences, suggesting that semantic plausibility is an important cue for the encoding of structure, arguably more so than in human language processing. finally, we showed that priming effects remain positive in the presence of phrases with differing degrees of complexity across prime and target. this offers some insight into the nature of the representations learned by the model: it suggests that, in addition to abstract sequential structure, some degree of hierarchical syntactic information is being represented. the current work does not reveal, for the various conditions tested, what the mechanics of the boosting or suppressing effects are. for example, we do not know whether the boosts from lexical overlap or semantic similarity are the result of an improved match with the same structural representations, or of independent factors that influence priming behaviour. similarly, the precise interplay between semantic plausibility and structural encoding remains unclear. overall, the pattern of results calls for further investigation using interpretability methods, such as probing and feature attributions, which we plan to pursue in future work. an additional aspect that requires further study is the role of the training data and its statistics, for example, regarding the frequency of the different constructions under investigation and the impact this may have on priming asymmetries within an alternation, and on priming behaviour more generally. an important future step to disentangle the factors that may give rise to priming behavior would involve training a range of different model types on the same data. this way it becomes possible to interpret the role that model architecture, model size, training objective, and corpus statistics play in shaping the behavior of the model. an important class of models to include in such studies are masked language models. we conducted a preliminary experiment on three such models, which resulted in biased priming behavior for all (see figure 6 ). we postulate that these models may rely less on the structure of a prime because their bi-directional nature allows them to take the entire target sentence into account. however, in order to adequately determine that this is entirely due to their training objective, and not due to external factors stemming from corpus statistics, future work could control for this with newly trained models. our study reveals novel details about the potential of lms to represent structural information and the persistence of this information when making predictions about upcoming sentences. but more generally, we believe our findings also figure 6 : priming effects for three masked language models on the core corpus: bert @xcite , roberta @xcite , and albert @xcite . to compute sentence probabilities we utilize the pseudo-log-likelihood of @xcite , masking out one token at a time. results for dative yield a similar pattern.