How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure (2023)

Abstract: Language models are typically evaluated on their success at predicting the distribution of specific words in specific contexts. Yet linguistic knowledge also encodes relationships between contexts, allowing inferences between word distributions. We investigate the degree to which pre-trained transformer-based large language models (LLMs) represent such relationships, focusing on the domain of argument structure. We find that LLMs perform well in generalizing the distribution of a novel noun argument between related contexts that were seen during pre-training (e.g., the active object and passive subject of the verb spray), succeeding by making use of the semantically organized structure of the embedding space for word embeddings. However, LLMs fail at generalizations between related contexts that have not been observed during pre-training, but which instantiate more abstract, but well-attested structural generalizations (e.g., between the active object and passive subject of an arbitrary verb). Instead, in this case, LLMs show a bias to generalize based on linear order. This finding points to a limitation with current models and points to a reason for which their training is data-intensive.1

## introduction competent speakers of a language know how likely a word w is to appear in a specific context c, which we define as the sequence of words that surrounds a certain position in a sentence. any speaker of english knows, for example, that the word kids is reasonably likely to occur in the context the hit the ball and that ball is likely to appear in the context the kid hit the . we call this kind of knowledge about specific words in specific contexts ''type 0 knowledge.'' type 0 knowledge: for a specific word w, w occurs with probability p in context c. by itself, such type 0 knowledge is woefully incomplete. competent speakers need to be able to generalize their knowledge of distributions across certain pairs of contexts: type 1 knowledge: for all words w, if w occurs in c, w can also occur in c . type 1 knowledge tells a speaker that appearance in one context is predictive of appearance in another. not only do english speakers know that kids or ball are likely to occur in the specific contexts given above, they also know that occurrence in these contexts is predictive of occurrence in others, where the determiner is changed, other nouns are used, adverbs are added, etc., essentially any other context in which the relation of the predicted noun to the verb hit remains. the relevant bit of type 1 knowledge thus implicates sensitivity to linguistic abstractions like grammatical role: a noun that is a likely subject of a particular sentence with active transitive hit will also be a likely subject of another sentence in which it is also the subject of hit. in fact, type 1 knowledge goes further still: any noun that is a likely object of an active transitive hit is a likely subject of passive hit. one kind of type 1 generalization can thus be characterized in terms of thematic role: both the object of active transitive hit and the subject of passive hit are mapped to the theme role of an hitting event, and this thematic role imposes certain selectional preferences on the choice of noun. similarly, both the subject of active transitive hit and the object of the by-phrase in passive hit are mapped to the agent role, which gives rise to other selectional preferences that apply to both contexts. in this case, the relevant bit of type 1 knowledge says that contexts for a word that put it in the same thematic role with respect to hit support type 1 generalization. in fact, a competent speaker has knowledge of even more abstract generalizations over contexts: type 2 knowledge: for all contexts c and c , if c and c are appropriately related, then for all w, if w can occur in c, w can also occur in c . characterizing type 2 knowledge requires defining what it means for two contexts to be appropriately related, and for the purposes of this paper we will focus on contexts in which the same thematic role is assigned. the type 1 knowledge just discussed concerning the arguments of hit is a specific instance of such a relation. the relevant type 2 knowledge is a meta-level generalization, holding across argument positions in active and passive sentences regardless of the specific verb or the roles it assigns. for example, object experiencer verbs like bother show a thematic mapping distinct from the one found with hit: active transitive subjects are the theme, while the object is the experiencer (and hence necessarily animate). this means that likely subjects and objects of active experiencer verbs differ substantially from those of verbs like hit. nonetheless, when each type of verbs is passivized, the same type 2 generalization holds: the role assigned to the passive subject, and hence which nouns are likely subjects, is the same as the role of the corresponding active object. the range of pairs of contexts that share thematic roles, and hence fall under the type 2 generalization under discussion, goes well beyond voice alternations like active and passive. for instance, in a cleft, any argument of a verb may appear in a position to the left of its unmarked (active) use. and once again, english speakers understand that the argument out of its canonical position bears the same thematic relation to the event, and shows the same preferences for possible nouns. in other words, speakers of a language understand not only relationships between specific structures with specific lexical items, but also higher level productive relationships between structures. recent work has demonstrated that large language models (llms) exhibit sensitivity to a wide range of subtle regularities of linguistic form @xcite @xcite . yet because of the vast quantities of data on which llms are trained, it is difficult to determine the degree to which these results derive from type 0, type 1, or type 2 knowledge. given sufficient quantities of training data, a llm can behave as though it abides by type 1 or type 2 knowledge, so long as it has received explicit and consistent evidence about each context. but in order to be robust in the face of infrequent words and rare constructions, true type 1 and type 2 knowledge will be crucial. in this paper, we assess the presence of type 1 and type 2 generalizations in llms, specifically in the domain of the mapping between structural position and thematic role as diagnosed by selectional preferences. focusing on the bert family of transformerbased language models (bert, @xcite @xcite , our strategy is to introduce novel tokens into the models' vocabularies. we fine-tune pre-trained llms in a single structural context that links each novel token to a specific participant role (cf. @xcite , and then examine predictions in unobserved but related structural contexts for the novel tokens. we consider a different set of verbs and a wider range of structural contexts than were investigated in this previous work, and investigate what underlies the models' behavior in more detail by examining the embeddings of the novel words. in addition, while this work has focused on type 1 knowledge, we investigate type 2 knowledge as well, in experiment 2. since the llms have no experience with the novel tokens we teach them in the test contexts, whatever success they have must be due to knowledge acquired during pre-training. this allows us to address our general question about llms' knowledge of relationships between contexts by focusing on specific questions about their knowledge of argument structure. we find that llms demonstrate relatively robust type 1 knowledge: they generalize the use of novel argument tokens across different structures for existing verbs, suggesting that their knowledge of position-role mappings is abstract enough to recognize a notion of construction over which generalizations can be stated. however, deeper probing reveals that they do not exhibit human-like type 2 knowledge of generalization across construction types. we show this by introducing a new verb into the models' vocabulary that has novel selectional preferences. though llms learned these novel preferences and generalized them to structures that are superficially similar to those in the fine-tuning data, they failed to consistently generalize them to other contexts in a structurally defined, human-like manner. instead, their generalizations were derived through heuristics based on linear order, yielding high performance in structures where the relative linear order of arguments and main verb is preserved (e.g., polar questions), and poor performance in structures where the relative linear order of arguments is reversed (e.g., canonical passives). we conclude that the models represent abstract structures, but that the type 2 generalizations they seem to represent rely on surface properties of the training data. ## experiment 1: assessing type 1 generalization with novel nouns experiment 1 examines the degree to which llms show evidence of type 1 knowledge by examining position-role mappings/selectional preferences across constructions. we explore whether fine-tuning on a novel noun, presented in a single context, will allow the llm to predict its appearance in a related context. our primary type of relation among contexts derives from argument structure alternations, where the same set of thematic roles can be realized with distinct structures. we adopt this strategy following @xcite , who examine type 1 knowledge with novel arguments of dative verbs. we focus instead on the spray/load alternation. in (3) we see that spray/load alternating verbs can occur in a theme-object (to) structure, with the theme as the object and the goal as prepositional argument; or in a goal-object (go) structure, with the goal as the object and the theme as the prepositional argument. i sprayed the paint onto the gorx. i sprayed the gorx with the paint. we introduced a novel token that can appear in each of the non-subject argument positions: thax for theme arguments and gorx for goal arguments. 2 we fine-tuned llms on a small set of simple sentences containing spray/load verbs in one of their alternating forms: to or go. all of the sentences for a given run were headed by one of two spray/load verbs: spray or load. this meant that there were four distinct fine-tuning sets, one for each combination of the syntactic structure (to or go) and lexical verb. every example in these sets contained exactly one novel token (as in (3)), with the remaining arguments drawn from a small set of semantically plausible nouns. this yielded a total of 12 fine-tuning examples per set. 3 our validation set includes four subparts, whose performance was averaged: the fine-tuning dataset evaluated without dropout, sentences instantiating the opposite form of the alternation (e.g., spray go used spray to in the validation set, 2 for both experiments 1 and 2, we added the novel tokens to the llms' vocabularies such that they were not split into subword tokens during tokenization (verified during each run). our scripts also verified during each run that all training and test sentences were tokenized identically when using a tokenizer with the added tokens and the same tokenizer without the added tokens (except in the position of the added tokens). 3 an anonymous reviewer notes that this makes our fine-tuning sets quite small, which may affect their ability to learn a good representation of the novel tokens. however, we note that @xcite find that llms of the sort we investigate are able to glean relevant distributional information from a fine-tuning set consisting of as little as two short sentences. in addition, the models achieved high levels of success in experiment 1. for experiment 2, where less success is seen, our fine-tuning sets were larger (288 sentences). thus, we do not believe the small size of the fine-tuning sets in experiment 1 is likely to have had a negative impact on our results. and vice versa), the dataset with the other finetuning verb for that alternation (e.g., spray go used load to in the validation set, and vice versa), and the dataset in passive (e.g., active spray go used passive spray go in the validation set, and vice versa). the validation set was thus constructed to penalize overfitting to our fine-tuning sentences at the expense of sentences with various other structures compatible with our verbs. to assess an llm's sensitivity to type 1 generalizations, our test set considered the fine-tuned model's predictions on sentences that differed from the fine-tuning data along three structural dimensions: (i) the argument structure alternation, (ii) passivization, and (iii) other syntactic transformations, including clefting, polar and (matrix/embedded) wh-question formation, negation, raising, relative clause formation, and particle shift. in contrast to @xcite test data, which included only four test structures (crossing the alternation and passivization), the way we combined our three dimensions gave rise a much broader range of 78 test structures. we show a subset of these here, with the positions corresponding to each novel word filled in with the expected token. (cleft p-obj go passive) e. which thax was the gorx sprayed with? (matrix wh-p-obj go passive) f. i wonder which gorx the man seems to have sprayed onto the thax. to examine generalization to sentences with verbs distinct from those used in the fine-tuning data, but which share their thematic roles and participate in the alternation, we used the following verbs in test data, grouped into relevant lexical semantic subclasses (based on @xcite . during evaluation, novel nouns were replaced by a mask token, and the predicted probabilities for these positions were extracted from the llm. a prediction for a position was taken to be accurate if the model assigns higher probability to the expected novel token than the unexpected one. foot_2 ## experiment 2: assessing type 2 generalization with novel verbs the results of experiment 1 show that llms can generalize the use of novel tokens to related structural contexts, demonstrating knowledge of position-role mappings that cuts across many constructions. our exploration of the learned embeddings of the novel tokens suggests that the llms' ability to generalize in this way may be linked to semantic properties encoded in individual token embeddings. if the model knows that words represented in a certain embedding subspace are compatible with multiple structural contexts and it learns that a novel word's representation falls in that subspace, it will generalize the distribution of a word learned in one context to another. however, such a strategy does not suffice for type 2 generalization, which requires the recognition of systematic relationships across contexts, independent of the specific properties of the contexts. a model could succeed in experiment 1 by learning preferences for each context separately, so long as these are characterized by the same embedding subspace. we would then expect the syntactic complexity effect shown in experiment 2 examines type 2 generalization more directly. we modify the experiment 1 paradigm to introduce a novel verb token with novel selectional preferences. this renders the novel verb unlike existing ones, so the llm cannot succeed by assimilating the novel verb to a known class of verbs with associated embedding subspace, as it did with experiment 1's novel nouns. instead, success requires recognizing regularities that cut across all verbs, which will apply to the novel verb. we are not aware of studies that explicitly demonstrate a human capacity for this kind of generalization, though the following thought experiment suggests it would be possible. suppose we invent a new video game which includes an action called blorking that can be done only to an idiosyncratic set of objects-say mushrooms, clocks, and starfish. if we expose new gamers to sentences with blork only in the active voice as in (9a), it seems clear that they will generalize their knowledge of its selectional preferences in the way that a type 2 generalization would suggest, for example preferring the appropriate nouns in the passive subject position, as in (9b). are llms capable of a similar feat? note that we are not saying that people do not represent type 0 and type 1 generalizationsthey clearly do (e.g., @xcite . however, we believe people represent type 2 generalizations, too. our question is simply whether models like bert have such generalizations, or whether they are limited to the type 0 and type 1 generalizations we examined in experiment 1. experiment 2 focuses specifically on the generalization from active to passive sentences discussed above, where active transitive objects are subject to the same preferences as passive subjects, regardless of the specific choice of verb. our fine-tuning dataset consists of 8 sentences with a novel verb used in the perfect, like the following. sentences differ only in the choice of preverbal adverb and presence (or not) of a sentence-final modifier. for each of blork's argument positions, [subj] and [obj], we select 6 nouns in the llm vocabulary as possible arguments. to ensure that assimilation to the selectional properties of an existing verb was not a viable strategy, we chose nouns that did not exhibit a clear preference for one position over the other prior to fine-tuning. 9 during fine-tuning, sentences were presented with both the novel verb and its argument nouns masked, as this maximized performance. as validation sets, we used the fine-tuning set with dropout disabled, as well as perfect passive, past-tense active, and past-tense passive sentences with the novel verb. we use these validation set subparts to ensure that the model did not overfit to active structures (via the inclusion of passive sentences), nor to perfect structures (via the 9 we did this as follows: for each model we randomly initialized the novel verb embedding. across the set of 8 fine-tuning sentences as well as 5 more complex sentences, we compared the model's predictions in subject and object positions for all nouns from the subtlex corpus @xcite @xmath0. from this set of predictions, we identified the 12 nouns with the lowest average sum of squared log odds ratios, i.e., s log p(@xmath1. we found finetuning until convergence did not always produce the best generalization, as overall validation loss would decrease despite overfitting to one of our validation set subparts. instead, we fine-tuned for a maximum of 260 epochs (= weight updates), with a minimum of 100 epochs and a patience of 30, a regimen that yielded the most success for bert. 10 our test set consisted of a similar set of sentence types to those from experiment 1, which included passive sentences containing the verb blork. to form such sentences, we exploited the widely attested homophony of english past and passive participles in order to create passive sentences using the same, fine-tuned, novel verb token for blorked. ## related work past work has probed token embeddings for knowledge of argument structure @xcite @xcite @xcite . other work has focused on neural networks' ability to predict the likelihood of a verb or noun in forms of an argument structure alternation @xcite @xcite @xcite @xcite , and whether llms distinguish plausible from implausible argument-role mappings in role-reversal sentences @xcite . though revealing of type 0 knowledge, that work does not address whether llms can apply such knowledge productively, which is what drives our study. experiment 1 adapts @xcite approach to probe type 1 knowledge, but we consider different verbs in a wider range of structures, allowing us to assess this knowledge in a more refined way along dimensions such as linear order and movement type. however, our work extends these previous investigations by considering in more detail than this work how the models achieved success by examining the learned embeddings of the novel words, finding that the possibility of assimilation seemed to underlie success, and by examining type 2 knowledge directly. @xcite found bert predicted unseen in-group pairings of novel verbs with novel nouns to be more likely than unseen out-group pairings, showing bert can generalize selectional preferences across verbs. they also showed bert can generalize the use of novel verbs that occurred in contexts compatible with one form of an argument structure alternation to the other context (e.g., a novel verb v in a do context to v in a pd context). while this work also focuses on the generalization of selectional preferences, we examine structures beyond the argument structure alternation itself, giving us a better assessment of bert's abilities with type 2 generalization over structurally defined relations between contexts. methodologically, @xcite work is most similar to ours. they reinvestigate @xcite claim that bert can generalize its knowledge of subject-verb number agreement in english to novel structures and lexical items. @xcite show that goldberg's results hold only for simple sentences and those whose lexical patterns match naturally occurring data. bert fails to generalize correctly to grammatical but nonsensical sentences in more complex structures, where as little as a single attractor noun intervenes between the correct agreement target and the verb. @xcite came to similar conclusions on the basis of a different approach to subject-verb agreement, finding that models were less likely to predict correct agreement morphology for verbs that were improbable in particular contexts. this work provides a distinct but complementary perspective regarding our interpretation of our results, given our observation that models seem to do well only when they can assimilate novel words to existing words (experiment 1, but not experiment 2). this suggests that a general reliance on surface-level similarities to the pre-training data underlies success in generalization. @xcite show that this is true of the morphological fact of subject-verb agreement, and we show it is true for the structural fact of argument structure alternations (and movement more generally). ## conclusion we investigated the performance of transformerbased language models on two tasks requiring structural generalization. experiment 1 revealed that llms were generally able to succeed at predicting novel arguments to known verbs in correct positions in structures different from those in the fine-tuning data. an investigation of the learned embeddings of the novel tokens showed that the llms can use a syntactic bootstrapping strategy @xcite , where information about syntactic distribution is used to inform hypotheses about meaning. specifically, llms assimilate the embeddings of novel tokens to those of existing words that share their distribution. if an llm constrains distributions on the basis of embedding subspaces, the distribution of the novel token in a previously observed context in which it has not appeared can thereby be driven by its similarity to other tokens, giving rise to successful type 1 generalization. experiment 2 found that, when this strategy was unavailable, models were able to generalize successfully only when the relative linear order of the novel verb's arguments matched in the fine-tuning and test data, but degraded noticeably for reversed orders. we conclude that the models we investigated can represent type 2 knowledge across structural contexts, but that the kind of type 2 knowledge they represent is based on surface level properties like the relative linear order of corresponding elements. while this study focuses on one microscopic aspect of linguistic knowledge, the lessons it offers about what llms are capable of are significant. impressive as their performance on a wide range of downstream tasks is they appear to have limited ability to generalize in a human-like structure-sensitive fashion. appearances to the contrary may stem from the separate learning of distributional properties across multiple contexts. the necessity for such redundant learning imposes a high demand on the amount of training data that is required for llms, which we expect can be satisfied for only a very few high resource languages. and at the university of toronto, northwestern university, stony brook university, ucla, and heinrich heine university düsseldorf. the comments and suggestions of the anonymous tacl reviewers were also invaluable. this work was made possible by support from national science foundation grant @xcite