Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attention (2021)

Abstract: We advance a novel explanation of similarity-based interference effects in subject-verb and reflexive pronoun agreement processing, grounded in surprisal values computed from a pretrained large-scale Transformer model, GPT-2. Specifically, we show that surprisal of the verb or reflexive pronoun predicts facilitatory interference effects in ungrammatical sentences, where a distractor noun that matches in number with the verb or pronouns leads to faster reading times, despite the distractor not participating in the agreement relation. We review the human empirical evidence for such effects, including recent meta-analyses and large-scale studies. We also show that attention patterns (indexed by entropy and other measures) in the Transformer show patterns of diffuse attention in the presence of similar distractors, consistent with cue-based retrieval models of parsing. But in contrast to these models, the attentional cues and memory representations are learned entirely from the simple self-supervised task of predicting the next word.

## introduction deep neural network (dnn) language models @xcite @xcite have recently attracted the attention of researchers interested in assessing their linguistic competence @xcite @xcite @xcite and potential to provide accounts of psycholinguistic phenomena in sentence processing @xcite @xcite . in this paper we show how attention-based transformer models (we use a pre-trained version of gpt-2) provide the basis for a new theoretical account of facilitatory interference effects in subject-verb and reflexive agreement processing. these effects, which we review in detail below, have played an important role in psycholinguistic theory because they show that properties of noun phrases that are not the grammatical targets of agreement relations may nonetheless exert an influence on processing time at points where those agreement relations are computed. the explanation we propose here is a novel one grounded in surprisal @xcite , but with origins in graded attention and similaritybased interference @xcite @xcite . we use surprisal as the key predictor of reading time @xcite , and through targeted analyses of patterns of attention in the transformer, show that the model behaves in ways consistent with cue-based retrieval theories of sentence processing. the account thus provides a new integration of surprisal and similarity-based interference theories of sentence processing, adding to a growing literature of work integrating noisy memory and surprisal @xcite . in this case, the noisy representations arise from training the transformer, and interference must exert its influence on reading times through a surprisal bottleneck @xcite . the remainder of this paper is organized as follows. we first provide an overview of some of key empirical work in human sentence processing concerning subject-verb and reflexive pronoun agreement. we then provide a brief overview of the gpt-2 architecture, its interesting psycholinguistic properties, and the method and metrics that we will use to examine the agreement effects. we then apply gpt-2 to the materials used in several different human reading time studies. we conclude with some theoretical reflections, identification of weaknesses, and suggestions for future work. ## agreement interference effects in human sentence processing one long-standing focus of work in sentence comprehension is understanding how the structure of human short-term memory might support and con-strain the incremental formation of linguistic dependencies among phrases and words @xcite @xcite @xcite . a key property of human memory thought to shape sentence processing is similarity-based interference @xcite @xcite . figure 1 shows a simple example of how such interference arises in cue-based retrieval models of sentence processing, as a function of the compatibility of retrieval targets and distractors with retrieval cues @xcite @xcite (corresponding sentences are from wagers et al. (2009)'s exp 4-6 shown in in this study, we focus on interference effects in subject-verb number agreement and reflexive pronoun-antecedent agreement, specifically in languages where the agreement features include syntactic number which is morphologically marked on the verb or pronoun. in such cases, number is plausibly a useful retrieval cue, and it is easy to manipulate the number of distractor noun phrases to allow for carefully controlled empirical contrasts. interference in subject-verb agreement. previous studies @xcite @xcite @xcite attest to both inhibitory interference (slower processing in the presence of an interfering distractor) and facilitatory interference (faster processing in the presence of an interfering distractor), but the existing empirical support for inhibitory interference is weak, and many studies fail to find any evidence for it @xcite @xcite . there is stronger evidence for facilitatory effects, which arise in ungrammatical structures where the verb or pronoun fails to agree in number with the structurally correct target noun phrase, but where either an intervening or preceding distractor noun phrase does match in number. example a. below illustrates, taken from @xcite , where the subject and verb are boldfaced and the distractor noun is underlined: a. the slogan on the posters were designed to get attention. figure 1 : how facilitatory and inhibitory interference effects arise in subject-verb dependency creation in cuebased retrieval parsing. the critical manipulation concerns the overlap of number feature between the distractor, target, and retrieval cue. a bayesian meta-analysis of agreement phenomena was recently conducted with an extensive set of studies @xcite . their analysis of first-pass reading times from eye-tracking experiments on subjectverb number agreement is shown in figure 1 . the evidence from the meta-analysis is consistent with a very small or nonexistent inhibitory interference effect in in the grammatical conditions, with a small but robust facilitatory interference effects in the ungrammatical conditions. concerned that the existing experiments did not have sufficient power to detect the inhibitory effects, @xcite ran a large scale eye-tracking study (185 participants) with materials designed to increase the inhibition effect, and did detect a 9ms effect (95% credible posterior interval 0-18ms). this represents the strongest evidence to date for inhibitory effects in grammatical agreement structures, but even this evidence indicates the effect may be near zero. interference in reflexive pronoun agreement. example b. below shows a pair of sentences from @xcite used to probe facilitatory effects in reflexive pronoun agreement (again, the target antecedent and pronoun are boldfaced and the distractor is underlined): b. (1) interfering the basketball coach who trained the star players usually blamed themselves for the ... (2) non-interfering the basketball coach who trained the star player usually blamed themselves for the ... the empirical record concerning facilitatory effects in reflexive agreement is mixed. some have claimed that such effects do not arise @xcite @xcite , and that this is expected under a model in which the structural constraints from binding theory @xcite serve to effectively filter candidates for retrieval-in short, the parser does not consider or make contact with the ungrammatical distractor noun phrases @xcite . however, a recent bayesian meta-analysis of key experiments by @xcite indicates substantially overlapping posterior estimates of facilitatory effects for subject-verb agreement and reflexive agreement @xcite . concerned again about under-powered studies, @xcite undertook a large scale (181 participants) eye-tracking replication and did find evidence for nearly equivalent facilitatory speedups for reflexive and subject-verb agreement (figure 3 ). this result is not inconsistent with the metaanalysis, but provides stronger evidence that the facilitation effects in reflexives are real. ## gpt-2 for psycholinguistic analysis the psycholinguistic relevance of gpt-2 and its training method. gpt-2 (generative pre-trained transformer-2), introduced by openai in radford et al. (2019) , is a language model with a decoder-only transformer architecture @xcite , and has achieved state-of-the-art performance in diverse downstream tasks. gpt-2 and other large-scaled language models based on transformer architectures were trained on billions of words of text, and engineered with performance in mind, not with concern for psycholinguistic plausibility. why then should we then take them seriously as the basis of psycholinguistic models? we believe that the new transformer-based models have three important properties that make them of psycholinguistic interest. (a) the models are among the first to serve as the basis of systems that achieve human-level performance on a range of linguistic tasks, and they directly generate a key quantity, surprisal of the next word, that we know is an important predictor of reading times in humans @xcite . (b) although the data requirements are currently much greater than that for human language acquisition, the models are trained on a simple task-predict the next word-that may plausibly serve as the basis of a self-supervised learning signal in human language acquisition. the representations that arise from such learning are thus psycholinguistically interesting. (c) the learned soft-attention and parallel content-based retrieval of representations of prior input are architectural properties of the gpt models that align very closely with retrieval-based models of sentence comprehension @xcite . and the structure of these psycholinguistic models was proposed as a response to the challenges of computing long-distance dependencies-the same challenge that motivated the transformer as a departure from standard recurrent architectures @xcite . identifying specialized heads in gpt-2. here we use the medium-sized gpt-2 which is constructed with 12 layers, each of which includes 12 attention heads. previous studies have revealed that individual attention heads in transformer models serve are at least partially specialized in function @xcite @xcite . specifically, @xcite found that certain attention heads are specialized for different dependency relations. following @xcite 's method, we identified heads that are specialized for subject-verb relations and reflexive anaphora resolution. @xcite 's method works as follows. first, sentences are parsed using corenlp dependency parser @xcite . then, relative string positions (e.g., one token back, two tokens back) of all instances in each syntactic dependency were counted. considering the proportion of the most frequent relative position as the baseline, attention heads are selected as specialized for a particular dependency relation if attention is paid for the corresponding dependent at least 10% more often than the baseline. in other words, there must be some evidence that the attention head is sensitive to the dependency and not merely string position. to find attention heads responsible for the relation between subjects and verbs, we used the corenlp parser on 148,376 sentences from the brown corpus and gutenberg corpus provided via natural language toolkit (nltk) @xcite , extracting 49,145 nsubj relations, which associate nominal subjects and their governors which are mostly verbs. the most frequent relative position for nsubj dependency relation is -1, which means that the nominal subjects usually come right before their governor, taking up 42% of the cases. after analyzing the attention distribution pattern using gpt-2, we obtained four syntactic heads that were found to be partly specialized for nsubj dependency relations: head4_3 (59%); head3_6 (51%); head6_0 (49%); head2_9 (49%) foot_0 . although we expect that the four syntactic heads responsible for nsubj dependency relation may play distinct roles, in our analyses here we simply use the best performing head (head4_3). the same method was implemented to find attention heads responsible for reflexive anaphora resolution. the only difference was that we used neuralcoref @xcite to count relative position of antecedents to reflexive anaphora since the dependency parser does not associate antecedents and anaphora. out of 2,660 sentences that includes reflexive anaphora, we extracted 510 sentences where neuralcoref identified a single unique antecedent for the reflexive pronoun. the most frequent relative position for reflexive anaphora and their antecedents was -2, meaning that antecedents appear before reflexive anaphora having one word in between. the proportion of the highest relative position was 22%, requiring 24.2 % of accuracy for attention heads to be considered responsible for reflexive anaphora resolution. we found four heads whose accuracies are higher than the threshold: head1_5 (44%); head3_5 (39%); head4_3 (27%); head6_0 (25%), and we again take the best performing head (head1_5) for further analysis. metrics. we define here three metrics for our analyses: surprisal, attention entropy from syntactic heads, and attention to target. we use surprisal for making reading time predictions, but use the attention metrics to provide insight into the processing at the critical region and therefore the representations computed in the prefix before the critical region. surprisal is thus based on the final prediction of the entire model, but the attention metrics are associated with the attention heads most specialized for our dependencies of interest. surprisal @xcite is defined as the negative log probability of the word given left context. surprisal(w) = -log 2 p (w|context) (1) any use of surprisal requires adoption of some kind of language model; e.g. some past work has used probabilistic cfgs @xcite . here we use gpt-2, which computes after each word a probability distribution over its large lexicon that is conditioned on its internal representation of the left context. attention to target is simply the value of the soft attention vector element that corresponds to the target word position, which we denote attn(w cue , w target ), and indicates how much attention is allocated to the target by one of the specialized attention heads (head4_3 for subject-verb and head1_5 for reflexives.) attention entropy is a variant of @xcite 's information entropy that we use as a measure of how sharply focused (low entropy) or diffuse (high entropy) the attention pattern is. (it may be thought of as a measure of the uncertainty about the attentional target, but because the attention values are not probabilities from which targets are sampled, this interpretation is not strictly warranted). ## subject-verb agreement experiments to investigate whether gpt-2 may predict facilitatory interference effects in subject-verb agreement, we ran gpt-2 on materials from three studies @xcite : 48 sets of sentences from experiments 2-3 in wagers et al. (2009) foot_1 ; 24 sets of sentences from experiments 4-7 in wagers et al. (2009); 48 sets of sentences from dillon et al. (2013) (see these three sets of sentences have in common a 2 × 2 structure with the factors grammaticality (grammatical/ungrammatical) and interference (interfering/non-interfering), as described above. additionally, wagers et al. ( 2009 )'s exp 3 also includes an additional condition, subject (singular/plural) for investigating a possible singularplural asymmetry, i.e., asking whether interference effects are equivalent for plural (for plural verbs) and singular (for singular verbs) distractors. note that sentences from experiments 2-3 in wagers et al. ( 2009 ) involve structures in which the distractor appears before the target, and so test effects of proactive interference. thus the distractors are also more distant from verbs than in the other experimental materials. results of surprisal analyses. figure 4 shows the surprisal computed at the critical verbs in each of the experiments and in each of the four conditions separately (red dots and intervals represent means and conventional 95% confidence intervals). surprisal matches the important qualitative pattern found in the meta-analysis of first-pass reading times: lower surprisal-facilitatory effects-are found in the ungrammatical conditions when the distractor matches the verb's number, and no inhibitory effects are found in the grammatical conditions. furthermore, the effects are largest for the case of retroactive interference, where the distractor follows the target and immediately precedes the verb (figure 4a ), compared to proactive inteference, where the distractor precedes the target (figure 4c ). the exception is that no facilitatory effects were found when the verb is singular and the target subject is plural (see figure 4d ). but the facilitatory effect in this condition was not reliably different from zero in the meta-analysis, and it mirrors a plural-singular asymmetry (or markedness effect) found in agreement attraction in production. results of attention analyses. our conjecture is that in the interfering conditions where the distractor matches the verb in number that the attention of the nsubj-specialized attention head head4_3 will be distributed to both the target and the distractor. it is possible to visualize exactly this pattern using a tool developed by @xcite . figure 5 shows an example visualization. analyses of the attention entropy and attention to target metrics provide quantitative evidence for this conjecture: figure 6 shows two metrics across the four datasets. the interfering conditions always show the highest value of attention entropy and the lowest value of attention to target, which means that the head most specialized for subject-verb relations distributes attention more diffusely and away from the target subject. there is evidence for the expected attention effects even in the grammatical conditions, but in these conditions there is no effect of surprisal. thus, under a theory in which similarity-based interference exerts its effects on reading time through a surprisal bottleneck @xcite , no reading time differences are expected here-even though the underlying representations and attention patterns may reflect the interference. preliminary corpus analysis of ungrammatical subject-verb agreement sentences. one possible explanation for the observed facilitatory interference effects is that gpt-2 was exposed to ungrammatical sentences in the training data that have precisely the interference patterns of the ungrammatical sentences in our experiments. to examine such possibility, we analyzed 241 sentences randomly extracted from a reddit corpus @xcite whose subjects and verbs do not agree in number, and have either interfering or non-interfering distractors in between. the results shown in in the key case of a singular subject and a plural verb, the number of an intervening distractor is about twice as likely to be plural (interfering) rather than singular (non-interfering). see text for a discussion. ## reflexive agreement experiments to examine whether the prediction of gpt-2 are consistent with the null interference effects argued for by @xcite , or show facilitatory interference effects as in the large scale jäger et al. results of the surprisal analyses. summaries of the surprisal (and attention metrics) measured at reflexive anaphora are provided in figure 7. consistent with the large scale replication of dillon et al. (2013) conducted by jäger et al. ( 2020 ) (but inconsistent with the null results reported by dillon et al), we found lower surprisal values in the ungrammatical interfering conditions, consistent with a facilitatory interference effect. results of the attention analyses. we found little or no differences between interfering and noninterfering cases in the two attention metrics at- ## discussion and future directions effects of similarity-based interference have been the province of models of noisy memory rather than models of probabilistic expectations, because in standard probabilistic grammars the expectation for the agreement features of a licensor such as a verb or pronoun should not be conditioned upon the agreement features of constituents other than the target licensee. but we show here that a largescale transformer language model, gpt-2, trained only to predict the next word, nevertheless yields surprisal values that are consistent with facilitatory interference effects due to distractor noun phrases that do not participate in the agreement relations. we also confirmed that two metrics that are easily computed from the transformers' attention mechanism, attention entropy and attention to target, show patterns in the subject-verb experiments that are consistent with cue-based retrieval models. our results are suggestive of a possible interesting link between surprisal and noisy memory representations. the attention patterns that we have discovered must reflect similarity between the representations of the target and distractor noun phrases. this representational similarity is the source of great generalization power, but this generalization can lead to linguistic expectations that are not derived by conventional grammatical analyses. one limitation of our analyses of attention is that they depend on methods for identifying specialized heads for specific dependency types. it is not clear that we understand enough about transformer models to do this reliably. but our results suggest that for at least some dependencies, these simple attention metrics and head selection methods can yield interesting insights. the approach outlined may provide an important way to combine surprisal and noisy memory accounts, maintaining a surprisal bottleneck. using trained transformers has the significant theoretical advantage that the memory representations, the attention/retrieval cues, and thus the predicted similarity effects are learned via a self-supervised prediction task. and so such models naturally yield experience-driven sources of noisy representations that are independent of the process noise assumed in existing memory-based models. combining the process-and experience-based noise in a single model is an important goal for psycholinguistic theory.