Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models (2021)

Abstract: Targeted syntactic evaluations have demonstrated the ability of language models to perform subject-verb agreement given difficult contexts. To elucidate the mechanisms by which the models accomplish this behavior, this study applies causal mediation analysis to pre-trained neural language models. We investigate the magnitude of models’ preferences for grammatical inflections, as well as whether neurons process subject-verb agreement similarly across sentences with different syntactic structures. We uncover similarities and differences across architectures and model sizes—notably, that larger models do not necessarily learn stronger preferences. We also observe two distinct mechanisms for producing subject-verb agreement depending on the syntactic structure of the input sentence. Finally, we find that language models rely on similar sets of neurons when given sentences with similar syntactic structure.

## introduction targeted syntactic evaluations have shown that neural language models (lms) are able to predict the correct token from a set of grammatically minimally different continuations with high accuracy, even in difficult contexts @xcite , for constructions such as subject-verb agreement @xcite , filler-gap dependencies @xcite , and reflexive anaphora @xcite . as an illustration of the targeted syntactic evaluation paradigm, consider the following example, which demonstrates subject-verb agreement across an agreement attractor. here, a model using a linear analysis (i.e., inflecting based on the most recent noun) would choose the ungrammatical inflection, while a model using a hierarchical analysis would choose the grammatical inflection: while we have a reasonable understanding of the generally correct behavior of lms in such contexts, the mechanisms that underlie models' sensitivity to syntactic agreement are still not well understood. recent work has performed causal analyses of syntactic agreement units in lstm (hochreiter and schmidhuber, 1997)-based lms @xcite or causal analyses of lstm hidden representations' impact on syntactic agreement @xcite , but the agreement mechanisms of transformer-based lms have not been as extensively investigated. transformerbased lms' syntactic generalization abilities are superior to those of lstms @xcite , which makes transformer-based models enticing candidates for further analysis. we apply the behavioral-structural method of causal mediation analysis @xcite to investigate syntactic agreement in transformers, following the approach used by @xcite for interpreting gender bias in pre-trained english lms. this method allows us to implicate specific model components in the observed behavior of a model. if we view a neural lm as a causal graph proceeding from inputs to outputs, we can view each model component (e.g., a neuron) as a mediator. we measure the contribution of a mediator to the observed output behavior by performing controlled interventions on input sentences and observing how they change the probabilities of continuation pairs. we focus primarily on gpt-2 @xcite , although we also analyze transformerxl @xcite and xlnet @xcite . we find that both gpt-2 and transformer-xl use two distinct mechanisms to accomplish subjectverb agreement, one of which is active only when the subject and verb are adjacent. conversely, xlnet uses one unified mechanism across syntactic structures. even though larger models assign a higher probability to the correct inflection more often, this does not necessarily translate to a larger margin between the probability of the correct and incorrect options. additionally, in larger models, agreement mechanisms are similar to those in smaller models, but are more distributed across layers. finally, we find that the most important neurons for agreement are shared across different structures to various extents, and that the degree of neuron overlap matches well with human intuitions of syntactic similarity between structures. ## conclusions this study applied causal mediation analysis to discover and interpret the mechanisms behind syntactic agreement in pre-trained neural language models. our results reveal the location and importance of various neurons within various models, and provide insights into the inner workings of these lms. for future work, we suggest intervening on groups of neurons and attention heads to see how these components work together, and extending the analysis to phenomena such as filler-gap dependencies and negative polarity items. further work should also explore the impact of specific verbs on syntactic agreement mechanisms @xcite . lastly, we suggest examining examples where the model makes incorrect predictions to determine how models misuse the mechanisms from section 6.1.