Frequency Effects on Syntactic Rule Learning in Transformers (2021)

Abstract: Pre-trained language models perform well on a variety of linguistic tasks that require symbolic reasoning, raising the question of whether such models implicitly represent abstract symbols and rules. We investigate this question using the case study of BERT’s performance on English subject–verb agreement. Unlike prior work, we train multiple instances of BERT from scratch, allowing us to perform a series of controlled interventions at pre-training time. We show that BERT often generalizes well to subject–verb pairs that never occurred in training, suggesting a degree of rule-governed behavior. We also find, however, that performance is heavily influenced by word frequency, with experiments showing that both the absolute frequency of a verb form, as well as the frequency relative to the alternate inflection, are causally implicated in the predictions BERT makes at inference time. Closer analysis of these frequency effects reveals that BERT’s behavior is consistent with a system that correctly applies the SVA rule in general but struggles to overcome strong training priors and to estimate agreement features (singular vs. plural) on infrequent lexical items.

## introduction many natural language phenomena are best described as the product of applying rules to abstract symbols, without access to the content of these symbols @xcite . most speakers of english will agree, for example, that if "gorp" is a singular noun, then, regardless of the meaning of "gorp", the utterance "the gorp adds nothing" is grammatical, but "the gorp add nothing" is not. the success of contemporary neural language models such as bert @xcite on language understanding tasks, as well as in more targeted linguistic evaluations @xcite , raises the question of whether these systems acquire such symbolic rules. while previous studies have attempted to address such questions, particularly in relation to bert @xcite , prior work has generally not analyzed the relationship between the model's pretraining data and its behavior. as a result, it has been difficult to tease apart the many factors that may influence a model's test time performance. in this paper, we investigate whether pre-trained transformer-based language models learn and apply symbolic rules, focusing on bert's ability to follow the english subject-verb number agreement rule ( §3) as a case study. on our evaluation stimuli ( §4), we find that bert achieves high performance, even on subject-verb pairs that never occurred together in the training set ( §5.1- §5.2). in exploratory data analysis, however, we find that this performance is also influenced by effects from both absolute and relative frequency of verb forms in the training data ( §5.3- §5.4). to confirm these phenomena causally, we perform a series of training interventions where we pre-train bert models on training data for which we have carefully manipulated the frequencies of verb forms ( §6). we further use probing classifiers to attribute observed mistakes either to errors in rule-following or to errors in lexical categorization ( §7). these experiments reveal several insights about bert in the context of rule-governed tasks. first, the high performance of bert on subject-verb combinations that never occurred in the training set is consistent with a model that learns abstract representations of lexical items and patterns, i.e., abstract features and rules. second, bert's performance is influenced by absolute frequency effects, but probing classifiers show that this influence can be explained by the model's inability to learn the features of a verb form (singular vs. plural) for infrequent lexical items, rather than a failure to apply the rule when the verb form has been classified. fi-nally, although bert generally applies rules with high accuracy, it fails to overcome strong priors during training-when one verb form is much more frequent than another, bert tends to produce the more common form, even when it is not consistent with the rule. ## related work targeted syntactic evaluation. we use the targeted syntactic evaluation framework of @xcite and @xcite to measure the model's ability to learn and apply the sva rule. following the setup from @xcite , each test instance consists of a sentence in which a verb has been masked out, and bert's masked language modeling (mlm) parameters are used to score whether the singular or plural form of the verb is a better fit for the masked position. for example, given the sentence "the section [mask] nothing to the info." and set of verb inflections {"add", "adds"}, the model would be considered correct if the mlm prediction assigns a higher score to the singular form "adds" than the plural form "add" since the subject of the masked verb position is "section," which is singular. due to the particulars of bert's mlm task setup, the model is only able to score words that are represented by a single wordpiece. while @xcite dealt with this limitation by restricting evaluation to just those verbs that appear in the original bert model's vocabulary as a single wordpiece, we are able to avoid such compromises because pre-training the models ourselves means that we can add any entries we want to the vocabulary. ## exploratory analyses: what factors correlate with error rates? we first perform an exploratory analysis of how the model's abilities on the sva task vary as a function of pre-training frequency. as discussed in §2, we consider generalization to unseen subject-verb pairs to be evidence of symbolic reasoning (h1 or h3), and strong frequency effects to suggest itemspecific learning (h2 or h3). note that in these experiments, it is not the individual lexical itemsthe subject and verb-that are unseen, only the combination of them in a single sentence. therefore, this analysis evaluates the model's ability to perform abstract reasoning about individual items for which it has learned representations. ## confirmatory analysis: manipulating the training data to better understand the above trends, we design a set of experiments in which we manipulate one variable (absolute or relative frequency of a verb form in pre-training) while holding the other fixed. ## discussion the goal of this work is to determine whether bert performs sva by implicitly representing rules defined over abstract agreement features and characterize the training conditions under which such representations emerge. we differentiate between the representation of the rule ("if x then y") and that of observations (containing the correct agreement features). we draw two conclusions, which suggest a mix of systematic rule-like generalization and unsystematic item-specific inferences. bert appears to represent the correct rule, but fails to predict agreement features for lowfrequency verb forms. although the error rate decreases as a function of frequency of target verb v ( §6.2), bert's ability to predict the agreement feature of v ( §7.2) follows the same trend. this observed behavior is thus consistent with a model that correctly represents the sva rule ( §2), but makes mistakes at inference time due to noise in the represented observations for low frequency verb forms (for example, producing "run" in the context "the dog run" because "run" is incorrectly encoded as singular), rather than due to a failure to represent the concept of singular altogether. bert fails to apply the rule when doing so requires overcoming strong item-specific priors. similar to the absolute frequency trend, we see that bert's error rate on sva also decreases as a function of the frequency of the target verb v relative to its competing form v ( §6.3). unlike above, however, we see no effect of the frequency ratio n v : n v on bert's ability to predict the agreement feature of v when the frequency of v is fixed ( §7.2). these results suggest that bert is heavily influenced by skewed training distributions, preferring to produce more common verb forms over forms consistent with the rule. such behavior could either mean that, when p (v) << p (v ), (1) bert represents the correct sva rule but it is overridden in favor of the prior, or (2) bert does not represent the rule at all. teasing apart these possibilities is a valuable direction for future work. open questions. our results on absolute frequency effects indicate that bert does not infer agreement features until it sees 10-100 examples of a verb, even though it is possible, in principle, to infer agreement features from a single training example (e.g., "all of the dogs dax" implies "dax" is a plural verb form). future controlled studies could investigate how the sample efficiency of inferring agreement information depends on factors such as architecture (e.g., access to morphological signals), size of the model, and amount of training data. analysis of such patterns would elucidate how models like bert (and by extension, transformers and neural networks more generally) learn and generalize, enabling more principled development and deployment. ## conclusions we have studied whether bert's performance on subject-verb agreement exhibits rule-governed behavior. we focus on frequency effects, pre-training multiple bert instances in order to isolate how the model's predictions are affected by absolute and relative verb frequency. our results suggest that bert's behavior is consistent with a system that correctly applies the sva rule in general but struggles to overcome strong training priors and to estimate agreement features (singular vs. plural) on infrequent lexical items.