BERT s of a feather do not generalize together: Large variability in generalization across models with similar test set performance (2020)

Abstract: If the same neural network architecture is trained multiple times on the same dataset, will it make similar linguistic generalizations across runs? To study this question, we fine-tuned 100 instances of BERT on the Multi-genre Natural Language Inference (MNLI) dataset and evaluated them on the HANS dataset, which evaluates syntactic generalization in natural language inference. On the MNLI development set, the behavior of all instances was remarkably consistent, with accuracy ranging between 83.6% and 84.8%. In stark contrast, the same models varied widely in their generalization performance. For example, on the simple case of subject-object swap (e.g., determining that “the doctor visited the lawyer” does not entail “the lawyer visited the doctor”), accuracy ranged from 0.0% to 66.2%. Such variation is likely due to the presence of many local minima in the loss surface that are equally attractive to a low-bias learner such as a neural network; decreasing the variability may therefore require models with stronger inductive biases.

## introduction generalization is a crucial component of learning a language. no training set can contain all possible sentences, so learners must be able to generalize to sentences that they have never encountered before. we differentiate two types of generalization: standard test sets in natural language processing are generated in the same way as the corresponding training set, therefore testing only in-distribution generalization. current neural architectures perform very well at this type of generalization. for example, on the natural language understanding tasks included in the glue benchmark @xcite , several transformer-based models @xcite have surpassed the human baselines from @xcite . however, this strong performance does not necessarily indicate mastery of language. because of biases in training distributions, it is often possible for a model to achieve strong in-distribution generalization by using shallow heuristics rather than deeper linguistic knowledge. therefore, evaluating only on standard test sets cannot reveal whether a model has learned abstract properties of language or if it has only learned shallow heuristics. an alternative evaluation approach addresses this flaw by testing how the model handles particular linguistic phenomena, using datasets designed to be impossible to solve using shallow heuristics. in this line of investigation, which tests out-of-distribution generalization, the results are more mixed. some works have found successful handling of phenomena such as subject-verb agreement @xcite and filler-gap dependencies @xcite . other works, however, have illuminated surprising failures even on seemingly simple types of examples @xcite . such results make it clear that there is still much room for improvement in how neural models perform on syntactic structures that are rare in training corpora. in this work, we investigate whether the linguistic generalization behavior of a given neural architecture is consistent across multiple instances of that architecture. this question is important because, in order to tell which types of architectures generalize best, we need to know whether suc-cesses and failures of generalization should be attributed to aspects of the architecture or to random luck in the choice of the model's initial weights. we investigate this question using the task of natural language inference (nli). we fine-tuned 100 instances of bert @xcite on the mnli dataset @xcite . 1 these 100 instances differed only in (i) the initial weights of the classifier trained on top of bert, and (ii) the order in which training examples were presented. all other aspects of training, including the initial weights of bert, were held constant. we evaluated these 100 instances on both the in-distribution mnli development set and the out-of-distribution hans evaluation set @xcite , which tests syntactic generalization in nli models. we found that these 100 instances were remarkably consistent in their in-distribution generalization accuracy, with all accuracies on the mnli development set falling in the range 83.6% to 84.8%, and with a high level of consistency on labels for specific examples (e.g., we identified 526 examples that all 100 instances labeled incorrectly). in contrast, these 100 instances varied dramatically in their out-of-distribution generalization performance; for example, on one of the thirty categories of examples in the hans dataset, accuracy ranged from 4% to 76%. these results show that, when assessing the linguistic generalization of neural models, it is important to consider multiple training runs of each architecture, since models can differ vastly in how they perform on examples drawn from a different distribution than the training set, even when they perform similarly on an in-distribution test set. ## discussion we have found that models that differ only in their initial weights and the order of training examples can vary substantially in out-of-distribution linguistic generalization. we found this variation even with the vast majority of initial weights held constant (i.e., all the weights in the bert component of the model). we conjecture that models might be even more variable if the pre-training of bert were also redone across instances. these results underscore the importance of evaluating models on multiple restarts, as conclusions drawn from a single instance of a model might not hold across instances. further, these results highlight the importance of evaluating out-of-distribution generalization; since all of our instances displayed similar in-distribution generalization, only their out-ofdistribution generalization illuminates the substantial differences in what they have learned. in stark contrast to the models we have looked at-which generalized in highly variable ways despite being trained on the same set of exampleshumans tend to converge to similar linguistic generalizations despite major differences in the linguistic input that they encounter as children @xcite . this suggests that reducing the generalization variability of nlp models may help bring them closer to human performance in one major area where they still dramatically lag behind humans, namely in out-of-distribution generalization. how could the out-of-distribution generalization of models be made more consistent? the variability that we have observed likely reflects the presence of many local minima in the loss surface, all of which are equally attractive to our models. this makes the model's choice of a minimum essentially arbitrary and easily affected by the initial weights and the order of training examples. to reduce this variability, then, one approach would be to use models with stronger inductive biases, which can help distinguish between the many local minima. an alternate approach would be to use training sets that better represent a large set of linguistic phenomena, to decrease the probability of there being local minima that ignore certain phenomena. figure 7 : results for models trained on mnli. the mnli column reports accuracy on the mnli matched development set, where there are three possible labels (entailment, contradiction, and neutral). the remaining columns are subsets of the hans dataset, with neutral and contradiction merged into a single label, non-entailment, such that there are only two possible labels: entailment and non-entailment. the examples that are consistent with the heuristics are those that have a correct label of entailment, while the examples that are inconsistent with the heuristics are those with a correct label of non-entailment. all statistics are based on 100 runs.