P REDICTING I NDUCTIVE B IASES OF P RE -T RAINED M ODELS
A BSTRACT
Most current NLP systems are based on a pre-train-then-ﬁne-tune paradigm, in which a large neural network is ﬁrst trained in a self-supervised way designed to encourage the network to extract broadly-useful linguistic features, and then ﬁnetuned for a speciﬁc task of interest.
Recent work attempts to understand why this recipe works and explain when it fails.
Currently, such analyses have produced two sets of apparently-contradictory results.
Work that analyzes the representations that result from pre-training (via “probing classiﬁers”) ﬁnds evidence that rich features of linguistic structure can be decoded with high accuracy, but work that analyzes model behavior after ﬁne-tuning (via “challenge sets”) indicates that decisions are often not based on such structure but rather on spurious heuristics speciﬁc to the training set.
In this work, we test the hypothesis that the extent to which a feature inﬂuences a model’s decisions can be predicted using a combination of two factors:
The feature’s extractability after pre-training (measured using information-theoretic probing techniques), and the evidence available during ﬁnetuning (deﬁned as the feature’s co-occurrence rate with the label).
In experiments with both synthetic and naturalistic data, we ﬁnd strong evidence (statistically signiﬁcant correlations) supporting this hypothesis.
I NTRODUCTION
Large pre-trained language models (LMs) (Devlin et al., 2018; Raffel et al., 2019; Brown et al., 2020) have demonstrated impressive empirical success on a range of benchmark NLP tasks.
However, analyses have shown that such models are easily fooled when tested on distributions that differ from those they were trained on, suggesting they are often “right for the wrong reasons” (McCoy et al., 2019).
Recent research which attempts to understand why such models behave in this way has primarily made use of two analysis techniques: probing classiﬁers (Adi et al., 2017; Hupkes et al., 2018), which measure whether or not a given feature is encoded by a representation, and challenge sets (Cooper et al., 1996; Linzen et al., 2016b; Rudinger et al., 2018), which measure whether model behavior in practice is consistent with use of a given feature.
The results obtained via these two techniques currently suggest different conclusions about how well pre-trained representations encode language.
Work based on probing classiﬁers has consistently found evidence that models contain rich information about syntactic structure (Hewitt & Manning, 2019;
Bau et al., 2019; Tenney et al., 2019a), while work using challenge sets has frequently revealed that models built on top of these representations do not behave as though they have access to such rich features, rather they fail in trivial ways (Dasgupta et al., 2018; Glockner et al., 2018a;
Naik et al., 2018).
In this work, we attempt to link these two contrasting views of feature representations.
We assume the standard recipe in NLP, in which linguistic representations are ﬁrst derived from large-scale selfsupervised pre-training intended to encode broadly-useful linguistic features, and then are adapted for a task of interest via transfer learning, or ﬁne-tuning , on a task-speciﬁc dataset.
We test the hypothesis that the extent to which a ﬁne-tuned model uses a given feature can be explained as a function of two metrics: The extractability of the feature after pre-training (as measured by probing classiﬁers) and the evidence available during ﬁne-tuning (deﬁned as the rate of co-occurrence with the label).
We ﬁrst show results on a synthetic task, and second using state-of-the-art pre-trained
LMs on language data.
Our results suggest that probing classiﬁers can be viewed as a measure of the pre-trained representation’s inductive biases: The more extractable a feature is after pre-training, the less statistical evidence is required in order for the model to adopt the feature during ﬁne-tuning.
Contribution.
This work establishes a relationship between two widely-used techniques for analyzing LMs.
Currently, the question of how models’ internal representations (measured by probing classiﬁers) inﬂuence model behavior (measured by challenge sets) remains open (Belinkov & Glass, 2019;
Belinkov et al., 2020).
Understanding the connection between these two measurement techniques can enable more principled evaluation of and control over neural NLP models.
S ETUP AND T ERMINOLOGY 2. F ORMULATION
Our motivation comes from McCoy et al.
(2019), which demonstrated that, when ﬁne-tuned on a natural language inference task (Williams et al., 2018, MNLI), a model based on a state-of-the-art pre-trained LM (Devlin et al., 2018, BERT) categorically fails on test examples which defy the expectation of a “lexical overlap heuristic”.
For example, the model assumes that the sentence “the lawyer followed the judge” entails “the judge followed the lawyer” purely because all the words in the latter appear in the former.
While this heuristic is statistically favorable given the model’s training data, it is not infallible.
Speciﬁcally, McCoy et al.
(2019) report that 90% of the training examples containing lexical overlap had the label “entailment”, but the remaining 10% did not.
Moreover, the results of recent studies based on probing classiﬁers suggest that more robust features are extractable with high reliability from BERT representations.
For example, given the example “the lawyer followed the judge” / “the judge followed the lawyer” , if the model can represent that “lawyer” is the agent of “follow” in the ﬁrst sentence, but is the patient in the second, then the model should conclude that the sentences have different meanings.
Such semantic role information can be recovered at > 90% accuracy from BERT embeddings (Tenney et al., 2019b).
Thus, the question is:
Why would a model prefer a weak feature over a stronger one, if both features are extractable from the model’s representations and justiﬁed by the model’s training data?
Abstracting over details, we distill the basic NLP task setting described above into the following, to be formalized in the Section 2.2.
We assume a binary sequence classiﬁcation task where a target feature t perfectly predicts the label (e.g., the label is 1 iff t holds).
Here, t represents features which actually determine the label by deﬁnition, e.g., whether one sentence semantically entails another.
Additionally, there exists a spurious feature s that frequently co-occurs with t in training but is not guaranteed to generalize outside of the training set.
Here, s (often called a “heuristic” or “bias” elsewhere in the literature) corresponds to features like lexical overlap, which are predictive of the label in some datasets but are not guaranteed to generalize.
Assumptions.
In this work, we assume there is a single t and a single s ; in practice there may be many s features.
Still, our deﬁnition of a feature accommodates multiple spurious or target features.
In fact, some of our spurious features already encompass multiple features: the lexical feature, for example, is a combination of several individual-word features because it holds if one of a set of words is in the sentence.
This type of spurious feature is common in real datasets: E.g., the hypothesis-only baseline in NLI is a disjunction of lexical features (with semantically unrelated words like “no”, “sleeping”, etc.)
(Poliak et al., 2018b; Gururangan et al., 2018).
We assume that s and t frequently co-occur, but that only s occurs in isolation.
This assumption reﬂects realistic NLP task settings since datasets always contain some heuristics, e.g., lexical cues, cultural biases, or artifacts from crowdsourcing (Gururangan et al., 2018).
Thus, our experiments focus on manipulating the occurrence of s alone, but not t alone: This means giving the model evidence against relying on s .
This is in line with prior applied work that attempts to inﬂuence model behavior by increasing the evidence against s during training (Elkahky et al., 2018; Zmigrod et al., 2019; Min et al., 2020).
2. D EFINITIONS
Let X be the set of all sentences and S be the space of all sentence-label pairs ( x, y ) ∈
X × { 0 , 1 } .
We use D ⊂ S to denote a particular training sample drawn from S .
We deﬁne two types of binary features: target ( t ) and spurious ( s ).
Each is a function from sentences x ∈ X to a binary label { 0 , 1 } that indicates whether the feature holds.
Target and spurious features.
The target feature t is such that there exists some function f : { 0 , 1 } → { 0 , 1 } such that ∀ ( x, y ) ∈ S , f ( t ( x )) =
y .
In other words, the label can always be perfectly predicted given the value of t .
1 A feature s is spurious if it is not a target feature.
Partitions of S .
To facilitate analysis, we partition S in four regions (Figure 1).
We deﬁne S s -only to be the set of examples in which the spurious feature occurs alone (without the target).
Similarly,
S t -only is the set of examples in which the target occurs without the spurious feature.
S both and S neither are analogous.
For clarity, we sometimes drop the S ∗ notation (e.g., s -only in place of S s -only ). [Equation]
only
[Equation]
<latexit sha1_base64="bSiqwQANSN306jL/eI6aFqdfpQw=">AAA00nicnVtbc9zKcaYcx3EYxzlOHv2CCiX7nBTFInWYil9cZd5JcXlfXqSzOiosthcLETdhZrGk4E1VKi
[Table]
/OP/
AsWXYDg=</latexit>
Evidence from Spurious-Only Examples.
We are interested in spurious features which are highly correlated with the target during training.
Given a training sample D and features s and t , we deﬁne the s -only example rate as the evidence against the use of s as a predictor of y .
Concretely, s -only rate =
| D s -only | (cid:14) | D | , the proportion of training examples in which s occurs without t (and y = 0 ).
Use of Spurious Feature.
If a model has falsely learned that the spurious feature s alone is predictive of the label, it will have a high error rate when classifying examples for which s holds but t does not.
We deﬁne the s -only error to be the classiﬁer’s error on examples from S s -only .
When relevant, t -only error , both error , and neither error are deﬁned analogously.
In this work, “feature use” is a model’s predictions consistency with that feature; we are not making a causal argument.
Extractability of a Feature.
We want to compare features in terms of how extractable they are given a representation.
For example, given a sentence embedding, it may be possible to predict multiple features with high accuracy, e.g., whether the word “dog” occurs, and also whether the word “dog” occurs as the subject of the verb “run” .
However, detecting the former will no doubt be an easier task than detecting the latter.
We use the prequential minimum description length (MDL) Rissanen (1978)–ﬁrst used by Voita & Titov (2020) for probing–to quantify this intuitive difference.
2 MDL is an information-theoretic metric that measures how accurately a feature can be decoded and the amount of effort required to decode it.
Formally, MDL measures the number of bits required to communicate the labels given the representations.
Conceptually, MDL can be understood as a measure of the area under the loss curve: If a feature is highly extractable , a model trained to detect that feature will converge quickly to high accuracy, resulting in a low MDL.
Computing MDL requires repeatedly training a model over a dataset labeled by the feature in question.
To compute
MDL ( s ) , we train a classiﬁer (without freezing any parameters) to differentiate S s -only vs. S neither , and similarly compute MDL( t ).
See Voita & Titov (2020) for additional details on MDL.
2. H YPOTHESIS
Stated using the above-deﬁned terminology, our hypothesis is that a model’s use of the target feature is modulated by two factors: The relative extractability of the target feature t (compared to the spurious feature s ), and the evidence from s -only examples provided by the training data.
In particular, we expect that higher extractability of t (relative to s ), measured by MDL ( s ) / MDL ( t ) , will yield models that achieve better performance despite less training evidence.
E XPERIMENTS WITH S YNTHETIC D ATA
Since it is often difﬁcult to fully decouple the target feature from competing spurious features in practice, we ﬁrst use synthetic data in order to test our hypothesis in a clean setting.
We use a simple classiﬁer with an embedding layer, a 1-layer LSTM, and an MLP with 1 hidden layer with tanh activation.
We use a synthetic sentence classiﬁcation task with k -length sequences of numbers as input and binary labels as output.
We use a symbolic vocabulary V with the integers 0 . . .
| V | − 1 .
We ﬁx k = 10 and | V | =
K. We begin with an initial training set of 200 K, evenly split between examples from S both and S neither .
Then, varied across runs, we manipulate the evidence against the spurious feature (i.e., the s -only rate) by replacing a percentage p of the initial data with examples from S s -only for p ∈ { 0% , 0 .
Test and validation sets consist of 1, examples each from S both , S neither , S t -only , S s -only .
In all experiments, we set the spurious feature s to be the presence of the symbol 2 .
We consider several different target features t (Table 1), intended to vary in their extractability.
Table 1 contains MDL metrics for each feature (computed on training sets of 200 K , averaged over 3 random seeds).
We see some gradation of feature extractability, but having more features with wider variation would help solidify our results.
[Table]
Figure 2 shows model performance as a function of s -only rate for each of the four features described above.
Here, performance is reported using error rate (lower is better) on each partition ( S s -only ,
S t -only , S both , S neither ) separately.
We are primarily interested in whether the relative extractability of the target feature (compared to the spurious feature) predicts model performance.
We indeed see a fairly clear relationship between the relative extractability (MDL ( s ) (cid:14) MDL ( t ) ) and model performance, at every level of training evidence ( s -only rate).
For example, when t is no less extractable than s (i.e., contains-1 ), the model achieves zero error at an s -only rate of 0 .
001 , meaning it learns that t alone predicts the label despite having only a handful of examples that support this inference.
In contrast, when t is harder to extract than s (e.g., first-last ), the model fails to make this inference, even when a large portion of training examples provide evidence supporting it.
E XPERIMENTS WITH N ATURALISTIC D ATA
We investigate whether the same trend holds for language models ﬁne-tuned with naturalistic data, e.g grammar-generated English sentences.
To do this, we ﬁne-tune models for the linguistic acceptability task, a simple sequence classiﬁcation task as deﬁned in Warstadt & Bowman (2019), in which longer blocks, using classiﬁers that are trained on previously transmitted data. The high MDL’s are a result of overﬁtting by classiﬁers that are trained on limited data–and therefore, the classiﬁers have worse compression performance than the uniform baseline. [Figure]
[Figure] the goal is to differentiate grammatical sentences from ungrammatical ones. We focus on acceptability judgments since formal linguistic theory guides how we deﬁne the target features, as well as recent work in computational linguistics showing that neural language models can be sensitive to spurious features in this task (Marvin & Linzen, 2018; Warstadt et al., 2020a)
[Figure]
4. D ATA
We design a series of simple natural language grammars that generate a variety of feature pairs ( s, t ) , which we expect will exhibit different levels of relative extractability (MDL ( s ) (cid:14) MDL ( t ) ).
We focus on three syntactic phenomena (described below).
In each case, we consider the target feature t to be whether a given instance of the phenomenon obeys the expected syntactic rules.
We then introduce several spurious features s which we deliberately correlate with the positive label during ﬁne-tuning.
The Subject-Verb Agreement (SVA) construction requires detecting whether the verb agrees in number with its subject, e.g., “the girls are playing” is acceptable while “the girls is playing” is not.
In general, recognizing agreement requires some representation of hierarchical syntax, since subjects may be separated from their verbs by arbitrarily long clauses.
We introduce four spurious features: 1) lexical, grammatical sentences begin with speciﬁc lexical items (e.g., “often” ); 2) length, grammatical sentences are longer; 3) recent-noun, verbs in grammatical sentences agree with the immediately preceding noun (in addition to their subject); and 4) plural, verbs in grammatical sentences are preceded by singular nouns as opposed to plural ones.
The Negative Polarity Items (NPI) construction requires detecting whether a negative polarity item (e.g., “any” , “ever” ) is grammatical in a given context, e.g., “no girl ever played” is acceptable while “a girl ever played” is not.
In general, NPIs are only licensed in contexts that fall within the scope of a downward entailing operator (such as negation).
We again consider four types of spurious features: 1) lexical, in which grammatical sentences always include one of a set of lexical items ( “no” and “not” ); 2) length (as above); 3) plural, in which each noun in a grammatical sentence is singular, as opposed to plural; and 4) tense, in which grammatical sentences are in present tense.
Some verbs (e.g. “recognize” ) require a direct object.
However, in the right syntactic contexts (i.e., when in the correct syntactic relation with a wh -word), the object position can be empty, creating what is known as a “gap”.
E.g., “I know what you recognized ” is acceptable while “I know that you recognized ” is not.
The Filler-Gap Dependencies (GAP) construction requires detecting whether a sentence containing a gap is grammatical.
For our GAP tasks, we again consider four spurious features (lexical, length, plural, and tense), deﬁned similarly to above. [Table]
The templates above (and slight variants) result in 20 distinct ﬁne-tuning datasets, over which we perform our analyses (see Appendix for details).
Table 2 shows several examples.
For the purposes of this paper, we are interested only in the relative extractability of t vs. s given the pre-trained representation; we don’t intend to make general claims about the linguistic phenomena per se .
Thus, we do not focus on the details of the features themselves, but rather consider each template as generating one data point, i.e., an ( s, t ) pair representing a particular level of relative extractability.
4. S ETUP
We evaluate T5, BERT, RoBERTa, GPT-2 and an LSTM with GloVe embeddings (Raffel et al., 2019; Devlin et al., 2018; Liu et al., 2019b; Radford et al., 2019; Pennington et al., 2014).
5 Both T and BERT learn to perform well over the whole test set, whereas the GloVe model struggles with many of the tasks.
We expect that this is because the contextualized pre-training encodes certain syntactic features which let the models better leverage small training sets (Warstadt & Bowman, 2020).
Again, we begin with an initial training set of 2000 examples, evenly split between both and neither , and then introduce s -only examples at rates of 0%, 0.1%, 1%, 5%, 10%, 20%, and 50%, using three random seeds each.
Test and validation sets consist of 1000 examples each from S both , S neither , S s -only .
In the natural language setting, it is often difﬁcult to generate t -only examples, and thus we cannot compute extractability of the target feature t by training a classiﬁer to distinguish S t -only from a random subset of S neither , as we did in Section 3.
Therefore, we estimate MDL by training a classiﬁer to distinguish between examples from S s -only and examples from S both .
Using the simulated data from Section 3, we conﬁrm that both methods ( S s -only vs. S both and S t -only vs. S neither ) produce similar estimates of MDL( t ) (see Appendix). Per model, we ﬁlter out feature pairs that the model could not solve both probing tasks in isolation (less than 90% accuracy).
4. R ESULTS
For each ( s, t ) feature pair, we plot the use of the spurious feature ( s -only error) as a function of the evidence against the spurious feature seen in training ( s -only example rate).
7 We expect to see the same trend we observed in our synthetic data, i.e., the more extractable the target feature t is relative to the spurious feature s , the less evidence the model will require before preferring t over s .
To quantify this trend, we compute correlations between 1) the relative extractability of t compared to s and 2) the average test f. score across all rates and partitions of the data 8 , capturing how well and how quickly the model uses (i.e. is consistent with) the target feature Initially, we used a more complicated metric based on the s example rate required for the model to solve the test set. Both report similar trends and correlations.
For posterity, we include details in the Appendix A.2. [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] Figure 3 shows these correlations and associated scatter plots.
We can see that relative extractability is strongly correlated with average test f. score (Figure 3a), showing high correlations for both BERT ( ρ = 0 . 89 ) and T5 ( ρ = 0 . 77 ).
That is, the more extractable t is relative to s , the less evidence the model requires before preferring t , performing better across all partitions.
This relationship holds regardless of whether relative extractability is computed using a ratio of MDL scores or an absolute difference.
We also see that, in most cases, the relative extractability explains the model’s behavior better than does the extractability of s or t alone.
This trend is also apparent, albeit weak, for the
GloVe model.
The pre-trained transformer models get > 0 .
99 accuracy on all feature pairs when testing the target feature in isolation, but the GloVe model solved the target task in isolation for only 55% (11/20) feature pairs.
This partially explains why the GloVe model results are less clear.
Figure 4 shows that T5 (compared to BERT) requires more data to perform well.
This may be because we ﬁne-tuned T5 with a linear classiﬁcation head, rather than only-text input and output it was pre-trained on.
We made this decision (1) because we had trouble training T5 in the original manner, and (2) using a linear classiﬁcation head was consistent with the other model architectures.
Figure 4 shows the performance curves for BERT and T5 (with others the in Appendix A), i.e., use of the spurious feature ( s -only error) as a function of the evidence from s -only examples seen in training ( s -only example rate).
Each data point is the test performance on a dataset that varies by the amount of evidence from spurious-only examples along the x-axis (with each line corresponding to a different s, t feature pair.)
For pairs with high MDL ratios (i.e., when t is actually easier to extract than s ), the model learns to solve the task “the right way” even when the training data provides no incentive to do so:
In such cases, the models’ decisions do not appear to depend on the spurious feature s even when s and the target feature t perfectly co-occur in the ﬁne-tuning data.
D ISCUSSION
Our experimental results provide support for our hypothesis: the relative extractability of features given an input representation (as measured by information-theoretic probing techniques) is predictive of the decisions a trained model will make in practice.
In particular, we see evi- [Figure] dence that models will tend to use imperfect features that are more readily extractable over perfectly predictive features that are harder to extract.
This insight is highly related to prior work which has shown, e.g., that neural networks learn “easy” examples before they learn “hard” examples (Mangalam & Prabhu, 2019).
Our ﬁndings additionally connect to new probing techniques which have received signiﬁcant attention in NLP but have yet to be connected to explanations of or predictions about state-of-the-art models’ decisions in practice.
Fine-tuning may not uncover new features.
The models are capable of learning both the s and t features in isolation, so our experiments show that if the relative extractibility is highly skewed, one feature may hide the other – a ﬁne-tuned model may not use the harder-to-extract feature.
Thus, if one classiﬁer does not pick up on a feature readily enough, another classiﬁer (or, rather, the same classiﬁer trained with different data) may not be sensitive to that feature at all.
This has ramiﬁcations for how we view ﬁne-tuning, which is generally considered to be beneﬁcial because it allows models to learn new, task-relevant features.
Our ﬁndings suggest that if the needed feature is not already extractable-enough after pretraining, ﬁne-tuning may not have the desired effect.
Probing classiﬁers can be viewed as measures of a pre-trained representation’s inductive biases.
Analysis with probing classiﬁers has primarily focused on whether important linguistic features can be decoded from representations at better-than-baseline rates, but there has been little insight about what it would mean for a representations’ encoding of a feature to be “sufﬁcient”.
Based on these experiments, we argue that a feature is “sufﬁciently” encoded if it is as available to the model as are surface features of the text.
For example, if a ﬁne-tuned model can access features about a word’s semantic role as easily as it can access features about that word’s lexical identity, the model may need little (or no explicit) training signal to prefer a decision rule based on the former structural feature.
The desire for models with such behavior motivates the development of architectures with explicit inductive biases (e.g., TreeRNNs).
Evidence that similar generalization behavior result from pre-trained representations has exciting implications for those interested in sample efﬁciency and cognitively-plausible language learning (Warstadt & Bowman, 2020; Linzen, 2020).
We note this work has not established that the relationship between extractability and feature use is causal.
This could be explored using intermediate task training (Pruksachatkun et al., 2020) in order to inﬂuence the extractability of features prior to ﬁne-tuning for the target task; Merchant et al.
(2020) suggests that ﬁne-tuning on parsing might improve the extractability of syntactic features.
R ELATED W ORK
Signiﬁcant prior work analyzes the representations and behavior of pre-trained LMs.
Work using probing classiﬁers (Veldhoen et al., 2016; Adi et al., 2017; Conneau et al., 2018; Hupkes et al., 2018) suggests that such models capture a wide range of relevant linguistic phenomena (Hewitt &
Manning, 2019;
Bau et al., 2019; Dalvi et al., 2019; Tenney et al., 2019a;b).
Similar techniques include attention maps/visualizations (Voita et al., 2019; Serrano & Smith, 2019), and relational similarity analyses (Chrupała & Alishahi, 2019).
A parallel line of work uses challenge sets to understand model behavior in practice.
Some works construct evaluation sets to analyze weaknesses in the decision procedures of neural NLP models (Jia & Liang, 2017b; Glockner et al., 2018b;
Dasgupta et al., 2018; Gururangan et al., 2018; Poliak et al., 2018b; Elkahky et al., 2018;
Ettinger et al., 2016; Linzen et al., 2016b; Isabelle et al., 2017; Naik et al., 2018; Jia & Liang, 2017a; Linzen et al., 2016a; Goldberg, 2019, and others).
Others use such datasets to improve models’ handling of linguistic features (Min et al., 2020; Poliak et al., 2018a;
Liu et al., 2019a), or to mitigate biases (Zmigrod et al., 2019; Zhao et al., 2018; 2019; Hall Maudslay et al., 2019; Lu et al., 2018).
Nie et al.
(2020) and Kaushik et al.
(2020) explore augmenting training sets with human-in-the-loop methods.
Our work is related to work on generalization of neural NLP models.
Geiger et al.
(2019) discusses ways in which evaluation tasks should be sensitive to models’ inductive biases and Warstadt & Bowman (2020) discusses the ability of language model pre-training to encode such inductive biases.
Work on data augmentation (Elkahky et al., 2018; Min et al., 2020; Zmigrod et al., 2019) is relevant, as the approach relies on the assumption that altering the training data distribution (analogous to what we call s -only rate in our work) will improve model behavior in practice.
Kodner &
Gupta (2020); Jha et al.
(2020) discuss concerns about ways in which such approaches can be counterproductive, by introducing new artifacts.
Work on adversarial robustness (Ribeiro et al., 2018;
Iyyer et al., 2018; Hsieh et al., 2019; Jia et al., 2019; Alzantot et al., 2018; Hsieh et al., 2019; Ilyas et al., 2019; Madry et al., 2017; Athalye et al., 2018) is also relevant, as it relates to the inﬂuence of dataset artifacts on models’ decisions.
A still larger body of work studies feature representation and generalization in neural networks outside of NLP.
Mangalam & Prabhu (2019) show that neural networks learn “easy” examples (as deﬁned by shallow machine learning model performance) before they learn “hard” examples.
Zhang et al.
(2016) and Arpit et al.
(2017) show that neural networks which are capable of memorizing noise nonetheless acheive good generalization performance, suggesting that such models might have an inherent preference to learn more general features.
Finally, ongoing theoretical work characterizes the ability of over-parameterized networks to generalize in terms of complexity (Neyshabur et al., 2019) and implicit regularization (Blanc et al., 2019).
Concurrent work (Warstadt et al., 2020b) also investigates the inductive biases of large pre-trained models (RoBERTa), in particular, they ask when (at what amount of pre-training data) such models shift from a surface feature (what we call spurious features) to a linguistic feature (what we call a target feature).
In our work, we focus on how to predict which of these two biases characterize the model (via relative MDL).
C ONCLUSION
This work bears on an open question in NLP, namely, the question of how models’ internal representations (as measured by probing classiﬁers) inﬂuence model behavior (as measured by challenge sets).
We ﬁnd that the feature extractability can be viewed as an inductive bias: the more extractable a feature is after pre-training, the less statistical evidence is required in order for the model to adopt the feature during ﬁne-tuning.
Understanding the connection between these two measurement techniques can enable more principled evaluation of and control over neural NLP models. A CKNOWLEDGEMENTS
We would like to thank Michael Littman for helpful suggestions on how to better present our ﬁndings and Ian Tenney for insightful comments on a previous draft of this work.
We also want to thank our reviewers were their detailed and helpful comments.
This work is supported by DARPA under grant number HR00111990064.
This research was conducted using computational resources and services at the Center for Computation and Visualization, Brown University
A A DDITIONAL R ESULTS
Figure 6, 7, 8, 9, 10 show additional results for all models over all partitions ( both accuracy, neither accuracy, and F-score).
These charts appear at the end of the Appendix.
Details on the MDL statistics are available in Table 3.
[Table]
RoBERTa
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
332 npi
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
MDL( s )
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
A. B EYOND A CCURACY ?
For the transformer models, for 18/20 feature pairs
[Table]
the models are able to solve all the spurious and target features in isolation during probing.
(They do solve the test set in all cases–its that two
[Table]
of the spurious features
ended up being very difﬁcult for the models.)
During the reviews, we did not control for the cases where the model did not solve the probing task.
These 2 extraneous points accentuate the lineplot curves, but do not change the character of the results (nor much adjust the correlations).
In the paper, now, we control for accuracy by ﬁltering out these cases.
With or without this control, the accuracy provides no predictive power about the inductive biases.
We present the correlations without ﬁltering for these cases for consistency with the reviews (Table 4 above); we believe it is important to control for these cases because they could have acted as giveaways, where even accuracy might have worked.
A. A LTERNATE M ETRIC : s - RATE (cid:63)
We initially used a different metric when computing the correlations to compact the lineplots.
Rather than using the average test performance, we looked at the evidence required for the model to solve the test set.
Both of these metrics conceptually capture what we are interested in, but the new one (simply averaging test performance) is much easier to understand, and captures the performance across all partitions.
Here we report the correlations with this evidence required metric instead, which we called s -rate (cid:63) .
Speciﬁcally, we deﬁned it to be: s -rate (cid:63) is the lowest s -only example rate at which the ﬁne-tuned model achieves essentially perfect performance (F-score > 0 . 99 ) (see Figure 5a).
Intuitively, s -rate (cid:63) is the (observed) minimum amount of evidence from which the model infer that t alone is predictive of the label.
See Table 5b for the results.
[Table]
A. R ESULTS USING AUC INSTEAD OF MDL
See Table 5.
A metric similar to MDL for capturing the same intuition is the area under the validation loss curve (AUC) .
This metric is highly related to online MDL in computation.
B I MPLEMENTATION D ETAILS & R EPRODUCIBLITY
Our code is available at: https://github.com/cjlovering/predicting-inductive-biases .
[Table]
There are two major parts to this project in terms of reproducibility: (1) the data and (2) the model implementations.
We describe the templates for the data below in Appendix D – the full details are in the project source.
For the transformer models, we use Hugging Face for the implementations and access to the pre-trained embeddings (Wolf et al., 2019).
We use PyTorch Lightning to organize the training code (Falcon, 2019).
We ﬁx all hyperparameters, which are reported in Table 6.
We want to call attention to BERT requiring much less data than T5 to capture our target features.
At face value, it seems that BERT requires much less data than T5 to capture our target features.
However, we are wary about making such strong claims.
Something to consider here (noted in the
Appendix B is that for T5 we used a linear model rather than formatting the task in text (which is how T5 is trained).
We made this decision (1) because we had trouble training T5 in this purely textual manner, and (2) using a linear classiﬁcation head over two classes is consistent with the other model architectures.
Again, GPT2 and RoBERTa performed on par with BERT, so the difference between the performance of BERT and T5 may be due to how we trained T5.
C M EASURING E XTRACTABILITY I NDIRECTLY
We measure the MDL for t with both and s -only examples.
In the simulated setting we can compare this approach with measuring the MDL directly ( t -only vs neither ).
See Table 7 for MDL results.
The ordering of the feature’s difﬁculty holds across the two methods.
D T EMPLATES FOR N ATURALISTIC D ATA
Each template corresponds to a combination of target features, grammars, and spurious features (the target and spurious features are discussed in Section 4.1).
See Table 8 for a complete list of templates.
See Table 10 for further details about the templates that are used for each of the the target features.
Complete details about implementation of these templates (and all data) will be released upon acceptance.
E W HY EXACTLY IS IT HARD TO GENERATE T - ONLY EXAMPLES ?
Target features may be unavoidably linked to spurious ones.
For example, for a Negative Polarity
Item to be licensed (perhaps smoothing over some intricacies) the NPI (“any”, “all”, etc) must be a downward entailing context.
These downward entailing contexts are created by triggers, e.g., if a negative word like “no” or “not” or a quantiﬁer like “some”.
Linguists who study the problem have assembled a list of such triggers (see Hoeksema (2008)).
Arguably, one cannot write down a correct example of NLP licensing that doesn’t contain one of these memorizable triggers.
Thus, we cannot train or test models on correct examples of NPI usage while simultaneously preventing it from having access to trigger-speciﬁc features.
Hyperparameter
Value
Hyperparameters random seed 1, 2,batch size cumulative mdl block sizes (%) 0.1, 0.1, 0.2, 0.4, 0.8, 1.6, 3.05, 6.25, 12.5,s -only rates (%) 0, 0.1, 1, 5, 10, 20,
T5 Hyperparameters model keycode t5-base model architecture
T5Model warmup linear (default values) optimizer
AdamW pooler last hidden state lr 2ebatch size classiﬁer head
Linear
BERT/RoBERTa/GPT2 Hyperparameters model keycode bert-base-uncased model architecure
BertForSequenceClassiﬁcation warmup cosine (default values) optimizer
AdamW pooler ﬁrst hidden state lr 2ebatch size classiﬁer head
MLP
LSTM (Toy and GloVe)
Hyperparameters optimizer
Adam pooler last hidden state lr 1ebatch size
[Table]
MLP with 1 hidden-layer tanh activation
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
not generate examples with the target feature (free of spurious features), so we used a dataset comprising both and s -only examples.
In the simulated setting, our approach for measuring the target extractability indirectly by using both and s -only examples reports results similar to those when directly using target and neither .
These values are in k bits.
[Table]
[Table]
[Table]
x 1 of the ( N i of the ) ∗ N 2 V x
[Table]
N
[Table]
1 of the ( N i of the ) ∗ N 2 V ythe N
[Table]
that)
[Table]
i V
[Table]
Similar to the NPI example, it’s not possible (to our knowledge) to construct target-only examples for ﬁller-gap since construction requires a wh-word and syntactic gap; thus, we can’t create a positively labeled, grammatical sentence that exhibits a Filler Gap without these elements.
In summary
[Figure]
target-only examples may add new spurious features (as with NPI)
[Figure]
or be impossible to construct because the presence of the target feature implies the presence of the spurious feature (as with ﬁller gaps).
Still, our setup permits the MDL to be computed directly with target-only examples
[Figure]
and so, in cases where it is feasible to create target-only examples (e
[Figure]
g. the Subject-Verb
Agreement templates), it would have bolstered our argument to do so
[Figure]
e s t
F - S c o r e
[Figure]
F MDL ISSUES : O VERFITTING IN THE S YNTHETIC E XPERIMENTS
We found that the MDL exceeds the uniform code length in some of the synthetic experiments.
We found that this occurs because the model overﬁts on the small early-block sizes.
See Figure 11.
[Figure]
[Figure]
[Figure]
[Figure]
Models Overfit on Early Blocks
[Figure]