I NFORMATION - THEORETIC P ROBING E XPLAINS R ELIANCE ON S PURIOUS H EURISTICS
Anonymous authors
Paper under double-blind review
A BSTRACT
Most current NLP systems are based on a pre-train-then-ﬁne-tune paradigm, in which a large neural network is ﬁrst trained in a self-supervised way designed to encourage the network to extract broadly-useful linguistic features, and then ﬁnetuned for a speciﬁc task of interest.
Recent work attempts to understand why this recipe works and explain when it fails.
Currently, such analyses have produced two sets of apparently-contradictory results.
Work that analyzes the representations that result from pre-training (via “probing classiﬁers”) ﬁnds evidence that rich features of linguistic structure can be decoded with high accuracy, but work that analyzes model behavior after ﬁne-tuning (via “challenge sets”) indicates that decisions are often not based on such structure but rather on spurious heuristics speciﬁc to the training set.
In this work, we test the hypothesis that the extent to which a feature inﬂuences a model’s decisions can be predicted using a combination of two factors:
The feature’s extractability after pre-training (measured using information-theoretic probing techniques), and the evidence available during ﬁnetuning (deﬁned as the feature’s co-occurrence rate with the label).
In experiments with both synthetic and natural language data, we ﬁnd strong evidence (statistically signiﬁcant correlations) supporting this hypothesis.
I NTRODUCTION
Large pre-trained language models (LMs) (Devlin et al., 2018; Raffel et al., 2019; Brown et al., 2020) have demonstrated impressive empirical success on a range of benchmark NLP tasks.
However, analyses have shown that such models are easily fooled when tested on distributions that differ from those they were trained on, suggesting they are often “right for the wrong reasons” (McCoy et al., 2019).
Recent research which attempts to understand why such models behave in this way has primarily made use of two analysis techniques: probing classiﬁers (Adi et al., 2017; Hupkes et al., 2018), which measure whether or not a given feature is encoded by a representation, and challenge sets (Cooper et al., 1996; Linzen et al., 2016b; Rudinger et al., 2018), which measure whether model behavior in practice is consistent with use of a given feature.
The results obtained via these two techniques currently suggest different conclusions about how well pre-trained representations encode language.
Work based on probing classiﬁers has consistently found evidence that models contain rich information about syntactic structure (Hewitt & Manning, 2019;
Bau et al., 2019; Tenney et al., 2019a), while work using challenge sets has frequently revealed that models built on top of these representations do not behave as though they have access to such rich features, rather they fail in trivial ways (Dasgupta et al., 2018; Glockner et al., 2018a;
Naik et al., 2018).
In this work, we attempt to link these two contrasting views of feature representations.
We assume the standard recipe in NLP, in which linguistic representations are ﬁrst derived from large-scale selfsupervised pre-training intended to encode broadly-useful linguistic features, and then are adapted for a task of interest via transfer learning, or ﬁne-tuning , on a task-speciﬁc dataset.
We test the hypothesis that the extent to which a ﬁne-tuned model uses a given feature can be explained as a function of two metrics: The extractability of the feature after pre-training (as measured by probing classiﬁers) and the evidence available during ﬁne-tuning (deﬁned as the rate of co-occurrence with the label).
We ﬁrst show results on a synthetic task, and second using state-of-the-art pre-trained
LMs on language data.
Our results suggest that probing classiﬁers can be viewed as a measure of the pre-trained representation’s inductive biases: The more extractable a feature is after pre-training, the less statistical evidence is required in order for the model to adopt the feature during ﬁne-tuning.
Contribution.
This work establishes a relationship between two widely-used techniques for analyzing LMs.
Currently, the question of how models’ internal representations (measured by probing classiﬁers) inﬂuence model behavior (measured by challenge sets) remains open (Belinkov & Glass, 2019;
Belinkov et al., 2020).
Understanding the connection between these two measurement techniques can enable more principled evaluation of and control over neural NLP models.
S ETUP AND T ERMINOLOGY 2. F ORMULATION
Our motivation comes from McCoy et al.
(2019), which demonstrated that, when ﬁne-tuned on a natural language inference task (Williams et al., 2018, MNLI), a model based on a state-of-the-art pre-trained LM (Devlin et al., 2018, BERT) categorically fails on test examples which defy the expectation of a “lexical overlap heuristic”.
For example, the model assumes that the sentence “the lawyer followed the judge” entails “the judge followed the lawyer” purely because all the words in the latter appear in the former.
While this heuristic is statistically favorable given the model’s training data, it is not infallible.
Speciﬁcally, McCoy et al.
(2019) report that 90% of the training examples containing lexical overlap had the label “entailment”, but the remaining 10% did not.
Moreover, the results of recent studies based on probing classiﬁers suggest that more robust features are extractable with high reliability from BERT representations.
For example, given the example “the lawyer followed the judge” / “the judge followed the lawyer” , if the model can represent that “lawyer” is the agent of “follow” in the ﬁrst sentence, but is the patient in the second, then the model should conclude that the sentences have different meanings.
Such semantic role information can be recovered at > 90% accuracy from BERT embeddings (Tenney et al., 2019b).
Thus, the question is:
Why would a model prefer a weak feature over a stronger one, if both features are extractable from the model’s representations and justiﬁed by the model’s training data?
Abstracting over details, we distill the basic NLP task setting described above into the following, to be formalized in the Section 2.2.
We assume a binary sequence classiﬁcation task where a target feature t perfectly predicts the label (e.g., the label is 1 iff t holds).
Here, t represents features which actually determine the label by deﬁnition, e.g., whether one sentence semantically entails another.
Additionally, there exists a spurious feature s that frequently co-occurs with t in training but is not guaranteed to generalize outside of the training set.
Here, s (often called a “heuristic” or “bias” elsewhere in the literature) corresponds to features like lexical overlap, which are predictive of the label in some datasets but are not guaranteed to generalize.
Assumptions.
In this work, we assume there is a single t and a single s ; in practice there may be many s features.
Still, our deﬁnition of a feature accommodates multiple spurious or target features.
In fact, some of our spurious features already encompass multiple features: the lexical feature, for example, is a combination of several individual-word features because it holds if one of a set of words is in the sentence.
This type of spurious feature is common in real datasets: E.g., the hypothesis-only baseline in NLI is a disjunction of lexical features (with semantically unrelated words like “no”, “sleep”, etc.)
(Poliak et al., 2018b; Gururangan et al., 2018).
We assume that s and t frequently co-occur, but that only s occurs in isolation.
This assumption reﬂects realistic NLP task settings since datasets always contain some heuristics, e.g., lexical cues, cultural biases, or artifacts from crowdsourcing (Gururangan et al., 2018).
Thus, our experiments focus on manipulating the occurrence of s alone, but not t alone: This means giving the model evidence against relying on s .
This is in line with prior applied work that attempts to inﬂuence model behavior by increasing the evidence against s during training (Min et al., 2020; Zmigrod et al., 2019; Elkahky et al., 2018).
2. D EFINITIONS
Let X be the set of all sentences and S be the space of all sentence-label pairs ( x, y ) ∈
X × { 0 , 1 } .
We use D ⊂ S to denote a particular training sample drawn from S .
We deﬁne two types of binary features: target ( t ) and spurious ( s ).
Each is a function from sentences x ∈ X to a binary label { 0 , 1 } that indicates whether the feature holds.
Target and spurious features.
The target feature t is such that there exists some function f : { 0 , 1 } → { 0 , 1 } such that ∀ ( x, y ) ∈ S , f ( t ( x )) =
y .
In other words, the label can always be perfectly predicted given the value of t .
1 A feature s is spurious if it is not a target feature.
Partitions of S .
To facilitate analysis, we partition S in four regions (Figure 1).
We deﬁne S s -only to be the set of examples in which the spurious feature occurs alone (without the target).
Similarly,
S t -only is the set of examples in which the target occurs without the spurious feature.
S both and S neither are analogous.
For compactness, we sometimes drop the S ∗ notation (e.g., s -only in place of S s -only ).
t-
only both s- only neither
[Equation]
<latexit sha1_base64="bSiqwQANSN306jL/eI6aFqdfpQw=">AAA00nicnVtbc9zKcaYcx3EYxzlOHv2CCiX7nBTFInWYil9cZd5JcXlfXqSzOiosthcLETdhZrGk4E1VKi9+8Evyg/I//F/8kJ4ZDLoHC/LEYZVE9PfNNOby9XQvFhzmcSTk+vqfXvzor3781z/5m5/+7fLf/ezvf/4PX/3iH29ENi0CuA6yOCvuhr6AOErhWkYyhru8AD8ZxnA7vN9R
[Table]
/op4EuOxxb87fYDruLWzgjV/QCJTR9i/xl1/g3hcZa3m1sAE7WUm0MpRAb7N4XPiJeiA1mWUFFqjCfxTey973b16qt3f0nw1MU/MWqshx/4V+e+zlAOKYtbEPRF9525gAMeRT9d8jxjsk6i02VQUbp6y1eoU0m4Y6Z+qiOJKwqt2LzBtloNzNovsoh1Hkr7VeQc6KJFYP7+dV7/v1eQeZpaC4jS5OznS/N11crpi8g9Fa6H0/iNKxfGyHTu4X6uEwHhu+CpYrwLNW+CF4UeqlWV3QS3hY83YmmVDLk6kCMJh4u/jZN4VfC0/9FcTasvM45yxXp3NW/AtqvAj1APD3YFVdPddQnZOmIV51u9RqxWb6/yda9FFQffWGXwxy4A8xzuJsNizAv2/NXk7zGExhE/tpGIN5QVBf4v0/frWy0f77lsWLmzdrG5tr/3qxufK77fpvX3669Mulf176emlj6d+Wfrd0uHS+dL0UvBi++MOL/3rx35v9zS+b/7H5n6bpj17Uff5pyfnZ/OP/AsWXYDg=</latexit> .e.
, 20% of examples in D provide evidence that s alone should not be used to predict y .
Evidence from Spurious-Only Examples.
We are interested in spurious features which are highly correlated with the target during training.
Given a training sample D and features s and t , we deﬁne the s -only example rate to be a measure of the model’s evidence against the use of s as a predictor of y .
Concretely, s -only rate =
| D s -only | (cid:14) | D | , the proportion of training examples in which s occurs without t (and y = 0 ).
Use of Spurious Feature.
If a model has falsely learned that the spurious feature s alone is predictive of the label, it will have a high error rate when classifying examples for which s holds but t does not.
We deﬁne the s -only error to be the classiﬁer’s error computed only over examples drawn from S s -only .
When relevant, t -only error , both error , and neither error are deﬁned analogously.
Extractability of a Feature.
We want to compare features in terms of how extractable they are given a representation.
For example, given a sentence embedding, it may be possible to predict multiple features with high accuracy, e.g., whether the word “dog” occurs, and also whether the word “dog” occurs as the subject of the verb “run” .
However, detecting the former will no doubt be an easier task than detecting the latter.
We use the prequential minimum description length (MDL) Rissanen (1978)–ﬁrst used by Voita & Titov (2020) for probing–in order to quantify this intuitive difference.
2 MDL is an information-theoretic metric that measures how accurately a feature can be decoded and the amount of effort required to decode it.
Formally, MDL measures the number of bits required to communicate the labels given the representations.
Conceptually, MDL can be understood as a measure of the area under the loss curve: If a feature is highly extractable , a model trained to detect that feature will converge quickly to high accuracy, resulting in a low MDL.
Computing MDL requires repeatedly training a model over a dataset labeled by the feature in question.
To compute
MDL ( s ) , we train a classiﬁer (without freezing any parameters) to differentiate S s -only vs. S neither , and similarly compute MDL( t ).
See Voita & Titov (2020) for additional details on MDL.
We observe similar overall trends when using an alternative metric based on validation loss (Appendix C).
Note that our reported MDL is higher in some cases than that given by the uniform code (the number of sentences that are being encoded).
The MDL is computed as a sum of the costs of transmitting successively longer blocks, using classiﬁers that are trained on previously transmitted data.
The high MDL’s are a result of overﬁtting by classiﬁers that are trained on limited data–and therefore, the classiﬁers have worse compression performance than the uniform baseline.
2. H YPOTHESIS
Stated using the above-deﬁned terminology, our hypothesis is that a model’s use of the target feature is modulated by two factors: The relative extractability of the target feature t (compared to the spurious feature s ), and the evidence from s -only examples provided by the training data.
In particular, we expect that higher extractability of t (relative to s ), measured by MDL ( s ) / MDL ( t ) , will yield models that achieve better performance despite less training evidence.
E XPERIMENTS WITH S YNTHETIC D ATA
Since it is often difﬁcult to fully decouple the target feature from competing spurious features in practice, we ﬁrst use synthetic data in order to test our hypothesis in a clean setting.
We use a simple classiﬁer with an embedding layer, a 1-layer LSTM, and an MLP with 1 hidden layer with tanh activation.
We use a synthetic sentence classiﬁcation task with k -length sequences of numbers as input and binary labels as output.
We use a symbolic vocabulary V with the integers 0 . . .
| V | − 1 .
We ﬁx k = 10 and | V | =
K. We begin with an initial training set of 200 K, evenly split between examples from S both and S neither .
Then, varied across runs, we manipulate the evidence against the spurious feature (i.e., the s -only rate) by replacing a percentage p of the initial data with examples from S s -only for p ∈ { 0% , 0 .
Test and validation sets consist of 1, examples each from S both , S neither , S t -only , S s -only .
In all experiments, we set the spurious feature s to be the presence of the symbol 2 .
We consider several different target features t (Table 1), intended to vary in their extractability.
Table 1 contains MDL metrics for each feature (computed on training sets of 200 K , averaged over 3 random seeds).
We see a gradation of feature extractability, as desired.
[Table]
Figure 2 shows model performance as a function of s -only rate for each of the four features described above.
Here, performance is reported using error rate (lower is better) on each partition ( S s -only ,
S t -only , S both , S neither ) separately.
We are primarily interested in whether the relative extractability of the target feature (compared to the spurious feature) predicts model performance.
We indeed see a fairly clear relationship between the relative extractability ( MDL ( s ) MDL ( t ) ) and the performance of the model, at every level of training evidence ( s -only rate).
For example, when t is no less extractable than s (i.e., contains-1 ), the model achieves zero error at an s -only rate of 0 .
001 , meaning it learns that t alone predicts the label despite having only a handful of examples that support this inference.
In contrast, when t is harder to extract than s (e.g., first-last ), the model fails to make this inference, even when a large portion of training examples provide evidence supporting it.
E XPERIMENTS ON N ATURAL L ANGUAGE
We now investigate whether the same trend holds for widely-used language models ﬁne-tuned on natural language data.
To do this, we ﬁne-tune models for the linguistic acceptability task, a simple sequence classiﬁcation task as deﬁned in Warstadt & Bowman (2019), in which the goal is to differentiate grammatical sentences from ungrammatical ones.
We focus on acceptability judgments since there exists substantial formal linguistic theory that can inform how we deﬁne the target features, as well as recent work in computational linguistics showing that neural language models can be sensitive to spurious features in this task (Warstadt et al., 2020a; Marvin & Linzen, 2018). [Figure] [Figure]
4. D ATA
We design a series of simple natural language grammars that generate a variety of feature pairs ( s, t ) , which we expect will exhibit different levels of relative extractability (MDL ( s ) (cid:14) MDL ( t ) ).
We focus on three syntactic phenomena (described below).
In each case, we consider the target feature t to be whether a given instance of the phenomenon obeys the expected syntactic rules.
We then introduce several spurious features s which we deliberately correlate with the positive label during ﬁne-tuning.
The Subject-Verb Agreement (SVA) construction requires detecting whether the verb agrees in number with its subject, e.g., “the girls are playing” is acceptable while “the girls is playing” is not.
In general, recognizing agreement requires some representation of hierarchical syntax, since subjects may be separated from their verbs by arbitrarily long clauses.
We introduce four spurious features: 1) lexical, in which grammatical sentences begin with speciﬁc lexical items (e.g., “often” ); 2) length, in which grammatical sentences are longer; 3) recent-noun, in which verbs in grammatical sentences agree with the immediately preceding noun (in addition to their subject); and 4) plural, in which verbs in grammatical sentences are preceded by singular nouns as opposed to plural ones.
The Negative Polarity Items (NPI) construction requires detecting whether a negative polarity item (e.g., “any” , “ever” ) is grammatical in a given context, e.g., “no girl ever played” is acceptable while “a girl ever played” is not.
In general, NPIs are only licensed in contexts that fall within the scope of a downward entailing operator (such as negation).
We again consider four types of spurious features: 1) lexical, in which grammatical sentences always include one of a set of lexical items ( “no” and “not” ); 2) length (as above); 3) plural, in which each noun in a grammatical sentence is singular, as opposed to plural; and 4) tense, in which grammatical sentences are in present tense.
Some verbs (e.g. “recognize” ) require a direct object.
However, in the right syntactic contexts (i.e., when in the correct syntactic relation with a wh -word), the object position can be empty, creating what is known as a “gap”.
E.g., “I know what you recognized ” is acceptable while “I know that you recognized ” is not.
The Filler-Gap Dependencies (GAP) construction requires detecting whether a sentence containing a gap is grammatical.
For our GAP tasks, we again consider four spurious features (lexical, length, plural, and tense), deﬁned similarly to above.
The templates above (and slight variants) result in 20 distinct ﬁne-tuning datasets, over which we perform our analyses (see Appendix for details).
Table 2 shows several examples.
For the purposes of this paper, we are interested only in the relative extractability of t vs. s given the pre-trained [Table] representation; we don’t intend to make general claims about the linguistic phenomena per se .
Thus, we do not focus on the details of the features themselves, but rather consider each template as generating one data point, i.e., an ( s, t ) pair representing a particular level of relative extractability.
4. S ETUP
We use three models: T5, BERT, and an LSTM with GloVe embeddings (Devlin et al., 2018; Raffel et al., 2019; Pennington et al., 2014).
5 Both T5 and BERT learn to perform well over the whole test set, whereas the GloVe model struggles with many of the tasks.
We expect that this is because the contextualized pre-training encodes certain syntactic features which let the models better leverage small training sets (Warstadt & Bowman, 2020).
Again, we begin with an initial training set ofexamples, evenly split between both and neither , and then introduce s -only examples at rates of 0%, 0.1%, 1%, 5%, 10%, 20%, and 50%, using three random seeds each.
Test and validation sets consist of 1000 examples each from S both , S neither , S s -only .
In the natural language setting, it is often difﬁcult to generate t -only examples, and thus we cannot compute extractability of the target feature t by training a classiﬁer to distinguish S t -only from a random subset of S neither , as we did in Section 3.
Therefore, we estimate MDL by training a classiﬁer to distinguish between examples from S s -only and examples from S both .
Using the simulated data from Section 3, we conﬁrm that both methods ( S s -only vs. S both and S t -only vs. S neither ) produce similar estimates of MDL( t ) (see Appendix).
4. R ESULTS
For each of our ( s, t ) feature pairs, we plot the use of the spurious feature ( s -only error) as a function of the evidence against the spurious feature seen in training ( s -only example rate).
6 We expect to see the same trend we observed in our synthetic data, i.e., the more extractable the target feature t is relative to the spurious feature s , the less evidence the model will require before preferring t over s .
To quantify this trend, we compute correlations between 1) the relative extractability of t compared to s and 2) the amount of training evidence required for the model to adopt the target feature.
For (2), we deﬁne s -rate (cid:63) to be the lowest s -only example rate at which the ﬁne-tuned model is able to achieve essentially perfect performance (F-score > 0 . 99 ) (see Fig. 3a).
Intuitively, s -rate (cid:63) is the (observed) minimum amount of evidence from which the model is able to infer that the t alone is predictive of the label.
Figure 3 shows these correlations and associated scatter plots.
We can see that relative extractability is strongly correlated with s -rate (cid:63) (Fig. 3b), showing highly signiﬁcant negative correlation for both BERT ( ρ = − 0 . 82 ) and T5 ( ρ = − 0 .
That is, the more extractable t is relative to s , the less evidence the model requires before preferring t .
This relationship holds regardless of whether relative extractability is computed using a ratio of MDL scores or an absolute difference.
We also see that, in most cases, the relative extractability explains the model’s behavior better than does the [Figure] Relative [Figure]
to s [Figure] Target Spurious
Ratio
Difference
T 0.
BERT 0.
RoBERTa 0.
GPT -0.
GloVe 0.
[Figure]
[Figure]
extractability of s or t alone.
This trend is also apparent, albeit weak, for the GloVe model.
BERT,
T5, get > 0 .
99 accuracy on all feature pairs when testing the target feature in isolation, but the
GloVe model solved the target task in isolation for only 55% (11/20) feature pairs.
This partially explains why the GloVe model results are less clear.
Figure 3c shows that T5 (compared to BERT, GPT2, and RoBERTa) requires more data (a higher s -rate (cid:63) ) to perform well. We believe that this may be because we ﬁne-tuned T5 with a linear classiﬁcation head, rather than the purely textual input and output that it used in pre-training.
We made this decision (1) because we had trouble training T5 in this purely textual manner, and (2) using a linear classiﬁcation head over two classes is consistent with the other model architectures.
Figure 4 shows the performance curves for T5 (with BERT, GPT2, RoBERTA, and GloVe in Appendix A), i.e., use of the spurious feature ( s -only error) as a function of the evidence from s -only examples seen in training ( s -only example rate).
Note that each data point is the test performance on a dataset that varies by the amount of evidence from spurious-only examples along the x-axis (with each line corresponding to a different s, t feature pair.)
For pairs with high MDL ratios (i.e., when t is actually easier to extract than s ), the model learns to solve the task “the right way” even when
[Figure]
the training data provides no incentive to do so.
That is, in such cases, the models’ decisions do not appear to depend on the spurious feature s even in cases when s and the target feature t perfectly co-occur in the ﬁne-tuning data.
D ISCUSSION
Our experimental results provide support for our hypothesis: the relative extractability of features given an input representation (as measured by information-theoretic probing techniques) is predictive of the decisions a trained model will make in practice.
In particular, we see evidence that models will tend to use imperfect features that are more readily extractable over perfectly predictive features that are harder to extract.
This insight is highly related to prior work which has shown, e.g., that neural networks learn “easy” examples before they learn “hard” examples (Mangalam & Prabhu, 2019).
Our ﬁndings additionally connect to new probing techniques which have received signiﬁcant attention in NLP but have yet to be connected to explanations of or predictions about SOTA models’ decisions in practice. Fine-tuning may not uncover new features.
The models are capable of learning both the s and t features in isolation, so our experiments show that if the relative extractibility is highly skewed, one feature may hide the other – a ﬁne-tuned model may not use the harder-to-extract feature.
Thus, if one classiﬁer does not pick up on a feature readily enough, another classiﬁer (or, rather, the same classiﬁer trained with different data) may not be sensitive to that feature at all.
This has ramiﬁcations for how we view ﬁne-tuning, which is generally considered to be beneﬁcial because it allows models to learn new, task-relevant features.
Our ﬁndings suggest that if the needed feature is not already extractable-enough after pretraining, ﬁne-tuning may not have the desired effect.
Probing classiﬁers can be viewed as measures of a pre-trained representation’s inductive biases.
Thus far, analysis using probing classiﬁers has primarily focused on whether important linguistic features can be decoded from representations at better-than-baseline levels, but there has been little insight about what it would mean for a representations’ encoding of a feature to be “sufﬁcient”.
Based on these experiments, we argue that a feature is “sufﬁciently” encoded if it is as available to the model as are surface features of the text.
For example, if a ﬁne-tuned model can access features about a word’s semantic role as easily as it can access features about that word’s lexical identity, the model may need little (or no explicit) training signal in order to prefer a decision rule based on the former structural feature.
The desire for models with such behavior motivates the development of architectures with explicit inductive biases (e.g., TreeRNNs).
Evidence that similar generalization behavior can result from pre-trained representations has exciting implications for those interested in sample efﬁciency and cognitively-plausible language learning (Warstadt & Bowman, 2020; Linzen, 2020).
We note that this work has not established that the relationship between extractability and feature use is causal.
This could be explored, for example, using intermediate task training (Pruksachatkun et al., 2020) in order to inﬂuence the extractability of features prior to ﬁne-tuning for the target task.
Recent work suggests, e.g., that ﬁne-tuning on parsing might improve the extractability of syntactic features (Merchant et al., 2020).
R ELATED W ORK
Signiﬁcant prior work analyzes the representations and behavior of pre-trained LMs.
Work using probing classiﬁers (Veldhoen et al., 2016; Adi et al., 2017; Conneau et al., 2018; Hupkes et al., 2018) has suggested that such models capture a wide range of relevant linguistic phenomena (Hewitt &
Manning, 2019;
Bau et al., 2019; Dalvi et al., 2019; Tenney et al., 2019a;b).
Other techniques in this vein include attention maps/visualizations (Voita et al., 2019; Serrano & Smith, 2019), and relational similarity analyses (Chrupała & Alishahi, 2019).
A parallel line of work uses challenge sets to understand model behavior in practice.
Some works construct evaluation sets to analyze weaknesses in the decision procedures of neural NLP models (Jia & Liang, 2017b; Glockner et al., 2018b; Dasgupta et al., 2018; Gururangan et al., 2018;
Poliak et al., 2018b; Elkahky et al., 2018;
Ettinger et al., 2016; Linzen et al., 2016b; Isabelle et al., 2017; Naik et al., 2018; Jia & Liang, 2017a; Linzen et al., 2016a; Goldberg, 2019, and others).
Others use such datasets to improve models’ handling of linguistic features (Min et al., 2020; Poliak et al., 2018a;
Liu et al., 2019), or to mitigate biases (Zmigrod et al., 2019; Zhao et al., 2018; 2019; Hall Maudslay et al., 2019; Lu et al., 2018).
Nie et al.
(2020) and Kaushik et al.
(2020) explore augmenting training sets with a human-in-the-loop methods.
Our work is related to work on generalization of neural NLP models.
Geiger et al.
(2019) discusses ways in which evaluation tasks should be sensitive to models’ inductive biases and Warstadt & Bowman (2020) discusses the ability of language model pre-training to encode such inductive biases.
Work on data augmentation (Elkahky et al., 2018; Min et al., 2020; Zmigrod et al., 2019) is relevant, as the approach relies on the assumption that altering the training data distribution (analogous to what we call s -only rate in our work) will improve model behavior in practice.
Kodner &
Gupta (2020); Jha et al.
(2020) discuss concerns about ways in which such approaches can be counterproductive, by introducing new artifacts.
Work on adversarial robustness (Ribeiro et al., 2018;
Iyyer et al., 2018; Hsieh et al., 2019; Jia et al., 2019; Alzantot et al., 2018; Hsieh et al., 2019; Ilyas et al., 2019; Madry et al., 2017; Athalye et al., 2018) is also relevant, as it relates to the inﬂuence of dataset artifacts on models’ decisions.
A still larger body of work studies feature representation and generalization in neural networks outside of NLP.
Mangalam & Prabhu (2019) show that neural networks learn “easy” examples (as deﬁned by shallow machine learning model performance) before they learn “hard” examples.
Zhang et al.
(2016) and Arpit et al.
(2017) show that neural networks which are capable of memorizing noise nonetheless acheive good generalization performance, suggesting that such models might have an inherent preference to learn more general features.
Finally, ongoing theoretical work characterizes the ability of over-parameterized networks to generalize in terms of complexity (Neyshabur et al., 2019) and implicit regularization (Blanc et al., 2019).
Concurrent work (Warstadt et al., 2020b) also investigates the inductive biases of large pre-trained models (RoBERTa), and when theses models shift from a surface feature (what we call spurious features) to a linguistic feature (what we call a target feature).
In our work, we focus on how to predict which of these two biases characterize the model (via relative MDL).
C ONCLUSION
This work bears on an open question in NLP, namely, the question of how models’ internal representations (as measured by probing classiﬁers) inﬂuence model behavior (as measured by challenge sets).
We ﬁnd that the feature extractability can be viewed as an inductive bias: the more extractable a feature is after pre-training, the less statistical evidence is required in order for the model to adopt the feature during ﬁne-tuning.
Understanding the connection between these two measurement techniques can enable more principled evaluation of and control over neural NLP models.