C URRICULUM : A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding
Abstract
In the age of large transformer language models, linguistic evaluation play an important role in diagnosing models’ abilities and limitations on natural language understanding.
However, current evaluation methods show some significant shortcomings.
In particular, they do not provide insight into how well a language model captures distinct linguistic skills essential for language understanding and reasoning.
Thus they fail to effectively map out the aspects of language understanding that remain challenging to existing models, which makes it hard to discover potential limitations in models and datasets.
In this paper, we introduce C URRICULUM as a new format of NLI benchmark for evaluation of broad-coverage linguistic phenomena.
C URRICULUM contains a collection of datasets that covers 36 types of major linguistic phenomena and an evaluation procedure for diagnosing how well a language model captures reasoning skills for distinct types of linguistic phenomena.
We show that this linguisticphenomena-driven benchmark can serve as an effective tool for diagnosing model behavior and verifying model learning quality. In addition, our experiments provide insight into the limitation of existing benchmark datasets and state-of-the-art models that may encourage future research on re-designing datasets, model architectures, and learning objectives.
Introduction
With the rising power of pre-trained language models, large-scale benchmarks serve as an important factor driving the future progress of NLP.
These benchmarks can provide a tool for analyzing the strengths and weaknesses of pre-trained language models.
In recent years, many benchmarks (Wang et al., 2019, 2020; Rajpurkar et al., 2018) have been proposed that offer a diverse set of evaluation objectives.
However, recent criticisms have been a broad-coverage diagnostic benchmark for linguistic-phenomena-driven evaluation.
Our benchmark includes both a dataset collection and an evaluation procedure for evaluating model performance and diagnosing linguistic skills captured by a model.
We evaluate models ﬁne-tuned on large NLI datasets through four types of diagnostic tests: zero-shot, inoculation, hypothesis-only, and cross-distribution.
made that these benchmarks fail to serve as effective measures of progress in machine learning (Raji et al., 2021). In particular, the task design does not formulate speciﬁc linguistic skills required for understanding.
They lack the effectiveness in helping researchers understand how certain systems or models work and how they fail. Although many stateof-the-art language models have shown impressive performance on these common benchmarks, their performance degrades considerably on adversarial or out-of-distribution samples (Bras et al., 2020). The performance drop shows that models may not be learning the required linguistic skills for solving the tasks of these benchmarks but exploit spurious dataset biases (Poliak et al., 2018b). Overall, the current benchmark format seems to be more like a contest than a tool that can explain how well a language model captures distinct linguistic skills essential to language understanding and reasoning.
In this paper, we propose a new form of benchmark that serves as a diagnostic evaluation tool for analyzing model linguistic skills. We present C URRICULUM benchmark: a framework for diagnosing neural language models through broad-coverage linguistic phenomena. Our benchmark includes (1) a large-scale collection of natural language inference (NLI) datasets covering 36 linguistic phenomena and (2) an evaluation procedure for probing and evaluating how well a language model captures reasoning skills for distinct types of linguistic phenomena.
Targeted linguistic phenomena in C URRICULUM range from fundamental properties like named entity and coreference to complex ones like commonsense and deductive reasoning.
With the C URRICULUM benchmark, we aim to investigate the following research questions:
• Q1 :
Do language models trained on benchmark datasets have the ability to reason over a wide range of linguistic phenomena?
• Q2 :
Are linguistic phenomena missing from the training data recoverable through inoculation (i.e., continuing to train models on a small sample of examples)
(Liu et al., 2019a)?
• Q3 :
Do language models learn a general reasoning skill of a phenomenon through inoculation?
To address the above questions, we empirically analyze NLI models trained on popular benchmark datasets through a pipeline of evaluations that in cludes : a zero-shot diagnostic test, inoculation retraining, hypothesis-only sanity check, and cross cross-distribution generalization tests
For Q1 , we observe that models trained on benchmark datasets, including adversarial data, do not have the reasoning ability for a large set of linguistic phenomena.
Our results show that training on more datasets can help the model learn more types of reasoning but does not help the model acquire complex reasoning skills such as deductive and commonsense reasoning.
Our benchmark exposes multiple knowledge gaps in large NLI models regarding diverse linguistic phenomena, particularly in the categories of commonsense and comprehension.
For Q2 , our analysis provides empirical evidence that exposes the lack of recoverable linguistic phenomena in benchmark datasets and models’ inability to learn certain linguistic phenomena.
We also show that, on some phenomena, models may rely heavily on spurious dataset bias existing in the hypothesis to reach high accuracy.
For Q3 ,
Our experiments show that models can adapt between distributions with different difﬁculties only on 22.2% of the phenomena such as Boolean, conditional, and comparative logic. In the majority (58.3 %) of the phenomena, models fail to generalize when the difﬁculties of the train and test distributions are different, for example, relational knowledge, puns, and contextual commonsense reasoning.
A model’s learning performance may not align with its generalization ability, suggesting the lack of a general reasoning skill.
Overall, our proposed benchmark systematically maps out a wide range of speciﬁc linguistic skills required for language understanding and inference.
We envision linguistic-phenomena-based evaluation to be an integral component of general linguistic intelligence.
We hope C URRICULUM can serve as a useful evaluation tool that can map out which aspects of the problem space remain challenging for existing systems and models.
Related Work NLU Benchmarks
In recent years, multiple large-scale benchmarks for evaluating models’ general language understanding performance have been proposed.
Similar to our benchmark’s task format, SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018) are the two common benchmarks for Natural Language Inference (NLI).
GLUE and SuperGLUE are the two most popular benchmarks that aim to provide a straightforward comparison between task-agnostic transfer learning techniques.
They cover various task formats, task domains, and training volumes, with datasets all collected from publicly available sources.
The construction of our benchmark is similar in that we also collect publicly available datasets from peer-reviewed papers.
Adversarial NLI (ANLI) is a new benchmark collected "via an iterative, adversarial human-and-model-in-the-loop procedure."
(Nie et al., 2020).
ANLI is shown to be a more difﬁcult challenge than previous benchmarks.
A part of our study focuses on the low-data generalization performance on ANLI.
Different from these benchmarks, our work aims to map out and evaluate speciﬁc linguistic skills a model needs for language understanding. Fine-grained NLU Evaluation
On top of largescale benchmarks, there are several works (Joshi et al., 2020; Tarunesh et al., 2021) contributing to the ﬁne-grained analysis of model performance.
They collect data examples from existing benchmarks by attaching taxonomic category labels to each data. Or, they build semi-synthetic data allowing analysis on 17 reasoning dimensions. Our data collection and categorization concepts are similar [Table]
to them. However, our work covers more linguistic phenomena that are difﬁcult but important such as commonsense and non-monotonic reasoning. Challenge Datasets for NLU Many challenge datasets have been developed to evaluate models on speciﬁc linguistic skills for understanding.
These datasets are in different formats such as NLI, Question Answering (QA), and Reading Comprehension (RC).
They target a large set of skills including monotonicity (Yanaka et al., 2019a), deductive logic (Liu et al., 2020), event semantics (Han et al., 2021), physical and social commonsense (Sap et al., 2019; Bisk et al., 2019), defeasible reasoning (Rudinger et al., 2020), and more.
Our work brings together a set of challenge datasets to build a benchmark covering a large set of speciﬁc linguistic skills.
We also merge different evaluation methods proposed by these works into a complete evaluation pipeline for our benchmark.
Probing Linguistic Knowledge
Several works have found evidence that pre-trained models’ representations encode knowledge about linguistic phenomena. Tenney et al. (2019) probe contextual representations from four pre-trained language models through the edge-probing method across tasks ranging from syntactic and semantic phenomena.
They ﬁnd that pre-trained models encode rich information on syntactic phenomena but only weakly encode information on semantic tasks compared to non-contextual baselines. Chen and Gao (2021)’s linguistic-information-probing framework extends the edge-probing study by focusing on different semantic phenomena that are important for logical inference in natural language. Their results show that pre-trained contextual embeddings encode more linguistic information on simple semantic phenomena than complex phenomena.
Our work is partly motivated by this line of work in which our evaluation is based on the fact that pre-trained models can capture speciﬁc linguistic skills from learning. Other work investigates if models use speciﬁc linguistic skills to solve a downstream task.
The DNC benchmark (Poliak et al., 2018a) provides a collection of datasets for analyzing if models use distinct linguistic phenomena to conduct natural language inference. Several tasks in our benchmark come directly from this collection.
However, our benchmark covers a wider range of linguistic phenomena from more categories than DNC.
In particular, our benchmark contains semantic phenomena and includes phenomena from funda- mental linguistic properties to complex reasoning types.
In addition, our benchmark includes a systematic evaluation methodology that allows a more in-depth analysis of model behavior.
The C URRICULUM Benchmark 3. A New Form of Benchmark
Recently, Raji et al.
(2021) suggested that good benchmark construction should focus on mapping out a speciﬁc set of linguistic skills required for language understanding. They recommend a future benchmark should provide interpretation on how systems work and how they fail on particular aspects of a problem space. Following this suggestion, we propose a new form of benchmark: linguistic-phenomena-driven evaluation. Our main objective is to reformulate the benchmark not simply to be a scoreboard for SOTA model contest but rather as a real measurement and standardization tool for (1) analyzing model performance, (2) exposing model and dataset weakness and (3) providing insights for future research directions.
The curriculum benchmark aims to map out a speciﬁc set of linguistic skills required for language understanding.
Our benchmark will serve as a diagnostic framework for linguistic-phenomena-driven probing and evaluation.
The targeted linguistic skills should range from fundamental linguistic properties to complex reasoning types.
Our linguistic phenomena selection is motivated by three benchmarks: GLUE Diagnostic, Rainbow, and DNC.
In addition, we include many more phenomena focusing on complex reasoning types such as deductive logic and analytical thinking.
Our ﬁnalized benchmark covers eight categories of linguistic phenomena.
Each linguistic phenomenon is considered one task, and one should train, evaluate, and analyze models on each phenomenon individually.
We brieﬂy describe the types of reasoning skill each category focus on in Table 1.
Appendix A and B shows a list of references and dataset details for the train and test datasets used for each linguistic phenomenon.
3. Dataset
We collect many challenge NLI or NLU datasets and ﬁlter them individually with the following criteria: (1) We focus on datasets that evaluate a speciﬁc or a set of speciﬁc linguistic phenomena.
(2) We focus on English monolingual datasets that are institutional and publicly available.
(3) We exclude tasks that require domain-speciﬁc knowledge that we would not expect a model to learn through pretraining, such as medical knowledge.
We ﬁnalize our selection with 36 datasets.
Figure 1 shows a detailed ontology of our selected linguistic phenomena and their abbreviations.
Our motivation for dataset selection is mainly based on the linguistic phenomena categories that we aim to cover which will range from a simple to complex setting.
3. Uniﬁed Task Format
We uniﬁed the task formats into a single linguistic task, Natural Language Inference (NLI). NLI is a task for Natural Language Understanding.
The task requires a model to classify the logical relationship between premise and a hypothesis. This logical relationship can either be Entailment (premise is true implies the hypothesis is absolutely true), Contradiction (premise is true implies the hypothesis is absolutely false), and Neutral (one cannot determine if the hypothesis is true or false based on the premise) (Dagan et al., 2013).
We select NLI as the universal task format because NLI often serves as a general evaluation method for models on different downstream tasks.
A model would need to handle nearly the full complexity of natural language understanding in order to solve the NLI task (Poliak et al., 2018b).
Our benchmark contains two types of NLI problems: (1) the 3-way NLI with
Entailment , Contradiction , and Neutral ; (2) the 2-way NLI with Entailed and Not-Entailed .
Each example has a premise and a hypothesis with 2-way or 3-way labels.
3. Automatic Recast
To convert non-NLI datasets into the NLI task format, we follow the dataset recast procedure (Poliak et al., 2018b): automatically convert from nonNLI datasets with minimum human intervention.
We design algorithmic ways to generate sentence pairs from the input text and convert the original labels into the NLI labels.
Question Answering (QA) and Reading Comprehension (RC) are the two major tasks we need to convert.
To convert datasets into NLI format, we follow the standard procedure (Khot et al., 2018).
In QA datasets, if choices are given as declarative statements, we consider them as hypotheses and the question context as the premise.
If choices are given as phrases answering the question, we concatenate the context and question to form a premise and consider the answers as hypotheses. Several datasets are tasks P I v
P
I v
P I v lex-ent 0. transit 0. hyper -0.
hypo -0.
ner 0.
vbn 0.
vbc -0.
syn-alt 0.
syn-var 0. bool 1.
cond 1.
cont 0.
comp 0.
negat 1.
quant 0.
monot -1.
kg-rel 0.
coref -0.
senti 0.
ctx-align -0.
puns 0.
sprl -0.
ent-tree 0.
analytic 0. temp 0.
spat 0.
counter 0.
defeas -0.
social -0.
physic -0. swag -0. cosmo -0. drop 0.
ester -0. logi -0. control -0. The lower I v is the more difﬁculty a dataset will be for the model.
P here are the abbreviations of linguistic phenomena listed in Tablewith free-response problems, and an answer can only be converted to an entailed hypothesis. To generate non-entailed hypotheses, we use several techniques during recasting. We show more details on our conversion techniques in Appendix C. As a sanity check on our resulting datasets, we empirically ﬁnd low performance on standard partialinput baselines (Poliak et al., 2018b), suggesting that our conversion yields data of high quality.
3. Dataset Difﬁculty
To enhance our benchmark to provide more information on each dataset for in-depth evaluation and analysis, we provide each phenomenon a difﬁculty level.
We use the predictive V -information (Ethayarajh et al., 2021) as a measurement for dataset difﬁculty.
The V -information can measure how much information an input variable X can provide about Y when constrained to functions V .
Intuitively, more usable infromation X can provide, the easier a dataset is for the functions V . Formally, let ∅ denote a null input that provides no information about Y and V as a predictive family, we can compute the V -information I v (X → Y) as follows: [Equation] where X , Y denote random variables with sample spaces X , Y .
According to Ethayarajh et al. (2021), ∅ can be an empty string here as f [ ∅ ] [Table] models the label entropy.
This framework can naturally adapt to the calculation of the point-wise
V -information ( PVI ) where we measure the difﬁculty of each data example.
Given a training dataset D train = { ( x i , y i ) } ni =1 , and the predictive family
V , the PVI of a data instance ( x , y ) ∈ D train is computed as:
[Equation]
where ∅ is an empty string (null input) and { f, f ′ } ⊆ V . f ′ and f are models ﬁne-tuned from D train and { ( ∅ , y i )
| ( x i , y i ) ∈ D train } respectively.
The V -information framework can also serve as a difﬁculty measurement for datasets and can be computed explicitly by averaging over PVI :
[Equation]
As Table 2 shows, the difﬁculty level ranges from negative to positive.
The higher the V -information is, the easier a dataset is for the model. Dataset Controlled Split
For our model evaluation pipeline, we are interested in verifying model’s ability to learn a generalizable reasoning skill on linguistic phenomena. In particular, we want to check if a model can generalize when its training and testing data distributions have different measurement of difﬁculty. Thus, we need to conduct controlled split on datasets based on the point-wise difﬁculty, i.e. the point-wise V -information of their data examples. We ﬁrst calculate the PVI( x → y ) for each phenomenon dataset, then we split each dataset into two portions: simple and hard, based on the calculation of each example’s PVI .
Evaluation Methodology
We deﬁne an evaluation process for the C URRICULUM benchmark that aims to bring different types of evaluation and diagnosing methods used by previous challenge NLI datasets.
Following Raji et al.
(2021)’s suggestion, we want our evaluation process to both to analyze the model output in detail and explore which aspects of the inference problem space remain challenging to current models.
Zero-shot Diagnostic Test
This test is motivated by the diagnostic test in GLUE.
We focus on providing ﬁne-grained analysis of zero-shot system performance on a broad range of linguistic phenomena. We follow the GLUE diagnostic dataset and use the
Matthews Correlation Coefﬁcient (MCC) (Jurman et al., 2012) as the evaluation metric.
MCC computes the correlation coefﬁcient of the predicted labels and the true labels. The correlation coefﬁcient value is between -1 and +1. A coefﬁcient of +1 indicates a perfect prediction. A 0 indicates average random prediction A -1 indicates the classiﬁer always miss-classiﬁes.
MCC is perfectly symmetric, so it can be used even if the dataset has classes with different sizes.
We use inoculation (Liu et al., 2019a) to further analyze model failures on target linguistic phenomena.
This method ﬁne-tunes the model on a down-sampled training section of a phenomenon dataset (inoculation).
One can interpret inoculation performance in two ways:
Good performance: the original training set of the model, prior to inoculation, did not sufﬁciently cover the target phenomenon, but it is recoverable through through additional training on a small sample of data.
Poor performance: there exists a model weakness to handle the target phenomenon.
Hypothesis-only Bias Analysis
We conduct analysis on hypothesis-only bias as (1) a sanity check for our converted datasets and also and (2) a veriﬁcation on whether model’s good performance is from leveraging artifacts in the hypotheses. We train a hypothesis-only baseline (Poliak et al., 2018b) for each phenomenon and compare their performance against the best models from the inoculation experiment.
We want to ensure that models’ improved performance after inoculation is due to their ability to reason about a hypothesis and the given context together.
If the hypothesis-only baseline shows good performance, we interpret this as a sign that the datasets contain artifact. If the baseline shows poor performance, it gives evidence that the model is not taking short-cuts.
Cross-Distribution Generalization
We conduct the cross-distribution generalization test (Rozen et al., 2019) to verify if the model learns a general reasoning skill from inoculation. The good inoculation performance does not ensure that the model’s learned skill is generalizable. The model can likely over-ﬁt the dataset distribution by adopting superﬁcial cues. We evaluate the model’s generalization ability by training and testing the model on distributions yielding different difﬁculty levels within the same dataset.
For example, we train the model on the simple part of the dataset (data with high V -information) and test it on the hard part (data with low V -information). 4. Experiment Setup For the zero-shot test, we test a model on each test set without additional ﬁne-tuning.
We select
NLI models with top performance on NLI benchmarks MNLI and ANLI.
We list these models in
Table 3.
We are interested in evaluating models with both the single-encoder and the text2text architecture.
All models are publicly available from
Huggingface (Wolf et al., 2019).
For inoculation, we ﬁne-tune models on training examples with a size ranging from 10 to 1000 examples per label.
For the cross-distribution generalization test, we ﬁrst create variant data distributions for train and test sets using the V -information-based dataset split method from Section 3.5.
We split each dataset into two portions (simple and hard) according to the point-wise V information.
Next, we either train and test the model on the same difﬁculty distribution or train it on one portion and test it on a different portion.
In the inoculation, hypothesis-only, and generalization experiments, we all use robertaanli-mix as our NLI model because its training set covers all the major NLI training datasets: SNLI, MNLI, FEVER (Thorne et al., 2018), and ANLI. We use accuracy as our evaluation metric for all these three experiments. For all the experiments excluding zero-shot test, we run several turns and select the best performance for analysis.
First, we report the results on zero-shot diagnostic evaluation for each baseline model. From Figure 2a, we observe that both single-encoder and text2text models trained on MultiNLI show a negative correlation in the majority of linguistic phe- l e x - e n t t r a n s i t
h
y p e r h y p o n e r v b n v b c s y n - a l t s y n - v a r b oo l c o n d c o n t c o m p n e g a t q u a n t m o n o t k g - r e l c o r e f s e n t
i c t x - a li g n p un s s p r l e n t - t r ee a n a l y t i c t e m p s p a t
c o un t e
r d e f e a s s o c i
a l p
h
y s i
c s w a g
c
o s m o d r o p e s t e r l o g
i c
o
n t r o l 1.
M a tt h e w s C o rr e l a t i o n C
o e ff i n
[Figure]
[Figure]
[Table]
l e x - e n t t r a n s i t h y p e r h y p o n e r v b n v b c
s y n - a l t b o o l c o n d c o n t c o m p n e g a t q u a n t m o n o t s y ll o k g - r e l c o r e f s e n t
i c t x - a li g n p un s s p r l e n t - t r ee a n a l y t i c t e m p s p a t c o un t e r d e f e a s s o c i a l p
h
y s i
c s w a g
c
o s m o l o g i d r o
[Table]
e s t e r
[Table]
o n [Table]
[Figure]
[Table]
[Figure]
nomena.
Meanwhile, anli-mix models (robertaanli-mix, xlnet-anli-mix) are positively correlated on most (77.8 %) of the phenomena and they show high correlation ( > 0 . 50 ) on 27.8 % of the phenomena.
On average, models trained on the large dataset mixture show better performance than models trained on MultiNLI alone, suggesting that training on more datasets help models capture more types of linguistic phenomena.
However, most of the phenomena captured by the anli-mix models are easier to learn (higher V information).
On harder phenomena, models did not beneﬁt from the training dataset mixture.
For instance, both the anli-mix models have a low correlation on deductive and analytical reasoning.
Overall, we ﬁnd that NLI datasets from common benchmarks lack examples of a diverse set of reasoning skills.
5. Inoculation
Based on Figure 2b, the model can reach high accuracy on about 64 % of the phenomena as the training examples accumulate.
Most of these phenomena have higher V information ( > 0 . 0 ) that should relatively be easier to learn.
We are surprised that for some hard phenomena ( ≤ 0 . 0 ) such as commonsense contextual reasoning (cosmo, - 0.67), the model’s performance improved after inoculation.
The improvement shows an gap in the original training data mixture.
On 25 % of the phenomena, the model’s performance did not improve simple hard simple hard    lex-ent simple hard    transit simple hard    hyper simple hard    hypo simple hard    ner simple hard    vbn simple hard    vbc simple hard    syn-alt simple
hard    syn-var simple hard    bool    cond    comp    cont    quant    negat    monot    coref    senti simple hard    kg-rel    puns    sprl    ctx-align    analytic    ent-tree    social    physic    swag simple hard    cosmo    temp    spat    counter    defeas    logi    ester    drop    control signiﬁcantly after inoculation, meaning that it fails to learn the reasoning skills for these phenomena.
Most of these phenomena are difﬁcult, with a low V information, such as monotonicity(mono) and deductive (logi) reasoning.
The accuracy is consistently low when training examples accumulate.
We also observe that model struggles to learn phenomena that require complex reasoning, such as phenomena from the comprehension category.
This trends show inherent weaknesses in the model or its training strategy that cause its failure to learn complex and hard phenomena.
Overall, results from this experiment, combined with the zero-shot evaluation, suggest that many linguistic phenomena are missing from different large-scale NLI datasets but are recoverable through additional training examples.
However, the model fails to learn the skills for hard and complex phenomena.
In summary, our diagnostic study through inoculation exposes a diverse set of dataset and model weaknesses.
5. Hypothesis-only Bias
To determine if models can leverage spurious artifacts in the hypotheses of each phenomenon, we compare full models to hypothesis-only baselines.
From Figure 2b, we observe that hypothesis-only baseline performs poorly on a majority of the phenomena.
This indicates that our benchmark generally requires the model to learn an inference process between contexts and hypotheses for good performance.
We observe that on 30.6% of the phenomena, the full-model can reach a high accuracy while the baseline has low accuracy, suggesting the model can learn the phenomenon without relying on hypothesis artifacts.
On 36 % of the phenomena, the model does not show a signiﬁcant performance gain compared to the baseline.
Most of these are complex reasoning phenomena like deductive and analytical reasoning.
The result validates that the model struggles more with complex linguistic phenomena.
On 33.3 % of the phenomena, both the full-model and the baseline achieve high accuracy showing the possibility that the model exploits artifacts from the hypothesis to reach high accuracy.
Also, note that the hypothesis-only baseline performs better for some tasks than the ﬁne-tuned model, which can be interpreted in two ways. When both the baseline and ﬁne-tuned model achieve high accuracy (vbc, syn-alt), higher accuracy on baseline indicates that the hypothesis-only bias is pretty strong in the dataset. When the intervention from the premise is removed (hypothesisonly input), the models can easily exploit the bias to achieve higher accuracy. In contrast, when both the baseline and ﬁne-tuned model achieve low accuracy (hypo, analytic, social, ester), higher accuracy on baseline indicates that the task is very difﬁcult for a model to master successfully.
Low baseline accuracy means that the dataset does not contain much bias, so a model must learn the correct reasoning to perform well. However, the ﬁnetuned model has even worse performance than the baseline, meaning that it fails to learn the skill required for these tasks. Our main ﬁnding here is that good performance on a linguistic phenomenon dataset does not mean the model captured the associated phenomena. The model can learn short-cuts through hypothesis-only bias and artifacts.
5. Generalization
As Figure 3 show, the model can adapt between different distributions only on 22.2 % of the phenomena.
The model achieves high accuracy consistently for all four categories in the generalization matrix suggesting the learned skills are generalizable.
On 58.3 % phenomena, models can not generalize between different difﬁculty distributions.
They show higher accuracy when trained and tested on the same distribution but low accuracy when the test distribution shifted.
For example, on relational knowledge reasoning (kg-rel), the model achieves 83% for simple → simple and 98 % for hard → hard.
Nevertheless, the performance drops to 53 % for hard → simple and 38 % for simple → hard.
Notice that model’s good performance on inoculation does not align with its generalization ability.
For example, the model reaches 90.9 % accuracy on kg-rel, but its generalization performance is poor.
This behavior highlights a model weakness: can over-ﬁt to a particular distribution but fail to learn a general reasoning skill for the target phenomenon.
We observe an interesting behavior that models struggle to generalize from hard to simple distribution on about 14 % of the phenomena while showing good generalization from simple to hard distribution.
We think the possible reason is that the hard distribution contains data with relatively low V information.
A low amount of usable information makes it hard for the model to learn the phenomena sufﬁciently for generalization.
Conclusion and Future Work
In this paper, we introduce a new form of benchmark that can serve as an effective tool for evaluating and analyzing model outcomes.
We propose a linguistic-phenomena-driven benchmark that aims to diagnose neural language models to discover types of linguistic skills that remain challenging to models.
We compiled a dataset collection covering 36 types of linguistic phenomena ranging from fundamental linguistic properties to complex reasoning skills.
In addition, we deﬁne an evaluation procedure that can provide an in-depth analysis of model and dataset weaknesses.
Using our benchmark, we comprehensively study how well language models capture speciﬁc linguistic skills essential for understanding.
Our major ﬁndings include:
• Models trained on benchmark NLI datasets fail to reason over a diverse set of linguistic phenomena.
• Good inoculation performance on some phenomena results from the model leveraging superﬁcial artifacts in the hypothesis.
• The model tends to over-ﬁt the dataset distribution without learning a general reasoning skill on a majority of phenomena.
Overall, our benchmark effectively evaluates a model on speciﬁc linguistic skills and exposes a list of model and training data weaknesses.
We hope that our benchmark and empirical ﬁndings can encourage the communicate to rethink dataset construction and model architecture design. In particular, we hope to encourage the the development of new datasets that cover richer types of linguistic phenomena and language models that can learn essential linguistic skills that are generalizable. For future work, we plan to add more datasets to cover more phenomena such as psycholinguistics (Laverghetta Jr. et al., 2021).
We envision our benchmark to be dynamic, meaning that a higher-quality and more difﬁcult dataset for a phenomenon should replace the current ones in the future.
For example, the StepGame benchmark (Shi et al., 2022) provides better data for spatial reasoning, which can replace the current spatial reasoning dataset. We also plan to explore new learning methods to help models overcome the weakness of learning non-generalizable skills, such as calibration through symbolic loss functions. Acknowledgment
We thank the anonymous reviewers for their thoughtful and constructive comments. We thank Kyle Richardson from AI2 for his insights and suggestions on improving our camera-ready version.
Thanks also to our advisors Laurence S. Moss and Michael Wollowski for their feedback on earlier drafts of this work. Special thanks to the Machine Learning for Language Group at NYU for their wonderful NLP toolkit, JIANT (Phang et al., 2020)
A Linguistic Phenomena in C URRICULUM Phenomena
Train Reference
Test Reference
Lexical Phenomena
Lexical Entailment
Schmitt and Schütze
Schmitt and Schütze 2021; Glockner et al.
Hypernymy
Richardson and Sabharwal
Richardson and Sabharwal 2020 Hyponymy
Richardson and Sabharwal
Richardson and Sabharwal 2020 Named Entity
Poliak et al.
2018a
Poliak et al. 2018a
Veridicality and Transitivity
Poliak et al. 2018a;
Yanaka et al.
Poliak et al. 2018a;
Yanaka et al.
Syntactic Phenomena
VerbNet
Poliak et al.
2018a
Poliak et al. 2018a
VerbCorner
Poliak et al.
2018a
Poliak et al.
2018a
Syntactic Variation
Dolan and Brockett Dolan and Brockett
Syntactic Alternations
Kann et al. Kann et al.
Semantic Phenomena
Coreference & Anaphora
Sakaguchi et al. 2019; Wang et al.
Sakaguchi et al. 2019;
Wang et al. Webster et al. Webster et al.
Sentiment
Poliak et al.
2018a
Poliak et al. 2018a
Relational Knowledge
Poliak et al.
2018a
Poliak et al.
2018a
Puns
Poliak et al.
2018a
Poliak et al. 2018a
Semantic Proto Label
White et al. White et al.
2017 Context Alignment
White et al. White et al.
2017; BIG-bench collaboration
Logical Phenomena
Boolean
Richardson et al. Richardson et al.
Conditional
Richardson et al. Richardson et al.
Comparative
Richardson et al. Richardson et al.
Counting
Richardson et al. Richardson et al.
2019 Quantiﬁer
Richardson et al. Richardson et al.
2019 Negation
Richardson et al. Richardson et al.
Monotonicity
Yanaka et al.
2019b
Yanaka et al. 2019a; Richardson et al.
Analytic Phenomena
Entailment Tree
Dalvi et al. Dalvi et al. Analytical
[Table]
B C URRICULUM Dataset Details in C URRICULUM
[Table]
C Data Recasting Details
Here we provide more details on the major techniques we used to convert Question Answering (QA) and
Reading Comprehension (RC) datasets into recast NLI datasets.
C. Entity Swapping
[Table]
C. Question/Answer Concatenation
[Table]
D Reproducibility
Implementation.
Our model training and testing pipeline is modiﬁed from the JIANT toolkit.
We mainly adapted several components on classes and functions involving task, dataset, reprocessing, tokenization, model version control, and evaluation metrics.
All our experiments are implemented with models publicly available from Huggingface Transformers (Wolf et al., 2020) 2 .
Hyper-parameters
We mainly follow the practice in (Nie et al., 2020).
For all the experiments excluding the zero-shot test in Section 5.1, we use a learning rate of 1 e − 5 with a batch size of 8.
We set the number of warmup updates to be 1000.
We set the epoch number to be 3 and 5.
We evaluate the model on D dev every 200 steps for the inoculation and generalization experiments, and 500 steps for the hypothesis-only experiment.
For the low-data generalization on ANLI, we evaluate on the full-test set according to the number of training examples listed in Figure ?? .
We use the AdamW (Loshchilov and Hutter, 2019) as our optimizer.
Infrastructure
All experiments are done with one single Geforce RTX 3090 (24GB).
A single inoculation or generalization job ﬁnishes within 0.5 hours on average.
A single hypothesis-only job ﬁnishes within 1-2 hours on average.
A single job on sequential training and low-data ﬁne-tuning ﬁnishes within approximately 1.5 hours on average.
Number of Parameters.
RoBERTa-large model contains 355 million parameters.
BART-large model contains 139 million parameters.
BART-Large model contains 406 million parameters.
XLNet-large model contains 340 million parameters.