Curriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding (2022)

Abstract: In the age of large transformer language models, linguistic evaluation play an important role in diagnosing models’ abilities and limitations on natural language understanding. However, current evaluation methods show some significant shortcomings. In particular, they do not provide insight into how well a language model captures distinct linguistic skills essential for language understanding and reasoning. Thus they fail to effectively map out the aspects of language understanding that remain challenging to existing models, which makes it hard to discover potential limitations in models and datasets. In this paper, we introduce Curriculum as a new format of NLI benchmark for evaluation of broad-coverage linguistic phenomena. Curriculum contains a collection of datasets that covers 36 types of major linguistic phenomena and an evaluation procedure for diagnosing how well a language model captures reasoning skills for distinct types of linguistic phenomena. We show that this linguistic-phenomena-driven benchmark can serve as an effective tool for diagnosing model behavior and verifying model learning quality. In addition, our experiments provide insight into the limitation of existing benchmark datasets and state-of-the-art models that may encourage future research on re-designing datasets, model architectures, and learning objectives.

## introduction with the rising power of pre-trained language models, large-scale benchmarks serve as an important factor driving the future progress of nlp. these benchmarks can provide a tool for analyzing the strengths and weaknesses of pre-trained language models. in recent years, many benchmarks @xcite @xcite have been proposed that offer a diverse set of evaluation objectives. however, recent criticisms have been our benchmark includes both a dataset collection and an evaluation procedure for evaluating model performance and diagnosing linguistic skills captured by a model. we evaluate models fine-tuned on large nli datasets through four types of diagnostic tests: zero-shot, inoculation, hypothesis-only, and cross-distribution. in this paper, we propose a new form of benchmark that serves as a diagnostic evaluation tool for analyzing model linguistic skills. we present cur-riculum benchmark: a framework for diagnosing neural language models through broad-coverage linguistic phenomena. our benchmark includes (1) a large-scale collection of natural language inference (nli) datasets covering 36 linguistic phenomena and (2) an evaluation procedure for probing and evaluating how well a language model captures reasoning skills for distinct types of linguistic phenomena. targeted linguistic phenomena in cur-riculum range from fundamental properties like named entity and coreference to complex ones like commonsense and deductive reasoning. with the curriculum benchmark, we aim to investigate the following research questions: to address the above questions, we empirically analyze nli models trained on popular benchmark datasets through a pipeline of evaluations that includes: a zero-shot diagnostic test, inoculation retraining, hypothesis-only sanity check, and cross cross-distribution generalization tests. for q1, we observe that models trained on benchmark datasets, including adversarial data, do not have the reasoning ability for a large set of linguistic phenomena. our results show that training on more datasets can help the model learn more types of reasoning but does not help the model acquire complex reasoning skills such as deductive and commonsense reasoning. our benchmark exposes multiple knowledge gaps in large nli models regarding diverse linguistic phenomena, particularly in the categories of commonsense and comprehension. for q2, our analysis provides empirical evidence that exposes the lack of recoverable linguistic phenomena in benchmark datasets and models' inability to learn certain linguistic phenomena. we also show that, on some phenomena, models may rely heavily on spurious dataset bias existing in the hypothesis to reach high accuracy. for q3, our experiments show that models can adapt between distributions with different difficulties only on 22.2% of the phenomena such as boolean, conditional, and comparative logic. in the majority (58.3 %) of the phenomena, models fail to gen-eralize when the difficulties of the train and test distributions are different, for example, relational knowledge, puns, and contextual commonsense reasoning. a model's learning performance may not align with its generalization ability, suggesting the lack of a general reasoning skill. overall, our proposed benchmark systematically maps out a wide range of specific linguistic skills required for language understanding and inference. we envision linguistic-phenomena-based evaluation to be an integral component of general linguistic intelligence. we hope curriculum can serve as a useful evaluation tool that can map out which aspects of the problem space remain challenging for existing systems and models. fine-grained nlu evaluation on top of largescale benchmarks, there are several works @xcite contributing to the fine-grained analysis of model performance. they collect data examples from existing benchmarks by attaching taxonomic category labels to each data. or, they build semi-synthetic data allowing analysis on 17 reasoning dimensions. our data collection and categorization concepts are similar to them. however, our work covers more linguistic phenomena that are difficult but important such as category description phenomena lexical testing a model's word-level reasoning lexical entailment (lex-ent), named entity (ner) skill on lexical semantic, direct, transitive, hypernymy (hyper), hyponymy (hypo) and compositional lexical relationships. veridicality & transitivity (transit) syntactic testing a model's reasoning skill on syntactic alternation (syn-alt), verbnet (vbn) syntactic structure and compositionality. syntactic variation (syn-var), verbcorner (vbc) semantic testing a model's reasoning skill on sentence-level reasoning sentiment (senti), relational knowledge (kg-rel), involving diverse semantic properties: entity relations, puns (puns), semantic proto label (sprl) context, events, subjectivity, and semantic proto roles. context alignment (ctx-align), coreference (coref) logical testing a model's reasoning skill on logical operations: boolean (bool), counting (count), conditional (cond) propositional structure, quantification, negation, comparative (comp), negation (negat) and monotonicity reasoning. monotonicity (monot), quantifier (quant) analytical testing a model's knowledge exploitation ability: drawing entailment tree (ent-tree) accurate conclusions based on domain-specific knowledge, analytical reasoning (analytic) symbolic knowledge, and interpretable reasoning steps. commonsense testing a model's reasoning skill on commonsense knowledge physical (physic), social (social), hellaswag (swag) independent of cultural and educational background. contextual commonsense reasoning (cosmo) comprehension testing a model's reasoning skill on complex reading event semantics (ester), discrete reasoning (drop) comprehension and inference, covering aspects of deductive reasoning (logi) semantic, context, logic, and numerical long contextual reasoning (control) special testing a model's everyday reasoning skill. including spatial reasoning (spat), temporal reasoning (temp) non-monotonic reasoning about valid but defeasible hypothesis defeasible reasoning (defeas) from hypothetical context and spatial-temporal reasoning. challenge datasets for nlu many challenge datasets have been developed to evaluate models on specific linguistic skills for understanding. these datasets are in different formats such as nli, question answering (qa), and reading comprehension (rc). they target a large set of skills including monotonicity @xcite , deductive logic @xcite , event semantics @xcite , physical and social commonsense @xcite , defeasible reasoning @xcite , and more. our work brings together a set of challenge datasets to build a benchmark covering a large set of specific linguistic skills. we also merge different evaluation methods proposed by these works into a complete evaluation pipeline for our benchmark. probing linguistic knowledge several works have found evidence that pre-trained models' representations encode knowledge about linguistic phenomena. @xcite probe contextual representations from four pre-trained language models through the edge-probing method across tasks ranging from syntactic and semantic phenomena. they find that pre-trained models encode rich in-formation on syntactic phenomena but only weakly encode information on semantic tasks compared to non-contextual baselines. chen and gao (2021)'s linguistic-information-probing framework extends the edge-probing study by focusing on different semantic phenomena that are important for logical inference in natural language. their results show that pre-trained contextual embeddings encode more linguistic information on simple semantic phenomena than complex phenomena. our work is partly motivated by this line of work in which our evaluation is based on the fact that pre-trained models can capture specific linguistic skills from learning. other work investigates if models use specific linguistic skills to solve a downstream task. the dnc benchmark @xcite provides a collection of datasets for analyzing if models use distinct linguistic phenomena to conduct natural language inference. several tasks in our benchmark come directly from this collection. however, our benchmark covers a wider range of linguistic phenomena from more categories than dnc. in particular, our benchmark contains semantic phenomena and includes phenomena from fundamental linguistic properties to complex reasoning types. in addition, our benchmark includes a sys-tematic evaluation methodology that allows a more in-depth analysis of model behavior. ## evaluation methodology we define an evaluation process for the curricu-lum benchmark that aims to bring different types of evaluation and diagnosing methods used by previous challenge nli datasets. following raji et al. zero-shot diagnostic test this test is motivated by the diagnostic test in glue. we focus on providing fine-grained analysis of zero-shot system performance on a broad range of linguistic phenomena. we follow the glue diagnostic dataset and use the matthews correlation coefficient (mcc) @xcite as the evaluation metric. mcc computes the correlation coefficient of the predicted labels and the true labels. the correlation coefficient value is between -1 and +1. a coefficient of +1 indicates a perfect prediction. a 0 indicates average random prediction a -1 indicates the classifier always miss-classifies. mcc is perfectly symmetric, so it can be used even if the dataset has classes with different sizes. ## conclusion and future work in this paper, we introduce a new form of benchmark that can serve as an effective tool for evaluating and analyzing model outcomes. we propose a linguistic-phenomena-driven benchmark that aims to diagnose neural language models to discover types of linguistic skills that remain challenging to models. we compiled a dataset collection covering 36 types of linguistic phenomena ranging from fundamental linguistic properties to complex reasoning skills. in addition, we define an evaluation procedure that can provide an in-depth analysis of model and dataset weaknesses. using our benchmark, we comprehensively study how well language models capture specific linguistic skills essential for understanding. our major findings include: • models trained on benchmark nli datasets fail to reason over a diverse set of linguistic phenomena.