T axi NLI : Taking a Ride up the NLU Hill (2020)

Abstract: Pre-trained Transformer-based neural architectures have consistently achieved state-of-the-art performance in the Natural Language Inference (NLI) task. Since NLI examples encompass a variety of linguistic, logical, and reasoning phenomena, it remains unclear as to which specific concepts are learnt by the trained systems and where they can achieve strong generalization. To investigate this question, we propose a taxonomic hierarchy of categories that are relevant for the NLI task. We introduce TaxiNLI, a new dataset, that has 10k examples from the MNLI dataset with these taxonomic labels. Through various experiments on TaxiNLI, we observe that whereas for certain taxonomic categories SOTA neural models have achieved near perfect accuracies—a large jump over the previous models—some categories still remain difficult. Our work adds to the growing body of literature that shows the gaps in the current NLI systems and datasets through a systematic presentation and analysis of reasoning categories.

## introduction the natural language inference (nli) task tests whether a hypothesis (h) in text contradicts with, is entailed by, or is neutral with respect to a given premise (p) text. this 3-way classification task, popularized by @xcite , which was in turn inspired by @xcite , now serves as a benchmark for evaluation of natural language understanding (nlu) capability of models; for example, nli datasets @xcite are included in all nlu benchmarks such as glue and superglue @xcite . these corpora, in turn, have been successfully used to train models such as bert @xcite † denotes equal contribution. * work was done while authors were at microsoft research india. herein lie the central questions of our work: 1) what is the distribution of various categories of reasoning tasks in the nli datasets? 2) which categories of tasks are rarely captured by current nli datasets (owing to the nature of the task and the non-expert annotators)? 3) which categories are well-understood by the sota models? and 4) are there categories where transformer-based architectures are consistently deficient? in order to answer these questions, we first discuss why performance-specific error analysis categories @xcite , and stress testing categories @xcite are inadequate. we then propose a taxonomy of the various reasoning tasks that are commonly covered by the current nli datasets (sec 2). next, we annotate 10,071 p-h pairs from the mnli dataset @xcite with the lowest level taxonomic categories, 18 in total (sec 3). then we conduct various experiments and careful error analysis of the sota models-bert and roberta, as well as other baselines, such as bag-of-words naïve bayes and esim, on their performance across these categories (sec 4). our analyses indicate that while these models perform well on some categories such as linguistic reasoning, the performance on many other categories, such as those that require world knowledge or temporal reasoning, are quite poor. we also look into the embeddings of the p-h pairs to understand which of these categorical distinctions are captured well in the learnt representations, and which get conflated (sec 5). inline with our previous finding, we observe strong correlation between the level of clustering within the representation of the examples from a category, and the performance of the models for that particular category. existing categorization efforts have centred around informing feature creation in the pre-transformer era, and model-specific error analysis in more recent times. previously, (lobue and yates, 2011) enumerated the type of commonsense knowledge required for nli. among recent error analysis efforts, the glue diagnostic dataset @xcite , inference types for adversarial nli @xcite , the new checklist (ribeiro et al., 2020) system and the stress tests @xcite are mentionworthy. as we attempted to group the categorizations in @xcite and @xcite into four high-level categories (lexical, syntactic, semantic, and pragmatic) foot_0 , we observe that there is a lack of consensus, nonuniformity and repetitiveness of these categories. for example, the tricky label in @xcite groups examples that involve "wordplay, linguistic strategies such as syntactic transformations, or inferring writer intentions from contexts"; thereby spanning aspects of syntax and pragmatics. similarly, reference and names requires both reasoning and knowledge. the glue diagnostic categories @xcite does not include interesting reasoning categories such as temporal, and spatial. the stress types proposed by @xcite are specific to mostly lexical and some semantic corner cases. this is expected, as these categorizations are analysis-oriented and often dependent on the performance of a set of models in question. here, we propose a taxonomic categorization that delineates a set of necessary uniform inferencing capabilities for the nli task. ## taxinli: dataset details we present taxinli, a dataset collected based on the principles and categorizations of the aforementioned taxonomy. we curate a subset of examples from multinli @xcite by sampling uniformly based on the entailment label and the domain. we then annotate this dataset with finegrained category labels. ## (re)evaluation of sota models we re-evaluate two transformer-based and two standard baseline machine learning models on taxinli, under the lens of the taxonomic categories. as baselines, we choose bert-base (devlin et al., 2019), and roberta-large @xcite as two state-of-the-art nli systems. for our experiments, we use the pre-trained bertbase and roberta models from huggingface's transformers implementation @xcite . as pre-transformer baselines, we use the bidirectional lstm-based enhanced sequential inference model (esim) @xcite . we also train a naive bayes (nb) model using bag-of-words features for the p-h pairs after removing stop words foot_2 . ## discussion visual analysis section 4 paints a thorough picture by analysing the fine-grained capabilities of sota nli systems at a behavioral foot_3 level. whereas we can say the systems are lacking in certain aspects despite their high overall performance, it naturally also raises questions at the understanding level: 1) is there any implicit knowledge acquired by the nli-finetuned systems about the kinds of reasoning required in the inference task? 2) if not, do the systems simply lack the understanding of what kind of reasoning is required per example, or despite understanding that, are unable to do the reasoning? 3) can we make an argument for future work and model architecture that can more consciously use this information? in light of recent probing task literature @xcite @xcite , we specifically investigate whether representations of examples cluster meaningfully into taxonomic categories relevant to the reasoning required for nli. we use the t-sne @xcite algorithm to visualize pooled contextualized representations of nli examples, under the lens of our taxonomy. for an nli example, we construct the embeddings at a transformer layer by maxpooling hidden states over all input token positions concatenated with the [cls] token (typically used for classification tasks) representation. the resulting visualizations (fig. 7 ) reveal definitive patterns of clustering by taxonomic categories. the earliest separation is observed for the lexical category, at layer 3 in bert (and layer 1 in roberta), much before any other categories are realized. at layers 6 in bert, and 19 in roberta, about the same time as clustering by nli label is seen, the connectives cluster is revealed. the deductions (see sec. 2), and syntactic categories are revealed in later layers (layer 11 and 12 in bert and layer 21 in roberta). the knowledge cate-gory is revealed more prominently in roberta at layer 21, while bert does not seem to show such a cluster. by the last few layers, separation into most categories becomes apparent. this means, along various layers of a nli finetuned language model, taxonomic information is implicitly captured. despite this, as discussed in the previous sections, sota models seem to be deficient in some of the categories-certain categories remain harder to perform inference on. in the latter layers, the separation along taxonomic categories also corresponds strongly with separation along nli labels. for instance, in fig. 7 (c) , the examples categorized as syntactic almost entirely lie in the entailment cloud, which matches our intuition based on the statstics in fig. 3 . the layer-wise separation of examples by taxonomy raises an interesting possibility to motivate model architectures that may attempt to use its discriminative power to identify such taxonomic categories, for specialized treatment to examples requiring certain reasoning capabilities. recasting: the under-representation of certain categories in the mnli dataset raises a need for more balanced data collection. a possible alternative is to build recast diagnostic datasets for each category, and create probing tasks. some datasets @xcite ) can be recast to the syntactic and logical categories respectively, as their data creation aligns with our category definitions. however, most categories lack such aligned synthetic data, and crowdsourced data would require manual annotation as above. this poses an avenue for future work. ## conclusion to bridge the gap between accuracy-led performance measurement and linguistic analysis of stateof-the-art nli systems, we propose a taxonomic categorization of necessary inferencing capabilities for the nli task, and a re-evaluation framework of systems on a re-annotated nli dataset using this categorization; which underscores the reasoning categories that current systems struggle with. we would like to emphasize that unlike the case with challenge and adversarial datasets, taxinli re-annotates samples from existing nli datasets which the sota models have been exposed to. therefore, a lower accuracy in certain taxonomic categories in this case cannot be simply explained away by the "lack of data" and "unnatural distribution" arguments. anli inference types glue diagnostic lexical standard inference lexical inference lexical entailment morphological negation factivity, redundancy syntactic tricky syntactic ambiguity, prepositional phrase alternations: active/passive, genitives/partitives, nominalization, datives semantic standard inference propositional structure, intersectivity quantifiers, restrictivity, quantification numerical, quantitative counting reference and names symmetry/collectivity, coreference, richer logical structures reference and names reasoning about facts named entities knowledge and commonsense pragmatics tricky ellipsis/implicits . we learn the parameters by fitting this bn to the observed data. in figure 9 , we see from the bayesian estimate again that, the improvements by bert in categories such as relational reasoning has been low. it also shows, that there is a sharp decrease in accuracy for examples requiring the use of taxonomic knowledge. however, roberta improves over nb and esim by large margins, albeit non-uniformly.