To What Extent Do Natural Language Understanding Datasets Correlate to Logical Reasoning? A Method for Diagnosing Logical Reasoning. (2022)

Abstract: Reasoning and knowledge-related skills are considered as two fundamental skills for natural language understanding (NLU) tasks such as machine reading comprehension (MRC) and natural language inference (NLI). However, it is not clear to what extent an NLU task defined on a dataset correlates to a specific NLU skill. On the one hand, evaluating the correlation requires an understanding of the significance of the NLU skill in a dataset. Significance judges whether a dataset includes sufficient material to help the model master this skill. On the other hand, it is also necessary to evaluate the dependence of the task on the NLU skill. Dependence is a measure of how much the task defined on a dataset depends on the skill. In this paper, we propose a systematic method to diagnose the correlations between an NLU dataset and a specific skill, and then take a fundamental reasoning skill, logical reasoning, as an example for analysis. The method adopts a qualitative indicator to indicate the significance while adopting a quantitative indicator to measure the dependence. We perform diagnosis on 8 MRC datasets (including two types) and 3 NLI datasets and acquire intuitively reasonable results. We then perform the analysis to further understand the results and the proposed indicators. Based on the analysis, although the diagnostic method has some limitations, it is still an effective method to perform a basic diagnosis of the correlation between the dataset and logical reasoning skill, which also can be generalized to other NLU skills.

## introduction machine reading comprehension (mrc) and natural language inference (nli) are used to benchmark natural language understanding (nlu) capabilities. although a large number of nlu-related † these authors contributed equally. ‡ corresponding author. based on different deduction methods, reasoning skills can be divided into logical reasoning, coreferential reasoning, numerical reasoning, causal reasoning, etc @xcite . among all these reasoning skills, logical reasoning is a fundamental skill widely used to understand natural language @xcite . in this paper, we take logical reasoning as an example and construct a systematic method to analyze the correlations between nlu datasets and a specific reasoning skill. the correlations include two aspects: significance and dependence. 1) significance aims to judge whether a dataset includes sufficient (explicit or implicit) logical expressions; 2) dependence is used to measure how much the task on a dataset depends on the logical reasoning skill. to aid the diagnosis, a logical probe, consisting of a probe model and a probe dataset, is introduced. we select 8 mrc datasets (including two types) and 3 nli datasets to perform diagnosis. we create a qualitative and quantitative indicator to reflect the association between the dataset and logical reasoning after training the probe model on various datasets in various ways. the results show: 1) most nli datasets are relatively strongly correlated to logical reasoning. 2) as for the comprehensive mrc datasets (type1), the correlations are moderate which means that logical reasoning is not the only dominant reasoning skill in this type; 3) the correlations of different mrc datasets for specific anticipations (type2) vary remarkably according to their purposes of design. further cause analysis, which is consistent with the results, confirms the rationality of our method to a certain degree. in conclusion, this method offers a reasonable view of exploring the correlations between datasets and logical reasoning. our contributions are as follows: analysis for nlu. recently, more and more researches focus on what models have really learned in nlu tasks. in this field, some researches attempt to understand the knowledge @xcite and reasoning skills @xcite in language models, while others mainly focus on systematic evaluations of nlu models @xcite . these studies have provided foundations to further understand nlu. in addition, some researchers begin to pay attention to the analysis of nlu datasets @xcite . for example, sugawara et al. (2020) have provided an ablation-based method to understand the tasks defined on the nlu datasets. however, such researches are still rare and further investigations are required. ## methodology in this paper, we limit our investigation to a fundamental form of logical reasoning: conjunctive implications with negation (musen and @xcite which uses multiple conditions to derive the final conclusion and can be expressed as: our method adopts control variates to control what models learn. supported by a logical probe, we train models in the different manners on the diverse datasets to adjust the skills mastered by models. based on the method, we design a qualitative indicator and a quantitative indicator to indicate significance and dependence, respectively. next, we will firstly introduce the logical probe and then introduce two indicators and two corresponding processes. ## conclusions and future work in this work, we present a novel framework, which can diagnose the correlation between the nlu dataset and a specific skill and we probe a fundamental reasoning skill, logical reasoning, on 11 nlu datasets. our framework involves a logical probe to conduct diagnosis and defines a qualitative process and a quantitative process to calculate two indicators. from the results, we observe that 1) most nli datasets have a relatively strong correlation with logical reasoning. 2) the correlations between type 1 mrc datasets and logical reasoning are moderate because logical reasoning is not the only dominant skill in these datasets. 3) the dependences of type 2 mrc datasets are not always exactly consistent with their intended purpose. based on the analysis, although there are several limitations in our proposed method, this work is still a reasonable attempt to deeply understand the relationship between the dataset and a specific nlu skill. in future works, we will focus on: 1) exploring the solution to the limitations of the proposed method; 2) build associations for different datasets that require the same nlu capabilities.