Machine Reading, Fast and Slow: When Do Models “Understand” Language? (2022)

Abstract: Two of the most fundamental issues in Natural Language Understanding (NLU) at present are: (a) how it can established whether deep learning-based models score highly on NLU benchmarks for the ”right” reasons; and (b) what those reasons would even be. We investigate the behavior of reading comprehension models with respect to two linguistic ”skills”: coreference resolution and comparison. We propose a definition for the reasoning steps expected from a system that would be ”reading slowly”, and compare that with the behavior of five models of the BERT family of various sizes, observed through saliency scores and counterfactual explanations. We find that for comparison (but not coreference) the systems based on larger encoders are more likely to rely on the ”right” information, but even they struggle with generalization, suggesting that they still learn specific lexical patterns rather than the general principles of comparison.

## introduction generally, human decisions may be based on deliberate, careful reasoning ('slow thinking') or quick heuristics ('fast thinking') @xcite . these two processes have parallels in the realm of reading comprehension (rc): a human reader would ideally fully process the text to answer questions, but in practice, we may deliberately skim rather than read to save effort. even capable students may be misled by superficial cues @xcite . the previous generations of nlp models have already achieved high performance on many rc benchmarks, but they were found to often 'read fast', i.e. rely on shallow patterns @xcite @xcite . fine-tuned transformer-based models @xcite still have similar shortcomings figure 1 : a sample question from the squad @xcite dataset. green tokens are the words that a reader relying on coreference resolution would take into account, and red tokens are the words that could be used to answer the question with entity type matching. @xcite @xcite in rc, as well as other tasks @xcite . consider the example in figure 1 . a human reader would ideally construct the coreference chain resolving the pronoun 'he' to 'leo strauss'. a possible heuristic-based solution is entity type matching @xcite : a model could observe that a 'where' question can only be answered by a 'location' and among two such entities @xcite ) the correct answer ('germany') is closer to 'born'. such heuristic reasoners will not generalize to unseen examples. thus a key challenge in building trustworthy and explainable rc systems is to make sure their decisions are based on valid reasoning steps. however, it is difficult to establish: @xcite what that reasoning should be; and (b) whether a blackbox system adheres to it. the present study proposes a framework for the analysis of rc models that includes: (a) defining the expected reasoning; (b) analysing model performance using explainability techniques. in particular, we contribute a case study for rc questions involving coreference resolution and comparison: we define the expected 'reasoning' for them ( §2) and use a combination of saliency-based and coun-terfactual explanations ( §3) to analyze rc systems based on bert and roberta encoders of various sizes ( §4). overall, we find that the larger models are more likely to rely on the 'right' information, but even they seem to learn specific lexical patterns rather than underlying linguistic phenomena. ## discussion and related work our work continues the emerging trend of research on being 'right for the right reasons' @xcite @xcite . we contribute stricter success criteria for behavioral tests of nlp models (def. 1), and, for the rc task, develop the methodology of: (a) defining what information the model should rely on for a given linguistic, logical, or world knowledge 'skill'; (b) systematically testing the behavior of rc models with interpretability techniques for whether they rely on that information. this is most closely related to the work on 'defining comprehension' by @xcite , though their testing is limited to probing the models with rc questions. another related study is the qed framework @xcite , annotating natural questions @xcite with 'explanations' of the expected reasoning process. their expected reasoning process also contains 3 steps, partly similar to ours: selecting a relevant sentence, referential equality, and deciding on whether this sentence entails the predicate in the question. however, the goals of qed are to: (a) predict both the answer and the explanation for a question; and (b) understand if explanations help qa models. such explanation annotations are unavailable for most datasets, and few qa models produce explanations. therefore, our approach of: (a) defining expected reasoning steps; and (b) using model interpretations to validate such steps applies to a broader class of models. this study is also related to the overall efforts to define what kinds of 'skills' rc models can be expected to exhibit @xcite @xcite . while these works focus on the high-level taxonomies of 'skills', we contribute practical definitions for two linguistic 'skills' (comparison and coreference resolution) which could be used for analyzing model performance. implicitly, research proposing rc resources that target various specific 'skills' (e.g. tempquestions @xcite for temporal order, mathqa @xcite for numerical reasoning, etc.) also contributes to this area, but they typically rely on broad linguistic definitions rather than on steps for machine reasoning. the saliency techniques we rely on have previously been used for extractive qa @xcite , but we are among the first @xcite to investigate their correlation with counterfactual explanations. for counterfactual perturbations, we also ensure that the perturbations are humaninterpretable and change the prediction, which is not the case for adding incomprehensible text @xcite , removing words from questions, shuffling the context @xcite , or replacing context tokens with random tokens @xcite . ## conclusion making progress towards trustworthy nlp models requires specific definitions for the behavior expected of these models in different situations. we propose a framework for rc model analysis that involves: (a) the definition of the expected 'reasoning' steps; (b) analysis of model behavior. we contribute such definitions for two linguistic 'skills' (comparison and coreference resolution), and use parallel explainability techniques to investigate whether rc models based on bert family encoders answer such questions correctly for the right reasons. we find that to be the case for comparison, but not for coreference. moreover, we find that, even for comparison, the models 'break' when encountering out-of-distribution counterfactual perturbations, suggesting that they memorize specific lexical patterns rather than learn more general reasoning 'skills'. as such, more research is needed on developing definitions and tests for specific 'skills' expected of nlu models, as well as on more faithful interpretability techniques.