Discovering Highly Influential Shortcut Reasoning: An Automated Template-Free Approach (2023)

Abstract: Shortcut reasoning is an irrational process of inference, which degrades the robustness of an NLP model. While a number of previous work has tackled the identification of shortcut reasoning, there are still two major limitations: (i) a method for quantifying the severity of the discovered shortcut reasoning is not provided; (ii) certain types of shortcut reasoning may be missed. To address these issues, we propose a novel method for identifying shortcut reasoning. The proposed method quantifies the severity of the shortcut reasoning by leveraging out-of-distribution data and does not make any assumptions about the type of tokens triggering the shortcut reasoning. Our experiments on Natural Language Inference and Sentiment Analysis demonstrate that our framework successfully discovers known and unknown shortcut reasoning in the previous work.

## introduction while transformer-based large language models have remarkably improved various nlp tasks, the issue of shortcut reasoning has been identified as a severe problem @xcite @xcite . shortcut reasoning usually refers to the irrational inference of a model, which is derived from spurious correlations in the training data @xcite @xcite . for example, sentiment analysis models could learn to classify any sentences containing the word spielberg into posi-tive, given a training dataset with many positive movie reviews containing spielberg (e.g., spielberg is a great director!). shortcut reasoning makes models brittle against out of distribution (ood) data (i.e., data from a different distribution from the training data) compared to independent and identically distributed (iid) data (i.e., data from an identical distribution as the training data) @xcite ). in the aforementioned example, the reasoning for movie reviews would not be valid for ood data (e.g., news articles) because the sentiment of news articles containing spielberg could be arbitrary. although many studies have explored the detection of spurious correlations or shortcut reasoning @xcite @xcite , several challenges persist. @xcite propose a state-of-the-art method for discovering shortcut reasoning, which implements an automated framework to discover shortcuts without predefining shortcut templates. still, their approach suffers from two major limitations. firstly, their framework lacks a method for quantifying the severity of the discovered shortcut reasoning on ood data. even if the shortcuts are identified, we do not have to necessarily be concerned about them as long as they have little negative impact on the model's robustness. secondly, their approach assumes that genuine tokens, useful tokens for predicting labels across different datasets (e.g., "good", "bad"), do not lead to shortcut reasoning. while this assumption seems reasonable, @xcite argue that such tokens are still prevalent among spurious correlations. this is because such tokens are indeed necessary to predict the label, but these tokens alone may not provide sufficient information to accurately predict labels. for example, a genuine token "good" in a sentence this movie is not good can be spurious, since "good" is a necessary but insufficient token for determining its sentiment label. therefore, genuine tokens can not be ignored when identifying shortcut reasoning. to address these problems, we propose a new method for discovering shortcut reasoning. our contributions can be summarized as follows: ## discovering shortcut reasoning fig. 1 shows the overall procedure of the proposed method. given (i) a target model f , (ii) iid data d iid , and (iii) ood data d ood as inputs, shortcut reasoning is extracted as an output. the procedure consists of the following three steps. step 1 extracts inference patterns, an abstract representation that characterizes the inferential process of a given model ( §2.1). to extract inference patterns, we use input reduction, an algorithm that automatically derives the inference patterns ( §2.2). step 2 estimates the generality of the extracted inference patterns. generality is a measure of the strength of an inference pattern, indicating its degree of regularity ( §2.3). step 3 identifies shortcut reasoning. we automatically determine whether an inference pattern exhibits shortcut reasoning without human intervention, comparing the effectiveness of inference patterns in d iid with that in d ood and leveraging the estimated generality as a proxy for the severity of the identified shortcut reasoning ( §2.4). ## related work numerous studies have tackled the problem of detecting spurious correlations or shortcut reasoning @xcite . one major limitation of earlier studies is that they predefine a specific format or structure for the shortcuts (e.g., a single token or a predefined set of tokens). this can hinder the discovery of new and unexplored shortcuts, which may be manifested in diverse forms. recently, @xcite address this problem by combining multiple interpretability techniques, such as influence function @xcite and feature attribution methods (e.g., integrated gradient). however, they rely on human assessment to identify shortcut reasoning, which can result in misjudgment between rational and irrational reasoning. besides, human evaluation is laborious and time-consuming. @xcite solve this issue by automatically identifying genuine tokens, important tokens that appear across different datasets, and spurious tokens, important tokens that appear only in an indomain dataset. still, as discussed in §1, there is a major limitation with their approach in that they fail to consider the influence of the identified shortcut reasoning on ood data. our work attempts to address this issue by estimating the generality of inference patterns. besides, our definition of shortcut reasoning aligns well with more practical scenarios. while they rely on a subjective definition of shortcut reasoning (i.e., whether reasoning is irrational from humans' point of view), our work targets shortcut reasoning that performs well on iid but underperforms on ood @xcite , namely the one that clearly hurts the robustness of nlp models by definition. ## conclusion we introduced a method to automatically discover shortcut reasoning. with minimal predefinition, our method successfully identified known and previously unknown examples of shortcut reasoning. for future research, we plan to adapt our method to large language models and other tasks such as machine reading comprehension. overall, we hope that our study provides a promising approach towards understanding the behavior of deep learning models and improving their trustworthiness.