Statistically Profiling Biases in Natural Language Reasoning Datasets and Models (2023)

Abstract: Recent studies have shown that many natural language understanding and reasoning datasets contain statistical cues that can be exploited by NLP models, resulting in an overestimation of their capabilities. Existing methods, such as “hypothesis-only” tests and CheckList, are limited in identifying these cues and evaluating model weaknesses. We introduce ICQ (I-See-Cue), a lightweight, general statistical profiling framework that automatically identifies potential biases in multiple-choice NLU datasets without requiring additional test cases. ICQ assesses the extent to which models exploit these biases through black-box testing, addressing the limitations of current methods. In this work, we conduct a comprehensive evaluation of statistical biases in 10 popular NLU datasets and 4 models, confirming prior findings, revealing new insights, and offering an online demonstration system to encourage users to assess their own datasets and models. Furthermore, we present a case study on investigating ChatGPT’s bias, providing valuable recommendations for practical applications.

## introduction deep neural models have made remarkable strides in a broad spectrum of natural language understanding (nlu) tasks @xcite @xcite @xcite . these tasks often employ a multiple-choice framework, as illustrated in example 1. however, the inherent sensitivity of these models to minute variations calls for a robust and precise evaluation mechanism @xcite . example 1 natural language inference in the snli dataset, with the correct answer bolded. premise: a swimmer playing in the surf watches a low flying airplane headed inland. in tasks akin to example 1, humans typically rely on the logical relationship between the premise and hypothesis. contrarily, some nlp models might bypass this logical reasoning, focusing instead on the biases embedded within the dataset and, more specifically, within the hypotheses @xcite . these biases-such as sentiment or shallow n-grams-could provide misleading cues for correct predictions. we refer to these biases as "artificial spurious cues" when they pervade both the training and test datasets, maintaining a similar distribution over predictions. an example of such a cue is a model's disproportionate dependence on the word "someone" in example 1. these cues, when absent or altered, can significantly impair a model's performance, underlining the importance of identifying them to enhance model robustness in the future. to tackle the issue of cues, it's crucial to distinguish between cues embedded in the dataset and those learned by the model. conventional bias detection and mitigation tools, such as the ai fairness 360 toolkit @xcite , primarily target dataset biases, inadequately addressing those learned by models during training. while existing methods like "hypothesis-only" tests and checklist can uncover model vulnerabilities, they're not expressly designed to identify model-learned cues. "hypothesis-only" tests can highlight dataset issues where the hypothesis alone can provide a correct answer but fail to realistically depict the model's capabilities as they don't evaluate the model using the full data context used during both training and prediction. drawing from the tenets of black-box testing in software engineering, checklist scrutinizes model weaknesses without detailed knowledge of the model's internal architecture. it achieves this by delivering additional stress test cases premised on predefined linguistic features. however, checklist's dependence on meticulously designed templates limits its scope, and it also falls short of illuminating the knowledge the model has actually gleaned from the data. to address these limitations, we introduce icq ("i-see-cue"), a resilient statistical analysis framework foot_0 engineered to identify model-learned cues. diverging from traditional methods, icq identifies biases in multiple-choice nlu datasets without necessitating additional test cases. employing blackbox testing, icq assesses how models utilize these biases, delivering a comprehensive understanding of the bias in nlu tasks. we authenticate icq's efficacy by deploying it on various nlu datasets to probe potential cues learned by models during training. icq facilitates an in-depth understanding of how models like chat-gpt foot_1 learn potential cues, and it offers illustrative examples to guide the selection of suitable prompts, providing invaluable guidance for model optimization. in summary, this paper contributes the following: ## approach the icq framework, depicted in figure 2 , consists of three phases: data extraction, cue discovery, and model probing. in the data extraction phase, instances containing a specific linguistic feature f are extracted from the dataset. the cue discovery phase identifies potential cues among pre-defined features. finally, the model probing phase conducts two tests: the "accuracy test" and "distribution test". we will discuss these phases in more detail below. ## evaluation we first present the experimental setup, followed by results on cue discovery, model probing, and analysis. the entire framework is implemented in an online demo. ## case study recently, chatgpt, a large language model (llm) released by openai, has garnered significant interest from the nlp community. chatgpt, a gpt- ## related work our work is related to three research directions: spurious features analysis, bias calculation, and dataset filtering. spurious features analysis has been increasingly studied recently. much work @xcite @xcite has observed that some nlp models can surprisingly get good results on natural language understanding questions in mcq form without even looking at the stems of the questions. such tests are called "hypothesis-only" tests in some works. further, some research @xcite discovered that these models suffer from insensitivity to certain small but semantically significant alterations in the hypotheses, leading to speculations that the hypothesis-only performance is due to simple statistical correlations between words in the hypothesis and the labels. spurious features can be classified into lexicalized and unlexicalized @xcite : lexicalized features mainly contain indicators of n-gram tokens and cross-ngram tokens, while unlexicalized features involve word overlap, sentence length, and bleu score between the premise and the hypothesis. @xcite refined the lexicalized classification to negation, numerical reasoning, spelling error. @xcite refined the word overlap features to lexical overlap, subsequence, and constituent which also considers the syntactical structure overlap. @xcite provided unseen tokens an extra lexicalized feature. bias calculation is concerned with methods to quantify the severity of the cues. some work @xcite @xcite attempted to encode the cue feature implicitly by hypothesis-only training or by extracting features associated with a certain label from the embeddings. other methods compute the bias by statistical metrics. for example, @xcite used the probability of seeing a word conditioned on a specific label to rank the words by their biasness. lmi @xcite was also used to evaluate cues and re-weight in some models. however, these works did not give the reason to use these metrics, one way or the other. separately, @xcite gave a test data augmentation method, without assessing the degree of bias. dataset filtering is one way of achieving higher quality in datasets by reducing artifacts. in fact, datasets such as swag and reclor evaluated in this paper were produced using variants of this filter approach which iteratively perturb the data instances until a target model can no longer fit the resulting dataset well. some methods @xcite , instead of preprocessing the data by removing biases, leave out samples with biases in the middle of training according to the decision made between epoch to epoch. @xcite investigated model-based reduction of dataset cues and designed an algorithm using iterative training. any model can be used in this framework. although such an approach is more general and more efficient than human annotating, it heavily depends on the models. unfortunately, different models may catch different cues. thus, such methods may not be complete. ## conclusion our lightweight framework, icq, identifies biases and cues in multiple-choice nlu datasets, illuminating model behaviors from a statistical perspective. extensive experimentation on diverse tasks validates icq's efficiency in revealing dataset and model biases. using a case study on chatgpt, we explore its cues, offering practical guidance. icq advances our understanding and optimization of large language models, promoting the creation of robust, unbiased ai systems.