Being Right for Whose Right Reasons? (2023)

Abstract: Explainability methods are used to benchmark the extent to which model predictions align with human rationales i.e., are ‘right for the right reasons’. Previous work has failed to acknowledge, however, that what counts as a rationale is sometimes subjective. This paper presents what we think is a first of its kind, a collection of human rationale annotations augmented with the annotators demographic information. We cover three datasets spanning sentiment analysis and common-sense reasoning, and six demographic groups (balanced across age and ethnicity). Such data enables us to ask both what demographics our predictions align with and whose reasoning patterns our models’ rationales align with. We find systematic inter-group annotator disagreement and show how 16 Transformer-based models align better with rationales provided by certain demographic groups: We find that models are biased towards aligning best with older and/or white annotators. We zoom in on the effects of model size and model distillation, finding –contrary to our expectations– negative correlations between model size and rationale agreement as well as no evidence that either model size or model distillation improves fairness.

## introduction transparency of nlp models is essential for enhancing protection of user rights and improving model performance. a common avenue for providing such insight into the workings of otherwise opaque models come from explainability methods @xcite @xcite @xcite . explanations for model decisions, also called rationales, are extracted to detect when models rely on spurious correlations, i.e., are right for the wrong reasons @xcite , or to analyze if they exhibit human-like inferential ## fairness and rationales fairness generally concerns the distribution of resources, often across society as a whole. in nlp, the main resource is system performance. others include computational resources, processing speed and user friendliness, but performance is king. ai fairness is an attempt to regulate the distribution of performance across subgroups, where these are defined by the product of legally protected attributes @xcite . nlp researchers have uniformly adopted american philosopher john rawls' definition of fairness @xcite @xcite @xcite , defining fairness as performance parity, except where it worsens the conditions of the least advantaged. several dozen metrics have been proposed, based on rawls' definition @xcite , some of which are argued to be in-consistent or based on mutually exclusive normative values @xcite . @xcite grouped these metrics into metrics based only on predicted outcome, e.g., statistical parity, and metrics based on both predicted and actual outcome, e.g., performance parity and accuracy equality. @xcite argue that metrics such as predictive parity and accuracy equality do not track fairness in case of infra-marginality, i.e., when the error distributions of two subgroups are different. for a better understanding of the consequences of inframarginality we refer to @xcite and @xcite . generally, there is some consensus that fairness in nlp is often best evaluated in terms of performance parity using standard performance metrics @xcite @xcite . we do the same and evaluate fairness in group-model rationale agreement quantifying performance differences (understanding performance as degree of rationale agreement) across end user demographics. in doing so, we are embodying group fairness values: that individuals should be treated equally regardless of their protected attributes, i.e., group belonging. fairness and explainability are often intertwined in the literature due to the assumption that transparency, through explainability methods, makes it possible to identify which models are right for the right reasons or, on the contrary, right by relying on spurious, potentially harmful, patterns @xcite . this study tightens the connection between fairness and explainability, investigating whether model rationales align better with those of some groups rather than others. if so, this would indicate that models can be more robust for some groups rather than others, even in the face of performance parity on dedicated evaluation data. that is: we ask whether models are equally right for the right reasons (with the promise of generalization) across demographic groups. ## data we augment a subset of data from three publicly available datasets spanning two tasks: dynasent @xcite and sst @xcite 3 , for sentiment classification and cos-e (talmor et al., 2019; rajani et al., 2019) for common-sense age below 36 or above 37 years old black/african american or latino/hispanic or white/caucasian recruitment (prolific filters) have not participated in previous studies annotation (qualtrics) informed consent gender? age? ethnicity? english speaking ability? language disorders? demographics survey instruc. positive negative no sentiment the (...) has a lot to offer. 10 exclusion survey age recruitment age survey ethnicity recruitment ethnicity attention check failed figure 2 : overview of the annotation collection process from annotator recruitment criteria, to the annotation itself, and finally annotator exclusion criteria. separately for each dataset, annotators are recruited via prolific using specific filters for age, ethnicity and participation status. recruits are directed to a qualtrics survey containing, in consecutive order, a consent form, a short demographics survey, instructions for the annotation task and then approx. 10 randomly selected instances of which annotators provide both labels and rationales for. after annotation, some annotators' responses are excluded from our analysis due to certain mismatches in responses. the annotation process is detailed further in section 3.1 and we show the instructions and task examples in appendix a. ## experiments we first conduct an analysis of group-group label agreement (i.e., comparing human annotator groups with each other, measuring human agreement on the sentiment and answer labels) and rationale agreement (measuring human agreement on rationale annotations) to characterize inter-group differences. we then move to group-model agreement (comparing the labels and rationales of our annotator groups to model predictions and model rationales) and ask: do models' explanations align better with certain demographic groups compared to others? in our analysis, we further focus on how rationale agreement and fairness behave depending on model size and model distillation. we probe 16 transformer-based models 8 . to ease readability, we will use abbreviations following their original naming when depicting models' performance 9 . we fine-tune the models individually on each dataset (see figure 3 ). sst-2 and cos-e simplified 10 are modeled as binary classification tasks; dynasent is modeled as a ternary (positive/negative/neutral) sentiment analysis task. we exclude all annotated instances from the training splits; for cos-e, we downsample the negative examples to balance both classes in the training split. after fine-tuning for 3 epochs, we select the checkpoint with the highest validation accuracy to run on our test (annotated) splits and apply two explainability methods to obtain input-based explanations, i.e., rationales, for the predictions made. 8 all pretrained models can be downloaded at huggingface.co/models. we measure label agreement with appropriate variants of f 1 (sst-2 binary-f 1 ; dynasent macro-f 1 ; cos-e mean of binary-f 1 towards the negative and the positive class). cos-e simplified represents a slightly different task (see footnote 10) from what the annotators were presented to solve (a multiclass question-answering task). to correctly measure label agreement, we evaluate whether a model predicts 'true' for the question-answer pair with the answer selected by the annotator. therefore, to avoid misleading f 1 scores if, for example, a model predominantly predict true, we report the mean of the f 1 towards each class. we explain below how we measure rationale agreement. ## results and discussion figure 3 shows group-model label agreement over our annotated data. 13 error bars show the variability between best and worst performing groups. cos-e exhibits the lowest variability, indicating less variability in label agreement between groups. when annotators disagree on the label of an instance, it is to be expected that their rationales will subsequently be different. therefore, to compare group-group ( § 5.1) and group-model ( § 5.2) rationales more fairly, we focus on the subset of instances where all groups are in agreement about 12 formally, these metrics account for plausibility (deyoung et al., 2019) of the models' rationales, i.e., the degree to which they are agreeable to humans, as well as the extent to which models are 'right for the right reasons' @xcite . since we are interested in comparing rationale alignment between groups and between groups and models, measuring plausability is our go-to. other research @xcite focus on properties like faithfulness, which reflect a model's true decision process, i.e., whether the provided rationale influenced the corresponding decision, generally measured through perturbation experiments. ## conclusion in this paper, we present a new collection of three existing datasets with demographics-augmented annotations, balanced across age and ethnicity. by having annotators choose the right label and marking supporting evidence for their choice, we find that what counts as a rationale differs depending on peoples' socio-demographic backgrounds. through a series of experiments with 16 popular model architectures and two families of explainability methods, we show that model rationales align better with older individuals, especially on sentiment classification. we look closer at model size and the influence of distilled pretraining: despite the fact that larger models perform better in general nlp tasks, we find negative correlations between model size and rationale agreement. furthermore, from the point of view of performance parity, we find no evidence that increasing model size improves fairness. likewise, distilled models do not seem to be more fair in terms of rationale agreement, however they do present overall higher scores. this work indicates the presence of undesired biases that do not necessarily surface in task performance. we believe this provides an important addendum to the fairness literature: even if models are fair in terms of predictive performance, they may still exhibit biases that can only be revealed by considering model rationales. if models are equally right, but only right for the right reasons in the eyes of some groups rather than others, they will likely be less robust for the latter groups.