Wrapper Boxes for Faithful Attribution of Model Predictions to Training Data (2024)

Abstract: Can we preserve the accuracy of neural models while also providing faithful explanations of model decisions to training data? We propose a “wrapper box” pipeline: training a neural model as usual and then using its learned feature representation in classic, interpretable models to perform prediction. Across seven language models of varying sizes, including four large language models (LLMs), two datasets at different scales, three classic models, and four evaluation metrics, we first show that the predictive performance of wrapper classic models is largely comparable to the original neural models. Because classic models are transparent, each model decision is determined by a known set of training examples that can be directly shown to users. Our pipeline thus preserves the predictive performance of neural language models while faithfully attributing classic model decisions to training data. Among other use cases, such attribution enables model decisions to be contested based on responsible training instances. Compared to prior work, our approach achieves higher coverage and correctness in identifying which training data to remove to change a model decision. To reproduce findings, our source code is online at: https://github.com/SamSoup/WrapperBox.

## introduction opaque predictive models are challenging to trust and reason about, prompting calls for greater transparency and interpretability in automated decisions @xcite . in critical sectors like law, health, and finance, interpretability may be essential to prevent catastrophic failures @xcite @xcite . furthermore, interpretability may be required for regulatory compliance @xcite . however, popular pre-trained language models @xcite @xcite are inscrutable, making it difficult to explain model decisions @xcite . in contrast, classic "white box" methods such as k-nearest neighbor (knn) and decision tree (dt) are inherently interpretable @xcite : each model decision is determined by a known set of training examples that can be directly shown to users. nevertheless, classic models tend to underperform today's neural models. recent work has pursued ways to blend the interpretability of classic models with the predictive performance of today's neural models @xcite @xcite @xcite . however, prior models face limitations in efficiency and scalability, requiring training from scratch or expensive computation and storage. in addition, research on interpretable nlp has largely focused on feature-style explanations, with far less work on example-based explanations @xcite . because people naturally reason by analogy @xcite @xcite , explaining predictions to specific training data is intuitively appealing. example-based explanations also connect to work on case-based reasoning @xcite by relating new problem instances to similar past ones, a problem-solving strategy people naturally use in decision-making @xcite . @xcite thus argues for developing modern case-based methods as one of the grand challenges in interpretable machine learning. in this work, we synthesize existing black-box and white-box methods toward building (training data) attributable-by-design models. specifically, we introduce the wrapper box pipeline to combine the accuracy of modern neural models with the faithful, example-based explanations of classic models. our approach effectively "wraps" a given neural model with one or more transparent classic models to maintain neural performance while im-figure 1 : three wrapper boxes illustrated for toxic language detection @xcite . red and blue dots denote harmful vs. benign speech. smaller dots represent examples, while larger dots represent clusters (e.g., dt leaf nodes). a neural model's penultimate layer provides the feature representation for the white wrapper boxes. our results show that classic models can achieve comparable performance to the underlying neural models while also providing intuitive, example-based explanations (described in section 4). note that wrapper boxes do not attempt to approximate the underlying neural model, i.e., wrapper box vs. neural model predictions can differ. our claim is that wrapper box predictions can be faithfully explained and that the predictive performance of wrapper boxes is largely comparable to the underlying neural models (see below). in contrast with prior techniques for generating example-based explanations ( our first evaluation assesses the predictive performance of wrapper boxes on two classification tasks with varying data scales: toxic speech detection (toxigen) and natural language inference (e-snli). we show that wrapper box predictive performance is largely comparable to neural baselines. while some statistically significant differences are observed (12%-27% across our datasets), this also includes cases in which the wrapper box actually performs significantly better than the base neural model. in the few cases where performance is worse, the value of interpretability may still justify use, bolstered by the vast majority of cases where no statistically significant difference is observed. we also evaluate the effectiveness of wrapper boxes using representations from modern large language models (llms). namely, we experiment with llama 2-7b instruct @xcite , llama 3-8b-instruct @xcite , mistral-7b instruct @xcite , and gemma-7b instruct @xcite . results show that wrapper boxes using zero-shot llm representations strongly outperform baseline llm performance across both tasks. next, we demonstrate the usefulness of examplebased explanations from wrapper boxes to attribute model decisions to specific training data. such attribution supports intuitive model explanations for end-users @xcite and enabling data-centric approaches for model developers, such as data cleaning @xcite . finally, we evaluate another use case: enabling model decisions to be contested based on the training data responsible for those decisions. specifically, we consider identifying which training data needs to be removed to change a model decision @xcite . this task offers a form of algorithmic recourse @xcite , which emphasizes providing actionable explanations to users unfavorably treated by automated systems. users are provided a foundation for contesting model decisions by attributing model decisions to specific training data. compared to @xcite , we show higher coverage and correctness in identifying which training data to remove while also generalizing beyond simple linear models and scaling to more modern neural networks. ## related work explainable models most work on interpretability focuses on post hoc methods that explain a pretrained model retroactively @xcite . this includes input attribution @xcite @xcite example-based explanations both input attribution and example-based explanations seek to explain model predictions in relation to observable data (i.e., inputs and training examples) rather than latent representations. this allows feature representations to be optimized for predictive performance without complicating explanations for end-users. unlike input attribution methods, example-based explanations (keane and kenny, 2019) aim to identify similar training inputs as analogical justification for model predictions. early work @xcite proposed treating the activation patterns of hidden nodes in a multi-layer perceptron as features for 1-nearest neighbor and decision trees. most prior work offers post hoc case-based reasoning via influence functions that show the training points most critical to a specific prediction as explanations @xcite @xcite . @xcite offer an inherently interpretable model, although derived concepts (non-terminal phrases) for explanation are at best partially faithful. prior work has consistently validated the significance, utility, and effectiveness of example-based explanations @xcite @xcite . benefits for users include increased model understanding (simulatability), complementary performance, and trust. @xcite @xcite @xcite @xcite @xcite . for developers, tying inference to specific training examples can uncover artifacts @xcite , errors (koh and liang, 2017), and gaps @xcite in training data, which can be addressed by label cleaning @xcite , data augmentation @xcite , and other data-centric techniques (anik and bunt, 2021). these studies show that example-based explanations are especially effective in the vision and text domains, given the intuitive nature of images and words @xcite . furthermore, in health and law, where decisions rely on historical precedents, case-based reasoning can assist users in developing intuitions for a model's inference procedure @xcite . of course, if training data is private, then example-based explanations are not possible @xcite . section e further examines the suitability of example-based explanations. while some studies have reported other forms of explanation being preferred over case-based explanations @xcite @xcite , none of the case-based systems evaluated provided faithful explanations. deep knn we build on @xcite 's dknn, which has been applied to text classification @xcite @xcite . our work generalizes dknn both conceptually and empirically to a broader suite of wrapper box models: decision trees (dts) and clustering-based classification alongside knn. this differs from prior approaches, which rely on traditionally learned linear components to forecast decisions @xcite @xcite . our framework is arguably the easiest to understand, implement, and reproduce. we do not modify the original model nor require additional computation beyond fitting white boxes and additional pass over training data to extract representations (which may be done offline). we thus avoid expensive operations required by prior work, such as approximating inverted hessian gradients @xcite or training a network from scratch @xcite . unlike prior work, the wrapper box framework is designed to be dataset, model, and task-agnostic. model auditing/algorithmic recourse model auditing and algorithmic recourse are commonly cited goals for fair and accountable ai systems and, thus, are closely related to explainability @xcite . model auditing @xcite @xcite involves systematically examining a model's behavior to identify problematic behaviors, potential biases, and errors in the training data. a natural next step is algorithmic recourse @xcite , which emphasizes providing actionable explanations and recommendations to users unfavorably treated by automated systems. by faithfully attributing model decisions to specific training data, wrapper boxes provide an avenue for contesting unjust decisions to support algorithmic recourse. ## example-based explanation tradeoffs we focus on interpretable predictive models that tie inference directly to specific training examples, enabling each prediction to be faithfully explained via those same training examples that determined the model's prediction. appendix d further discusses user perceptions of machine-retrieved examples. to better elucidate the design space for working with such models, this section illustrates possible tradeoffs between three key variables of interest: predictive performance, explanation faithfulness, and explanation simplicity. following jacovi and goldberg (2020), we conceptually define faithfulness as how accurately presented explanations reflect the actual reasoning process of the inference model. concretely, we evaluate the faithfulness of example-based explanations by completeness @xcite , where derived examples are faithful to the extent that all instances that support the test prediction are selected. the simplicity of examplebased explanations can be intuitively quantified as the number of presented instances @xcite . section 4 discusses how such tradeoffs can be operationalized in practice for our specific wrapper box models. ## wrapper boxes our wrapper box pipeline essentially "wraps" a given neural model with one or more white box classic models fitted on extracted neural representations for inference. note that the resultant classic models do not attempt to approximate the underlying neural model faithfully. while both classifiers leverage learned linearly separable neural representations, the underlying decision-making process differs. hence, derived example-based explanations faithfully explain the inference procedure of the wrapper boxes (interpretable classic models), not the original neural model. post hoc methods evaluate fidelity for the neural model they seek to explain @xcite since explanations can diverge from actual model behavior. in contrast, we leverage case-based classifiers where derived example-based explanations by construction must have been consulted during inference. loss in fidelity can only occur intentionally if fewer training examples are shown in the explanation to reduce information overload. ## evaluation: prediction performance we first compare the predictive performance of wrapper boxes vs. underlying neural models. because neural models forecast via linear layers, we expect wrapper boxes to benefit from this learned linear separability and perform comparably. we consider two tasks and datasets: toxigen @xcite consists of offensive and benign english statements generated by gpt-3 @xcite . we use the 9,900 human-labeled instances, ignoring other instances without gold labels. each instance is assigned toxicity labels on a 5-point scale. we binarize labels by mapping values 1-3 to non-toxic and 4-5 as toxic for binary classification. 90/10 train-test split, we section off a validation set, resulting in a 70/20/10 train-eval-test split. the dataset is highly skewed, with a 3:1 ratio of benign vs. toxic speech. we employ stratified sampling to maintain this ratio in each split. e-snli @xcite adds crowdsourced natural language explanations for the 569,033 english premise-hypothesis pairs originally annotated in snli @xcite . we follow the predefined training-eval-test splits. each split contains a balanced label distribution. appendix g.2 compares wrapper box explanations vs. those obtained via crowdsourcing. models we report on three language models: bart-large @xcite , deberta-large @xcite , and flan-t5-large (chung et al., 2022), based on checkpoints from huggingface @xcite . representations are extracted from the layer immediately preceding the linear classification head for bart-large and debertalarge models. for flan-t5, representations are extracted from the layer preceding the language generation head. implementation details for neural and white box models are discussed in appendix a. ## evaluation: training data attribution the ability to attribute model decisions to specific training data enables decisions to be contested on the basis of the training data responsible. to evaluate how well wrapper boxes support this use case, we adopt @xcite 's task formulation of finding a subset of training data s t that, if removed, would change the model decision for a given input. we use the same two datasets but only with de-berta representations (best performing model). baselines yang et al. (2023)'s two algorithms are limited to convex linear classifiers (e.g., logistic regression). we report these as baselines. appendix c.5 details our reproduction of their reported results, further validating the new results we correctness is the % of test inputs for which a s t was proposed and verified that their removal and retraining led to a prediction flip. median is the median set cardinality across only the verified subsets that lead to prediction flip. yang slow (algorithm 2) starts with all training data and seeks to iteratively reduce size of s t by approximating expected changes to model parameters θ upon removal. like yang fast, s t is only found if the expected output change exceeds τ . of note, yang et al. report on five binary datasets in their work: three balanced, and two highly skewed 9:1 ("hate" and "essays"). while results are strong on the balanced datasets, coverage is low on hate (67%) and very low on essays (11-12%). yang et al. remark upon hate's severe label skew, and to address it, select a post hoc τ = 0.25 for this dataset only (using τ = 0.5 for all others). oddly, they do not note or address the same skew in essays, which may lead the very low coverage reported. our approach algorithm 1 defines a greedy approach to derive s t from wrapper box explanations. for knn, c tr includes all neighbors of the input, ranked by proximity. for dt, c tr comprises all examples in the same leaf, ranked by proximity. for l-means, c tr consists of all points in the same cluster, ranked by proximity to the cluster centroid. post-filtering, we remove examples in chunks until a prediction flip is observed. s t is then refined (iteratively or in chunks, depending on ϕ) until no size reduction is possible. this encourages the derived s t to be minimal (but still leads to a prediction flip). see appendices c.1 and c.2 for further details and an optimized algorithm for knn (no training). ## conclusion we propose wrapper boxes to provides faithful, example-based explanations for classic case-based model predictions, attributing decisions to specific training data. our proposed pipeline is quite general and agnostic to the underlying neural architecture, training procedure, and input data. after training a neural model, the learned feature representation is input to white-box case-based reasoning models for prediction. because case-based models tie inference directly to specific training data, each prediction can be faithfully attributed to the training examples responsible. our first evaluation showed that white casebased models could deliver predictive performance largely comparable to baseline transformers, as seen across seven large pre-trained language models, two datasets of varying scale, three classic models, and four metrics. in addition, we discussed how such attribution enables automated decisions to be contested based on the training data responsible for those decisions. in comparison to prior work @xcite , our approach achieves both higher coverage and correctness in identifying which training data to remove to change a model decision. beyond contesting model decisions, other use cases include intuitively explaining decisions to end-users based on past examples or supporting data-centric ai operations for model developers (e.g., training data augmentation and cleaning).