Original: Use-Case-Grounded Simulations for Explanation Evaluation
Revised: Use-Case-Grounded Simulations for Explanation Evaluation

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: A growing body of research runs human subject evaluations to study whetherproviding users with explanations of machine learning models can help them withpractical real-world use cases.
Revised: A growing body of research runs human subject evaluations to study whetherproviding users with explanations of machine learning models can help them withpractical real-world use cases.

Original: However, running user studies is challenging andcostly, and consequently each study typically only evaluates a limited numberof different settings, e.g., studies often only evaluate a few arbitrarily selectedmodel explanation methods.
Revised: However, running user studies is challenging andcostly, and consequently each study typically only evaluates a limited numberof different settings, e.g., studies often only evaluate a few arbitrarily selectedexplanation methods.

Original: To address these challenges and aid user study design,we introduce Sim ulated Eval uations ( SimEvals ).
Revised: To address these challenges and aid user study design, weintroduce Use-Case-Grounded Sim ulated Eval uations ( SimEvals ).

Original: SimEvals involve trainingalgorithmic agents that take as input the information content (such as modelexplanations) that would be presented to the user, to predict answers to the usecase of interest.
Revised: SimEvals involve training algorithmic agents that take as input the information content (such asmodel explanations) that would be presented to each participant in a human subjectstudy, to predict answers to the use case of interest.

Original: The algorithmic agent’s test set accuracy provides a measure ofthe predictiveness of the information content for the downstream use case.
Revised: The algorithmic agent’s test setaccuracy provides a measure of the predictiveness of the information content forthe downstream use case.

Original: Werun a comprehensive evaluation on three real-world use cases (forward simulation,model debugging, and counterfactual reasoning) to demonstrate that SimEvals caneffectively identify which explanation methods will help humans for each use case.
Revised: We run a comprehensive evaluation on three real-worlduse cases (forward simulation, model debugging, and counterfactual reasoning) todemonstrate that SimEvals can effectively identify which explanation methodswill help humans for each use case.

Original: These results provide evidence that SimEvals can be used to efficiently screenan important set of user study design decisions, e.g. selecting which explanationsshould be presented to the user, before running a potentially costly user study.
Revised: These results provide evidence that SimEvalscan be used to efficiently screen an important set of user study design decisions,e.g. selecting which explanations should be presented to the user, before running apotentially costly user study.

Original:  Introduction
Revised:  Introduction

Original: The field of interpretable machine learning has proposed a large and diverse number of techniquesto explain model behavior (e.g., [35, 26, 36, 32, 27]).
Revised: The field of interpretable machine learning has proposed a large and diverse number of techniquesto explain model behavior (e.g., [37, 28, 38, 34, 29]).

Original: A growing body of work studies what modelexplanation techniques can help humans with practical real-world use cases [2, 18, 14, 16, 31, 33, 8,14].
Revised: A growing body of work studies which ofthese techniques can help humans with practical real-world use cases [2, 20, 16, 18, 33, 35, 8, 16].

Original: Because it is difficult to anticipate exactly which explanations may help humans for a particularuse case [9], the gold standard to evaluate explanations’ usefulness to humans is to run a humansubject study [10].
Revised: Because it is difficult to anticipate exactly which explanations may help humans with a particular usecase [9, 10], the gold standard to evaluate an explanation’s usefulness to humans is to run a humansubject study [11].

Original: Theseuser studies ask users a series of questions about the use case and providethe users with relevant information (such as model explanations) to answer each question.
Revised: These studies ask users a series of questions about the use case and provide theusers with relevant information (such as model explanations) to answer each question.

Original: Selecting the relevant information, which we call information content , to provide each user in thehuman subject study comprises an important set of design decisions [16, 5].
Revised: Selecting the relevant information, which we call information content , to provide each user in thehuman subject study comprises an important set of design decisions [18, 5].

Original: The researcher needs toconsider the choice of explanation methods, hyperparameters that are used to calculate the modelexplanations, and whether additional pieces of information should be provided to the user (such asthe model’s prediction on the data-point being explained).
Revised: The researcher needs toconsider the choice of explanation methods, hyperparameters that are used to calculate the modelexplanations, and whether additional pieces of information should be provided to the user (such asthe model’s prediction on the data-point being explained).

Original: 1 While a researcher may wish to runseveral user studies to evaluate many different types of information content, in practice this is ofteninfeasible: user studies are resource intensive and time-consuming, requiring the recruitment andcompensation of real users. Due to these constraints, most studies are far from comprehensive in
Revised: 1 While a researcher may wish to runseveral user studies to evaluate many different types of information content, in practice this is ofteninfeasible: user studies are resource intensive and time-consuming, requiring the recruitment and

Original: [Figure]
Revised: [Figure]

Original: : “Which explanation methodswould help
Revised: “Which explanation methods

Original: [Figure]
Revised: [Figure]

Original: this
Revised: compensation of real users.

Original: [Figure] case? [Figure] terms of the number of design decisions evaluated.
Revised: Due to these constraints, most studies are far from comprehensive interms of the number of design decisions evaluated.

Original: For example, most user studies typically onlyevaluate a few explanation methods, with default hyperparameter settings, that are often arbitrarilyselected based on their perceived popularity (Figure 1, Left).
Revised: For example, most user studies typically onlyevaluate a few explanation methods, with default hyperparameter settings, that are often arbitrarilyselected based on their perceived popularity (Figure 1, Left).

Original: For example, forward simulation is a canonical use case where users may be asked “Can you predictwhat the model’s prediction is for this individual?”
Revised: For example, forward simulation is a canonical use case where users are asked “Can you predictwhat the model’s prediction is for this individual?”

Original: To answer this question, the user isprovided with information content, which may include data-points (e.g., the individual’s features) andmodel explanations for those data-points (e.g., a SHAP [27] importance score for each feature).
Revised: To answer this question, the user isprovided with information content, which may include data-points (e.g., the individual’s features) andmodel explanations for those data-points (e.g., a SHAP [29] importance score for each feature).

Original: Theresearcher can then measure which set of information content (and consequently which explanationmethod ) enables users to achieve the best performance (e.g. the highest accuracy when answeringthe use case questions).
Revised: Theresearcher can then measure which set of information content (and consequently which explanationmethod ) enables users to achieve the best performance (e.g. the highest accuracy when answeringthe use case questions).

Original: Beyond forward simulation, other user studies have investigated use casesranging from model debugging to counterfactual reasoning [2, 18, 14, 16, 31].
Revised: Beyond forward simulation, other user studies have investigated use casesranging from model debugging to counterfactual reasoning [2, 20, 16, 18, 33].

Original: In this work, we introduce  Sim ulated Eval uations ( SimEvals ): a general framework for conducting use-case-grounded, algorithmic evaluations of the information content present in a user study(Figure 1, Right).
Revised: In this work, we introduce Use-Case-Grounded Sim ulated Eval uations ( SimEvals ): a general framework for conducting use-case-grounded, algorithmic evaluations of the information content present ina user study (Figure 1, Right).

Original: Each SimEval involves training  an agent, which is an ML model, in a use-casegrounded manner.
Revised: Each SimEval involves training and evaluating an algorithmic agent,which is an ML model.

Original: The agent learns to predict the ground truth label for the use case given the sameinformation that would be presented to a user in a human subject study.
Revised: The agent learns to predict the ground truth label for the use case given thesame information that would be presented to a user in a human subject study.

Original: The agent’s test set accuracyprovides away to measure  whether the information content provided (which critically includes themodel explanation) is predictive for the use case.
Revised: The agent’s test setaccuracy is a measure of whether the information content provided (which critically includes themodel explanation) is predictive for the use case.

Original: By comparing the agent’s test set accuracy for different types of information content, the researchercan efficiently identify promising and eliminate unhelpful information content types (e.g. explanationmethods as shown in Figure 1) to present to real humans in their final human subject study.
Revised: By comparing the agent’s test set accuracy for different types of information content, the researchercan efficiently identify promising and eliminate unhelpful information content types (e.g. explanationmethods as shown in Figure 1) to present to real humans in their final human subject study.

Original: Since
Revised: Since

Original: SimEvals are intended to inform, not replace, human subject evaluations, we note that our algorithmicagent does not incorporate  factorsother than predictive information, such as the effects of cognitivebiases [24, 33], that may affect human decision-making (see Appendix J).
Revised: SimEvals are intended to inform, not replace, human subject evaluations, we note that our algorithmicagent does not incorporate additional factors such as the effects of cognitive biases [26, 35], that mayaffect human decision-making (see Appendix J).

Original: In our experiments, weshow that using SimEvals to measure predictive information can indeed identify which informationcontent can help a human with a downstream use case.
Revised: In our experiments, we show that using SimEvalsto measure predictive information can indeed identify which information content can help a humanwith a downstream use case.

Original: In this work, we focus primarily on the design choice of selecting which explanation methods toevaluate.
Revised: In this work, we focus primarily on the design choice of selecting which explanation methods toevaluate.

Original: We conduct an evaluation on a diverse set of use cases and explanation methods: (1) Wecompare SimEvals with findings from prior user studies, and (2) we design our own user study on anew use case.
Revised: We conduct an evaluation on a diverse set of use cases and explanation methods: (1) Wecompare SimEvals with findings from prior user studies, and (2) we run our own user study motivatedby use cases studied in prior work.

Original: In our own user study, we use SimEvals to first pose hypotheses and then verify thosehypotheses by running an MTurk human subject study.

Original: In both cases, we findstrong correspondences between the SimEval agents’ and humans’ accuracy on the use case, indicating that SimEvals canprovide guidance on which explanations may be helpful  to humans.
Revised: In both cases, we find that SimEval effectively distinguishesbetween explanations that are ultimately helpful versus unhelpful to human subjects in the user study.

Original: Our  contributions in this work are as follows:
Revised: Our primary contributions in this work are as follows:

Original: Introducing SimEvals Framework.
Revised: Introducing SimEvals Framework.

Original: We are the first to propose a use-case-groundedevaluation
Revised: We are the first to propose an automated use-case-grounded

Original:  framework to measure the predictive information contained in the information content provided to auser in a human subject study.
Revised: evaluation framework to measure the predictive information contained in the information content provided to a user in a human subject study.

Original: We instantiate our framework on three diverse use cases to illustrateimportant differences in how SimEvals can be instantiated, showcasing how the SimEval frameworkcovers a wide variety of real-world use cases, explanation types, and user study design choices.2.
Revised: We instantiate our framework on three diverse usecases to illustrate important differences in how SimEvals can be instantiated, showcasing how theSimEval framework covers a wide variety of real-world use cases and explanation methods.

Original: Corroborating Prior User Studies.
Revised: Identifying Promising Explanations for Humans.

Original: We demonstrate that SimEvals can corroborate findings onpost-hoc and by-design explanations from two prior works with user studies. Specifically, using theagent test accuracy recovers the relative ordering of explanations found in the user study, providingevidence that SimEvals can be used as a measure to efficiently screen for promising explanationmethods that will help real users on the use case. Identifying Promising Explanations.
Revised: We demonstrate that SimEvals can distin-

Original: For a third use case, we first use SimEvals to hypothesizewhich explanation will be most useful for  the human and then run our own Mechanical Turk userstudy to validate these hypotheses. We find strong correspondence between  the MTurk and humanresults, demonstrating how SimEval can inform user study design.
Revised: guish between which explanations are promising versus unhelpful for humans on each of the threeuse cases we consider. Specifically, we find that when there is a significant gap between SimEvalperformance for two explanations, we observe a similar significant gap in human performance whenusing the same explanations.

Original:  Related Work
Revised:  Related Work

Original: Types of User Studies on Explanations: Many prior works run user studies to evaluate IML methods,covering a diverse set of goals [28].
Revised: Types of User Studies on Explanations: Many prior works run user studies to evaluate IML methods,covering a diverse set of goals [30].

Original: Specifically, IML user studies can differ in the independentvariable that is changed (e.g., the explanation method used [18, 14] or the visualization of theexplanation [22]) and the dependent variable(s) being measured (e.g., the user’s accuracy on a usecase [18, 14, 24, 26, 25], time taken to complete the use case [24], or trust in the ML model [7]).
Revised: Specifically, IML user studies can differ in the independentvariable that is changed (e.g., the explanation method used [20, 16] or the visualization of theexplanation [24]) and the dependent variable(s) being measured (e.g., the user’s performance on a usecase [20, 16, 26, 28, 27, 14], time taken to complete the use case [26], or trust in the ML model [7]).

Original: We focus on the set of user studies that measure how different explanation methods (independentvariable) affect how accurately users can complete a use case (dependent variable).
Revised: We focus on the set of user studies that measure how different explanation methods (independentvariable) affect how accurately users can complete a use case (dependent variable).

Original: We group existing user studies on explanations that measure the accuracy of users on a use caseinto two categories.
Revised: We group existing user studies on explanations that measure the accuracy of users on a use caseinto two categories.

Original: The first category studies the utility of explanations in use cases where thepredictiveness of the explanation method is already known to researchers.
Revised: The first category studies the utility of explanations in use cases where thepredictiveness of the explanation method is already known to researchers.

Original: For example, [24, 26, 25]ask users to answer simple comprehension questions about a decision set explanation, where theseexplanations contain the information necessary to complete the use case with perfect accuracy.
Revised: For example, [26, 28, 27]ask users to answer simple comprehension questions about a decision set explanation, where theseexplanations contain the information necessary to complete the use case with perfect accuracy.

Original: Thus,these studies measure humans’ ability to comprehend and interpret the explanations without error.
Revised: Thus,these studies measure humans’ ability to comprehend and interpret the explanations without error.

Original: Incontrast, the second category includes studies where the predictiveness of the explanation methodis not known to researchers in advance of their study [22, 14, 2].
Revised: Incontrast, the second category includes studies where the predictiveness of the explanation methodis not known to researchers in advance of their study [24, 16, 2].

Original: Our framework was developed toaid researchers in this second setting to quickly validate whether or not an explanation method ispredictive for their use case.
Revised: Our framework was developed toaid researchers in this second setting to quickly validate whether or not an explanation method ispredictive for their use case.

Original: Algorithmic frameworks for evaluating explanations: One line of evaluation frameworks focuseson automating the ability to measure desirable properties relating to the faithfulness of the explanationto the model [20, 1, 42], which does not account for any downstream use cases for the explanation.
Revised: Algorithmic frameworks for evaluating explanations: One line of evaluation frameworks focuseson automating the ability to measure desirable properties relating to the faithfulness of the explanationto the model [22, 1, 44], which does not account for any downstream use cases for the explanation.

Original: In contrast, our algorithmic evaluation framework allows researchers to customize and perform a usecase grounded evaluation of explanation methods.
Revised: In contrast, our algorithmic evaluation framework allows researchers to customize and perform a usecase grounded evaluation of explanation methods.

Original: We were able to identify one-off examples that use algorithmic agents to perform use case groundedevaluations in prior work [31, 43, 21, 34].
Revised: We were able to identify one-off examples that use algorithmic agents to perform use case groundedevaluations in prior work [33, 45, 23, 36].

Original: All of these examples propose evaluations designed aroundone specific use case.
Revised: All of these examples propose evaluations designed aroundone specific use case.

Original: Our framework is more general, and can be applied to a wide variety of usecases for explanations.
Revised: Our framework is more general, and can be applied to a wide variety of usecases for explanations.

Original: Our work is the first to propose a learning-based agent to identify whichexplanation methods may be predictive for a downstream use case.
Revised: Our work is the first to propose a learning-based agent to identify whichexplanation methods may be predictive for a downstream use case.

Original: We provide a more in-depthdiscussion of how these examples can be contextualized in our general framework in Appendix A.
Revised: We provide a more in-depthdiscussion of how these examples can be contextualized in our general framework in Appendix A.

Original:  General Framework
Revised:  General Framework

Original: We present a general framework to train SimEvals , which measure the predictiveness of severaldifferent types of information content for a downstream use case.
Revised: We present a general framework to train SimEvals , which measure the predictiveness of informationcontent choices for a downstream use case.

Original: Recall that when designing  the userstudy, the researcher must identify candidate choices of information content (e.g. which explanationmethods, hyperparameters, or other additional baseline information) to evaluate.
Revised: Recall that when designing a user study, the researchermust identify candidate choices of information content (e.g. which explanation methods, hyperparameters, or other additional baseline information) to evaluate.

Original: Our frameworktrains a SimEval agent for each choice of information 
Revised: Our framework trains a SimEval agentfor each choice of information content that the researcher intends to evaluate for a downstream usecase.

Original: [Figure]

Original: The researcher can then interpret the test set accuracy of each SimEval agent as a measure of thepredictiveness of that information content for the downstream use case.
Revised: The researcher can then interpret the test set accuracy of each SimEval agent as a measureof the predictiveness of that information content.

Original: Before describing how totrain each agent, we overview three use-case-specific componentsthat the researcher must instantiatethat are shared across all agents/types of information content:
Revised: Before describing how to train each agent, weoverview three use-case-specific components shared across all agents/types of information contentthat the researcher must instantiate:

Original: A base dataset ( D = { ( x , y ) } ), on which the explanation method’s utility will be evaluated.
Revised: A base dataset ( D = { ( x , y ) } ), on which the explanation method’s utility will be evaluated.

Original: The base model family, F , which is the family of prediction models.
Revised: The base model family, F , which is the family of prediction models.

Original: The trained base model
Revised: The trained base model

Original:  f ∈ F , which is trained on D , is used to generate explanations.
Revised: [Figure] f ∈ F , which is trained on D , is used to generate explanations.3.

Original: A function that defines a use case label, which is a ground truth label for the use case.
Revised: A function that defines a use case label, which is a ground truth label for the use case.

Original: This label corresponds to the correct answer for each question that a user must answer in the user study.
Revised: This labelcorresponds to the correct answer for each question that a user must answer in the user study.

Original: Now, we describe how to train and evaluate each agent model.
Revised: Now, we describe how to train and evaluate each agent model.

Original: 2 We introduce a three-step generalframework: (1) data generation, (2) agent training, and (3) agent evaluation.
Revised: 2 We introduce a three-step generalframework: (1) data generation, (2) agent training, and (3) agent evaluation.

Original: Since the three stepsof our framework closely parallel the steps in a standard ML workflow for model training andevaluation, canonical ML training techniques can also be used to train an algorithmic agent.
Revised: Since the three stepsof our framework closely parallel the steps in a standard ML workflow for model training andevaluation, canonical ML training techniques can also be used to train an algorithmic agent.

Original: Theprimary difference from the standard ML workflow is that in our framework, Step (1) is much moreinvolved to ensure that the data generated as input to the algorithmic agent reflects the informationcontent that would be presented to a human subject in a user study as shown in Figure 2.
Revised: Theprimary difference from the standard ML workflow is that in our framework, Step (1) is much moreinvolved to ensure that the data generated as input to the algorithmic agent reflects the informationcontent that would be presented to a human subject in a user study as shown in Figure 2.

Original: Step 1: Data Generation.
Revised: Step 1: Data Generation.

Original: The goal of this step is to use the researcher’s information content ofinterest and use-case-specific components to generate a dataset for training and evaluating the agent.
Revised: The goal of this step is to use the researcher’s choice of informationcontent and use-case-specific components to generate a dataset to train and evaluate the agent.

Original: The dataset consists of observation and use case label pairs, where the information content defineswhat information is included in each observation .
Revised: Thedataset consists of observation and use case label pairs, where the information content defines whatinformation is included in each observation .

Original: We show a concrete example of data generation forthe forward simulation use case in Figure 3.
Revised: We show a concrete example of data generation forthe forward simulation use case in Figure 3.

Original: The specific construction of each observation can varydepending on several factors to accommodate a diverse range of use cases and user study designs:
Revised: The specific construction of each observation can varydepending on several factors to accommodate a diverse range of use cases and user study designs:( x 1 , E ( x 1 , f )), f ( x 1 ) ... ( x N , E ( x N , f )), f ( x N ) ( x 2 , E ( x 2 , f )), f ( x 2 ) x i , y i From : UCI [Figure]

Revised: f

Original: Local vs. global.
Revised: [Figure] x [Figure] Model:

Original:  The SimEval framework can be used to evaluate both local and global explanations:
Revised: Explanation: [Figure] Local vs. global. The SimEval framework can be used to evaluate both local and global explanations:

Original: A local explanation method E ( f, x ) takes as input prediction model f and a single data-point x ∈ X .
Revised: A local explanation method E ( f, x ) takes as input prediction model f and a single data-point x ∈ X .

Original: A global explanation method E ( f, D ) takes as  input model f and base dataset D .
Revised: A global explanation method E ( f, D ) takes as the input model f and a dataset D .

Original: The specificconstruction of each observation (i.e., whether the observation contains a single explanation vs. anaggregation of explanations) depends on whether the use case is a global problem (using explanationsto answer a question about the entire model) or a local problem (using an explanation to answer aquestion about the model’s behavior at a particular point).
Revised: The specific construction of each observation (i.e., whether the observation contains a single explanation vs. anaggregation of explanations) depends on whether the use case is a global problem (using explanationsto answer a question about the entire model) or a local problem (using an explanation to answer aquestion about the model’s behavior at a particular point).

Original: For global problems, the use case label isthe same for a given model as opposed to for local problems, the use case label varies depending onthe data-point of interest.
Revised: For global problems, the use case label isthe same for a given model as opposed to for local problems, the use case label varies depending onthe data-point of interest.

Original: A longer discussion can be found in the Appendix E.
Revised: A longer discussion can be found in the Appendix E.

Original: Multiple datasets D i and models f i .
Revised: Multiple datasets D i and models f i .

Original: Instead of building each observation using a single base datasetand base model, for some use cases we sample each observation i from a different base dataset D i(and use it to train a new base model f i ).We do so because (a) global problems require multiple ( x 1 , E ( x 1 , f )), f ( x 1 ) ... ( x N , E ( x N , f )), f ( x N ) ( x 2 , E ( x 2 , f )), f ( x 2 ) x i , y i
Revised: Instead of building each observation using a single base datasetand base model, for some use cases we sample each observation i from a different base dataset D i(and use it to train a new base model f i ).

Original: From : UCI Adult f

Original: From : Random Forest

Original: E ( x i , f ) LIME

Original: Forward simulate f ’s prediction on x i

Original: Dataset: Model: Explanation: Use case: models to have a diverse set of use case labels and (b) it may be undesirable for the agent to overfit toa single base dataset.
Revised: We do so because (a) global problems require multiplemodels to have a diverse set of use case labels and (b) it may be undesirable for the agent to overfit toa single base dataset.

Original: Human Factors.
Revised: Human Factors.

Original: When selecting explanation methods for evaluation, it is important to considerfactors that influence the way humans reason about explanations.
Revised: When selecting explanation methods for evaluation, it is important to consider factorsthat influence the way humans reason about explanations.

Original: To avoid potentially misleadingexplanations, we advocate for researchers to select explanations that also satisfy desirable proxymetrics such as faithfulness and stability.
Revised: We advocate for researchers to selectexplanations that also satisfy desirable proxy metrics such as faithfulness and stability.

Original: Additionally, it is important to select explanations that aresuitable for human use.
Revised: Additionally,it is important to select explanations that are suitable for human use.

Original: We discuss these considerations in the context of our experiments and userstudy in more detail in Appendix J.
Revised: We discuss these considerationsin the context of our experiments and user study in more detail in Appendix J.

Original: Step 2: Agent training.
Revised: Step 2: Agent training.

Original: After generating a dataset, the researcher trains an algorithmic agent (e.g.,a machine learning model) to predict the use case label for a given observation.
Revised: After generating a dataset, the researcher trains an algorithmic agent (e.g.,a machine learning model) to predict the use case label for a given observation.

Original: While this step canbe viewed as equivalent to training an ML model (i.e., requiring standard hyperparameter search andtuning), this step is also analogous to “training” phases in human subject studies in which humansare provided with labeled examples to convey relevant knowledge for the use case (e.g., presentedexample explanations and expected conclusions).
Revised: While this step canbe viewed as equivalent to training an ML model (i.e., requiring standard hyperparameter search andtuning), this step is also analogous to “training” phases in human subject studies in which humansare provided with labeled examples to convey relevant knowledge for the use case (e.g., presentedexample explanations and expected conclusions).

Original: One important difference is that while humansubjects may have varying degrees of prior knowledge about the use case that may inform how theycomplete it, the algorithmic agent does not have  knowledge a priori. Thus the training process allowsthe agent to acquire the relevant domain knowledge from labeled examples.
Revised: The training process allows the algorithmic agentto implicitly learn relevant domain knowledge about the use case from labeled examples, even whenthe researcher does not have any knowledge a priori of how to predict the use case labels.

Original: In terms of model architecture selection, the researcher should select an agent model that is able toencode and learn meaningful representations of the observations that are input to the agent (whichinclude the data-point and explanations).
Revised: In terms of model architecture selection, the researcher should select an agent model that is able toencode and learn meaningful representations of the observations that are input to the agent (whichinclude the data-point and explanations).

Original: The researcher needs to define the encoding for the choiceof explanation.
Revised: For example, the researcher needs to define the encoding forthe choice of explanation.

Original: There are some standard ways to encode certain types of explanations (e.g., afeature attribution explanation for tabular data can be encoded as a vector of length d
Revised: There are some standard ways to encode certain types of explanations (e.g.,a feature attribution explanation for tabular data can be encoded as a vector of length d

Original: if x ∈ R d andconcatenated to the data-point to create an observation vector of length 2 d ).
Revised: if x ∈ R d andconcatenated to the data-point to create an observation vector of length 2 d ).

Original: However, in general,the researcher can leverage their domain knowledge in selecting the model: for example, a CNNarchitecture is appropriate for a base dataset containing images.
Revised: However, in general,the researcher can leverage their domain knowledge in selecting the model: for example, a CNNarchitecture is appropriate for a base dataset containing images.

Original: We discuss in Appendix E how weencode explanation methods used in our experimental evaluations.
Revised: We discuss in Appendix E how weencode the explanations evaluated in our experiments.

Original: Step 3: Agent evaluation.
Revised: Step 3: Agent evaluation.

Original: After training the agent, the researcher evaluates the agent’s abilityto make predictions on unseen test examples.
Revised: After training the agent, the researcher evaluates the agent’s ability tomake predictions on unseen test examples.

Original: This phase is analogous to the evaluation portionof typical user studies, where humans are tested on new examples.
Revised: This phase is analogous to the evaluation portion of typicaluser studies, where humans are tested on new examples.

Original: The agent’s accuracy on theseexamples is a measure of the predictiveness of the explanation for the use case: a relative higheraccuracy is evidence that a particular type of explanation is more useful.
Revised: The agent’s accuracy on these examples is ameasure of the predictiveness of the information content for the use case: a relative higher accuracyis evidence that a particular type of information content is more useful.

Original: We note that the agent’sperformance (i.e. test set accuracy)with a given explanation method is not intended to be interpretedas an anticipated human subject’s performance.
Revised: We note that the agent’sperformance (i.e. test set accuracy) is not intended to be interpreted as the exact, anticipated humansubject’s performance when given the same set of information content as the algorithmic agent.

Original: We discuss several factors beyond the predictiveinformation in the explanation that may affect human performance in Appendix J. In our experiments,we show that the relative ordering of explanation method performance is consistent for SimEvalsand human subject evaluations.
Revised: Wediscuss several factors beyond the predictive information in the explanation that may affect humanperformance in Appendix J. As such, researchers should focus on statistically significant differencesin agent performance when interpreting their SimEvals, as smaller differences between informationcontent may not generalize to human performance.

Original:  Instantiating SimEvals
Revised:  Instantiating SimEvals

Original: We instantiate our general framework to study the utility of popular post-hoc and by-design explanations (e.g., LIME, SHAP, Anchors, GAM) for three different use cases, spanning a diverse set of problems (forward simulation [14], counterfactual reasoning [23], and data bugs [18]).
Revised: We instantiate our general framework to study the utility of popular post-hoc and by-design explanations (e.g., LIME, SHAP, Anchors, GAM) for three different use cases, spanning a diverse set ofproblems (forward simulation [16], counterfactual reasoning [25], and data bugs [20]).

Original: The first twouse cases are local problems that focus on base model behavior at a specific point, while the third is aglobal problem that asks questions about the base model behavior more broadly.
Revised: The first twouse cases are local problems that focus on base model behavior at a specific point, while the third is aglobal problem that asks questions about the base model behavior more broadly.

Original: The human subjectevaluations that studied these use cases primarily focused on the design choice of which explanationmethod best helps the user answer the use case questions.
Revised: The human subjectevaluations that studied these use cases primarily focused on studying which explanation method besthelps users answer the use case questions.

Original: In the following sections, we focus on  key portions of the SimEval framework that must be carefullydefined based on the specifics of each individual use case: the data generation process in Step (1)and the agent model architecture in Step (2).
Revised: Next, we focus on a key portion of the SimEval framework that must be carefully defined, discussingthe data generation process in Step (1) for each use case.

Original: Further discussion including detailed data generationalgorithms and agent architectures  for each use case can be found in Appendices B and E.
Revised: Further details on data generation algorithmsand agent architectures used for each use case can be found in Appendices B and E.

Original: Forward simulation measures whether a user correctly simulates  the ML model’s prediction on agiven data-point [33, 8, 14].
Revised: Forward simulation measures whether a user correctly simulates (i.e., predicts) the ML model’sprediction on a given data-point [35, 8, 16].

Original: Our particular set-up is motivated by Hase and Bansal [14], where theauthors test whether providing users with explanations helps users better predict a given model’sbehavior when compared to a control setting in which users are not given explanations.
Revised: Our particular set-up is motivated by Hase and Bansal [16],where the authors test whether providing users with explanations helps users better predict a basemodel’s behavior when compared to a control setting in which users are not given explanations.

Original: Here, eachobservation given as input to the  agent consists of a single data-point and explanation ( x i , E ( x i , f )) ,where x i is sampled from the base dataset D .
Revised: Here,each observation given as input to the SimEval agent consists of a single data-point and explanation( x i , E ( x i , f )) , where x i is sampled from the base dataset D .

Original: Following the same procedure as Hase and Bansal, wesplit D into train, test, and validation sets.
Revised: Following the same procedure as Haseand Bansal, we split D into train, test, and validation sets.

Original: The agent is trained on a dataset of observations from thetraining set and is then evaluated on observations from the unseen validation set.
Revised: The agent is trained on a dataset ofobservations from the training set and is then evaluated on observations from the unseen validationset.

Original: The use case labelu i = f
Revised: The use case label u i = f

Original: ( x i ) is the model f ’s output.
Revised: ( x i ) is the model f ’s output.

Original: Counterfactual reasoning is a concept from philosophy used to describe “what if” conditionalsthat are contrary to fact [41].
Revised: Counterfactual reasoning is a concept from philosophy used to describe “what if” conditionalsthat are contrary to fact [43].

Original: Counterfactual reasoning can enable individuals to reason abouthow to achieve a desired outcome from a prediction model.
Revised: Counterfactual reasoning can enable individuals to reason abouthow to achieve a desired outcome from a prediction model.

Original: We study a specific instantiation ofcounterfactual reasoning from Kumar et al.
Revised: We study a specific instantiation ofcounterfactual reasoning from Kumar et al.

Original: Our use case examines whether a model’s outputf ( x i ) increases for some input x i in a counterfactual setting where a given feature of x i is increased.
Revised: Our use case examines whether a model’s outputf ( x i ) increases for some input x i in a counterfactual setting where a given feature of x i is increased.

Original: Like forward simulation, each observation used to train and evaluate the agent consists of a singledata-point but differs in that each observation also includes the base model prediction on that datapoint: ( x i , f ( x i ) , E ( x i , f )) .
Revised: Like forward simulation, each observation used to train and evaluate the agent consists of a singledata-point but differs in that each observation also includes the base model prediction on that datapoint: ( x i , f ( x i ) , E ( x i , f )) .

Original: For a given input x i , the use case label u i answers the question: “Doesf ( x i ) increase if a particular feature of x i is increased?”.
Revised: For a given input x i , the use case label u i answers the question: “Doesf ( x i ) increase if a particular feature of x i is increased?”.

Original: We use the same agent model architectureas in the forward simulation use case.
Revised: We use the same agent model architectureas in the forward simulation use case.

Original: One important difference for counterfactual reasoning is that each  dataset D i and its correspondingprediction model f i can differ across observations.
Revised: One important difference for counterfactual reasoning is that each base dataset D i and its corresponding base model f i can differ across observations.

Original: We construct the agent’s observations in thisway to prevent the agent from simply memorizing where an individual baseprediction model isincreasing versus decreasing with respect to a particular feature, instead encouraging the agent tolearn a heuristic for this use case that generalizes across different base models (further discussionprovided in Appendix D).
Revised: We construct the agent’s observations in this way toprevent the agent from simply memorizing where an individual base model is increasing versus decreasing with respect to a particular feature, instead encouraging the agent to learn a heuristic for thisuse case that generalizes across different base models (further discussion provided in Appendix D).

Original: Data bugs is a use case where the user is asked to predict whether the base model was trained on adataset containing a systematic error.
Revised: Data bugs is a use case where the user is asked to predict whether the base model was trained on adataset containing a systematic error.

Original: For this use case, the researcher additionally needs to providea “bug definition" B , which corrupts the given base dataset.
Revised: For this use case, the researcher additionally needs to providea “bug definition" B , which corrupts the given base dataset.

Original: We consider bug definitions studied in
Revised: We consider bug definitions studied in

Original: Kaur et al.
Revised: Kaur et al.

Original: [18] and summarize their motivation for identifying each of these bugs: The missing valuebug occurs when missing values for a feature are imputed using the mean feature value.
Revised: [20] and summarize their motivation for identifying each of these bugs: The missing valuebug occurs when missing values for a feature are imputed using the mean feature value.

Original: This can biasthe prediction model towards the mean value.
Revised: This can biasthe prediction model towards the mean value.

Original: The redundant feature bug occurs when the datasetcontains features that are redundant, or contain the same predictive information.
Revised: The redundant feature bug occurs when the datasetcontains features that are redundant, or contain the same predictive information.

Original: This can distributethe true importance of the measure between the redundant features, failing to convey the measure’sactual importance.
Revised: This can distributethe true importance of the measure between the redundant features, failing to convey the measure’sactual importance.

Original: We note that while explanations may not be the only approach to identify thesedata bugs, the authors demonstrate that model explanations can help users with this use case.
Revised: We note that while explanations may not be the only approach to identify thesedata bugs, the authors demonstrate that model explanations can help users with this use case.

Original: A distinguishing feature of the data bugs use case is that the study participants in Kaur et al.
Revised: A distinguishing feature of the data bugs use case is that the study participants in Kaur et al.

Original: [18]were given a set of data-points and explanations per observation because it might be easier to detect amodel bug if given more than one data-point (an illustration provided in the Appendix F.3).
Revised: [20]were given a set of data-points and explanations per observation because it might be easier to detect amodel bug if given more than one data-point (an illustration provided in the Appendix F.3).

Original: As such,each observation for the agent is also a set of data-point, model prediction, and explanation tuples:{ ( x ( j ) i , f ( x ( j ) i ) , E ( x ( j ) i , f i )) }
Revised: As such,each observation for the agent is also a set of data-point, model prediction, and explanation tuples:{ ( x ( j ) i , f ( x ( j ) i ) , E ( x ( j ) i , f i )) }

Original: Sj =1 , where x ( j ) i denotes the j th element in the set which is part of thei th observation.
Revised: Sj =1 , where x ( j ) i denotes the j th element in the set which is part of thei th observation.

Original: In our experiments, we vary the proportion of datasets D i that have bug B .
Revised: In our experiments, we vary the proportion of datasets D i that have bug B .

Original: For thedatasets that are bugged, we set u i = 1 and u i = 0 otherwise.
Revised: For thedatasets that have the bug, we set u i = 1 and u i = 0 otherwise.

Original: Again, for data bugs each  D i and its corresponding prediction model f i differ across observationsbecause data bugs is a global problem , e.g. the use case label u
Revised: For data bugs, each base dataset D i and its corresponding base model f i differ across observationsbecause data bugs is a global problem , i.e. the use case label u

Original: i is a property of base model f i .
Revised: i is a property of base model f i .

Original: Specifically, we derive many D i by subsampling onebase dataset D .
Revised: Specifically, we derive many D i by subsampling one dataset D .

Original: This allows us to create a variety of training (and evaluation) examples for the agent with varying use case labels, encouraging theagent to learn a heuristic that generalizes better across different sets of points.
Revised: This allows us to create a variety oftraining (and evaluation) examples for the agent with varying use case labels, encouraging the agentto learn a heuristic that generalizes better across different sets of points.

Original:  Recreating Human Subject Study Findings
Revised:  Comparing Simulation and Humans

Original: We ran SimEvals that replicate the settings of two existing human subject studies [18, 14] todemonstrate that SimEvals can correctly measure the predictiveness of different explanation methods.
Revised: To demonstrate that SimEvals are able to identify promising explanations, we run SimEvals onthe three use cases discussed in Section 4 and compare the algorithmic agent results to user studyresults, which come from either prior user studies or our own Mechanical Turk studies.

Original: Specifically, we validate that the algorithmic  agent performs  well when given an explanation that ahuman can perform well with and performs poorly when given an explanation that is not useful toa human.
Revised: Specifically,we validate that the algorithmic SimEval agent performs relatively well when given an explanationthat a human can perform well with, and performs relatively worse when given an explanation that isnot useful to a human.

Original:  In  the following result tables, we denote explanation settings that were considered in theoriginal user study with ⋆ .
Revised: We include training details for our SimEvals in Appendix F, extended results(with error bars) and comparisons with human results in Appendix G. 5.1 User study details In our experiments, we compare the algorithmic SimEval agent’s performance to the average humansubject performance at predicting the use case label in a user study. For the Forward Simulationuse case, we compare the algorithmic agent’s performance to human performance as reported in apreviously conducted independent user study by Hase and Bansal [16].

Original: We then used our framework to screen additional explanation candidates.
Revised: We conduct new user studiesto measure human performance on two of the use cases.

Original: Training details can be found in Appendix F and extended results (with error bars) in Appendix G.
Revised: For Counterfactual Reasoning , thereis no existing user study from the motivating work from Kumar et al.

Original: 5.1 Forward Simulation
Revised: [25], necessitating that werun our own. For Data Bugs , we conducted our own study that, unlike the original one by Kaur etal [20], individually measures the usefulness of candidate explanations. We expand on the differencesbetween the two studies in Section 5.4. Our user studies were conducted on Amazon Mechanical Turk, where we presented Turkers withthe observations sampled from the dataset that SimEvals were trained and evaluated on.

Revised: We choosea between-subjects study design, where each participant is randomly assigned to one explanationsettings from the SimEvals .

Revised: In non-baseline settings, participants are also provided with explanationsfor each x i , which we described to the Turkers as importance scores (Figure 7).

Revised: The study is splitinto two parts.

Revised: First, each participant completes a “Train” survey (analogous to agent training) wherethey are asked to predict the use case label for 15 observations and provided feedback on whethertheir responses were correct after every 5 observations.

Revised: Then participants complete a “Test” survey(analogous to agent evaluation) where they predict the use case label for another 15 observationswithout feedback.

Revised: Each question corresponds directly to 1 randomly sampled observation used totrain or test the algorithmic agent.

Revised: For each use case, we recruit 80 total participants: 20 participantsfor each explanation setting.

Revised: Other details, including the study interface and instructions for each usecase, participant recruitment process, details of initial pilot studies are provided in Appendix H.

Revised: 5.2 Forward Simulation

Original: Following Hase and Bansal [14], we compare the prediction accuracy of our agent on the ‘Noexplanation’ baseline setting (where it receives only the data-point without explanation) with itsprediction accuracy when given both the data-point and its corresponding explanation.
Revised: Following Hase and Bansal [16], we compare the prediction accuracy of our agent in the ‘Noexplanation’ baseline setting (where the agent/human receives only the data-point without explanation)with its prediction accuracy when given both the data-point and its corresponding explanation.

Original: The authorsevaluate 5 explanations: 2 are well-known explanations (LIME [35], Anchors [36]) and the restare bespoke.
Revised: Theauthors evaluate 5 explanations in their user study: 2 are well-known explanations (LIME [37], Anchors [38]) and the rest are bespoke.

Original: The authors find that LIME achieves an accuracy increase that is more than twicethat achieved by other methods.

Original: We do not evaluate the authors’ bespoke methods since noneoutperformed the standard methods.
Revised: In this work, we do not evaluate the authors’ bespokemethods since none outperformed the standard methods as described in Appendix C.

Original: [Table]
Revised: [Table]

Original: Findings:  As shown in Table 1, using 16 or 32 training observations (as done in the original study),we found that  the difference between LIME and the baseline is greater than the difference between
Revised: Findings: Hase and Bansal [16] found that participants who were given the LIME explanation forforward simulation achieved an accuracy increase that is more than twice that achieved by othermethods. As shown in Table 1, we find that SimEvals corroborates this result from the user studyeven when varying the agent’s train set size. In Appendix G.1, we provide additional SimEvalswhich hypothesize that SHAP and GAM are also promising choices for this use case. 5.3 Counterfactual reasoning

Original: Anchors and the baseline, corroborating findings from Hase and Bansal [14].
Revised: The counterfactual reasoning use case is introduced by Kumar et al. [25], where the authors discusswhy SHAP feature importance scores [29] do not explicitly attempt to provide guidance for theuse case. Specifically, a positive SHAP feature importance score does not necessarily imply thatincreasing that feature will increase the model’s prediction.

Original: However, if the paperauthors had performed SimEvals , they would have noticed that SHAP could have also been apromising explanation method.
Revised: We additionally hypothesize that anapproximation-based explanation like LIME might be more useful for this use case than SHAP. Weconduct a user study for this use case

Original: We note thatfor forward simulation, there exist known heuristics topredict the model’s output using some explanation methods (i.e., it is possible for a human or agentto achieve (near) perfect accuracy). For example, SHAP feature attributions by definition sum tothe model prediction [27].
Revised: [Table] Findings: As shown in Table 2 (Left), we find that the simulated agent had significantly higheraccuracy when given LIME explanations. Additionally, we find that this trend is consistent acrossall training set sizes: the agent achieves a 37% increase in validation accuracy when given LIMEinstead of any other explanation after observing only 4 train observations.

Original: With sufficient examples (i.e. N > 1000 ), the predictiveness of SHAPexplanations for this use case isdemonstrated by the agent’s near perfect accuracy.
Revised: These results suggest that,of the set of explanations considered, LIME is the most promising explanation for the counterfactualreasoning use case.

Original: 5.2 Data bugs
Revised: In Appendix G.5 we demonstrate that LIME outperforms all other explanationmethods consistently across different types of model architectures. As shown in Table 2 (Right), the gap in agent performance between LIME and the other explanationsis reflected in the human test accuracy in our Turk study. In particular, a number of Turkers areable to use the predictiveness of LIME to complete the use case with high accuracy; whereas, ingeneral, Turkers using SHAP, GAM, and No Explanation perform no better than random guessing. We ran an ANOVA test and found a statistically significant difference between the Turkers’ accuracyusing LIME vs. all other explanation settings and no statistically significant difference between theother settings (Appendix G.2).

Revised: Note that there is also significant overlap in the error bars of agentperformance on SHAP, GAM, No Explanation (Table 6).

Revised: We observe greater variance in participantaccuracy between participants for LIME compared to the other explanations.

Revised: We believe that this isbecause many, but not all, Turkers were able to learn how to use the LIME explanation successfully.

Revised: 5.4 Data bugs

Original: Kaur et al.
Revised: Kaur et al.

Original: [18] evaluates two explanations (SHAP [27] and GAMs [15]) on two  data bugs (missingvalues and redundant features).
Revised: [20] study two explanations (SHAP [29] and GAMs [17]) on two types of data bugs(missing values and redundant features).

Original: In their study, users were prompted to suggest bugs that they believedmay be present.
Revised: In their study, users were prompted to suggest bugs that theybelieved may be present via semi-structured explorations using these explanation tools.

Original: The authors found that 4 out of 11 users mentioned a missing values bug and 3 ofthem mentioned redundant features.
Revised: They foundthat 4 out of 11 users mentioned a missing values bug and 3 users mentioned redundant features.

Original: Their results suggest that both SHAP and GAM may be usefulfor finding  data bugs.
Revised: Their results suggest that both SHAP and GAM may be useful for finding these types of data bugs,though they do not individually test each explanation’s usefulness

Original: The base dataset was UCI Adult and the base model  for SHAP was Light-GBM.
Revised: and they also did not considerproviding users with baseline information content (without model explanations) to compare withsettings where users were presented with explanations. Thus, for this use case, we conducted our own

Original: Since GAMs are interpretable by design, the GAM model is the base prediction model in the GAMexplanation setting only.
Revised: Turk user study on the missing values bug, evaluating both the explanations considered in the originalstudy and including additional explanation/baseline comparisons.

Original: We present SimEval results for the missing values bug below (redundantfeatures can be found in Appendix G.4).
Revised: We present SimEval results forthe missing values bug below (redundant features can be found in Appendix G.4).

Original: Missing Values: Kaur et al.
Revised: Missing Values: Kaur et al.

Original: [18] implements the missing values bug by replacing the ‘Age’ valuewith the mean value of 38 for 10% of adults with > 50 k income.
Revised: [20] implement the missing values bug by replacing the ‘Age’ valuewith the mean value of 38 for 10% of adults with > 50 k income.

Original: We detail how we recreate this bugin Appendix F.3.
Revised: We detail how we recreate thisbug in Appendix F.3.

Original: We additionally vary the number of data-points per train set observation (e.g. thesize S of each set), another type of information content that can inform human subject study design.
Revised: In addition to comparing various explanation methods, we also use SimEvalsto evaluate the effect of varying the number of data-points per train set observation (e.g. the size Sof each set). We note is another type of information content that the researcher must select whenrunning a user study.

Original: [Table] Findings: Using SimEvals , we verified that SHAP and GAM contained predictive information toidentify the missing values bug.
Revised: Findings: As shown in Table 3 (Left), we verify using SimEvals that SHAP and GAM explanationscontain predictive information to identify the missing values bug.

Original: Furthermore, as shown in Table 11, we found that the accuracy ofthe algorithmic agent drops when given too few data-points (i.e., for observation sets of size ≤ 10 ).
Revised: We also find that the accuracy ofthe algorithmic agent drops when given too few data-points (i.e., for observation sets of size ≤ 10 ).

Original: We note that S , the number of data-points per observation, is yet another information content choicethat SimEval can efficiently evaluate.

Original: This means that a small number of data-points do not containsufficient signal to detect the bug.
Revised: This means that a small number of data-points do not contain sufficient signal to detect the bug.

Original: Since the researcher would need to present a large number ofdata-points and explanations to a human user in a human-interpretable manner, a user interface isneeded where the explanations are presented inan aggregated fashion, which is indeed the approachtaken by Kaur et al. [18] for their study’s interface. In Appendix G, we find similar conclusions evenwhen varying the bug strength and the parameterization of the agent model.  MTurk User Study
Revised: Sincethe researcher would need to present a large number of data-points and explanations to a human userin a human-interpretable manner, a user interface is needed where the explanations are presented in

Original: The counterfactual reasoning use case is introduced by Kumar et al.

Original: [23], where the authors discusswhy SHAP feature importance scores do not explicitly attempt to provide guidance for the use case.

Original: Specifically, a positive SHAP feature importance score does not necessarily imply that increasing thatfeature will increase the model’s prediction.

Original: We additionally hypothesize that an approximation-basedexplanation like LIME might be more useful for this use case than SHAP.

Original: [Table]
Revised: [Table]

Original: We first performed SimEvals to compare candidate explanation methods and then we ran a user studyon Amazon Mechanical Turk to validate that there is a match in the relative rankings of differentexplanations based on the accuracy of the simulated agent and human subject.

Original: 3 We hope that thissection can serve as a guiding example for others who want to perform SimEvals for their use cases.

Original: [Table] For this use case [Table] we construct a 2 D synthetic regression dataset inspired by Kumar et al.
Revised: an aggregated fashion, which is indeed the approach taken by both Kaur et al.

Original: We [Table]
Revised: [20] and our study.

Original: define the use case label as 1 if increasing a data-point’s specific feature x 2 increases the regression [Table] model prediction f ( x ) (holding the other feature x 1 constant), 0 otherwise. The data generation
Revised: In Appendix G, we observe similar trends in the relative performance of explanation methods evenwhen varying the bug strength and the parameterization of the agent model.

Original: [Table] and other use case details are described extensively in Appendix D. In  the baseline setting, theagent/human receives no explanation. We evaluate the same local explanations as previous usecases (Section 5).
Revised: In our Turk study, we utilized the findings from our SimEvals experiments and presented theexplanations for S = 1000 data-points per observation in an aggregated fashion (via scatter plotsas shown in Figure 11).

Original: We use a Light-GBM Regressor [19] as the base prediction model class for allexplanation settings except GAM, in which the GAM is the base model. Findings: As shown in Table 3, we find that the simulated agent had significantly higher accuracywhen given LIME explanations.
Revised: As shown in Table 3 (Right), participants assigned to SHAP or GAMconsistently outperformed the participants assigned to LIME or the Model Prediction baseline, eventhough our participant population for the user study were dissimilar from the participant population(data scientists) of the original user study.

Original: Additionally, we find that the results are consistent across all trainingset sizes: the agent achieves a 37% increase in validation accuracy when given LIME instead of anyother explanation after observing only 4 train observations. These results affirm our hypothesis that LIME is more useful than SHAP for the counterfactual reasoning use case presented in Kumar etal [23].
Revised: We conducted pairwise statistical analyses and foundstatistical significance between SHAP and LIME/Model Prediction as well as GAM and LIME/Model

Original: In Appendix G.5 we demonstrate these results are consistent across model architectures. 6.2 User study Findings
Revised: Prediction and no statistical significance between other pairs of experimental conditions.

Original:  We ran a human subject study on Amazon Mechanical Turk where we presented Turkers with theobservations sampled from the dataset that SimEvals were trained on. We introduced the Turkersto the use case using the context of furniture pricing: Turkers were shown data about a furnitureitem (height and length measurements x i and the item’s price f ( x i ) ) and were asked to performcounterfactual reasoning by choosing whether or not increasing the item’s length would increase itsprice (a screenshot of the interface is shown in Figure 8).
Revised:  Discussion

Original: Participants were instructed to set asideany prior knowledge they may have about furniture pricing and only use the information providedduring the study. In some settings, users were also provided with explanations for each x i , which wedescribed to  the Turkers as importance scores (Figure 7). The study was split into two parts. First, each participant completed a “Train” survey (analogousto agent training) where they were asked to perform counterfactual reasoning on 15 observations(furniture items).
Revised: Our experiments and analyses in Section 5 demonstrate that SimEvals are able to provide aninformative measure of the predictive information contained in a design set-up, and as such canidentify promising explanations from a set of candidate explanations.

Original: Each participant was provided feedback on whether their responses were correctafter every 5 observations. Then participants completed a “Test” survey (analogous toagent evaluation) where they performed counterfactual reasoning on another 15 observations without feedback. We chose a between-subjects study design, where each participant was randomly assigned to 1 ofthe 4 explanation settings from the SimEvals . The measurements and importance scores for eachfurniture item shown to participants in the Turk Study correspond directly to 1 randomly sampledobservation used to train or test the algorithmic agent.
Revised: However, we emphasize thatbecause SimEvals only measure predictive information, the algorithmic agent’s accuracy cannot bedirectly interpreted as the explanation’s utility to a human, as evidenced by the differences betweenraw agent and human test accuracies in Tables 1, 2, and 3.

Original: In total, we recruited 80 total participants: 20participants for each explanation setting. Other details, including a description of the study interface,participant recruitment process, details of initial pilot studies are provided in Appendix H. Findings: As shown in Table 3, we found that  in general, human test accuracy matches the orderingfound using the SimEvals .
Revised: Furthermore, we note that small differencesbetween SimEval agent’s relative performance on explanations (e.g., in Table 2) may not generalizeto humans.

Original: In particular, a number of Turkers were able to use the predictiveness of LIME to complete the use case with high accuracy; whereas, in general, Turkers using SHAP, GAM,and No Explanation performed no better than random guessing. We ran an ANOVA test and found astatistically significant difference betweenthe Turkers’ accuracy using LIME vs. all other explanationsettings ( p < 0 . 006 for all pairs).
Revised: We discuss potential human factors that may contribute to these differences between

Original: We also observed greater variance in participant accuracy betweenparticipants for LIME compared to the other explanations. We believe that this is because many, butnot all, Turkers were able to learn how to use the LIME explanation successfully. On the other hand,like the algorithmic agent, the Turkers were uniformly unable to find a predictive heuristic to use SHAP, GAM, and No Explanation.
Revised: SimEval and human performance in Appendix J. These findings suggest an important direction forfuture research to better understand these differences between agent and human, i.e., how humanfactors beyond predictive information affect humans’ ability to reason about model explanations.

Original:  Conclusion
Revised:  Conclusion

Original: We proposed a use-case-grounded algorithmic evaluation called SimEvals to efficiently screenchoices of information content and identify good candidates for a user study.
Revised: We proposed a use-case-grounded algorithmic evaluation called SimEvals to efficiently screenchoices of information content and identify good candidates for a user study.

Original:  SimEval involvestraining an agent to predict the use case label given the information content that would be presented toa user.
Revised: Each SimEval involvestraining an agent to predict the use case label given the information content that would be presentedto a user.

Original: We instantiated SimEvals on three use cases motivated by prior work, and demonstrated thatthe agent’s test set accuracy can be used as a measure of the predictiveness of information content(more specifically, of an explanation method) for each use case.
Revised: We instantiated SimEvals on three use cases motivated by prior work, and demonstratedthat the agent’s test set accuracy can be used as a measure of the predictiveness of informationcontent (more specifically, of an explanation method) for each use case.

Original: We find that the agent recovers therelative ordering of explanations found in both prior user studies and our own MTurk study.
Revised: We found that humansperform significantly better on explanations that SimEvals selects as promising compared to otherexplanations.

Original: Whileour experiments focus primarily on the design choice of selecting explanation methods, we note thatresearchers can also use SimEvals to screen the predictiveness of other types of information content,such as hyperparameter selection for explanation methods or the baseline information given witheach observation.
Revised: While our experiments focus primarily on the design choice of selecting explanationmethods, we note that researchers can also use SimEvals to screen the predictiveness of other typesof information content, such as hyperparameter selection for explanation methods or the baselineinformation given with each observation.

Original: We hope that this work can be incorporated into future explanation evaluationworkflows to enhance the efficiency and effectiveness of human subject experiments.
Revised: We hope that this work can be incorporated into futureexplanation evaluation workflows to enhance the efficiency and effectiveness of user study design.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately 
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: reflect the

Original: paper’scontributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 1 and Appendix J
Revised: [Yes] See Section 1 and Appendix J(c)

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See
Revised: [Yes] See

Original: Appendix J
Revised: Section

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Revised: [Table]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Revised: [Table]

Original: If you ran experiments...
Revised: If you ran experiments...

Original:  (a) Did you include the code, data, and instructions needed to reproduce the main experi- 
Revised: [Table] (a) Did you include the code, data, and instructions needed to reproduce the main experi- [Table]

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] See supplemen- tal material. 
Revised: [Yes] See supplemental material. [Table]

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Appendix F
Revised: [Yes] See Appendix F

Original:  (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: [Table] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Appendix G
Revised: [Yes] See Appendix G

Original:  (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [Table] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix F.
Revised: [Yes] See Appendix F.

Original: [Table]
Revised: [Table]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: [Table]
Revised: [Table]

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Appendix E.2and Appendix F.
Revised: [Yes] See Appendix E. [Table] and Appendix F.

Original: [Table]
Revised: [Table]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See Appendix F.
Revised: [Yes] See Appendix F.

Original: [Table]
Revised: [Table]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: [Table]
Revised: [Table]

Original: Code included in the supplemental material.
Revised: Code included in the supplemental material.

Original: [Table]
Revised: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]
Revised: [Table]

Original: using/curating?
Revised: using/curating?

Original: [Yes] See Appendix F.1.
Revised: [Yes] See Appendix F.1.

Original: [Table]
Revised: [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: [Table]
Revised: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: [Table]
Revised: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]
Revised: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if [Table] applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [Yes] See Appendix H
Revised: [Yes] See Appendix H

Original: [Table]
Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks

Original: [Table]
Revised: Board

Original: Board (IRB) approvals, if applicable?
Revised: (IRB) approvals, if applicable?

Original: [Yes] See AppendixH
Revised: [Yes] See Appendix

Original: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [Yes] See Appendix H
Revised: [Yes] See Appendix H

Original: [Table]

Original: A Examples of algorithmic agents in prior work
Revised: A Examples of algorithmic agents in prior work

Original: We can unify existing ad-hoc examples that use algorithmic agents to perform use case groundedevaluations under our framework.
Revised: We can unify existing ad-hoc examples that use algorithmic agents to perform use case groundedevaluations under our framework.

Original: We group existing algorithmic evaluations into two categories:
Revised: We group existing algorithmic evaluations into two categories:

Original: Heuristic-Based and Learning-Based.
Revised: Heuristic-Based and Learning-Based.

Original: A.1 Heuristic-Based Evaluations
Revised: A.1 Heuristic-Based Evaluations

Original: These algorithmic agents were not trained, and instead were defined using hand-designed heuristics.
Revised: These algorithmic agents were not trained, and instead were defined using hand-designed heuristics.

Original: In contrast, our agent is trained on observations and use case labels.
Revised: In contrast, our agent is trained on observations and use case labels.

Original: Expo [31]: The use case studied is whether an explanation can be used to determine how to modifyfeatures of an instance to achieve a target prediction.
Revised: Expo [33]: The use case studied is whether an explanation can be used to determine how to modifyfeatures of an instance to achieve a target prediction.

Original: There is no agent training process: the agent usesa randomized greedy heuristic (which involves picking the largest feature).
Revised: There is no agent training process: the agent usesa randomized greedy heuristic (which involves picking the largest feature).

Original: The agent is evaluated bythe number of modifications that it makes to reach the target prediction.
Revised: The agent is evaluated bythe number of modifications that it makes to reach the target prediction.

Original: Influence functions [21] / RPS [43]: The use case studied is whether explanations can be used toidentify mislabeled examples in a dataset.
Revised: Influence functions [23] / RPS [45]: The use case studied is whether explanations can be used toidentify mislabeled examples in a dataset.

Original: The agent is also heuristic-based, flagging points withthe largest influences to be inspected for potentially being mislabeled.
Revised: The agent is also heuristic-based, flagging points withthe largest influences to be inspected for potentially being mislabeled.

Original: The agent is evaluated by thenumber of correct points it selects for inspection.
Revised: The agent is evaluated by thenumber of correct points it selects for inspection.

Original: Anchors [36]: The use case studied is whether explanations can help a user make accurate predictionson unseen instances.
Revised: Anchors [38]: The use case studied is whether explanations can help a user make accurate predictionson unseen instances.

Original: The authors specify a heuristic that they use to compare Anchors with LIME.
Revised: The authors specify a heuristic that they use to compare Anchors with LIME.

Original: The heuristic for Anchors is simply to check whether the conditions are satisfied.
Revised: The heuristic for Anchors is simply to check whether the conditions are satisfied.

Original: The heuristic for
Revised: The heuristic for

Original: LIME is to use the linear approximation directly (while additionally checking whether the point ofprediction is near the approximation point).
Revised: LIME is to use the linear approximation directly (while additionally checking whether the point ofprediction is near the approximation point).

Original: The agent is evaluated using its precision when used topredict labels on unseen instances.
Revised: The agent is evaluated using its precision when used topredict labels on unseen instances.

Original: A.2 Learning-Based Evaluations.
Revised: A.2 Learning-Based Evaluations

Original: Student-teacher Models [34]: The use case studied is whether explanations can help a student modellearn to simulate a teacher model for a sentiment analysis task and question answering task.
Revised: Student-teacher Models [36]: The use case studied is whether explanations can help a student modellearn to simulate a teacher model for a sentiment analysis task and question answering task.

Original: Theauthors quantify the benefit of an explanation as the improvement in simulation accuracy when theagent has access to explanations during the training process versus when the agent does not.
Revised: Theauthors quantify the benefit of an explanation as the improvement in simulation accuracy when theagent has access to explanations during the training process versus when the agent does not.

Original: Oneimportant distinction from our approach is that it is noted in [34] that their algorithmic evaluationdoes not attempt to replicate the evaluation protocols used in a human subject study.
Revised: Oneimportant distinction from our approach is that it is noted in [36] that their algorithmic evaluationdoes not attempt to replicate the evaluation protocols used in a human subject study.

Original: This point isaffirmed by the significant difference between the results of the authors’ human subject study vs. theiralgorithmic evaluation.
Revised: This point isaffirmed by the significant difference between the results of the authors’ human subject study vs. theiralgorithmic evaluation.

Original: In contrast, our approach is designed with a human subject study in mind: weexplicitly do attempt to construct the agent’s observations in a way that reflects the information thatwould be presented to a human subject.
Revised: In contrast, our approach is designed with a human subject study in mind: weexplicitly do attempt to construct the agent’s observations in a way that reflects the information thatwould be presented to a human subject.

Original: Other work [12, 40] similar to [34] that evaluates explanations without a human in the loop typicallyexamines how explanations can improve the model training process.
Revised: Other work [13, 42] similar to [36] that evaluates explanations without a human in the loop typicallyexamines how explanations can improve the model training process.

Original: In contrast, our work focuses onuse cases where explanations are intended to be shown to and used by humans.
Revised: In contrast, our work focuses onuse cases where explanations are intended to be shown to and used by humans.

Revised: A.3 Distinction from prior work

Revised: In Table A.3, we summarize prior work along two main axes: their algorithmic evaluation ofexplanations (if they conducted them) and their human evaluation of explanations (if they conductedthem).

Revised: On the algorithmic evaluation front, no prior work allows their agent to learn how to useexplanations (i.e., a researcher does not need to specify a heuristic for how the explanation will beused) and provides a general framework that encompasses many use cases.

Revised: Furthermore, no priorwork extensively verifies their evaluation framework with human evaluation to show the comparabilitybetween algorithmic and human performance, whereas our work evaluates as many explanations asmany user studies.

Original: B Use Case Algorithms
Revised: B Use Case Algorithms

Original: We provide more detailed algorithms to describe how the dataset is generated for the forward simulation and counterfactual reasoning use cases. 
Revised: We provide more detailed algorithms to describe how the dataset is generated for the forwardsimulation and counterfactual reasoning use cases. Algorithmic Evaluation Human Evaluation Learning-based?

Revised: (i.e.,noheuristic required).

Original: B.1 Forward simulation
Revised: Does framework generalize? Conducts a user study? Agent matches human? Num. explanations (baselines) evaluated

Revised: Expo

Revised: No

Revised: No

Revised: Yes

Revised: No 1 (1)

Revised: IF/RPS

Revised: No

Revised: No

Revised: No – –

Revised: S-T Models

Revised: Yes

Revised: No

Revised: Yes

Revised: No 2 (1)

Revised: Anchors

Revised: No

Revised: No

Revised: Yes

Revised: Yes 2 (0)

Revised: User studies – –

Revised: Yes – ∼ 2-4 (1-2)

Revised: Ours (SimEvals)

Revised: Yes

Revised: Yes

Revised: Yes

Revised: Yes 4 (2)

Revised: B.1 Forward simulation

Original: • Given base dataset D , split into train/test/validation
Revised: • Given base dataset D , split into train/test/validation

Original: • Train model f on the train set
Revised: • Train model f on the train set

Original: • Let N T be size of desired training dataset and N V be size of the desired validation dataset
Revised: • Let N T be size of desired training dataset and N V be size of the desired validation dataset

Original: • Sample N T points from the train set and for each point x i ∈ D train generate an explanationof f
Revised: • Sample N T points from the train set and for each point x i ∈ D train generate an explanation

Original:  at that point, add the tuple ( x i , E ( x i , f ))to the training
Revised: of f at that point, add the tuple ( x i , E ( x i , f ))

Original: dataset.•
Revised: to the training dataset.

Original:  Repeat process for N V points but from the validation set
Revised: • Repeat process for N V points but from the validation set

Original: Note the baseline condition would be generated in the same way, but excluding the explanation.
Revised: Note the baseline condition would be generated in the same way, but excluding the explanation.

Original: B. Reasoning
Revised: B.2 Counterfactual Reasoning

Original: • Given base model family F , desired dataset size N•
Revised: • Given base model family F , desired dataset size N

Original:  For i = 1 , ..., N :– Sample saddle-point function g i as detailed in Appendix D.2–
Revised: • For i = 1 , ..., N : – Sample saddle-point function g i as detailed in Appendix D.

Original:  Train f i on g
Revised: – Train f i on g

Original: i as detailed in Appendix D.2–
Revised: i as detailed in Appendix D.2–

Original: Sample new point x i from g i– Get prediction f i ( x i )
Revised: Sample new point x i from g i – Get prediction f i ( x i )

Original: – Generate E ( x i , f i ) of f i at x i
Revised: – Generate E ( x i , f i ) of f i at x i

Original: • Set the i th observation to be: ( x i , f i ( x i ) , E ( x i , f i ))
Revised: • Set the i th observation to be: ( x i , f i ( x i ) , E ( x i , f i ))

Original: B.3 Data Bugs
Revised: B.3 Data Bugs

Original: Algorithm 1 generates a dataset of N observations for the data bug use case.
Revised: Algorithm 1 generates a dataset of N observations for the data bug use case.

Original: 1: Given base dataset D , base model family F and bug specification B 2: Let N be the size of desired dataset,
Revised: 1: Given base dataset D , base model family F and bug specification B 2: Let N be the size of desired dataset,

Original: S be the observation set size 3: for i in 1 ...
Revised: S be the observation set size 3: for i in 1 ...

Original: N do 4:
Revised: N do 4:

Original: Subsample D i from D 5:
Revised: Subsample D i from D 5:

Original: With probability 0.5, apply B to D i 6:
Revised: With probability 0.5, apply B to D i 6:

Original: Train f i on D i 7: for j in 1 ...
Revised: Train f i on D i 7: for j in 1 ...

Original: S do 8:
Revised: S do 8:

Original: Sample x ( j ) i from D i 9:
Revised: Sample x ( j ) i from D i 9:

Original: Get prediction f i ( x ( j ) i ) 10:
Revised: Get prediction f i ( x ( j ) i ) 10:

Original: Generate E ( x ( j ) i , f i ) of f i at x ( j ) i 11: end for 12:
Revised: Generate E ( x ( j ) i , f i ) of f i at x ( j ) i 11: end for 12:

Original: Set the i th observation to be: { ( x ( j ) i , f i ( x ( j ) i ) , E ( x ( j ) i , f i )) }
Revised: Set the i th observation to be: { ( x ( j ) i , f i ( x ( j ) i ) , E ( x ( j ) i , f i )) }

Original: Sj = 13: end for
Revised: Sj = 13: end for

Original: C Summary of prior user studies
Revised: C Summary of prior user studies

Original: We discuss the findings from the prior user studies that we recreate results from.
Revised: We discuss the findings from the prior user studies that we recreate results from.

Original: Forward Simulation from [14].
Revised: Forward Simulation from [16].

Original: The authors evaluate 5 explanations in this paper: 2 (LIME,
Revised: The authors evaluate 5 explanations in this paper: 2 (LIME,

Original: Anchors) are well-known, open-source explanations and 3 (Prototype, DB, and Composite) arebespoke explanations.
Revised: Anchors) are well-known, open-source explanations and 3 (Prototype, DB, and Composite) arebespoke explanations.

Original: In the tabular setting, the authors find that LIME outperforms all otherexplanations (change of 11.25% in user accuracy compared to Anchors 5.01%, Prototype 1.68%,5.27%, 0.33%).
Revised: In the tabular setting, the authors find that LIME outperforms all otherexplanations (change of 11.25% in user accuracy compared to Anchors 5.01%, Prototype 1.68%,5.27%, 0.33%).

Original: The base accuracy before explanations was 70.74%.
Revised: The base accuracy before explanations was 70.74%.

Original: Since none of the bespokeexplanations outperformed the open-source explanations, we chose not to re-implement them in ourstudy.
Revised: Since none of the bespokeexplanations outperformed the open-source explanations, we chose not to re-implement them in ourstudy.

Original: Data Bugs from [18].
Revised: Data Bugs from [20].

Original: The authors evaluate 2 explanations: SHAP and GAM.
Revised: The authors evaluate 2 explanations: SHAP and GAM.

Original: The set-up of thisstudy did not explicitly ask the users to answer whether the bug was present, but rather asked theuser to identify the bugs themselves (i.e., if a user could identify the existence of the bug, then theexplanation was useful).
Revised: The set-up of thisstudy did not explicitly ask the users to answer whether the bug was present, but rather asked theuser to identify the bugs themselves (i.e., if a user could identify the existence of the bug, then theexplanation was useful).

Original: The authors find that 4 out of 11 data scientists were able to identify themissing values bug and 3 out of 11 the redundant features bug.
Revised: The authors find that 4 out of 11 data scientists were able to identify themissing values bug and 3 out of 11 the redundant features bug.

Original: While the authors do not distinguishwhen SHAP is helpful versus GAM.
Revised: While the authors do not distinguishwhen SHAP is helpful versus GAM.

Original: From Figure 1 of their paper, SHAP and GAM seem to havesimilar visualizations and thus might equally help a user.
Revised: From Figure 1 of their paper, SHAP and GAM seem to havesimilar visualizations and thus might equally help a user.

Original: We find similar results with our simulatedagent.
Revised: We find similar results with our simulatedagent.

Original: D Counterfactual Reasoning Use Case Details
Revised: D Counterfactual Reasoning Use Case DetailsD.1 Distinction from Counterfactual Explanations

Original: D.1 Distinction from Counterfactual Explanations

Original: There are significant differences between our use case and the desiderata for which counterfactualexplanation methods for ML models are developed.
Revised: There are significant differences between our use case and the desiderata for which counterfactualexplanation methods for ML models are developed.

Original: Why existing counterfactual explanation methods do not apply: A growing body of IMLresearch has proposed new methods to provide counterfactual explanations for a black-box model [39,29, 17].
Revised: Why existing counterfactual explanation methods do not apply: A growing body of IMLresearch has proposed new methods to provide counterfactual explanations for a black-box model [41,31, 19].

Original: These methods provide explanations that fulfill the desiderata introduced in [41], definedformally as:
Revised: These methods provide explanations that fulfill the desiderata introduced in [43], definedformally as:

Original: Definition D.1.
Revised: Definition D.1.

Original: Given input x , model f , and desired outcome set Y , counterfactual explanation x ′ isthe closest possible point to x such that the model’s prediction on the counterfactual point is in thedesired outcome set: f ( x ′ ) ∈ Y .
Revised: Given input x , model f , and desired outcome set Y , counterfactual explanation x ′ isthe closest possible point to x such that the model’s prediction on the counterfactual point is in thedesired outcome set: f ( x ′ ) ∈ Y .

Original: The real-world utility of counterfactual explanations relies on a large number of assumptions [6].
Revised: The real-world utility of counterfactual explanations relies on a large number of assumptions [6].

Original: Many variants of the counterfactual problem have been proposed to account for a wide range ofreal world objectives and constraints, such as the feasibility or difficulty of the proposed actions(recourse)[39].
Revised: Many variants of the counterfactual problem have been proposed to account for a wide range ofreal world objectives and constraints, such as the feasibility or difficulty of the proposed actions(recourse)

Original: Despite these varying goals, however, to our knowledge prior work on counterfactualexplanations shares the general problem set-up in Definition D.1.
Revised: Despite these varying goals, however, to our knowledge prior work on counterfactualexplanations shares the general problem set-up in Definition D.1.

Original: While this problem set-up in Definition D.1 is related to our use case, it is significantly different for anumber of reasons.
Revised: While this problem set-up in Definition D.1 is related to our use case, it is significantly different for anumber of reasons.

Original: Primarily, while the counterfactual explanation x ′ provided by these methods isencouraged to be “close” to the original input x , the counterfactual x ′ can increase or decrease anysubset of input features within the problem’s constraints.
Revised: Primarily, while the counterfactual explanation x ′ provided by these methods isencouraged to be “close” to the original input x , the counterfactual x ′ can increase or decrease anysubset of input features within the problem’s constraints.

Original: Thus, there may be a wide range of possiblecounterfactuals x ′ of varying “closeness” to x that result in the desired outcome f ( x ′ ) .
Revised: Thus, there may be a wide range of possiblecounterfactuals x ′ of varying “closeness” to x that result in the desired outcome f ( x ′ ) .

Original: In contrast,the use case asks a more specific question of what happens when a specific feature i is increased andall other features are held constant.
Revised: In contrast,the use case asks a more specific question of what happens when a specific feature i is increased andall other features are held constant.

Original: In a scenario where it may be possible to achieve an increased model prediction f ( x ′ ) by changingother subsets of features excluding feature i , then the provided counterfactual x ′ will give noinformation relevant to the use case.
Revised: In a scenario where it may be possible to achieve an increased model prediction f ( x ′ ) by changingother subsets of features excluding feature i , then the provided counterfactual x ′ will give noinformation relevant to the use case.

Original: Similarly, it may be possible that f ( x ′ ) may decrease whensome feature j ̸
Revised: Similarly, it may be possible that f ( x ′ ) may decrease whensome feature j ̸

Original: = i is changed and feature i is increased, suggesting that f decreases when featurei is increased; but if all other features j ̸ = i are held constant , then f increases when feature i isincreased.
Revised: = i is changed and feature i is increased, suggesting that f decreases when featurei is increased; but if all other features j ̸ = i are held constant , then f increases when feature i isincreased.

Original: In these cases, the optimal counterfactual explanation x ′ may give no information abouthow f ( x ) varies with feature i .
Revised: In these cases, the optimal counterfactual explanation x ′ may give no information abouthow f ( x ) varies with feature i .

Original: Due to this misalignment between the goals of existing counterfactual explanation methods and ourparticular use case, we exclude these counterfactual explanation methods from our evaluation.
Revised: Due to this misalignment between the goals of existing counterfactual explanation methods and ourparticular use case, we exclude these counterfactual explanation methods from our evaluation.

Original: Weinstead focus our evaluation on common feature attribution methods.
Revised: Weinstead focus our evaluation on common feature attribution methods.

Original: D.2 Synthetic Data
Revised: D.2 Synthetic Data

Original: Inspired by Kumar et al.
Revised: Inspired by Kumar et al.

Original: [23], we construct a 2D toy base dataset comprising of points sampled froma saddle-point function.
Revised: [25], we construct a 2D toy base dataset comprising of points sampled froma saddle-point function.

Original: We construct a low-dimensional dataset to reduce the complexity of the taskgiven to MTurkers, as prior work [33] found that Turkers achieved higher accuracy when presentedwith data that had fewer features.
Revised: We construct a low-dimensional dataset to reduce the complexity of the taskgiven to MTurkers, as prior work [35] found that Turkers achieved higher accuracy when presentedwith data that had fewer features.

Original: Each data-point ( x i : [ x i, 1 , x i, 2 ] , y i ) used to train predictive model
Revised: Each data-point ( x i : [ x i, 1 , x i, 2 ] , y i ) used to train predictive model

Original: f i is sampled from its ownsaddle-point function g i with parameters chosen stochastically (detailed further in Section D.2.1below).
Revised: f i is sampled from its ownsaddle-point function g i with parameters chosen stochastically (detailed further in Section D.2.1below).

Original: Because each observation is sampled from a different function, each observation also has itsown predictive model f i .
Revised: Because each observation is sampled from a different function, each observation also has itsown predictive model f i .

Original: Importantly, we chose to sample points from a saddle-point function (withno noise) because saddle-point functions are either strictly concave or strictly convex with respect toeach feature x
Revised: Importantly, we chose to sample points from a saddle-point function (withno noise) because saddle-point functions are either strictly concave or strictly convex with respect toeach feature x

Original: i, 1 , x i, 2 .
Revised: i, 1 , x i, 2 .

Original: Thus, if the predictive model f i correctly learns the saddle-point function g i ,then “ground truth” use case labels u i of whether or not the predictive model’s output increases withfeature x i, 2 are available for all points.
Revised: Thus, if the predictive model f i correctly learns the saddle-point function g i ,then “ground truth” use case labels u i of whether or not the predictive model’s output increases withfeature x i, 2 are available for all points.

Original: Why each point from a separate function?
Revised: Why each point from a separate function?

Original: : We chose to sample every observation given to theagent from its own separate saddle-point function to increase the difficulty of the agent prediction task.
Revised: : We chose to sample every observation given to theagent from its own separate saddle-point function to increase the difficulty of the agent prediction task.

Original: Consider for comparison an alternative naive dataset generation procedure where all observations{ ( x i , f ( x i )) , u i } Ni =1 are sampled from the same saddle-point function g (and consequently share thesame predictive model f ).
Revised: Consider for comparison an alternative naive dataset generation procedure where all observations{ ( x i , f ( x i )) , u i } N i =1 are sampled from the same saddle-point function g (and consequently share thesame predictive model f ).

Original: Given enough observations N , because f is smooth and either strictly concave or convex, an agent can memorize for which points ( x i , f ( x i ))
Revised: Given enough observations N , because f is smooth and either strictlyconcave or convex, an agent can memorize for which points ( x i , f ( x i ))

Original: the function f is increasingvs.
Revised: the function f is increasingvs.

Original: decreasing with respect to feature x i, 2 .
Revised: decreasing with respect to feature x i, 2 .

Original: Thus, the agent can effectively memorize the use caselabels without needing to use explanations.
Revised: Thus, the agent can effectively memorize the use caselabels without needing to use explanations.

Original: This is undesirable as our intention in training the agentis to evaluate candidate explanation methods.
Revised: This is undesirable as our intention in training the agentis to evaluate candidate explanation methods.

Original: Therefore instead of using the same predictive model for all observations, we trained differentpredictive models on observations sampled from different saddle-point functions to (a) prevent theagent from learning a heuristic specific to any singular function and (b) to encourage the agent tolearn a heuristic that will generalize well on new observations from different functions.
Revised: Therefore instead of using the same predictive model for all observations, we trained differentpredictive models on observations sampled from different saddle-point functions to (a) prevent theagent from learning a heuristic specific to any singular function and (b) to encourage the agent tolearn a heuristic that will generalize well on new observations from different functions.

Original: Below, we detail how data-points and use case labels are defined for each saddle-point function(Section D.2.1) and how the agent’s train and validation sets are constructed (Section D.2.2).
Revised: Below, we detail how data-points and use case labels are defined for each saddle-point function(Section D.2.1) and how the agent’s train and validation sets are constructed (Section D.2.2).

Original: D.2.1 Saddle-Point Functions
Revised: D.2.1 Saddle-Point Functions

Original: For each observation i , we construct saddle-point function g i as follows:
Revised: For each observation i , we construct saddle-point function g i as follows:

Original: First, critical points x ∗ 1 , x ∗ 2 are chosen uniformly at random in range [10 , 15] .
Revised: First, critical points x ∗ 1 , x ∗ 2 are chosen uniformly at random in range [10 , 15] .

Original: Second, an indicatorvariable Z is sampled from a Bernoulli distribution with parameter p = 0 .
Revised: Second, an indicatorvariable Z is sampled from a Bernoulli distribution with parameter p = 0 .

Original: We define saddle-point function g i as:
Revised: We define saddle-point function g i as:

Original: [Equation]
Revised: [Equation]

Original: To sample data-point pairs ( x i : [ x i, 1 , x i, 2 ] , y i ) and their corresponding ground truth use case labelsu i from g i :• x i, 1 and x i, 2 are sampled uniformly in the ranges [ x ∗ 1 ± 5] and [ x ∗ 2 ± 5] respectively.
Revised: To sample data-point pairs ( x i : [ x i, 1 , x i, 2 ] , y i ) and their corresponding ground truth use case labelsu i from g i : • x i, 1 and x i, 2 are sampled uniformly in the ranges [ x ∗ 1 ± 5] and [ x ∗ 2 ± 5] respectively.

Original:  Generate outcomes
Revised: • Generate outcomes

Original: y i as
Revised: y i as:

Original: [Equation]
Revised: [Equation]

Original: • Generate use case labels 
Revised: • Generate use case labels u

Original: u i using indicator variables 1 ( · ) as:
Revised: i using indicator variables 1 ( · ) as:

Original: [Equation]
Revised: [Equation]

Original: Figure 4 shows the 2D toy dataset generated using critical points x ∗ 1 = 10 , x ∗ 2 = 15 and Z = 1 .
Revised: Figure 4 shows the 2D toy dataset generated using critical points x ∗ 1 = 10 , x ∗ 2 = 15 and Z = 1 .

Original: Notice that because y = g i ( x 1 , x 2 ) + 30 is strictly concave with respect to feature x 2 , the function’soutput increases as x 2 increases for all points where x 2 ≤ x ∗ 2 .
Revised: Notice that because y = g i ( x 1 , x 2 ) + 30 is strictly concave with respect to feature x 2 , the function’soutput increases as x 2 increases for all points where x 2 ≤ x ∗ 2 .

Original: At the critical point of x ∗ 2 , the use caselabels u switch from 1 to 0 : for points where x 2 > x ∗ 2 , then the function’s output decreases as x 2increases.
Revised: At the critical point of x ∗ 2 , the use caselabels u switch from 1 to 0 : for points where x 2 > x ∗ 2 , then the function’s output decreases as x 2increases.

Original: In contrast, when Z = 0 , then data-points y = − g i ( x 1 , x 2 ) + 30 are generated from a function thatis the negation of Figure 4 that is strictly convex with respect to feature x 2 .
Revised: In contrast, when Z = 0 , then data-points y = − g i ( x 1 , x 2 ) + 30 are generated from a function thatis the negation of Figure 4 that is strictly convex with respect to feature x 2 .

Original: Thus the use case labelsare the negation of the use case labels for Z = 1 : they are now 1 when x 2 > x ∗ 2 .
Revised: Thus the use case labelsare the negation of the use case labels for Z = 1 : they are now 1 when x 2 > x ∗ 2 .

Original: D.2.2 Running the Simulated Evaluation
Revised: D.2.2 Running the Simulated Evaluation

Original: Below we detail how predictive models f i and the agent’s train and validation sets are constructed.
Revised: Below we detail how predictive models f i and the agent’s train and validation sets are constructed.

Original: Training predictive models f i .
Revised: Training predictive models f i .

Original: For each observation i , we generate N = 5000 data-points in range[ x ∗ 1 ± 5 , x ∗ 2 ± 5] following Section D.2.1 and train the Light-GBM Regressor f i .
Revised: For each observation i , we generate N = 5000 data-points in range[ x ∗ 1 ± 5 , x ∗ 2 ± 5] following Section D.2.1 and train the Light-GBM Regressor f i .

Original: We validate byvisual inspection that the learned models
Revised: We validate byvisual inspection that the learned models

Original: f i are “smooth”, i.e. that the use case labels u
Revised: f i are “smooth”, i.e. that the use case labels u

Original: i derivedusing the definition in Section D.2.1 are correct by visualizing the model’s output f i ( x i ) over thedomain x i ∈
Revised: i derivedusing the definition in Section D.2.1 are correct by visualizing the model’s output f i ( x i ) over thedomain x i ∈

Original: [ x ∗ 1 ± 5 , x ∗ 2 ± 5] .
Revised: [ x ∗ 1 ± 5 , x ∗ 2 ± 5] .

Original: Sampling observations ( x i , f i ( x i ) , u i ) to give to the agent.
Revised: Sampling observations ( x i , f i ( x i ) , u i ) to give to the agent.

Original: After a predictive model f i has beentrained, we sample a single point x
Revised: After a predictive model f i has beentrained, we sample a single point x

Original: i from the fixed range [11 . 25 , 13 . 75] to construct observation
Revised: i from the fixed range [11 . 25 , 13 . 75] to construct observation

Original: i forthe agent.
Revised: i forthe agent.

Original: While this range is a subset of the range in which the critical points x ∗ 1 , x ∗ 2 and the trainingdata points were sampled, we intentionally sample the input points x i in a more narrow range toincrease the difficulty of the agent prediction task.
Revised: While this range is a subset of the range in which the critical points x ∗ 1 , x ∗ 2 and the trainingdata points were sampled, we intentionally sample the input points x i in a more narrow range toincrease the difficulty of the agent prediction task.

Original: Specifically, for the chosen range, the probabilitythat the randomly selected critical point x ∗ 2 lies outside of this range is 50% , making it difficult forthe agent to easily infer the use case label from the model prediction
Revised: Specifically, for the chosen range, the probabilitythat the randomly selected critical point x ∗ 2 lies outside of this range is 50% , making it difficult forthe agent to easily infer the use case label from the model prediction

Original: f i ( x i ) and covariates x i alone.
Revised: f i ( x i ) and covariates x i alone.

Original: Repeating this process several times, we construct a dataset of 10 , 000 different observations (e.g. wetrain 10 , 000 different models
Revised: Repeating this process several times, we construct a dataset of 10 , 000 different observations (e.g. wetrain 10 , 000 different models

Original: f
Revised: f

Original: i corresponding to 10 , 000 different saddle-point functions g i ).
Revised: i corresponding to 10 , 000 different saddle-point functions g i ).

Original: Forthe 4 different explanation settings, we use each observation’s respective trained predictive model f ito generate explanations.
Revised: Forthe 4 different explanation settings, we use each observation’s respective trained predictive model f ito generate explanations.

Original: To train the agent, we split these 10 , 000 observations into a train, test, andvalidation set.
Revised: To train the agent, we split these 10 , 000 observations into a train, test, andvalidation set.

Original: D.3 Limitations & Future Work
Revised: D.3 Limitations & Future Work

Original: We note that there are a few limitations to our use case definition and evaluation.
Revised: We note that there are a few limitations to our use case definition and evaluation.

Original: First, our use casedefinition does not account for the feasibility or cost of increasing the specific feature under study,which may be of practical importance in a real-world application.
Revised: First, our use casedefinition does not account for the feasibility or cost of increasing the specific feature under study,which may be of practical importance in a real-world application.

Original: Second, we note that it may bedifficult or subjective to derive ground-truth use case labels of whether or not a predictive model’soutcome will increase locally if a feature is increased for a non-convex prediction model f .
Revised: Second, we note that it may bedifficult or subjective to derive ground-truth use case labels of whether or not a predictive model’soutcome will increase locally if a feature is increased for a non-convex prediction model f .

Original: Thenaive approach of taking the gradient of f at point x may not be representative of the behavior offunction f in a region around x [38].
Revised: Thenaive approach of taking the gradient of f at point x may not be representative of the behavior offunction f in a region around x [40].

Original: In practice, we recommend that ground-truth use case labelsare defined empirically by averaging the model’s predictions in a local region around input x usingexpert knowledge to define a meaningful and representative local region.
Revised: In practice, we recommend that ground-truth use case labels are defined empirically by averaging the model’s predictions in a local region around input x usingexpert knowledge to define a meaningful and representative local region.

Original: Third, we note that to ourknowledge, there is no explanation method that is explicitly developed to address this use case, whichis an open direction for future work.
Revised: Third, we note that to ourknowledge, there is no explanation method that is explicitly developed to address this use case, whichis an open direction for future work.

Original: E Explanation EncodingE.1 Global vs. Local
Revised: E Explanation EncodingE.1 Global vs. Local

Original: The number of data-points included in each observation can vary depending on the explanation type(global vs. local explanations) and problem type (global vs. local problems).
Revised: The number of data-points included in each observation can vary depending on the explanation type(global vs. local explanations) and problem type (global vs. local problems).

Original: Table 4 summarizeshow we recommend constructing each observation for each combination of problem and explanationtype.
Revised: Table 4 summarizeshow we recommend constructing each observation for each combination of problem and explanationtype.

Original: We elaborate on several of our choices below.
Revised: We elaborate on several of our choices below.

Original: [Table]
Revised: [Table]

Original: • Global Problem + Global Explanation: As discussed in Section 3, global problemsrequire that each observation uses a different base predictive model
Revised: • Global Problem + Global Explanation: As discussed in Section 3, global problems require that each observation uses a different base predictive model

Original: f i because the use caseu i labels are a property of the model (and thus would be the same for all observations
Revised: f i because the use caseu i labels are a property of the model (and thus would be the same for all observations

Original: if they
Revised: if they

Original: [Table]

Original: all shared the same model).
Revised: all shared the same model).

Original: Therefore each observation i contains the global explanation forthat model f i .•
Revised: Therefore each observation i contains the global explanation for that model f i .

Original:  Global Problem + Local Explanation: Global problems require that each observationuses a different predictive model, but local explanations only explain the model’s predictionon a single data-point.
Revised: • Global Problem + Local Explanation: Global problems require that each observation [Table] uses a different predictive model, but local explanations only explain the model’s prediction on a single data-point.

Original: A single data-point may not contain enough information to infer aglobal property of predictive model f i .
Revised: A single data-point may not contain enough information to infer a global property of predictive model f i .

Original: To address this problem, the agent can take multiple(e.g.
Revised: To address this problem, the agent can take multiple

Original:  a set of S ) data-points and explanations per observation.• Local Problem + Global Explanation: A local problem defines a different use-case labelfor each individual data-point x i .
Revised: (e.g. a set of S ) data-points and explanations per observation. • Local Problem + Global Explanation: A local problem defines a different use-case labelfor each individual data-point x i .

Original: However, each base model f and base dataset D onlyhas 1 global explanation E ( f, D ) .
Revised: However, each base model f and base dataset D only has 1 global explanation E ( f, D ) .

Original: As such, if the same model f and dataset D are used forall observations i (as in the Forward Simulation use case introduced in Section ?? ), thenthe global explanation E ( f, D ) will be constant across all observations i .
Revised: As such, if the same model f and dataset D are used for all observations i (as in the Forward Simulation use case introduced in the main text, then the global explanation E ( f, D ) will be constant across all observations i .

Original: We note that a
Revised: We note that a

Original: [Table]

Original: researcher can choose to vary the predictive model f i and dataset D i across observationsi (as in the Counterfactual Reasoning use case introduced in Section ?? ) ; in this case the
Revised: researcher can choose to vary the predictive model f i and dataset D i across observations i (as in the Counterfactual Reasoning use case introduced in the main text; in this case the

Original: global explanation E ( f i , D i ) will vary across observations.
Revised: global explanation E ( f i , D i ) will vary across observations.

Original: • Local Problem + Local Explanation: Like the previous example, the predictive base
Revised: • Local Problem + Local Explanation: Like the previous example, the predictive base

Original: model can vary or be held constant across different observations i .
Revised: model can vary or be held constant across different observations i .

Original: Each observation contains
Revised: Each observation contains

Original: the base model’s explanation on sampled point x i .
Revised: the base model’s explanation on sampled point x i .

Original: E.2 Encoding explanations in our experiments
Revised: E.2 Encoding explanations in our experiments

Original: The explanations we consider in our experiment are LIME, SHAP, Anchors, and GAM:
Revised: The explanations we consider in our experiment are LIME, SHAP, Anchors, and GAM:

Original: • LIME and SHAP are both local explanations that return an importance score for each
Revised: • LIME and SHAP are both local explanations that return an importance score for each

Original: feature.
Revised: feature.

Original: We encode each explanation as a vector of length d if x ∈ R d and concatenate
Revised: We encode each explanation as a vector of length d if x ∈ R d and concatenate

Original: the explanation to the data-point to create an observation vector of length 2 d .
Revised: the explanation to the data-point to create an observation vector of length 2 d .

Original: Note that inForward Simulation, to recreate the original setting, we extend the vector of length d with
Revised: Note that in Forward Simulation, to recreate the original setting, we extend the vector of length d with

Original: additional information from the LIME package as discussed in Appendix G.1.
Revised: additional information from the LIME package as discussed in Appendix G.1.

Original: • To get the GAM explanation, we used method explain_local out-of-the-box from the
Revised: • To get the GAM explanation, we used method explain_local out-of-the-box from the

Original: InterpretML package [30].
Revised: InterpretML package [32].

Original: • To get the Anchors explanation, we call the Anchors implementation [36] which returns a set of anchors for a given data-point.
Revised: • To get the Anchors explanation, we call the Anchors implementation [38] which returns aset of anchors for a given data-point.

Original: For categorical features, the anchor is one of the values that the feature can take on.
Revised: For categorical features, the anchor is one of the valuesthat the feature can take on.

Original: For continuous features, the anchor can take the form of a lowerand upper bound.
Revised: For continuous features, the anchor can take the form of a lowerand upper bound.

Original: We include both the upper and lower bound for each continuous feature inthe observation.
Revised: We include both the upper and lower bound for each continuous feature inthe observation.

Original: For a data-point x with d cat categorical and d cont continuous features, thefinal observation has dimension at most 2 d cat + 3 d cont .
Revised: For a data-point x with d cat categorical and d cont continuous features, thefinal observation has dimension at most 2 d cat + 3 d cont .

Original: E.3 Encoding other kinds of explanations
Revised: E.3 Encoding other kinds of explanations

Original: There are other types of explanations that we do not consider in this work, but could also be studiedusing our proposed framework.
Revised: There are other types of explanations that we do not consider in this work but could also be studiedusing our proposed framework.

Original: We discuss some potential ways to encode these other types ofexplanations:
Revised: We discuss some potential ways to encode these other types ofexplanations:

Original: • Saliency maps: These types of explanations provide a “heat map" over an image input,where larger values signify a larger contribution of that pixel to the model’s prediction.
Revised: • Saliency maps: These types of explanations provide a “heat map" over an image input,where larger values signify a larger contribution of that pixel to the model’s prediction.

Original: Suppose the dimensions of the input image are d × d × c and the dimensions of the saliencymap are d × d × k , one could define the observation as a d × d × c + k input and instantiatethe agent as a convolutional network.
Revised: Suppose the dimensions of the input image are d × d × c and the dimensions of the saliencymap are d × d × k , one could define the observation as a d × d × c + k input and instantiate the agent as a convolutional network.

Original: In Appendix I, we use this set up to replicate resultsfor an image user study.
Revised: In Appendix I, we use this set up to replicate results for an image user study.

Original: Note that these types of explanations will most likely require aneural network-based agent architecture.• Counterfactual explanations: These types of explanations provide the closest possible pointthat achieves the desired outcome for a given point.
Revised: Note that these types of explanations will most likely require a [Table] neural network-based agent architecture. • Counterfactual explanations: These types of explanations provide the closest possible point

Revised: [Table]

Revised: that achieves the desired outcome for a given point.

Original: Suppose the data-points come from R d ,then one possible encoding of the input observation is concatenating the two points into a2 d vector.• Decision Sets: Suppose we are trying to encode an interpretable decision set [26], whereeach rule is written in if-else form and the data-points come from R d .
Revised: Suppose the data-points come from R d , then one possible encoding of the input observation is concatenating the two points into a 2 d vector. • Decision Sets: Suppose we are trying to encode an interpretable decision set [28], whereeach rule is written in if-else form and the data-points come from R d .

Original: One possible encodingof each rule is via a 2 d vector where each feature has 2 slots (one for the equivalenceoperator and one for the value) and the slots would be filled in accordingly if the featureappeared in the rule.
Revised: One possible encodingof each rule is via a 2 d vector where each feature has 2 slots (one for the equivalence [Table] operator and one for the value) and the slots would be filled in accordingly if the feature [Table]

Original:  This way, if the interpretable decision set consists of k rules, then thefinal encoding would be of the size k × 3 d .
Revised: appeared in the rule This way, if the interpretable decision set consists of k rules, then thefinal encoding would be of the size k × 3 d .

Original: F Experimental DetailsF.1 Dataset preprocessing
Revised: F Experimental DetailsF.1 Dataset preprocessing

Original: We used the UCI Adult (“adult”) dataset as found here [11].
Revised: For forward simulation and data bugs, we used the UCI Adult (“adult”) dataset as done in [20, 16](which is linked: here [12]).

Original: The data was extracted by Barry Beckerfrom the 1994 Census database.
Revised: The data was extracted by Barry Becker from the 1994 Census database.

Original: The citizens of the United States consent to having their data includedin the Census.
Revised: The citizens of the United States consent to having their data included in the Census.

Original: According to the U.S. Census Bureau, the United States/Commerce grants users aroyalty-free, nonexclusive license to use, copy, and create derivative works of the Software.
Revised: Accordingto the U.S. Census Bureau, the United States/Commerce grants users a royalty-free, nonexclusivelicense to use, copy, and create derivative works of the Software.

Original: In the adult dataset, we dropped the finalweight feature.
Revised: In the adult dataset, we dropped thefinalweight feature.

Original: For all datasets, we one-hot encoded thecategorical features and MinMax scaled all features.
Revised: For all datasets, we one-hot encoded the categorical features and MinMaxscaled all features.

Revised: For counterfactual reasoning, we construct a 2 D synthetic regression dataset inspired by Kumar etal.

Revised: We define the use case label as 1 if increasing a data-point’s specific feature x 2 increasesthe regression model prediction f ( x ) (holding the other feature x 1 constant), 0 otherwise.

Revised: The datageneration and other use case details are described extensively in Appendix D. In the baseline setting,the agent/human receives no explanation.

Revised: We evaluate the same local explanations as previous usecases (Section 5).

Revised: We use a Light-GBM Regressor [21] as the base prediction model class for allexplanation settings except GAM, in which the GAM is the base model.

Original: F.2 Agent architecture
Revised: F.2 Agent architecture

Original: We use the same Deepset architecture [44] for all three use cases (though, note that when the input isof set size 1 in the forward simulation and counterfactual reasoning use cases, Deepsets is effectivelya standard feedfoward network).
Revised: We use the same Deepset architecture [46] for all three use cases (though, note that when the input isof set size 1 in the forward simulation and counterfactual reasoning use cases, Deepsets is effectivelya standard feedfoward network).

Original: The Deepset architecture is characterized by two parts: (1) thefeature extractor ϕ , which encodes each item in the set, and (2) the standard network ρ , which takesin a summed representation of all of the representations from ϕ .
Revised: The Deepset architecture is characterized by two parts: (1) thefeature extractor ϕ , which encodes each item in the set, and (2) the standard network ρ , which takesin a summed representation of all of the representations from ϕ .

Original: The feature extractor ϕ that we usehas a sequence of Dense layers of sizes ( m , where m is the shape of each observation; 200; 100)respectively, where each layer is followed by an ELU activation with the exception of the last layer.
Revised: The feature extractor ϕ that we use has a sequence of Dense layers of sizes ( m , where m is the shape of each observation; 200; 100)respectively, where each layer is followed by an ELU activation with the exception of the last layer.

Original: The ρ network is also a sequence of Dense layers of size (100, 30, 30, 10), each followed by an ELUactivation with the exception of the last layer which is followed by a Sigmoid activation.
Revised: The ρ network is also a sequence of Dense layers of size (100, 30, 30, 10), each followed by an ELUactivation with the exception of the last layer which is followed by a Sigmoid activation.

Original: F.3 Missing Values Adjustment
Revised: F.3 Missing Values Adjustment

Original:  Figure 5 and Figure 6 show our attempt to recreate the original bug [18] as closely as possible.
Revised: [Figure] Figure 5 and Figure 6 show our attempt to recreate the original bug [20] as closely as possible.

Original: Theauthors state that they apply the bug to 10% of data-points.
Revised: Theauthors state that they apply the bug to 10% of data-points.

Original: However, we observe that the 30% bug forboth explanation methods much more closely resembles the figure that was presented in the originalpaper.
Revised: However, we observe that the 30% bug forboth explanation methods much more closely resembles the figure that was presented in the originalpaper.

Original: F.4 Base Models
Revised: F.4 Base Models

Original: Following Kaur et al.
Revised: Following Kaur et al.

Original: [18], we evaluate both glass-box models that are designed to be interpretable(e.g., GAMs) and post-hoc explanation methods for black-box models (e.g., LIME, SHAP, Anchors).
Revised: [20], we evaluate both glass-box models that are designed to be interpretable(e.g., GAMs) and post-hoc explanation methods for black-box models (e.g., LIME, SHAP, Anchors).

Original: Since GAM is glass-box, the GAM from which local explanations are derived is the base predictivemodel.
Revised: Since GAM is glass-box, the GAM from which local explanations are derived is the base predictivemodel.

Original: On the other hand, LIME, SHAP, and Anchors are model-agnostic post-hoc explanationtechniques, so they can be used to explain any predictive base model class that is compatible withtheir Python package implementations.
Revised: On the other hand, LIME, SHAP, and Anchors are model-agnostic post-hoc explanationtechniques, so they can be used to explain any predictive base model class that is compatible withtheir Python package implementations.

Original: This means that we need to additionally specify the basemodel family for post-hoc explanation techniques.
Revised: This means that we need to additionally specify the basemodel family for post-hoc explanation techniques.

Original: Ideally, we would have used the same underlying base predictive model for all explanation settings.
Revised: Ideally, we would have used the same underlying base predictive model for all explanation settings.

Original: When the underlying base model is controlled and held constant, then differences in the agent’saccuracy across explanation settings can be attributed to differences in the explanation methods - thevariable we’re interested in studying - instead of differences in the underlying base models.
Revised: When the underlying base model is controlled and held constant, then differences in the agent’saccuracy across explanation settings can be attributed to differences in the explanation methods - thevariable we’re interested in studying - instead of differences in the underlying base models.

Original: However, we encountered the same issue as Kaur et al.
Revised: However, we encountered the same issue as Kaur et al.

Original: [18]: the InterpretML GAM implementationis not supported by the SHAP Python library; thus we could not generate SHAP explanations fora GAM predictive model.
Revised: [20]: the InterpretML GAM implementationis not supported by the SHAP Python library; thus we could not generate SHAP explanations for a GAM predictive model.

Original: As a result, we also chose to use a different base predictive model classfor the 3 post-hoc explanation settings.
Revised: As a result, we also chose to use a different base predictive model classfor the 3 post-hoc explanation settings.

Original: We emphasize that the comparisons that we make betweenexplanation settings are particular to the base predictive model classes chosen: e.g. between the
Revised: We emphasize that the comparisons that we make betweenexplanation settings are particular to the base predictive model classes chosen: e.g. between the

Original: InterpretML implementation of a GAM and the SHAP Python package used to explain a LightGBMmodel.
Revised: InterpretML implementation of a GAM and the SHAP Python package used to explain a LightGBMmodel.

Original: F.5 Framework Hyperparameters
Revised: F.5 Framework Hyperparameters

Original: For all use cases, we generate an observation dataset of up to size 1000 and evaluate on 250 newobservations.
Revised: For all use cases, we generate an observation dataset of up to size 1000 and evaluate on 250 newobservations.

Original: In our experiments we train our agent with the Adam optimizer with learning rate10 − 4 for our optimizer (and weight decay 10 − 4 for counterfactual reasoning), weight decay ofbinary cross-entropy for the loss function, and batch size of 16 .
Revised: In our experiments we train our agent with the Adam optimizer with learning rate10 − 4 for our optimizer (and weight decay 10 − 4 for counterfactual reasoning), weight decay ofbinary cross-entropy for the loss function, and batch size of 16 .

Original: We train for 350 epochs for forwardsimulation and data bugs and 1000 epochs for counterfactual reasoning.
Revised: We train for 350 epochs for forwardsimulation and data bugs and 1000 epochs for counterfactual reasoning.

Original: F.6 Computational Resources
Revised: F.6 Computational Resources

Original: In general, training an agent using the hyperparameters specified above does not require a GPU,making SimEvals on particularly tabular datasets very accessible.
Revised: In general, training an agent using the hyperparameters specified above does not require a GPU,making SimEvals on particularly tabular datasets very accessible.

Original: When scaling up the data complexity, for example considering image-based data, it may be necessary to use GPUs to train both theprediction model and the agent model.
Revised: When scaling up the data complexity, for example considering image-based data, it may be necessary to use GPUs to train both theprediction model and the agent model.

Original: G Full results
Revised: G Full results

Original: We include a more complete set of experimental results for all three use cases.
Revised: We include a more complete set of experimental results for all three use cases.

Original: Average agentvalidation set accuracies (percentages) and their standard error are reported over 3 train-test splits.
Revised: Average agentvalidation set accuracies (percentages) and their standard error are reported over 3 train-test splits.

Original: G.1 Forward Simulation
Revised: G.1 Forward Simulation

Original: In Hase and Bansal [14] [Table] we noticed that the LIME explanation provided included (a) the LIMEapproximation model’s weights, (b) the approximation model’s intercept, (c) the sum of the approximation model’s weights, and (d) the approximation model’s prediction on the point.
Revised: In Hase and Bansal [16], we noticed that the LIME explanation provided included (a) the LIMEapproximation model’s weights, (b) the approximation model’s intercept, (c) the sum of the approximation model’s weights, and (d) the approximation model’s prediction on the point.

Original: We also includeresults for a setting where the agent is given the LIME approximation model’s weights only.
Revised: We also includeresults for a setting where the agent is given the LIME approximation model’s weights only.

Original: Whilethe agent’s accuracy in this setting is lower, it still outperforms the baseline setting.
Revised: Whilethe agent’s accuracy in this setting is lower, it still outperforms the baseline setting.

Revised: varying the training observations the agent receives to perform forward simulation.

Original: [Table]
Revised: [Table]

Revised: We note that for forward simulation, there exist known heuristics to predict the model’s outputusing some explanation methods (i.e., it is possible for a human or agent to achieve (near) perfectaccuracy).

Revised: For example, SHAP feature attributions by definition sum to the model prediction [29].

Revised: With sufficient examples ( N > 1000 ), the predictiveness of SHAP explanations for this use case isdemonstrated by the agent’s near perfect accuracy.

Revised: For completeness we provide the p -values from Hase and Bansal [16] which are: LIME (0.014) and

Revised: Anchors (0.234).

Revised: Their statistical analysis only compares each explanation to the baseline of noexplanation, so the p -values provided signify whether the difference in test accuracy of the humanwhen given an explanation versus not is statistically significant.

Original: G.2 Counterfactual Reasoning
Revised: G.2 Counterfactual Reasoning

Revised: We provide results for the counterfactual reasoning use case for SimEvals as well as our MTurkstudy.

Revised: We find that SimEvals performs well using the LIME explanation which is what we observein human performance as well.

Original: [Table]
Revised: [Table]

Original: reasoning.

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: G.3 Missing Values
Revised: G.3 Missing Values

Revised: We provide results for the missing values data bug use case for SimEvals as well as our MTurkstudy.

Revised: We find that SimEvals performs well using the SHAP and GAM explanation which is whatwe observe in human performance as well.

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: G.4 Redundant Features
Revised: G.4 Redundant Features

Original: [Table]

Original: To create a setting where there is no bug, we randomize the value of one of the features so there is nocorrelation between the two and only one feature contains the original information.
Revised: To create a setting where there is no bug, we randomize the value of one of the features so there is nocorrelation between the two and only one feature contains the original information.

Original: Note that, in theoriginal user study, the authors do not vary the strength of the bug, but we are easily able to study thisvariant of the bug with our algorithmic framework.
Revised: Note that, in theoriginal user study, the authors do not vary the strength of the bug, but we are easily able to study thisvariant of the bug with our algorithmic framework.

Original: We increase the strength of the bug by increasingthe number of data-points in dataset D i that have the two correlated features.
Revised: We increase the strength of the bug by increasingthe number of data-points in dataset D i that have the two correlated features.

Original: The agent’s accuracy was near random guessing when given LIME explanations or the modelprediction baseline, suggesting that neither would be helpful for a human user.
Revised: The agent’s accuracy was near random guessing when given LIME explanations or the modelprediction baseline, suggesting that neither would be helpful for a human user.

Original: [Table]
Revised: [Table]

Original: G.5 Ablation results

Original: Single data-point observation: Counterfactual reasoning is one of the use cases where the agent ispresented with a single data-point (and explanation).
Revised: Single data-point observation: Counterfactual reasoning is one of the use cases where the agent ispresented with a single data-point (and explanation).

Original: In the main text, we presented results using the
Revised: In the main text, we presented results using the

Original: DeepSet architecture.
Revised: DeepSet architecture.

Original: Here, we swap out DeepSet with a non-neural network architecture, LightGBM.
Revised: Here, we swap out DeepSet with a non-neural network architecture, LightGBM.

Original: We find that the rankings of the explanations are consistent (e.g., compare Table 6 and Table 10).
Revised: We find that the rankings of the explanations are consistent (e.g., compare Table 6 and Table 14).

Original: We find that the rankings are still consistent with the DeepSet results.

Original: [Table]
Revised: [Table]

Original: Multiple data-point observation.
Revised: Multiple data-point observation.

Original: Data bugs is a use case where the user receives multiple datapoints as the observation.
Revised: Data bugs is a use case where the user receives multiple datapoints as the observation.

Original: We do not include these results, but we find low accuracy when usingnon-neural network models like LightGBM.
Revised: We do not include these results, but we find low accuracy when usingnon-neural network models like LightGBM.

Original: However, we find similar results when ablating theneural network architecture.
Revised: However, we find similar results when ablating theneural network architecture.

Original: For example, instead of summing the representations (as is done in
Revised: For example, instead of summing the representations (as is done in

Original: DeepSet), we instead concatenate representations.
Revised: DeepSet), we instead concatenate representations.

Original: We find comparable results (with the caveat that one might need to scale up the number of training points in the dataset).
Revised: We find comparable results (with the caveat thatone might need to scale up the number of training points in the dataset).

Original: For example, for SHAP withset size 100, one would need 4.5x larger dataset to achieve the same accuracy as the original Deepsetmodel (99.8%): 1x data (57.9%), 1.5x data (78.9%), 2.5x data (92.3%), 3.5x data (95.6%), 4.5x data(99.5%).
Revised: For example, for SHAP withset size 100, one would need 4.5x larger dataset to achieve the same accuracy as the original Deepsetmodel (99.8%): 1x data (57.9%), 1.5x data (78.9%), 2.5x data (92.3%), 3.5x data (95.6%), 4.5x data(99.5%).

Original: This is because concatenating representations drastically increases the model parametersize, particularly for larger set sizes.
Revised: This is because concatenating representations drastically increases the model parametersize, particularly for larger set sizes.

Original: We also find that the agent architecture is not drastically affectedby adding/removing a layer (Table ?? vs Table 11)
Revised: We also find that the agent architecture is not drastically affectedby adding/removing a layer (Table 9 vs Table 3)

Original: [Table]
Revised: [Table]

Original: H MTurk Details
Revised: H MTurk DetailsH.1 Participants.

Original: [Table] H.1 Participants.

Original: A total of 80 participants were recruited to complete the task via Amazon MTurk in 4 batches (20for each explanation setting).
Revised: A total of 80 participants were recruited to complete the task via Amazon MTurk in 4 batches (20for each explanation setting).

Original: We controlled the quality of the Turkers by limiting participation inour MTurk HIT adults located in an English-speaking countries (specifically the United States and
Revised: We controlled the quality of the Turkers by limiting participation inour MTurk HIT adults located in an English-speaking countries (specifically the United States and

Original: Canada) with greater than a 97% HIT approval rate for quality control.
Revised: Canada) with greater than a 97% HIT approval rate for quality control.

Original: Each worker was only allowedto complete the study 1 time.
Revised: Each worker was only allowedto complete the study 1 time.

Original: All participants were retained for final analysis.
Revised: All participants were retained for final analysis.

Original: The estimated time tocomplete the study was 15 minutes.
Revised: The estimated time tocomplete the study was 15 minutes.

Original: The study took 21 minutes on average (however, this includesany pauses/breaks the Turker might have taken between the two portions of the task).
Revised: The study took 21 minutes on average (however, this includesany pauses/breaks the Turker might have taken between the two portions of the task).

Original: Each workerwas compensated 2.50 USD for an estimated hourly wage of 10 USD.
Revised: Each workerwas compensated 2.50 USD for an estimated hourly wage of 10 USD.

Original: Our study was approved by the IRB (STUDY2021_00000286) for exempt Category 3 research.
Revised: Our study was approved by the IRB (STUDY2021_00000286) for exempt Category 3 research.

Original: Wedid not anticipate any potential participant risks as we described in the IRB: “The risks and discomfortassociated with participating in this study are no greater than those ordinarily encountered in dailylife or operation of a smartphone or laptop, such as boredom or fatigue due to the length of thequestionnaires or discomfort with the software.”
Revised: Wedid not anticipate any potential participant risks as we described in the IRB: “The risks and discomfortassociated with participating in this study are no greater than those ordinarily encountered in dailylife or operation of a smartphone or laptop, such as boredom or fatigue due to the length of thequestionnaires or discomfort with the software.”

Original: H.2 Study Interface.
Revised: H.2 Study Context.

Revised: Counterfactual Reasoning.

Revised: We introduce the Turkers to the use case using the context of furniturepricing: Turkers are shown data about a furniture item (height and length measurements x i and the item’s price f ( x i ) ) and are asked to perform counterfactual reasoning by choosing whether or notincreasing the item’s length would increase its price (a screenshot of the interface is shown in Figure8).

Original: Participants were first presented with brief information about  the study and an informed consent form.
Revised: The Turkers are instructed to set aside any prior knowledge they may have about furniture pricingand only use the information provided during the study. Data Bugs. We introduce the Turkers to the use case using the context of identifying “bugs” orproblems in an Artificial Intelligence system which is used to make predictions about individualincomes. Turkers are shown scatter plots that follow the presentation from the original user studyconducted by Kaur et al.

Revised: [20], reflecting how the system is making predictions along each attribute.

Revised: The user is provided a demo showing that, for example, as the amount of education a person receivesincreases, the system deems that attribute to be more important to income prediction.

Revised: The participantsare then asked to determine whether the system contains a “bugs” and learn from the feedback.

Revised: Ascreenshot of the interface is shown in Figure 10.

Revised: H.3 Study Interface.

Revised: Participants were first presented with brief information about the study and an informed consent form.

Original: To complete the HIT, participants were instructed to finish both parts (corresponding to a “Train” and“Test” phase) of the HIT via external URLs to Qualtrics surveys.
Revised: To complete the HIT, participants were instructed to finish both parts (corresponding to a “Train” and“Test” phase) of the HIT via external URLs to Qualtrics surveys.

Original: Both phases of the MTurk evaluationtask described in the main text were conducted entirely on Qualtrics.
Revised: Both phases of the MTurk evaluationtask described in the main text were conducted entirely on Qualtrics.

Original: In both phases of the study, participants are presented with a series of furniture items, and areinstructed to answer survey questions where they predict if increasing the item’s length will increaseits price.
Revised: In both phases of the study, participants are presented with a series of furniture items, and areinstructed to answer survey questions where they predict if increasing the item’s length will increaseits price.

Original: Figure 8 illustrates how the information shown to the user varies across different explanationsettings.
Revised: Figure 8 illustrates how the information shown to the user varies across different explanationsettings.

Original: After opening the Train survey, participants are presented with instructions about the counterfactualreasoning task (Figure 7).
Revised: After opening the Train survey, participants are presented with instructions about the counterfactualreasoning task (Figure 7).

Original: After reading the instructions, participants begin the Train phase of thestudy, where they receive feedback after submitting their responses (Figure 9).
Revised: After reading the instructions, participants begin the Train phase of thestudy, where they receive feedback after submitting their responses (Figure 9).

Original: Participants thencomplete the Test phase of the study, where they are not given any feedback after submitting theirresponses (but are presented with an otherwise identical interface to the Train phase).
Revised: Participants thencomplete the Test phase of the study, where they are not given any feedback after submitting theirresponses (but are presented with an otherwise identical interface to the Train phase).

Original: For each question, we recorded the participants’ response and the time the participant took to respond.
Revised: For each question, we recorded the participants’ response and the time the participant took to respond.

Original: At the end of the Test phase, we asked participants to describe the strategy that they used to answerthe questions.
Revised: At the end of the Test phase, we asked participants to describe the strategy that they used to answerthe questions.

Original: We also included an attention question where we asked the Turkers to list the twomeasurements provided.
Revised: We also included an attention question where we asked the Turkers to list the twomeasurements provided.

Original: We found that all Turkers either fully answered this question correctly (bysaying Height and Length) or gave a response that was reasonable (by saying Price and Length).
Revised: We found that all Turkers either fully answered this question correctly (bysaying Height and Length) or gave a response that was reasonable (by saying Price and Length).

Original: Each participant is assigned to 1 explanation setting for the of the study, andso willonly ever see 1 of these possible interfaces.
Revised: participant is assigned to 1 explanation setting for the of the study, and will ever see 1 of these possible interfaces.

Original: H.3 MTurk Full Results

Original: [Table]

Original: H.4 Statistical testing details

Original: We run an ANOVA test for LIME versus each other experimental setting (SHAP, GAM, and Noexplanation) to determine if the higher accuracy observed for the LIME condition is statisticallysignificant.

Original: We find that LIME vs. SHAP has p = 0 . 0007 , LIME vs GAM has p = 0 . 008 , and LIME vs. Noexplanation has p = 0 .

Original: Since all three p < 0 .

Original: 05 , we conclude that Turkers achieve significantlyhigher accuracy when given LIME explanations in comparison to all other explanation settings.

Original: We also provide 95% confidence intervals for the test accuracies: LIME 69.4% ± 12.4%, SHAP:41.4% ± 6.14%, GAM: 45.7% ± 7.19%, No explanation: 48.6% ± 5.61%.

Original: The CI still aligns with

Original: SimEval conclusions: Turkers were able to use LIME to get more correct predictions than any othersetting.

Original: H.5 Pilot Studies
Revised: H.4 Pilot Studies

Original: We conducted several pilot human subject studies that informed the design of our final MTurk study.
Revised: We conducted several pilot human subject studies that informed the design of our final MTurk study.

Original: Our pilot studies showed that subjects struggled to complete the counterfactual reasoning task whenpresented with data x with a 3 or more features, likely due to information overload [33].
Revised: Our pilot studies showed that subjects struggled to complete the counterfactual reasoning task whenpresented with data x with a 3 or more features, likely due to information overload [35].

Original: Thus wechose to construct a 2D dataset for our final study.
Revised: Thus wechose to construct a 2D dataset for our final study.

Original: Our pilot studies also showed
Revised: Our pilot studies also showed

Original: that instructions presented to subjects influenced the strategy that theyused to respond to the questions.
Revised: that instructions presented to subjects influenced the strategy that theyused to respond to the questions.

Original: Particularly, we noticed that the way we defined importance scoresinfluenced how participants used the scores when answering each question.
Revised: Particularly, we noticed that the way we defined importance scoresinfluenced how participants used the scores when answering each question.

Original: In an initial trial, weprovided participants with the following more detailed definition of importance scores:
Revised: In an initial trial, weprovided participants with the following more detailed definition of importance scores:

Original: • A positive importance score means that the measurement’s valuecontributes to the furniture item having a high price.•
Revised: • A positive importance score means that the measurement’s value [Figure] contributes to the furniture item having a high price.

Revised: [Figure]

Original:  A negative importance score means that the measurement’s valuecontributes to the furniture item having a low price.•
Revised: • A negative importance score means that the measurement’s value [Figure] contributes to the furniture item having a low price.

Revised: [Figure]

Original:  An importance score with a large magnitude (absolute value)means that the measurement’s value has a larger contribution todetermining the item’s price.
Revised: • An importance score with a large magnitude (absolute value)means that the measurement’s value has a larger contribution todetermining the item’s price.

Original: However when we reviewed the participants’ self-reported descriptions of the strategy they used,several participants reported that they “used the strategy provided by the instructions” rather thanlearn from the feedback given during the Train phase.
Revised: However when we reviewed the participants’ self-reported descriptions of the strategy they used,several participants reported that they “used the strategy provided by the instructions” rather thanlearn from the feedback given during the Train phase.

Original: While the given instructions do not directlystate a strategy for how the importance scores should be used for the task when literally interpreted,the instructions can be misinterpreted to mean that a positive length importance score implies thatthe item’s price will increase when its length is increased.
Revised: While the given instructions do not directlystate a strategy for how the importance scores should be used for the task when literally interpreted,the instructions can be misinterpreted to mean that a positive length importance score implies thatthe item’s price will increase when its length is increased.

Original: Indeed, this is the strategy that manyparticipants used when given the above instructions.
Revised: Indeed, this is the strategy that manyparticipants used when given the above instructions.

Original: As such, we decided to provide a more minimaldescription of importance scores in the final study (shown in Figure 7) to encourage participants tolearn their own strategy using the information provided as opposed to simply following the giveninstructions.
Revised: As such, we decided to provide a more minimaldescription of importance scores in the final study (shown in Figure 7) to encourage participants to learn their own strategy using the information provided as opposed to simply following the giveninstructions.

Original: I Image Experiment
Revised: I Image Experiment

Original: We also conducted SimEvals for a user study which evaluated saliency maps for model debugging [2].
Revised: We also conducted SimEvals for a user study which evaluated saliency maps for model debugging [2].

Original: We provide this example primarily to illustrate how to perform SimEvals on saliency maps andchose not to include these experiments in the main text because there were no positive results forsaliency maps (e.g., providing the model prediction alone was enough to detect the bug).
Revised: We provide this example primarily to illustrate how to perform SimEvals on saliency maps andchose not to include these experiments in the main text because there were no positive results forsaliency maps (e.g., providing the model prediction alone was enough to detect the bug).

Original: The basedataset for this user study was a dog breed dataset (the image label is the breed of the dog).
Revised: The basedataset for this user study was a dog breed dataset (the image label is the breed of the dog).

Original: Theauthors conducted a user study on 4 different bugs: out-of-distribution input, label errors, spuriouscorrelations, and top layer randomization.
Revised: Theauthors conducted a user study on 4 different bugs: out-of-distribution input, label errors, spuriouscorrelations, and top layer randomization.

Original: The authors find that for all of the use cases exceptfor spurious correlations, users relied on the model prediction to detect the bug as opposed to theexplanation .
Revised: The authors find that for all of the use cases exceptfor spurious correlations, users relied on the model prediction to detect the bug as opposed to theexplanation .

Original: Unfortunately, due to the lack of available code, we were unable to recreate the samespurious correlations that were studied in the paper.
Revised: Unfortunately, due to the lack of available code, we were unable to recreate the samespurious correlations that were studied in the paper.

Original: We trained SimEvals on the label error bug.
Revised: We trained SimEvals on the label error bug.

Original: We followed a similar data generation process as inthe Data Bugs use cases described in the main text and introduced label errors by randomizing thelabels before the prediction model was trained and attempt to recreate the prediction model describedin the paper.
Revised: We followed a similar data generation process as inthe Data Bugs use cases described in the main text and introduced label errors by randomizing thelabels before the prediction model was trained and attempt to recreate the prediction model describedin the paper.

Original: We instantiated the agent architecture using a convolutional Deepset architecture bymodifying the ϕ network to be a series of convolutions.
Revised: We instantiated the agent architecture using a convolutional Deepset architecture bymodifying the ϕ network to be a series of convolutions.

Original: The architecture for the agent that is provided with saliency maps: For both the saliency map and theoriginal image, we pass each through a ResNet50 model as a feature extractor, flatten, dropout withprobability 0.5, Linear layer with 128 nodes and ReLU non-linearity.
Revised: The architecture for the agent that is provided with saliency maps: For both the saliency map and theoriginal image, we pass each through a ResNet50 model as a feature extractor, flatten, dropout withprobability 0.5, Linear layer with 128 nodes and ReLU non-linearity.

Original: We concatenate the two vectorsand then pass through another Linear layer with 100 nodes and ReLU non-linearity, and finally a
Revised: We concatenate the two vectorsand then pass through another Linear layer with 100 nodes and ReLU non-linearity, and finally a

Original: Linear layer with 1 output and Sigmoid non-linearity.
Revised: Linear layer with 1 output and Sigmoid non-linearity.

Original: The architecture for the agent that is provided with only the model prediction: For the original image,we pass each through a ResNet50 model as a feature extractor, flatten, dropout with probability 0.5,
Revised: The architecture for the agent that is provided with only the model prediction: For the original image,we pass each through a ResNet50 model as a feature extractor, flatten, dropout with probability 0.5,

Original: Linear layer with 128 nodes and ReLU non-linearity.
Revised: Linear layer with 128 nodes and ReLU non-linearity.

Original: The model prediction is encoded as a one-hotvector and is passed through a Linear layer with 100 nodes and ReLU non-linearity.
Revised: The model prediction is encoded as a one-hotvector and is passed through a Linear layer with 100 nodes and ReLU non-linearity.

Original: We concatenatethe two vectors and then pass through another Linear layer with 100 nodes and ReLU non-linearity,and finally a Linear layer with 1 output and Sigmoid non-linearity.
Revised: We concatenatethe two vectors and then pass through another Linear layer with 100 nodes and ReLU non-linearity,and finally a Linear layer with 1 output and Sigmoid non-linearity.

Original: We use batch size of 16, Adam optimizer with learning rate 0.001, binary cross entropy loss, andtrain for 10 epochs.
Revised: We use batch size of 16, Adam optimizer with learning rate 0.001, binary cross entropy loss, andtrain for 10 epochs.

Original: Findings: We find that the agent is able to achieve > 95% accuracy when provided with the modelprediction.
Revised: Findings: We find that the agent is able to achieve > 95% accuracy when provided with the modelprediction.

Original: This makes sense because provided that the model has a reasonable mental model ofclassifying which breed a dog is, then it is easy to tell whether the image is correctly classified or not.
Revised: This makes sense because provided that the model has a reasonable mental model ofclassifying which breed a dog is, then it is easy to tell whether the image is correctly classified or not.

Original: However, when provided with saliency maps (e.g., Gradient, Integrated Gradient), the agent accuracyis 50 − 65% , despite attempts to ablate the CNN architecture.
Revised: However, when provided with saliency maps (e.g., Gradient, Integrated Gradient), the agent accuracyis 50 − 65% , despite attempts to ablate the CNN architecture.

Original: After visual inspection (similar to
Revised: After visual inspection (similar to

Original: Figure 13 of their paper) of the saliency map for both cases, it is not particularly evident why it wouldbe helpful in distinguishing when there is a label error.
Revised: Figure 13 of their paper) of the saliency map for both cases, it is not particularly evident why it wouldbe helpful in distinguishing when there is a label error.

Original: J Discussion: The Agent-Human GapJ.1 Human Factors
Revised: J Discussion: The Agent-Human GapJ.1 Human Factors

Original: We describe several human factors that should be considered when researchers are designing
Revised: We describe several human factors that should be considered when researchers are designing

Original: SimEvals and interpreting a SimEval agent’s accuracy.
Revised: SimEvals and interpreting a SimEval agent’s accuracy.

Original: We also discuss how we considered thesefactors when designing our own MTurk study.
Revised: We also discuss how we considered thesefactors when designing our own MTurk study.

Original: Proxy Metrics.
Revised: Proxy Metrics.

Original: Since SimEvals is agnostic to whether a provided explanation is faithful or notto the model that it is explaining.
Revised: Since SimEvals is agnostic to whether a provided explanation is faithful or notto the model that it is explaining.

Original: A researcher using SimEvals to select candidate explanationmethods should be careful to check that the explanations under consideration also satisfy desirableproxy metrics.
Revised: A researcher using SimEvals to select candidate explanationmethods should be careful to check that the explanations under consideration also satisfy desirableproxy metrics.

Original: This way one would not select explanations that may inadvertently mislead the users.
Revised: This way one would not select explanations that may inadvertently mislead the users.

Original: Important proxy metrics include faithfulness to the model [35, 32] and stability over runs [4, 3, 13].
Revised: Important proxy metrics include faithfulness to the model [37, 34] and stability over runs [4, 3, 15].

Original: In our user study, we computed explanations using well known, open-source explanation methodsand ensured not to “adversarially” select explanations which would mislead the humans but enablethe agent to achieve high accuracy.
Revised: In our user study, we computed explanations using well known, open-source explanation methods and ensured not to “adversarially” select explanations which would mislead the humans but enablethe agent to achieve high accuracy.

Original: Complexity of Explanation.
Revised: Complexity of Explanation.

Original: A well-studied human factor is that more complex explanations willnegatively impact a human’s ability to use the explanation [33, 24].
Revised: A well-studied human factor is that more complex explanations willnegatively impact a human’s ability to use the explanation [35, 26].

Original: Provided with enough data,the algorithmic agent may in some cases perform better with more complex explanations (if theadditional dimensions of complexity also contain more predictive information).
Revised: Provided with enough data,the algorithmic agent may in some cases perform better with more complex explanations (if theadditional dimensions of complexity also contain more predictive information).

Original: When designing
Revised: When designing

Original: SimEvals and interpreting agent accuracy, one should factor in the complexity of the explanationsthat are being evaluated.
Revised: SimEvals and interpreting agent accuracy, one should factor in the complexity of the explanationsthat are being evaluated.

Original: If the explanation is complex and high-dimensional, then one might discountthe agent’s accuracy as a measure of informing human performance.
Revised: If the explanation is complex and high-dimensional, then one might discountthe agent’s accuracy as a measure of informing human performance.

Original: In our user study, we limitedthe dataset to 2 dimensions to control explanation complexity and reduce the likelihood that the userexperiences cognitive overload.
Revised: In our user study, we limitedthe dataset to 2 dimensions to control explanation complexity and reduce the likelihood that the userexperiences cognitive overload.

Original: User Interfaces.
Revised: User Interfaces.

Original: The way information is presented to the user through the user study interface orvisualization of the explanations can also affect a human’s ability to make decisions [37].
Revised: The way information is presented to the user through the user study interface orvisualization of the explanations can also affect a human’s ability to make decisions [39].

Original: In our userstudy, we control these factors by designing a simple MTurk interface using Qualtrics (a widely-usedsurvey platform).
Revised: In our userstudy, we control these factors by designing a simple MTurk interface using Qualtrics (a widely-usedsurvey platform).

Original: J.2 Societal Implications

Original: While we demonstrate that SimEvals is able to provide an informative measure of the predictiveinformation contained in a design set-up that, we acknowledge (as discussed in the previous twosubsections) that there are ways in which SimEvals can potentially lead a researcher to draw anincorrect conclusion that would affect the design of their human subject evaluation.

Original: Our work alsoillustrates the many challenges faced when trying to compare an agent accuracy to human accuracy,which is a relevant consideration as automation is increasingly considered in decision-making.

Original: J.3 Future Work
Revised: J.2 Future Work

Original: It is evident that ML models and humans learn and reason differently.
Revised: It is evident that ML models and humans learn and reason differently.

Original: While we found overall thatthe accuracy of an algorithmic agent can be interpreted as a measure of an explanation’s utility forhumans, we observe in our MTurk Study that there is a gap.
Revised: While we found overall thatthe accuracy of an algorithmic agent can be interpreted as a measure of an explanation’s utility forhumans, we observe in our MTurk Study that there is a gap.

Original: Our proposed framework does not intendto measure potential cognitive factors that may affect the ability of humans to use explanations.
Revised: Our proposed framework does not intendto measure potential cognitive factors that may affect the ability of humans to use explanations.

Original: Aninteresting and challenging direction for future work would attempt to (a) understand what suchfactors are, and (b) measure the extent to which a factor is present in the set of candidate explanations.
Revised: Aninteresting and challenging direction for future work would attempt to (a) understand what suchfactors are, and (b) measure the extent to which a factor is present in the set of candidate explanations.

Original: A better understanding of such factors could also inform development of new explanation methodsthat meet these desiderata.
Revised: A better understanding of such factors could also inform development of new explanation methodsthat meet these desiderata.

Original: It’s also worth noting that there are many domains/types of data where humans are still much betterat learning than machines.
Revised: It’s also worth noting that there are many domains/types of data where humans are still much betterat learning than machines.

Original: In these settings, an algorithmic agent would also fail to measure theutility of the explanation to a human.
Revised: In these settings, an algorithmic agent would also fail to measure theutility of the explanation to a human.

Original: Future work may include bringing a human in the loop to definehuman strategies or behaviors that the agent should follow as to make the agent’s reasoning moresimilar to a human’s.
Revised: Future work may include bringing a human in the loop to definehuman strategies or behaviors that the agent should follow as to make the agent’s reasoning moresimilar to a human’s.