Original: The Reverse Turing Test for Evaluating Interpretability Methods on Unknown Tasks
Revised: The Reverse Turing Test for Evaluating Interpretability Methods on Unknown Tasks

Original: Abstract
Revised: Abstract

Original: The Turing Test evaluates a computer program’s ability to mimic human behaviour.
Revised: The Turing Test evaluates a computer program’s ability to mimic human behaviour.

Original: The Reverse Turing Test, reversely, evaluates a human’s ability to mimicmachine behaviour in a forward prediction task.
Revised: The Reverse Turing Test, reversely, evaluates a human’s ability to mimic machine behaviour in a forward prediction task.

Original: We propose to use the Reverse Turing Test to evaluate the quality of interpretability methods.
Revised: We propose to use the Reverse Turing Test to evaluate the quality of interpretability methods.

Original: The Reverse Turing Test improves on previous experimental protocols for human evaluation ofinterpretability methods by a) including a training phase and b) masking the task,which, combined, enables us to evaluate models independently of their quality,in a way that is unbiased by the participants’ previous exposure to the task.
Revised: The Reverse Turing Test improves on previous experimental protocols for human evaluation of interpretability methods by a) including a training phase, and b) masking the task, which, combined, enables us to evaluate models independently of their quality, in a way that is unbiased by the participants’ previous exposure to the task.

Original: Wepresent a human evaluation of LIME across ﬁve NLP tasks in a Latin Square design and analyze the effect of masking the task in forward prediction experiments.
Revised: We present a human evaluation of LIME across ﬁve NLP tasks in a Latin Square design and analyze the effect of masking the task in forward prediction experiments.

Original: Additionally, we demonstrate a fundamental limitation of LIME and show how this limitation is detrimental for human forward prediction in some NLP tasks
Revised: Additionally, we demonstrate a fundamental limitation of LIME and show how this limitation is detrimental for human forward prediction in some NLP tasks.

Original:  Introduction
Revised:  Introduction

Original: Machine learning models have tremendous impact on our daily lives, from information storing andtracking (i.e. Google Search and Facebook News Feed), as well as on other scientiﬁc disciplines.
Revised: Machine learning models have tremendous impact on our daily lives, from information storing and tracking (i e. Google Search and Facebook News Feed), as well as on other scientiﬁc disciplines.

Original: Modern-day NLP models, for example, are complex neural networks with millions or billions ofparameters trained with multiple objectives and often in multiple stages (Devlin et al., 2019; Raffelet al., 2019); they are often seen for that reason, as black boxes whose rationales cannot easily bequeried.
Revised: Modern-day NLP models, for example, are complex neural networks with millions or billions of parameters trained with multiple objectives and often in multiple stages (Devlin et al., 2019; Raffel et al., 2019); they are often seen for that reason, as black boxes whose rationales cannot easily be queried.

Original: In other words, we are increasingly relying on models that we do not understand or cannotexplain, in science, as well as in our daily lives.
Revised: In other words, we are increasingly relying on models that we do not understand or cannot explain, in science, as well as in our daily lives.

Original: Model interpretability, however, is desired for severalreasons: Humans often ask for the motivation behind advice, and in the same way, users are likelyto trust model decisions more if they can ask for the rationale behind them.
Revised: Model interpretability, however, is desired for several reasons: Humans often ask for the motivation behind advice and in the same way, users are likely to trust model decisions more if they can ask for the rationale behind them.

Original: Model interpretabilityenables us to inspect whether models are fair and unbiased, and it enables engineers to detect whenmodels rely on mere confounds.
Revised: Model interpretability enables us to inspect whether models are fair and unbiased and it enables engineers to detect when models rely on mere confounds.

Original: Combatting this type of overﬁtting will lead to more robust (orless error-prone) decision making with better generalization to unseen data (and, hence, safer modelemployment).
Revised: Combatting this type of overﬁtting will lead to more robust (or less error-prone) decision making with better generalization to unseen data (and, hence, safer model employment).

Original: Recent years has seen a surge in work on post-hoc interpretability methods for neural networkswhich aim to approximate complex decision boundaries with less complex models, for example,locally linear models.
Revised: Recent years has seen a surge in work on post-hoc interpretability methods for neural networks which aim to approximate complex decision boundaries with less complex models, for example, locally linear models.

Original: See § 5 in Murdoch et al.
Revised: See §5 in Murdoch et al.

Original: for a brief survey.
Revised: for a brief survey.

Original: Unfortunately, there is littleconsensus on how to compare interpretability methods.
Revised: Unfortunately, there is little consensus on how to compare interpretability methods.

Original: Some benchmarks have been introduced (Reiand Søgaard, 2018; Poerner et al., 2018; DeYoung et al., 2020), but some of these are ﬂawed, andthey are all only applicable to some of the proposed interpretablitity methods.
Revised: Some benchmarks have been introduced (Rei and Søgaard, 2018; Poerner et al., 2018; DeYoung et al., 2020), but some of these are ﬂawed, and they are all only applicable to some of the proposed interpretablitity methods.

Original: See § for discussion.
Revised: See §5 for discussion.

Original: Inour view, a more promising approach to evaluating interpretability methods is by human forwardprediction experiments.
Revised: In our view, a more promising approach to evaluating interpretability methods is by human forward prediction experiments.

Original: Nguyen (2018) presented the ﬁrst evaluations of LIME (Ribeiro et al.,2016) for sentiment analysis using human subjects through a series of Mechanical Turk experiments.
Revised: Nguyen (2018) presented the ﬁrst evaluations of LIME (Ribeiro et al., 2016) for sentiment analysis using human subjects through a series of Mechanical Turk experiments.

Original: Their study had two limitations: (a) They did not allow for a training phase for the human participantsto learn model idiosyncracies, and participants instead had to rely on the assumption that the modelwas near-perfect.
Revised: Their study had two limitations: (a) They did not allow for a training phase for the human participants to learn model idiosyncracies, and participants instead had to rely on the assumption that the model was near-perfect.

Original: (b) Since the participants thus had to rely on their own sentiment predictions, theirevaluations are biased by their beliefs about the sentiment of the input documents.
Revised: (b) Since the participants thus had to rely on their own sentiment predictions, their evaluations are biased by their beliefs about the sentiment of the input documents.

Original: Hase and Bansal(2020) recently presented evaluations of LIME with human participants that involved a trainingphase, enabling them to predict poor model behavior, and thereby addressing limitation (a), butthey still only included known tasks for which forward prediction is biased by the participants’ ownbeliefs.
Revised: Hase and Bansal (2020) recently presented evaluations of LIME with human participants that involved a training phase, enabling them to predict poor model behavior, and thereby addressing limitation (a), but they still only included known tasks for which forward prediction is biased by the participants’ own beliefs.

Original: This paper aims to ﬁll this gap.
Revised: This paper aims to ﬁll this gap.

Original: Contributions This work presents a simple-yet-insightful method for evaluating interpretabilitymethods - which return feature importances or saliency maps with rationales- based on simple 15minute experiments with human participants.
Revised: Contributions This work presents a simple-yet-insightful method for evaluating interpretability methods - which return feature importances or saliency maps with rationales- based on simple 15-minute experiments with human participants.

Original: Our experiments differ from previous work in a veryimportant way: our proposed evaluation of interpretability involves conditions in which human subjects are less likely to rely on their cognitive biases .
Revised: Our experiments differ from previous work in a very important way: our proposed evaluation of interpretability involves conditions in which human subjects are less likely to rely on their cognitive biases .

Original: As our test case, we evaluate LIME (Ribeiroet al., 2016) - which has been a popular feature attribution method in the last few years- acrossﬁve NLP tasks in a Latin Square design Shah and Sinha (1989), including three tasks which werekept secret to our participants.
Revised: As our test case, we evaluate LIME (Ribeiro et al., 2016) - which has been a popular feature attribution method in the last few years- across ﬁve NLP tasks in a Latin Square design Shah and Sinha (1989), including three tasks which were kept secret to our participants.

Original: We argue that keeping the tasks secret to the participants makes theevaluation of interpretability methods more reliable and investigate the impact of this difference inexperimental design.
Revised: We argue that keeping the tasks secret to the participants makes the evaluation of interpretability methods more reliable and investigate the impact of this difference in experimental design.

Original: Additionally, we also point out a weakness of LIME -which is shared acrossmany word attribution methods- namely, that its input/output dimensions are occasionally orthogonal to the relevant dimensions for interpretability.
Revised: Additionally, we also point out a weakness of LIME -which is shared across many word attribution methods- namely, that its input/output dimensions are occasionally orthogonal to the relevant dimensions for interpretability.

Original: We include a task in which this happens and showhow detrimental the interpretability method can be in such cases.
Revised: We include a task in which this happens and show how detrimental the interpretability method can be in such cases.

Original:  Human Biases in Forward Prediction
Revised:  Human Biases in Forward Prediction

Original: One thing sets our experiments in this paper apart from previous evaluations of interpretability methods by A/B testing with human forward prediction (Nguyen, 2018; Hase and Bansal, 2020):
Revised: One thing sets our experiments in this paper apart from previous evaluations of interpretability methods by A/B testing with human forward prediction (Nguyen, 2018; Hase and Bansal, 2020):

Original: We willpresent participants with decisions by models trained on tasks that are unknown to the participants.
Revised: We will present participants with decisions by models trained on tasks that are unknown to the participants.

Original: In other words, humans are simply asked to predict y from x , with no prior knowledge of the relationthat may exist between them, beyond an initial training phase.
Revised: In other words, humans are simply asked to predict y from x , with no prior knowledge of the relation that may exist between them, beyond an initial training phase.

Original: Several different cognitive biases areparticularly important for motivating and analyzing our experimental design:
Revised: Several different cognitive biases are particularly important for motivating and analyzing our experimental design:

Original: Belief bias An effect where someone’s evaluation of the logical strength of an argument is biased by the plausibility of the conclusion (Klauer et al., 2000).
Revised: Belief bias An effect where someone’s evaluation of the logical strength of an argument is biased by the plausibility of the conclusion (Klauer et al., 2000).

Original: In human forward prediction ofmodel behavior, this happens when the plausibility of the conclusion, e.g., this review is positive,biases the subject’s evaluation of her own conclusions, e.g., the model will predict this review isnegative, because it includes this or that term.
Revised: In human forward prediction of model behavior, this happens when the plausibility of the conclusion, e.g., this review is positive, biases the subject’s evaluation of her own conclusions, e.g., the model will predict this review is negative, because it includes this or that term.

Original: We argue that it is particularly important to evaluateinterpretability methods with human forward prediction on unknown tasks to avoid belief bias.
Revised: We argue that it is particularly important to evaluate interpretability methods with human forward prediction on unknown tasks to avoid belief bias.

Original: Conﬁrmation bias This bias occurs when individuals seek information which supports their priorbelief while disproportionately disregarding information that challenges this belief Mynatt et al.(1977).
Revised: Conﬁrmation bias This bias occurs when individuals seek information which supports their prior belief while disproportionately disregarding information that challenges this belief Mynatt et al.

Original: In our context, such a bias could, for example, lead subjects that already classiﬁed a document in one way to disregard LIME mark-up.
Revised: In our context, such a bias could, for example, lead subjects that already classiﬁed a document in one way to disregard LIME mark-up.

Original: In the extreme, conﬁrmation bias could cancel outany effect of interpretability methods in human forward prediction, but our results below show thatin practice, LIME has a strong (positive or negative) effect on human forward prediction.
Revised: In the extreme, conﬁrmation bias could cancel out any effect of interpretability methods in human forward prediction, but our results below show that in practice, LIME has a strong (positive or negative) effect on human forward prediction.

Original: Curse of knowledge This is the phenomenon when better-informed people ﬁnd it extremelydifﬁcult to think about problems from the perspective of lesser-informed people (Ackermanet al., 2003).
Revised: Curse of knowledge This is the phenomenon when better-informed people ﬁnd it extremely difﬁcult to think about problems from the perspective of lesser-informed people (Ackerman et al., 2003).

Original: In our case, the model plays the role of a lesser-informed agent.
Revised: In our case, the model plays the role of a lesser-informed agent.

Original: We believe thecurse of knowledge ampliﬁes belief bias and makes it very hard for participants to unlearn theirprior knowledge of the underlying task relation.
Revised: We believe the curse of knowledge ampliﬁes belief bias and makes it very hard for participants to unlearn their prior knowledge of the underlying task relation.

Original: This bias is very evident in our experimentsbelow and additional motivation for including a training phase in the Reverse Turing test (see
Revised: This bias is very evident in our experiments below and additional motivation for including a training phase in the Reverse Turing test (see

Original: our
Revised: our

Original: Pre-Experiment).
Revised: Pre-Experiment).

Original: Our experimental design is motivated by a desire to reduce the above biases in our forward predictionexperiments.
Revised: Our experimental design is motivated by a desire to reduce the above biases in our forward prediction experiments.

Original: Cognitive biases can interact with human forward prediction in a number of ways, e.g., 
Revised: Cognitive biases can interact with human forward prediction in a number of ways, e.g., making participants less conﬁdent about predictions that do not align with their prior beliefs, or leading them to ignore explanations that are inconsistent with their beliefs.

Original:  LIME – and its Limitations
Revised:  LIME – and its Limitations

Original: The Local Model-agnostic Explanations (LIME) method (Ribeiro et al., 2016) has become one ofthe most widely used post-hoc model interpretability methods in NLP.
Revised: The Local Model-agnostic Explanations (LIME) method (Ribeiro et al., 2016) has become one of the most widely used post-hoc model interpretability methods in NLP.

Original: LIME aims to interpret modelpredictions by locally approximating a model’s decision boundary around an individual prediction.
Revised: LIME aims to interpret model predictions by locally approximating a model’s decision boundary around an individual prediction.

Original: This is done by training a linear classiﬁer on perturbations of this example.
Revised: This is done by training a linear classiﬁer on perturbations of this example.

Original: Several weaknesses of LIME have been identiﬁed in the literature: LIME is linear (Bramhall et al. ,2020), unstable (Elshawi et al., 2019) and very sensitive to the width of the kernel used to assignweights to input example perturbations (Vlassopoulos, 2019; Kopper, 2019), an increasing numberof features also increases weight instability (Gruber, 2019), and Vlassopoulos (2019) argues thatwith sparse data, sampling is insufﬁcient.
Revised: Several weaknesses of LIME have been identiﬁed in the literature: LIME is linear (Bramhall et al., 2020), unstable (Elshawi et al., 2019) and very sensitive to the width of the kernel used to assign weights to input example perturbations (Vlassopoulos, 2019; Kopper, 2019), an increasing number of features also increases weight instability (Gruber, 2019), and Vlassopoulos (2019) argues that with sparse data, sampling is insufﬁcient.

Original: Laugel et al.
Revised: Laugel et al.

Original: (2018) argues the speciﬁc sampling techniqueis suboptimal.
Revised: (2018) argues the speciﬁc sampling technique is suboptimal.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: we train our using standard datasets and evaluate the model on held out training data and testing data to be used for the training and evalu ation sessions involving humans. We also extract LIME explanations. In the human experiments phase, the humans train and evaluate in theseconditions (LIME explanation or no explanation).

Original:  the feature dimensions of LIME.
Revised: Finally, we compare the results. We point to an additional, albeit perhaps obvious, weakness of LIME’s : It can only explain the decisions of a classiﬁer in so far as the decision boundary of the classiﬁer aligns with the feature dimensions of LIME.

Original: In most applications of feature attribution interpretability to NLP problems, the feature dimensions are theinput words.
Revised: In most applications of feature attribution interpretability to NLP problems, the feature dimensions are the input words.

Original: That is to say, such methods canonly explain the decisions of a classiﬁer if thedecision boundary aligns with the dimensionsalong which the occurrences of words are encoded.
Revised: That is to say, such methods can only explain the decisions of a classiﬁer if the decision boundary aligns with the dimensions along which the occurrences of words are encoded.

Original: LIME can, for example, not explain thedecisions of a classiﬁer ” 1 if sentence lengthodd, else 0 ”.
Revised: LIME can, for example, not explain the decisions of a classiﬁer " 1 if sentence length odd, else 0 ".

Original: In our experiments, we include atask in which a classiﬁer is trained to predictthe length of the input sentence (from a lowrank representation), as a way of evaluating theeffect of LIME on human forward prediction,on tasks that LIME is, for this reason, not ableto explain.
Revised: In our experiments, we include a task in which a classiﬁer is trained to predict the length of the input sentence (from a low-rank representation), as a way of evaluating the effect of LIME on human forward prediction, on tasks that LIME is, for this reason, not able to explain.

Original: Examples of real tasks where this limitation is aproblem, include, for example, all tasks wheresentence length is predictive, including readability assessment (Kincaid et al., 1975), authorship attribution (Stamatatos, 2009), or sentence alignment (Brown et al., 1991).
Revised: Examples of real tasks where this limitation is a problem, include, for example, all tasks where sentence length is predictive, including readability assessment (Kincaid et al., 1975), authorship attribution (Stamatatos, 2009), or sentence alignment (Brown et al., 1991).

Original: We notethis limitation is not unique to LIME, but shared among most post-hoc interpretability methodswhich output word or span importance, e.g., hot ﬂip (Ebrahimi et al., 2018), attention (Rei and Søgaard, 2018), and back-propagation (Rei and Søgaard, 2018).
Revised: We note this limitation is not unique to LIME, but shared among most post-hoc interpretability methods which output word or span importance, e.g., hot ﬂip (Ebrahimi et al., 2018), attention (Rei and Søgaard, 2018), and back-propagation (Rei and Søgaard, 2018).

Original: Other approaches to interpretabilitysuch as using inﬂuence functions (Koh and Liang, 2017) may have more explanatory power for suchproblems however we choose to focus our experiments on LIME, as a vast number of interpretabilitymethods return explanations which are extractive similarly to this method.
Revised: Other approaches to interpretability such as using inﬂuence functions (Koh and Liang, 2017) may have more explanatory power for such problems however we choose to focus our experiments on LIME, as a vast number of interpretability methods return explanations which are extractive similarly to this method.

Original:  Human Forward Prediction Experiments
Revised:  Human Forward Prediction Experiments

Original: The experiments we describe below are examples of the Reverse Turing Test.
Revised: The experiments we describe below are examples of the Reverse Turing Test.

Original: The test resemblesthe Turing Test (Turing, 1950; Horn, 1995) in that it focuses on the differences between the behavior of humans and computer programs.
Revised: The test resembles the Turing Test (Turing, 1950; Horn, 1995) in that it focuses on the differences between the behavior of humans and computer programs.

Original: In the Reverse Turing Test, we quantify humans’ ability tosimulate computer programs, however; rather than computer programs’ ability to simulate humans.
Revised: In the Reverse Turing Test, we quantify humans’ ability to simulate computer programs, however; rather than computer programs’ ability to simulate humans.

Original: Speciﬁcally, we quantify humans’ ability to predict the output of machine learning models givenpreviously unseen examples.
Revised: Speciﬁcally, we quantify humans’ ability to predict the output of machine learning models given previously unseen examples.

Original: The test is deﬁned (for classiﬁcation models) as follows:
Revised: The test is deﬁned (for classiﬁcation models) as follows:

Original: The Reverse
Revised: The Reverse

Original: Turing test is an experimental protocol according to which participants are presented with k examples of (cid:104)I ( x ) , ˆ y (cid:105) pairs, with ˆ y =
Revised: Turing test is an experimental protocol according to which participants are presented with k examples of (cid:104)I ( x ) , ˆ y (cid:105) pairs, with ˆ y =

Original: f
Revised: f

Original: ( x ) the labeling of x by some unknown machine learning modelf ( · ) , and I is a possibly empty interpretation function, which, in the case of post-hoc interpretabilitymethods, highlights parts of the input, e.g., input words.
Revised: ( x ) the labeling of x by some unknown machine learning model f ( · ) , and I is a possibly empty interpretation function, which, in the case of post-hoc interpretability methods, highlights parts of the input, e.g., input words.

Original: The training phase is timed.
Revised: The training phase is timed.

Original: Subsequently, participants are presented with m unseen examples x 1 . . .
Revised: Subsequently, participants are presented with m unseen examples x 1 . . .

Original: x m and asked to predict f ( x 1 ) . . .
Revised: x m and asked to predict f ( x 1 ) . . .

Original: f
Revised: f

Original: ( x m ) .
Revised: ( x m ) .

Original: The evaluation phase is also timed.
Revised: The evaluation phase is also timed.

Original: The result of the Reverse Turing test is the accuracy or F 1 of theparticipants’ predictions compared to ˆ y 1 , . . .
Revised: The result of the Reverse Turing test is the accuracy or F 1 of the participants’ predictions compared to ˆ y 1 , . . .

Original: , ˆ y m , as well as the training and inference times.
Revised: , ˆ y m , as well as the training and inference times.

Original: Thetest is meant to evaluate the quality of different interpretations, I ( · ) and can be used for evaluationmethods, like we do, or for evaluating models or interpretability methods during development (Lageet al., 2018).
Revised: The test is meant to evaluate the quality of different interpretations, I ( · ) and can be used for evaluation methods, like we do, or for evaluating models or interpretability methods during development (Lage et al., 2018).

Original: We believe our test is in some ways more critical than previous, as we are attemptingto evaluate interpretability methods more reliably by reducing human belief bias.
Revised: We believe our test is in some ways more critical than previous, as we are attempting to evaluate interpretability methods more reliably by reducing human belief bias.

Original:  Tasks and Data
Revised: 4.1 Tasks and Data

Original: Based on the efforts of 30 annotators, we collected a total of 3000 example annotations in humanforward prediction experiments, distributed across ﬁve different tasks (two known; three unknown)and two experimental conditions (with and without explanations).
Revised: Based on the efforts of 30 annotators, we collected a total of 3000 example annotations in human forward prediction experiments, distributed across ﬁve different tasks (two known; three unknown) and two experimental conditions (with and without explanations).

Original: The overall experimental protocolis shown in Figure 1.
Revised: The overall experimental protocol is shown in Figure 1.

Original: All code for preprocessing data, training the models, and the experimental setups are publicly available at https://github.com/anonymous repo.
Revised: All code for preprocessing data, training the models, and the experimental set ups are publicly available at https://github.com/anonymous_repo.

Original: Known Tasks For our known tasks
Revised: Known Tasks For our known tasks

Original: , we focus on two very common text classiﬁcation tasks: sentiment analysis and hate/offensive speech detection.
Revised: , we focus on two very common text classiﬁcation tasks: sentiment analysis and hate/offensive speech detection.

Original: For sentiment analysis we use the Stanford
Revised: For sentiment analysis we use the Stanford

Original: Sentiment Treebank (SST) (Socher et al., 2013).
Revised: Sentiment Treebank (SST) (Socher et al., 2013).

Original: The SST dataset consists of 6920 documents fortraining, 872 documents for development and 1820 documents for testing.
Revised: The SST dataset consists of 6920 documents for training, 872 documents for development and 1820 documents for testing.

Original: For hate speech detection,we use the HatEval dataset from SemEval 2019 (Basile et al., 2019).
Revised: For hate speech detection, we use the HatEval dataset from SemEval 2019 (Basile et al., 2019).

Original: The dataset consists of several binary tasks, however we focus on the task of detecting presence of hate speech (disregardingwhich group is being targeted as this is considered a separate task).
Revised: The dataset consists of several binary tasks, however we focus on the task of detecting presence of hate speech (disregarding which group is being targeted as this is considered a separate task).

Original: In total, there are 9000 tweetsfor training, 1000 for development and 3000 for testing.
Revised: In total, there are 9000 tweets for training, 1000 for development and 3000 for testing.

Original: Unknown Tasks As our unknown tasks, we use 3 of the 10 probing tasks introduced in Conneauet al.
Revised: Unknown Tasks As our unknown tasks, we use 3 of the 10 probing tasks introduced in Conneau et al.

Original: The probing tasks were originally designed to evaluate the linguistic properties of sentence embedding models.
Revised: The probing tasks were originally designed to evaluate the linguistic properties of sentence embedding models.

Original: In this study we are mostly interested in the differences in performancebetween humans and machines, and are not looking to evaluate linguistic properties of representations in depth, therefore chose only a few of theses tasks.
Revised: In this study we are mostly interested in the differences in performance between humans and machines, and are not looking to evaluate linguistic properties of representations in depth, therefore chose only a few of theses tasks.

Original: The ﬁrst task is sentence length predictionin which the sentences are grouped in 6 bins indicating length in terms of number of words.
Revised: The ﬁrst task is sentence length prediction in which the sentences are grouped in 6 bins indicating length in terms of number of words.

Original: Thistask was chosen in order to examine the effect on LIME in a task where LIME offers poor explanations.
Revised: This task was chosen in order to examine the effect on LIME in a task where LIME offers poor explanations.

Original: The second probing task is tense prediction , which involves predicting whether the verb inthe main clause is present or past tense.
Revised: The second probing task is tense prediction , which involves predicting whether the verb in the main clause is present or past tense.

Original: The third task is subject number prediction , which focuseson predicting whether the subject in the main clause is plural or singular.
Revised: The third task is subject number prediction , which focuses on predicting whether the subject in the main clause is plural or singular.

Original: These last two are simpletasks where we expect LIME to offer good enough explanations.
Revised: These last two are simple tasks where we expect LIME to offer good enough explanations.

Original: The training data for each of theprobing tasks consists of 100k sentences, 10k sentences for validation and 10k sentences for testing.
Revised: The training data for each of the probing tasks consists of 100k sentences, 10k sentences for validation and 10k sentences for testing.

Original: The sentences are taken from the Toronto Book Corpus (Zhu et al., 2015).
Revised: The sentences are taken from the Toronto Book Corpus (Zhu et al., 2015).

Original: More details on dataextraction can be found on Conneau et al.
Revised: More details on data extraction can be found on Conneau et al.

Original:  Classiﬁcation Model
Revised: 4.2 Classiﬁcation Model

Original: For training sentiment and hate speech classiﬁers, we pass as our input pretrained BERT representations (Devlin et al., 2019) through an LSTM layer (Hochreiter and Schmidhuber, 1997)
Revised: For training sentiment and hate speech classiﬁers, we pass as our input pretrained BERT representations (Devlin et al., 2019) through an LSTM layer (Hochreiter and Schmidhuber, 1997)

Original: ( d = 100 )followed by a multi-layered perceptron with a single hidden layer ( d = 100 ).
Revised: ( d = 100 ) followed by a multi-layered perceptron with a single hidden layer ( d = 100 ).

Original: We use a learningrate of 0.001 and Adam optimizer.
Revised: We use a learning rate of 0.001 and Adam optimizer.

Original: The hyper-parameters were not tuned for optimal performance.
Revised: The hyper-parameters were not tuned for optimal performance.

Original: We use the same architecture for all tasks, except for sentence length prediction.
Revised: We use the same architecture for all tasks, except for sentence length prediction.

Original: For the sentencelength prediction task, we use BERT token representations and pass them through a mean poolinglayer followed by a multi-layered perceptron with a single hidden layer ( d = 100 ).
Revised: For the sentence length prediction task, we use BERT token representations and pass them through a mean pooling layer followed by a multi-layered perceptron with a single hidden layer ( d = 100 ).

Original: Both models aretrained for 20 epochs.
Revised: Both models are trained for 20 epochs.

Original: Note also that we do not ﬁne-tune the BERT representations.
Revised: Note also that we do not ﬁne-tune the BERT representations.

Original: This, togetherwith our hyper-parameters, gives us suboptimal performance, especially on the known tasks, butthis was done on purpose to make our predictions different from the gold labels for the known tasks,in order to make it possible to quantify participants’ belief bias: If results are too close to humanperformance, it would not be possible to distinguish human forward prediction performance withrespect to model predictions from human performance with respect to predicting the true class.
Revised: This, together with our hyper-parameters, gives us suboptimal performance, especially on the known tasks, but this was done on purpose to make our predictions different from the gold labels for the known tasks, in order to make it possible to quantify participants’ belief bias: If results are too close to human performance, it would not be possible to distinguish human forward prediction performance with respect to model predictions from human performance with respect to predicting the true class.

Original: Ourperformance on the unknown probing tasks is comparable to the results in Conneau et al.
Revised: Our performance on the unknown probing tasks is comparable to the results in Conneau et al.

Original:  Stimulus Presentation
Revised: 4.3 Stimulus Presentation

Original: [Figure] Each human forward prediction experiment consists of a training session where we present the participant with 25 training samples with model predictions, with or without explanations, followedby an evaluation session with 15 testing samples (without model predictions), also with or without
Revised: Each human forward prediction experiment consists of a training session where we present the participant with 25 training samples with model predictions, with or without explanations, followed by an evaluation session with 15 testing samples (without model predictions), also with or without

Original: LIME explanations.
Revised: LIME explanations.

Original: The participant is asked to predict the model’s labeling of these items.
Revised: The participant is asked to predict the model’s labeling of these items.

Original: We usea Latin Square design (Shah and Sinha, 1989)to control for idiosyncratic differences between ourparticipants.
Revised: We use a Latin Square design (Shah and Sinha, 1989)to control for idiosyncratic differences between our participants.

Original: For each of the tasks t t , we therefore randomly sample 120 examples, 75 of which weuse for training our participants, and 45 of which we use for evaluation.
Revised: For each of the tasks t t , we therefore randomly sample 120 examples, 75 of which we use for training our participants, and 45 of which we use for evaluation.

Original: We divide the 75 trainingsamples into groups of 25: t t 1 , t t 2 , t t 3 .
Revised: We divide the 75 training samples into groups of 25: t t 1 , t t 2 , t t 3 .

Original: We have three different presentation conditions: no explanation, LIME explanation, or random explanation (for control).
Revised: We have three different presentation conditions: no explanation, LIME explanation, or random explanation (for control).

Original: For the LIME explanations, we removeinformation about model decision and present participants with the original LIME output images,after turning them into grayscale in order to avoid revealing the class label.
Revised: For the LIME explanations, we remove information about model decision and present participants with the original LIME output images, after turning them into grayscale in order to avoid revealing the class label.

Original: We rely on 500 perturbations of each data sample in order to obtain the top 3 most informative input tokens.
Revised: We rely on 500 perturbations of each data sample in order to obtain the top 3 most informative input tokens.

Original: See Figure2 for an example of the visual stimuli under this condition.
Revised: See Figure 2 for an example of the visual stimuli under this condition.

Original: The training sessions are interactive,simulating the test interface, but providing the true answer whenever the participant has provided aninitial guess.
Revised: The training sessions are interactive, simulating the test interface, but providing the true answer whenever the participant has provided an initial guess.

Original: We shufﬂe the training sessions at random.
Revised: We shufﬂe the training sessions at random.

Original: The evaluation sets for each task t e consistof 45 samples in total, split into chunks of 15: t e 1 , t e 2 , t e 3 .
Revised: The evaluation sets for each task t e consist of 45 samples in total, split into chunks of 15: t e 1 , t e 2 , t e 3 .

Original: In the evaluation session, subjects are notprovided with the true model responses, to avoid biases from additional training.
Revised: In the evaluation session, subjects are not provided with the true model responses, to avoid biases from additional training.

Original: We divide our participants in three groups, and for each task, the groups are assigned task subsamples in the following Latin Square design:
Revised: We divide our participants in three groups, and for each task, the groups are assigned task subsamples in the following Latin Square design:

Original: [Figure]
Revised: [Figure]

Original: tt 3 ,t et t 1 ,t e 1 Subjectst t 3 ,t et t 1 ,t et t 2 ,t e
Revised: t et t 2 ,t e

Original: We include 3 unknown tasks, meaning that no information about the tasks was provided to theparticipants in advance of the experiment.
Revised: We include 3 unknown tasks, meaning that no information about the tasks was provided to the participants in advance of the experiment.

Original: Instead, subjects had to try to infer patterns from thedata sample, possibly augmented with LIME explanations.
Revised: Instead, subjects had to try to infer patterns from the data sample, possibly augmented with LIME explanations.

Original: For the known tasks, we follow Nguyen(2018) and Hase and Bansal (2020) and provide subjects with a brief explanation of the task, butemphasize the fact that the participants should predict model decisions , not the true labels; and hence,they should avoid being inﬂuenced by their own beliefs of whether a text is positive or an instanceof hate speech.
Revised: For the known tasks, we follow Nguyen (2018) and Hase and Bansal (2020) and provide subjects with a brief explanation of the task, but emphasize the fact that the participants should predict model decisions , not the true labels; and hence, they should avoid being inﬂuenced by their own beliefs of whether a text is positive or an instance of hate speech.

Original: As in Hase and Bansal (2020), we make sure that true positives, false positives, truenegatives, and false negatives are balanced across the training and test data.
Revised: As in Hase and Bansal (2020), we make sure that true positives, false positives, true negatives, and false negatives are balanced across the training and test data.

Original: In total we have 30participants, all with at least undergraduate education and some knowledge of computer science andmachine learning.
Revised: In total we haveparticipants, all with at least undergraduate education and some knowledge of computer science and machine learning.

Original: We collect 3000 human forward predictions: 1800 from training sessions and 1200from the evaluation sessions.
Revised: We collect 3000 human forward predictions: 1800 from training sessions andfrom the evaluation sessions.

Original: For each condition and item in the evaluation set, we have at least twohuman forward predictions.
Revised: For each condition and item in the evaluation set, we have at least two human forward predictions.

Original: Some of the participants gave us optional feedback on strategies theyused.
Revised: Some of the participants gave us optional feedback on strategies they used.

Original: This, as well as the distribution of data points across tasks and conditions and some examplesof our interface can be found in the Appendix.
Revised: This, as well as the distribution of data points across tasks and conditions and some examples of our interface can be found in the Appendix.

Original:  Pre-Experiment: The Effect of Training on Forward Prediction
Revised: 4.4 Pre-Experiment: The Effect of Training on Forward Prediction

Original: In addition to our main experiment with 30 participants, we also performed a human forward prediction pre-experiment with a single participant.
Revised: In addition to our main experiment with 30 participants, we also performed a human forward prediction pre-experiment with a single participant.

Original: In the pre-experiment we compare human forwardprediction with and without training ; we do so to motivate our experimental design, in which we follow Hase and Bansal (2020), but depart from Nguyen (2018), in including a training phase in whichhumans can learn the idiosyncracies of the machine learning model.
Revised: In the pre-experiment we compare human forward prediction with and without training ; we do so to motivate our experimental design, in which we follow Hase and Bansal (2020), but depart from Nguyen (2018), in including a training phase in which humans can learn the idiosyncracies of the machine learning model.

Original: In the pre-experiment, we onlyexplore the effects of the training phase for the known tasks.
Revised: In the pre-experiment, we only explore the effects of the training phase for the known tasks.

Original: We ﬁrst ran the experiment withouttraining; then ran the experiment with training.
Revised: We ﬁrst ran the experiment without training; then ran the experiment with training.

Original: To clearly be able to quantify the effect of our interactive training phase, we only use examples with false model predictions in the pre-experiment.
Revised: To clearly be able to quantify the effect of our interactive training phase, we only use examples with false model predictions in the pre-experiment.

Original: For each of the two tasks, sentiment analysis and hate speech detection, we use: (a) 20 distinct examples for evaluation for each of the two conditions; and (b) 25 distinct examples for training for
Revised: For each of the two tasks, sentiment analysis and hate speech detection, we use: (a) 20 distinct examples for evaluation for each of the two conditions; and (b) 25 distinct examples for training for

Original: H UMAN A CC .
Revised: H UMAN A CC .

Original: (ˆ p )
Revised: (ˆ p )

Original: H UMAN T IME (ˆ p )
Revised: H UMAN T IME (ˆ p )

Original: M ODEL A CC .
Revised: M ODEL A CC .

Original: ( p )
Revised: ( p )

Original: H UMAN A CC .
Revised: H UMAN A CC .

Original: ( p )
Revised: ( p )

Original: Task x
Revised: [Table] x

Original: LIME (x ) x
Revised: LIME (

Original: LIME ( x ) x x
Revised: [Table] ) x LIME ( [Table]

Original:  LIME (x )
Revised: x x LIME (

Original: K NOWN TASKS

Original: [Table]
Revised: [Table]

Original: the second experimental condition.
Revised: the second experimental condition.

Original: Note that since we only use a single human participant in thepre-experiment, controlling for individual differences, we cannot control for the difﬁculty of datapoints and use different data points across the two experimental conditions.
Revised: Note that since we only use a single human participant in the pre-experiment, controlling for individual differences, we cannot control for the difﬁculty of data points and use different data points across the two experimental conditions.

Original: The effect of training is positive.
Revised: The effect of training is positive.

Original: On the SST dataset, accuracy with respect to model predictions ( ˆ p )increases from 0.400 to 0.550 ; 1 on the HatEval2019 dataset, performance increases from 0.3690 to0.526 .
Revised: On the SST dataset, accuracy with respect to model predictions ( ˆ p ) increases from 0.400 to 0.550 ; 1 on the HatEval2019 dataset, performance increases from 0.3690 to 0.526 .

Original: We see this as a very strong motivation for including a training phase.
Revised: We see this as a very strong motivation for including a training phase.

Original: A training phase alsomakes it possible to perform human forward prediction experiments on tasks that are unknown tothe participants, removing any belief bias that may otherwise affect results.
Revised: A training phase also makes it possible to perform human forward prediction experiments on tasks that are unknown to the participants, removing any belief bias that may otherwise affect results.

Original: We note that a trainingphase does not necessarily lead to faster inference times.
Revised: We note that a training phase does not necessarily lead to faster inference times.

Original: On HatEval, average inference time wasreduced from 08:56 to 07:21 , but on STS, it increased from 06:24 to 08:53 .
Revised: On HatEval, average inference time was reduced from 08:56 to 07:21 , but on STS, it increased from 06:24 to 08:53 .

Original: This suggests thatuntrained annotators (after a few instances) learn superﬁcial heuristics that enable them to draw fast,yet inaccurate, inferences.
Revised: This suggests that untrained annotators (after a few instances) learn superﬁcial heuristics that enable them to draw fast, yet inaccurate, inferences.

Original:  Main Experiment: The Effect of LIME on Forward Prediction
Revised: 4.5 Main Experiment: The Effect of LIME on Forward Prediction

Original: We report the results of our main experiment in Table 1.
Revised: We report the results of our main experiment in Table 1.

Original: Results show that LIME helps, both in termsof accuracy and time, on known and unknown target tasks, except when the decisions boundary doesnot align with LIME dimensions (Sent Len) (columns 1–4); and that while humans are biased bytheir beliefs and knowledge of the known tasks, they are not biased during unknown tasks, whichcan be seen by their decrease in accuracy with respect to human annotation.
Revised: Results show that LIME helps, both in terms of accuracy and time, on known and unknown target tasks, except when the decisions boundary does not align with LIME dimensions (Sent Len) (columns 1–4); and that while humans are biased by their beliefs and knowledge of the known tasks, they are not biased during unknown tasks, which can be seen by their decrease in accuracy with respect to human annotation.

Original: We make the followingobservations:
Revised: We make the following observations:

Original: The Effect of LIME on Known Tasks This is the standard set-up considered also in previouswork (Nguyen, 2018; Hase and Bansal, 2020); see columns 1–2 and rows 1–2 in Table 1.
Revised: The Effect of LIME on Known Tasks This is the standard set-up considered also in previous work (Nguyen, 2018; Hase and Bansal, 2020); see columns 1–2 and rows 1–2 in Table 1.

Original: We seethat LIME leads to signiﬁcantly better human forward prediction performance on both tasks.
Revised: We see that LIME leads to signiﬁcantly better human forward prediction performance on both tasks.

Original: It alsoleads to (statistically) signiﬁcantly faster inference times, approximately halving the time participants spend on classifying the test examples.
Revised: It also leads to (statistically) signiﬁcantly faster inference times, approximately halving the time participants spend on classifying the test examples.

Original: This shows that LIME, in spite of its limitations ( § 3),is a very useful tool in some cases.
Revised: This shows that LIME, in spite of its limitations (§3), is a very useful tool in some cases.

Original: The Effect of LIME on Unknown Tasks The effect of LIME on human forward prediction accuracy on 2 of the unknown tasks is not signiﬁcant.
Revised: The Effect of LIME on Unknown Tasks The effect of LIME on human forward prediction accuracy on 2 of the unknown tasks is not signiﬁcant.

Original: On the two tasks, where LIME provides meaningfulexplanations (subject number and tense prediction), LIME does lead to smaller reductions in inference time which are not statistically signiﬁcant.
Revised: On the two tasks, where LIME provides meaningful explanations (subject number and tense prediction), LIME does lead to smaller reductions in inference time which are not statistically signiﬁcant.

Original: The effect on the participants’ accuracy is mixedand insigniﬁcant.
Revised: The effect on the participants’ accuracy is mixed and insigniﬁcant.

Original: In addition, LIME is signiﬁcantly detrimental on human forward prediction accuracy for the task of sentence length prediction; it also leads to longer inference times, although thisdifference was not statistically signiﬁcant.
Revised: In addition, LIME is signiﬁcantly detrimental on human forward prediction accuracy for the task of sentence length prediction; it also leads to longer inference times, although this difference was not statistically signiﬁcant.

Original: This shows that while LIME is useful in some cases, this is not always the case.
Revised: This shows that while LIME is useful in some cases, this is not always the case.

Original: We speculate that since LIME explanations are partial, they are only effectivewhen supplemented by (approximately correct) belief bias.
Revised: We speculate that since LIME explanations are partial, they are only effective when supplemented by (approximately correct) belief bias.

Original: If true, this suggests that LIME, even forthe tasks that can be explained in terms of input words, is, nevertheless, only applicable to tasks thathumans have experience with, and when the underlying models perform reasonably well.
Revised: If true, this suggests that LIME, even for the tasks that can be explained in terms of input words, is, nevertheless, only applicable to tasks that humans have experience with, and when the underlying models perform reasonably well.

Original: Known and Unknown Tasks In general, our participants are much slower at classifying exampleswhen the task is unknown.
Revised: Known and Unknown Tasks In general, our participants are much slower at classifying examples when the task is unknown.

Original: This shows the efﬁciency of the belief biases our participants have in sentiment analysis and hate speech classiﬁcation.
Revised: This shows the efﬁciency of the belief biases our participants have in sentiment analysis and hate speech classiﬁcation.

Original: The effectiveness of these biases is also demonstratedby the performance gaps between humans and models when comparing their predictions to groundtruth labels.
Revised: The effectiveness of these biases is also demonstrated by the performance gaps between humans and models when comparing their predictions to ground truth labels.

Original: To see this, consider columns 5–7 in Table 1.
Revised: To see this, consider columns 5–7 in Table 1.

Original: Participants, while instructed to predictmodel output ( ˆ p ), actually signiﬁcantly outperform our classiﬁer in predicting the true labels (0.706vs. 0.573)!
Revised: Participants, while instructed to predict model output ( ˆ p ), actually signiﬁcantly outperform our classiﬁer in predicting the true labels (0. vs. 0.573)!

Original: In contrast, participants perform subject number and tense prediction at chance levels(0.491 and 0.500), while a simple classiﬁer achieves accuracy greater than 0 .
Revised: In contrast, participants perform subject number and tense prediction at chance levels (0.491 and 0.500), while a simple classiﬁer achieves accuracy greater than 0 .

Original: 9 on both tasks.
Revised: 9 on both tasks.

Original: Thisclearly demonstrates belief bias in human forward prediction experiments.
Revised: This clearly demonstrates belief bias in human forward prediction experiments.

Original: Human Inference Time
Revised: Human Inference Time

Original: In addition to considering performance, we also recorded thetime our participants spent on completing theforward prediction tasks.
Revised: In addition to considering performance, we also recorded the time our participants spent on completing the forward prediction tasks.

Original: We present the average times of each condition in Table 1 withshorter times bolded.
Revised: We present the average times of each condition in Table 1 with shorter times bolded.

Original: We used the MannWhitney U test to determine signiﬁcance forthese, which is also shown in the same table.
Revised: We used the Mann-Whitney U test to determine signiﬁcance for these, which is also shown in the same table.

Original: We plot the total averages in Figure 3.
Revised: We plot the total averages in Figure 3.

Original: All theresults shown in the plot are signiﬁcant withα < 0 .
Revised: All the results shown in the plot are signiﬁcant with α < 0 .

Original:  Related Works 
Revised:  Related Works Interpretability methods Interpretability

Original: Interpretability

Original: methodsInterpretabilitymethods come in different ﬂavors: (a) post-hocanalysis methods that estimate input featureimportance for decisions, including LIME, (b)post-hoc analysis methods that estimate theinﬂuence of training instances on decisions,e.g., inﬂuence functions (Koh and Liang,2017) and (c) strategies for making complexmodels interpretable by learning to generateexplanations (Narang et al., 2020) or uptrainingsimpler models (Agarwal et al., 2020).
Revised: methods come in different ﬂavors: (a) post-hoc analysis methods that estimate input feature importance for decisions, including LIME, (b) post-hoc analysis methods that estimate the inﬂuence of training instances on decisions, e.g., inﬂuence functions (Koh and Liang, 2017) and (c) strategies for making complex models interpretable by learning to generate explanations (Narang et al., 2020) or uptraining simpler models (Agarwal et al., 2020).

Original: In thispaper we have focused on post-hoc interpretability methods, but it is equally important, we argue,to evaluate other types of interpretability methods on unknown tasks, when running human forwardprediction experiments, to avoid participants’ cognitive biases.
Revised: In this paper we have focused on post-hoc interpretability methods, but it is equally important, we argue, to evaluate other types of interpretability methods on unknown tasks, when running human forward prediction experiments, to avoid participants’ cognitive biases.

Original: Intrinsic evaluation of interpretability methods One standard approach to evaluating explanations is to remove the parts of the input detected by the interpretability method and see whether classiﬁer performance degrades (Samek et al., 2017).
Revised: Intrinsic evaluation of interpretability methods One standard approach to evaluating explanations is to remove the parts of the input detected by the interpretability method and see whether classiﬁer performance degrades (Samek et al., 2017).

Original: One drawback of this method is that the corrupted examples are now out-of-distribution, and classiﬁers will generally perform worse on suchexamples.
Revised: One drawback of this method is that the corrupted examples are now out-of-distribution, and classiﬁers will generally perform worse on such examples.

Original: Hooker et al.
Revised: Hooker et al.

Original: (2019) improve on this by evaluating classiﬁers retrained on the corruptedexamples.
Revised: (2019) improve on this by evaluating classiﬁers retrained on the corrupted examples.

Original: This approach, however, now suffers from another drawback: If classiﬁers perform wellon the corrupted examples, that does not mean the interpretability methods were wrong.
Revised: This approach, however, now suffers from another drawback: If classiﬁers perform well on the corrupted examples, that does not mean the interpretability methods were wrong.

Original: 2 Jain and
Revised: 2 Jain and

Original: Wallace (2019) evaluate attention functions as explanations and argue that they do not provide usefulexplanations, in part because they do not correlate with gradient-based approaches to determining feature importance; Wiegreffe and Pinter (2019), in return, show this test is not sufﬁcient to showattention functions do not provide useful explanations.
Revised: Wallace (2019) evaluate attention functions as explanations and argue that they do not provide useful explanations, in part because they do not correlate with gradient-based approaches to determining feature importance; Wiegreffe and Pinter (2019), in return, show this test is not sufﬁcient to show attention functions do not provide useful explanations.

Original: Extrinsic benchmarks for interpretability methods Rei and Søgaard (2018) show how tokenlevel annotated corpora can be converted to benchmarks for evaluating post-hoc interpretabilitymethods.
Revised: Extrinsic benchmarks for interpretability methods Rei and Søgaard (2018) show how tokenlevel annotated corpora can be converted to benchmarks for evaluating post-hoc interpretability methods.

Original: They train sentence classiﬁers to predict whether sentences contain labels or not, use interpretability methods to predict what input words were important, and use the F 1 score of thosepredictions to evaluate the interpretability methods.
Revised: They train sentence classiﬁers to predict whether sentences contain labels or not, use interpretability methods to predict what input words were important, and use the F 1 score of those predictions to evaluate the interpretability methods.

Original: Their method, however, only works as an evaluation of interpretability methods under the assumption that the classiﬁer is near-perfect (since otherwise the token-level annotations cannot be assumed to be explanations of model decisions); furthermore, it is only applicable to tasks for which we have token-level annotations.
Revised: Their method, however, only works as an evaluation of interpretability methods under the assumption that the classiﬁer is near-perfect (since otherwise the token-level annotations cannot be assumed to be explanations of model decisions); furthermore, it is only applicable to tasks for which we have token-level annotations.

Original: Poerner et al.
Revised: Poerner et al.

Original: (2018)adopt a slightly different approach, augmenting real documents with random text passages to seewhether interpretability methods focus on the original text passages.
Revised: (2018) adopt a slightly different approach, augmenting real documents with random text passages to see whether interpretability methods focus on the original text passages.

Original: This method suffers from thesame drawback, that it assumes near-perfect performance.
Revised: This method suffers from the same drawback, that it assumes near-perfect performance.

Original: It is also only designed to capture falsepositives; it cannot distinguish between true or false negatives.
Revised: It is also only designed to capture false positives; it cannot distinguish between true or false negatives.

Original: Finally, DeYoung et al.
Revised: Finally, DeYoung et al.

Original: (2020) recently introduced ERASER, 3 a suite of NLP datasets augmented with rationales, including readingcomprehension, natural language inference, and fact checking.
Revised: (2020) recently introduced ERASER, 3 a suite of NLP datasets augmented with rationales, including reading comprehension, natural language inference, and fact checking.

Original: ERASER also assumes near-perfectperformance, and can be seen as extending the set of tasks for which the method proposed in Reiand Søgaard (2018), is applicable.
Revised: ERASER also assumes near-perfect performance, and can be seen as extending the set of tasks for which the method proposed in Rei and Søgaard (2018), is applicable.

Original: Our method, in contrast, is independent of model quality.
Revised: Our method, in contrast, is independent of model quality.

Original: Human evaluation of explanations
Revised: Human evaluation of explanations

Original: The idea of evaluating explanations by testing human participants’ ability to predict model decisions with and without explanations is not novel.
Revised: The idea of evaluating explanations by testing human participants’ ability to predict model decisions with and without explanations is not novel.

Original: Nguyen (2018),
Revised: Nguyen (2018),

Original: Lage et al.
Revised: Lage et al.

Original: (2018) and Hase and Bansal (2020), as already discussed, present such experiments.
Revised: (2018) and Hase and Bansal (2020), as already discussed, present such experiments.

Original: Schmidt and Biessmann (2019) is another example of human forward prediction experiments in acrowdsourcing platform.
Revised: Schmidt and Biessmann (2019) is another example of human forward prediction experiments in a crowdsourcing platform.

Original: They perform experiments on the effect of LIME and COVAR on humanforward prediction for a sentiment task that is known to be participants, in advance.
Revised: They perform experiments on the effect of LIME and COVAR on human forward prediction for a sentiment task that is known to be participants, in advance.

Original: Our criticismof Nguyen (2018) also applies to their study.
Revised: Our criticism of Nguyen (2018) also applies to their study.

Original: Narayanan et al.
Revised: Narayanan et al.

Original: (2018) also present evaluations ofinterpretability methods with humans; they design simple tasks in which humans verify whether anoutput is consistent with an input and an explanation.
Revised: (2018) also present evaluations of interpretability methods with humans; they design simple tasks in which humans verify whether an output is consistent with an input and an explanation.

Original: The human participants are provided withexplanations of what the tasks are, and they only consider a handful of input features.
Revised: The human participants are provided with explanations of what the tasks are, and they only consider a handful of input features.

Original: The Reverse Turing Test that we propose here is different from previous proposals to use humanforward prediction to evaluate interpretability methods, in that it a) includes a training phase whichis important for subjects to learn model nuances and which in turn, allows us to b) include humanforward prediction on unknown tasks, i.e., tasks about which they have no prior beliefs.
Revised: The Reverse Turing Test that we propose here is different from previous proposals to use human forward prediction to evaluate interpretability methods, in that it a) includes a training phase which is important for subjects to learn model nuances and which in turn, allows us to b) include human forward prediction on unknown tasks, i.e., tasks about which they have no prior beliefs.

Original: We are, tothe best of our knowledge, the ﬁrst to propose such a protocol.
Revised: We are, to the best of our knowledge, the ﬁrst to propose such a protocol.

Original: In the above experiments, designed tomotivate the design of the Reverse Turing Test , we see the limitations of a widely used interpretability method, LIME.
Revised: In the above experiments, designed to motivate the design of the Reverse Turing Test , we see the limitations of a widely used interpretability method, LIME.

Original: On some tasks, i.e., tasks which cannot be explained by the occurrence of inputwords, the effect of LIME is detrimental; and on unknown tasks, for which LIME interpretations arenot supported by participants’ cognitive biases, its effect on human forward prediction is insigniﬁcant.
Revised: On some tasks, i.e., tasks which cannot be explained by the occurrence of input words, the effect of LIME is detrimental; and on unknown tasks, for which LIME interpretations are not supported by participants’ cognitive biases, its effect on human forward prediction is insigniﬁcant.

Original: Overall, our experiments show that our proposed design offers interesting insights into the rolethat cognitive biases play in the evaluation of interpretability, and propose that such a set up be usedin further research to explore the effect of cognitive biases for other interpretability methods whichprovide ﬁnal rationales similar to those provided by LIME.
Revised: Overall, our experiments show that our proposed design offers interesting insights into the role that cognitive biases play in the evaluation of interpretability, and propose that such a set up be used in further research to explore the effect of cognitive biases for other interpretability methods which provide ﬁnal rationales similar to those provided by LIME.

Original:  Conclusion
Revised:  Conclusion

Original: We presented an evaluation protocol for interpretability methods, which differs from previous workby including a training phase and by including unknown tasks.
Revised: We presented an evaluation protocol for interpretability methods, which differs from previous work by including a training phase and by including unknown tasks.

Original: This makes our protocol work independently of model quality, and controls for belief bias.
Revised: This makes our protocol work independently of model quality, and controls for belief bias.

Original: Using LIME as our test case, we ﬁnd thaton known tasks, LIME leads to statistically signiﬁcant improvements in human forward prediction,both in accuracy and inference time.
Revised: Using LIME as our test case, we ﬁnd that on known tasks, LIME leads to statistically signiﬁcant improvements in human forward prediction, both in accuracy and inference time.

Original: However, when tasks are unknown, differences are no longersigniﬁcant.
Revised: However, when tasks are unknown, differences are no longer signiﬁcant.

Original: We see this as evidence of bias in the standard protocols, and argue that making tasksunknown, leads to more reliable evaluations.
Revised: We see this as evidence of bias in the standard protocols, and argue that making tasks unknown, leads to more reliable evaluations.

Original: We also identify tasks, where model decisions cannotbe explained in terms of input word occurrences, and for which the effect of LIME is detrimental for human forward prediction performance
Revised: We also identify tasks, where model decisions cannot be explained in terms of input word occurrences, and for which the effect of LIME is detrimental for human forward prediction performance.

Original:  Presentation of stimuli
Revised: A Presentation of stimuli

Original: We created a web application using Flask 4 in order to collect participant data.
Revised: We created a web application using Flask 4 in order to collect participant data.

Original: Participants wouldget assigned a known or unknown task and LIME explanations or no explanations.
Revised: Participants would get assigned a known or unknown task and LIME explanations or no explanations.

Original: For all tasks weprovide the same general instructions.
Revised: For all tasks we provide the same general instructions.

Original: See top of Figure 4 for a screenshot of our general instructions.
Revised: See top of Figure 4 for a screenshot of our general instructions.

Original: In addition, we had task speciﬁc instructions.
Revised: In addition, we had task speciﬁc instructions.

Original: For known tasks we provided short descriptions of thetask, while emphasizing the fact that subjects should imitate the model rather than follow their ownopinions about the true labels.
Revised: For known tasks we provided short descriptions of the task, while emphasizing the fact that subjects should imitate the model rather than follow their own opinions about the true labels.

Original: For unknown tasks, we provided instructions as seen in Figure 4.
Revised: For unknown tasks, we provided instructions as seen in Figure 4.

Original: The training and evaluation sessions were almost the same, with the only difference being that duringtraining, subjects could check the model’s answer after making an initial guess.
Revised: The training and evaluation sessions were almost the same, with the only difference being that during training, subjects could check the model’s answer after making an initial guess.

Original: See Figure 5 for anexample of what the items looked like.
Revised: See Figure 5 for an example of what the items looked like.

Original: The example here is for the task of sentence length predictionusing LIME explanations.
Revised: The example here is for the task of sentence length prediction using LIME explanations.

Original: [Figure]
Revised: [Figure]

Original:  Subject Feedback
Revised: B Subject Feedback

Original: [Figure] As an optional part of our tests, subjects provided some insight into the strategies they came up with[Figure] or troubles they had when solving a task.
Revised: As an optional part of our tests, subjects provided some insight into the strategies they came up with or troubles they had when solving a task.

Original: We only had this feedback from some of the participants,[Figure] which can be found in Table
Revised: We only had this feedback from some of the participants, which can be found in Table

Original: [Figure]
Revised: Explanation Task

Original: [Table]
Revised: Strategy none

Original: Binary
Revised: Binary

Original: Tried to identify what kinds of data the ML model fails 
Revised: Tried to identify what kinds of data the ML model fails lime

Original: [Table]

Original: Binary 1.
Revised: Binary 1.

Original: Read the sentence.
Revised: Read the sentence.

Original: Paid attention to the shaded words: if the overall sentiment of these words was clear I assumed the model would classify them accordingly.
Revised: Paid attention to the shaded words: if the overall sentiment of these words was clear I assumed the model would classify them accordingly.

Original: Otherwise I tried to consider how easy it would be for the model to understand the compositional meaning of the sentence assuming it will make mistakes at phenomena involving ironies or comparsions to proper names etc. 
Revised: Otherwise I tried to consider how easy it would be for the model to understand the compositional meaning of the sentence assuming it will make mistakes at phenomena involving ironies or comparsions to proper names etc. none Binary logical none

Original: [Table]
Revised: Hateval

Original: Keywords
Revised: Keywords, the sentimental polarity of the sentence lime

Original: [Table] only look at highlighted [Table]
Revised: Hateval only look at highlighted words none

Revised: Hateval logical lime

Original: Sent 
Revised: Sent Len

Original: [Table]
Revised: Haven’t got the faintest idea.

Original: Haven’t
Revised: none Sent Len

Original: [Table]
Revised: I was very lost in this task.

Original: I coud not ﬁnd topics in the sentences so I tried to focus whether sentences contained similar words guessing that these would be mapped to the same class... 
Revised: I coud not ﬁnd topics in the sentences so I tried to focus whether sentences contained similar words guessing that these would be mapped to the same class... lime

Original: [Table]
Revised: Tense no clue none Tense 1st Person 1 Person vs 2nd Person Multiple participants