Original: D O U SERS B ENEFIT F ROM I NTERPRETABLE V ISION ? A U SER S TUDY , B ASELINE , A ND D ATASET
Revised: D O U SERS B ENEFIT F ROM I NTERPRETABLE V ISION ? A U SER S TUDY , B ASELINE , A ND D ATASET

Original: [Table]
Revised: [Table]

Original: A BSTRACT
Revised: A BSTRACT

Original: A variety of methods exist to explain image classiﬁcation models.
Revised: A variety of methods exist to explain image classiﬁcation models.

Original: However, it remains unclear whether they provide any beneﬁt to users over simply comparing various inputs and the model’s respective predictions.
Revised: However, it remains unclear whether they provide any beneﬁt to users over simply comparing various inputs and the model’s respective predictions.

Original: We conducted a user study (N=240) to test how such a baseline explanation technique performs against concept-based and counterfactual explanations.
Revised: We conducted a user study (N=240) to test how such a baseline explanation technique performs against concept-based and counterfactual explanations.

Original: To this end, we contribute a synthetic dataset generator capable of biasing individual attributes and quantifying their relevance to the model.
Revised: To this end, we contribute a synthetic dataset generator capable of biasing individual attributes and quantifying their relevance to the model.

Original: In a study, we assess if participants can identify the relevant set of attributes compared to the ground-truth.
Revised: In a study, we assess if participants can identify the relevant set of attributes compared to the ground-truth.

Original: Our results show that the baseline outperformed concept-based explanations.
Revised: Our results show that the baseline outperformed concept-based explanations.

Original: Counterfactual explanations from an invertible neural network performed similarly as the baseline.
Revised: Counterfactual explanations from an invertible neural network performed similarly as the baseline.

Original: Still, they allowed users to identify some attributes more accurately.
Revised: Still, they allowed users to identify some attributes more accurately.

Original: Our results highlight the importance of measuring how well users can reason about biases of a model, rather than solely relying on technical evaluations or proxy tasks.
Revised: Our results highlight the importance of measuring how well users can reason about biases of a model, rather than solely relying on technical evaluations or proxy tasks.

Original: We open-source our study and dataset so it can serve as a blue-print for future studies.
Revised: We open-source our study and dataset so it can serve as a blue-print for future studies.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks have been widely adopted in many domains.
Revised: Deep neural networks have been widely adopted in many domains.

Original: Yet, for some applications, their use may be limited by how little we understand which features are relevant.
Revised: Yet, for some applications, their use may be limited by how little we understand which features are relevant.

Original: Whether engineer or user, insurance company, or regulatory body; all require reliable information about what the model has learned or why the model provides a certain output.
Revised: Whether engineer or user, insurance company, or regulatory body; all require reliable information about what the model has learned or why the model provides a certain output.

Original: Numerous methods have been proposed to explain deep neural networks (Gilpin et al., 2018; Molnar et al., 2020).
Revised: Numerous methods have been proposed to explain deep neural networks (Gilpin et al., 2018; Molnar et al., 2020).

Original: Ultimately, to evaluate whether such explanations are helpful, we need user studies (Doshi-Velez & Kim, 2017; Wortman Vaughan & Wallach, 2020).
Revised: Ultimately, to evaluate whether such explanations are helpful, we need user studies (Doshi-Velez & Kim, 2017; Wortman Vaughan & Wallach, 2020).

Original: In fact, some studies provided evidence that interpretable ML techniques may be helpful to ﬁnd biases or spurious correlations (Ribeiro et al., 2016b; Adebayo et al., 2020a).
Revised: In fact, some studies provided evidence that interpretable ML techniques may be helpful to ﬁnd biases or spurious correlations (Ribeiro et al., 2016b; Adebayo et al., 2020a).

Original: However, a substantial body of work shows that they may not be as helpful as claimed (Kaur et al., 2020; Alqaraawi et al., 2020;
Revised: However, a substantial body of work shows that they may not be as helpful as claimed (Kaur et al., 2020; Alqaraawi et al., 2020;

Original: Chu et al., 2020; Shen & Huang, 2020).
Revised: Chu et al., 2020; Shen & Huang, 2020).

Original: Consequently, it seems that in real-world applications, biases are often found by simply inspecting the model’s predictions rather than applying interpretable ML.
Revised: Consequently, it seems that in real-world applications, biases are often found by simply inspecting the model’s predictions rather than applying interpretable ML.

Original: A recent example is the Twitter image cropping algorithm: it was the users who discovered that it favored white people over people of color (Yee et al., 2021).
Revised: A recent example is the Twitter image cropping algorithm: it was the users who discovered that it favored white people over people of color (Yee et al., 2021).

Original: In this work, we ask: do modern interpretability methods enable users to discover biases better than by simply inspecting input/output pairs?
Revised: In this work, we ask: do modern interpretability methods enable users to discover biases better than by simply inspecting input/output pairs?

Original: To investigate this question empirically, we ﬁrst propose T WO 4T WO : a synthetic dataset depicting two abstract animals.
Revised: To investigate this question empirically, we ﬁrst propose T WO 4T WO : a synthetic dataset depicting two abstract animals.

Original: Its data-generating factors can be correlated with the binary target class, thereby creating arbitrarily strong biases.
Revised: Its data-generating factors can be correlated with the binary target class, thereby creating arbitrarily strong biases.

Original: We designed a baseline explanation technique for bias discovery using only the model’s output: input images are arranged in a grid grouped by the model’s logit predictions.
Revised: We designed a baseline explanation technique for bias discovery using only the model’s output: input images are arranged in a grid grouped by the model’s logit predictions.

Original: This design allows a user to inspect all attributes that potentially predict the target class.
Revised: This design allows a user to inspect all attributes that potentially predict the target class.

Original: In an initial user study (N=50), we validated that participants were struggling to ﬁnd both biases contained in our dataset using this technique.
Revised: In an initial user study (N=50), we validated that participants were struggling to ﬁnd both biases contained in our dataset using this technique.

Original: Hence, more elaborate methods can improve over the baseline on this dataset.
Revised: Hence, more elaborate methods can improve over the baseline on this dataset.

Original: In the main study (N=240), we compared the baseline against two state-of-the-art explanations: automatically-discovered concepts and counterfactual interpolations generated with an invertible neural network.
Revised: In the main study (N=240), we compared the baseline against two state-of-the-art explanations: automatically-discovered concepts and counterfactual interpolations generated with an invertible neural network.

Original: [Figure]
Revised: [Figure]

Original: We found that none of these explanations outperformed the baseline, even though some features were identiﬁed more accurately with counterfactuals.
Revised: We found that none of these explanations outperformed the baseline, even though some features were identiﬁed more accurately with counterfactuals.

Original: The textual justiﬁcations of participants revealed several usability issues in all methods.
Revised: The textual justiﬁcations of participants revealed several usability issues in all methods.

Original: This highlights the necessity to validate any claims about the beneﬁts of explanation techniques in user studies.
Revised: This highlights the necessity to validate any claims about the beneﬁts of explanation techniques in user studies.

Original: This work represents substantial empirical novelty and signiﬁcance in the ﬁeld of interpretable ML:
Revised: This work represents substantial empirical novelty and signiﬁcance in the ﬁeld of interpretable ML:

Original: • The T WO 4T WO dataset generator provides control over features and biases.
Revised: • The T WO 4T WO dataset generator provides control over features and biases.

Original: It is designed speciﬁcally for human subject studies and to challenge existing interpretability approaches, • Methods to quantify ground-truth feature importance when the data-generating process is known, • A study design that provides a uniﬁed approach to evaluating interpretable vision methods on the task of bias discovery.
Revised: It is designed speciﬁcally for human subject studies and to challenge existing interpretability approaches, • Methods to quantify ground-truth feature importance when the data-generating process is known, • A study design that provides a uniﬁed approach to evaluating interpretable vision methods on the task of bias discovery.

Original: It is suitable for lay-users and includes several measures to ensure high-quality crowd-sourced responses, • A strong and simple baseline explanation technique using only the model output, which we propose as a benchmark for future studies, • We open-source our dataset, explanation techniques, model, study design, including instructions and videos to support replicating our results as well as adapting our design to other explanation techniques.
Revised: It is suitable for lay-users and includes several measures to ensure high-quality crowd-sourced responses, • A strong and simple baseline explanation technique using only the model output, which we propose as a benchmark for future studies, • We open-source our dataset, explanation techniques, model, study design, including instructions and videos to support replicating our results as well as adapting our design to other explanation techniques.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Interpretable ML for Vision Different explanation approaches have been proposed: saliency maps (Bach et al., 2015; Ancona et al., 2018; Sundararajan et al., 2017), example-based explanations (Cai et al., 2019), counterfactual examples (Singla et al., 2020), activation-concept approaches (Kim et al., 2018), or models with built-in interpretability (Chen et al., 2019; Brendel & Bethge, 2018).
Revised: Interpretable ML for Vision Different explanation approaches have been proposed: saliency maps (Bach et al., 2015; Ancona et al., 2018; Sundararajan et al., 2017), example-based explanations (Cai et al., 2019), counterfactual examples (Singla et al., 2020), activation-concept approaches (Kim et al., 2018), or models with built-in interpretability (Chen et al., 2019; Brendel & Bethge, 2018).

Original: For a detailed review about the ﬁeld, we refer to (Gilpin et al., 2018; Molnar et al., 2020).
Revised: For a detailed review about the ﬁeld, we refer to (Gilpin et al., 2018; Molnar et al., 2020).

Original: Our work focuses on counterfactual explanations and automatically-discovered concepts.
Revised: Our work focuses on counterfactual explanations and automatically-discovered concepts.

Original: Counterfactual explanations are samples that change the model output, e.g., ﬂip the output class (Wachter et al., 2018).
Revised: Counterfactual explanations are samples that change the model output, e.g., ﬂip the output class (Wachter et al., 2018).

Original: We generated counterfactuals with invertible neural networks (INNs) (Jacobsen et al., 2018;
Revised: We generated counterfactuals with invertible neural networks (INNs) (Jacobsen et al., 2018;

Original: Kingma & Dhariwal, 2018).
Revised: Kingma & Dhariwal, 2018).

Original: This approach has recently gained momentum (Hvilshøj et al., 2021;
Revised: This approach has recently gained momentum (Hvilshøj et al., 2021;

Original: Dombrowski et al., 2021; Mackowiak et al., 2020).
Revised: Dombrowski et al., 2021; Mackowiak et al., 2020).

Original: Previous works have also used GANs and VAEs for counterfactual generation (Goyal et al., 2019; Mertes et al., 2020; Sauer & Geiger, 2021;
Revised: Previous works have also used GANs and VAEs for counterfactual generation (Goyal et al., 2019; Mertes et al., 2020; Sauer & Geiger, 2021;

Original: Singla et al., 2020; Liu et al., 2019; Baumgartner et al., 2018; Chang et al., 2019).
Revised: Singla et al., 2020; Liu et al., 2019; Baumgartner et al., 2018; Chang et al., 2019).

Original: The main advantage of using INNs for counterfactuals is that the generative function is perfectly aligned with the forward function, as an analytic inverse exists.
Revised: The main advantage of using INNs for counterfactuals is that the generative function is perfectly aligned with the forward function, as an analytic inverse exists.

Original: Concepts represent abstract properties, which can be used to explain a model.
Revised: Concepts represent abstract properties, which can be used to explain a model.

Original: For example, the classiﬁcation of an image as ”zebra” could be explained by a pronounced similarity to the ”stripe” concept.
Revised: For example, the classiﬁcation of an image as ”zebra” could be explained by a pronounced similarity to the ”stripe” concept.

Original: This similarity is determined by the dot product of the network’s internal activations with a concept vector.
Revised: This similarity is determined by the dot product of the network’s internal activations with a concept vector.

Original: TCAV (Kim et al., 2018) required manually deﬁned concepts.
Revised: TCAV (Kim et al., 2018) required manually deﬁned concepts.

Original: Recent works proposed to discover concepts automatically (Ghorbani et al., 2019; Zhang et al., 2021).
Revised: Recent works proposed to discover concepts automatically (Ghorbani et al., 2019; Zhang et al., 2021).

Original: [Figure]
Revised: [Figure]

Original: Peeky
Revised: Peeky

Original: Stretchy
Revised: Stretchy

Original: User Studies for Interpretability Previous works with the task of bias discovery have mainly evaluated saliency maps and used datasets with a single, simple bias, e.g. background Adebayo et al.
Revised: User Studies for Interpretability Previous works with the task of bias discovery have mainly evaluated saliency maps and used datasets with a single, simple bias, e.g. background Adebayo et al.

Original: (2020a)
Revised: (2020a)

Original: ; Ribeiro et al.
Revised: ; Ribeiro et al.

Original: (2016a) or image watermarks Kim et al.
Revised: (2016a) or image watermarks Kim et al.

Original: User studies for concept-based methods tested only the accessibility of the explanations by asking users to assign images to a concept (Zhang et al., 2021; Ghorbani et al., 2019).
Revised: User studies for concept-based methods tested only the accessibility of the explanations by asking users to assign images to a concept (Zhang et al., 2021; Ghorbani et al., 2019).

Original: Counterfactual explanations have been evaluated by Mertes et al.
Revised: Counterfactual explanations have been evaluated by Mertes et al.

Original: (2020) on a forward-prediction task.
Revised: (2020) on a forward-prediction task.

Original: We thus believe that we are the ﬁrst to extensively test counterfactual-based and concept-based explanations on bias discovery using a challenging dataset.
Revised: We thus believe that we are the ﬁrst to extensively test counterfactual-based and concept-based explanations on bias discovery using a challenging dataset.

Original: Recently, a study on exemplary-based explanations focused on understanding internal activations of a neural network (Borowski et al., 2020).
Revised: Recently, a study on exemplary-based explanations focused on understanding internal activations of a neural network (Borowski et al., 2020).

Original: It showed that for this task, examples could be more beneﬁcial than complex feature visualizations (Olah et al., 2017).
Revised: It showed that for this task, examples could be more beneﬁcial than complex feature visualizations (Olah et al., 2017).

Original: Similarly, there is evidence that participants often rely on model predictions rather than on explanations (Alqaraawi et al., 2020; Adebayo et al., 2020a).
Revised: Similarly, there is evidence that participants often rely on model predictions rather than on explanations (Alqaraawi et al., 2020; Adebayo et al., 2020a).

Original: Synthetic Datasets for Interpretable Vision Datasets with known ground-truth biases have been proposed before.
Revised: Synthetic Datasets for Interpretable Vision Datasets with known ground-truth biases have been proposed before.

Original: BAM is an artiﬁcial dataset (Yang & Kim, 2019) where spurious background correlations are introduced by pasting segmented objects on different textures, e.g. dogs on bamboo forests.
Revised: BAM is an artiﬁcial dataset (Yang & Kim, 2019) where spurious background correlations are introduced by pasting segmented objects on different textures, e.g. dogs on bamboo forests.

Original: However, the resulting images are unsuitable for user studies as they look artiﬁcial and make it easy for participants to suspect that the background is important.
Revised: However, the resulting images are unsuitable for user studies as they look artiﬁcial and make it easy for participants to suspect that the background is important.

Original: Additionally, it would be difﬁcult to introduce more than one bias.
Revised: Additionally, it would be difﬁcult to introduce more than one bias.

Original: A limitation that also the synthetic dataset in (Chen et al., 2018) shares.
Revised: A limitation that also the synthetic dataset in (Chen et al., 2018) shares.

Original: The synthetic dataset created by Arras et al.
Revised: The synthetic dataset created by Arras et al.

Original: (2021) created a dataset to technically evaluate saliency methods on a visual question answering task technically.
Revised: (2021) created a dataset to technically evaluate saliency methods on a visual question answering task technically.

Original: T WO 4T WO is the ﬁrst dataset designed explicitly for human subject evaluations.
Revised: T WO 4T WO is the ﬁrst dataset designed explicitly for human subject evaluations.

Original: To the best of our knowledge, we provide the ﬁrst uniﬁed approach to evaluate interpretable vision on a bias-discovery task.
Revised: To the best of our knowledge, we provide the ﬁrst uniﬁed approach to evaluate interpretable vision on a bias-discovery task.

Original:  T WO 4T WO : D ATASETS W ITH K NOWN F EATURE I MPORTANCE Dataset Description
Revised:  T WO 4T WO : D ATASETS W ITH K NOWN F EATURE I MPORTANCE Dataset Description

Original: Datasets generated with T WO 4T WO consist of two abstract animal classes, called Peeky and Stretchy .
Revised: Datasets generated with T WO 4T WO consist of two abstract animal classes, called Peeky and Stretchy .

Original: Both consist of eight blocks: four for the spine and four for the legs.
Revised: Both consist of eight blocks: four for the spine and four for the legs.

Original: For both animals, one pair of legs is always at an extended position.
Revised: For both animals, one pair of legs is always at an extended position.

Original: The other pair moves parallel to the spine inward and outward.
Revised: The other pair moves parallel to the spine inward and outward.

Original: The attribute legs’ position , a scalar in [0,1], controls the position.
Revised: The attribute legs’ position , a scalar in [0,1], controls the position.

Original: At a value of 0.5, the pair of legs are at the same vertical position as the last block of the spine.
Revised: At a value of 0.5, the pair of legs are at the same vertical position as the last block of the spine.

Original: Peekies have a leg position ≤ 0 .
Revised: Peekies have a leg position ≤ 0 .

Original: 52 which means legs are moved mostly inwards to the body center.
Revised: 52 which means legs are moved mostly inwards to the body center.

Original: In the same fashion, Stretchies are extended outwards, legs’ position ≥ 0 .
Revised: In the same fashion, Stretchies are extended outwards, legs’ position ≥ 0 .

Original: We added some ambiguity to ensure a model has an incentive to use possible biases.
Revised: We added some ambiguity to ensure a model has an incentive to use possible biases.

Original: Therefore, Peekies and Stretchies are equally likely for a legs’ position between 0.48 and 0.52.
Revised: Therefore, Peekies and Stretchies are equally likely for a legs’ position between 0.48 and 0.52.

Original: It is also difﬁcult for humans to tell if the legs are outward or inwards in this range.
Revised: It is also difﬁcult for humans to tell if the legs are outward or inwards in this range.

Original: Besides the legs’ position, the dataset has the following parameters: body posture (bending and three rotation angles), position , animal color (from red to blue), blocks’ shape (from cubes to spheres), and background color (from red to blue).
Revised: Besides the legs’ position, the dataset has the following parameters: body posture (bending and three rotation angles), position , animal color (from red to blue), blocks’ shape (from cubes to spheres), and background color (from red to blue).

Original: Each can be changed arbitrarily and continuously (See Appendix Table 5).
Revised: Each can be changed arbitrarily and continuously (See Appendix Table 5).

Original: When designing the dataset, we wanted to ensure that (1) participants can become experts within a few minutes of training, (2) it allows for the creation of multiple biases that are difﬁcult to ﬁnd, and (3) that it provides a challenge for existing interpretability methods.
Revised: When designing the dataset, we wanted to ensure that (1) participants can become experts within a few minutes of training, (2) it allows for the creation of multiple biases that are difﬁcult to ﬁnd, and (3) that it provides a challenge for existing interpretability methods.

Original: Goal (1) is met as participants can be instructed using only a few examples (see the tutorial video in Appendix C).
Revised: Goal (1) is met as participants can be instructed using only a few examples (see the tutorial video in Appendix C).

Original: The high number of controllable attributes achieve Goal (2).
Revised: The high number of controllable attributes achieve Goal (2).

Original: We biased the attributes such that they do not stand out, which we validated in the ﬁrst user study.
Revised: We biased the attributes such that they do not stand out, which we validated in the ﬁrst user study.

Original: Goal (3) is met by spatially overlapping attributes and long-
Revised: Goal (3) is met by spatially overlapping attributes and long-

Original: Legs' Position 0.
Revised: Legs' Position 0.

Original: B a c k g r o un d
Revised: B a c k g r o un d

Original: Legs' Position 0.
Revised: Legs' Position 0.

Original: S h a p e
Revised: S h a p e

Original: Legs' Position 0.
Revised: Legs' Position 0.

Original: C o l o r range image dependencies.
Revised: C o l o r range image dependencies.

Original: Spatially overlapping attributes, like color and shape, directly challenge saliency map explanations.
Revised: Spatially overlapping attributes, like color and shape, directly challenge saliency map explanations.

Original: Long-range image dependencies, like the legs’ positions relative to the spine, can not be explained when analyzing patches separately as done in (Chen et al., 2019; Brendel & Bethge, 2018).
Revised: Long-range image dependencies, like the legs’ positions relative to the spine, can not be explained when analyzing patches separately as done in (Chen et al., 2019; Brendel & Bethge, 2018).

Original: Both properties are common in real-world datasets: For example, race and gender in facial datasets are encoded by spatially overlapping features.
Revised: Both properties are common in real-world datasets: For example, race and gender in facial datasets are encoded by spatially overlapping features.

Original: Long-range image dependencies are particularly relevant for pose estimation and visual reasoning (Johnson et al., 2017).
Revised: Long-range image dependencies are particularly relevant for pose estimation and visual reasoning (Johnson et al., 2017).

Original: Introducing Biases For our studies’ dataset, we sampled the block’s shape in a non-predictive biased fashion.
Revised: Introducing Biases For our studies’ dataset, we sampled the block’s shape in a non-predictive biased fashion.

Original: This means that for legs’ positions that clearly showed a Peeky [0, 0.45] most blocks were rather cubic, while for legs’ positions that clearly showed a Stretchy [0.55, 1] most blocks were rather round.
Revised: This means that for legs’ positions that clearly showed a Peeky [0, 0.45] most blocks were rather cubic, while for legs’ positions that clearly showed a Stretchy [0.55, 1] most blocks were rather round.

Original: However, for the legs’ positions between [0.45, 0.55] the blocks shape was uniformly distributed.
Revised: However, for the legs’ positions between [0.45, 0.55] the blocks shape was uniformly distributed.

Original: In particular, in the even narrower interval [0.48, 0.52] where a classiﬁer can only be as good as random guessing, the block’s shape does not provide any additional information about the target class.
Revised: In particular, in the even narrower interval [0.48, 0.52] where a classiﬁer can only be as good as random guessing, the block’s shape does not provide any additional information about the target class.

Original: In Figure 3, we show the joint distribution of the block’s shape and legs’ position.
Revised: In Figure 3, we show the joint distribution of the block’s shape and legs’ position.

Original: We sampled the animals’ color to be predictive for the target class.
Revised: We sampled the animals’ color to be predictive for the target class.

Original: At the small interval where the legs overlap [0.48; 0.52], we distributed the animal color to provide additional class information.
Revised: At the small interval where the legs overlap [0.48; 0.52], we distributed the animal color to provide additional class information.

Original: Stretchies were more likely to be red, and Peekies were more likely to be blue.
Revised: Stretchies were more likely to be red, and Peekies were more likely to be blue.

Original: Outside of this centered interval, the color gradually became uniformly distributed (see Figure 3).
Revised: Outside of this centered interval, the color gradually became uniformly distributed (see Figure 3).

Original: Hence, color was more equally distributed than the shape, making the color bias harder to detect visually.
Revised: Hence, color was more equally distributed than the shape, making the color bias harder to detect visually.

Original: The remaining attributes, background color and body posture, were sampled independently of the class, and we expected our model to ignore them.
Revised: The remaining attributes, background color and body posture, were sampled independently of the class, and we expected our model to ignore them.

Original: Measuring Ground-Truth Feature Importance
Revised: Measuring Ground-Truth Feature Importance

Original: Even if a dataset contains biases, it is unclear how relevant they will be to a neural network after training.
Revised: Even if a dataset contains biases, it is unclear how relevant they will be to a neural network after training.

Original: Feature importance also depends on the network architecture, the optimization process, and even the weight initialization.
Revised: Feature importance also depends on the network architecture, the optimization process, and even the weight initialization.

Original: As T WO 4T WO allows us to change any parameter in isolation, we can directly compare the model prediction between two images that differ in only one parameter.
Revised: As T WO 4T WO allows us to change any parameter in isolation, we can directly compare the model prediction between two images that differ in only one parameter.

Original: For these two images, we measured both the median absolute logit change and also for how many samples the predicted class was ﬂipped.
Revised: For these two images, we measured both the median absolute logit change and also for how many samples the predicted class was ﬂipped.

Original: Both measures quantify how inﬂuential each parameter is (see Table 1).
Revised: Both measures quantify how inﬂuential each parameter is (see Table 1).

Original: As expected, the legs’ position had a strong inﬂuence on the prediction.
Revised: As expected, the legs’ position had a strong inﬂuence on the prediction.

Original: The model relied more on animal color than on the blocks’ shape, which is expected as the color contains additional information about the class.
Revised: The model relied more on animal color than on the blocks’ shape, which is expected as the color contains additional information about the class.

Original: Surprisingly, the prediction ﬂip for unrelated attributes such as background was only slightly lower than for blocks’ shape.
Revised: Surprisingly, the prediction ﬂip for unrelated attributes such as background was only slightly lower than for blocks’ shape.

Original: To analyze this further, we calculated a linear ﬁt for each parameter change to the logit change.
Revised: To analyze this further, we calculated a linear ﬁt for each parameter change to the logit change.

Original: We reported the coefﬁcient of determination R 2 , which indicates how much of the variance in the prediction can be explained linearly by the analyzed property.
Revised: We reported the coefﬁcient of determination R 2 , which indicates how much of the variance in the prediction can be explained linearly by the analyzed property.

Original: While the unrelated properties sometimes ﬂip a prediction, the direction of that ﬂip is random ( R 2 ≈ 0).
Revised: While the unrelated properties sometimes ﬂip a prediction, the direction of that ﬂip is random ( R 2 ≈ 0).

Original: In contrast, the biased parameters inﬂuence predictions in a directed fashion, with animal color ( R 2 =0.751) being clearly more directed than blocks’ shape ( R 2 =0.307).
Revised: In contrast, the biased parameters inﬂuence predictions in a directed fashion, with animal color ( R 2 =0.751) being clearly more directed than blocks’ shape ( R 2 =0.307).

Original:  M ODEL AND E VALUATED M ETHODS
Revised:  M ODEL AND E VALUATED M ETHODS

Original: As discussed in section 3, T WO 4T WO was designed to challenge existing interpretability methods, e.g., saliency map explanations and patch-based models.
Revised: As discussed in section 3, T WO 4T WO was designed to challenge existing interpretability methods, e.g., saliency map explanations and patch-based models.

Original: We selected two methods that might provide how often the model’s prediction changes the sign when changing the attribute.
Revised: We selected two methods that might provide how often the model’s prediction changes the sign when changing the attribute.

Original: The Mean Logit
Revised: The Mean Logit

Original: Change reports the median of the absolute change in logit values.
Revised: Change reports the median of the absolute change in logit values.

Original: The R 2 score is calculated on an ordinary least squares from the changes of each factor to the changes in the model’s logit.
Revised: The R 2 score is calculated on an ordinary least squares from the changes of each factor to the changes in the model’s logit.

Original: For more attributes, see Appendix Table 3.
Revised: For more attributes, see Appendix Table 3.

Original: [Table]
Revised: [Table]

Original: the user with the necessary information: counterfactuals generated with an invertible neural network (INN) and concept-based explanations (Zhang et al., 2021).
Revised: the user with the necessary information: counterfactuals generated with an invertible neural network (INN) and concept-based explanations (Zhang et al., 2021).

Original: INN Counterfactuals We trained an INN using both a supervised and an unsupervised objective (Dinh et al., 2016; 2015).
Revised: INN Counterfactuals We trained an INN using both a supervised and an unsupervised objective (Dinh et al., 2016; 2015).

Original: To predict the target class, the model ﬁrst applies the forward function ϕ to map a data point x to a feature vector z
Revised: To predict the target class, the model ﬁrst applies the forward function ϕ to map a data point x to a feature vector z

Original: = ϕ ( x ) .
Revised: = ϕ ( x ) .

Original: Then, a linear classiﬁer takes those features z and predicts the logit score f ( x ) =
Revised: Then, a linear classiﬁer takes those features z and predicts the logit score f ( x ) =

Original: w T z + b .
Revised: w T z + b .

Original: Any input can be reconstructed from the feature vector by applying the inverse function x = ϕ − 1 ( z ) .
Revised: Any input can be reconstructed from the feature vector by applying the inverse function x = ϕ − 1 ( z ) .

Original: The model has a test accuracy of 96 . 7% .
Revised: The model has a test accuracy of 96 . 7% .

Original: Further details can be found in Appendix A.2.
Revised: Further details can be found in Appendix A.2.

Original: The baseline and concept techniques are also applied to this model.
Revised: The baseline and concept techniques are also applied to this model.

Original: To create a counterfactual example ˜ x for a data point x , we can exploit the linearity of the classiﬁer.
Revised: To create a counterfactual example ˜ x for a data point x , we can exploit the linearity of the classiﬁer.

Original: Moving along the weight vector w , i.e., adding w to the features z , changes the model’s prediction.
Revised: Moving along the weight vector w , i.e., adding w to the features z , changes the model’s prediction.

Original: By controlling the step size with a scalar α , we can directly quantify the change in the logit value ∆ y =
Revised: By controlling the step size with a scalar α , we can directly quantify the change in the logit value ∆ y =

Original: α w T w .
Revised: α w T w .

Original: The modiﬁed feature vector z + α w can be inverted back to the input domain, resulting in a counterfactual ˜ x
Revised: The modiﬁed feature vector z + α w can be inverted back to the input domain, resulting in a counterfactual ˜ x

Original: = ϕ − 1 ( z + α w ) which visualizes the changes introduced by a step α w in z -space.
Revised: = ϕ − 1 ( z + α w ) which visualizes the changes introduced by a step α w in z -space.

Original: The INN’s explanations are visualized in a grid where each row shows a single counterfactual interpolation (see Figure 1b).
Revised: The INN’s explanations are visualized in a grid where each row shows a single counterfactual interpolation (see Figure 1b).

Original: Automatically-discovered Concepts We adapted the NMF approach of Zhang et al.
Revised: Automatically-discovered Concepts We adapted the NMF approach of Zhang et al.

Original: (2021) to our speciﬁc network architecture.
Revised: (2021) to our speciﬁc network architecture.

Original: Because the network’s internal representations also contain negative values, we used matrix factorization instead of NMF.
Revised: Because the network’s internal representations also contain negative values, we used matrix factorization instead of NMF.

Original: We generated the concepts using layer 342 (from a total of 641 layers).
Revised: We generated the concepts using layer 342 (from a total of 641 layers).

Original: The layer has a feature map resolution of 8x8.
Revised: The layer has a feature map resolution of 8x8.

Original: This choice represents a trade-off between enough spatial resolution and high-level information.
Revised: This choice represents a trade-off between enough spatial resolution and high-level information.

Original: We ran the matrix factorization with 10 components and selected the ﬁve components that correlated most with the logit score ( r is in the range [0.21, 0.34]).
Revised: We ran the matrix factorization with 10 components and selected the ﬁve components that correlated most with the logit score ( r is in the range [0.21, 0.34]).

Original: Our presentation of concept-based explanations was very similar to (Zhang et al., 2021): we visualized concepts with ﬁve exemplary images per row and highlighted regions corresponding to a concept.
Revised: Our presentation of concept-based explanations was very similar to (Zhang et al., 2021): we visualized concepts with ﬁve exemplary images per row and highlighted regions corresponding to a concept.

Original: Since our classiﬁer is binary, a negative contribution for Stretchy actually means a positive contribution for Peeky.
Revised: Since our classiﬁer is binary, a negative contribution for Stretchy actually means a positive contribution for Peeky.

Original: Hence, we could have characterized a concept as more Peeky and more Stretchy , to make the design similar to the other two explanation techniques.
Revised: Hence, we could have characterized a concept as more Peeky and more Stretchy , to make the design similar to the other two explanation techniques.

Original: However, as the concepts did not strongly correlate with the model’s output, presenting them as class-related could confuse participants: a more Peeky column would have contained some images showing Stretchies and vice versa.
Revised: However, as the concepts did not strongly correlate with the model’s output, presenting them as class-related could confuse participants: a more Peeky column would have contained some images showing Stretchies and vice versa.

Original: Thus, we presented them separately in two consecutive rows (See Figure 1c).
Revised: Thus, we presented them separately in two consecutive rows (See Figure 1c).

Original: Presenting concepts in this fashion gives them a fair chance in the study because participants rated the relevance of each attribute for the model rather than for each class separately.
Revised: Presenting concepts in this fashion gives them a fair chance in the study because participants rated the relevance of each attribute for the model rather than for each class separately.

Original:  H UMAN S UBJECT S TUDY
Revised:  H UMAN S UBJECT S TUDY

Original: We share the view of Doshi-Velez & Kim (2017) and Wortman Vaughan & Wallach (2020) that user-testing of explanation techniques is a crucial but challenging endeavor.
Revised: We share the view of Doshi-Velez & Kim (2017) and Wortman Vaughan & Wallach (2020) that user-testing of explanation techniques is a crucial but challenging endeavor.

Original: As our second main contribution, we propose and conduct a user study based on the Two4Two dataset which can act as a blue-print for future investigations.
Revised: As our second main contribution, we propose and conduct a user study based on the Two4Two dataset which can act as a blue-print for future investigations.

Original: Our design has been iterated in over ten pilot studies and proposes solutions to common problems that arise when evaluating explanation techniques on crowd-sourcing platforms with lay participants.
Revised: Our design has been iterated in over ten pilot studies and proposes solutions to common problems that arise when evaluating explanation techniques on crowd-sourcing platforms with lay participants.

Original: 5. D ESIGN CONSIDERATIONS
Revised: 5. D ESIGN CONSIDERATIONS

Original: Data without Prior Domain Knowlege We speciﬁcally designed the Two4Two dataset to avoid overburdening participants, as might be the case with other types of data.
Revised: Data without Prior Domain Knowlege We speciﬁcally designed the Two4Two dataset to avoid overburdening participants, as might be the case with other types of data.

Original: Within a few minutes, participants can easily become domain experts.
Revised: Within a few minutes, participants can easily become domain experts.

Original: Since the data is unknown to them prior to the study, we avoid introducing any prior domain knowledge as a confounding factor, which can be an issue (Alqaraawi et al., 2020).
Revised: Since the data is unknown to them prior to the study, we avoid introducing any prior domain knowledge as a confounding factor, which can be an issue (Alqaraawi et al., 2020).

Original: Manageable but not Oversimpliﬁed Tasks
Revised: Manageable but not Oversimpliﬁed Tasks

Original: We propose the task of bias-discovery : participants had to rate features as either relevant or irrelevant to a model.
Revised: We propose the task of bias-discovery : participants had to rate features as either relevant or irrelevant to a model.

Original: The task directly reﬂects users’ perception of feature importance.
Revised: The task directly reﬂects users’ perception of feature importance.

Original: Furthermore, bias-discovery has the advantage of being suitable for lay participants.
Revised: Furthermore, bias-discovery has the advantage of being suitable for lay participants.

Original: At the same time, it is also grounded in the model’s behavior.
Revised: At the same time, it is also grounded in the model’s behavior.

Original: This is an advantage over tasks used in several previous studies, which only evaluated whether explanations were accessible to users, e.g. by identifying the target property smiling using image interpolations (Singla et al., 2020) or assigning images to a concept class (Zhang et al., 2021; Ghorbani et al., 2019).
Revised: This is an advantage over tasks used in several previous studies, which only evaluated whether explanations were accessible to users, e.g. by identifying the target property smiling using image interpolations (Singla et al., 2020) or assigning images to a concept class (Zhang et al., 2021; Ghorbani et al., 2019).

Original: However, these tasks are an oversimpliﬁcation and cannot measure any insights the users gained about the model.
Revised: However, these tasks are an oversimpliﬁcation and cannot measure any insights the users gained about the model.

Original: In contrast, Alqaraawi et al.
Revised: In contrast, Alqaraawi et al.

Original: (2020) employed the task of forward prediction of a neural network.
Revised: (2020) employed the task of forward prediction of a neural network.

Original: This requires substantial model understanding and is very challenging, as reﬂected by the participants’ low accuracy.
Revised: This requires substantial model understanding and is very challenging, as reﬂected by the participants’ low accuracy.

Original: Assessing trust in a human-in-the-loop task , despite its realistic setting, has the disadvantage that trust is inﬂuenced by many factors which are difﬁcult to control for (Lee & See, 2004; Springer et al., 2017).
Revised: Assessing trust in a human-in-the-loop task , despite its realistic setting, has the disadvantage that trust is inﬂuenced by many factors which are difﬁcult to control for (Lee & See, 2004; Springer et al., 2017).

Original: Another approach is to asks participants to assess whether a model is ﬁt for deployment (Ribeiro et al., 2016b; Adebayo et al., 2020b).
Revised: Another approach is to asks participants to assess whether a model is ﬁt for deployment (Ribeiro et al., 2016b; Adebayo et al., 2020b).

Original: However, in our own pilots studies, users deemed a model ﬁt for deployment even if they knew it was biased.
Revised: However, in our own pilots studies, users deemed a model ﬁt for deployment even if they knew it was biased.

Original: Baseline Explanation Technique To quantify whether an explanation is beneﬁcial for users
Revised: Baseline Explanation Technique To quantify whether an explanation is beneﬁcial for users

Original: , it must be compared to an alternative explanation.
Revised: , it must be compared to an alternative explanation.

Original: In this work, we argue that a very simple and reasonable alternative for users is to inspect the model’s logits assigned to a set of input images.
Revised: In this work, we argue that a very simple and reasonable alternative for users is to inspect the model’s logits assigned to a set of input images.

Original: We designed such a baseline explanation as shown in Figure 1a.
Revised: We designed such a baseline explanation as shown in Figure 1a.

Original: After several design iterations, we settled for a visually dense image grid with 5 columns sorted by the logit score, each column covering 20% of the logit values.
Revised: After several design iterations, we settled for a visually dense image grid with 5 columns sorted by the logit score, each column covering 20% of the logit values.

Original: The columns were labeled very certain for Peeky/Stretchy, certain for Peeky/Stretchy, and as unsure.
Revised: The columns were labeled very certain for Peeky/Stretchy, certain for Peeky/Stretchy, and as unsure.

Original: Pilot studies showed that participants’ attention is limited.
Revised: Pilot studies showed that participants’ attention is limited.

Original: We thus decided to display a total of 50 images, i.e. an image grid of 10 rows.
Revised: We thus decided to display a total of 50 images, i.e. an image grid of 10 rows.

Original: The number of images was held constant between explanation techniques to ensure the same amount of visual information and a fair comparison.
Revised: The number of images was held constant between explanation techniques to ensure the same amount of visual information and a fair comparison.

Original: In this work, we focused on binary classiﬁcations.
Revised: In this work, we focused on binary classiﬁcations.

Original: For a multi-class setting, one could adapt the baseline by contrasting one class verses another class.
Revised: For a multi-class setting, one could adapt the baseline by contrasting one class verses another class.

Original: High Response Quality We took extensive measures to ensure participants understood their task and the explanation techniques.
Revised: High Response Quality We took extensive measures to ensure participants understood their task and the explanation techniques.

Original: Participants were required to watch three professionally-spoken tutorial videos, each under four minutes long.
Revised: Participants were required to watch three professionally-spoken tutorial videos, each under four minutes long.

Original: The videos explained, on a high level, the Two4Two dataset, machine learning and how to use an assigned explanation technique to discover relevant features.
Revised: The videos explained, on a high level, the Two4Two dataset, machine learning and how to use an assigned explanation technique to discover relevant features.

Original: To avoid inﬂuencing participants, we prototyped idealized explanations using images from
Revised: To avoid inﬂuencing participants, we prototyped idealized explanations using images from

Original: T WO 4T WO .
Revised: T WO 4T WO .

Original: The explanations showed different biases than those in the study.
Revised: The explanations showed different biases than those in the study.

Original: Each video was followed by a written summary and set of multiple choice comprehension questions After failing such a test once, participants could study the video and summary again.
Revised: Each video was followed by a written summary and set of multiple choice comprehension questions After failing such a test once, participants could study the video and summary again.

Original: When failing a test for a second time, participants were excluded from the study.
Revised: When failing a test for a second time, participants were excluded from the study.

Original: We also excluded participants if their written answers reﬂected a serious misunderstanding of the task, indicated by very short answers copied for all attributes or reasoning that is very different from the tutorial.
Revised: We also excluded participants if their written answers reﬂected a serious misunderstanding of the task, indicated by very short answers copied for all attributes or reasoning that is very different from the tutorial.

Original: We recruited participants from
Revised: We recruited participants from

Original: Proliﬁc who are ﬂuent in English, hold an academic degree and have an approval rate of ≥ 90% .
Revised: Proliﬁc who are ﬂuent in English, hold an academic degree and have an approval rate of ≥ 90% .

Original: To ensure they are also motivated, we compensated them with an average hourly pay of £11.45 which included a bonus of £0.40 per correct answer.
Revised: To ensure they are also motivated, we compensated them with an average hourly pay of £11.45 which included a bonus of £0.40 per correct answer.

Original: 5. E XPERIMENTAL D ESIGN
Revised: 5. E XPERIMENTAL D ESIGN

Original: We conducted two online user studies.
Revised: We conducted two online user studies.

Original: Before starting the data collection, we formulated our hypotheses, chose appropriate statistical tests, and pre-registered our studies (see Appendix D).
Revised: Before starting the data collection, we formulated our hypotheses, chose appropriate statistical tests, and pre-registered our studies (see Appendix D).

Original: This way, we follow the gold-standard of deﬁning the statistical analysis before the data collection, thus ensuring that our statistical results are reliable (Cockburn et al., 2018).
Revised: This way, we follow the gold-standard of deﬁning the statistical analysis before the data collection, thus ensuring that our statistical results are reliable (Cockburn et al., 2018).

Original: The ﬁrst study (N=50) analyzed whether the task was challenging enough that other methods could potentially improve over the baseline.
Revised: The ﬁrst study (N=50) analyzed whether the task was challenging enough that other methods could potentially improve over the baseline.

Original: We tested if at least one bias in our model (either the animal’s color or the blocks’ shape) was difﬁcult to ﬁnd using the baseline technique.
Revised: We tested if at least one bias in our model (either the animal’s color or the blocks’ shape) was difﬁcult to ﬁnd using the baseline technique.

Original: Consequently, we used a within-subject design.
Revised: Consequently, we used a within-subject design.

Original: [Table]
Revised: [Table]

Original: In the second study (N=240)
Revised: In the second study (N=240)

Original: [Table]
Revised: [Table]

Original: we evaluated the two explanation techniques described in Section 4 against the baseline using a between-subjects design.
Revised: we evaluated the two explanation techniques described in Section 4 against the baseline using a between-subjects design.

Original: Participants were randomly, but equally assigned to one of the explanation techniques.
Revised: Participants were randomly, but equally assigned to one of the explanation techniques.

Original: We speciﬁed two directed hypotheses.
Revised: We speciﬁed two directed hypotheses.

Original: We expected participants in the INN condition to perform better than those in baseline, because the baseline does not clearly highlight relevant features, whereas interpolations highlight features in isolation.
Revised: We expected participants in the INN condition to perform better than those in baseline, because the baseline does not clearly highlight relevant features, whereas interpolations highlight features in isolation.

Original: We expected participants viewing concepts to perform worse than those in the baseline, due to their inability to highlight spatially overlapping features.
Revised: We expected participants viewing concepts to perform worse than those in the baseline, due to their inability to highlight spatially overlapping features.

Original: For both studies
Revised: For both studies

Original: [Table]
Revised: [Table]

Original: participants completed a tutorial phase ﬁrst.
Revised: participants completed a tutorial phase ﬁrst.

Original: Using their assigned explanations
Revised: Using their assigned explanations

Original: [Table]
Revised: [Table]

Original: they then assessed the relevance of ﬁve attributes: legs’ position relative to the spine, animal color, background, rotation or bending, and blocks’ shape.
Revised: they then assessed the relevance of ﬁve attributes: legs’ position relative to the spine, animal color, background, rotation or bending, and blocks’ shape.

Original: The questions were formulated as: ”How relevant is < attribute > for the system?”
Revised: The questions were formulated as: ”How relevant is < attribute > for the system?”

Original: [Table]
Revised: [Table]

Original: and participants had to choose between irrelevant or relevant .
Revised: and participants had to choose between irrelevant or relevant .

Original: The percentage of correct answers (accuracy) served as our primary metric.
Revised: The percentage of correct answers (accuracy) served as our primary metric.

Original: Participants also had to write a short, fully-sentenced justiﬁcation for their answers.
Revised: Participants also had to write a short, fully-sentenced justiﬁcation for their answers.

Original: For links to the study, see
Revised: For links to the study, see

Original: Appendix C.
Revised: Appendix C.

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: [Table]
Revised: [Table]

Original: #Correct
Revised: #Correct

Original: P r opo r t i on
Revised: P r opo r t i on

Original: BASE
Revised: BASE

Original: CON
Revised: CON

Original: INN
Revised: INN

Original: Data Exclusions As stated in the preregistration, we automatically excluded all participants who withdrew their consent, failed one of the comprehension questions twice, skipped a video, or exceeded Proliﬁc’s time limit for completion.
Revised: Data Exclusions As stated in the preregistration, we automatically excluded all participants who withdrew their consent, failed one of the comprehension questions twice, skipped a video, or exceeded Proliﬁc’s time limit for completion.

Original: If a participant was excluded, a new participant’s place was made available until the pre-registered number of completed responses was reached.
Revised: If a participant was excluded, a new participant’s place was made available until the pre-registered number of completed responses was reached.

Original: We excluded 63 study respondents for the ﬁrst study, and 145 for the second study in this fashion.
Revised: We excluded 63 study respondents for the ﬁrst study, and 145 for the second study in this fashion.

Original: We ensured that all participants were naive about the dataset.
Revised: We ensured that all participants were naive about the dataset.

Original: Once they participated in a study, they were blacklisted for future studies.
Revised: Once they participated in a study, they were blacklisted for future studies.

Original: For completed studies, two annotators independently marked the participants’ written answers and excluded those with copy and paste answers or indications of grave misunderstandings of the instructions.
Revised: For completed studies, two annotators independently marked the participants’ written answers and excluded those with copy and paste answers or indications of grave misunderstandings of the instructions.

Original: Participants were labeled as: include , unsure , or exclude .
Revised: Participants were labeled as: include , unsure , or exclude .

Original: Both anotators had an agreement of κ = 0 .
Revised: Both anotators had an agreement of κ = 0 .

Original: 545 for the ﬁrst study and κ = 0 .
Revised: 545 for the ﬁrst study and κ = 0 .

Original: 643 for the second (measured include vs. unsure and exclude ).
Revised: 643 for the second (measured include vs. unsure and exclude ).

Original: Disagreements were solved by discussion.
Revised: Disagreements were solved by discussion.

Original: In total, we excluded 7 participants from the ﬁrst study (14%) and 48 participants from the second study (20%).
Revised: In total, we excluded 7 participants from the ﬁrst study (14%) and 48 participants from the second study (20%).

Original: First Study For the accepted 43 participants, we used two-sided exact McNemar tests on their answers about the relevance of the legs position compared with animal color (ﬁrst test) and background (second test).
Revised: First Study For the accepted 43 participants, we used two-sided exact McNemar tests on their answers about the relevance of the legs position compared with animal color (ﬁrst test) and background (second test).

Original: Participants found the color bias less often than the legs’ positions ( P < 0.0001).
Revised: Participants found the color bias less often than the legs’ positions ( P < 0.0001).

Original: The success rate for the color attribute was 49% vs. 86% for legs.
Revised: The success rate for the color attribute was 49% vs. 86% for legs.

Original: The shape bias was not signiﬁcantly harder to ﬁnd than the legs’ positions and was identiﬁed correctly with 74% accuracy ( P =0.3036).
Revised: The shape bias was not signiﬁcantly harder to ﬁnd than the legs’ positions and was identiﬁed correctly with 74% accuracy ( P =0.3036).

Original: Hence, we conﬁrmed our hypothesis and concluded that other methods still have room for improvement over the baseline.
Revised: Hence, we conﬁrmed our hypothesis and concluded that other methods still have room for improvement over the baseline.

Original: Second Study
Revised: Second Study

Original: In the second study, we evaluated 192 valid participant responses (62 INN,
Revised: In the second study, we evaluated 192 valid participant responses (62 INN,

Original: BASE, 59 CON).
Revised: BASE, 59 CON).

Original: We expected data to be different from the normal distribution, and a Shapiro-Wilk test for all conditions conﬁrmed this ( P < 0 . 001 ).
Revised: We expected data to be different from the normal distribution, and a Shapiro-Wilk test for all conditions conﬁrmed this ( P < 0 . 001 ).

Original: We depict the number of correct answers per condition in Figure 4.
Revised: We depict the number of correct answers per condition in Figure 4.

Original: A Kruskal-Wallis test showed a signiﬁcant differences in accuracy scores between conditions ( P < 0 . 001 ).
Revised: A Kruskal-Wallis test showed a signiﬁcant differences in accuracy scores between conditions ( P < 0 . 001 ).

Original: For focused comparisons, we used two Wilcoxon-rank-sum tests with Bonferroni correction to correct for multiple comparisons.
Revised: For focused comparisons, we used two Wilcoxon-rank-sum tests with Bonferroni correction to correct for multiple comparisons.

Original: The accuracy scores differed signiﬁcantly between the baseline and concept conditions ( P < 0 .
Revised: The accuracy scores differed signiﬁcantly between the baseline and concept conditions ( P < 0 .

Original: 001 , r =0.778).
Revised: 001 , r =0.778).

Original: The performance of participants using concepts was rather poor, with only 31.7% accuracy, considering that random answers would yield a score of 50%.
Revised: The performance of participants using concepts was rather poor, with only 31.7% accuracy, considering that random answers would yield a score of 50%.

Original: For concepts, not a single attribute surpassed the 50% barrier.
Revised: For concepts, not a single attribute surpassed the 50% barrier.

Original: We found no signiﬁcant difference when comparing the baseline and counterfactuals ( P =0.441, r =0.091).
Revised: We found no signiﬁcant difference when comparing the baseline and counterfactuals ( P =0.441, r =0.091).

Original: Their mean accuracies are close, with 80.8% for baseline and 84.5% for counterfactuals.
Revised: Their mean accuracies are close, with 80.8% for baseline and 84.5% for counterfactuals.

Original: INN counterfactuals helped users to discover the main attribute, legs’ position, ( P < 0.001) and color bias ( P =0.033) more reliably.
Revised: INN counterfactuals helped users to discover the main attribute, legs’ position, ( P < 0.001) and color bias ( P =0.033) more reliably.

Original: However, counterfactuals performed signiﬁcantly worse for the background attribute ( P =0.033), while for blocks’ shape and position we found no signiﬁcant difference (for both, P =1).
Revised: However, counterfactuals performed signiﬁcantly worse for the background attribute ( P =0.033), while for blocks’ shape and position we found no signiﬁcant difference (for both, P =1).

Original: Qualitative Results To understand how participants integrated the explanation techniques into their reasoning, we analyzed the textual answers of each feature qualitatively.
Revised: Qualitative Results To understand how participants integrated the explanation techniques into their reasoning, we analyzed the textual answers of each feature qualitatively.

Original: Two annotators ﬁrst applied open coding to the answers.
Revised: Two annotators ﬁrst applied open coding to the answers.

Original: They performed another pass of closed coding after agreeing on a subset of the relevant codes, on which the following analysis is based.
Revised: They performed another pass of closed coding after agreeing on a subset of the relevant codes, on which the following analysis is based.

Original: Overall, the participants perceived the task as challenging, as they expressed being unsure about their answers (N=71).
Revised: Overall, the participants perceived the task as challenging, as they expressed being unsure about their answers (N=71).

Original: We designed our image grid to show both possible classes and provide information about the model’s certainty.
Revised: We designed our image grid to show both possible classes and provide information about the model’s certainty.

Original: We found that many participants integrated this additional source of information into their reasoning.
Revised: We found that many participants integrated this additional source of information into their reasoning.

Original: This was especially prevalent in the baseline condition (N=51).
Revised: This was especially prevalent in the baseline condition (N=51).

Original: Participants particularly focused on the columns ’very certain Peeky’ and ’very certain Stretchy’, as well as on the column ’unsure’.
Revised: Participants particularly focused on the columns ’very certain Peeky’ and ’very certain Stretchy’, as well as on the column ’unsure’.

Original: While this may have helped conﬁrm or reject their own hypotheses, it sometimes led to confusion; for example, when an image that exhibited a pronounced leg position, and therefore could easily be identiﬁed as Peeky or Stretchy, was classiﬁed by the model as ’unsure’ (N=14).
Revised: While this may have helped conﬁrm or reject their own hypotheses, it sometimes led to confusion; for example, when an image that exhibited a pronounced leg position, and therefore could easily be identiﬁed as Peeky or Stretchy, was classiﬁed by the model as ’unsure’ (N=14).

Original: Across conditions, we also observed that participants expect that all images needed to support a hypothesis. ”
Revised: Across conditions, we also observed that participants expect that all images needed to support a hypothesis. ”

Original: The animals are in different colors, there are blue stretchy and also blue peeky animals, If the color was relevant peeky/stretchy would be in one color etc” (P73, BASE).
Revised: The animals are in different colors, there are blue stretchy and also blue peeky animals, If the color was relevant peeky/stretchy would be in one color etc” (P73, BASE).

Original: Across conditions, most participants that applied such deterministic reasoning failed to ﬁnd the color bias.
Revised: Across conditions, most participants that applied such deterministic reasoning failed to ﬁnd the color bias.

Original: In contrast, other participants applied more probabilistic reasoning, which helped them deal with such contradictions: ”Peeky is more likely to be blue in colour, whereas Stretchy is more likely to be pink.
Revised: In contrast, other participants applied more probabilistic reasoning, which helped them deal with such contradictions: ”Peeky is more likely to be blue in colour, whereas Stretchy is more likely to be pink.

Original: This is not always true (e.g. the shapes can be white in colour at either ends of the spectrum) but it might be somewhat relevant to help the system decide” (P197, INN).
Revised: This is not always true (e.g. the shapes can be white in colour at either ends of the spectrum) but it might be somewhat relevant to help the system decide” (P197, INN).

Original: Another observed strategy of participants was to reference how often they saw evidence for the relevance of a feature (N=35), which was very prevalent in the concepts condition (N=20).
Revised: Another observed strategy of participants was to reference how often they saw evidence for the relevance of a feature (N=35), which was very prevalent in the concepts condition (N=20).

Original: Especially concepts were rather difﬁcult for participants to interpret.
Revised: Especially concepts were rather difﬁcult for participants to interpret.

Original: A common issue was that they expected a relevant feature to be highlighted completely and consistently (N=38).
Revised: A common issue was that they expected a relevant feature to be highlighted completely and consistently (N=38).

Original: Several instances show that participants struggled to interpret how a highlighted region can explain the relevance of a feature, ”If this [the legs position] were relevant I would have expected the system to highlight only the portion of the image that contains the legs and spine.
Revised: Several instances show that participants struggled to interpret how a highlighted region can explain the relevance of a feature, ”If this [the legs position] were relevant I would have expected the system to highlight only the portion of the image that contains the legs and spine.

Original: (e.g. only the legs and one block of the spine at each end).
Revised: (e.g. only the legs and one block of the spine at each end).

Original: Instead, every image had minimally the entire animal highlighted” (P82, CON).
Revised: Instead, every image had minimally the entire animal highlighted” (P82, CON).

Original: Furthermore, spatially overlaping features were another cause of confusion: ”there are rows in which the animal is highlighted but not the background so it could be because of color, shape or rotation” (P157, CON)
Revised: Furthermore, spatially overlaping features were another cause of confusion: ”there are rows in which the animal is highlighted but not the background so it could be because of color, shape or rotation” (P157, CON)

Original: Participants erred more often for the background in the INN condition than for the baseline.
Revised: Participants erred more often for the background in the INN condition than for the baseline.

Original: We conducted an analysis to investigate this issue.
Revised: We conducted an analysis to investigate this issue.

Original: We found that 29 participants stated that they perceived no changes in the background of the counterfactuals and hence considered this feature irrelevant.
Revised: We found that 29 participants stated that they perceived no changes in the background of the counterfactuals and hence considered this feature irrelevant.

Original: Another 21 participants noted that they saw such a change, which let 12 of them to believe its a relevant feature. ”
Revised: Another 21 participants noted that they saw such a change, which let 12 of them to believe its a relevant feature. ”

Original: The background color changes in every case, it’s also a little subtle but it does” (P205).
Revised: The background color changes in every case, it’s also a little subtle but it does” (P205).

Original: Another 9 participants decided that the changes were too subtle to be relevant. ”
Revised: Another 9 participants decided that the changes were too subtle to be relevant. ”

Original: The background colour does not change an awful lot along each row, maybe in a couple of rows it changes slightly but I do not feel the change is signiﬁcant enough that this is a relevant factor in the machine decision” (P184).
Revised: The background colour does not change an awful lot along each row, maybe in a couple of rows it changes slightly but I do not feel the change is signiﬁcant enough that this is a relevant factor in the machine decision” (P184).

Original: Do Counterfactuals Highlight Irrelevant Features?
Revised: Do Counterfactuals Highlight Irrelevant Features?

Original: Indeed, subtle perceptual changes in background color are present (Figure 1b).
Revised: Indeed, subtle perceptual changes in background color are present (Figure 1b).

Original: To quantify these changes, we decided to use an objective observer: a convolutional neural network.
Revised: To quantify these changes, we decided to use an objective observer: a convolutional neural network.

Original: We trained a MobileNetV2 (Sandler et al., 2018) to predict
Revised: We trained a MobileNetV2 (Sandler et al., 2018) to predict

Original: −
Revised: −

Original: Logit
Revised: Logit

Original: L e g s ’
Revised: L e g s ’

Original: P o s iti on −
Revised: P o s iti on −

Original: Logit
Revised: Logit

Original: C
Revised: C

Original: o l o r −
Revised: o l o r −

Original: Logit
Revised: Logit

Original: B ac kg r ound −
Revised: B ac kg r ound −

Original: Logit
Revised: Logit

Original: S h a p e the parameter values of individual attributes of an image (e.g., background color, object color, etc.) using a completely unbiased version of T WO 4T WO .
Revised: S h a p e the parameter values of individual attributes of an image (e.g., background color, object color, etc.) using a completely unbiased version of T WO 4T WO .

Original: After training, the model could predict the parameter values almost exactly (MSE < 0.0022, see Table 7).
Revised: After training, the model could predict the parameter values almost exactly (MSE < 0.0022, see Table 7).

Original: We then used this model to evaluate the parameter values of counterfactual INN interpolations, each spanning 99% of the logit distribution.
Revised: We then used this model to evaluate the parameter values of counterfactual INN interpolations, each spanning 99% of the logit distribution.

Original: We visualize the predictions of MobileNetV2 in Figure 5.
Revised: We visualize the predictions of MobileNetV2 in Figure 5.

Original: All predictive properties (legs’ position, body color, blocks’ shape) are changed by the counterfactuals consistently.
Revised: All predictive properties (legs’ position, body color, blocks’ shape) are changed by the counterfactuals consistently.

Original: For the background, the changes are subtle but present.
Revised: For the background, the changes are subtle but present.

Original: We also quantiﬁed the change in parameters using the difference between the maximum and minimum predicted value per individual interpolation (See Table 6).
Revised: We also quantiﬁed the change in parameters using the difference between the maximum and minimum predicted value per individual interpolation (See Table 6).

Original: This supports the ﬁnding that relevant attributes change the most – legs’ position: 0.662; shapes: 0.624; color: 0.440.
Revised: This supports the ﬁnding that relevant attributes change the most – legs’ position: 0.662; shapes: 0.624; color: 0.440.

Original: The background changes less with 0.045, which seems enough to give some participants a false impression about its relevance.
Revised: The background changes less with 0.045, which seems enough to give some participants a false impression about its relevance.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Contributions We contribute a dataset with full control over the biases it contains and methods to verify the feature importances of a given model.
Revised: Contributions We contribute a dataset with full control over the biases it contains and methods to verify the feature importances of a given model.

Original: The dataset was speciﬁcally designed for user studies and we contribute a carefully crafted study design as a template for future empirical evaluations of interpretable vision.
Revised: The dataset was speciﬁcally designed for user studies and we contribute a carefully crafted study design as a template for future empirical evaluations of interpretable vision.

Original: Our design includes a simple, yet powerful baseline technique that relies on the model’s outputs only.
Revised: Our design includes a simple, yet powerful baseline technique that relies on the model’s outputs only.

Original: While interpretability methods had room to improve over the baseline, we showed that two state-of-the-art methods did not perform signiﬁcantly better.
Revised: While interpretability methods had room to improve over the baseline, we showed that two state-of-the-art methods did not perform signiﬁcantly better.

Original: Our results emphasize that any explanation technique needs to be evaluated in extensive user studies.
Revised: Our results emphasize that any explanation technique needs to be evaluated in extensive user studies.

Original: Limitations Due to budget constraints, we limited the number of factors in our experimental design (external vs. internal validity trade-off).
Revised: Limitations Due to budget constraints, we limited the number of factors in our experimental design (external vs. internal validity trade-off).

Original: Our study introduced a predictive bias for the animal’s color and a non-predictive bias for the blocks’ shape.
Revised: Our study introduced a predictive bias for the animal’s color and a non-predictive bias for the blocks’ shape.

Original: It remains unclear how our results may have changed for a different dataset conﬁguration: certain biases could exhibit different visual saliency.
Revised: It remains unclear how our results may have changed for a different dataset conﬁguration: certain biases could exhibit different visual saliency.

Original: It remains also left for future work to determine which visual interface design is optimal for a given method.
Revised: It remains also left for future work to determine which visual interface design is optimal for a given method.

Original: Furthermore, our study design restricted participants to make binary choices and provide textual justiﬁcations – limiting our understanding of the participants issues.
Revised: Furthermore, our study design restricted participants to make binary choices and provide textual justiﬁcations – limiting our understanding of the participants issues.

Original: Take-Aways We were surprised by the performance of the two tested techniques.
Revised: Take-Aways We were surprised by the performance of the two tested techniques.

Original: Users had problems interpreting the automatically-discovered concepts and could not identify the relevant attributes.
Revised: Users had problems interpreting the automatically-discovered concepts and could not identify the relevant attributes.

Original: As we expected, explaining spatially overlapping features by highlighting important regions limited the concepts’ expressiveness.
Revised: As we expected, explaining spatially overlapping features by highlighting important regions limited the concepts’ expressiveness.

Original: On the other hand, INN counterfactuals also did not perform signiﬁcantly better than the baseline.
Revised: On the other hand, INN counterfactuals also did not perform signiﬁcantly better than the baseline.

Original: Still, counterfactuals were more helpful to discover the strongest bias in the model.
Revised: Still, counterfactuals were more helpful to discover the strongest bias in the model.

Original: However, some participants rated the relevance of the background incorrectly, as slight changes in the interpolations were still salient enough.
Revised: However, some participants rated the relevance of the background incorrectly, as slight changes in the interpolations were still salient enough.

Original: It is therefore important for future work to develop counterfactuals that alter only relevant attributes.
Revised: It is therefore important for future work to develop counterfactuals that alter only relevant attributes.

Original: We presented a user study on a synthetic dataset.
Revised: We presented a user study on a synthetic dataset.

Original: We believe that the results also have implications for natural image data.
Revised: We believe that the results also have implications for natural image data.

Original: When we created Two4Two, our objective was to translate challenges faced on ”real” computer vision data (like spatially overlapping features) into an abstract domain.
Revised: When we created Two4Two, our objective was to translate challenges faced on ”real” computer vision data (like spatially overlapping features) into an abstract domain.

Original: Although some properties of photorealistic datasets are lost in this abstraction, a method performing poorly on
Revised: Although some properties of photorealistic datasets are lost in this abstraction, a method performing poorly on

Original: T WO 4T WO would likely not perform well on a natural dataset with spatially overlapping features.
Revised: T WO 4T WO would likely not perform well on a natural dataset with spatially overlapping features.

Original: Outlook The user study was a reality check for two interpretability methods.
Revised: Outlook The user study was a reality check for two interpretability methods.

Original: Such studies can guide technical innovations by identifying areas where users still struggle with current explanation methods.
Revised: Such studies can guide technical innovations by identifying areas where users still struggle with current explanation methods.

Original: They are laborious and expensive, but at the same time, they are crucial for future interpretability research.
Revised: They are laborious and expensive, but at the same time, they are crucial for future interpretability research.

Original: Future work could focus on creating a more realistic, controllable dataset, e.g. using augmented reality (Alhaija et al., 2018).
Revised: Future work could focus on creating a more realistic, controllable dataset, e.g. using augmented reality (Alhaija et al., 2018).

Original: By open-sourcing our videos, study, and code we encourage the community to take on the challenge to beat the simple baseline.
Revised: By open-sourcing our videos, study, and code we encourage the community to take on the challenge to beat the simple baseline.

Original:  E THICS S TATEMENT
Revised:  A CKNOWLEDGMENTS

Revised: We thank Jonas K¨ohler and Benjamin Wild for their feedback on the manuscript.

Revised: We also thank our reviewers for their time.

Revised: LS was supported by the Elsa-von-Neumann Scholarship of the state of

Revised: Berlin.

Revised: MS and PW were funded by the German Federal Ministry of Education and Research (BMBF) - NR 16DII113.

Revised: In addition, the work was partially funded by the the German Federal Ministry of Education and Research (BMBF), under grant number 16DHB4018.

Revised: We thank the Center for

Revised: Information Services and High Performance Computing (ZIH) at Dresden University of Technology and the HPC Service of ZEDAT, Freie Universit¨at Berlin, for generous allocations of computation time (Bennett et al., 2020).

Revised:  E THICS S TATEMENT

Original: In our human subject evaluation, it was important for us to pay crowd workers £11.45 per hour which is above UK minimum wage of £8.91.
Revised: In our human subject evaluation, it was important for us to pay crowd workers £11.45 per hour which is above UK minimum wage of £8.91.

Original: The crowd workers consented to the use of their answers in the study.
Revised: The crowd workers consented to the use of their answers in the study.

Original: No personal-data was collected.
Revised: No personal-data was collected.

Original: Our organization does not require an approval of online user studies through an ethics review board.
Revised: Our organization does not require an approval of online user studies through an ethics review board.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: Our work brings some additional challenges to reproducibility beyond compared to other machine learning research.
Revised: Our work brings some additional challenges to reproducibility beyond compared to other machine learning research.

Original: The results of the human subject study depend on the dataset, model and even the presentations in the video tutorials.
Revised: The results of the human subject study depend on the dataset, model and even the presentations in the video tutorials.

Original: We decided to tackle this challenge by ﬁrst open-sourcing our dataset, model, videos, code, and even the study itself.
Revised: We decided to tackle this challenge by ﬁrst open-sourcing our dataset, model, videos, code, and even the study itself.

Original: For now, we share the links to the study and videos in the Appendix.
Revised: For now, we share the links to the study and videos in the Appendix.

Original: We will share an machine-readable export of the study, the source-code, and the model with our reviewers via OpenReview once the discussion period start and will make them public at a later point in time.
Revised: We will share an machine-readable export of the study, the source-code, and the model with our reviewers via OpenReview once the discussion period start and will make them public at a later point in time.

Original: In total, we spent around 5000 GBP including prestudies, software licenses and our two studies in the paper.
Revised: In total, we spent around 5000 GBP including prestudies, software licenses, and our two studies in the paper.

Original: We estimate the costs of reproducing the studies in our work at around 2500 GBP excluding software licenses.
Revised: We estimate the costs of reproducing the studies in our work at around 2500 GBP excluding software licenses.

Original: For a study comparing only one condition against the baseline, we would estimate the costs to be around 1400 GBP.
Revised: For a study comparing only one condition against the baseline, we would estimate the costs to be around 1400 GBP.

Original: A A PPENDIX : T ECHNICAL D ETAILS A. 
Revised: A A PPENDIX : T ECHNICAL D ETAILS A.

Original: [Table]
Revised: [Table]

Original: Yes -
Revised: Yes -

Original: [Table]
Revised: [Table]

Original: See Figure
Revised: See Figure

Original: Yes
Revised: Yes

Original: Yes Shape [0, 1]
Revised: Yes Shape [0, 1]

Original: See Figure
Revised: See Figure

Original: Yes
Revised: Yes

Original: No
Revised: No

Original: [Table]
Revised: [Table]

Original: Uniform
Revised: Uniform

Original: No
Revised: No

Original: [Table]
Revised: [Table]

Original: Uniform
Revised: Uniform

Original: No
Revised: No

Original: [Table]
Revised: [Table]

Original: Uniform
Revised: Uniform

Original: No No
Revised: No No

Original: [Table]
Revised: [Table]

Original: Uniform No
Revised: Uniform No

Original: No
Revised: No

Original: [Table]
Revised: [Table]

Original: No No
Revised: No No

Original: [Table]
Revised: [Table]

Original: No
Revised: No

Original: [Table]
Revised: [Table]

Original: No No
Revised: No No

Original: [Table]
Revised: [Table]

Original: Attribute
Revised: Attribute

Original: [Table]
Revised: [Table]

Original: Shapes
Revised: Shapes

Original: [Table]
Revised: [Table]

Original: Background
Revised: Background

Original: [Table]
Revised: [Table]

Original: Rotation Pitch
Revised: Rotation Pitch

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Rotation Yaw
Revised: Rotation Yaw

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Rotation Roll
Revised: Rotation Roll

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Position X
Revised: Position X

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Position Y 0.
Revised: Position Y 0.

Original: [Figure]
Revised: [Figure]

Original: Logit
Revised: Logit

Original: L e g s ’
Revised: L e g s ’

Original: P o s iti on (a) −
Revised: P o s iti on (a) −

Original: Logit
Revised: Logit

Original: C o l o r (b) −
Revised: C o l o r (b) −

Original: Logit
Revised: Logit

Original: B ac kg r ound (c) −
Revised: B ac kg r ound (c) −

Original: Logit
Revised: Logit

Original: S h a p e (d) −
Revised: S h a p e (d) −

Original: Logit − 0.
Revised: Logit − 0.

Original: [Figure]
Revised: [Figure]

Original: e nd i ng (e) −
Revised: e nd i ng (e) −

Original: Logit −
Revised: Logit −

Original: R o t a ti on Y a w (f) −
Revised: R o t a ti on Y a w (f) −

Original: Logit − 0.
Revised: Logit − 0.

Original: P o s iti on X (g) −
Revised: P o s iti on X (g) −

Original: Logit − 0.
Revised: Logit − 0.

Original: − 0. − 0.
Revised: − 0. − 0.

Original: [Figure]
Revised: [Figure]

Original: s iti on Y (h)
Revised: s iti on Y (h)

Original: 1 import dataclasses 2 import numpy as np 3 import matplotlib.pyplot as plt 4 5 from two4two.blender import render 6 from two4two.bias import Sampler, Continouos 7 from two4two.scene_parameters import SceneParameters 8 9 @dataclasses.dataclass 10 class RotationBiasSampler(Sampler): """A rotation-biased sampler.
Revised: 1 import dataclasses 2 import numpy as np 3 import matplotlib.pyplot as plt 4 5 from two4two.blender import render 6 from two4two.bias import Sampler, Continouos 7 from two4two.scene_parameters import SceneParameters 8 9 @dataclasses.dataclass 10 class RotationBiasSampler(Sampler): """A rotation-biased sampler.

Original: The rotation is sampled conditionally depending on the object type.
Revised: The rotation is sampled conditionally depending on the object type.

Original: Positive rotations for peaky and negative rotations for stretchy.
Revised: Positive rotations for peaky and negative rotations for stretchy.

Original: """ 16obj_rotation_yaw: Continouos = dataclasses.field( default_factory=lambda
Revised: """ 16obj_rotation_yaw: Continouos = dataclasses.field( default_factory=lambda

Original: { ’peaky’: np.
Revised: { ’peaky’: np.

Original: .uniform
Revised: .uniform

Original: ), ’stretchy’:
Revised: ), ’stretchy’:

Original: np
Revised: np

Original: .uniform np images 24 sampler = RotationBiasSampler() 25 params =
Revised: .uniform np images 24 sampler = RotationBiasSampler() 25 params =

Original: [sampler.sample( for _ in range(4)] 26 for in render params): plt
Revised: [sampler.sample( for _ in range(4)] 26 for in render params): plt

Original: plt show()
Revised: plt show()

Original: A. A RCHITECTURE OF THE I NVERTIBLE N EURAL N ETWORK
Revised: A. A RCHITECTURE OF THE I NVERTIBLE N EURAL N ETWORK

Original: Our model is based on the Glow architecture (Kingma & Dhariwal, 2018) and contains 7 blocks.
Revised: Our model is based on the Glow architecture (Kingma & Dhariwal, 2018) and contains 7 blocks.

Original: A block is a collection of 32 ﬂow steps, followed by a down-sampling layer, and ends with a fade-out layer.
Revised: A block is a collection of 32 ﬂow steps, followed by a down-sampling layer, and ends with a fade-out layer.

Original: A single ﬂow step consists of actnorm , invertible 1 × 1 convolution and afﬁne coupling layer.
Revised: A single ﬂow step consists of actnorm , invertible 1 × 1 convolution and afﬁne coupling layer.

Original: The down-sampling keeps all dimensions, e.g. a shape of ( h, w, c ) becomes ( h/ 2 , w/ 2 , 4 c ) .
Revised: The down-sampling keeps all dimensions, e.g. a shape of ( h, w, c ) becomes ( h/ 2 , w/ 2 , 4 c ) .

Original: The fade-out layer maps removes half of the channels.
Revised: The fade-out layer maps removes half of the channels.

Original: The out-faded channels are than mapped to a standard normal distribution to compute the unsupervised loss.
Revised: The out-faded channels are than mapped to a standard normal distribution to compute the unsupervised loss.

Original: For generating counterfactuals, the out-faded values are not thrown away but rather stored to be used when computing the inverse.
Revised: For generating counterfactuals, the out-faded values are not thrown away but rather stored to be used when computing the inverse.

Original: The model is trained using a supervised loss and an unsupervised objective.
Revised: The model is trained using a supervised loss and an unsupervised objective.

Original: In total our model had 687 layers and 261 million parameters.
Revised: In total our model had 687 layers and 261 million parameters.

Original: The classiﬁer used the output of layer 641.
Revised: The classiﬁer used the output of layer 641.

Original: The remaining layers 642-687 were optimized using the standard unsupervised ﬂow objective.
Revised: The remaining layers 642-687 were optimized using the standard unsupervised ﬂow objective.

Original: For the ﬁrstlayers, we also trained on the classiﬁer’s supervised loss.
Revised: For the ﬁrstlayers, we also trained on the classiﬁer’s supervised loss.

Original: Let ϕ denote the ﬁrst 641 layers and µ : R n (cid:55)→ R n the last.
Revised: Let ϕ denote the ﬁrst 641 layers and µ : R n (cid:55)→ R n the last.

Original: We train ϕ both on a supervised loss from the classiﬁer f ( x ) and an unsupervised loss from matching the prior distribution N (0 , I ) and the log determinante of the Jacobian.
Revised: We train ϕ both on a supervised loss from the classiﬁer f ( x ) and an unsupervised loss from matching the prior distribution N (0 , I ) and the log determinante of the Jacobian.

Original: µ is only trained on the unsupervised loss:
Revised: µ is only trained on the unsupervised loss:

Original: [Equation]
Revised: [Equation]

Original: For the supervised loss L sup , we use the binary cross entropy.
Revised: For the supervised loss L sup , we use the binary cross entropy.

Original: As unsupervised loss L un , we use the commonly used standard ﬂow loss obtained from the change of variables trick
Revised: As unsupervised loss L un , we use the commonly used standard ﬂow loss obtained from the change of variables trick

Original: Dinh et al.
Revised: Dinh et al.

Original: The unsupervised loss ensures that inverting the function results in realistic looking images and can also be seen as a regularization.
Revised: The unsupervised loss ensures that inverting the function results in realistic looking images and can also be seen as a regularization.

Original: The layer 342 used for the concept explanations is an afﬁne coupling layer.
Revised: The layer 342 used for the concept explanations is an afﬁne coupling layer.

Original: [Table]
Revised: [Table]

Original: We used a MobileNet-V2 to predict the attributes values.
Revised: We used a MobileNet-V2 to predict the attributes values.

Original: The models test mean squared errors are denoted in Table
Revised: The models test mean squared errors are denoted in Table

Original: [Table]
Revised: [Table]

Original: B L INKS TO M ODEL , D ATASET AND S TUDY
Revised: B L INKS TO M ODEL , D ATASET AND S TUDY

Original: Model export: https://f002.backblazeb2.com/file/iclr2022/do_users_
Revised: Model export: https://f002.backblazeb2.com/file/iclr2022/do_users_

Original: benefit_from_interpretable_vision_model.tar.gz
Revised: benefit_from_interpretable_vision_model.tar.gz

Original: Unbiased dataset: https://f002.backblazeb2.com/file/iclr2022/two4two_ obj_color_and_spherical_finer_search_spherical_ uniform_0.33_uniform_0.15_unbiased.tar
Revised: Unbiased dataset: https://f002.backblazeb2.com/file/iclr2022/two4two_ obj_color_and_spherical_finer_search_spherical_ uniform_0.33_uniform_0.15_unbiased.tar

Original: Biased dataset: https://f002.backblazeb2.com/file/iclr2022/two4two_ obj_color_and_spherical_finer_search_spherical_ uniform_0.33_uniform_0.15.tar
Revised: Biased dataset: https://f002.backblazeb2.com/file/iclr2022/two4two_ obj_color_and_spherical_finer_search_spherical_ uniform_0.33_uniform_0.15.tar

Original: Export of study: https://f002.backblazeb2.com/file/iclr2022/ICLR2022_
Revised: Export of study: https://f002.backblazeb2.com/file/iclr2022/ICLR2022_

Original: Export_Do_Users_Benefit_From_Interpretable_Vision.qsf
Revised: Export_Do_Users_Benefit_From_Interpretable_Vision.qsf

Original: PDF print-out of the study: https://f002.backblazeb2.com/file/iclr2022/ICLR2022_
Revised: PDF print-out of the study: https://f002.backblazeb2.com/file/iclr2022/ICLR2022_

Original: Export_Do_Users_Benefit_From_Interpretable_Vision.pdf
Revised: Export_Do_Users_Benefit_From_Interpretable_Vision.pdf

Original: C U SER - STUDY L INKS AND V IDEOS
Revised: C U SER - STUDY L INKS AND V IDEOS

Original: The studies can be accessed on the Qualtrics platform (with anonymized consent form) under the following links:
Revised: The studies can be accessed on the Qualtrics platform (with anonymized consent form) under the following links:

Original: Baseline condition:
Revised: Baseline condition:

Original: https://wznbm.qualtrics.com/jfe/form/SV_ 7Umdmdaq8EHVRm INN condition:
Revised: https://wznbm.qualtrics.com/jfe/form/SV_ 7Umdmdaq8EHVRm INN condition:

Original: https://wznbm.qualtrics.com/jfe/form/SV_ dneHADG7BxjVurc Concepts condition:
Revised: https://wznbm.qualtrics.com/jfe/form/SV_ dneHADG7BxjVurc Concepts condition:

Original: https://wznbm.qualtrics.com/jfe/form/SV_
Revised: https://wznbm.qualtrics.com/jfe/form/SV_

Original: 0PWErBQmGL0lobk
Revised: 0PWErBQmGL0lobk

Original: The tutorial videos can be viewed under the following links:
Revised: The tutorial videos can be viewed under the following links:

Original: Introduction Tutorial for Peeky and Stretchy: https://f backblazeb2.com/file/iclr2022/Intro_
Revised: Introduction Tutorial for Peeky and Stretchy: https://f backblazeb2.com/file/iclr2022/Intro_

Original: Peeky_Stretchy.mp
Revised: Peeky_Stretchy.mp

Original: Second Introduction Tutorial for ML and Biases: https://f002.backblazeb2.com/file/iclr2022/Second_
Revised: Second Introduction Tutorial for ML and Biases: https://f002.backblazeb2.com/file/iclr2022/Second_

Original: Intro_ML.mp
Revised: Intro_ML.mp

Original: Tutorial for baseline condition: https://f002.backblazeb2.com/file/iclr2022/condition_
Revised: Tutorial for baseline condition: https://f002.backblazeb2.com/file/iclr2022/condition_

Original: BASE.mp
Revised: BASE.mp

Original: Tutorial for concept condition: https://f002.backblazeb2.com/file/iclr2022/condition_
Revised: Tutorial for concept condition: https://f002.backblazeb2.com/file/iclr2022/condition_

Original: CONCEPTS.mp
Revised: CONCEPTS.mp

Original: Tutorial for INN condition: https://f002.backblazeb2.com/file/iclr2022/condition_
Revised: Tutorial for INN condition: https://f002.backblazeb2.com/file/iclr2022/condition_

Original: INN.mp
Revised: INN.mp

Original: D U SER - STUDY P REREGISTRATION AND H YPOTHESIS
Revised: D U SER - STUDY P REREGISTRATION AND H YPOTHESIS

Original: The Preregistrations can also be viewed under the following URLs:
Revised: The Preregistrations can also be viewed under the following URLs:

Original: • Validation of T WO 4T WO : https://aspredicted.org/blind.php?x=/62X_ 15J • Study 2: Concepts vs. Baseline and INN vs. Baseline: https://aspredicted.org/ blind.php?x=/7XN_77P
Revised: • Validation of T WO 4T WO : https://aspredicted.org/blind.php?x=/62X_ 15J • Study 2: Concepts vs. Baseline and INN vs. Baseline: https://aspredicted.org/ blind.php?x=/7XN_77P

Original: We also paid the participants in both studies the more lucrative tariff included in the preregistration of study 2, e.g. for third comprehension task: 3.50GBP and so on.
Revised: We also paid the participants in both studies the more lucrative tariff included in the preregistration of study 2, e.g. for third comprehension task: 3.50GBP and so on.

Original: D. V ALIDATION OF T WO 4T WO
Revised: D. V ALIDATION OF T WO 4T WO

Original: 1) Have any data been collected for this study already?
Revised: 1) Have any data been collected for this study already?

Original: No, no data have been collected for this study yet.
Revised: No, no data have been collected for this study yet.

Original: What’s the main question being asked or hypothesis being tested in this study?
Revised: What’s the main question being asked or hypothesis being tested in this study?

Original: This study investigates whether users identify biases learned by a neural network.
Revised: This study investigates whether users identify biases learned by a neural network.

Original: The neural networks task is to discriminate between two abstract animals (”Peeky” and ”Stretchy”).
Revised: The neural networks task is to discriminate between two abstract animals (”Peeky” and ”Stretchy”).

Original: Each participant is presented with predictions of the system in a 10x5 image grid.
Revised: Each participant is presented with predictions of the system in a 10x5 image grid.

Original: After an initial tutorial phase, the participants have to ﬁnd biases in the model.
Revised: After an initial tutorial phase, the participants have to ﬁnd biases in the model.

Original: They do this by scoring different characteristics as relevant or irrelevant.
Revised: They do this by scoring different characteristics as relevant or irrelevant.

Original: The characteristics are: ”legs position relative to the spine (LEGS)”, ”object color (COLOR)”, ”background (BACK)”, ”rounded or rectangular shape of the blocks (SHAPE)”, and ”rotation and bending (ROT)”.
Revised: The characteristics are: ”legs position relative to the spine (LEGS)”, ”object color (COLOR)”, ”background (BACK)”, ”rounded or rectangular shape of the blocks (SHAPE)”, and ”rotation and bending (ROT)”.

Original: The main research question is whether we succeeded in creating a model that contains at least one bias that is hard to detect, i.e. either COLOR or SHAPE should be harder to detect than LEGS.
Revised: The main research question is whether we succeeded in creating a model that contains at least one bias that is hard to detect, i.e. either COLOR or SHAPE should be harder to detect than LEGS.

Original: HB: Participants can identify the biases in COLOR or SHAPE less frequently than LEGS.
Revised: HB: Participants can identify the biases in COLOR or SHAPE less frequently than LEGS.

Original: Describe the key dependent variable
Revised: Describe the key dependent variable

Original: (s) specifying how
Revised: (s) specifying how

Original: they will
Revised: they will

Original: be measured.
Revised: be measured.

Original: Participant will answer the following questions:
Revised: Participant will answer the following questions:

Original: • LEGS: How relevant is the legs position relative to the spine for the system?:
Revised: • LEGS: How relevant is the legs position relative to the spine for the system?:

Original: Relevant / Irrelevant • COLOR: How relevant is the color of the animal for the system?
Revised: Relevant / Irrelevant • COLOR: How relevant is the color of the animal for the system?

Original: Relevant / Irrelevant • BACK: How relevant is the background of the animal for the system?
Revised: Relevant / Irrelevant • BACK: How relevant is the background of the animal for the system?

Original: Relevant / Irrelevant • SHAPE: How relevant is the rounded or rectangular shape of the animal’s blocks for the system?
Revised: Relevant / Irrelevant • SHAPE: How relevant is the rounded or rectangular shape of the animal’s blocks for the system?

Original: Relevant / Irrelevant • ROT:
Revised: Relevant / Irrelevant • ROT:

Original: How relevant is the rotation and bending of the animal for the system?
Revised: How relevant is the rotation and bending of the animal for the system?

Original: Relevant / Irrelevant
Revised: Relevant / Irrelevant

Original: The ground truth answer is that LEGS, COLOR, SHAPE are relevant while BACK and ROT are irrelevant.
Revised: The ground truth answer is that LEGS, COLOR, SHAPE are relevant while BACK and ROT are irrelevant.

Original: Our ﬁrst dependent variable is the number of times the head position was selected as relevant.
Revised: Our ﬁrst dependent variable is the number of times the head position was selected as relevant.

Original: Our second dependent variable is the number of times the color of the animal was selected as relevant.
Revised: Our second dependent variable is the number of times the color of the animal was selected as relevant.

Original: Our third dependent variable is the number of times the rounded or rectangular shape of the animal’s blocks was selected as relevant.
Revised: Our third dependent variable is the number of times the rounded or rectangular shape of the animal’s blocks was selected as relevant.

Original: 4) How many and which conditions will participants be assigned to?
Revised: 4) How many and which conditions will participants be assigned to?

Original: Our study follows a within-subject design and has only one condition.
Revised: Our study follows a within-subject design and has only one condition.

Original: We ﬁrst show the participants introductory videos about the two abstract animals, the machine learning system, and some guidance on how to interpret the predictions of the system.
Revised: We ﬁrst show the participants introductory videos about the two abstract animals, the machine learning system, and some guidance on how to interpret the predictions of the system.

Original: Each video is accompanied by a written summary.
Revised: Each video is accompanied by a written summary.

Original: We then show the predictions of the system in a grid of images: 10 sorted rows of 5 images drawn from the validation set (50 original images).
Revised: We then show the predictions of the system in a grid of images: 10 sorted rows of 5 images drawn from the validation set (50 original images).

Original: Each of the ﬁve columns represents the neural netwoks’s logit range.
Revised: Each of the ﬁve columns represents the neural netwoks’s logit range.

Original: Similarly rated images are assigned to the same column.
Revised: Similarly rated images are assigned to the same column.

Original: 5) Specify exactly which analyses you will conduct to examine the main question/hypothesis.
Revised: 5) Specify exactly which analyses you will conduct to examine the main question/hypothesis.

Original: We will conduct two exact one-sided McNemar-tests with LEGS acting as our control: one between SHAPE and LEGS and a second between COLOR and LEGS.
Revised: We will conduct two exact one-sided McNemar-tests with LEGS acting as our control: one between SHAPE and LEGS and a second between COLOR and LEGS.

Original: We will use a one-sided test as we expect that SHAPE and COLOR are harder to identify.
Revised: We will use a one-sided test as we expect that SHAPE and COLOR are harder to identify.

Original: The signiﬁcance level of both tests will be Bonferroni adjusted to α = 0.025. 6)
Revised: The signiﬁcance level of both tests will be Bonferroni adjusted to α = 0.025. 6)

Original: Describe exactly how outliers will be deﬁned and handled, and your precise rule(s) for excluding observations.
Revised: Describe exactly how outliers will be deﬁned and handled, and your precise rule(s) for excluding observations.

Original: We reject participants with low effort responses or who failed to understand the dataset, machine learning concept, or explanation method.
Revised: We reject participants with low effort responses or who failed to understand the dataset, machine learning concept, or explanation method.

Original: We have implemented hard-coded exclusion criteria directly in the survey (implemented with Qulatrics and Proliﬁc).
Revised: We have implemented hard-coded exclusion criteria directly in the survey (implemented with Qulatrics and Proliﬁc).

Original: • did not ﬁnish experiment at all or in under 77 minutes • did not watch tutorial videos completely (there are 3 videos) or failed a multiple-choice comprehension test twice (there are four such tests), unless participants explicitly ask us to retake the study • using a device smaller than a tablet (min.
Revised: • did not ﬁnish experiment at all or in under 77 minutes • did not watch tutorial videos completely (there are 3 videos) or failed a multiple-choice comprehension test twice (there are four such tests), unless participants explicitly ask us to retake the study • using a device smaller than a tablet (min.

Original: 600 px in width or height) • provided answers about relevant characteristics in under 30 seconds • withdrawn data consent / returned task on Proliﬁc • circumvented Qualtrics protection against retaking the entire survey again (ﬁrst complete submission will be counted)
Revised: 600 px in width or height) • provided answers about relevant characteristics in under 30 seconds • withdrawn data consent / returned task on Proliﬁc • circumvented Qualtrics protection against retaking the entire survey again (ﬁrst complete submission will be counted)

Original: We do not plan to exclude any participants who passed all of the above criteria unless the qualitative answers reveal a serious misunderstanding of the study instructions that the multiple choice tests did not cover.
Revised: We do not plan to exclude any participants who passed all of the above criteria unless the qualitative answers reveal a serious misunderstanding of the study instructions that the multiple choice tests did not cover.

Original: We will report such exclusions in detail in the Appendix.
Revised: We will report such exclusions in detail in the Appendix.

Original: ) How many observations will be collected or what will determine sample size?
Revised: ) How many observations will be collected or what will determine sample size?

Original: No need to justify decision, but be precise about exactly how the number will be determined.participants from Proliﬁc with the background:
Revised: No need to justify decision, but be precise about exactly how the number will be determined.participants from Proliﬁc with the background:

Original: • Fluent in English • Hold an academic degree • Proliﬁc approval rate of at least 90% • Did not participate in pilot studies • Passed hard coded exclusion criteria (see 8).
Revised: • Fluent in English • Hold an academic degree • Proliﬁc approval rate of at least 90% • Did not participate in pilot studies • Passed hard coded exclusion criteria (see 8).

Original: We pay participants max.
Revised: We pay participants max.

Original: 8.00 GBP (6.00 GBP base salary + 2.00 GBP max bonus).
Revised: 8.00 GBP (6.00 GBP base salary + 2.00 GBP max bonus).

Original: For those failing any comprehension questions or not watching the video, we pay
Revised: For those failing any comprehension questions or not watching the video, we pay

Original: : • First comprehension task: no compensation • Second comprehension task: 0.5 GBP • Third comprehension task: 1.75 GBP • Failed to watch ﬁrst video: no compensation • Failed to watch second video: 1 GBP • Failed to watch third video: 2 GBP 8)
Revised: : • First comprehension task: no compensation • Second comprehension task: 0.5 GBP • Third comprehension task: 1.75 GBP • Failed to watch ﬁrst video: no compensation • Failed to watch second video: 1 GBP • Failed to watch third video: 2 GBP 8)

Original: Anything else you would like to pre-register?
Revised: Anything else you would like to pre-register?

Original: (e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?)
Revised: (e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?)

Original: We ask participants to answer three multiple choice comprehension tests in the form of true/false statements to ensure that they understood the task and the dataset.
Revised: We ask participants to answer three multiple choice comprehension tests in the form of true/false statements to ensure that they understood the task and the dataset.

Original: We also ask them to provide some free-text justiﬁcation of why they chose a relevant / irrelevant rating to the questions in Section 3.
Revised: We also ask them to provide some free-text justiﬁcation of why they chose a relevant / irrelevant rating to the questions in Section 3.

Original: D. S TUDY 2: C ONCEPTS VS . B ASELINE AND INN VS . B ASELINE
Revised: D. S TUDY 2: C ONCEPTS VS . B ASELINE AND INN VS . B ASELINE

Original: 1) Have any data been collected for this study already?
Revised: 1) Have any data been collected for this study already?

Original: No, no data have been collected for this study yet.
Revised: No, no data have been collected for this study yet.

Original: What’s the main question being asked or hypothesis being tested in this study?
Revised: What’s the main question being asked or hypothesis being tested in this study?

Original: This study investigates whether users identify biases learned by a neural network.
Revised: This study investigates whether users identify biases learned by a neural network.

Original: The neural networks task is to discriminate between two abstract animals (”Peeky” and ”Stretchy”).
Revised: The neural networks task is to discriminate between two abstract animals (”Peeky” and ”Stretchy”).

Original: Each participant is presented one of three different explanation methods: baseline (B), counterfactuals obtained using invertible neural networks (CF) and prototypes (P).
Revised: Each participant is presented one of three different explanation methods: baseline (B), counterfactuals obtained using invertible neural networks (CF) and prototypes (P).

Original: Each participant is randomly assigned to a method.
Revised: Each participant is randomly assigned to a method.

Original: After an initial tutorial phase, the participants have to ﬁnd biases in the model.
Revised: After an initial tutorial phase, the participants have to ﬁnd biases in the model.

Original: They do this by scoring different characteristics as relevant or irrelevant.
Revised: They do this by scoring different characteristics as relevant or irrelevant.

Original: The characteristics are: ”legs position relative to the spine (LEGS)”, ”object color (COLOR)”, ”background (BACK)”, ”rounded or rectangular shape of the blocks (SHAPE)”, and ”rotation and bending (ROT)”.
Revised: The characteristics are: ”legs position relative to the spine (LEGS)”, ”object color (COLOR)”, ”background (BACK)”, ”rounded or rectangular shape of the blocks (SHAPE)”, and ”rotation and bending (ROT)”.

Original: The main question of our study is whether the participants can correctly identify relevant and irrelevant attributes using these explanation methods (B, CF, P).
Revised: The main question of our study is whether the participants can correctly identify relevant and irrelevant attributes using these explanation methods (B, CF, P).

Original: This is reﬂected by two hypotheses:
Revised: This is reﬂected by two hypotheses:

Original: H1:
Revised: H1:

Original: Participants identify relevant and irrelevant attributes with less accuracy using P compared to B. H2: Participants identify relevant and irrelevant attributes with higher accuracy using CF compared to B. 3) Describe the key dependent variable(s) specifying how they will be measured.
Revised: Participants identify relevant and irrelevant attributes with less accuracy using P compared to B. H2: Participants identify relevant and irrelevant attributes with higher accuracy using CF compared to B. 3) Describe the key dependent variable(s) specifying how they will be measured.

Original: Participant will answer the following questions:
Revised: Participant will answer the following questions:

Original: • LEGS: How relevant is the legs position relative to the spine for the system?:
Revised: • LEGS: How relevant is the legs position relative to the spine for the system?:

Original: Relevant / Irrelevant • COLOR: How relevant is the color of the animal for the system?
Revised: Relevant / Irrelevant • COLOR: How relevant is the color of the animal for the system?

Original: Relevant / Irrelevant • BACK: How relevant is the background of the animal for the system?
Revised: Relevant / Irrelevant • BACK: How relevant is the background of the animal for the system?

Original: Relevant / Irrelevant • SHAPE: How relevant is the rounded or rectangular shape of the animal’s blocks for the system?
Revised: Relevant / Irrelevant • SHAPE: How relevant is the rounded or rectangular shape of the animal’s blocks for the system?

Original: Relevant / Irrelevant • ROT:
Revised: Relevant / Irrelevant • ROT:

Original: How relevant is the rotation and bending of the animal for the system?
Revised: How relevant is the rotation and bending of the animal for the system?

Original: Relevant / Irrelevant
Revised: Relevant / Irrelevant

Original: The ground truth answer is that LEGS, COLOR, SHAPE are relevant while BACK and ROT are irrelevant.
Revised: The ground truth answer is that LEGS, COLOR, SHAPE are relevant while BACK and ROT are irrelevant.

Original: Our dependent variable is the percentage of correctly
Revised: Our dependent variable is the percentage of correctly

Original: answered questions per participant (accuracy, which is computed as (true positives + true negatives)/number of total answers).
Revised: answered questions per participant (accuracy, which is computed as (true positives + true negatives)/number of total answers).

Original: 4) How many and which conditions will participants be assigned to?
Revised: 4) How many and which conditions will participants be assigned to?

Original: We run a between-subject study, with randomly but equally assigned participants to 1 of 3 conditions.
Revised: We run a between-subject study, with randomly but equally assigned participants to 1 of 3 conditions.

Original: We ﬁrst show introductory videos about the two abstract animals, the machine learning system, the explanation technique and some guidance on how to interpret the technique.
Revised: We ﬁrst show introductory videos about the two abstract animals, the machine learning system, the explanation technique and some guidance on how to interpret the technique.

Original: Each video is accompanied by a written summary.
Revised: Each video is accompanied by a written summary.

Original: We then show a grid of (10x5) images:
Revised: We then show a grid of (10x5) images:

Original: B: NN predictions explained with 10 sorted rows of 5 images drawn from the validation set ( original images).
Revised: B: NN predictions explained with 10 sorted rows of 5 images drawn from the validation set ( original images).

Original: Each of the ﬁve columns represents a score range.
Revised: Each of the ﬁve columns represents a score range.

Original: Similarly rated images are assigned to the same column.
Revised: Similarly rated images are assigned to the same column.

Original: 2.CF: Same grid layout as B, but the NN is explained by counterfactual interpolations.
Revised: 2.CF: Same grid layout as B, but the NN is explained by counterfactual interpolations.

Original: Each row contains interpolations which change the prediction of the NN to ﬁt the designated score.
Revised: Each row contains interpolations which change the prediction of the NN to ﬁt the designated score.

Original: Original images are used as starting points but are not shown.
Revised: Original images are used as starting points but are not shown.

Original: 3.P: We found concepts based on the work by (Zhang et al., 2020).
Revised: 3.P: We found concepts based on the work by (Zhang et al., 2020).

Original: Each row shows a set of relevant concepts.
Revised: Each row shows a set of relevant concepts.

Original: We only used concepts correlated with at least r=0.2 with the model logit values.
Revised: We only used concepts correlated with at least r=0.2 with the model logit values.

Original: In total, we display 10 rows where each row contains a concept.
Revised: In total, we display 10 rows where each row contains a concept.

Original: Each row contains a set of 5 example images for which the concept is relevant.
Revised: Each row contains a set of 5 example images for which the concept is relevant.

Original: (Zhang et al., 2020)
Revised: (Zhang et al., 2020)

Original: https://arxiv.org/abs/2006.
Revised: https://arxiv.org/abs/2006.

Original: Specify exactly which analyses you will conduct to examine the main question/
Revised: Specify exactly which analyses you will conduct to examine the main question/

Original: hypothesis
Revised: hypothesis

Original: We will compute the accuracy scores for each participant and then compare the accuracy scores between the conditions.
Revised: We will compute the accuracy scores for each participant and then compare the accuracy scores between the conditions.

Original: We expect the data to be non-normally distributed, and will test this assumption using a Shapiro-Wilk test with a signiﬁcance level of α = 0.05.
Revised: We expect the data to be non-normally distributed, and will test this assumption using a Shapiro-Wilk test with a signiﬁcance level of α = 0.05.

Original: If our assumption is true, we plan to conduct a Kruskal-Wallis test, followed by post-hoc analysis using Wilcoxon’s-rank-sum tests for focused comparison between the groups CF and B (expecting higher accuracy in CF) and P and B (expecting lower accuracy in P).
Revised: If our assumption is true, we plan to conduct a Kruskal-Wallis test, followed by post-hoc analysis using Wilcoxon’s-rank-sum tests for focused comparison between the groups CF and B (expecting higher accuracy in CF) and P and B (expecting lower accuracy in P).

Original: If the data is normally distributed, we will conduct a one-way ANOVA with planned contrasts, if the following assumptions of ANOVAs are met:
Revised: If the data is normally distributed, we will conduct a one-way ANOVA with planned contrasts, if the following assumptions of ANOVAs are met:

Original: • Homogeneity of the variance of the population (assessed with a Levene-Test with a signiﬁcance level of α = 0.05.)
Revised: • Homogeneity of the variance of the population (assessed with a Levene-Test with a signiﬁcance level of α = 0.05.)

Original: If the homogeneity of variance assumption of ANOVA is violated (assessed with a Levene-Test with a signiﬁcance level of α = 0.05.), we plan to perform Welch’s Anova. 6) Describe exactly how outliers will be deﬁned and handled, and your precise rule(s) for excluding observations.
Revised: If the homogeneity of variance assumption of ANOVA is violated (assessed with a Levene-Test with a signiﬁcance level of α = 0.05.), we plan to perform Welch’s Anova. 6) Describe exactly how outliers will be deﬁned and handled, and your precise rule(s) for excluding observations.

Original: We reject participants with low effort responses or who failed to understand the dataset, machine learning concept, or explanation method.
Revised: We reject participants with low effort responses or who failed to understand the dataset, machine learning concept, or explanation method.

Original: We have implemented hard-coded exclusion criteria directly in the survey (implemented with Qulatrics and Proliﬁc)
Revised: We have implemented hard-coded exclusion criteria directly in the survey (implemented with Qulatrics and Proliﬁc)

Original: • did not ﬁnish experiment at all or in under 77 minutes • did not watch the tutorial videos completely (there are 3 videos) or failed a multiple-choice comprehension test twice (there are four such tests), unless participants explicitly ask us to retake the study • using a device smaller than a tablet (min.
Revised: • did not ﬁnish experiment at all or in under 77 minutes • did not watch the tutorial videos completely (there are 3 videos) or failed a multiple-choice comprehension test twice (there are four such tests), unless participants explicitly ask us to retake the study • using a device smaller than a tablet (min.

Original: 600 px in width or height) • provided answers about relevant characteristics in under 30 seconds • withdrawn data consent / returned task on Proliﬁc • circumvented Qualtrics protection against retaking the entire survey again (ﬁrst complete submission will be counted)
Revised: 600 px in width or height) • provided answers about relevant characteristics in under 30 seconds • withdrawn data consent / returned task on Proliﬁc • circumvented Qualtrics protection against retaking the entire survey again (ﬁrst complete submission will be counted)

Original: We do not plan to exclude any participants who passed all of the above criteria unless the qualitative answers reveal a serious misunderstanding of the study instructions that the multiple choice tests did not cover.
Revised: We do not plan to exclude any participants who passed all of the above criteria unless the qualitative answers reveal a serious misunderstanding of the study instructions that the multiple choice tests did not cover.

Original: We will report such exclusions in detail in the Appendix.
Revised: We will report such exclusions in detail in the Appendix.

Original: 7) How many observations
Revised: 7) How many observations

Original: will be
Revised: will be

Original: collected
Revised: collected

Original: or what will determine sample size?
Revised: or what will determine sample size?

Original: No need to justify
Revised: No need to justify

Original: decision,
Revised: decision,

Original: but
Revised: but

Original: be precise about exactly how the number will be determined.
Revised: be precise about exactly how the number will be determined.

Original: 240 (80 per condition) participants from Proliﬁc with the background:
Revised: 240 (80 per condition) participants from Proliﬁc with the background:

Original: • Fluent in English •
Revised: • Fluent in English •

Original: First, we sample participants with an academic degree.
Revised: First, we sample participants with an academic degree.

Original: If we do not reach the desired participant number, which is likely given the limited availability of such subjects, we will supplement with participants with an academic degree in other subjects.
Revised: If we do not reach the desired participant number, which is likely given the limited availability of such subjects, we will supplement with participants with an academic degree in other subjects.

Original: All participants will be randomly and equally split into the 4 conditions.
Revised: All participants will be randomly and equally split into the 4 conditions.

Original: • Proliﬁc approval rate of at least 90% • Did not participate in pilot studies • Passed hard coded exclusion criteria (see 8).
Revised: • Proliﬁc approval rate of at least 90% • Did not participate in pilot studies • Passed hard coded exclusion criteria (see 8).

Original: We pay participants max.
Revised: We pay participants max.

Original: 6.50 GBP (4.50 GBP base salary + 2.00 GBP max bonus).
Revised: 6.50 GBP (4.50 GBP base salary + 2.00 GBP max bonus).

Original: For those failing any comprehension questions or not watching the video, we pay:
Revised: For those failing any comprehension questions or not watching the video, we pay:

Original: • First comprehension task: 0.5 GBP • Second comprehension task: 1.75 GBP
Revised: • First comprehension task: 0.5 GBP • Second comprehension task: 1.75 GBP

Original: • Third comprehension task: 3.50GBP • Failed to watch ﬁrst video: no compensation • Failed to watch second video: 1 GBP • Failed to watch third video: 2 GBP 8) Anything else you would like to pre-register?
Revised: • Third comprehension task: 3.50GBP • Failed to watch ﬁrst video: no compensation • Failed to watch second video: 1 GBP • Failed to watch third video: 2 GBP 8) Anything else you would like to pre-register?

Original: (e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?)
Revised: (e.g., secondary analyses, variables collected for exploratory purposes, unusual analyses planned?)

Original: In a previous study, we collected 50 responses for the baseline condition only (Preregistration #75056)).
Revised: In a previous study, we collected 50 responses for the baseline condition only (Preregistration #75056)).

Original: We do not plan to use the data for this study.
Revised: We do not plan to use the data for this study.

Original: We ask participants to answer three multiple choice comprehension tests in the form of true/false statements to ensure that they understood the task and the dataset.
Revised: We ask participants to answer three multiple choice comprehension tests in the form of true/false statements to ensure that they understood the task and the dataset.

Original: We also ask them to provide some free-text justiﬁcation of why they chose a relevant / irrelevant rating to the questions in Section 3.
Revised: We also ask them to provide some free-text justiﬁcation of why they chose a relevant / irrelevant rating to the questions in Section 3.

Original: Additionally, we ask the participants about their machine learning expertise level.
Revised: Additionally, we ask the participants about their machine learning expertise level.

Original: Participants can rate their expertise as: complete novice, some expertise, or expert in the topic.
Revised: Participants can rate their expertise as: complete novice, some expertise, or expert in the topic.

Original: We plan to use descriptive statistics to see how accuracies change per condition for each expertise level and how expertise was distributed within our sample.
Revised: We plan to use descriptive statistics to see how accuracies change per condition for each expertise level and how expertise was distributed within our sample.

Original: We are also planning a qualitative thematic analysis of the open-text questions in our survey via open and axial coding, with the aim of understanding how participants integrated explanations in their reasoning about the relevance of attributes.
Revised: We are also planning a qualitative thematic analysis of the open-text questions in our survey via open and axial coding, with the aim of understanding how participants integrated explanations in their reasoning about the relevance of attributes.