Original: B EYOND E XAMPLES : C ONSTRUCTING E XPLANATION S PACE FOR E XPLAINING P ROTOTYPES
Revised: B EYOND E XAMPLES : C ONSTRUCTING E XPLANATION S PACE FOR E XPLAINING P ROTOTYPES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: As deep learning has been successfully deployed in diverse applications, there is ever increasing need for explaining its decision.
Revised: As deep learning has been successfully deployed in diverse applications, there is ever increasing need for explaining its decision.

Original: Most of the existing methods produced explanations with a second model that explains the ﬁrst black-box model, but we propose an inherently interpretable model for faithful explanations.
Revised: Most of the existing methods produced explanations with a second model that explains a given black-box model, but we propose an inherently interpretable model for faithful explanation.

Original: Our method constructs an explanation space in which similarities in terms of humaninterpretable features at images share similar latent representations by using a variational autoencoder.
Revised: Our method constructs an explanation space in which similarities in terms of humaninterpretable features at images share similar latent representations by using a variational autoencoder.

Original: This explanation space provides additional explanations of the relationships, going beyond previous classiﬁcation networks that provide explanations by distances and learned prototypes.
Revised: This explanation space provides additional explanation of the relationship, going beyond previous classiﬁcation networks that provide explanation with distances and learned prototypes.

Original: In addition, our distance has more intrinsic meaning by VAE training techniques that regulate the latent space.
Revised: In addition, our distance has more intrinsic meaning by VAE training techniques that regulate a latent space.

Original: With user study, we validate the quality of explanation space and additional explanations.
Revised: With human evaluation, we validate the quality of explanation space and additional explanations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: "Explanation" is essentially based on understanding.
Revised: " Explanation " is essentially based on understanding.

Original: When a fact is in doubt, an explanation is provided to resolve a question.
Revised: When a fact is in doubt, an explanation is provided to resolve a question.

Original: Therefore explanation must be human-interpretable and reliable.
Revised: Therefore, explanation must be human-interpretable and reliable.

Original: An explanation that is not human-interpretable does not solve the question, and an explanation that is not reliable creates another question.
Revised: An explanation that is not human-interpretable does not resolve the issue, and an explanation that is not reliable may create confusion.

Original: Unfortunately in the real world, it is difﬁcult to ﬁnd an explanation that perfectly satisﬁes even one condition.
Revised: In practice, it is difﬁcult to ﬁnd an explanation that perfectly satisﬁes even one condition.

Original: In this situation, people receive an insufﬁcient explanationand they may ask - "Why is this explanation given?".
Revised: In this situation, when an insufﬁcient explanation is given, a use may wonder why the explanation is given.

Original: We aim to address this question by providing a reliable explanation and providing additional explanations for the given explanation in this paper.
Revised: We aim to address this question by providing a reliable explanation and providing additional explanations for a given explanation in this paper.

Original: With recent interest in deep learning,  performance of deep learning has evolved at a tremendous rate.
Revised: With advances in deep learning, its performance improves rapidly.

Original: Advances have increased the use of deep neural networks (DNNs), resulting in utilization in areas where model decisions signiﬁcantly affect people - such as healthcare (Caruana et al., 2015), radiation (Reardon, 2019), and autonomous driving.
Revised: Advances have increased the use of deep neural networks (DNNs), which is used in areas where model decisions signiﬁcantly affect people - such as healthcare (Caruana et al., 2015), radiation (Reardon, 2019), and autonomous driving.

Original: As a result, questions about the rationality of model predictions for industrial and social use have been raised, and calls for "explanation" are growing (Goodman & Flaxman, 2017; Doshi-Velez & Kim, 2017).
Revised: As a result, questions about the rationality of model predictions for industrial and social use have been raised, and the call for explanation is growing (Goodman & Flaxman, 2017; Doshi-Velez & Kim, 2017).

Original: Since most DNNs are black-box models, explanation of model prediction is often provided by a second model that explains the ﬁrst black-box model (Ribeiro et al., 2016b;
Revised: Since most DNNs are black-box models, explanation of model prediction is often provided by a second model that explains a given black-box model (Ribeiro et al., 2016b;

Original: Chen et al., 2018a; Lundberg & Lee, 2017; Guo et al., 2018).
Revised: Chen et al., 2018a; Lundberg & Lee, 2017; Guo et al., 2018).

Original: This second model is called a post-hoc interpretable model, which has been used for ﬂexibility (Ribeiro et al., 2016a).
Revised: This second model is called a post-hoc interpretable model, which has been used for ﬂexibility (Ribeiro et al., 2016a).

Original: However, this can lead people to misunderstanding the explanation because it is not offered by the model that provides the prediction.
Revised: However, this can lead people to misunderstanding the explanation because it is not offered by the model that provides the prediction.

Original: Therefore, as Rudin (2019) claimed, it is necessary to use an inherently-interpretable model in order to accurately explain DNNs without any mistakes, which are due to its nature of providing its own explanation.
Revised: Therefore, as Rudin (2019) claimed, it is necessary to use an inherently-interpretable model in order to accurately explain DNNs without any mistakes, which results from an architecture to provide intrinsic explanation.

Original: We build an inherently interpretable model, by referring to methods that generate prototypes and conduct classiﬁcation using Euclidean-distance in the latent space (Li et al., 2018; Chen et al., 2019).
Revised: We build an inherently interpretable model, by referring to methods that generate prototypes and conduct classiﬁcation using Euclidean-distance in a latent space (Li et al., 2018; Chen et al., 2019).

Original: These prototype-based explanation methods provide explanations with examples and distances and prove their ability in diverse domains.
Revised: These prototype-based explanation methods provide explanations with examples and distances and show efﬁcacy in diverse domains.

Original: However, prototype-based methods are powerful but not enough for fully understanding the model’s prediction, as explanations must be both humaninterpretable and reliable.
Revised: However, prototype-based methods are powerful but not enough to understand a model prediction fully, as explanations must be both human-interpretable and reliable.

Original: As most of the existing prototype-based explanation models suffer from the problem that similar latent representations do not guarantee similarity in shape,  distances often provide explanations that are not human-interpretable.
Revised: As most of the existing prototype-based explanation models suffer from the problem that similar latent representations do not guarantee similarity in shape, and distances often provide explanations that are not human-interpretable.

Original: Users can raise doubts but have no choice but to accept the given distances only as numerical values.
Revised: Users can raise doubts, but have no choice but to accept the given distances only as numerical values.

Original: Since there is no way to verify how distances are composed, the reliability of the explanations is compromised.
Revised: Since there is no way to verify how distances are constructed, the reliability of the explanations is compromised.

Original: [Figure]
Revised: [Figure]

Original: To solve these problems, we construct the explanation space that provides gradual explanations and reliable distance.
Revised: To solve these problems, we construct an explanation space that provides gradual explanations and reliable distance.

Original: By using VAE, the latent space can be visualized as an image that users can recognize.
Revised: By using VAE, a latent space can be visualized as an image that users can recognize.

Original: Also, we leverage VAE training techniques to regulate the latent space to ensure latent representations share similarities in appearance.
Revised: We also use VAE training techniques to regulate a latent space to ensure latent representations share similarities in appearance.

Original: This means that the distance in the latent space can have more "intrinsic meanings".
Revised: This means that a distance in the latent space can have more "intrinsic meaning".

Original: We call the latent space obtained through these processes as the explanation space.
Revised: We call the latent space obtained through the process the explanation space.

Original: Our model provides additional gradual explanations asillustrated in Figure 1.
Revised: Our model provides additional gradual explanation as in Figure 1.

Original: Meanwhile,  prototypes and  distances between the input data and prototypes are given asa basic explanation in Figure 1(a).
Revised: Meanwhile, the prototypes and the distances between the input data and prototypes are given as basic explanation in Figure 1(a).

Original: Through these relationships, users are able to understand how distances are composed.
Revised: Through these relationships, users are able to understand how distances are composed.

Original: Rather than only providing  distances with the input and prototypes, it is possible to increase understanding of the model by showing which features are changing through relationships.
Revised: Instead of providing the distances between the input and prototypes, it is possible to better understand a model by showing which features are changing in terms of relationship.

Original: Therefore, our proposed model allows users to conﬁrm the reliability of the distance.
Revised: Therefore, our proposed model allows users to interpret distances more reliably.

Original: In this paper, we developed the model that performs classiﬁcation through distances between prototype distributions and the encoded input distribution in the latent space of the Variational Autoencoder (VAE)(Kingma & Welling, 2014).
Revised: In this paper, we developed the model that performs classiﬁcation through distances between prototype distributions and the encoded input distribution in the latent space of the Variational Autoencoder (VAE)(Kingma & Welling, 2014).

Original: After VAE encodes input data in a Gaussian distribution form, we compare the encoded input distribution and generated prototype distributions through four measures with various number of prototype distributions to provide explanations with high classiﬁcation accuracy.
Revised: After VAE encodes input data in a Gaussian distribution form, we compare the encoded input distribution and generated prototype distributions through four measures with various number of prototype distributions to provide explanations with high classiﬁcation accuracy.

Original: Furthermore, we use the method (Sensoy et al., 2018) of placing a Dirichlet distribution on the class probabilities to improve accuracy by reducing the uncertainty of the model.
Revised: Furthermore, we use the method (Sensoy et al., 2018) of placing a Dirichlet distribution on the class probabilities to improve accuracy by reducing the uncertainty of the model.

Original: Contributions.
Revised: Contributions.

Original: Our key contributions are as follows:
Revised: Our key contributions are as follows:

Original: • By constructing the explanation space, we can visualize additional gradual explanations.
Revised: • By constructing the explanation space, we can visualize additional gradual explanations.

Original: Through gradual explanations, relationships are provided rather than just providing each prototype image.
Revised: Through gradual explanations, relationships are provided rather than just providing each prototype image.

Original: • Similar latent representations sharing similarities in appearance are ensured by ourconstructed dense explanation space  using VAE.
Revised: • By our dense explanation space approach using VAE, similar latent representations tend to resemble each other in appearance.

Original: Therefore the distanceprovided asan explanation becomes possible to function as a numerical value that guarantees similarity.
Revised: Therefore, distance as explanation is more meaningful because similarity is guaranteed.

Original: • We present a method to ﬁnd a baseline number of prototypes to describe the training dataset.
Revised: • We present a method to ﬁnd a baseline number of prototypes to describe the training dataset.

Original: In experiments we show changes in accuracy and explanations of the model by varying numbers of prototypes.
Revised: In experiments, we show changes in accuracy and explanations of the model by varying numbers of prototypes.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Explanation Methods for DNN While there are diverse suites of methods to explain DNN, it can be categorized based on the access of the model’s internal information and how they provide explanations (Jeyakumar et al., 2020).
Revised: Explanation Methods for DNN While there are diverse suites of methods to explain DNN, it can be categorized based on the access of the model’s internal information and how they provide explanations (Jeyakumar et al., 2020).

Original: The model-agnostic method is to approximate the relationships between input data and output decision while treating the model as a black-box.
Revised: The model-agnostic method is to approximate the relationships between input data and output decision while treating the model as a black-box.

Original: There are methods to create a local explanation (Ribeiro et al., 2016b; Chen et al., 2018a; Lundberg & Lee, 2017; Chen et al., 2018b) by selecting features that play an essential role in making decisions and methods providing an interpretable model through a global approximate (Guo et al., 2018).
Revised: There are methods to create a local explanation (Ribeiro et al., 2016b; Chen et al., 2018a; Lundberg & Lee, 2017; Chen et al., 2018b) by selecting features that play an essential role in making decisions and methods providing an interpretable model through a global approximate (Guo et al., 2018).

Original: On the other hand, in knowing the internal information of the model, the model-transparent method provides explanations by saliency map using gradient in classiﬁer (Simonyan et al., 2014; Smilkov et al., 2017; Sundararajan et al., 2017), class activation map using global average pooling (Zhou et al., 2016; Selvaraju et al., 2017), and feature map based on attention (Liu et al., 2020).
Revised: On the other hand, in knowing the internal information of the model, the model-transparent method provides explanations by saliency map using gradient in classiﬁer (Simonyan et al., 2014; Smilkov et al., 2017; Sundararajan et al., 2017), class activation map using global average pooling (Zhou et al., 2016; Selvaraju et al., 2017), and feature map based on attention (Liu et al., 2020).

Original: The third method provides explanations by examples capturing relationship between test input and the model’s decision.
Revised: The third method provides explanations by examples capturing relationship between test input and the model’s decision.

Original: Considering that examples are based on training data (Bien & Tibshirani, 

Revised: Considering that examples are based on training data (Bien & Tibshirani,

Original: ; Koh & Liang, 2017;
Revised: ; Koh & Liang, 2017;

Original: Arık & Pﬁster, 2020) or learned prototype data (Li et al., 2018; Chen et al., 2019; Hase et al., 2019), our work is closely aligned with these prototype-based classiﬁcation models.
Revised: Arık & Pﬁster, 2020) or learned prototype data (Li et al., 2018; Chen et al., 2019; Hase et al., 2019), our work is closely aligned with these prototype-based classiﬁcation models.

Original: The example-based method proves their ability to explain complex concepts in psychology (Aamodt & Plaza, 1994) and education (Renkl, 2014) domains and is also accepted by the general public as the most preferred explanation method to explain DNN (Jeyakumar et al., 2020).
Revised: The example-based method shows its efﬁcacy to explain complex concepts in psychology (Aamodt & Plaza, 1994) and education (Renkl, 2014) domains, and is also accepted by the general public as the most preferred explanation method to explain DNN (Jeyakumar et al., 2020).

Original: The following three prototype-based classiﬁcation methods are similar to our method in that they use the distance between the encoded input and prototypes for classiﬁcation and explanation.
Revised: The following three prototype-based classiﬁcation methods are similar to our method in that they use the distance between the encoded input and prototypes for classiﬁcation and explanation.

Original: Li et al.
Revised: Li et al.

Original: (2018) learn and visualize the prototypes for the entire data through autoencoder.
Revised: (2018) learn and visualize the prototypes for the entire data through autoencoder.

Original: Chen et al.
Revised: Chen et al.

Original: (2019) go further from here, characterizing training image patches as class-speciﬁc prototypes for partial explanations, while Hase et al.
Revised: (2019) go further from here, characterizing training image patches as class-speciﬁc prototypes for partial explanations, while Hase et al.

Original: (2019) feature a hierarchical demonstration of prototypes through related groups for non-ﬂat explanations.
Revised: (2019) feature a hierarchical demonstration of prototypes through related groups for non-ﬂat explanations.

Original: As most of prototype-based explanation methods are based on Li et al.
Revised: As most of prototype-based explanation methods are based on Li et al.

Original: (2018), we use it as a baseline model.
Revised: (2018), we use it as a baseline model.

Original: Our framework goes further from these methods, expands prototypes from vectors to distribution in latent space and constructs explanation space for diverse explanations with reliability.
Revised: Our framework goes further from these methods, expands prototypes from vectors to distribution in latent space and constructs explanation space for diverse explanations with reliability.

Original: In addition tothe prototype-based methods, there are several methods that use the distance in latent space to understand the internal behavior of a model or to providean explanation using VAE (Antorán et al., 2021; O’Shaughnessy et al., 2020) or GAN (Lang et al., 2021;
Revised: In addition to prototype-based methods, there are several methods that use the distance in latent space to understand the internal behavior of a model or to provide explanation using VAE (Antorán et al., 2021; O’Shaughnessy et al., 2020) or GAN (Lang et al., 2021;

Original: Yang et al., 2021).
Revised: Yang et al., 2021).

Original: Uncertainty in deep learning The largest body of research on estimating uncertainty in deep learning is Bayesian neural networks (Neal, 2012; Gal & Ghahramani, 2015; Kendall & Gal, 2017).
Revised: Uncertainty in deep learning The largest body of research on estimating uncertainty in deep learning is Bayesian neural networks (Neal, 2012; Gal & Ghahramani, 2015; Kendall & Gal, 2017).

Original: To obtain predictive uncertainty, Bayesian neural networks use prior distributions on model parameters and infer the posterior distribution.
Revised: To obtain predictive uncertainty, Bayesian neural networks use prior distributions on model parameters and infer the posterior distribution.

Original: Although exact inference in Bayesian methods is intractable, a range of approximate techniques has been proposed, such as variational inference (Kingma et al., 2015; Gal & Ghahramani, 2016) and Stochastic Gradient Hamiltonian Monte Carlo (SG-HMC) (Chen et al., 2014).
Revised: Although exact inference in Bayesian methods is intractable, a range of approximate techniques has been proposed, such as variational inference (Kingma et al., 2015; Gal & Ghahramani, 2016) and stochastic gradient hamiltonian monte carlo (SG-HMC) (Chen et al., 2014).

Original: However, in practice, these Bayesian methods are outperformed by Deep Ensembles (Lakshminarayanan et al., 2017).
Revised: However, in practice, these Bayesian methods are outperformed by Deep Ensembles (Lakshminarayanan et al., 2017).

Original: This non-bayesian technique uses multiple neural networks trained from different initializations and averages their predictions as the model output.
Revised: This non-bayesian technique uses multiple neural networks trained from different initializations and averages their predictions as the model output.

Original: It is simple and effective but requires high computational costs at training and test time.
Revised: It is simple and effective but requires high computational costs at training and test time.

Original: Another non-bayesian technique is based on evidential deep learning (Sensoy et al., 2018;
Revised: Another non-bayesian technique is based on evidential deep learning (Sensoy et al., 2018;

Original: Amini et al., 2019), which we use to estimate uncertainty.
Revised: Amini et al., 2019), which we use to estimate uncertainty.

Original: These methods use a single deterministic neural network and apply prior distributions on model predictions, not model parameters.
Revised: These methods use a single deterministic neural network and apply prior distributions on model predictions, not model parameters.

Original:  M ETHODS
Revised:  M ETHODS

Original: Our model consists of three parts: (i) VAE layer, (ii) Distance measuring layer (iii) Uncertainty layer, as shown inFigure 2. The VAE layer is used to encode input into a form ofdistribution and decodethe values obtained at prototype distributions and relationships with recognizable image data.
Revised: Our model consists of three parts: (i) VAE layer, (ii) Distance measuring layer (iii) Uncertainty layer, as shown in form of and decode obtained at prototype distributions and relationships with recognizable image data.

Original: The distance between prototype distributions and encoded input distribution is used as input for one fully-connected layer to create an inherently interpretable classiﬁcation model.
Revised: The distance between prototype distributions and encoded input distribution is used as input for one fully-connected layer to create an inherently interpretable classiﬁcation model.

Original: Moreover, since classiﬁcation is conducted through distance in the latent space of VAE, we use the uncertainty layer to increase accuracy.
Revised: Moreover, since classiﬁcation is performed with distances in the latent space of VAE, we use an uncertainty layer to increase accuracy.

Original: Encoder 𝑓
Revised: Encoder 𝑓

Original: Decoder 𝑔
Revised: Decoder 𝑔

Original: Input image x
Revised: Input image x

Original: Reconstructed image ො x 𝜇 𝑧 𝜎 𝑧 z d 𝑧 … 𝜇𝜇𝜇𝜇 𝑚 dddd 𝑚
Revised: Reconstructed image ො x 𝜇 𝑧 𝜎 𝑧 z d 𝑧 … 𝜇𝜇𝜇𝜇 𝑚 dddd 𝑚

Original: VAE layers … 𝜇 𝑧
Revised: VAE layers … 𝜇 𝑧

Original: Distance measuring layer …
Revised: Distance measuring layer …

Original: Fully-connected layer ℎ
Revised: Fully-connected layer ℎ

Original: Uncertainty layer
Revised: Uncertainty layer

Original: Uncertainty calculation 𝛼𝛼𝛼 𝑘 𝑢 l ogits 𝑙
Revised: Uncertainty calculation 𝛼𝛼𝛼 𝑘 𝑢 l ogits 𝑙

Original: 3. U SING V ARIATIONAL A UTO E NCODER
Revised: 3. U SING V ARIATIONAL A UTO E NCODER

Original: Let D =
Revised: Let D =

Original: { X, Y } be the training dataset consists of images x ∈ R N and labels y ∈ { 1 , . .
Revised: { X, Y } be the training dataset consists of images x ∈ R N and labels y ∈ { 1 , . .

Original: . , K } .
Revised: . , K } .

Original: The encoder f : q ( z | x ) of the VAE maps an input image to a Gaussian distribution d z ∼
Revised: The encoder f : q ( z | x ) of the VAE maps an input image to a Gaussian distribution d z ∼

Original: N ( µ z , σ z 2 ) ( µ z , σ z ∈ R d ) which is reparameterized to generate latent vector z ∈ R d .
Revised: N ( µ z , σ z 2 ) ( µ z , σ z ∈ R d ) which is reparameterized to generate latent vector z ∈ R d .

Original: After that, the generated z is reconstructed in the form of image ˆ x = g ( z ) through decoder g : p ( x | z ) .
Revised: After that, the generated z is reconstructed in the form of image ˆ x = g ( z ) through decoder g : p ( x | z ) .

Original: Unlike autoencoder, VAE regulates the latent space created by the encoder as isotropic unit Gaussian ( p ( z ) ∼ N (0 , I ) ), which is why we used VAE as a feature extractor.
Revised: Unlike autoencoder, VAE regulates the latent space created by the encoder as isotropic unit Gaussian ( p ( z ) ∼ N (0 , I ) ), which is why we used VAE as a feature extractor.

Original: By regularizing latent space and sampling latent vectors from distribution, VAE constructs appropriate dense latent space that creates similar latent representations from similar-shaped images, which plays an essential role in the interpretability of our method described in Section 4.
Revised: By regularizing latent space and sampling latent vectors from distribution, VAE constructs appropriate dense latent space that creates similar latent representations from similar-shaped images, which plays an essential role in the interpretability of our method described in Section 4.

Original: Furthermore, our model learns prototype distributions as m independent multivariate Gaussian distributions D = { d 1 , d 2 , . . .
Revised: Furthermore, our model learns prototype distributions as m independent multivariate Gaussian distributions D = { d 1 , d 2 , . . .

Original: , d m } , d i ∼ N ( µ i , σ 2 i )
Revised: , d m } , d i ∼ N ( µ i , σ 2 i )

Original: ∈ R d and visualizes µ i as prototype.
Revised: ∈ R d and visualizes µ i as prototype.

Original: For that, the structures of disentangled VAE are applied in order to disentangle the explanation components at prototype distributions through dimensions.
Revised: For that, the structures of disentangled VAE are applied in order to disentangle the explanation components at prototype distributions through dimensions.

Original: A method of encouraging the independence of representation distribution q ( z ) by lowering the KL divergence between q ( z ) and ¯ q ( z ) :
Revised: A method of encouraging the independence of representation distribution q ( z ) by lowering the KL divergence between q ( z ) and ¯ q ( z ) :

Original: = (cid:81) dj
Revised: = (cid:81) dj

Original: =1 q ( z j ) , and methods of lowering the KL divergence of encoder f and isotropic unit Gaussian p ( z ) to constrain the latent space stronger are additionally used with vanilla VAE.
Revised: =1 q ( z j ) , and methods of lowering the KL divergence of encoder f and isotropic unit Gaussian p ( z ) to constrain the latent space stronger are additionally used with vanilla VAE.

Original: Applying these methods would result in the following objective:
Revised: Applying these methods would result in the following objective:

Original: [Equation]
Revised: [Equation]

Original: In this objective function, β and γ are hyperparameters that control the power of the constraint terms.
Revised: In this objective function, β and γ are hyperparameters that control the power of the constraint terms.

Original: 3. C LASSIFICATION NETWORK USING DISTANCE
Revised: 3. C LASSIFICATION NETWORK USING DISTANCE

Original: Once the input is encoded by latent distribution d z , logits based on the distance from the prototype distributions are produced for the classiﬁcation.
Revised: Once the input is encoded by latent distribution d z , logits based on the distance from the prototype distributions are produced for the classiﬁcation.

Original: This process is divided into (i) measuring the distance between learned D = { d 1 , d 2 , . . .
Revised: This process is divided into (i) measuring the distance between learned D = { d 1 , d 2 , . . .

Original: , d m } and d z in the latent space and (ii) producing the logits by measured distance using the fully-connected layer h for the classiﬁcation.
Revised: , d m } and d z in the latent space and (ii) producing the logits by measured distance using the fully-connected layer h for the classiﬁcation.

Original: Since prototype distributions D and encoded input d z both are in the form of Gaussian distribution, we use 2-Wasserstein distance (Givens et al., 1984) as a distance measure.
Revised: Since prototype distributions D and encoded input d z both are in the form of Gaussian distribution, we use 2-Wasserstein distance (Givens et al., 1984) as a distance measure.

Original: The 2-Wasserstein distance
Revised: The 2-Wasserstein distance

Original: W for two multivariate Gaussian distribution N ( µ 1 , Σ 1 ) , N ( µ 2 , Σ 2 )( µ i ∈ R d , Σ i ∈ R
Revised: W for two multivariate Gaussian distribution N ( µ 1 , Σ 1 ) , N ( µ 2 , Σ 2 )( µ i ∈ R d , Σ i ∈ R

Original: d × R d ) can be computed as:
Revised: d × R d ) can be computed as:

Original: [Equation]
Revised: [Equation]

Original: Since we assume the prototype distributions D to be independent and encoder f predicts diagonal covariance matrices, Σ
Revised: Since we assume the prototype distributions D to be independent and encoder f predicts diagonal covariance matrices, Σ

Original: i = diag( σ 1 i , . . .
Revised: i = diag( σ 1 i , . . .

Original: , σ di ) and the above equation simpliﬁed as:
Revised: , σ di ) and the above equation simpliﬁed as:

Original: [Equation]
Revised: [Equation]

Original: Three other measurements KL-divergence (KLD), Jensen-Shannon divergence (JSD), Jensen-Tsallis distance (JTD) (Choi et al., 2019) are also experimented in Section 5.1.
Revised: Three other measurements KL-divergence (KLD), Jensen-Shannon divergence (JSD), Jensen-Tsallis distance (JTD) (Choi et al., 2019) are also experimented in Section 5.1.

Original: The distances between D and d z are used as input of fully-connected layer h to generate logits.
Revised: The distances between D and d z are used as input of fully-connected layer h to generate logits.

Original: The reason for classiﬁcation by passing only one fully-connected layer is to construct a weight matrix.
Revised: The reason for classiﬁcation by passing only one fully-connected layer is to construct a weight matrix.

Original: By constructing a weight matrix using h , the model can numerically conﬁrm the impact of each prototype distribution on class selection.
Revised: By constructing a weight matrix using h , the model can numerically conﬁrm the impact of each prototype distribution on class selection.

Original: This weight matrix gives interpretability to the model’s prediction even when the number of given prototype distributions m and the class K are different.
Revised: This weight matrix gives interpretability to the model’s prediction even when the number of given prototype distributions m and the class K are different.

Original: 3. U SING UNCERTAINTY LAYER
Revised: 3. U SING UNCERTAINTY LAYER

Original: Repeating the samplingprocess could increase the uncertainty ofthe classiﬁcation process through the distance at latent space.
Revised: To repeat sampling could increase the uncertainty of classiﬁcation based on distances in a latent space.

Original: Therefore, with consideringthe uncertainty, we  improve the classiﬁcation performance by about 1 % with a method (Sensoy et al., 2018) that uses  Dirichlet distribution to reduce the uncertainty of the model.
Revised: Therefore, by considering uncertainty, we also improve the classiﬁcation performance by about 1 % with the method (Sensoy et al., 2018) that uses the Dirichlet distribution to reduce the uncertainty of a model.

Original: In the training process, the uncertainty loss L uncertain is calculated using the Dirichlet parameters obtained through logits l .
Revised: In the training process, the uncertainty loss L uncertain is calculated using the Dirichlet parameters obtained through logits l .

Original: The detailed method for uncertainty layer is in Appendix A.5.
Revised: The detailed method for uncertainty layer is in Appendix A.5.

Original: After going through all of these processes, the ﬁnal objective function is composed as follows:
Revised: After going through all of these processes, the ﬁnal objective function is composed as follows:

Original: [Equation]
Revised: [Equation]

Original: where L uncertain is the uncertainty classiﬁcation term based on cross-entropy loss and D and d z is the VAE term that described at Section 3 . 1 .
Revised: where L uncertain is the uncertainty classiﬁcation term based on cross-entropy loss and D and d z is the VAE term that described at Section 3 . 1 .

Original: Moreover, λ is a hyperparameter for adjusting the ratio between them.
Revised: Moreover, λ is a hyperparameter for adjusting the ratio between them.

Original:  C ONSTRUCTING INTERPRETABLE EXPLANATIONS 4. R ELIABLE EXPLANATION SPACE
Revised:  C ONSTRUCTING INTERPRETABLE EXPLANATIONS 4. R ELIABLE EXPLANATION SPACE

Original: Our method increases the reliability of the explanation through distance and the prototypes.
Revised: Our method increases the reliability of the explanation through distance and the prototypes.

Original: As noted in Brendel & Bethge (2019), the interpretation of previous prototype-based classiﬁcation methods is difﬁcult because only the Euclidean distance between prototypes and latent representations is given, while having a similar latent representation does not guarantee that images share similarities in human-interpretable features.
Revised: As noted in Brendel & Bethge (2019), the interpretation of previous prototype-based classiﬁcation methods is difﬁcult because only the Euclidean distance between prototypes and latent representations is given, while having a similar latent representation does not guarantee that images share similarities in human-interpretable features.

Original: Therefore, by sampling with distributions and regularizing latent space through VAE, our model constructs dense explanation space that ensures similar latent representations share similarities in appearance.
Revised: Therefore, by sampling with distributions and regularizing latent space through VAE, our model constructs dense explanation space that ensures similar latent representations share similarities in appearance.

Original: Figure 3 makes this more evident.
Revised: Figure 3 makes this more evident.

Original: Looking at the visualized latent space by t-SNE (Van der Maaten & Hinton, 2008), embedded data by the previous method tends to separate clusters, often make the case that the distance between two points in the same class farther than the distance between two different class points, as shown in Figure 3(a).
Revised: Looking at the visualized latent space by t-SNE (Van der Maaten & Hinton, 2008), embedded data by the previous method tends to separate clusters, often make the case that the distance between two points in the same class farther than the distance between two different class points, as shown in Figure 3(a).

Original: In contrast, the proposed model increases the reliability of explanation through distance by showing that each class is clustered together and exist close to each other that forms dense latent space.
Revised: In contrast, the proposed model increases the reliability of explanation through distance by showing that each class is clustered together and exist close to each other that forms dense latent space.

Original: Furthermore, by showing latent representations contained within the prototype distribution d i as images, the prototype’s reliability can also be increased by their similarities.
Revised: Furthermore, by showing latent representations contained within the prototype distribution d i as images, the prototype’s reliability can also be increased by their similarities.

Original: As illustrated in Figure 1(d), features within the distribution are similar to the prototype µ i at basic explanation, demonstrating that prototypes are not coincidentally chosen.
Revised: As illustrated in Figure 1(d), features within the distribution are similar to the prototype µ i at basic explanation, demonstrating that prototypes are not coincidentally chosen.

Original: We made quantitative comparisons through user study as well as qualitative comparisons to ensure the reliability of the explanation space.
Revised: We made quantitative comparisons through user study as well as qualitative comparisons to ensure the reliability of the explanation space.

Original: We measured the user’s understanding through an interactive reconstruction experiment (Ross et al., 2021) to show the reliability of the explanation space of our 
Revised: We measured the user’s understanding through an interactive reconstruction experiment (Ross et al., 2021) to show the reliability of the explanation space of our method.

Revised: This experiment is to generate an image similar to a randomly sampled image by changing

Original: [Figure]
Revised: [Figure]

Original: method. This experiment is to generate an image similar to a randomly sampled image by changing each dimension value of the latent space through a slide using the decoder of the models.
Revised: each dimension value of the latent space through a slide using the decoder of the models.

Original: As the user explores the latent space and generates an image similar to the sampled image, we can see how the conﬁguration of the latent space is easy to understand for the user.
Revised: As the user explores the latent space and generates an image similar to the sampled image, we can see how the conﬁguration of the latent space is easy to understand for the user.

Original: The evaluation was conducted through Amazon Mechanical Turk.
Revised: The evaluation was conducted through Amazon Mechanical Turk.

Original: In order to improve the quality of the evaluation, all experiments were conducted through 20 Mechanical Turk Masters.
Revised: In order to improve the quality of the evaluation, all experiments were conducted through 20 Mechanical Turk Masters.

Original: The success rate by model is shown in Table 1.
Revised: The success rate by model is shown in Table 1.

Original: Detailed settings for experiments are in Appendix A.2.
Revised: Detailed settings for experiments are in Appendix A.2.

Original: [Table]
Revised: [Table]

Original: As Ross et al.
Revised: As Ross et al.

Original: (2021) concluded that a higher success rate on interactive reconstruction tasks shows improved interpretability, we concluded that the explanation space by our method improved interpretability because the success rate obtained through the interactive reconstruction experiment showed a higher value than the baseline model.
Revised: (2021) concluded that a higher success rate on interactive reconstruction tasks shows improved interpretability, we concluded that the explanation space by our method improved interpretability because the success rate obtained through the interactive reconstruction experiment showed a higher value than the baseline model.

Original: The distance value in the explanation space having more intrinsic meaning because the explanation space created by our method showed improved interpretability compared to the baseline model.
Revised: The distance value in the explanation space having more intrinsic meaning because the explanation space created by our method showed improved interpretability compared to the baseline model.

Original: Therefore, the experimental results could be used as quantitative evaluation to support that our method created a reliable explanation space.
Revised: Therefore, the experimental results could be used as quantitative evaluation to support that our method created a reliable explanation space.

Original: 4. S ELECTING THE NUMBER OF PROTOTYPE DISTRIBUTIONS
Revised: 4. S ELECTING THE NUMBER OF PROTOTYPE DISTRIBUTIONS

Original: In the prototype-based explanation model, the number of prototypes is critical.
Revised: In the prototype-based explanation model, the number of prototypes is critical.

Original: As can be seen in the model structure, the number of prototypes affects the input of the fully-connected layer and inevitably affects accuracy.
Revised: As can be seen in the model structure, the number of prototypes affects the input of the fully-connected layer and inevitably affects accuracy.

Original: Moreover, since explanations are also provided through prototypes, it is important to set the number of prototypes.
Revised: Moreover, since explanations are also provided through prototypes, it is important to set the number of prototypes.

Original: However, since there was no information to refer other than the number of classes to determine the number of prototypes that can represent the dataset, the previous methods (Li et al., 2018; Chen et al., 2019; Hase et al., 2019) speciﬁed the number of prototypes to an arbitrary and heuristic value determined by a human.
Revised: However, since there was no information to refer other than the number of classes to determine the number of prototypes that can represent the dataset, the previous methods (Li et al., 2018; Chen et al., 2019; Hase et al., 2019) speciﬁed the number of prototypes to an arbitrary and heuristic value determined by a human.

Original: We propose a method to produce the baseline number of prototype distributions in our framework.
Revised: We propose a method to produce the baseline number of prototype distributions in our framework.

Original: Since we have set the prototype distributions as independent Gaussian distributions rather than vectors, prototype distributions cover the latent space.
Revised: Since we have set the prototype distributions as independent Gaussian distributions rather than vectors, prototype distributions cover the latent space.

Original: Furthermore, as prototype distributions are representative of the whole dataset, it shares similarities with the Gaussian Mixture Model (GMM) in that the whole dataset is represented in the form of multiple Gaussian distributions.
Revised: Furthermore, as prototype distributions are representative of the whole dataset, it shares similarities with the Gaussian Mixture Model (GMM) in that the whole dataset is represented in the form of multiple Gaussian distributions.

Original: We choose an optimal number of components of the GMM in latent space as a baseline number of prototype distributions.
Revised: We choose an optimal number of components of the GMM in latent space as a baseline number of prototype distributions.

Original: We use the Bayesian Information Criterion (BIC), which is known to be the most appropriate criterion for GMM (Steele & Raftery, 2010).
Revised: We use the Bayesian Information Criterion (BIC), which is known to be the most appropriate criterion for GMM (Steele & Raftery, 2010).

Original: The experimental results for this can be seen in the Section 5.3.
Revised: The experimental results for this can be seen in the Section 5.3.

Original: 4. E XPLAINING THE RELATIONSHIP OF EXPLANATIONS
Revised: 4. E XPLAINING THE RELATIONSHIP OF EXPLANATIONS

Original: The explanations of our model are divided into basic explanation and explanations of the basic explanation.
Revised: The explanations of our model are divided into basic explanation and explanations of the basic explanation.

Original: For fundamental explanations, visualized images of the learned prototypes and the distance between encoded input and prototype distributions are given as previous prototype-based explanation methods.
Revised: For fundamental explanations, visualized images of the learned prototypes and the distance between encoded input and prototype distributions are given as previous prototype-based explanation methods.

Original: To explain this basic explanation, we additionally provide two gradual explanations.
Revised: To explain this basic explanation, we additionally provide two gradual explanations.

Original: These explanations can be obtained because our method constructs dense explanation space, unlike previous methods.
Revised: These explanations can be obtained because our method constructs dense explanation space, unlike previous methods.

Original: As a result, it is possible to generate interconnection between latent representations as images.
Revised: As a result, it is possible to generate interconnection between latent representations as images.

Original: We represent changing elements by interpolating these latent representations and deﬁne them as relationships.
Revised: We represent changing elements by interpolating these latent representations and deﬁne them as relationships.

Original: The relationships between inputs and prototypes can be visualized by our model.
Revised: The relationships between inputs and prototypes can be visualized by our model.

Original: In situations where distance is given for interpretable information, the factors that can reduce the distance are critical for understanding the model’s prediction.
Revised: In situations where distance is given for interpretable information, the factors that can reduce the distance are critical for understanding the model’s prediction.

Original: Additionally, relationships between prototypes are given.
Revised: Additionally, relationships between prototypes are given.

Original: Since the prototypes are representative of the whole dataset, relationships between prototypes show which features are needed to change from one class to another class.
Revised: Since the prototypes are representative of the whole dataset, relationships between prototypes show which features are needed to change from one class to another class.

Original: This explanation can be used as basic information about how the model discriminates classes.
Revised: This explanation can be used as basic information about how the model discriminates classes.

Original: After the model is trained, basic explanation and explanations for basic explanation are visualized as follows.
Revised: After the model is trained, basic explanation and explanations for basic explanation are visualized as follows.

Original: First, the prototype, provided as the fundamental explanation, is visualized by sending µ i of the prototype distribution through the decoder g as a prototype.
Revised: First, the prototype, provided as the fundamental explanation, is visualized by sending µ i of the prototype distribution through the decoder g as a prototype.

Original: Next, the relationships between prototype µ i and prototype µ j are visualized as sending µ i + ( µ j − µ i ) r/k for r = 0 , 1 , . . .
Revised: Next, the relationships between prototype µ i and prototype µ j are visualized as sending µ i + ( µ j − µ i ) r/k for r = 0 , 1 , . . .

Original: , k through the decoder g , which can be controlled by increasing the value of k for detailed explanation and decreasing the value of k for a brief explanation.
Revised: , k through the decoder g , which can be controlled by increasing the value of k for detailed explanation and decreasing the value of k for a brief explanation.

Original: Finally, the relationships between inputs and prototypes are provided in the same way, and visualizations of prototype distributions are based on µ values and σ values.
Revised: Finally, the relationships between inputs and prototypes are provided in the same way, and visualizations of prototype distributions are based on µ values and σ values.

Original: The disadvantage of our model comes from this visualization.
Revised: The disadvantage of our model comes from this visualization.

Original: Many training samples are required to generate explanation images.
Revised: Many training samples are required to generate explanation images.

Original: In addition, visualized images of our explanations are fundamentally generated through VAE, making it challenging to generate human-recognizable images for complex datasets that are difﬁcult for VAE to generate the observable image.
Revised: In addition, visualized images of our explanations are fundamentally generated through VAE, making it challenging to generate human-recognizable images for complex datasets that are difﬁcult for VAE to generate the observable image.

Original: However, this is a problem related to VAE’s expressive power, and there are many high-resolution expressive methods (Lee et al., 2020; Vahdat & Kautz, 2020) based on VAE, so applying these methods would solve these problems in future work.
Revised: However, this is a problem related to VAE’s expressive power, and there are many high-resolution expressive methods (Lee et al., 2020; Vahdat & Kautz, 2020) based on VAE, so applying these methods would solve these problems in future work.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Experiment details MNIST and Fashion-MNIST (Xiao et al., 2017) datasets are used for our experiments and are splited by 0.9, 0.1 for training dataset and test dataset.
Revised: Experiment details MNIST and Fashion-MNIST (Xiao et al., 2017) datasets are used for our experiments and are splited by 0.9, 0.1 for training dataset and test dataset.

Original: For training our model, we use Adam (Kingma & Ba, 2015) optimizer with learning rate of 0.0001 and weight decay of 0.00005, and set hyperparameter λ to 50.
Revised: For training our model, we use Adam (Kingma & Ba, 2015) optimizer with learning rate of 0.0001 and weight decay of 0.00005, and set hyperparameter λ to 50.

Original: To compare various VAE structures, we use vanilla VAE ( β = 1 , γ = 0 ), β -VAE ( β = 4 , γ = 0 )
Revised: To compare various VAE structures, we use vanilla VAE ( β = 1 , γ = 0 ), β -VAE ( β = 4 , γ = 0 )

Original: (Higgins et al., 2017) and FactorVAE ( β = 1 , γ = 6 . 4 ) (Kim & Mnih, 2018).
Revised: (Higgins et al., 2017) and FactorVAE ( β = 1 , γ = 6 . 4 ) (Kim & Mnih, 2018).

Original: We train 500 epochs with VAE and β -VAE for about 2 hours, and about 3 hours were used with FactorVAE using a single 1080 Ti GPU.[Table]
Revised: We train 500 epochs with VAE and β -VAE for about 2 hours, and about 3 hours were used with FactorVAE using a single 1080 Ti GPU.

Original: [Table]
Revised: [Table]

Original:  C LASSIFICATION ACCURACY
Revised: 5. C LASSIFICATION ACCURACY

Original: The accuracy of our model on MNIST is 99.33 ± 0.04 % with providing visualized explanations in
Revised: The accuracy of our model on MNIST is 99.33 ± 0.04 % with providing visualized explanations in

Original: Figure 1.
Revised: Figure 1.

Original: Furthermore, as shown in Table 2, our model achieved testing accuracy 93.00 ± 0.13 % on Fashion-MNIST.
Revised: Furthermore, as shown in Table 2, our model achieved testing accuracy 93.00 ± 0.13 % on Fashion-MNIST.

Original: This result is comparable to non-interpretable classiﬁcation networks such as
Revised: This result is comparable to non-interpretable classiﬁcation networks such as

Original: VGG16 (93.5 % ) and GoogleNet (93.7 % ) reported in the dataset website.
Revised: VGG16 (93.5 % ) and GoogleNet (93.7 % ) reported in the dataset website.

Original: Moreover, we achieve higher accuracy than the baseline model’s
Revised: Moreover, we achieve higher accuracy than the baseline model’s

Original: (Li et al., 2018) accuracy 91.03 ± 0.32 % that only provides basic explanation.
Revised: (Li et al., 2018) accuracy 91.03 ± 0.32 % that only provides basic explanation.

Original: We construct baseline model with same layers and same hyperparameters.
Revised: We construct baseline model with same layers and same hyperparameters.

Original: [Figure]

Original: As shown in Table 2, we conduct experiments with four measurements and three VAE structures.
Revised: As shown in Table 2, we conduct experiments with four measurements and three VAE structures.

Original: According to measures, results of experiments using Wasserstein-distance show the highest accuracy  through three VAE structures.
Revised: According to measures, results of experiments using Wasserstein-distance show the highest accuracy [Figure] through three VAE structures.

Original: In addition, it achieves the highest diversity and stability of the learned prototypes as it provides a usable gradient when distributions are supported on non-overlapping domains (Kolouri et al., 2019).
Revised: In addition, it achieves the highest diversity and stability of the learned prototypes as it provides a usable gradient when distributions are supported on non-overlapping domains (Kolouri et al., 2019).

Original: This indicates that Wasserstein-distance is the most suitable measurement in creating prototypes using distances between distributions.
Revised: This indicates that Wasserstein-distance is the most suitable measurement in creating prototypes using distances between distributions.

Original: The role of the uncertainty layer to increase accuracy is also clearly revealed.
Revised: The role of the uncertainty layer to increase accuracy is also clearly revealed.

Original: In most of the 12 cases, accuracy increased when the uncertainty layer is applied.
Revised: In most of the 12 cases, accuracy increased when the uncertainty layer is applied.

Original: Among three VAE structures, the vanilla VAE shows the highest accuracy, but FactorVAE also shows no signiﬁcant difference in accuracy.
Revised: Among three VAE structures, the vanilla VAE shows the highest accuracy, but FactorVAE also shows no signiﬁcant difference in accuracy.

Original: However, FactorVAE forms more reliable explanation space as Section 4.1, it is used to create Figures 1, 4, 5.
Revised: However, FactorVAE forms more reliable explanation space as Section 4.1, it is used to create Figures 1, 4, 5.

Original: 5. P ROVIDED EXPLANATIONS : B EYOND EXAMPLES
Revised: 5. P ROVIDED EXPLANATIONS : B EYOND EXAMPLES

Original: Our model generates explanations for prediction as Figure 4.
Revised: Our model generates explanations for prediction as Figure 4.

Original: In Figure 4(a), for the black T-shirt input image, our model shows prototypes with the lowest distance having shapes of tops.
Revised: In Figure 4(a), for the black T-shirt input image, our model shows prototypes with the lowest distance having shapes of tops.

Original: On the other hand, the distances from the prototypes that looked like bags, trousers, and shoes are higher than 6.5, indicating that the prototypes and distance information provide accurate explanations.
Revised: On the other hand, the distances from the prototypes that looked like bags, trousers, and shoes are higher than 6.5, indicating that the prototypes and distance information provide accurate explanations.

Original: Furthermore, a fully-connected layer is applied as a weight matrix corresponding to a similarity connection between these distances and logits, showing relevance between prototype distributions and classes.
Revised: Furthermore, a fully-connected layer is applied as a weight matrix corresponding to a similarity connection between these distances and logits, showing relevance between prototype distributions and classes.

Original: However, this explanation alone makes it difﬁcult to understand why the distance from the prototypes in the same T-shirt class is different and whether the different distances came out even if the prototypes were in the form of dresses.
Revised: However, this explanation alone makes it difﬁcult to understand why the distance from the prototypes in the same T-shirt class is different and whether the different distances came out even if the prototypes were in the form of dresses.

Original: Therefore, our method provides additional explanations for this basic explanation.
Revised: Therefore, our method provides additional explanations for this basic explanation.

Original: Additionally, our model provides the relationships between prototypes and inputs as an explanation of the factors that inﬂuenced distance, such as Figure 4(b).
Revised: Additionally, our model provides the relationships between prototypes and inputs as an explanation of the factors that inﬂuenced distance, such as Figure 4(b).

Original: For example, from relationships between the input and prototypes for class dress, the nearest prototype can be seen with only the thickness of the torso changing, followed by the next prototype with longer sleeves and the farthest prototype with no sleeves.
Revised: For example, from relationships between the input and prototypes for class dress, the nearest prototype can be seen with only the thickness of the torso changing, followed by the next prototype with longer sleeves and the farthest prototype with no sleeves.

Original: Since prototypes with changes in the sleeves have similar distances, we can assume that the model is sensitive to sleeves in the relationship between input and dress prototypes.
Revised: Since prototypes with changes in the sleeves have similar distances, we can assume that the model is sensitive to sleeves in the relationship between input and dress prototypes.

Original: The bigger the change in the sleeves of the clothes, the farther the distance becomes.
Revised: The bigger the change in the sleeves of the clothes, the farther the distance becomes.

Original: In addition, relationships between prototypes are visualized as additional information for how the model discriminates prototypes, such as Figure 4(c).
Revised: In addition, relationships between prototypes are visualized as additional information for how the model discriminates prototypes, such as Figure 4(c).

Original: We conduct a user study to show that the relationship helps understanding model predictions.
Revised: We conduct a user study to show that the relationship helps us to understand model predictions.

Original: With the explanations obtained through our model, we tried to compare human understandings when (i) basic explanation (Figure 4(a)),  (ii) basic explanation and the relationship between input and prototype (Figure 4(a), (b)) are given.
Revised: With the explanations from our model, subjective comparison is made when (i) basic explanation (Figure 4(a)), and (ii) basic explanation and the relationship between input and prototype (Figure 4(a), (b)) are given.

Original: For each set of explanations, we survey the case of correct/wrong classiﬁcation.
Revised: For each set of explanations, we studied the results of correct/wrong classiﬁcation.

Original: We prepare the following questions to check whether users were able to respond to factors that inﬂuenced model predictions.
Revised: We prepare the following questions to check whether users were able to respond to factors that inﬂuenced model predictions.

Original: In the case of correct classiﬁcation, we ask  factors that affect differences in the distance between prototypes within the same class.
Revised: In the case of correct classiﬁcation, we ask the factors that affect differences in the distance between prototypes within the same class.

Original: In the case of misclassiﬁcation, we asked  factors that made the distance from the prototype of the wrong class close.
Revised: In the case of misclassiﬁcation, we asked the factors that made the distance from the prototype of the wrong class close.

Original: The relationships with the input and following prototypes were given during the survey: (i) Correct classiﬁcation: two prototypes of the same class as input.
Revised: The relationships with the input and following prototypes were given during the survey: (i) Correct classiﬁcation: two prototypes of the same class as input.

Original: (ii) Wrong classiﬁcation:  prototype[Figure] closest to the input among misclassiﬁed classes, and  prototype closest to the input among true classes.
Revised: (ii) Wrong classiﬁcation: the prototype closest to the input among misclassiﬁed classes, and the prototype closest to the input among true classes.

Original: Additional information about the survey is included in Appendix A.3.
Revised: Additional information about the survey is included in Appendix A.3.

Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: We conducted two surveys through Amazon Mechanical Turk with 50 Mechanical Turk masters.
Revised: We conducted two surveys through Amazon Mechanical Turk with 50 Mechanical Turk masters.

Original: Each survey consists of 12 questions, 6 per correct classiﬁcation and misclassiﬁcation.
Revised: Each survey consists of 12 questions, 6 per correct classiﬁcation and misclassiﬁcation.

Original: The average response rate for answering the factor is in Table 3.
Revised: The average response rate for answering the factor is in Table 3.

Original: This result shows that participants were better able to answer factors for model predictions when given the additional explanations of the relationship between inputs and prototypes.
Revised: This result shows that participants were better able to answer factors for model predictions when given the additional explanations of the relationship between inputs and prototypes.

Original: It shows that relationships increase human understandings of factors that compute distances.
Revised: It shows that relationships enables us to better understand the factors that compute distances.

Original: 5. C HANGING THE NUMBER OF PROTOTYPE DISTRIBUTIONS
Revised: 5. C HANGING THE NUMBER OF PROTOTYPE DISTRIBUTIONS

Original: As mentioned above, we chose the optimal number of Gaussian Mixture Model’s components as a proper number of prototype distributions.
Revised: As mentioned above, we chose the optimal number of Gaussian Mixture Model’s components as a proper number of prototype distributions.

Original: We use the EM algorithm to ﬁt Gaussian Mixture Model to the dataset on latent space.
Revised: We use the EM algorithm to ﬁt Gaussian Mixture Model to the dataset on latent space.

Original: First, trained VAE encodes µ values of the entire dataset to place them on the latent space.
Revised: First, trained VAE encodes µ values of the entire dataset to place them on the latent space.

Original: Since our prototype distributions consist of independent multivariate Gaussian distributions, we use the Gaussian Mixture Model with each component having its own diagonal covariance matrix.
Revised: Since our prototype distributions consist of independent multivariate Gaussian distributions, we use the Gaussian Mixture Model with each component having its own diagonal covariance matrix.

Original: As the ﬁrst local minimum BIC score indicates strong evidence for the model (Fraley & Raftery, 1998), we determine that the number of components at the ﬁrst local minimum is the optimal number of components.
Revised: As the ﬁrst local minimum BIC score indicates strong evidence for the model (Fraley & Raftery, 1998), we determine that the number of components at the ﬁrst local minimum is the optimal number of components.

Original: The number of components obtained is 20.3, 19.8, 23.8 on average in VAE, β -VAE, and FactorVAE.
Revised: The number of components obtained is 20.3, 19.8, 23.8 on average in VAE, β -VAE, and FactorVAE.

Original: Through these results, the baseline number of prototype distributions is set as 20.
Revised: Through these results, the baseline number of prototype distributions is set as 20.

Original: To verify this, we experiment with numbers of prototype distributions m =
Revised: To verify this, we experiment with numbers of prototype distributions m =

Original: and the results are in Figure 5.
Revised: and the results are in Figure 5.

Original: In Figure 5(a), the accuracy begins to converge around 20.
Revised: In Figure 5(a), the accuracy begins to converge around 20.

Original: In addition, when the number of prototypes are small, the model converges extensively slowly.
Revised: In addition, when the number of prototypes are small, the model converges extensively slowly.

Original: In Figure 5(b), when the number of prototypes increases from less than the  dataset class, various appearances appear.
Revised: In Figure 5(b), when the number of prototypes increases from a number smaller than the number of dataset classes, various appearances appear.

Original: However, when the number of prototypes increases excessively, similar appearance prototypes are inevitably raised.
Revised: However, when the number of prototypes increases excessively, similar appearance prototypes are inevitable.

Original: These results demonstrate that it is proper to reference on theoptimal number ofcomponents of the Gaussian Mixture Model for the number of prototype distributions.
Revised: These results indicate that it is better to set the number of Gaussian mixture model components as the number of prototype distributions.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The main contribution of our paper is to further enhance human understanding of model predictions by providing additional explanations of relationships.
Revised: The main contribution of our paper is to further enhance human understanding of model predictions by providing additional explanations of relationships.

Original: Further to existing prototype-based explanation methods that generate explanations by prototype examples and distances, our model visualizes relationships between input and prototypes that can explain the factors relevant to distance.
Revised: Further to existing prototype-based explanation methods that generate explanations by prototype examples and distances, our model visualizes relationships between input and prototypes that can explain the factors relevant to distance.

Original: Moreover, due to creating a dense explanation space, the distance became possible to function as a numerical value that guarantees similarity, which led to increase reliability.
Revised: Moreover, due to creating a dense explanation space, the distance became possible to function as a numerical value that guarantees similarity, which led to increase reliability.

Original: ODEL A RCHITECTURE D ETAILS
Revised: ODEL A RCHITECTURE D ETAILS

Original: In our proposed model, we use vanilla VAE, β -VAE, and FactorVAE for the VAE layer.
Revised: In our proposed model, we use vanilla VAE, β -VAE, and FactorVAE for the VAE layer.

Original: For VAE and β -VAE, we use a Convolutional Neural network for the encoder and a Deconvolutional Neural
Revised: For VAE and β -VAE, we use a Convolutional Neural network for the encoder and a Deconvolutional Neural

Original: Network for the decoder.
Revised: Network for the decoder.

Original: Detailed architecture explanation is provided in Table 4.
Revised: Detailed architecture explanation is provided in Table 4.

Original: For FactorVAE, discriminator is additionally used for distinguishing whether the input was drawn from q ( z ) or ¯ q ( z ) :
Revised: For FactorVAE, discriminator is additionally used for distinguishing whether the input was drawn from q ( z ) or ¯ q ( z ) :

Original: = (cid:81) dj
Revised: = (cid:81) dj

Original: =1 q ( z j ) to calculate KL divergence between q ( z ) and ¯ q ( z ) .
Revised: =1 q ( z j ) to calculate KL divergence between q ( z ) and ¯ q ( z ) .

Original: The discriminator is made of 6 MLP layers with 1000 hidden units each and leaky ReLU is used for non-linearity, that generates 2 logits as outputs.
Revised: The discriminator is made of 6 MLP layers with 1000 hidden units each and leaky ReLU is used for non-linearity, that generates 2 logits as outputs.

Original: We use Adam optimizer for discriminator with learning rate 0.0001 and weight decay 0.00005.
Revised: We use Adam optimizer for discriminator with learning rate 0.0001 and weight decay 0.00005.

Original: A. D ETAILS FOR INTERACTIVE RECONSTRUCTION EXPERIMENT
Revised: A. D ETAILS FOR INTERACTIVE RECONSTRUCTION EXPERIMENT

Original: We constructed the experiments in the same way as in the paper Ross et al.
Revised: We constructed the experiments in the same way as in the paper Ross et al.

Original: Through the interactive reconstruction method shown in the paper, we evaluate the models: Li et al.
Revised: Through the interactive reconstruction method shown in the paper, we evaluate the models: Li et al.

Original: (2018), proposed model with VAE, and proposed model with FactorVAE.
Revised: (2018), proposed model with VAE, and proposed model with FactorVAE.

Original: The evaluation was conducted through Amazon Mechanical Turk, and for a fair evaluation, everyone paid $3 .
Revised: The evaluation was conducted through Amazon Mechanical Turk, and for a fair evaluation, everyone paid $3 .

Original: 75 per evaluation.
Revised: 75 per evaluation.

Original: In order to improve the quality of the evaluation, all experiments were conducted through Mechanical
Revised: In order to improve the quality of the evaluation, all experiments were conducted through Mechanical

Original: Turk Masters.
Revised: Turk Masters.

Original: The experiments were conducted with models trained through the MNIST dataset.
Revised: The experiments were conducted with models trained through the MNIST dataset.

Original: Success was considered if the user-generated image and the sampled image had a similarity of at least 75 % .
Revised: Success was considered if the user-generated image and the sampled image had a similarity of at least 75 % .

Original: In addition, it is designed so that users can skip the question only if they generate images for at leastseconds.
Revised: In addition, it is designed so that users can skip the question only if they generate images for at leastseconds.

Original: Experiments were conducted with 20 participants per model and 7 questions with random images were provided.
Revised: Experiments were conducted with 20 participants per model and 7 questions with random images were provided.

Original: An example of interactive reconstruction experiment is shown in Figure 6.
Revised: An example of interactive reconstruction experiment is shown in Figure 6.

Original: A. U SER STUDY ABOUT RELATIONSHIPS
Revised: A. U SER STUDY ABOUT RELATIONSHIPS

Original: The survey was conducted after showing precautions to the participants ﬁrst.
Revised: The survey was conducted after showing precautions to the participants ﬁrst.

Original: Participants were asked to answer the factors that inﬂuenced the model’s decision by looking at the given information.
Revised: Participants were asked to answer the factors that inﬂuenced the model’s decision by looking at the given information.

Original: At this time, since the given information may be insufﬁcient to judge the factor, two options were provided: the option of not knowing and the option of guessing.
Revised: At this time, since the given information may be insufﬁcient to judge the factor, two options were provided: the option of not knowing and the option of guessing.

Original: In the case of the participants who judged that they could guess factors, they were also asked to answer which factors they guessed.
Revised: In the case of the participants who judged that they could guess factors, they were also asked to answer which factors they guessed.

Original: Consent form and example of survey without relationships is shown in Figure 7.
Revised: Consent form and example of survey without relationships is shown in Figure 7.

Original: Consent form and example of survey with relationships is shown in Figure 8.
Revised: Consent form and example of survey with relationships is shown in Figure 8.

Original: A. D ETAIL EXPLANATIONS FOR MEASUREMENTS
Revised: A. D ETAIL EXPLANATIONS FOR MEASUREMENTS

Original: We use KL-divergence, Jensen-Shannon divergence (JSD), Jensen-Tsallis distance (JTD) and Wasserstein distance for measuring distance between prototype distributions D
Revised: We use KL-divergence, Jensen-Shannon divergence (JSD), Jensen-Tsallis distance (JTD) and Wasserstein distance for measuring distance between prototype distributions D

Original: = { d 1 , d 2 , . . .
Revised: = { d 1 , d 2 , . . .

Original: , d m } and encoded input distribution d z .
Revised: , d m } and encoded input distribution d z .

Original: Since it was not described in detail in this paper, we provide additional explanations in this section.
Revised: Since it was not described in detail in this paper, we provide additional explanations in this section.

Original: For two multivariate Gaussian distributions p = N ( µ 1 , Σ 1 ) , q = N ( µ 2 , Σ 2 )( µ i ∈ R d , Σ i ∈
Revised: For two multivariate Gaussian distributions p = N ( µ 1 , Σ 1 ) , q = N ( µ 2 , Σ 2 )( µ i ∈ R d , Σ i ∈

Original: R
Revised: R

Original: d × R d ) ,
Revised: d × R d ) ,

Original: KL-divergence is as follows:
Revised: KL-divergence is as follows:

Original: [Equation]
Revised: [Equation]

Original: From independent distributions D and encoded input distribution that has diagonal covariance matrix Σ i = diag( σ 1 i , . . .
Revised: From independent distributions D and encoded input distribution that has diagonal covariance matrix Σ i = diag( σ 1 i , . . .

Original: , σ di ) , the above equation simpliﬁes to:
Revised: , σ di ) , the above equation simpliﬁes to:

Original: [Equation]
Revised: [Equation]

Original: Through this equation, with m = 12 ( p + q ) , Jensen-Shannon divergence also simpliﬁes in the same manner because Jensen-Shannon divergence deﬁned as:
Revised: Through this equation, with m = 12 ( p + q ) , Jensen-Shannon divergence also simpliﬁes in the same manner because Jensen-Shannon divergence deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Unlike JSD, Jensen-Tsallis divergence uses the Tsallis entropy instead of the Shannon entropy.
Revised: Unlike JSD, Jensen-Tsallis divergence uses the Tsallis entropy instead of the Shannon entropy.

Original: The
Revised: The

Original: Tsallis entropy S q,k ( p ) is generalization of the Shannon entropy which becomes Shannon entropy when q converges to 1 and k equals 1.
Revised: Tsallis entropy S q,k ( p ) is generalization of the Shannon entropy which becomes Shannon entropy when q converges to 1 and k equals 1.

Original: In previous works, they use speciﬁc Jensen-Tsallis divergence with q = 2 and k = 4 and refer to JTD since it satisﬁes the conditions of a metric.
Revised: In previous works, they use speciﬁc Jensen-Tsallis divergence with q = 2 and k = 4 and refer to JTD since it satisﬁes the conditions of a metric.

Original: In our condition,
Revised: In our condition,

Original: JTD is as follows:
Revised: JTD is as follows:

Original: [Equation]
Revised: [Equation]

Original: By using KL-divergence, JSD, and JTD as a distance measure, the proposed model learns prototypes as illustrated in Figure 9.
Revised: By using KL-divergence, JSD, and JTD as a distance measure, the proposed model learns prototypes as illustrated in Figure 9.

Original: Using these metric for distance, the model accuracy is high enough, but the prototypes do not form observable images, which is contrary to our purpose of explaining the model predictions.
Revised: Using these metric for distance, the model accuracy is high enough, but the prototypes do not form observable images, which is contrary to our purpose of explaining the model predictions.

Original: This is because these three measures fail to provide usable gradients when distributions are supported on non-overlapping domains.
Revised: This is because these three measures fail to provide usable gradients when distributions are supported on non-overlapping domains.

Original: For this reason, the model fails to provide usable gradients when the prototype distributions and the embedded input distribution do not overlap, thus not forming a proper latent space as shown in the Figure 9, which makes prototypes unstable.
Revised: For this reason, the model fails to provide usable gradients when the prototype distributions and the embedded input distribution do not overlap, thus not forming a proper latent space as shown in the Figure 9, which makes prototypes unstable.

Original: A. D ETAILED ANALYSIS DUE TO APPLICATION OF UNCERTAINTY LAYER
Revised: A. D ETAILED ANALYSIS DUE TO APPLICATION OF UNCERTAINTY LAYER

Original: In the process of forming the latent space in our proposed model, VAE formulates the reconstruction loss through the latent vector sampled from the embedded input distribution.
Revised: In the process of forming the latent space in our proposed model, VAE formulates the reconstruction loss through the latent vector sampled from the embedded input distribution.

Original: Since classiﬁcation was performed through the distance in the latent space, we tried to improve the classiﬁcation accuracy by lowering the uncertainty of the model.
Revised: Since classiﬁcation was performed through the distance in the latent space, we tried to improve the classiﬁcation accuracy by lowering the uncertainty of the model.

Original: By adding the calculation process to logits without changing the structure of the model, it was possible to predict uncertainty and increase the accuracy by making the model robust against adversarial perturbations.
Revised: By adding the calculation process to logits without changing the structure of the model, it was possible to predict uncertainty and increase the accuracy by making the model robust against adversarial perturbations.

Original: Furthermore, since the model is not trained to handle uncertain information, it not only increases accuracy but also makes similar features exist in similar positions in the latent space.
Revised: Furthermore, since the model is not trained to handle uncertain information, it not only increases accuracy but also makes similar features exist in similar positions in the latent space.

Original: The embedded dataset visualized with t-SNE is illustrated in
Revised: The embedded dataset visualized with t-SNE is illustrated in

Original: Figure 10 showing that data of the same class forms more aggregated clusters in latent space when the uncertainty layer is applied.
Revised: Figure 10 showing that data of the same class forms more aggregated clusters in latent space when the uncertainty layer is applied.

Original: Through this latent space property, prototypes form more obvious observations that can express the whole dataset better, as Figure 11, because prototype distributions are learned through distances at latent space.
Revised: Through this latent space property, prototypes form more obvious observations that can express the whole dataset better, as Figure 11, because prototype distributions are learned through distances at latent space.

Original: For detailed explanations of the uncertainty layer setting in our model is as follows.
Revised: For detailed explanations of the uncertainty layer setting in our model is as follows.

Original: Given a sample x i , let there be one-hot vector
Revised: Given a sample x i , let there be one-hot vector

Original: y i with ground truth class j , then y ij = 1 and y ik = 0 for all k (cid:54) =
Revised: y i with ground truth class j , then y ij = 1 and y ik = 0 for all k (cid:54) =

Original: j , and logit l i that comes through the distance measuring layer of the model.
Revised: j , and logit l i that comes through the distance measuring layer of the model.

Original: We compute the evidence e i as exponential of logit l i .
Revised: We compute the evidence e i as exponential of logit l i .

Original: Then use the evidence to derive the uncertainty probability u i
Revised: Then use the evidence to derive the uncertainty probability u i

Original: = K/S i , where total S i
Revised: = K/S i , where total S i

Original: = (cid:80) Kj
Revised: = (cid:80) Kj

Original: =1 ( e ij + 1) , and parameters of Dirichlet distribution
Revised: =1 ( e ij + 1) , and parameters of Dirichlet distribution

Original: α ik = e ik
Revised: α ik = e ik

Original: + 1 for class
Revised: + 1 for class

Original: [Table]
Revised: [Table]

Original: Input
Revised: Input

Original: [Table]
Revised: [Table]

Original: image
Revised: image

Original: Input z
Revised: Input z

Original: [Table]
Revised: [Table]

Original: R
Revised: R

Original: Conv2d (32 channels, 4 × 4 kernels, stride 2, pad 1)
Revised: Conv2d (32 channels, 4 × 4 kernels, stride 2, pad 1)

Original: Conv2d (256 channels, 1 × 1 kernels, stride 1, pad 0)
Revised: Conv2d (256 channels, 1 × 1 kernels, stride 1, pad 0)

Original: BatchNorm, LeakyReLU(0.2) BatchNorm, LeakyReLU(0.2)
Revised: BatchNorm, LeakyReLU(0.2) BatchNorm, LeakyReLU(0.2)

Original: Conv2d (32 channels, 4 × 4 kernels, stride 2, pad 1)
Revised: Conv2d (32 channels, 4 × 4 kernels, stride 2, pad 1)

Original: ConvTranspose2d (64 channels, 4 × 4 kernels, stride 1, pad 0)
Revised: ConvTranspose2d (64 channels, 4 × 4 kernels, stride 1, pad 0)

Original: BatchNorm, LeakyReLU(0.2) BatchNorm, LeakyReLU(0.2)
Revised: BatchNorm, LeakyReLU(0.2) BatchNorm, LeakyReLU(0.2)

Original: Conv2d (64 channels, 4 × 4 kernels, stride 2, pad 1)
Revised: Conv2d (64 channels, 4 × 4 kernels, stride 2, pad 1)

Original: ConvTranspose2d (32 channels, 4 × 4 kernels, stride 2, pad 1)
Revised: ConvTranspose2d (32 channels, 4 × 4 kernels, stride 2, pad 1)

Original: BatchNorm, LeakyReLU(0.2) BatchNorm, LeakyReLU(0.2)
Revised: BatchNorm, LeakyReLU(0.2) BatchNorm, LeakyReLU(0.2)

Original: Conv2d (256 channels, 4 × 4 kernels, stride 1, pad 0)
Revised: Conv2d (256 channels, 4 × 4 kernels, stride 1, pad 0)

Original: ConvTranspose2d (32 channels, 4 × 4 kernels, stride 2, pad 1)
Revised: ConvTranspose2d (32 channels, 4 × 4 kernels, stride 2, pad 1)

Original: BatchNorm, LeakyReLU(0.2)
Revised: BatchNorm, LeakyReLU(0.2)

Original: BatchNorm, LeakyReLU(0.2)
Revised: BatchNorm, LeakyReLU(0.2)

Original: Conv2d (2*10 channels, 1 × 1 kernels, stride 1, pad 0)
Revised: Conv2d (2*10 channels, 1 × 1 kernels, stride 1, pad 0)

Original: ConvTranspose2d (1 channels, 4 × 4 kernels, stride 2, pad 1)
Revised: ConvTranspose2d (1 channels, 4 × 4 kernels, stride 2, pad 1)

Original: Sigmoid
Revised: Sigmoid

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: k .
Revised: k .

Original: Through the obtained parameter α ik and S i , which is the sum of total evidence and class K , the cross-entropy loss changes as follows:
Revised: Through the obtained parameter α ik and S i , which is the sum of total evidence and class K , the cross-entropy loss changes as follows:

Original: [Equation]
Revised: [Equation]

Original: Where ψ ( · ) is a digamma function, and after this process it has the same effect as specifying the class of uncertainty.
Revised: Where ψ ( · ) is a digamma function, and after this process it has the same effect as specifying the class of uncertainty.

Original: Also, we add a regularizing term for misleading evidence.
Revised: Also, we add a regularizing term for misleading evidence.

Original: For predicted parameters α i with true class j, misleading evidence is represented as α ik for all k (cid:54) =
Revised: For predicted parameters α i with true class j, misleading evidence is represented as α ik for all k (cid:54) =

Original: j .
Revised: j .

Original: With equation 5,
Revised: With equation 5,

Original: we
Revised: we

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: regularize them using KL divergence with the uniform Dirichlet distribution as follows:
Revised: regularize them using KL divergence with the uniform Dirichlet distribution as follows:

Original: [Equation]
Revised: [Equation]

Original: where λ t is the annealing coefﬁcient that starts from 0 and becomes 1 over training epoch t = 50 ,
Revised: where λ t is the annealing coefﬁcient that starts from 0 and becomes 1 over training epoch t = 50 ,

Original: D ( p i | (cid:104) 1 , . .
Revised: D ( p i | (cid:104) 1 , . .

Original: , 1 (cid:105) ) is the uniform Dirichlet distribution, and ˜ α
Revised: , 1 (cid:105) ) is the uniform Dirichlet distribution, and ˜ α

Original: i =
Revised: i =

Original: y i
Revised: y i

Original: + (1 − y i ) (cid:12) α
Revised: + (1 − y i ) (cid:12) α

Original: i for removing non-misleading evidence.
Revised: i for removing non-misleading evidence.

Original: A. A DDITIONAL RESULTS FOR MNIST EXPERIMENTS
Revised: A. A DDITIONAL RESULTS FOR MNIST EXPERIMENTS

Original: As shown in Figure 12(a), prototypes with the shape of 0 are the closest to the input.
Revised: As shown in Figure 12(a), prototypes with the shape of 0 are the closest to the input.

Original: Distances to numbers such as 2, 3, 8, 9, which have curved parts similar to 0, are closer than the distance to numbers such as 1, 4, 7, which have sharp shapes.
Revised: Distances to numbers such as 2, 3, 8, 9, which have curved parts similar to 0, are closer than the distance to numbers such as 1, 4, 7, which have sharp shapes.

Original: Moreover, in the same class, the distance from the input was similar.
Revised: Moreover, in the same class, the distance from the input was similar.

Original: Further explanations are given by the relationships between input and prototypes and the relationships between each prototype, as shown in Figure 12(b), (c).
Revised: Further explanations are given by the relationships between input and prototypes and the relationships between each prototype, as shown in Figure 12(b), (c).

Original: We can infer
Revised: We can infer

Original: [Figure] Prototypes
Revised: Prototype:Class Prototypes:Class 0~9 0 →

Original: [Figure] ↑ [Figure] Width 
Revised: Thickness ↑, Width ↑ 5.

Original: [Figure]
Revised: Smear ↑ 3.

Original: Rotation
Revised: Rotation

Original: [Figure]
Revised: Smear ↑

Original: [Figure]
Revised: Distanaces 6.

Original: [Figure] [Figure]
Revised: 9 →0 →0 →5.

Original: [Figure]
Revised: 6.84 5.06 Input Prototypes

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: the factors that caused the model to predict distance farther by the relationships between the input and the prototypes. Also, by examining the relationship between each prototype, we can ﬁnd out how the model distinguishes the classes. If the explanation is insufﬁcient, more detailed explanations can be obtained by visualizing more interpolation values between prototypes. A. S TABILITY : H OW SHOULD THE EXPLANATION RESPOND TO SIMILAR INPUTS ?

Original: [Figure]
Revised: Interpretable explanation models have to provide consistent explanations for similar inputs for stability. This is because the reliability of explanations is not guaranteed if different explanations are provided for similar inputs. To demonstrate the stability of our method, we measure the changes in distances provided as an explanation when the input is distorted. For fairness of measurements, the distance change is normalized by distance before the input is distorted.

Original: [Figure]
Revised: We measure these distance changes for each prototype of all inputs.

Original: [Figure] [Figure] [Figure] [Figure]
Revised: In Table 5, our method shows much more stability for input distortions.

Original: [Figure]
Revised: In particular, there is a large difference in terms of average and median in the case of the baseline model, which means explanations that provide signiﬁcantly different distances exist when the inputs are distorted. Based on our experiment, we claim our method provides a more reliable explanation to users than the baseline model.

Original: [Figure]
Revised: [Table]