Original: C O RTX: C ONTRASTIVE L EARNING FOR R EAL - TIME E XPLANATION
Revised: C O RTX: C ONTRASTIVE F RAMEWORK FOR R EAL - TIME E XPLANATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent advancements in explainable machine learning provide effective and faithful solutions for interpreting model behaviors.
Revised: Recent advancements in explainable machine learning provide effective and faithful solutions for interpreting model behaviors.

Original: However, most explanation methods encounter efficiency issues, which largely limits their deployments in practical scenarios.
Revised: However, many explanation methods encounter efficiency issues, which largely limit their deployments in practical scenarios.

Original: Real-time explainer (RTX) frameworks have thus been proposed to accelerate the model explanation process by learning an one-feed-forward explainer.
Revised: Real-time explainer (RTX) frameworks have thus been proposed to accelerate the model explanation process by learning an one-feed-forward explainer.

Original: Existing RTX frameworks typically build the explainer under the supervised learning paradigm, which require large amounts of explanation labels as  ground truth.
Revised: Existing RTX frameworks typically build the explainer under the supervised learning paradigm, which requires large amounts of explanation labels as the ground truth.

Original: Considering that accurate explanation labels are usually hard to obtained, due tothe constrained computational resources and limited human efforts, effective explainer training is still challenging in practice.
Revised: Considering that accurate explanation labels are usually hard to obtain, due to constrained computational resources and limited human efforts, effective explainer training is still challenging in practice.

Original: In this work, we propose a COntrastive Real-Time eXplanation (CoRTX) frameworkthat adopts contrastive learning to  relieve the intensive dependence of explainer training on explanation labels.
Revised: In this work, we propose a COntrastive RealTime eXplanation (CoRTX) framework to learn the explanation-oriented representation and relieve the intensive dependence of explainer training on explanation labels.

Original: Specifically, we design a synthetic strategy to select positive and negative samples for explanation representation learning.
Revised: Specifically, we design a synthetic strategy to select positive and negative instances for explanation representation learning.

Original: Theoretical analysis show that our selection strategy can benefit the contrastive learning process on explanation tasks.
Revised: Theoretical analysis show that our selection strategy can benefit the contrastive learning process on explanation tasks.

Original: Experimental results on three real-world datasets further demonstrate the efficiency and effectiveness of our proposed CoRTX framework.
Revised: Experimental results on three real-world datasets further demonstrate the efficiency and efficacy of our proposed CoRTX framework.

Revised: Our source code is available at: https://anonymous.4open.science/r/CoRTX-720-BDE

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The remarkable progress in explainable machine learning (ML) significantly improve the model transparency to human beings (Du et al., 2019).
Revised: The remarkable progress in explainable machine learning (ML) significantly improves the model transparency to human beings (Du et al., 2019).

Original: However, applying explainable ML techniques to real-time scenarios remains to be a challenging task.
Revised: However, applying explainable ML techniques to real-time scenarios remains to be a challenging task.

Original: Real-time systems typically require  explanationmethods to be not only human-understandable and faithful, but also efficient (Stankovic et al., 1992).
Revised: Real-time systems typically require model explanation to be not only effective but also efficient (Stankovic et al., 1992).

Original: Due to the requirements from both stakeholders and social regulations (Goodman & Flaxman, 2017; Floridi, 2019),  efficient  explanation methods are necessary for the real-time ML systems, such as thekey decisions in controlling systems (Steel & Angwin, 2010), advertisement recommendation on e-commerce systems (Yang et al., 2018), anddenotations in healthcare  systems (Esteva et al., 2019; Gao et al., 2017).
Revised: Due to the requirements from both stakeholders and social regulations (Goodman & Flaxman, 2017; Floridi, 2019), the efficient model explanation is necessary for the real-time ML systems, such as the controlling systems (Steel & Angwin, 2010), online recommender systems (Yang et al., 2018), and healthcare monitoring systems (Gao et al., 2017).

Original: Nevertheless, existing work on local explanation methods suffer from high explanation latency, including LIME (Ribeiro et al., 2016), KernelSHAP (Lundberg & Lee, 2017),
Revised: Nevertheless, existing work on non-amortized explanation methods have high explanation latency, including LIME (Ribeiro et al., 2016), KernelSHAP (Lundberg & Lee, 2017).

Original: GradCAM (Selvaraju et al., 2017), Integrated Gradient (Sundararajan et al., 2017) and RISE (Petsiuk et al., 2018).

Original: These methods rely on either multiple perturbations or backward propagation in deep neural networks (DNNs) for  explanation (Covert & Lee, 2021; Liu et al., 2021), which can be time-consuming and limited in deployment on the real-time scenarios.
Revised: These methods rely on either multiple perturbations or backpropagation in deep neural networks (DNN) for deriving explanation (Covert & Lee, 2021; Liu et al., 2021), which is time-consuming and limited for deployment in real-time scenarios.

Original: Real-time explainer (RTX) frameworks have thus been proposed to address such efficiency issue and provide faithful explanations for real-time systems (Dabkowski & Gal, 2017; Jethani et al., 2021b).
Revised: Real-time explainer (RTX) frameworks have thus been proposed to address such efficiency issues and provide effective explanations for real-time systems (Dabkowski & Gal, 2017; Jethani et al., 2021).

Original: Specifically, RTX learns a global explainer on the training set by using the ground-truth explanation labelswhich obtained from exact calculation or approximation.
Revised: Specifically, RTX learns an overall explainer on the training set by using the ground-truth explanation labels obtained through either exact calculation or approximation.

Original: RTXthen provides explanations for each testing instance via a single feed-forward process.
Revised: RTX provides the explanation for each local instance via a single feed-forward process.

Original: Existing efforts on RTX can be categorized into two lines of work.
Revised: Existing efforts on RTX can be categorized into two lines of work.

Original: The first line (Jethani et al., 2021b; Covert et al., 2022) explicitly learns an explainer to minimize the estimation error from the ground-truth explanation label.
Revised: The first line (Jethani et al., 2021; Covert et al., 2022) explicitly learns an explainer to minimize the estimation error regarding to the approximated explanation labels.

Original: The second line (Dabkowski & Gal, 2017; Chen et al., 2018; Kanehira & Harada, 2019) trains the feature mask generators based on certain constraints on predefined label distribution for feature selection.
Revised: The second line (Dabkowski & Gal, 2017; Chen et al., 2018; Kanehira & Harada, 2019) trains a feature mask generator subject to the certain constraints on pre-defined label distribution.

Original: Despite the effectiveness of existing RTX frameworks, recent advancements still rely on the large amounts of explanation label under the supervised learning paradigm.
Revised: Despite the effectiveness of existing RTX frameworks, recent advancements still rely on the large amounts of explanation labels under the supervised learning paradigm.

Original: The computational cost of deriving explanation labels is extremely high (Roth, 1988; Winter, 2002), indicating that existing supervised RTXs thereby limit the wide deployment in real-world scenarios.
Revised: The computational cost of obtaining explanation labels is extremely high (Roth, 1988; Winter, 2002), which thereby limits the RTX’s deployment in real-world scenarios.

Original: To tackle the aforementioned challenges, we propose a COntrastive Real-Time eXplanation (CoRTX) framework based on the contrastive learning techniques.
Revised: To tackle the aforementioned challenges, we propose a COntrastive Real-Time eXplanation (CoRTX) framework based on the contrastive learning techniques.

Original: CoRTX aims to learn the explanation representation of each data instance without any ground-truth explanation label.
Revised: CoRTX aims to learn the explanation representation of each data instance without any ground-truth explanation label.

Original: Contrastive learning has been widely exploited for improving the learning processes of downstream tasks by providing well pre-trained representative embeddings (Arora et al., 2019;
Revised: Contrastive learning has been widely exploited for improving the learning processes of downstream tasks by providing well-pretrained representative embeddings (Arora et al., 2019;

Original: He et al., 2020).
Revised: He et al., 2020).

Original: In particular, a task-oriented selection strategy of positive and negative pairs (Chen et al., 2020a; Khosla et al., 2020) can shape the representation properties through contrastive learning.
Revised: In particular, taskoriented selection strategies of positive and negative pairs (Chen et al., 2020; Khosla et al., 2020) can shape the representation properties through contrastive learning.

Original: Motivated by this existing contrastive schemes, CoRTX develops an explanation-oriented contrastivelearning framework to generate well pre-trained explanation representation, with the goal of further fine-tuning  in the downstreamexplanation tasks.
Revised: Motivated by the such contrastive scheme, CoRTX develops an explanation-oriented contrastive framework to learn the explanation representation, with the goal of further fine-tuning an explanation head in the downstream tasks.

Original: CoRTX learns the explanation representation to deal with  explanation tasks by minimizing the contrastive lossfunction (Van den Oord et al., 2018), Specifically, CoRTXframework designs a synthetic positive and negative sampling strategy to learn the explanation representation.
Revised: CoRTX learns the explanation representation to deal with the explanation tasks by minimizing the contrastive loss (Van den Oord et al., 2018). Specifically, CoRTX designs a synthetic positive and negative sampling strategy to learn the explanation representation.

Original: The pretrained explanation representation can then be transformed to feature attribution  by fine-tuning an explanation head using a tiny amount of explanation label.
Revised: The obtained explanation representation can then be transformed to feature attribution or ranking by fine-tuning a corresponding explanation head using a tiny amount of explanation labels.

Original: The theoretical analysis and experimental results demonstrate that CoRTX can successfully provides effective explanation representation for different explanation tasks.
Revised: Theoretical analysis and experimental results demonstrate that CoRTX can successfully provide effective explanation representation for feature attribution and ranking tasks.

Original: Overall, the contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • CoRTX provides a contrastivelearning framework for generating explanation representation, which can effectively reduce the required amount of explanation labels. • Theoretical analysis indicates our proposed CoRTX framework is able to generate effective explanation representation  and bounds the explanation error. • Empirical results demonstrate our proposed CoRTX can efficiently provide faithful explanation on both tabular and image datasets.
Revised: • CoRTX provides a contrastive framework for deriving explanation representation, which can effectively reduce the required amounts of explanation label; • Theoretical analysis indicate that CoRTX can effectively learn the explanation representation over the training set and strictly bound the explanation error; • Experimental results demonstrate that CoRTX can efficiently provide explanations to the target model, which is applicable to both tabular and image data.

Original:  P RELIMINARY 2. N OTATIONS
Revised:  P RELIMINARY 2. N OTATIONS

Original: We consider an arbitrary DNN f ( · )as the target model to interpret.
Revised: We consider an arbitrary target model f ( · ) to interpret.

Original: Let input feature be x = [ x 1 , · · · , x M ] ∈ X , where x 1 , · · · , x M denote the value ofinput feature  1 , · · · , M , respectively.
Revised: Let input feature be x = [ x 1 , · · · , x M ] ∈ X , where x i denote the value of feature i for 1 ≤ i ≤ M .

Original: The contribution of each feature to the model output can be treated as a cooperative game on the feature set X 
Revised: The contribution of each feature to the model output can be treated as a cooperative game on the feature set X .

Original: (Shapley, 1953).

Original: Specifically, the preceding difference f ( (cid:101) x S∪{ i } ) − f ( (cid:101) x S ) indicates the contribution of feature
Revised: Specifically, the preceding difference f ( (cid:101) x S∪{ i } ) − f ( (cid:101) x S ) indicates the contribution of feature

Original: i under feature subset S ⊆
Revised: i under feature subset S ⊆

Original: U \ { i } , where U is the entire feature set.
Revised: U \ { i } , where U is the entire feature space.

Original: The overall contribution of feature i is formalized as the average preceding difference considering all possible feature subsets S , which can be formally given by
Revised: The overall contribution of feature i is formalized as the average preceding difference considering all possible feature subsets S , which can be formally given by

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) x S = S ⊙ x
Revised: where (cid:101) x S = S ⊙ x

Original: +( 1 − S ) ⊙ x r denotes the perturbed sample, S = 1 S ∈ { 0 , 1 } M is a masking vector of S , and x r =
Revised: +( 1 − S ) ⊙ x r denotes the perturbed sample, S = 1 S ∈ { 0 , 1 } M is a masking vector of S , and x r =

Original: E [ x | x ∼ P ( x )] denotes the reference values 1 of each feature.
Revised: E [ x | x ∼ P ( x )] denotes the reference values 1 from feature distribution P ( x ) .

Original: The computational complexity of Equation 1 grows exponentially with the feature number M , which prevents its application to real-time explanation.
Revised: The computational complexity of Equation 1 grows exponentially with the feature number M , which cumbers its application to real-time scenarios.

Original: To this end, we propose an efficient and faithful framework forthe real-time explanation in this work.
Revised: To this end, we propose an efficient explanation framework for real-time scenarios in this work.

Original: 2. R EAL - TIME E XPLAINER
Revised: 2. R EAL - TIME E XPLAINER

Original: The Real-Time explainer framework (RTX) maintains a global model to providea fast explanationfor each data instance via one feed-forward process. 
Revised: Different from the existing non-amortized explanation methods (Lundberg & Lee, 2017; Lomeli et al., 2019) that utilize local surrogate models for explaining data instances, RTX trains a global model to provide fast explanation via one feed-forward process. Compared with the existing methods, the advantages of RTX mainly lie in two folds: (1) faster explanation generation; and (2) more robust explanation derivation.

Original: Generally, RTX attempts to learn the global explanation distribution under two different methodologies,  Shapley-sampling-based approaches (Wang et al., 2021;
Revised: Generally, existing RTXs attempt to learn the overall explanation distribution using two lines of methodologies, which are Shapley-sampling-based approaches (Wang et al., 2021;

Original: Jethani et al., 2021b; Covert et al., 2022) and feature-selection-based approaches (Chen et al., 2018; Dabkowski & Gal, 2017; Kanehira & Harada, 2019).
Revised: Jethani et al., 2021; Covert et al., 2022) and feature-selection-based approaches (Chen et al., 2018; Dabkowski & Gal, 2017; Kanehira & Harada, 2019).

Original: The first lineof approaches enforces a DNN explainer to simulate the given approximated Shapley distribution for generating explanation results.
Revised: The first line enforces the explainer to simulate a given approximated Shapley distribution for generating explanation results.

Original:  The second lineof approaches assumes the specific predefined feature patterns or distributions, and formulates the explainer learning strategy based onthe given hypothesis. As Shapley value is well supported with solid theoretical backbones, Shapley-sampling-based approaches are typically more faithful for training RTX frameworks. Different from the local explanation frameworks (Lundberg & Lee, 2017; Lomeli et al., 2019) that require multiple explaining models for each data instance, RTX frameworks only require one global
Revised: A representative work, FastSHAP (Jethani et al., 2021), exploits a DNN model to capture the Shapley distribution among training instances for efficient real-time explanations. The second line assumes the specific pre-defined feature patterns or distributions, and formulates the explainer learning process based on

Original: [Figure]
Revised: [Figure]

Original: [Figure] L model for explanation. Compared to the local explanation frameworks, the advantages of RTX are as follows: (1) obtaining faster inference utility and (2) ensuring similar data instances with similar explanations.
Revised: the given assumptions.

Original: One of the work (Chen et al., 2018) provides a feature masking generator for real-time feature selection.
Revised: One of the work (Chen et al., 2018) provides a feature masking generator for real-time feature selection.

Original: The training process of  mask generator is under the constraint of the given groundtruth label distribution.
Revised: The training process of the mask generator is under the constraint of the given ground-truth label distribution.

Original: Another work, FastSHAP (Jethani et al., 2021b), proposes a state-of-the-art

Original: Shapley-sampling-based framework for learning RTX to derive feature attribution.

Original: With sufficient amounts of masking sampling, FastSHAP exploits a DNN model to imitate the Shapley distribution among the training data instances to yield an efficient RTX.

Original: In general, Shapley-sampling-based approaches are more faithful but suffered from high computational costs when examining larger feature subsets, while feature-selection-based approaches are easy-to-train but constrained under the specific distributions.
Revised: Typically, Shapley-sampling-based approaches are more effective but suffer from high computational cost when examining larger feature subsets, while feature-selection-based methods are easy-to-train but constrained under the specific distributions.

Original: In this work, we propose an unsupervised framework, CoRTX, to benefit with the advantages of both works.
Revised: In this work, we propose an unsupervised framework CoRTX, achieving the advantages from both sides.

Original: This makes CoRTX to be more efficient and faithful.

Original: 2. L IMITATION OF S UPERVISED F RAMEWORK
Revised: 2. L IMITATION OF S UPERVISED F RAMEWORK

Original: [Figure]
Revised: [Figure]

Original: Supervised RTX  relies on enormous quantities of ground-truth explanation annotations, which limits its application to realworld scenarios.
Revised: Supervised RTX typically relies on large quantities of groundtruth explanation labels, which limits its application in practice.

Original: It is experimentally proved that supervised RTX suffers from  performance degradation without sufficient explanation annotations.
Revised: We experimentally show that supervised RTX suffers from the performance degradation when explanation labels are insufficient.

Original: Specifically, taking the Shapley values as the ground-truth explanation, the supervised RTX is learned to minimize the mean square error on the training dataset, and then estimates explanations on the testing dataset for evaluation.
Revised: Specifically, taking the Shapley value as the ground truth, supervised RTX learns to minimize the cross entropy on the training set, and then estimates the feature ranking on the testing set for evaluation.

Original: Preliminary experiments are conducted onthe CensusIncome dataset Dua & Graff (2017) which demonstrates the necessity of supervised RTX to use enormous quantities of explanation annotations.
Revised: The results on Census dataset (Dua & Graff, 2017) demonstrate the intensive dependence of supervised RTX on the amount of explanation labels.

Original: Theexplanation ranking performance  versus the exploitation ratio of explanation annotations for training is given in Figure 1. Implementation details are given Appendix B. According to Figure 1, we observe that the performance of explanation ranking significantly grows as the ratio of annotations increases.
Revised: The performance of feature ranking versus the exploitation ratio of explanation label is shown in Figure 1, where the implementation details are introduced in Appendix D. Here, we observe that the performance of feature ranking significantly enhances as the ratio of label increases.

Original: However, the complexity of estimating the ground-truth explanation is extremely high (i.e., thecomputational complexity of the Shapley values  grows exponentially with the feature number). It is challenging to have a well-trained RTX considering the limited computational resources in realworld scenarios.
Revised: Supervised RTXs outperform FastSHAP, which is trained with approximated labels, under 5% label usage. However, the complexity of calculating Shapley values generally grows exponentially with the feature number, which makes it challenging to train RTX with a large amount of labels.

Original: To tackle this problem, we proposea CoRTXframework to pre-train the explaners without any explanation labels on the training dataset. As shown in left-hand side of Figure 2, CoRTX first propose a explanation encoder to generate explanation representation without any supervision for downstream explanation task. After the explanation encoder has been well-trained, as shown inright-hand side of Figure 2, a explanation head can converge to optimal even very limited explanation labels are available to exploit.
Revised: We propose CoRTX in this paper to train the explainer in a light-label manner, where only a small amount of explanation labels are used for fine-tuning the downstream head.

Original:  C ONTRASTIVE R EAL - TIME E XPLANATION
Revised:  C ONTRASTIVE R EAL - TIME E XPLANATION

Original: We systematically introduce our CoRTX framework in this section.
Revised: We systematically introduce the CoRTX framework in this section.

Original: Figure 2 demonstrates the overall pipeline ofour proposed CoRTX.
Revised: Figure 2 illustrates the overall pipeline of CoRTX.

Original: CoRTX is built under a constrastive learning paradigm with  positivedata augmentation and negativedata sampling (Kim et al., 2016; Dhurandhar et al., 2018) for providing a real-time explanation.
Revised: In particular, our pipeline first uses an explanation encoder for the contrastive explanation representation learning, and then an explanation head is further tuned with a small amount of explanation labels. An explanation-oriented positive augmentation and negative sampling strategy is designed for the explanation representation learning process.

Original: 3. P OSITIVE D ATA A UGMENTATIONS TOWARD S IMILAR E XPLANATION
Revised: 3. P OSITIVE D ATA A UGMENTATION TOWARD S IMILAR E XPLANATION

Original: Different from  conventional data augmentation (Liu et al., 2021) for representation learning, CoRTX develops an explanation-oriented data augmenting strategy for training RTX.
Revised: Different from the conventional data augmentation (Liu et al., 2021) for representation learning, CoRTX develops an explanation-oriented data augmentation strategy for training RTX, where the augmented positive and negative instances of explanation are proved to be beneficial (Kim et al., 2016; Dhurandhar et al., 2018).

Original: Consider an anchor datapoint x ∈ X , the  synthetic positive data collection is randomly sampled by m times of perturbation of x , which is formally defined as:
Revised: Specifically, considering an anchor data instance x ∈ X , the set of synthetic positive instances X + is generated via m times of independent perturbations on x :

Original: [Equation]
Revised: [Equation]

Original: where B ( M, 0 . 5) denotes M -dimensional binomial distribution.
Revised: where S i is sampled from the M -dim binomial distribution, and λ controls the probability density.

Original: An optimal compact positive datapoint is ideally composed of similar important features to the anchor data point, which leads to a slight variation in prediction scores between the two datapoints.

Original: Thus, CoRTX determines an compact positive datapoint (cid:101) x + by holding the minimum difference gap between the prediction scores of x and (cid:101) x+ . Formally, we make the selection strategy as follows:
Revised: CoRTX provides the explanation-oriented data augmentation by selecting the compact positive instances (cid:101) x

Original: [Equation]
Revised: + from X + .

Original: We further propose Theorem 1 to provide a theoretical analysis to support the proposed positive data augmentation in CoRTX.
Revised: Intuitively, (cid:101) x + is expect to obtain the most similar feature attribution as the corresponding x .

Original: Theorem 1 basically shows that theideal greedy selectionstrategy in CoRTX can degrade the error difference ofexplanation values between the anchor point x and the positive sample (cid:101) x + .
Revised: However, the greedy selection of (cid:101) x + needs the explanation score of each instance in X + , which is hard to access in practice.

Original: This demonstrates the compact positive sample (cid:101) x
Revised: To tackle the issue, we propose Theorem 1 to guide the selection of (cid:101) x

Original: + has similar explanation to anchor point x .
Revised: + through a set of synthetic positive instances without accessing the explanation scores.

Original: The proof of Theorems 1 is provided in Appendix A.
Revised: The proof of Theorems 1 is shown in Appendix A.

Original: Theorem 1 ( Compact Alignment ) .
Revised: Theorem 1 ( Compact Alignment ) .

Original: Let f ( x ) be a K  -Lipschitz  continuous functionfor the given input sample x and Φ ( x ) =
Revised: Let f ( x ) be a K f -Lipschitz 2 continuous function and Φ ( x ) =

Original: [ ϕ 1 ( x ) , · · · , ϕ M ( x )] be the  importance score of each feature, where ϕ i ( x ):=
Revised: [ ϕ 1 ( x ) , · · · , ϕ M ( x )] be the feature importance scores, where ϕ i ( x ) :

Original: E
Revised: =

Original:  S⊆ U \{ i } (cid:2)f
Revised: E S⊆ U \{ i } (cid:2)

Original:  ( (cid:101) x
Revised: f ( (cid:101) x

Original: S∪{ i } ) − f ( (cid:101) x S ) (cid:3) .
Revised: S∪{ i } ) − f ( (cid:101) x S ) (cid:3) .

Original: Given a perturbed sample (cid:101) x ∈X + satisfying min 1 ≤ i ≤ M ϕ
Revised: Given a perturbed positive instance (cid:101) x ∈X + satisfying min 1 ≤ i ≤ M ϕ

Original: i ( (cid:101) x ) ≥ 0 , the explanation difference || ϕ
Revised: i ( (cid:101) x ) ≥ 0 , the explanation difference ∥

Original:  ( x ) − ϕ ( (cid:101) x ) || 2 is bounded by the prediction difference | f ( x ) − f ( (cid:101) x ) | as
Revised: Φ ( x ) − Φ ( (cid:101) x ) ∥ 2 is bounded by the prediction difference | f ( x ) − f ( (cid:101) x ) | as

Original: [Equation]
Revised: [Equation]

Original: where γ 

Revised: where γ

Original: =
Revised: =

Original: K || x || 2 and K  ≥ 0 is the Lipschitz constant of function f ( · ) .
Revised: K f ∥ x ∥ 2 and K f ≥ 0 is the Lipschitz constant of function f ( · ) .

Revised: Theorem 1 basically shows that the explanation difference can be bounded by the prediction difference between the anchor x and the perturbed positive instance (cid:101) x .

Original: According to Theorem 1, we can obtain an optimal positive sample for the least upper bound of || ϕ ( x ) − ϕ ( (cid:101) x ) || 2 , when CoRTX is able to select an optimal compact positive sample under the selected mask conditions.
Revised: The compact positive instance (cid:101) x + in Theorem 1 can derive the minimal difference of model outputs and further result in a similar explanation to the anchor x i . Following Theorem 1, CoRTX selects (cid:101) x + i ∈ X

Original: The selected masking condition in Theorem 1 ensures that each non-masking feature obtains positive contributions toward the model prediction f ( · ) , which prevents the ideal candidate of compact positive samples from being a set of noise sampling.
Revised: + by holding the minimum preceding difference between of f ( x ) and f ( (cid:101) x i ) . Formally, the selection strategy can be indicated as: [Equation] where f ( · ) is the target model for explanation.

Original: However, hard positive sample selection has been proved to benefitthe contrastive learning (Grill et al., 2020; Kalantidis et al., 2020).
Revised: Selecting hard positive instances has been proved to benefit contrastive learning (Grill et al., 2020; Zhu et al., 2021).

Original: The compact positive data may be a slightly perturbed sample when selecting from the universe set of perturbed datapoints and aggravate the contrastive learning process of CoRTX.
Revised: The hard positive instance is expected to be nearly positive to the anchor instance. However, our (cid:101) x + may become excessively positive when selected from the universe set of X + . Following this criterion, X + is set to be much smaller than the entire universe set,

Original:  In practice,considering to generate a hard positive sample, CoRTX receives an compact positive data point (cid:101) x
Revised: where |X + | = m ≪ 2 M . For example, we set |X + | =300 ≪ 2 96 on the Bankruptcy dataset (Liang et al., 2016). In practice, CoRTX uses Equation 4 to search the compact positive instance (cid:101) x

Original: +i by selecting fromthe subset of synthetic positive set X + .
Revised: + from X + , where the excessively positive instances may be avoided.

Revised: In this way, CoRTX can ensure hard positive instances to be as many as possible, which benefits the explanation representation learning process.

Original: 3. EXPLANATION C ONTRASTIVE L OSS
Revised: 3. E XPLANATION C ONTRASTIVE L OSS

Original: Unlike thedownstream tasks focusing by existing contrastive learningapproaches (He et al., 2020;
Revised: The proposed CoRTX adopts the contrastive learning (He et al., 2020;

Original: Chen et al., 2020b), CoRTX adopts the contrastive learning with positive data augmentation and negative sampling to generate the explanation representation  of each data instance.
Revised: Chen et al., 2020) to generate the explanation representation by training an explanation encoder g ( · | θ g ) : R M → R d . Here, a positive pair and several negative pairs are exploited during the training of g ( · | θ g ) .

Original: Specifically, a positive pair includes an anchordata instance x i and a compact perturbed augmentation (cid:101) x
Revised: A positive pair includes the anchor x i and a compact positive instance (cid:101) x

Original: + i .
Revised: + i .

Original: A negative pair contains an anchordata instance x i and another data point x j , where j ̸ = i . The proposed explanation encoder g ( · | θ g ) : R M → R d in CoRTX aims to generate the explanation representation.
Revised: A negative pair contains the anchor x i and a randomly selected instance x j ( j ̸ = i ), where the explanations are mostly different.

Original: Let h i
Revised: Let h i

Original: = g ( x i | θ g ) be the encoded explanation representation of x i and (cid:101)
Revised: = g ( x i | θ g ) , (cid:101)

Original: h
Revised: h

Original: + i
Revised: + i

Original: =
Revised: =

Original: g
Revised: g

Original: ( (cid:101) x
Revised: ( (cid:101) x

Original: + i | θ g ) be the encoded compact positive sampling (cid:101) x + i . CoRTX updates the explanation encoder g ( · | θ g ) by measuring thesimilarity through dot product. Given an anchor encoded representation 
Revised: + i | θ g ) be the explanation representation of the positive pair, and h i , h j = g ( x j | θ g ) be the representation for a negative pair.

Original: h i , theexplanation encoder g ( · | θ g ) can be optimized with one encoded positive sample pair ( h i , (cid:101) h + i ) and N encoded negative sample pairs ( h i , h j ) , where i ̸ =
Revised: CoRTX trains the encoder g ( · | θ g ) by maximizing the dot similarity of positive pairs and minimizing the dot similarity of negative pairs.

Original: j . We follow the existing work (He et al., 2020; Chen et al., 2020a) to construct the contrastive loss function of CoRTX and minimize the contrastive lossfunction illustrated as follows:[Equation]
Revised: Following the criterion, g ( · | θ g ) essentially minimizes the contrastive loss as follows:

Original: [Equation]
Revised: [Equation]

Original: where τ is a temperature hyper-parameter Wu et al. Note that the form of contrastive loss function can be replaced with other implementations (Jaiswal et al., 2020).
Revised: where τ is a temperature hyper-parameter (Wu et al., 2018) and N denotes the batch size for training.

Original: To provide downstream results from pre-train representation, a explanation head η  : R d → R M is exploited to generate the feature explanationin the downstream tasks.
Revised: To provide the explanation for f ( · ) from the learned representation in g ( · | θ g ) , an explanation head η ( · | θ η ) : R d → R M is further tuned based on the certain explanation tasks.

Original: Specifically, η ( · | θ η ) is learned with pre-trained explanation representation made from g ( x i | θ g ) and very few amount of explanation labels (e.g., the Shapley Values), which makes the computational cost be affordable in real-world scenarios.
Revised: Specifically, η ( · | θ η ) is tuned with a small amount of explanation labels (e.g., the Shapley values), where the computational cost is relatively affordable in real-world settings.

Original: Moreover, η ( · | θ η ) can be designed to adapt different scenarios of model interpretation.
Revised: η ( · | θ η ) is specifically designed to cope with the different scenarios of model explanation.

Original: In this work, CoRTX is built under two common explanation scenarios, which are the feature attribution task and featureimportance ranking task.
Revised: In this work, CoRTX is studied under two common explanation scenarios, i.e., feature attribution task and feature ranking task.

Original: To ensure theaccuracy of generated explanation from CoRTX, We here provide Theorem 2 to reveal the efficacy of the explanation error of CoRTX is under theoretical support.
Revised: To ensure the explanation quality of CoRTX, we here provide Theorem 2 to show that the explanation error of CoRTX is bounded with theoretical guarantees.

Original: The proof of Theoremsis provided in Appendix A.
Revised: The proof of Theorems 2 is provided in Appendix B.

Original: Theorem 2 ( Explanation Error Bound ) .
Revised: Theorem 2 ( Explanation Error Bound ) .

Original: Given a training set D , a testing set C , anda welltrained explainer ˆ ϕ =
Revised: Let η ( ·| θ η ) be K η -Lipschitz continuous, and ˆ Φ ( · )

Original: η ◦ g , where g denotes the contrastive explanation encoder to generate low rank explanation embeddings and η represents explanation head.
Revised: =

Original: Assume ˆ ϕ ( · ) is a K h -Lipschitz continuity.
Revised: η ( g ( ·| θ g ) | θ η ) be the estimated explanations.

Original: For all x j ∈ D in training set, if there exist E > 0 , such that the training error satisfies || ϕ ( xj ) − ˆ ϕ ( xj ) || 2 ≤ E . Then, for any testing datapoint x k ∈ C ,  thetesting explanation error 
Revised: If the encoder g ( ·| θ g ) is well-trained, satisfying ∥ Φ ( x ) − ˆ Φ ( x ) ∥ 2 ≤E ∈ R + , then the explanation error on testing instance x k can be bounded as:

Original: Err ( ˆ ϕ ) = || ϕ ( x k ) − ˆ ϕ ( x k )
Revised: [Equation]

Original: || 2 can be bounded as:

Original: [Equation]
Revised: [Equation]

Original: (6) where (cid:101) x
Revised: where (cid:101) x

Original: + k is the compact positive sample, h x i = g
Revised: + k is a compact positive instance, h x k = g

Original: ( x i ) ,  γ [-
-] = K ||
Revised: ( x k | θ g ) , and γ [-
-]k = K f

Original:  x k || 2 , and
Revised: ∥ x k ∥ 2 .

Original: K h ≥ 0 is the Lipschitz constant of prediction model f ( · ) . With theoretical analysis, we conclude two advantages brought from CoRTX in Remark 1. The first advantage lies in the explanation representation learning. The second advantage takes from the compact positive selection strategy.
Revised: Theorem 2 shows that the upper bound of explanation error depends on four different terms, which are γ k , training error E , | f ( x ) − f ( (cid:101) x + k ) | , and ∥ h (cid:101) x

Original: Remark 1.
Revised: k − h x k ∥ 2 .

Original: Within our proposed CoRTX framework, the upper bound of explanation error Err( ˆ ϕ ) can be reduced due to the following reasons:
Revised: CoRTX contributes to minimize the upper bound by explicitly minimizing | f ( x ) − f ( (cid:101) x + k ) | and ∥ h (cid:101) x

Original:  •Our proposed CoRTX  minimizes the representation distance of || h (cid:101) x 
Revised: k − h x k ∥ 2 as follows: • CoRTX selects the compact positive instances following the selection strategy in Equation 4, which obtains the minimal value of | f ( x ) − f ( (cid:101) x + k ) | ; • CoRTX maximizes the similarity between positive pairs while training the encoder g ( ·| θ g ) , which minimizes the value of ∥ h (cid:101) x k − h x k ∥ 2 .

Original: k − h x k || 2 which contributes to minimize thelast item of explanation error bound. • Our proposed CoRTX framework provides a compact positive selection strategy to get smaller | f ( x ) − f ( (cid:101) x + k ) | than randomly selection.
Revised: Besides the two terms mentioned above, the error E is naturally minimized after g ( · | θ g ) is trained, and γ k keeps constant for the given f ( · ) . Thus, we can theoretically guarantee the explanation error bound of CoRTX by Equation 6. This ensures the capability of CoRTX to provide effective explanations to the target model over the testing set.

Original: 3. A LGORITHM OF C O RTX
Revised: 3. A LGORITHM OF C O RTX

Original: The outline of CoRTX is given in Algorithm 1.
Revised: The outline of CoRTX is given in Algorithm 1.

Original: Specifically, CoRTXfirst follows Equation 3 to receive compact positive selection (lines 4-5), and then updates the explanation encoder g ( · | θ g ) according to Equation 5 (line 6).
Revised: CoRTX follows Equation 4 for compact positive selection (lines 4-5), and learns the explanation encoder g ( · | θ g ) according to Equation 5 (line 6).

Original: The training iterations stop once the explanation encoder g ( · | θ g ) converges to the optimal.
Revised: The training iteration terminates when g ( · | θ g ) is converged.

Original: After CoRTX framework converges, g ( · | θ g ) generates the explanation representation for eachdata instance x
Revised: Overall, g ( · | θ g ) learns the explanation representation for each instance x

Original: = [ x 1 , · · · , x M ] ∈X . A explanation head η ( · | θ η ) is then fine-tuned by involving very limited explanation labelsand explanation representation for generating efficient and faithful explanations.
Revised: = [ x 1 , · · · , x M ] ∈ X , and η ( · | θ η ) is then fine-tuned with a small amount of explanation labels for particular tasks. The algorithmic process of CoRTX can be expressed as η ( g ( x i | θ g ) | θ η ) in general.

Original: [Table]
Revised: [Table]

Original: Input DNN model f , input values x = [ x 1 , · · · ,x M ] .

Original: [Table]
Revised: [Table]

Original: Output Estimation contribution values of each feature [ˆ ϕ 1 , ..., ˆ ϕ M ] .

Original: 3 while not convergence do Generate the synthetic positive instances X + = { S i ⊙ x +( 1 − S i ) ⊙ x r | S i ∼B ( M, 0 . 5) } . Select the compact positive sampling (cid:101) x + = arg min (cid:101) x
Revised: [Table]

Original: i ∈X + | f ( x ) − f ( (cid:101) x i ) | and the set of negative samples { x j | j ̸ = i } .

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conductthe experiments to demonstrate the effectiveness of CoRTX, aiming to answer the following  research questions: RQ1: How does CoRTX perform  compared with state-of-the-art baselines?
Revised: In this section, we conduct experiments to evaluate the performance of CoRTX, aiming to answer the following three research questions: RQ1: How does CoRTX perform on explanation tasks in terms of the efficacy and efficiency compared with state-of-the-art baselines?

Original: RQ2: What is the impact brought frompre-trained CoRTX to reduce usage amount of ground-truth explanation labels?
Revised: RQ2: Is the explanation representation from CoRTX effective on fine-tuning the explanation head?

Original: RQ3: How does the explanation representation generated by CoRTX improve the generalization to the explanation tasks?
Revised: RQ3: Does synthetic augmentation contribute to the explanation performance of CoRTX?

Original: 4. D ATASETS AND B ASELINES
Revised: 4. D ATASETS AND B ASELINES

Original: Datasets.
Revised: Datasets.

Original: Our experiments consider two tabular datasets: CensusIncome (Dua & Graff, 2017) with 13 features, Bankruptcy (Liang et al., 2016) with 96 features and one image dataset: CIFAR10 (Krizhevsky et al., 2009) with 32 × 32 pixels.
Revised: Our experiments consider two tabular datasets: Census (Dua & Graff, 2017) with 13 features, Bankruptcy (Liang et al., 2016) with 96 features, and one image dataset: CIFAR-10 (Krizhevsky et al., 2009) with 32 × 32 pixels.

Original: The preprocessing and statisticsdetails of three datasets areall provided in Appendix B. Baseline Methods.
Revised: The preprocessing and statistics of three datasets are provided in Appendix C. Baseline Methods.

Original: On the CensusIncome and Bankruptcy datasets, CoRTX is compared with two RTX models, which are Supervised RTX, FastSHAP (Jethani et al., 2021b), and two representative local explanation models, KernelSHAP (KS) (Kokhlikyan et al., 2020) and Permutation Sampling (PS) (Mitchell et al., 2021).
Revised: In Census and Bankruptcy datasets, CoRTX is compared with two RTX methods, i.e., Supervised RTX and FastSHAP (Jethani et al., 2021), as well as two non-amortized explanation methods, i.e., KernelSHAP (KS) (Lundberg & Lee, 2017) and Permutation Sampling (PS) (Mitchell et al., 2021).

Original: On the CIFAR-10 dataset, CoRTX is compared withone RTX model, FastSHAP(Jethani et al., 2021b), and several local explanation models, DeepSHAP (Lundberg & Lee, 2017), Saliency (Simonyan et al., 2013), Integrated Gradients  (Sundararajan et al., 2017), SmoothGrad Smilkov et al.
Revised: In CIFAR-10 dataset, CoRTX is compared with FastSHAP and other non-amortized methods, including DeepSHAP (Lundberg & Lee, 2017), Saliency (Simonyan et al., 2013), Integrated Gradients (IG) (Sundararajan et al., 2017), SmoothGrad (Smilkov et al., 2017), and

Original: (2017), and GradCAM (Selvaraju et al., 2017).
Revised: GradCAM (Selvaraju et al., 2017).

Original: More details about the baseline methods can be referred to Appendix B.
Revised: More details about the baselines can be found in Appendix D.

Original:  E XPERIMENTAL S ETTINGS AND E VALUATION M ETRICS
Revised: 4. E XPERIMENTAL S ETTINGS AND E VALUATION M ETRICS

Original: In this section, we introduce the experimental settings and metrics used for evaluating CoRTX.
Revised: In this part, we introduce the experimental settings and metrics on evaluating CoRTX.

Original: The task settings and implementation details are shown as follows.
Revised: The considered explanation tasks and implementation details are shown as follows.

Original: Feature Attribution Task.
Revised: Feature Attribution Task.

Original: The task-specific CoRTX under such training is denoted as CoRTXMSE. CoRTX-MSE first provides pre-trained explanation representation h i of input instance x
Revised: This task aims to test the explanation performance on feature attribution. We here implement CoRTX-MSE on fine-tuning the explanation head η

Original: i from g ( x i | θ g ) .
Revised: ( x i | θ η ) .

Original: After this, feature attribution score is estimated based on a regression-oriented explanation head η ( h i | θ η ) with tiny amounts of Shapley-based explanation labels.

Original: The explanations are generated by [ˆ ϕ 1 , · · · , ˆ ϕ M ]= η
Revised: Given g ( x i | θ g ) , the explanation values are predicted through [ˆ ϕ 1 , · · · , ˆ ϕ M ] =

Original:  ( h i | θ  η ) , where ˆ ϕ i indicates the contribution of feature i .
Revised: η ( g ( x i | θ g ) | θ η ) , where ˆ ϕ i indicates the attribution scores of feature i .

Original: The explanation head η
Revised: Let [ ϕ 1 , · · · , ϕ M ] be the explanation label of an instance x i ,

Original:  ( h i | θ η ) learns to minimize the mean-square loss L MSE = 1 M (cid:80) [-Mi =
-]
Revised: CoRTX-MSE learns η ( · | θ η ) by minimizing the mean-square loss L MSE = 1 M (cid:80) [-Mi =
-]Mj

Original: (ˆ ϕ i − ϕ i )between estimated scores η ( h i | θ η ) = ˆ ϕ i and explanation labels ϕ i .Since CoRTX-MSE is evaluated on Shapley-based explanation values, we adjust the predicted explanation scores to fit the additive property of Shapley values on evaluation stage, following the common additive efficient normalization (Jethani et al., 2021b; Ruiz et al., 1998).
Revised: =1 (cid:0) ˆ ϕ j − ϕ j (cid:1) 2 .

Original: To evaluate the performance of CoRTX, the estimated feature attribution scores are measured by the ℓ 2 -error (Jethani et al., 2021b). The mean ℓ 2 -error indicates the Euclidean distance error from the explanation labels given by 
Revised: To evaluate the performance, we follow the existing work (Jethani et al., 2021) to estimate the ℓ 2 -error of each instance on the testing set, where the ℓ 2 -error is calculated by (cid:113)(cid:80) Mj

Original: [Equation]
Revised: =1 (cid:0) ϕ

Original: where N denotes the cardinality of testing set and M is feature numbers of each data instance.
Revised: j − ˆ ϕ j (cid:1) 2 .

Original: Feature Importance Ranking Task.
Revised: Feature Importance Ranking Task.

Original: In this set of experiments, our explanation head η ( h i | θ η ) aims to estimate the featureimportance ranking index  [ˆ r 1 , · · · , ˆ r M ] . The task-specific CoRTX in this scenario is denoted as CoRTX-CE.
Revised: This task aims to evaluate the explanation performance on feature ranking index. We here implement CoRTX-CE for fine-tuning the explanation head η ( · | θ η ) . Given g ( x i | θ g ) , the feature ranking index is generated by [ˆ r 1 , · · · , ˆ r M ] =

Original: Here, the ground-truth ranking annotations are assumed to be the ranking index from explanation scores. Given the ranking annotations [r 1 , · · · , r M ] of each training sample, the explanation head η
Revised: η ( g ( x i | θ g ) | θ η ) for each instance x i . The predicted feature ranking index [ˆr 1 , · · · , ˆr M ] is given by ˆr j = arg max ˆ r j for 1 ≤ j ≤ M , where arg max( · ) returns the index with the maximal importance score.

Original:  ( h i | θ η ) learns the feature importance ranking with pre-trained explanation representation. CoRTX-CE minimize the loss functiongiven by L CE =
Revised: Let [r 1 , · · · , r M ] 3 be the ground-truth ranking label, CoRTX-CE learns η ( · | θ η ) by minimizing the loss function L CE =

Original: (cid:80)
Revised: (cid:80)

Original: Mi =1 l (ˆ r i , r i ; θ h ) , where l (ˆ r i , r i ; θ ) denotes the cross entropy loss.
Revised: Mj = 1 CrossEntropy(ˆ r j , onehot (r j )) .

Original: The feature importance ranking is evaluated by the ranking accuracy (Wang et al., 2022). Specifically, it indicates the ratio of the correct-ranked features to all features .
Revised: To evaluate the feature importance ranking, we follow existing work (Wang et al., 2022), where the ranking accuracy of each instance is given by

Original: By considering more significance for the important features than the trivial features, the accuracy of descending feature ranking is given by:
Revised: Rank

Original: [Equation]
Revised: [Equation]

Original: where the factor 1 m at the numerator highlights the contributions of important features to the accuracy, and the factor (cid:80) Mm =1 1 m at the denominator normalizes the accuracy such that 0 ≤ Rank ACC ≤ 1 .
Revised: [Equation]

Original: Evaluation of Efficiency.
Revised: Evaluation of Efficiency.

Original: To evaluate speed performance of the above explanation scenarios, we adopt algorithmic throughput  to evaluate the efficiency (Wang et al., 2022).
Revised: The algorithmic throughput is exploited to evaluate the speed of explaining process (Wang et al., 2022; Teich & Teich, 2018).

Original: The measurement is calculated byThroughput = N test t total , where N test and t total denote the testing instance number and the overall time consumption of generating the explanations, respectively.
Revised: Specifically, the throughput is calculated by N test t total , where N test and t total denote the testing instance number and the overall time consumption of explanation derivation, respectively.

Revised: Higher throughput indicates the higher efficiency of the explanation process.

Original: For the three datasets, the testing instance number N test is given in Appendix B, and t total is measured based on the physical computing infrastructure given in Appendix B. [Figure]
Revised: In experiments, t total is measured based on the physical computing infrastructure given in Appendix E.3.

Original: Implementation Details.
Revised: Implementation Details.

Original: Three different prediction models f ( · ) are applied for verifying the agnostic property of proposed CoRTX.
Revised: In experiments, CoRTX is implemented under three different prediction models f ( · ) .

Original: AutoInt (Song et al., 2019) is adopted for Census Income dataset,
Revised: AutoInt (Song et al., 2019) is adopted for Census data, MLP model is used for

Original: MLP model is used for Bankruptcy dataset, and ResNet-18 is considered for CIFAR-10.
Revised: Bankruptcy data, and ResNet-18 is employed for CIFAR-10.

Original: To generate ground truth explanation labels, we calculate the exact Shapley values for CensusIncome dataset.
Revised: As for the ground-truth explanation labels, we calculate the exact Shapley values for Census dataset.

Original: However, due to the high computational resource requirements, we utilize the approximated methods to generate explanation labels until the convergence Covert & Lee (2021); Jethani et al. (2021b).
Revised: For the other two datasets, we utilize the approximated methods to generate explanation labels (Covert & Lee, 2021; Jethani et al., 2021), due to the high time complexity of exact Shapley value calculation.

Original: Since estimated values from Antithetical Permutation Sampling (APS) (Mitchell et al., 2021; Lomeli et al., 2019) and KernelSHAP Lundberg & Lee (2017) are proved to approach to  exact Shapley values when it obtains high samples Covert & Lee (2021).
Revised: The estimated values from Antithetical Permutation Sampling (APS) (Mitchell et al., 2021; Lomeli et al., 2019) and KS are proved to be infinitely closed to the exact Shapley values when they involve sufficient samples (Covert & Lee, 2021).

Original: Thus, APS is set to generate the explanation labels for Bankruptcydataset and KernelSHAP yields the explanation labels for CIFAR-10.
Revised: In experiments, we employ APS on Bankruptcy and KS on CIFAR-10 for obtaining explanation labels.

Original: Moreimplementation details are introduced at Appendix B.
Revised: More details are introduced in Appendix E. [Figure] [Figure]

Original: 4. T ABULAR E XPERIMENTS 4.3. E XPLANATION  AND E FFICIENCYP ERFORMANCE (RQ1)
Revised: 4. T ABULAR E XPERIMENTS 4.3. E XPLANATION E FFICACY AND E FFICIENCY (RQ1)

Original: We compare CoRTX with the baseline methods on efficacy and generating speed of the predicted explanations.
Revised: We compare CoRTX with the baseline methods on deriving model explanations.

Original: The results  are shown by ℓ 2 -error versus throughputon feature attribution task and Rank ACC versus throughput on feature importance ranking task.
Revised: The results in feature attribution and feature ranking are shown by ℓ 2 -error versus throughput and Rank ACC versus throughput, respectively.

Original: RTX frameworks, such as CoRTX and FastSHAP, build on the training set and evaluate on the testing dataset.

Original: For other local methods, such as KernelSHAP and Permutation Sampling, the explanations are generated based on 2 n times of model evaluation, where 3 ≤ n ≤ 11 .

Original: Larger times of model evaluation given in local methods indicate a smaller throughput.

Original: The reported results of CoRTX adopting 25% of the explanation labels on fine-tuning stage.
Revised: The reported results of CoRTX adopts 25% of the explanation labels on fine-tuning the explanation head.

Original: Theoutcomes of explanation throughput versus explanation performance on the Census Income dataset is illustrated in Figures 3 (a) and (c) and demonstrate Figures 3 (b)  and  (d) on the Bankruptcy dataset under feature attribution task and ranking task.
Revised: The performance comparisons are illustrated in Figures 3, where (a), (b) demonstrate the performance on feature ranking and (c), (d) show the performance on feature attribution. According to the experimental results, we have the following observations:

Original: • CoRTX vs. Local Methods :For KernelSHAP and PS, a sharp decrease of ranking accuracy and an increase of the ℓ 2 -error can be observed as the growth of throughput. Compared with local explanation methods obtaining large times of model evaluations, CoRTX achieves competitive explanation performance.
Revised: • CoRTX vs. Non-Amortized Methods : Compared with the non-amortized explanation methods with large times of model evaluations, CoRTX achieves competitive explanation performance.

Original: For example, CoRTX is competitive to PS  with  2 8 times of model evaluation andKernelSHAP with 2 10 times on Census Income dataset.
Revised: In Census data, we observe CoRTX is competitive to PS and KS with respectively 2 9 and 2 10 of model evaluations.

Revised: For KS and PS, it is noted that there is a significant decrease in Rank ACC and an obvious increase in ℓ 2 -error as the throughput grows.

Original: This indicates KernelSHAP and PS  suffer from an undesirable tradeoff between  explanation speed and performance.
Revised: This indicates that KS and PS both suffer from an undesirable trade-off between the explanation speed and performance.

Original: In contrast, our proposed CoRTX provides both fast and accurate explanation.
Revised: In contrast, the proposed CoRTX achieves both efficient and effective explanations for the target model.

Original: • CoRTX vs. FastSHAP : CoRTX outperforms FastSHAP on Rank ACC and ℓ 2 -error under the same level of throughput.
Revised: • CoRTX vs. FastSHAP : CoRTX outperforms FastSHAP on Rank ACC and ℓ 2 -error under the same level of throughput.

Original: This  shows thatour proposed CoRTX provides accurate and faithful explanations in the scenario of RTX.
Revised: This observation shows that CoRTX provides more effective explanation under the scenario of RTX.

Original: • CoRTX-MSE vs. CoRTX-CE : CoRTX successfully provides accurate solutions for  two explanation tasks.
Revised: • CoRTX-MSE vs. CoRTX-CE : CoRTX consistently provides effective solutions for the two explanation tasks.

Original: CoRTX-CE outperforms CoRTX-MSE in the feature  ranking task, while  CoRTX-MSE still remains competitive performance on feature attribution task.The results
Revised: CoRTX-MSE is competitive on the feature attribution and ranking task, while CoRTX -CE performs even better than CoRTX-MSE in the ranking task.

Original: indicate the necessity of selecting appropriate downstream loss on fine-tuning stage. To sum up, CoRTX has the potential capability and flexibility to be applied to different scenarios of model explanation.
Revised: The superiority of CoRTXCE on ranking task largely results from the loss function in fine-tuning, where appropriate tuning loss can help enhance the explanation quality for certain tasks.

Original: 4.3. I MPACT FROM P RE - TRAINED E XPLANATION R EPRESENTATION (RQ2)
Revised: 4.3. C ONTRIBUTIONS ON E XPLANATION R EPRESENTATION (RQ2)

Original: In thisset of experiments, we investigate the effects ofpre-trained explanation representation brought to the usage amount of explanation labels.
Revised: In this experiments, we study the effectiveness of explanation representation in deriving explanation results.

Original: Given the fixed pre-trained explanation representation, Figure 4 demonstrates the explanation performance of the RTX frameworks with different annotation usage ratios.
Revised: Given the obtained explanation representation, Figure 4 demonstrates the explanation performance of the RTX frameworks with different label usage ratios.

Original: We compare CoRTX withtwo baseline RTX frameworks (i.e., FastSHAP and supervised
Revised: We compare CoRTX with FastSHAP and Supervised RTX and summarize the key observations as below:

Original: RTX).
Revised: [Figure]

Original: For the fair comparison, CoRTX and supervised RTX adopt the same loss function under the same task.
Revised: u c k CoRTX FastSHAP KernelSHAP Saliency

Original: We summarize the key results as follows:
Revised: IG Smoothgrad Gradcam Deepshap

Original: [Figure]
Revised: [Figure]

Original: • Effectiveness of Pre-training :By providing pre-trained explanation representation, CoRTX-MSE and CoRTX-CE consistently outperform the supervised RTX on different label proportions.
Revised: • Effectiveness of Encoding : CoRTX-MSE and CoRTX-CE consistently outperform the Supervised RTX on different label proportions.

Original: The results on two explanation shows that CoRTX provides effective explanation representationh
Revised: The results on two explanation tasks show that CoRTX provides effective explanation representation

Original:  i on fine-tuning theexplanation head η
Revised: h i in fine-tuning the head η

Original: ( h i | θ η ) 
Revised: ( h i | θ η ) .

Original: • Sparsity of Annotations : CoRTX-MSE and CoRTX-CE canboth provide faithful explanation results when explanations labels arevery limited.
Revised: • Sparsity of Labels : CoRTX-MSE and CoRTX-CE can provide effective explanation outcomes when explanation labels are limited.

Original: This indicates CoRTX can be potentially applied to real-world large scale datasets because the computational complexity of generating the explanation labels is extremely high in practice.
Revised: The results indicate that CoRTX can be potentially applied to large-scale datasets, where the explanation labels are computationally expensive to obtain.

Original: • Effectiveness of Fine-tuning :Both supervised RTX and CoRTX  outperform FastSHAP on ranking accuracy and ℓ 2 -error under 5%of explanation label usage.
Revised: • Effectiveness of Labels : CoRTX and Supervised RTX consistently outperform FastSHAP when having more than 5% explanation labels.

Original: CoRTX can significantly reduce the usage of good quality labels by providing pre-trained explanation representation.

Original: For the fast explanation yielding, FastSHAP synchronously generates theapproximated explanation labels  during theexplainer training.
Revised: FastSHAP synchronously generates the explanation labels with low-sampling times during the training.

Original: However, comparing FastSHAP to supervised RTX, theresults indicates that exact explanation labels usage (e.g., exact Shapley values) benefits theexplanation performance than approximated labels. This reveals that CoRTX obtains the advantages from limited usage of exact labels and leads to fast estimation and accurate explanation.
Revised: In contrast, CoRTX and Supervised RTX are trained with high-quality labels, which are the exact Shapley values or the approximated ones with high-sampling times. The results indicate that high-quality explanation labels can significantly benefit the learning process of the explainer, even when exploiting only 5% explanation labels.

Original: 4.3. A BLATION S TUDIES ON S YNTHETIC P OSITIVE A UGMENTATIONS 
Revised: 4.3. A BLATION S TUDIES ON S YNTHETIC P OSITIVE A UGMENTATIONS (RQ3)

Original: [Figure]
Revised: [Figure]

Original: In this section, we conduct an ablation test on the greedy positive augmenting strategy in Section 3.1 to evaluate the efficacy on explanation.
Revised: We further conduct the ablation studies on synthetic positive augmentation in Section 3.1, aiming to prove its validity in learning explanation representation.

Revised: Since negative pairs are randomly selected in CoRTX, we here mainly focus on the positive augmentation in our studies.

Original: We perform the ablation experiments under the Census dataset on the feature ranking and feature attribution task.
Revised: The ablation studies are conducted under both attribution and ranking tasks on Census dataset.

Original: First, the greedy positive augmenting strategy is replaced with a random selection of candidate synthetic positive data, which is denoted as CoRTX w/o Compact Alignment (CA)  . 
Revised: In the experiments, CoRTX is compared with two methods: CoRTX w/o Compact Alignment (CA) and CoRTX w/ Maximum Alignment (MA) . To be concrete, CoRTX w/o

Original: Second, we replace the proposed greedy positive augmenting strategy with  a maximum positive augmenting strategy in our ablation studies, which is denoted as CoRTX w/ Maximum Alignment (MA) .
Revised: CA replaces Equation 4 with the existing settings of contrastive learning, which is a random masking strategy. CoRTX w/ MA replaces the minimum selecting strategy in Equation 4 with the maximum one on synthetic instance set, which is randomly masked from the anchor instance.

Original: The other settings of CoRTX w/o CA and
Revised: The remaining settings follow the same as CoRTX.

Original: CoRTX w/ MA follow the traditional contrastive learning, which uniformly samples the positive data.

Original: Figure 5 demonstrates the results of CoRTX w/o CA and CoRTX w/ MA comparing to our proposed CoRTX.
Revised: Figure 6 demonstrates the comparisons among CoRTX w/o CA , CoRTX w/ MA and CoRTX.

Original: As shown in the figure, we can observe that CoRTX outperforms CoRTX w/o CA and CoRTX w/ MA under different proportions of explanation label usage on two explanation tasks.
Revised: We can observe that CoRTX outperforms CoRTX w/o CA and CoRTX w/ MA under different proportions of explanation label usage.

Original: The results reveal that the proposed explanation-oriented sampling strategy in CoRTX can significantly benefit the efficacy of explanation results.
Revised: This demonstrates that the designed synthetic positive augmentation significantly benefits the efficacy of explanation results.

Original: The results in this ablation study demonstrate the necessity of adopting Compact Alignment in the CoRTX framework instead of positive sampling from conventional contrastive learning.

Original: 4. I MAGE E XPERIMENTS (RQ3)
Revised: 4. I MAGE E XPERIMENTS (RQ1)

Original: Unlike  tabular data that obtains relatively few features, image data is  composed of high-dimension pixel features.
Revised: Unlike the tabular cases, image data is typically composed of high-dimension pixel features.

Original: In this section, we evaluate the performance of explanation representation generated by CoRTX onthe CIFAR-10 dataset.
Revised: To further test the efficacy of explanation representation, we evaluate CoRTX on CIFAR-10 dataset.

Original: We compare CoRTX with FastSHAPutilizing all explanation labels and six other local image explaining methods.
Revised: We compare CoRTX with FastSHAP and six non-amortized image attribution methods.

Original: CoRTX generates the efficient image explanation through one-feed-forward model prediction process.

Original: The explanation results of CoRTX are finetuned using 5%  ground-truth explanation labels.
Revised: The explanation results of CoRTX are fine-tuned using 5% of ground-truth explanation labels.

Original: CoRTX and FastSHAP output 2 × 2 superpixel attribution (Jethani et al., 2021b) for explaining the outcome of image classification.[Figure] CoRTX FastSHAP
Revised: CoRTX and FastSHAP output 2 × 2 super-pixel attributions (Jethani et al., 2021) for explaining the prediction of image classification.

Original: KernelSHAP Saliency IG Smoothgrad
Revised: More details are provided in Appendix E.

Original: Gradcam
Revised: [Table]

Original: Deepshap

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: T o p

Original: A cc u r a c y    100 Inclusion % 0.

Original: T o p

Original: A cc u r a c y
Revised: cc u r a c y

Original: CoRTXFastSHAPKernelSHAPSaliencyIGSmoothgradGradcamDeepshap
Revised: CoRTXFastSHAPKernelSHAPSaliencyIGSmoothgradGradcamDeepshap

Original: [Figure]

Original:  A C ASE S TUDY OF C O RTX
Revised: 4.4. A C ASE S TUDY OF C O RTX

Original: We visualize thepredicted explanation results in Figure 6.
Revised: We visualize the explanation results in Figure 5.

Original: It shows that CoRTX, FastSHAP, and GradCAM are able to highlight the relevant objects corresponding to the image  labels.
Revised: It shows that CoRTX, FastSHAP, and GradCAM are able to highlight the relevant objects corresponding to the image attribution labels.

Original: Specifically,
Revised: Specifically,

Original: CoRTX providesa more faithful explanation result and highlights a precise region which is important for model prediction.
Revised: CoRTX provides more effective explanation results and highlights the particular regions which are essential to the model predictions.

Original: In contrast, we observe that the importance regions localized by FastSHAP are larger than relevant objects, which reveals to be less accurate to the explanations.
Revised: In contrast, we observe that the important regions localized by FastSHAP are larger than relevant objects, which are less accurate for explanation.

Original: Moreover,
Revised: Moreover,

Original: GradCAM provides importance regions only on partial relevant objects, which makes the explanations less faithful.
Revised: GradCAM provides the important regions only on tiny relevant objects, making the explanations less effective.

Original: The other baselines are less competitive due to the wrong regions highlighted, or noisy saliency maps provided.
Revised: Furthermore, CoRTX outperforms the other baselines in most of the examples in case studies.

Original: These observations validate that the explanations from CoRTX are more faithfulness.
Revised: These observations validate that the explanations from CoRTX are more effective.

Original: More case studies are available in Appendix D.
Revised: More case studies are available in Appendix H.

Original: 4.4. Q UANTITATIVE E VALUATION
Revised: 4.4. Q UANTITATIVE E VALUATION

Original: To  investigate thequality of the estimated explanation results on CIFAR-10, we compare CoRTX with two Shapley-based models and several gradient-based local explainers on exclusion and inclusion tasks.
Revised: To quantitatively investigate the explanation results on CIFAR-10, we test CoRTX with the exclusion and inclusion comparing with FastSHAP as well as Shapley-based and gradient-based non-amortized methods.

Original: Following the experimental setting from existing work (Petsiuk et al., 2018; Jethani et al., 2021b), we utilize  Top-1 accuracy and ∆ Log-odds from image classification task as the metric.
Revised: Following the experimental settings from existing work (Petsiuk et al., 2018; Jethani et al., 2021), we utilize the AUC of Top-1 accuracy and the AUC of Log - odds as the metrics for evaluation (Jethani et al., 2021).

Original: The evaluated images are orderly masked according to the estimation of feature importance scores.
Revised: Specifically, the testing images are orderly masked according to the estimated feature importance scores.

Original: Once the important pixels are removed from the input images on exclusion, we expect the Top-1 accuracy  to drop drastically and obtain a lower exclusionAUC. On the contrary, we expect to gain higher inclusion
Revised: Once the important pixels are removed from the input images, Top-1 accuracy and Log - odds are expected to drop drastically, obtaining a low exclusion

Original: AUC for better performance on the inclusion task.
Revised: AUC and a high inclusion AUC.

Original: ∆ Log-odds reveals the opposite instructions where higher exclusion AUC and lower inclusion task denotes better explanation performance. Figure 6 and
Revised: More details about Log - odds are given in Appendix G. Figure 6 and

Original: Table 1 reveal the results ofAUC curves for the exclusion and inclusiontasks under a set ofimages.
Revised: Table 1 demonstrate the results of the exclusion and inclusion under 1000 images.

Original: We observed that CoRTX outperforms allthe other baselines on the two tasks, which obtains the lowest exclusion AUC with Top-1 Accuracy and  lowest inclusion AUC with ∆ Log-odds.
Revised: We observe that CoRTX outperforms all other baselines in two evaluations, which obtains the lowest exclusion AUC of Top-1 Accuracy and the lowest inclusion AUC of Log - odds .

Original: CoRTX is also competitive with the state-of-the-art baseline, FastSHAP, on two other remaining tasks and performssignificantly better than other gradient-based explainers since gradient-based methods are less faithful than perturbation-based methods.
Revised: CoRTX is also competitive with the state-of-the-art baseline, FastSHAP, on the other two evaluations. Besides, CoRTX performs better than non-amortized baseline methods in all four evaluations.

Original:  C ONCLUSION
Revised:  C ONCLUSIONS

Original: In this work, we propose a contrastive RTX framework, CoRTX, which significantly reduces the usage amount of explanation labels.
Revised: In this paper, we propose a real-time explanation framework, CoRTX, based on contrastive learning techniques.

Original: Specifically, CoRTX introduces a synthetic positive samples selection on contrastive learning for generating pre-trained explanation representation and finetunes on the downstream explanation tasks. 
Revised: Specifically, CoRTX contains two components, i.e., explanation encoder and explanation head, which learns the explanation representation and then provides explanation results for certain explanation tasks. CoRTX introduces an explanation-oriented data augmentation strategy for learning explanation representation. An explanation head can be further fine-tuned with a small amount of explanation labels.

Original: We also provide theoretical analysis to support theeffectiveness of CoRTX in learning explanation representation.
Revised: Theoretical analysis indicate our proposed data augmentation and contrastive scheme can effectively bound the explanation error.

Original: The experimental results on three datasets demonstrate that CoRTX works more effectively and faithfully compared with other RTX frameworks and local explanation methods.
Revised: The experimental results on three datasets demonstrate that CoRTX outperforms the state-of-the-art baselines regarding to the explanation efficacy and efficiency.

Original: As forthe future exploration, we will explore more on explanation representation and reduce the limited labels requirement into zero label exploitation
Revised: As for future directions, we consider to extend CoRTX to multiple domains to obtain general pre-trained explanation representation for downstream scenarios. It is also promising to explore healthcare applications with CoRTX framework, where the explanation labels require intensive human knowledge.

Original: A PPENDIX A P ROOF OF T HEOREM
Revised: A PPENDIX A P ROOF OF T HEOREM

Revised: We prove Theorem 1 in this section.

Original: Theorem 1 ( Compact Alignment ) .
Revised: Theorem 1 ( Compact Alignment ) .

Original: Let f ( x ) be a K  -Lipschitz  continuous functionfor given input sample x and Φ ( x ) =
Revised: Let f ( x ) be a K f -Lipschitz 4 continuous function and Φ ( x ) =

Original: [ ϕ 1 ( x ) , · · · , ϕ M ( x )] be the  importance score of each feature, where ϕ i ( x ):=
Revised: [ ϕ 1 ( x ) , · · · , ϕ M ( x )] be the feature importance scores, where ϕ i ( x ) :

Original: E
Revised: =

Original:  S⊆ U \{ i } (cid:2)f
Revised: E S⊆ U \{ i } (cid:2)

Original:  ( (cid:101) x
Revised: f ( (cid:101) x

Original: S∪{ i } ) − f ( (cid:101) x S ) (cid:3) .
Revised: S∪{ i } ) − f ( (cid:101) x S ) (cid:3) .

Original: Given a perturbed sample (cid:101) x ∈X + satisfying min 1 ≤ i ≤ M ϕi ( (cid:101) x [Equation]
Revised: Given a perturbed positive instance (cid:101) x ∈X + satisfying min 1 ≤ i ≤ M ϕ

Original: [Equation] the explanation difference || ϕ
Revised: i ( (cid:101) x ) ≥ 0 , the explanation difference ∥

Original:  ( x ) − ϕ ( (cid:101) x ) || 2 is bounded by the prediction difference | f ( x ) − f ( (cid:101) x ) | as
Revised: Φ ( x ) − Φ ( (cid:101) x ) ∥ 2 is bounded by the prediction difference | f ( x ) − f ( [Equation] x ) | as

Original: [Equation]
Revised: [Equation]

Original: where γ 

Revised: where γ

Original: =
Revised: =

Original: K || x || 2 and K  ≥ 0 is the Lipschitz constant of function f ( · ) .
Revised: K f ∥ x ∥ 2 and K f ≥ 0 is the Lipschitz constant of function f ( · ) .

Original: Proof.
Revised: Proof.

Original: In order to calculate the importance score ϕ i ( x ) :=
Revised: In order to calculate the importance score Φ i ( x ):=

Original: E S∼ U \{ i } (cid:2)
Revised: E S∼ U \{ i } (cid:2)

Original: f
Revised: f

Original: ( (cid:101) x
Revised: ( (cid:101) x

Original: S∪{ i } ) − f ( (cid:101) x S ) (cid:3) of the feature subset S , we recast the formulation under the expression of S = 1 S ∈ { 0 , 1 } M, which is given as follows:
Revised: S∪{ i } ) − f ( (cid:101) x S ) (cid:3) of the feature subset S , we recast the formulation under the expression of S = 1 S ∈ { 0 , 1 } M as follows:

Original: [Equation]
Revised: [Equation]

Revised: where S ∪ [1] i denotes not to mask feature i -th from perturbed masking S .

Original: Following Equation 9, we can now discuss explanation difference by each feature.
Revised: Following Equation 8, we can now discuss explanation difference by each feature.

Original: Without lost of generality, we first discuss the score difference of  feature i , |
Revised: Without lost of generality, we consider the explanation difference of a single feature i as follows,

Original: [Equation]
Revised: [Equation]

Original: [Equation] We then discuss the case from Equation16. We have
Revised: Following Equation

Original: [Equation]
Revised: [Equation]

Revised: the upper bound of the explanation difference can be derived by

Original: [Equation]
Revised: [Equation]

Original: Note that the difference of prediction scores | f ( x ) − f ( (cid:101) x ) | is equal to the summation of contribution score among all features, where | f ( x ) − f ( (cid:101) x ) | = | (cid:80) Mi
Revised: Note that the difference of prediction scores | f ( x ) − f ( (cid:101) x ) | is equal to the summation of contribution score among all features (i.e., | f ( x ) − f ( (cid:101) x ) | = | (cid:80) Mi

Original: =

Revised: =

Revised: δ i | ).

Revised: Since we consider the assumption min 1 ≤ i ≤ M Φ i ( (cid:101) x ) ≥ 0 , we have | (cid:80)

Original: {+Mi =
+} δ i | = (cid:80) Mi
Revised: {+Mi =
+} δ i | = (cid:80) Mi

Original: =1 | δ i | .
Revised: =1 | δ i | .

Original: In this manner, we have the upper bound of the explanation difference || ϕ
Revised: In this manner [Equation] we have the upper bound of the explanation difference ∥

Original:  ( x ) − ϕ ( (cid:101) x ) || 2 given by
Revised: Φ ( x ) − Φ ( (cid:101) x ) ∥ 2 as follows:

Original: [Equation]
Revised: [Equation]

Original: where γ 

Revised: where γ

Original: =
Revised: =

Original: K || x 
Revised: K f ∥ x ∥ 2 .

Original: ||
Revised: B P ROOF OF T HEOREM

Original: Theorem 2 ( Explanation Error Bound ) .
Revised: Theorem 2 ( Explanation Error Bound ) .

Original: Given a training set D , a testing set C , and a welltrained explainer ˆ ϕ =

Original: η ◦ g , where g denotes the contrastive explanation encoder to generate d -dimensional explanation embeddings and η represents explanation head.

Original: Assume ˆ ϕ ( · )is a K hLipschitz continuity.
Revised: Let η ( ·| θ η ) be K η -Lipschitz continuous, and ˆ Φ ( · )

Original: For all x j ∈ D in training set, if there exist E > 0 , such that the training error satisfies || ϕ
Revised: =

Original:  ( x j ) − ˆ ϕ ( x j ) || 2 ≤ E .
Revised: η ( g ( ·| θ g ) | θ η ) be the estimated explanations.

Original: Then, for any testing datapoint x k ∈ C , the testing explanation error Err( ˆ ϕ ) = || ϕ ( xk ) − ˆ ϕ ( xk ) 
Revised: If the encoder g ( ·| θ g ) is well-trained, satisfying ∥ Φ ( x ) − ˆ Φ ( x ) ∥ 2 ≤E ∈ R + , then the explanation error on testing instance x k can be bounded as:

Original: || 2 can be bounded as:

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) x
Revised: where (cid:101) x

Original: + k is the compact positive sample, h x i = g
Revised: + k is a compact positive instance, h x k = g

Original: ( x i ) ,  γ [-
-] = K ||
Revised: ( x k | θ g ) , and γ [-
-]k = K f

Original:  x k || 2 , and
Revised: ∥ x k ∥ 2 .

Original: K h ≥ 0 is the Lipschitz constant of prediction model f ( · ) .

Original: Proof.
Revised: Proof.

Original: Without loss of generality, we consider ℓ 2 norm to evaluate the distance between predicted explanation scores and ground truth explanation scores.
Revised: Without loss of generality, we consider ℓ 2 norm to evaluate the distance between predicted explanation scores and ground-truth explanation scores.

Original: For any testing datapoint x k ∈ C , we have 
Revised: For all x k ∈ C , we have ∥

Original: [Equation]
Revised: [Equation]

Original: B D ATASETS AND I MPLEMENTATION D ETAILS ABOUT E XPERIMENTS
Revised: C D ETAILS ABOUT D ATASETS

Original: Our experiments are conductedwith the following details on two tabular datasets and one image dataset.
Revised: The experiments are conducted on two tabular datasets and one image dataset.

Original: The details of the datasets are provided as follows.
Revised: The details of the datasets are provided as follows:

Original:  Census Income: A collection of human social information with 26048 samples for training and validating; and 6513 samples for testing.
Revised: • Census Income: A collection of human social information with 26048 samples for training and validating; and 6513 samples for testing.

Original: Each sample has five continuous features and eightone-hot encoded categorical features.
Revised: Each instance has five continuous features and eight categorical features.

Original:  Bankruptcy: A  dataset contains 5455 samples ofvarious companies  training  and validating  and 1364 instances for testing.
Revised: • Bankruptcy: A financial dataset contains 5455 samples of companies in the training set and validating set; and 1364 instances for the testing set.

Original: Each sample has 96 features characterizing each company and whether it went bankrupt or not.
Revised: Each sample has 96 features characterizing each company and whether it went bankrupt.

Original:  CIFAR-10:
Revised: • CIFAR-10:

Original: An image dataset with 60000 images in 10 different classes, where each image has 32×32 pixels.
Revised: An image dataset with 60000 images in 10 different classes, where each image is composed of 32×32 pixels.

Original: We follow the original dataset division for training, validating, and testing.
Revised: We follow the original dataset division on training, validating, and testing process.

Original: For the tabular datasets, we consider two common explanation tasks: feature attribution estimation and feature importance ranking. For the image datasets, the explanation results are presented by using heatmap for the case study and evaluated by including and excluding the importance pixels for perturbed image classification.
Revised: D D ETAILS ABOUT B ASELINE M ETHODS

Original: All baseline algorithms are implemented under the open-source package 2 and the hyper-parameters are all decided with the optimal model convergence.
Revised: Some of the baseline algorithms are implemented under the open-source package 5 and the hyperparameters are all decided with the optimal model convergence.

Revised: In our experiments, the RTX frameworks, such as CoRTX and FastSHAP, are typically built on the training set and evaluated on the testing dataset.

Original:  We here provide some detailed information about the baselines on tabular dataset.
Revised: Tabular Dataset: We here provide some detailed information about the baselines on tabular dataset.

Original: Supervised RTX: A supervised RTX-based MLP model trains with raw features of data instances and ground-truth explanation labels from scratch.
Revised: Supervised RTX: A supervised RTX-based MLP model trains with raw features of data instances and ground-truth explanation labels from scratch.

Revised: The layer number in Supervised RTX is set to be six for purpose of fair comparison to CoRTX.

Original: FastSHAP:
Revised: FastSHAP:

Original: A state-of-the-art RTX method which adopts an DNN model to universally learn the  approximated Shapley values (Jethani et al., 2021b).
Revised: A state-of-the-art RTX method which adopts an DNN model to universally learn the distribution of approximated Shapley label (Jethani et al., 2021).

Original: KernelSHAP (KS): The model proposes the weighted linear regressions to estimate the Shapley additive explanations from the prediction model scores (Kokhlikyan et al., 2020).
Revised: KernelSHAP (KS): The model proposes the weighted linear regressions to estimate the Shapley additive explanations from the prediction model scores (Kokhlikyan et al., 2020).

Revised: Permutation

Original: Permutation Sampling (PS): PS estimates the feature attribution based on calculating the sensitivity gap of the scores from prediction model while inputting randomly masking data features (Mitchell et al., 2021).
Revised: Sampling (PS): PS estimates the feature attribution based on calculating the sensitivity gap of the scores from prediction model while inputting randomly masking data features (Mitchell et al., 2021).

Original: To evaluate proposed explanation representation, we follow thecommon protocol setting (He et al., 2020).
Revised: To achieve the competitive performance on the baselines, we demonstrate the following settings to the

Original: First, we freeze the input features (explanation representation).
Revised: KS and PS in the tabular experiments. On Census Income dataset, the numbers of model evaluations on KS is set from 2 5 to 2 10 , and the numbers of model evaluations on PS is given from 2 4 to 2 10 . On Bankruptcy dataset, the s of model evaluations on KS is set from 2 3 to 2 11 , and numbers of model evaluations on PS are from 2 3 to 2 10 .

Original: Second, we train the  explanation head η ( · | θ η ) with limited amount of explanation labels.
Revised: For a fair comparison, CoRTX and Supervised RTX adopt the same loss function while conducting on the same explanation task. Image Dataset: We evaluate the results with two non-amortized Shapley-based estimators: DeepSHAP and KS, where the first work utilizes a linear composition rule to estimate the Shapley values on the non-linear neural networks. Moreover, gradient-based methods are known for their efficiency compared to Shapley-based estimators.

Original:  The input of η ( · | θ η ) is thepre-trained explanation representation generated from explanation encoder g ( · | θ g ) , and the output is the corresponding explanation scores among each feature.
Revised: We list some gradient-based baselines: Saliency, Integrated Gradients (IG), SmoothGrad, and GradCAM. The mentioned four approaches derive the explanation based on the gradient values of prediction models.

Original: In this work, we adopt a 3-layer MLP model as g ( · | θ g ) in two tabular datasets (Census Income and Bankruptcy) and utilize a ResNet-18 model replacing the last layer with 1-layer MLP in image dataset (CIFAR-10).
Revised: Lastly, to evaluate the efficacy of the image explainers, we adopt a state-of-the-art baseline, FastSHAP, to test the explanation generating speed and accuracy. To achieve the better performance on baselines, the evaluation times on baselines is basically decided on grid search once achieving competitive performance. E E XPERIMENT D ETAILS We conduct the experiments on tabular dataset and image datasets.

Original: Considering to the  twodifferent common explanation task settings, we exploit a 3-layer MLP model as η ( · | θ η ) with cross-entropy (CE) loss on feature attributiontask andwith mean-square-error (MSE) loss on feature importance ranking task. 
Revised: For the tabular dataset, we consider two common explanation tasks: feature attribution and importance ranking task. For the image dataset, the explanation results are presented using heatmap in the case study and evaluated by including and excluding the important pixels for perturbed image classification.

Original: Tabular Dataset: The experiments details on each tabular dataset follow the pipeline below.
Revised: E. E XPERIMENTS ON T ABULAR D ATASET

Original: We verify the feature ranking task by using CoRTX-CE and the feature attribution task by exploiting CoRTXMSE.
Revised: We verify the feature ranking task by using CoRTX-CE and the feature attribution task by exploiting CoRTX-MSE.

Revised: The experiment on each tabular dataset follows the pipeline of Target Model Training ,

Revised: Explanation Benchmarks and Explainer Implementing .

Original: As for the Prediction Model , we exploit different prediction models f ( · ) among threedifferent datasets to evaluate the model-agnostic property of CoRTX.
Revised: Each step is shown as follows. Target Model Training : We exploit different prediction models f ( · ) on three datasets to evaluate the model-agnostic property of CoRTX.

Original: AutoInt (Song et al., 2019) is adopted for  Census Income dataset and  MLP model is for  Bankruptcy dataset.
Revised: AutoInt (Song et al., 2019) is adopted for the Census Income dataset, and the MLP model is for the Bankruptcy dataset.

Original: The two prediction models are trained until the convergence and the implementation are based on the DeepCTR 3 package (Shen, 2017).
Revised: The two prediction models are trained until the convergence, and the implementation is based on the DeepCTR 6 package (Shen, 2017).

Original: Due to the task given in two datasets, a binary cross entropy loss is given as the loss function of AutoInt and MLP model for  Census Incomedataset and Bankruptcy dataset, respectively.
Revised: Due to the tasks in two datasets, a binary cross entropy loss is given as the loss function of AutoInt and MLP model for the Census Income and Bankruptcy datasets, respectively.

Original: All hyperparameters are decided by grid search throughout the classification results, including model layers, hidden units,and etc.
Revised: All hyper-parameters on target model training are decided by grid search throughout the classification results, including model layers, hidden units, etc.

Original: Considering to the Explanation Benchmarks , we use brute-force algorithm to calculate the exact Shapley value  for  Census Income dataset as the explanation annotations.
Revised: Explanation Benchmarks : A brute-force algorithm for calculating the exact Shapley value is adopted for the Census Income dataset as the explanation labels.

Original: However,  Bankruptcy dataset contains large amount of features (96 features),  it is hard to gain the exact Shapley value for  Bankruptcy dataset due to the extremely high computational cost.
Revised: However, the Bankruptcy dataset contains https://captum.aihttps://github many features (96 features), so it is hard to gain the exact Shapley value for the Bankruptcy dataset due to the extremely high computational cost.

Original: We hereby adopt the proximity of Shapley values as the explanation annotations by using Antithetical Permutation Sampling(APS)
Revised: We hereby adopt the proximity of Shapley values as the explanation labels by using Antithetical Permutation Sampling(APS)

Original: (Mitchell et al., 2021; Lomeli et al., 2019) to convergence.
Revised: (Mitchell et al., 2021; Lomeli et al., 2019) to convergence.

Original: This is because APS has shown to converge to exact Shapley values when countering high permutation times on tabular datasets.
Revised: This is because APS has shown to converge to exact Shapley values when countering high permutation times on tabular datasets.

Original: As for other detailed hyper-parameter setting, the cardinality of synthetic positive data collection is set  to 30 and 300 for Census Income and Bankruptcy dataset, respectively.
Revised: In this work, we set the model permutation times as 2048 to generate the ground-truth labels. Explainer Implementation : The explainer is implemented on explanation representation learning and fine-tuning.

Original: The temperature hyper-parameter τ is given as 0.02 for both tabular datasets.
Revised: The explanation encoder and the explanation head are designed with the model structures and are learned with the hyper-parameters in Table 2.

Original: While conducting the feature attribution task, the explanation heads η ( · | θ η ) are set under the Adam optimizer with weight decay rate from 10 − 3 to 10 − 6 .
Revised: Following the existing works (Jethani et al., 2021; Dhurandhar et al., 2018), we adjust the feature importance scores via additive efficient normalization (Ruiz et al., 1998) for the feature attribution task. E. E XPERIMENTS ON I MAGE D ATASET

Original: Every training processes of CoRTX stop until the convergence. Image Dataset: Our experiments on the image dataset are conducted under CoRTX-MSE. 
Revised: Our experiments on the image dataset are conducted under CoRTX-MSE. We focus on the feature attribution task on the image datasets, which follows the pipeline of Target Model Training ,

Original: The experiments follow the pipeline as follows.
Revised: Explanation Benchmarks and Explainer Implementing . Each step is shown as follows.

Original: Prediction Model  : We utilize ResNet-18 as the prediction models f ( · ) and train it from scratch until themodel convergence.
Revised: Target Model Training : We utilize ResNet-18 as the prediction models f ( · ) in CIFAR-10 dataset. We train the target model from scratch until the convergence.

Revised: All hyper-parameters on model training are decided by grid search throughout the classification results and follow the training setting of the target model from (Jethani et al., 2021).

Original: Explanation Benchmarks : We calculate the approximation to Shapley values as the training annotations by adopting KernelSHAP until the convergence (Covert & Lee, 2021; Jethani et al., 2021b).
Revised: Explanation Benchmarks : We calculate the approximation to Shapley values as the training labels by adopting KS until the convergence (Covert & Lee, 2021; Jethani et al., 2021).

Original: The estimated values from KernelSHAP can infinitely approachto exact Shapley values when encountering the optimal convergence (Lundberg & Lee, 2017).
Revised: The estimated values from KS can infinitely approach exact Shapley values when encountering the optimal convergence (Lundberg & Lee, 2017).

Original: We observe that the important regions on images are typically consecutive.
Revised: In this work, we set the model permutation times as 2048 to generate the ground-truth labels.

Original: FastSAHP is better to meet the property since it adopts pre-trained weight of ResNet for training a CNN-based explainer, which provides average pooling for smoothing effects. Thus, for the fair comparison in case studies, the outcomes of CoRTX are processed through the moving average for smoothing visualization.
Revised: Explainer Implementation : The explanation encoder and the explanation head are designed with the model structures and are learned with the hyper-parameters in Table 2. Following the existing work (Jethani et al., 2021), we adopt the surrogate model with the same structure as ResNet-18.

Original: The adjustment maintains a similar trend of performance in visualization showcase from original of CoRTX. In other words, the adjusted explanation is not efficacy once the original explanation are definitely inaccurate. For quantitative experiments on exclusion and inclusion tasks, we exploit the original outputs from CoRTX and evaluate the performance on Top-1 Accuracy and ∆ Log-odds.
Revised: The surrogate model is distilled from the original prediction model with masked input images to overcome the effect of out-of-distribution in masking perturbation (Frye et al., 2020). Following existing work (Jethani et al., 2021), we adjust the feature attribution scores through additive efficient normalization (Ruiz et al., 1998) for generating the feature importance scores.

Original: Following thesame experiment setting from  (Jethani et al., 2021b;a; Frye et al., 2020), we adopt the supervised surrogate model while performing the explanation tasks on CoRTX and FastSHAP.
Revised: Moreover, we follow the setting from the existing work (Jethani et al., 2021) to conduct the bootstrapping evaluation.

Original: For more hyper-parameter settings, the temperature hyper-parameter τ is given as 0.02 and an Adam optimizer is given to train with both explanation encoder g ( · | θ g ) and explanation head η ( · | θ η ) . Every single training processes of CoRTX are guarantee to stop until converge.
Revised: The total testing candidates are chosen to be 1000 and we randomly select 666 instances with 20 times of repetitions for getting the bootstrap AUCs and Log - odds . [Table]

Original: Computation Infrastructure All experiments are conducted under the following physical computing infrastructure. The details of memory footprint and throughput are given in Table 2.
Revised: For the fair comparison on testing algorithmic throughput, the experiments are conducted based on the following physical computing infrastructure in Table 3.

Original: [Table]
Revised: [Table]

Original: C R ELATED W ORK
Revised: F R ELATED W ORK

Original: Local Methods.
Revised: Non-amortized methods.

Original: The local methods separately generate individual explanations for each data sample.

Original: Existing works of local methods can be categorized into three groups.
Revised: Existing work of non-amortized methods can be categorized into three groups.

Original: The first group of methods adopts linear regressions to fit the local explanation, such as LIME (Ribeiro et al., 2016) and KernelSHAP (Lundberg & Lee, 2017).
Revised: The first group of methods adopts linear regressions to fit the non-amortized explanation, such as LIME (Ribeiro et al., 2016) and KS (Lundberg & Lee, 2017).

Original: Another group of methods adopts the preceding difference of the value function for the explanation, such as RISE (Petsiuk et al., 2018),Permutation
Revised: Another group of methods adopts the preceding difference of the value function for the explanation, such as RISE (Petsiuk et al., 2018),

Original:  Sampling (Mitchell et al., 2021) and SHEAR (Wang et al., 2022).
Revised: Permutation Sampling (Mitchell et al., 2021) and SHEAR (Wang et al., 2022).

Original: The last group estimates the gradient towards the input data for the explanation, such as the GradCAM (Selvaraju et al., 2017), Integrated Gradient (Sundararajan et al., 2017) and SmoothGrad (Smilkov et al., 2017).
Revised: The last group estimates the gradient towards the input data for the explanation, such as the GradCAM (Selvaraju et al., 2017), Integrated Gradient (Sundararajan et al., 2017) and SmoothGrad (Smilkov et al., 2017).

Original: Even though the local methods can provide faithful explanation for DNN models, these group of methods suffer from high computational complexity since each data sample requires one local explainer to yield the explanation.
Revised: Even though the non-amortized method can provide faithful explanation for DNN models, this group of methods suffers from high computational complexity since each data instance requires one non-amortized explainer to yield the explanation.

Original: Real-time Explainer (RTX).
Revised: Real-time Explainer (RTX).

Original: We here review the existing RTX framework based on DNN approaches.

Original: Unlike the local methods,  RTX framework maintains an unified explainer to generate the explanation among each data sample.
Revised: Unlike the non-amortized methods, the RTX framework maintains an unified explainer to generate the explanation among each data instance.

Original: The explanation can be generated via  single feed-forward process, which is much faster than the local methods.
Revised: This way, the explanation can be generated via a single feed-forward process of the explainer, which is much faster than the non-amortized methods.

Original: One learning strategy of  RTX framework formulates the explainer learning by given strong assumption on prior feature distribution (Chen et al., 2018;
Revised: The learning strategy of the RTX framework formulates the explainer learning by giving strong assumptions on prior feature distribution (Chen et al., 2018;

Original: Dabkowski &
Revised: Dabkowski &

Original: Gal, 2017; Kanehira & Harada, 2019).
Revised: Gal, 2017; Kanehira & Harada, 2019).

Original: One of work (Chen et al., 2018) utilizes the instance-wise feature selection by maintaining a feature masking generator via maximizing the mutual information between selected features and corresponding labels.
Revised: Existing work (Chen et al., 2018) utilizes the instance-wise feature selection by maintaining a feature masking generator via maximizing the mutual information between selected features and corresponding labels.

Original: A well-trained feature masking generator is able to provide real-time explanation under  single feed-forward process.
Revised: A well-trained feature masking generator is able to provide real-time explanation under a single feed-forward process.

Original: Another framework of RTX is to adopt the exact or approximated Shapley values as the ground-truth annotations to learn the explainers (Wang et al., 2021;
Revised: Another framework of RTX is to adopt the exact or approximated Shapley values as the ground-truth labels to learn the explainers (Wang et al., 2021;

Original: Jethani et al., 2021b; Covert et al., 2022).
Revised: Jethani et al., 2021; Covert et al., 2022).

Original: However, the exploitation of exact Shapley values suffers from extremely high computational complexity.
Revised: However, the exploitation of exact Shapley values suffers from extremely high computational complexity.

Original: To address this problem,
Revised: To address this problem,

Original: FastSHAP (Jethani et al., 2021b) proposes a Monte-Carlo-based method to learn the explainer under  RTX framework.
Revised: FastSHAP (Jethani et al., 2021) proposes a Monte-Carlo-based method to learn the explainer under the RTX framework.

Original: Specifically, FastSHAP generates the approximated Shapley values by randomly samples batches of feature masks during the training process.
Revised: Specifically, FastSHAP generates the approximated Shapley values by randomly sampling batches of feature masks during the training process.

Original: Meanwhile, it updates the explainer to minimize the mean-square error between the overall contribution scores of masked features and outputs from  DNN explainer.
Revised: Meanwhile, it updates the explainer to minimize the mean-square error between the overall contribution scores of masked features and outputs from the DNN explainer.

Original: FastSHAP enforces the explanation performance without utilizing  ground-truth Shapley value which can be demonstrated by the experiment results.
Revised: FastSHAP enforces the explanation performance without utilizing the ground-truth Shapley value, which can be supported by the experiment results.

Original: D A DDITIONAL R ESULTS ON I MAGE D ATASET
Revised: G M EASUREMENT OF Log - odds

Revised: Definition.

Revised: Log - odds indicates the confidence of the prediction given by the target model f ( x ) .

Revised: Formally, Log - odds is defined as the log-likelihood ratio of f ( x ) (Ashton, 1972) as follows,

Revised: [Equation]

Revised: [Equation]

Revised: where p is the output probability of the target model f ( x ) .

Revised: Inclusion and Exclusion AUC of Log - odds .

Revised: To evaluate the estimated feature attribution on image data, we conduct the exclusion and inclusion test according to the feature importance scores.

Revised: Once the important features are removed from the input instances, the target model f ( x ) is expected to drop drastically on Log - odds which leading to a low exclusion AUC.

Revised: On the contrary, it is expected to have a high inclusion AUC of Log - odds since the important features are orderly added back to the input instances of target model.

Revised: The remaining experimental settings follows(Jethani et al., 2021).

Revised: Inclusion and Exclusion AUC of ∆ Log - odds : Besides exploiting Log - odds on measuring the performance, we additionally test CoRTX with ∆ Log - odds on the inclusion and exclusion task.

Revised: ∆ Log - odds calculates the preceding difference between the Log - odds of original input instances and masked instances.

Revised: Different from Log - odds and Top-1 Accuracy, Log - odds has the opposite indication where higher exclusion AUC and lower inclusion AUC represent better performance.

Revised: Table 4 shows the results of the exclusion and inclusion under 1000 images.

Revised: We observe that CoRTX outperforms other non-amortized baseline methods on both inclusion task and exclusion task.

Revised: CoRTX performs better than FastSHAP on the inclusion task and is competitive with FastSHAP on the exclusion task.

Revised: [Table]

Revised: H A DDITIONAL R ESULTS ON I MAGE D ATASET

Original: We demonstrate more explanation results on CIFAR-10 generated by CoRTX compared to other baselines.
Revised: We demonstrate more explanation results on CIFAR-10 generated by CoRTX compared to other baselines.

Original: The results show that CoRTX can identify more accurate regions as the explanation results toward the significant object related to the image classes.
Revised: The results show that CoRTX can identify more accurate regions as the explanation results toward the significant object related to the image classes.

Original: [Figure]
Revised: [Figure]

Original: CoRTX
Revised: CoRTX

Original: FastSHAP
Revised: FastSHAP

Original: KernelSHAP
Revised: KernelSHAP

Original: Saliency
Revised: Saliency

Original: IG
Revised: IG

Original: Smoothgrad
Revised: Smoothgrad

Original: Gradcam
Revised: Gradcam

Original: Deepshap
Revised: Deepshap

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]