Original: F AST G ENERIC I NTERACTION D ETECTION FOR M ODEL I NTERPRETABILITY AND C OMPRESSION
Revised: F AST G ENERIC I NTERACTION D ETECTION FOR M ODEL I NTERPRETABILITY AND C OMPRESSION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The ability of discovering feature interactions in a black-box model is vital to explainable deep learning.
Revised: The ability of discovering feature interactions in a black-box model is vital to explainable deep learning.

Original: We propose a principled, global interaction detection method by casting our target as a multi-arm bandits problem and solving it swiftly with the UCB algorithm.
Revised: We propose a principled, global interaction detection method by casting our target as a multi-arm bandits problem and solving it swiftly with the UCB algorithm.

Original: This adaptive method is free of ad-hoc assumptions and among the cutting-edge methods with outstanding detection accuracy and stability.
Revised: This adaptive method is free of ad-hoc assumptions and among the cutting-edge methods with outstanding detection accuracy and stability.

Original: Based on the detection outcome, a lightweight and interpretable deep learning model (called ParaACE) is further built using the alternating conditional expectation (ACE) method.
Revised: Based on the detection outcome, a lightweight and interpretable deep learning model (called ParaACE) is further built using the alternating conditional expectation (ACE) method.

Original: Our proposed ParaACE improves the prediction performance by 26% and reduces the model size by 100+ times as compared to its Teacher model over various datasets.
Revised: Our proposed ParaACE improves the prediction performance by 26% and reduces the model size by 100+ times as compared to its Teacher model over various datasets.

Original: Furthermore, we show the great potential of our method for scientiﬁc discovery through interpreting various real datasets in the economics and smart medicine sectors.
Revised: Furthermore, we show the great potential of our method for scientiﬁc discovery through interpreting various real datasets in the economics and smart medicine sectors.

Original: The code is available at http://github.com/xxx .
Revised: The code is available at http://github.com/xxx .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Explainable machine learning is an active research ﬁeld that focuses on providing interpretable models, transparent explanation, and conﬁdent decision to practical AI systems.
Revised: Explainable machine learning is an active research ﬁeld that focuses on providing interpretable models, transparent explanations, and conﬁdent decisions to practical AI systems.

Original: Investigating feature interaction is vital to model interpretability.
Revised: Investigating feature interaction is vital to model interpretability.

Original: Interaction detection should be able to reveal which subset of features inﬂuence the output jointly , and what the corresponding nonlinear transformation is.
Revised: Interaction detection should be able to reveal which subset of features inﬂuence the output jointly , and what the corresponding nonlinear transformation is.

Original: We aim to ﬁnd the underlying interactions from data, such that we can interpret the model properly.
Revised: We aim to ﬁnd the underlying interactions from data, such that we can interpret the model properly.

Original: To go one step further, we hope that new model can be builteconomically to avoid heavy parameterization, equipped with the  detected interaction knowledge.
Revised: To go one step further, we hope that new models can be built with the aid of the detected interaction knowledge economically to avoid heavy parameterization.

Original: In this paper, we ﬁrst restrict ourselves to interaction detection.
Revised: In this paper, we ﬁrst restrict ourselves to interaction detection.

Original: A novel method is derived directly from the most acknowledged deﬁnition of feature interaction (Friedman et al., 2008).
Revised: A novel method is derived directly from the most acknowledged deﬁnition of feature interaction (Friedman et al., 2008).

Original: We further apply the obtained interaction knowledge in the designof a transparent and reﬁned neural network (NN).
Revised: We further apply the obtained interaction knowledge to design a transparent and reﬁned neural network (NN).

Original: This approach ﬁts well into the Data Science Life Cycle (Yu & Kumbier, 2019), in particular the post-hoc analysis module, which was applied for instance in Tsang et al.
Revised: This approach ﬁts nicely into the Data Science Life Cycle (Yu & Kumbier, 2019), in particular the post-hoc analysis module, which was applied, for instance in Tsang et al.

Original: (2020a) successfully for interpretable recommender system design.
Revised: (2020a), successfully for interpretable recommender system design.

Original: The main contributions of this paper include: 1) a fast and principled interaction detection method, 2) a lightweight and interpretable neural network model that can surpass its Teacher, 3) thorough theoretical analyses and performance evaluations with real datasets.
Revised: The main contributions of this paper include: 1) a fast and principled interaction detection method, 2) a lightweight and interpretable neural network model that can surpass its Teacher, 3) thorough theoretical analysis and performance evaluations with real datasets.

Original: More details are given below.
Revised: More details are given below.

Original: We propose a generic interaction detection method based on a global statistical metric, namely the expected Hessian, H ij :=
Revised: We propose a generic interaction detection method based on a global statistical metric, namely the expected Hessian, H ij :=

Original: E x (cid:104) ∂ 2 F ( x ) ∂x i ∂x j (cid:105) .
Revised: E x (cid:104) ∂ 2 F ( x ) ∂x i ∂x j (cid:105) .

Original: Notably, our method is model-agnostic and applicable to any pre-trained learning model, F ( x ) ,  for instance, a deep neural network model or a tree model.
Revised: Notably, our method is model-agnostic and applicable to any pre-trained learning model, F ( x ) , being for instance, a deep neural network model or a tree model.

Original: Besides, it is ﬂexible to use for both two-way and multi-way interaction detection.
Revised: Besides, it is ﬂexible to use for both two-way and multi-way interaction detection.

Original: To speed up the detection process, we evaluate the expected Hessian via adaptive sampling using the Upper Conﬁdence Bound (UCB) algorithm (Lai & Robbins, 1985), which can signiﬁcantly reduce the computational complexity.
Revised: To speed up the detection process, we evaluate the expected Hessian via adaptive sampling using the Upper Conﬁdence Bound (UCB) algorithm (Lai & Robbins, 1985), which can signiﬁcantly reduce the computational complexity.

Original: Besides, thorough analyses on the proposed interaction detection method are conducted.
Revised: Besides, thorough analysis of the proposed interaction detection method are conducted.

Original: With the aid of the detected interaction pairs, we further design a compressed but interpretable Student model which can surpass its Teacher by 26% in terms of data ﬁtting performance averaged over various datasets.
Revised: With the aid of the detected interaction pairs, we further design a compressed but interpretable Student model which can surpass its Teacher by 26% in terms of data ﬁtting performance averaged over various datasets.

Original: Moreover, the compressed model reduces its
Revised: Moreover, the compressed model reduces its

Original: size over 100 times compared to the baseline fully-connected, over-parameterized neural network (OverparaFC).
Revised: size over 100 times compared to the baseline fully-connected, over-parameterized neural network (OverparaFC).

Original: We demonstrate the linkages between our compressed model and the classic alternating conditional expectation (ACE) model (Breiman & Friedman, 1985).
Revised: We demonstrate the linkages between our compressed model and the classic alternating conditional expectation (ACE) model (Breiman & Friedman, 1985).

Original: We conduct large-scale performance evaluations and further explain the obtained model interpretability with some real datasets.
Revised: We conduct large-scale performance evaluations and further explain the obtained model interpretability with some real datasets.

Original: The remainder of this paper is organized as follows.
Revised: The remainder of this paper is organized as follows.

Original: Section 2 introduces all related works.
Revised: Section 2 introduces all related works.

Original: Our proposed interaction detection method is introduced in Section 3.
Revised: Our proposed interaction detection method is introduced in Section 3.

Original: By exploiting the detection outcome, a new variant of lightweight and interpretable deep learning model is introduced in Section 4.
Revised: By exploiting the detection outcome, a new variant of lightweight and interpretable deep learning model is introduced in Section 4.

Original: Experimental results are given in Section 5.
Revised: Experimental results are given in Section 5.

Original: Finally, we conclude the paper in Section 6.
Revised: Finally, we conclude the paper in Section 6.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Interaction Detection: Early works adopt pure statistics for detecting feature interactions, and representatives include ANOVA and GUIDE (Wonnacott & Wonnacott, 1990; Fisher, 1992; Loh, 2002).
Revised: Interaction Detection: Early works adopt pure statistics for detecting feature interactions, and representatives include ANOVA and GUIDE (Wonnacott & Wonnacott, 1990; Fisher, 1992; Loh, 2002).

Original: These works have motivated a plethora of new methods with the aim to enhance the detection accuracy and/or efﬁciency.
Revised: These works have motivated a plethora of new methods with the aim to enhance the detection accuracy and/or efﬁciency.

Original: The ﬁrst class of methods centered around the GA 2 M and tree models, see for instance Lou et al.
Revised: The ﬁrst class of methods centered around the GA 2 M and tree models, see for instance Lou et al.

Original: ; Sorokina et al.
Revised: ; Sorokina et al.

Original: Friedman et al.
Revised: Friedman et al.

Original: ; Lundberg et al.
Revised: ; Lundberg et al.

Original: The second class of methods were built on the so-called factorization machines (Rendle, 2010) as well as its new variants (Xiao et al., 2017; Song et al., 2019) with attention mechanism.
Revised: The second class of methods were built on the so-called factorization machines (Rendle, 2010) as well as its new variants (Xiao et al., 2017; Song et al., 2019) with attention mechanism.

Original: The third class exploits the most recent advances in deep learning, including the Neural Interaction Detection (NID) and some new variants (Tsang et al., 2018a;b; Cui et al., 2019): Persistence Interaction
Revised: The third class exploits the most recent advances in deep learning, including the Neural Interaction Detection (NID) and some new variants (Tsang et al., 2018a;b; Cui et al., 2019): Persistence Interaction

Original: Detection (PID) (Liu et al., 2020), Integrated Hessians (IH) (Janizek et al., 2020), Shapley interaction (Zhang et al., 2020; Sundararajan et al., 2020), etc.
Revised: Detection (PID) (Liu et al., 2020), Integrated Hessians (IH) (Janizek et al., 2020), Shapley interaction (Zhang et al., 2020; Sundararajan et al., 2020), etc.

Original: Although we have witnessed well improved interaction detection performance for many datasets over the decades, the above methods still lead to inconsistent detection results for some other datasets.
Revised: Although we have witnessed well improved interaction detection performance for many datasets over the decades, the above methods still lead to inconsistent detection results for some other datasets.

Original: The reasons are twofold.
Revised: The reasons are twofold.

Original: Firstly, the interaction strength is empirically deﬁned, for instance NID method computes the interaction strength via summarizing the neural network weights.
Revised: Firstly, the interaction strength is empirically deﬁned, for instance, NID method computes the interaction strength via summarizing the neural network weights.

Original: Secondly,  speciﬁc deep learning model is required, for instance, an (cid:96) 1 -regularized ReLU network is required by the latest NID and PID methods to maintain high accuracy.
Revised: Secondly, a speciﬁc deep learning model is required, for instance, an (cid:96) 1 -regularized ReLU network is required by the latest NID and PID methods to maintain high accuracy.

Original: In contrast, our proposed method is derived directly from the deﬁnition of feature interaction (Friedman et al., 2008) and  not conﬁned to any speciﬁc learning model.
Revised: In contrast, our proposed method is derived directly from the deﬁnition of feature interaction (Friedman et al., 2008) and moreover is not conﬁned to any speciﬁc learning model.

Original: Model Interpretability: There are two categories of approaches to address model interpretability, namely the transparency-based and post-hoc approaches (Do ˇ silovi ´ c et al., 2018).
Revised: Model Interpretability: There are two categories of approaches to address model interpretability, namely the transparency-based and post-hoc approaches (Do ˇ silovi ´ c et al., 2018).

Original: Transparency-based approaches require the model itself to be simple and interpretable, like linear models, decision trees, etc., such that one can directly read off the interpretations from their coefﬁcients or decision rules.
Revised: Transparency-based approaches require the model itself to be simple and interpretable, like linear models, decision trees, etc. One can directly read off the interpretations from their coefﬁcients or decision rules.

Original: But often, they are less accurate due to limited representation power.
Revised: But often, they are less accurate due to limited representation power.

Original: In contrast, post-hoc approaches extract useful information from a pre-trained model, which is often complex and hard to interpret.
Revised: In contrast, post-hoc approaches extract useful information from a pre-trained model, which is often complex and hard to interpret.

Original: Well-known methods such as LIME (Ribeiro et al., 2016) and SHAP (Lundberg & Lee, 2017) fall in this category, but they did not take feature interaction into account.
Revised: Well-known methods such as LIME (Ribeiro et al., 2016) and SHAP (Lundberg & Lee, 2017) fall in this category, but they did not take feature interaction into account.

Original: Recently proposed symbolic metamodel (Alaa & van der Schaar, 2019) captures nonlinear interactions by approximating the black box model with explicit symbolic expressions.
Revised: Recently proposed symbolic metamodel (Alaa & van der Schaar, 2019) captures nonlinear interactions by approximating the black-box model with explicit symbolic expressions.

Original: There are also some other post-hoc approaches based on sensitivity analysis (Cortez & Embrechts, 2013), which return a quantiﬁcation of feature importance ν i =
Revised: There are also some other post-hoc approaches based on sensitivity analysis (Cortez & Embrechts, 2013), which return a quantiﬁcation of feature importance ν i =

Original: E [( ∂F ( x ) ∂x i ) 2 ] (Kucherenko et al., 2009) and interactions ν ij = E
Revised: E [( ∂F ( x ) ∂x i ) 2 ] (Kucherenko et al., 2009) and interactions ν ij = E

Original: [( ∂ 2 F ( x ) ∂x i ∂x j ) 2 ] (Roustant et al., 2014) by input perturbation.
Revised: [ | ∂ 2 F ( x ) ∂x i ∂x j | 2 ] (Roustant et al., 2014) by input perturbation.

Original: Our work aims to combine the strengths of the two categories.
Revised: Our work aims to combine the strengths of the two categories.

Original: Concretely, we ﬁrst extract the interaction knowledge by a post-hoc method, and then build a transparent and interpretable learning model as illustrated in Figure 3.
Revised: Concretely, we ﬁrst extract the interaction knowledge by a post-hoc method, and then build a transparent and interpretable learning model as illustrated in Figure 3.

Original: Model Compression and Knowledge Distillation (KD):
Revised: Model Compression and Knowledge Distillation (KD):

Original: Model compression (Bucilua et al., 2006) aims to learn a small compressed model (Student) from a large complex model (Teacher) with augmented training data produced by the Teacher.
Revised: Model compression (Bucilua et al., 2006) aims to learn a small compressed model (Student) from a large complex model (Teacher) with augmented training data produced by the Teacher.

Original: Compared to the Teacher, the Student can make similar or even better predictions.
Revised: Compared to the Teacher, the Student can make similar or even better predictions.

Original: Knowledge distillation (Hinton et al., 2015) mainly deals with multi-class classiﬁcation problems and extracts “valuable information that deﬁnes a rich similarity structure over the data”.
Revised: Knowledge distillation (Hinton et al., 2015) mainly deals with multi-class classiﬁcation problems and extracts “valuable information that deﬁnes a rich similarity structure over the data”.

Original: Our work introduces a novel viewpoint of knowledge (the interacted relationships), and targetson a lightweight but more accurate Student model.
Revised: Our work introduces a novel viewpoint of knowledge (the interacted relationships) and targets a lightweight but more accurate Student model.

Original:  P ROPOSED I NTERACTION D ETECTION M ETHOD
Revised:  P ROPOSED I NTERACTION D ETECTION M ETHOD

Original: Before diving into in-depth discussions of interaction detection, we need to formally deﬁne what feature interaction is.
Revised: Before diving into in-depth discussions of interaction detection, we need to formally deﬁne what feature interaction is.

Original: The textbook deﬁnition according to Friedman et al.
Revised: The textbook deﬁnition according to Friedman et al.

Original: (2008) is given below.
Revised: (2008) is given below.

Original: Deﬁnition 3.1 (Friedman & Popescu 2008) .
Revised: Deﬁnition 3.1 (Friedman & Popescu 2008) .

Original: A function F : R p
Revised: A function F : R p

Original: → R is said to exhibit an interaction between two of its variables x i and x j if the difference in the value of F ( x ) as a result of changing the value of x i depends on the value of x j .
Revised: → R is said to exhibit an interaction between two of its variables x i and x j if the difference in the value of F ( x ) as a result of changing the value of x i depends on the value of x j .

Original: Equivalently, if E x (cid:104)  ∂ 2 F ( x ) ∂x i ∂x j (cid:105) 2  > 0 , namely the partial derivative w.r.t x j turns out to be dependent on x i , then we say x i and x j are interacted.
Revised: Equivalently, if E x (cid:104) | ∂ 2 F ( x ) ∂x i ∂x j | 2 (cid:105) > 0 , namely the partial derivative w.r.t x j turns out to be dependent on x i , then we say x i and x j are interacted.

Original: Otherwise, x i and x j have no interaction, if F ( x ) can be expressed as the sum of two functions f \ i and f \ j (Sorokina et al., 2008), namely,
Revised: Otherwise, x i and x j have no interaction, if F ( x ) can be expressed as the sum of two functions f \ i and f \ j (Sorokina et al., 2008), namely,

Original: [Equation]
Revised: F ( x ) =

Original: [Equation] [Equation]
Revised: f \ i ( x 1 , . . . , x i − 1 , x i +1 , . . . , x p ) + f \ j ( x 1 , . . .

Original: [Equation] [Equation]
Revised: , x j − 1 , x j +1 , . . .

Original:  x p ) , where f \ i ( j ) is irrespective to x i ( j ) .
Revised: , x p ) , where f \ i ( j ) is irrespective to x i ( j ) .

Original: Similarly, higher-order (multi-way) interaction can be deﬁned.
Revised: Similarly, higher-order (multi-way) interaction can be deﬁned.

Original: In this paper, we mainly focus on pairwise interaction, while multi-way interaction is only brieﬂy discussed due to space limitations.
Revised: In this paper, we mainly focus on pairwise interaction, while multi-way interaction is only brieﬂy discussed due to space limitations.

Original: 3. I NTERACTION S TRENGTH M EASURE
Revised: 3. I NTERACTION S TRENGTH M EASURE

Original: Motivated by the above deﬁnition of interaction, it is natural to take advantage of the Hessian matrix  ∇ 2 xx F ( x ) .
Revised: Motivated by the above deﬁnition of interaction, it is natural to take advantage of the Hessian matrix H := ∇ 2 xx F ( x ) .

Original: The magnitude of its entry | ∂ 2 F ( x ) ∂x i ∂x j | contains rich information about the local curvature at a data point x .
Revised: The magnitude of its entry | ∂ 2 F ( x ) ∂x i ∂x j | contains rich information about the local curvature at a data point x .

Original: Note that F ( x ) is a regression function here, not a loss function.
Revised: Note that F ( x ) is a regression function here, not a loss function.

Original: This idea was originally considered in the economics community (Ai & Norton, 2003) and rediscovered for sensitivity analysis in Roustant et al.
Revised: This idea was originally considered in the economics community (Ai & Norton, 2003) and rediscovered for sensitivity analysis in Roustant et al.

Original: The goodness of F ( x ) as an approximator can essentially inﬂuence the interaction detection performance, see our Theorem M.1 in the supplement.
Revised: The goodness of F ( x ) as an approximator can essentially inﬂuence the interaction detection performance, see our Theorem M.2 in the supplement.

Original: We deﬁne g ( x , i, j ) := | ∂ 2 F ( x ) ∂x i ∂x j | 2 to measure the local interaction strength of the i -th and j -th features at point x .
Revised: We deﬁne g ( x , i, j ) := | ∂ 2 F ( x ) ∂x i ∂x j | 2 to measure the local interaction strength of the i -th and j -th features at point x .

Original: We then use f ( i, j ) :=
Revised: We then use f ( i, j ) :=

Original: E x
Revised: E x

Original: [ g ( x , i, j )] =
Revised: [ g ( x , i, j )] =

Original: E x (cid:104) | ∂ 2 F ( x ) ∂x
Revised: E x (cid:104) | ∂ 2 F ( x ) ∂x

Original: i ∂x j | 2 (cid:105) as a measure of the global interaction strength .
Revised: i ∂x j | 2 (cid:105) as a measure of the global interaction strength .

Original: If f ( i, j ) ≈ 0 , we say feature x i and x j have weak interaction; otherwise, if f ( i, j ) is signiﬁcantly larger than zero, then x i and x j have strong interaction.
Revised: If f ( i, j ) ≈ 0 , we say feature x i and x j have weak interaction; otherwise, if f ( i, j ) is signiﬁcantly larger than zero, then x i and x j have strong interaction.

Original: In this paper, we focus on the global interaction.
Revised: In this paper, we focus on the global interaction.

Original: 3. I NTERACTION S TRENGTH E VALUATION
Revised: 3. I NTERACTION S TRENGTH E VALUATION

Original: The above deﬁned global interaction strength f ( i, j ) =
Revised: The above deﬁned global interaction strength f ( i, j ) =

Original: E x (cid:104) | ∂ 2 F ( x ) ∂x
Revised: E x (cid:104) | ∂ 2 F ( x ) ∂x

Original: i ∂x j | 2 (cid:105) is mostly unavailable due to the unknown input distribution x ∼ P ( x ) .
Revised: i ∂x j | 2 (cid:105) is mostly unavailable due to the unknown input distribution x ∼ P ( x ) .

Original: The Monte Carlo method can be used to approximate the real interaction strength (Roustant et al., 2014) by computing the sample mean over the training data.
Revised: The Monte Carlo method can be used to approximate it by computing the sample mean over the training data(Roustant et al., 2014).

Original: Analytical Evaluation: The Hessian matrix ∇ 2 xx F ( x ) for neural networks at a certain data point can be calculated analytically and efﬁciently, by using the automatic differentiation (Paszke et al., 2017).
Revised: Analytical Evaluation: The Hessian matrix ∇ 2 xx F ( x ) for neural networks at a certain data point can be calculated analytically and efﬁciently, by using the automatic differentiation (Paszke et al., 2017).

Original: However, using this analytical solution is problematic for some learning models, such as the ReLU network,
Revised: However, using this analytical solution is problematic for some learning models, such as the ReLU network,

Original: Random Forest (RF), etc., see supplement A.
Revised: Random Forest (RF), etc., see supplement A.

Original: For example, the landscape of a ReLU network is piece-wise linear as shown in
Revised: For example, the landscape of a ReLU network is piece-wise linear as shown in

Original: Figure 1, thus the exact Hessian is a zero matrix at almost every point.
Revised: Figure 1, thus the exact Hessian is a zero matrix at almost every point.

Original: So, we turn to the following numerical evaluation for broader horizons.
Revised: So, we turn to the following numerical evaluation for broader horizons.

Original: Numerical Evaluation: Finite difference method is a common way to approximate the Hessian on a given data point (Campolongo & Braddock, 1999), i.e.,
Revised: Numerical Evaluation: Finite difference method is a common way to approximate the Hessian on a given data point (Campolongo & Braddock, 1999), i.e.,

Original: [Equation]
Revised: [Equation]

Original: where e i is a one-hot vector with the i -th element being equal to one, and zeros otherwise.
Revised: where e i is a one-hot vector with the i -th element being equal to one and the rest of elements being zeros.

Revised: We also note that if the computation of gradient ∂F∂ x is cheap (e.g., F is a neural network), then

Revised: H ij can be approximated as 12 h (cid:104) ∂F ( x +

Revised: [Equation]

Revised: j h

Revised: [Equation]

Revised: ∂x

Revised: [Equation]

Revised: to reduce computation, which is similar to the feature interaction score deﬁned in Greenside et al.

Original: The choice of perturbation size h i or h j (abbr.
Revised: The choice of perturbation size h i or h j (abbr.

Original: h i ( j ) ) is critical.
Revised: h i ( j ) ) is critical.

Original: Generally, we do not want h
Revised: Generally, we do not want h

Original: i ( j ) to be too small (incurring round-off error) or too large (incurring truncation error) to get a good overall approximation of the derivative (Jerrell, 1997; Baydin et al., 2018).
Revised: i ( j ) to be too small (incurring round-off error) or too large (incurring truncation error) to get a good overall approximation of the derivative (Jerrell, 1997; Baydin et al., 2018).

Original: For our problem, we particularly do not want h
Revised: For our problem, we particularly do not want h

Original: i ( j ) to be too small so that the four evaluated points (shown in the numerator of Equation 1) lie on the same hyperplane (e.g. region A in Figure 1), which makes the quantity in Equation 1 always zero.
Revised: i ( j ) to be too small so that the four evaluated points (shown in the numerator of Equation 1) lie on the same hyperplane (e.g., region A in Figure 1), which makes the quantity in Equation 1 always zero.

Original: The following theorem reveals that choosing a sufﬁciently small h i ( j ) is not necessary.
Revised: The following theorem reveals that choosing a sufﬁciently small h i ( j ) is not necessary.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: For any x and y , function F shows no interaction between x and y , i.e. it can be decomposed as F ( x, y )
Revised: For any x and y , function F shows no interaction between them, i.e., it can be decomposed as F ( x, y )

Original: = a ( x ) + b ( y ) if and only if , for any h, k > 0 , F ( x + h, y + k ) − F ( x + h, y − k ) − F ( x − h, y + k ) + F ( x − h, y − k )
Revised: = a ( x ) + b ( y ) if and only if , for any h, k > 0 , F ( x + h, y + k ) − F ( x + h, y − k ) − F ( x − h, y + k ) + F ( x − h, y − k )

Original: = 0 .
Revised: = 0 .

Original: The magnitude of the numerator in Equation 1 tells us whether F is locally separable for variables x i and x j at point x , see our proof in supplement B.
Revised: The magnitude of the numerator in Equation 1 tells us whether F is locally separable for variables x i and x j at point x , see our proof in supplement B.

Original: The main issue of the ﬁnite difference method lies in the computational complexity for approximating the global interaction strength f
Revised: The main issue of the ﬁnite difference method lies in the computational complexity for approximating the global interaction strength f

Original: ( i, j ) , especially when the function evaluation itself is expensive.
Revised: ( i, j ) , especially when the function evaluation itself is expensive.

Original: Using all the training samples for ﬁnding just a few strongest interaction pairs can be a total waste of computation resource.
Revised: Using all the training samples for ﬁnding just a few strongest interaction pairs can be a total waste of computation resources.

Original: In general, for a dataset with N samples in p -dimensional feature spaces, a total number of 4 Np ( p − 1) / 2 arithmetic evaluations of the surrogate regression function F are needed.
Revised: In general, for a dataset with N samples in p -dimensional feature spaces, a total number of 4 Np ( p − 1) / 2 arithmetic evaluations of the surrogate regression function F are needed.

Original: If the Hessian happens to be sparse, that is, there are only a few interactions existing in the ground truth function, massive evaluations on those feature pairs with zero interaction strength should be avoided.
Revised: If the Hessian happens to be sparse, that is, there are only a few interactions existing in the ground truth function, massive evaluations on those feature pairs with zero interaction strength should be avoided.

Original: 3. I DENTIFICATION OF THE k -S TRONGEST P AIRWISE I NTERACTIONS
Revised: 3. I DENTIFICATION OF THE k -S TRONGEST P AIRWISE I NTERACTIONS

Original: In practice, we do not have to obtain the interaction strengths for all interaction pairs. This is because many of them are simply too weak to have an impact on the output, and the top k strongest pairwise interactions are well sufﬁcient for data modeling and prediction.
Revised: In practice, we do not have to obtain the interaction strengths for all interaction pairs, because many of them are simply too weak to have an impact on the output, and the top k strongest pairwise interactions are well sufﬁcient for data modeling and prediction.

Original: Finding the top k strongest pairwise interactionsperfectly ﬁts  into the best k -arms identiﬁcation problem in the context of multi-armed bandits.
Revised: Finding the top k strongest pairwise interactions ﬁts perfectly into the best k -arms identiﬁcation problem in the context of multi-armed bandits.

Original: Inspired by the recent work (Bagaria et al., 2018b), we can treat each entry of the Hessian H ij as an arm A r (see Figure 2).
Revised: Inspired by the recent work (Bagaria et al., 2018b), we can treat each entry of the Hessian H ij as an arm A r (see Figure 2).

Original: Since Hessian is a symmetric matrix, we only need to consider n = p ( p − 1) / 2 arms in the set {A r : r ∈ [ n ] } .
Revised: Since Hessian is a symmetric matrix, we only need to consider n = p ( p − 1) / 2 arms in the set {A r : r ∈ [ n ] } .

Original: For example, if we choose to pull the arm for the i -th row and j -th column of the Hessian, we will evaluate the local interaction strength g ( x ξ , i, j ) on a randomly selected training data point x ξ .
Revised: For example, if we choose to pull the arm for the i -th row and j -th column of the Hessian, we will evaluate the local interaction strength g ( x ξ , i, j ) on a randomly selected training data point x ξ .

Original: The strength g ( x ξ , i, j ) can be regarded as a random reward, where x ξ is uniformly drawn from the training data { x 1 , . . .
Revised: The strength g ( x ξ , i, j ) can be regarded as a random reward, where x ξ is uniformly drawn from the training data { x 1 , . . .

Original: , x N } .
Revised: , x N } .

Original: For ease of notation, we let f ( r ; ξ ) :
Revised: For ease of notation, we let f ( r ; ξ ) :

Original: = g ( x ξ , i, j ) as the random reward of the arm r , and write µ r :=
Revised: = g ( x ξ , i, j ) as the random reward of the arm r , and deﬁne µ r :=

Original: E ξ [ f ( r ; ξ )] as the true mean value of the arm r ’s reward.
Revised: E ξ [ f ( r ; ξ )] as the true mean value.

Original: Our goal is to ﬁnd the best k arms with the highest mean reward.
Revised: Our goal is to ﬁnd the best k arms with the highest mean reward.

Original: This reformulation is reasonable since the following three essential assumptions for multi-armed bandits (Slivkins et al., 2019) are satisﬁed.
Revised: This reformulation is reasonable since the following three essential assumptions for multi-armed bandits (Slivkins et al., 2019) are satisﬁed.

Original: a. Only the reward will be observed after each pull; b. The reward for each arm is drawn independently and identically from its reward distribution;
Revised: a. Only the reward will be observed after each pull; b. The reward for each arm is drawn independently and identically from its reward distribution;

Original: c. The reward for each round is bounded.
Revised: c. The reward for each round is bounded.

Original: Since we only consider functions F ( x ) deﬁned on a compact set, if F is a continuous function and twice differentiable 1 , then both its ﬁrst- and second-order derivatives are bounded accordingly.
Revised: Since we only consider functions F ( x ) deﬁned on a compact set, if F is a continuous function and twice differentiable, its ﬁrst- and second-order derivatives are bounded accordingly.

Original: 3. D ETECTING THE k -S TRONGEST P AIRWISE I NTERACTIONS WITH UCB A LGORITHM
Revised: 3. D ETECTING THE k -S TRONGEST P AIRWISE I NTERACTIONS WITH UCB A LGORITHM

Original: In this section, we brieﬂy introduce the UCB algorithm (Lai & Robbins, 1985) for ﬁnding the k -strongest interactions.
Revised: In this section, we brieﬂy introduce the UCB algorithm (Lai & Robbins, 1985) for ﬁnding the k -strongest interactions.

Original: Here, we design a sequence of estimators { ˆ f (cid:96) ( r ) } for the mean reward of the arm A r and construct conﬁdence interval C ( (cid:96) ) .
Revised: Here, we design a sequence of estimators { ˆ f (cid:96) ( r ) } for the mean reward of the arm A r and construct conﬁdence interval C ( (cid:96) ) .

Original: We let ˆ f (cid:96) ( r ) denote the estimator after (cid:96) pulls of the arm A r .
Revised: We let ˆ f (cid:96) ( r ) denote the estimator after (cid:96) pulls of the arm A r .

Original: Several assumptions need to be made before we give out the main outcome.
Revised: Several assumptions need to be made before we give out the primary outcome.

Original: 1: For each arm {A r : r ∈ [ n ] } , compute an initial (1 − δ ) conﬁdence interval by running log( n ) steps of the estimator to obtain: [ˆ µ r (1) − C r (1) , ˆ µ r (1) + C r (1)] 2: B = {} // Set of k best arms 3: S =
Revised: 1: For each arm {A r : r ∈ [ n ] } , compute an initial (1 − δ ) conﬁdence interval by running log( n ) steps of the estimator to obtain: [ˆ µ r (1) − C r (1) , ˆ µ r (1) + C r (1)] 2: B = {} // Set of k best arms 3: S =

Original: {A r : r ∈ [ n ] } // Set of arms under consideration 4: while TRUE do 5:
Revised: {A r : r ∈ [ n ] } // Set of arms under consideration 4: while TRUE do 5:

Original: At iteration t , pick arm A r that maximizes ˆ µ r ( t − 1)
Revised: At iteration t , pick arm A r that maximizes ˆ µ r ( t − 1)

Original: + C r
Revised: + C r

Original: ( t − 1) among arms in S .
Revised: ( t − 1) among arms in S .

Original: 6: if arm A r is evaluated less than m times then 7:
Revised: 6: if arm A r is evaluated less than m times then 7:

Original: Reﬁne the conﬁdence interval and mean estimate of the arm A r by updating the estimator one more time.
Revised: Reﬁne the conﬁdence interval and mean estimate of the arm A r by updating the estimator one more time.

Original: 8: else 9:
Revised: 8: else 9:

Original: Set ˆ µ r ( t ) to be the current mean of the arm and set C r ( t ) = 0 .
Revised: Set ˆ µ r ( t ) to be the current mean of the arm and set C r ( t ) = 0 .

Original: 10: end if 11: if Arm A r is such that ∀
Revised: 10: end if 11: if Arm A r is such that ∀

Original: i (cid:54) =
Revised: i (cid:54) =

Original: r , ˆ µ i ( t ) + C i ( t ) <
Revised: r , ˆ µ i ( t ) + C i ( t ) <

Original: ˆ µ r ( t ) − C r ( t ) then 12:
Revised: ˆ µ r ( t ) − C r ( t ) then 12:

Original: Add A r to B 13:
Revised: Add A r to B 13:

Original: Remove A r from the set of arms under consideration, that is S = S − {A j } .
Revised: Remove A r from the set of arms under consideration, that is S = S − {A r } .

Original: 14: if  |B| =
Revised: 14: if the size of B , |B| =

Original: k  then 15: return B 16: end if 17: end if 18: end while
Revised: k , then 15: return B 16: end if 17: end if 18: end while

Original: Assumption 1.
Revised: Assumption 1.

Original: Finite m evaluations of one arm are sufﬁcient to obtain an accurate reward.
Revised: Finite m evaluations of one arm are sufﬁcient to obtain an accurate reward.

Original: Assumption 2.
Revised: Assumption 2.

Original: Estimators ˆ f (cid:96) ( r ) are σ r -subgaussian.
Revised: Estimators ˆ f (cid:96) ( r ) are σ r -subgaussian.

Original: Assumption 2 is satisﬁed because the reward is bounded (Hoeffding’s lemma, 1963).
Revised: Assumption 2 is satisﬁed because the reward is bounded (Hoeffding’s lemma, 1963).

Original: Here, σ r s are pre-deﬁned parameters.
Revised: Here, σ r s are pre-deﬁned parameters.

Original: In the following analysis, we will choose σ = max r { σ r } , such that all ˆ f (cid:96) ( r ) are σ -subgaussian.
Revised: In the following analysis, we will choose σ = max r { σ r } , such that all ˆ f (cid:96) ( r ) are σ -subgaussian.

Original: Let ˆ f (cid:96) ( r ) :
Revised: Let ˆ f (cid:96) ( r ) :

Original: = 1 (cid:96) (cid:80) (cid:96)i =1 f ( r ; ξ i ) , and we construct the 1 − δ conﬁdence intervals (see supplement C) of ˆ f (cid:96) ( r ) as,
Revised: = 1 (cid:96) (cid:80) (cid:96)i =1 f ( r ; ξ i ) , and we construct the 1 − δ conﬁdence intervals (see supplement C) of ˆ f (cid:96) ( r ) as,

Original: [Equation]
Revised: [Equation]

Original: where µ r denotes the true expectation.
Revised: where µ r denotes the true expectation.

Original: With the increasing number of pulls of one arm, the uncertainty of the reward for that arm will decrease.
Revised: Increasing number of pulls of one arm, the uncertainty of the reward for that arm will decrease.

Original: When the number of pulls reaches m , we can remove the uncertainty of the arm and set C ( (cid:96) )
Revised: When the number of pulls reaches m , we can remove the uncertainty of the arm and set C ( (cid:96) )

Original: = 0 .
Revised: = 0 .

Original: Let (cid:96) r ( t ) count the number of pulls of arm A r till iteration t .
Revised: Let (cid:96) r ( t ) count the number of pulls of arm A r till iteration t .

Original: To simplify the notation, we let ˆ µ r ( t ) =
Revised: To simplify the notations, we let ˆ µ r ( t ) =

Original: ˆ f (cid:96) r ( t ) ( r ) and C r ( t ) =
Revised: ˆ f (cid:96) r ( t ) ( r ) and C r ( t ) =

Original: C ( (cid:96) r
Revised: C ( (cid:96) r

Original: ( t )) , which are the sample means and conﬁdence intervals at iteration t .
Revised: ( t )) be the sample means and conﬁdence intervals at iteration t .

Original: Now, we formally introduce the complete procedure in
Revised: Now, we formally introduce the complete procedure in

Original: Algorithm 1.
Revised: Algorithm 1.

Original: The Algorithm 1 is initialized by evaluating all arms O (log( n )) times, then at each subsequent iteration, it picks the arm with the highest UCB, i.e., max j { ˆ µ j ( t )+ C j ( t ) } , to evaluate (see Figure 2).
Revised: The Algorithm 1 is initialized by evaluating all arms O (log( n )) times, then at each subsequent iteration, it picks the arm with the highest UCB, i.e., max j { ˆ µ j ( t )+ C j ( t ) } , to evaluate (see Figure 2).

Original: If one arm has been evaluated over m times, we willﬁx the current mean as its accurate result and set the uncertainty to zero.
Revised: If one arm has been evaluated for m times, we will its true mean value and set the uncertainty to zero.

Original: At the end of each iteration, if there exists one arm whose Lower Conﬁdence Bound (LCB) is higher than all the other’s UCB, we will put it into the set of k best arms.
Revised: At the end of each iteration, if there exists one arm whose Lower Conﬁdence Bound (LCB) is higher than all the other’s UCB, we will put it into the set of k best arms.

Original: In the following, we give a theoretical analysis on the complexity of this algorithm.
Revised: In the following, we give a theoretical analysis on the complexity of this algorithm.

Original: We have to further deﬁne ∆ ( k ) i := max(0 , µ i ∗ k − µ i ) , where i ∗ k is the index for the k -th best arm and ∆ ( k ) i indicates the gap of true values between the top- k arms and the rest.
Revised: We have to further deﬁne ∆ ( k ) i := max(0 , µ i ∗ k − µ i ) , where i ∗ k is the index for the k -th best arm and ∆ ( k ) i indicates the gap of true values between the top- k arms and the rest.

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: With probability 1 − Θ( 1 n 2 ) , given maximal pulling times m for each arm, Algorithmreturns the k-strongest interaction pairs in O (cid:18)(cid:80) ni
Revised: With probability 1 − Θ( 1 n 2 ) , given maximal pulling times m for each arm, Algorithmreturns the k-strongest interaction pairs in O (cid:18)(cid:80) ni

Original: =1 log( n ) (cid:18) σ 2 log( nm ) (∆ ( k ) i )
Revised: =1 log( n ) [Equation]

Original: ∧ m (cid:19)(cid:19) time, where ( · ∧ · ) is short for min( · , · ) .
Revised: time, where ( · ∧ · ) is short for min( · , · ) .

Original: This theorem gives the computational complexity for ﬁnding out k -best arms with high probability.

Original: For arm A i , one needs to pull it O (( σ 2 log( nm ) (∆ ( k ) i ) 2 ) ∧ m ) times.

Original: Before each pull, O (log( n )) complexity is needed to maintain a priority queue.

Original: Remark 3.1.
Revised: Remark 3.1.

Original: When k (cid:28) n , the running time of Algorithm 1 for ﬁnding the top k interactions is
Revised: When k (cid:28) n , the complexity of Algorithm 1 for ﬁnding the top k interactions is

Original: O ( n log( mn ) log( n ) + km log( n )) under some natural assumptions on the distribution of ∆ i , being superior to therunning time of the naive algorithm (every arm is pulled m times) which is O ( nm ) .
Revised: O ( n log( mn ) log( n ) + km log( n )) under some natural assumptions on the distribution of ∆ i , being superior to the naive algorithm (every arm is pulled m times) with complexity O ( nm ) .

Original: Remark 3.2.
Revised: Remark 3.2.

Original: For k = 1 , namely the strongest interaction pair identiﬁcation, it requires at most (cid:80) ni
Revised: For k = 1 , namely the strongest interaction pair identiﬁcation, it requires at most (cid:80) ni

Original: =1 (( 8 σ 2 (∆ (1) i ) 2 log( n 3 m ))
Revised: =1 (( 8 σ 2 (∆ (1) i ) 2 log( n 3 m ))

Original: ∧ m ) pulls.
Revised: ∧ m ) pulls.

Original: We leave the complete proof in supplement C.
Revised: We leave the complete proof in supplement C.

Original: 3. E XTENSION TO M ULTI -W AY I NTERACTION
Revised: 3. E XTENSION TO M ULTI -W AY I NTERACTION

Original: Similarly, the third-order derivatives ∂ 3 F ∂x i ∂x j ∂x k , ∀ ( i, j, k ) , and even the higher-order ones can be approximated by the ﬁnite difference method as well, but the computational complexity will increase geometrically.
Revised: Similarly, the third-order derivatives [Equation] ∂x k , ∀ ( i, j, k ) , and even the higher-order ones can be approximated by the ﬁnite difference method as well, but the computational complexity will increase geometrically.

Original: For a remedy, one can detect higher-order interactions from the obtained pairwise ones.
Revised: For a remedy, one can detect higher-order interactions from the obtained pairwise ones.

Original: For instance, based on the k -strongest pairwise interaction strengths, we can further construct an undirected graph G , in which each node represents a feature.
Revised: For instance, based on the k -strongest pairwise interaction strengths, we can further construct an undirected graph G , in which each node represents a feature.

Original: We develop Algorithm 2 to detect the multi-way interactions through checking all the cliques in the graph G ,  more details are given in supplement G. Despite the computation complexity, our proposed algorithm is more advantageous to use because the associated learning model is agnostic and the form of interaction is adaptive as compared with some existing methods, such as Min et al.
Revised: We develop Algorithm 2 to detect the multi-way interactions through checking all the cliques in the graph G , and more details are given in supplement G. Despite the computation complexity, our proposed algorithm is more advantageous to use because the associated learning model is agnostic and the form of interaction is adaptive as compared with some existing methods, such as Min et al.

Original: (2014), which relies on a logistic regression model and multiplicative interactions of a particular form.
Revised: (2014), which relies on a logistic regression model and multiplicative interactions of a particular form.

Original:  M ODEL C OMPRESSION WITH D ETECTED I NTERACTIONS
Revised:  M ODEL C OMPRESSION WITH D ETECTED I NTERACTIONS

Original: The detected interactions can be used for designing structured and more transparent models.
Revised: The detected interactions can be used for designing structured and more transparent models.

Original: Similar idea has been mentioned in Tsang et al.
Revised: A similar idea was mentioned in Tsang et al.

Original: (2020b).
Revised: 2020b

Original: But we propose a new architecture called Parametric
Revised: But we propose a new architecture called Parametric

Original: ACE (ParaACE), inspired by several seminal works on Generalized Additive Model (GAM) (Hastie, 2017) and (nonparametric)
Revised: ACE (ParaACE), inspired by several seminal works on Generalized Additive Model (GAM) (Hastie, 2017) and (nonparametric)

Original: Alternating Conditional Expectation (ACE) method (Breiman & Friedman, 1985).
Revised: Alternating Conditional Expectation (ACE) method (Breiman & Friedman, 1985).

Original: The original ACE method is based on the following model,
Revised: The original ACE method is based on the following model,

Original: [Equation]
Revised: [Equation]

Original: and solves for the optimal transformation functions { θ, φ 1 , · · · , φ p } nonparametrically in the Hilbert space.
Revised: and solves for the optimal transformation functions { θ, φ 1 , · · · , φ p } nonparametrically in the Hilbert space.

Original: However, it only considersthe transformation of each single feature and is time consuming for large feature dimension p .
Revised: However, it only considers transformation of each single feature and is time consuming for large feature dimension p .

Original: We build the ParaACE model with the detected interactions as
Revised: We build the ParaACE model with the detected interactions as

Original: [Equation]
Revised: [Equation]

Original: where {I i } Ri =1 is a set of indices for the detected interaction pairs, s i and r i are the transformation functions for a single feature and interacting features with the corresponding parameters { θ s i } and { θ r i } , β is a weight vector for the output of all the transformation functions.
Revised: where {I i } Ri =1 is a set of indices for the detected interaction pairs, s i and r i are the transformation functions for a single feature and interacting features with the corresponding parameters { θ s i } and { θ r i } , β is a weight vector for the output of all the transformation functions.

Original: [Figure]
Revised: [Figure]

Original: The new architecture is shown in Figure 3 with two meta layers: the optimal feature transformation layer (Equation 5, boxed in red and green) and an optional ﬁx-up layer (boxed in yellow).
Revised: The new architecture is shown in Figure 3 with two meta layers: the optimal feature transformation layer (Equation 5, boxed in red and green) and an optional ﬁx-up layer (boxed in yellow).

Original: The optimal transformation layer gives the transformation functions of both the main effects and interacting features explicitly.
Revised: The optimal transformation layer gives the transformation functions of both the main effects and interacting features explicitly.

Original: The ﬁx-up layer can be understood as the inverse function for the optimal transformation of the output, θ ( Y ) , in the original ACE (Equation 4).
Revised: The ﬁx-up layer can be understood as the inverse function for the optimal transformation of the output, θ ( Y ) , in the original ACE (Equation 4).

Original: Another functionality of the ﬁx-up layer is to alleviate the adversarial impacts of the wrongly detected interactions on the output.
Revised: Another functionality of the ﬁx-up layer is to alleviate the negative impacts of the wrongly detected interactions on the output.

Original: More explanations are given in supplement H.
Revised: More explanations are given in supplement H.

Original: We highlight the advantages of this model as follows: ) intelligent , unlike many other models, ParaACE model gives explicit transformations of features, making it interpretable.
Revised: We highlight the advantages of this model as follows: 1) interpretable , unlike many other models, ParaACE model gives explicit transformations of features, making it interpretable.

Original: 2) lightweight , this model can signiﬁcantly reduce the number of parameters from the original one, and unlike the pruning technique (Han et al., 2015), it does not need extra memory to store the indices of the preserved weights.
Revised: 2) lightweight , this model can signiﬁcantly reduce the number of parameters from the original one, and unlike the pruning technique (Han et al., 2015), it does not need extra memory to store the indices of the preserved weights.

Original: 3) ﬂexible , the structure of NN is modularized by blocks, which is easy to plug in/off for other applications.
Revised: 3) ﬂexible , the structure of NN is modularized by blocks, which is easy to plug in/off for other applications.

Original: The ﬁx-up layer can be redesigned as block-wise subnetworks to capture high-level interactions, end up in the shape of a hierarchical deep network.
Revised: The ﬁx-up layer can be redesigned as block-wise subnetworks to capture high-level interactions, end up in the shape of a hierarchical deep network.

Original: 4) parallelizable , each small subnetwork can be deployed on a portable device such as smartphone.

Original: This new-fashioned network can be trained in a distributed manner under the federated learning framework.

Original:  E XPERIMENTS 5. E XPERIMENTAL S ETUP
Revised:  E XPERIMENTS 5. E XPERIMENTAL S ETUP

Original: Datasets: We generated 10 synthetic datasets ( p = 10 ) as was used in Tsang et al.
Revised: Datasets: We generated 10 synthetic datasets ( p = 10 ) as was used in Tsang et al.

Original: (2018a) to test the accuracy of different interaction detection methods.
Revised: (2018a) to test the accuracy of different interaction detection methods.

Original: We added a scaled Gaussian noise η ∼ 0 .
Revised: We added a scaled Gaussian noise η ∼ 0 .

Original: 1 × N (0 , 1) to get the noisy data.
Revised: 1 × N (0 , 1) to get the noisy data.

Original: We also selected 5 real datasets, namely the Elevators for controlling an F16 aircraft (Itorgo, 2019),
Revised: We also selected 5 real datasets, namely the Elevators for controlling an F16 aircraft (Itorgo, 2019),

Original: Parkinsons for predicting the total UPDRS scores (Tsanas et al., 2009), Skillcraft for game player behavior analysis (Thompson et al., 2013), Cal housing for house price prediction (Pace & Barry, 1997), and Bike sharing for predicting hourly bike rental count (Fanaee-T & Gama, 2014).
Revised: Parkinsons for predicting the total UPDRS scores (Tsanas et al., 2009), Skillcraft for game player behavior analysis (Thompson et al., 2013), Cal housing for house price prediction (Pace & Barry, 1997), and Bike sharing for predicting hourly bike rental count (Fanaee-T & Gama, 2014).

Original: The datasets are preprocessed, and details are shown in supplement D.
Revised: The datasets are preprocessed, and details are shown in supplement D.

Original: Neural Network Conﬁguration:
Revised: Neural Network Conﬁguration:

Original: For the baseline OverparaFC (Teacher), we used the architecture of p -5000-900-400-100-30-1, which is a standard ReLU network with the last hidden layer being linear.
Revised: For the baseline OverparaFC (Teacher), we used the architecture of p -5000-900-400-100-30-1 (indicating a deep neural network with p inputs, one output, andhidden layers with 5000, 900, 400, 100, 30 neurons, respectively), which is a standard ReLU network with the last hidden layer being linear.

Original: For the ParaACE model (Student), the size of the network mainly depends on the number of candidate interactions we choose, more interactions imply more blocks.
Revised: For the ParaACE model (Student), the size of the network mainly depends on the number of candidate interactions we choose; more interactions imply more blocks.

Original: However, the conﬁguration for each subnetwork is ﬁxed, the structure of which is 1-50-8-1 for main effects and 2-50-8-1 for pairwise interactions.
Revised: However, the conﬁguration for each subnetwork is ﬁxed, the structure of which is 1-50-8for main effects and 2-50-8-1 for pairwise interactions.

Original: These subnetworks are ReLU networks with linear output.
Revised: These subnetworks are ReLU networks with linear output.

Original: The ﬁx-up layer is chosen as a single layer ResNet with the number of neurons equal to 15 .
Revised: The ﬁx-up layer is chosen as a single layer ResNet with the number of neurons equal to 15 .

Original: All  above networks were initialized with Kaiming’s strategy (He et al., 2015).
Revised: All the above networks were initialized with Kaiming’s strategy (He et al., 2015).

Original: Optimizer Setup:
Revised: Optimizer Setup:

Original: We chose adam (Kingma & Ba, 2014) as the optimizer, with the default setup in
Revised: We chose adam (Kingma & Ba, 2014) as the optimizer, with the default setup in

Original: MXNet (Chen et al., 2015), and the batch size is set to be 500 for all the datasets.
Revised: MXNet (Chen et al., 2015), and the batch size is set to be 500 for all the datasets.

Original: 5. P AIRWISE I NTERACTION D ETECTION
Revised: 5. P AIRWISE I NTERACTION D ETECTION

Original: We ﬁrst test on 10 different synthetic datasets ( p = 10 ) (Tsang et al., 2018a) to show the superiority of our proposed method.
Revised: We ﬁrst test on 10 different synthetic datasets ( p = 10 ) (Tsang et al., 2018a) to show the superiority of our proposed method.

Original: Datasets with higher input dimension ( p = 50 , 100 ) were compared as well, see supplement F. To compare the detection performance of different methods, we adopt the AUC metric (see supplement E).
Revised: Datasets with higher input dimensions ( p = 50 , 100 ) were compared as well, see supplement F. To compare the detection performance of different methods, we adopt the ROC-AUC metric (see supplement E).

Original: Here, we compared our proposed method with various existing methods, including ANOVA (Wonnacott & Wonnacott, 1990), AG 
Revised: In Table 1, we compared our proposed method with various existing methods, including ANOVA (Wonnacott & Wonnacott, 1990), AG (Sorokina et al., 2008),

Original: (Sorokina et al., 2008), IH (Janizek et al., 2020), NID (Tsang et al., 2018a), and PID (Liu et al., 2020).
Revised: IH (Janizek et al., 2020), NID (Tsang et al., 2018a), and PID (Liu et al., 2020).

Original: HierLasso (Bien et al., 2013) and RuleFit (Friedman et al., 2008) have been omitted due to their weak performances.
Revised: HierLasso (Bien et al., 2013) and RuleFit (Friedman et al., 2008) have been omitted due to their weak performances.

Original: For our method, we pick the same number of data samples n = 10000 for the training of OverparaFC, and runindependent Monte Carlo tests.
Revised: For our method, we pick the same number of data samples n = 10000 for the training of OverparaFC and run 5 independent Monte Carlo tests.

Original: In Algorithm 1, we pull each arm 3 times for initialization.
Revised: In Algorithm 1, we pull each arm 3 times for initialization.

Original: Each arm will be pulled at maximally m = 100 times, and we terminate when k = 20 strongest interactions stand out.
Revised: Each arm will be pulled at maximally m = 100 times, and we terminate when k = 20 strongest interactions stand out.

Original: We take ˆ µ r ( t ) upon termination as the ﬁnal scores for our method.
Revised: We take ˆ µ r ( t ) upon termination as the ﬁnal scores for our method.

Original: In this experiment, our proposed method needs around 1500 pulls of arms to pick out the top 20 interactions, however, naively pulling each arm 100 times needs 100 C 210 = 4500 pulls in total. 
Revised: In this experiment, our proposed method needs around 1500 pulls of arms to pick out the top 20 interactions, however, naively pulling each arm 100 times needs 100 C 210 = 4500 pulls in total. [Table]

Original: From the results we can conclude the following: (1) PID outperforms all others; (2) our proposed method is as competitive as the PID method; (3) while the average performance of our method is only slightly worse than the PID method due to the poor result on dataset 7; this is due to the term

Revised: From the results, we can conclude the following: (1) PID outperforms all others; (2) our proposed method is as competitive as the PID method; (3) while the average performance of our method is only slightly worse than the PID method due to the poor result on dataset 7; this is due to the term

Original: +( x 4 x 5 x 6 x 7 x 8 ) 2 in F 7 ( x ) has little impact on the output, and such interactions are hard to detect.
Revised: +( x 4 x 5 x 6 x 7 x 8 ) 2 in F 7 ( x ) has little impact on the output, and such interactions are hard to detect.

Original: Apart from the above facts, both NID and PID work by analyzing the connectivity of NN, which strongly relies on speciﬁc sparse ReLU networks and empirical interaction strength computations, and thus are risky to generalize and may fail for some other datasets.
Revised: Apart from the above facts, both NID and PID work by analyzing the connectivity of NN, which strongly relies on speciﬁc sparse ReLU networks and empirical interaction strength computations, and thus are risky to generalize and may fail for some other datasets.

Original: For instance, for the function
Revised: For instance, for the function

Original: F 11 ( x ) =
Revised: F 11 ( x ) =

Original: x 21 x 2 x 23 x 4 + x 25 x 6 x 27 x 8 , the AUC of NID degrades to around 0.69, while our method maintains 1.0 robustly.
Revised: x 21 x 2 x 23 x 4 + x 25 x 6 x 27 x 8 , the ROC-AUC of NID degrades to around 0.69, while our method maintains 1.0 robustly.

Original: Moreover, our method is model-agnostic and can be directly applied to any pre-trained learning models, such as tree models, kernel regression models, etc., which is impossible for other methods.[Table]
Revised: Moreover, our method is model-agnostic and can be directly applied to any pre-trained learning models, such as tree models, kernel regression models, etc., which is impossible for other methods.

Original: 5. M ODEL C OMPRESSION WITH I NTERACTION K NOWLEDGE
Revised: 5. M ODEL C OMPRESSION WITH I NTERACTION K NOWLEDGE

Original: In this section, we construct a Student model with the learned interactions.
Revised: In this section, we construct a Student model with the learned interactions.

Original: We ﬁrst show that, with the interaction knowledge extracted from the Teacher model,  the Student model trained under the same setup can achieve well-improved prediction accuracy (on average).
Revised: With the interaction knowledge extracted from the Teacher model, we ﬁrst show that the Student model trained under the same setup can achieve well-improved prediction accuracy (on average).

Original: We use normalized RMSE (NRMSE), which is deﬁned by
Revised: We use normalized RMSE (NRMSE), which is deﬁned by

Original: [Equation] to measure the performances of all competing models such that the comparison across different datasets is fair and convenient.
Revised: RMSE y max − y min to measure the performances of all competing models such that the comparison across different datasets is fair and convenient.

Original: For synthetic data, we use 800 training samples and 200 test samples.
Revised: For synthetic data, we use 800 training samples and 200 test samples.

Original: The noise η is injected as described earlier.
Revised: The noise η is injected as described earlier.

Original: We choose the top 20 pairwise interactions with our new detection method to build the ParaACE model.
Revised: We choose the top 20 pairwise interactions with our new detection method to build the ParaACE model.

Original: We compared our proposed ParaACE with OverparaFC, KD for regression (Chen et al., 2017), and one recent pruning method known as Lottery Ticket Hypothesis (LTH) (Frankle & Carbin, 2019).
Revised: We compared our proposed ParaACE with OverparaFC, KD for regression (Chen et al., 2017), and two recent pruning methods known as Lottery Ticket Hypothesis (LTH) (Frankle & Carbin, 2019) and SynFlow (Tanaka et al., 2020).

Original: The results are shown in Table 2.
Revised: The results are shown in Table 2.

Original: As we expected, the KD method shows similar performance as the OverparaFC, because the Student model tries hard to approximate its Teacher.
Revised: As we expected, the KD method shows similar performance as the OverparaFC, because the Student model tries hard to approximate its Teacher.

Original: However, our ParaACE model can improve the performance of the OverparaFC by 26.0% on average, and the model compression ratio (CR := # parameters in the uncompressed model # parameters in the compressed model ) is around 300.
Revised: However, our ParaACE model can improve the performance of the OverparaFC by 26.0% on average, and the model compression ratio (CR := [Table] uncompressed [Table]

Revised: ) is around 300.

Original: More details can be found in supplement L.
Revised: More details can be found in supplement L.

Original: We also found that the ParaACE model is sample efﬁcient.
Revised: We also found that the ParaACE model is sample efﬁcient.

Original: To be concrete, the ParaACE trained on a small training set can surpass the over-parameterized NN trained with substantially more data, see Figure 13 in supplement K.
Revised: To be concrete, the ParaACE trained on a small training set can surpass the over-parameterized NN trained with substantially more data, see Figurein supplement K.

Original: [Table]
Revised: [Table]

Original: For real world datasets
Revised: For real-world datasets

Original: [Table]
Revised: [Table]

Original: we built the ParaACE model with the top 50 pairwise interactions for
Revised: we built the ParaACE model with the top 50 pairwise interactions for

Original: Elevators, Parkinsons, Skillcraft, Bike sharing [Table] and the top 20 pairwise interactions for Cal housing (since the number of features is smaller).
Revised: Elevators, Parkinsons, Skillcraft, Bike sharing, and the top 20 pairwise interactions for Cal housing (since the number of features is smaller).

Original: For most datasets, the detected interactions can help improve the predictive performance and signiﬁcantly reduce the model size, see Table 3 for detailed results.
Revised: For most datasets, the detected interactions can help improve the predictive performance and signiﬁcantly reduce the model size, see Table 3 for detailed results.

Original: The comparison with KD is omitted as it is supposed to be close to the Teacher.
Revised: The comparison with KD is omitted as it is supposed to be close to the Teacher.

Original: [Table]
Revised: [Table]

Original: Due to space limitations
Revised: Due to space limitations

Original: [Table]
Revised: [Table]

Original: we give more results for classiﬁcation tasks in supplement I

Original: [Table] Furthermore [Table] we show the performance gain induced by interaction detection via ablation study in supplement N.
Revised: we give more results for classiﬁcation tasks in supplement I. Furthermore, we show the performance gain induced by interaction detection via ablation study in supplement N.

Original: 5. M ODEL I NTERPRETATION
Revised: 5. M ODEL I NTERPRETATION

Original: For synthetic data, we take F 10 ( x ) = sinh ( x 1 + x 2 ) + arccos (tanh( x 3 + x 5 + x 7 ))
Revised: For synthetic data, we take F 10 ( x ) = sinh ( x 1 + x 2 ) + arccos (tanh( x 3 + x 5 + x 7 ))

Original: + cos( x 4 + x 5 )
Revised: + cos( x 4 + x 5 )

Original: + sec( x 7 x 9 ) as an example.
Revised: + sec( x 7 x 9 ) as an example.

Original: Three dominant pairwise interactions { x 1 , x 2 } , { x 4 , x 5 } , { x 7 , x 9 } have been detected successfully and visualized in Figure 4 (left).
Revised: Three dominant pairwise interactions { x 1 , x 2 } , { x 4 , x 5 } , { x 7 , x 9 } have been detected successfully and visualized in Figure 4 (left).

Original: The ground truth transformations are shown in the upper part of the ﬁgure.
Revised: The ground truth transformations are shown in the upper part of the ﬁgure.

Original: The subnetwork failed to learn sec( x 7 x 9 ) due to the lack of representation power.

Original: Besides, the transformations for the single features can also be learned accurately.
Revised: Besides, the transformations for the single features can also be learned accurately.

Original: Taking F 3 ( x ) = exp | x 1 − x 2 | + | x 2 x 3 |− ( x 23 ) | x 4 | +log( x 24 + x 25 + x 27 + x 28 )+ x 9 + 1 1+ xfor example, wherein feature x 6 is not involved, x 9 is linearly transformed, and x 10 is transformed by an even function.
Revised: Taking F 3 ( x ) = exp | x 1 − x 2 | + | x 2 x 3 | − ( x 23 ) | x 4 | + log( x 24 + x 25 + x 27 + x 28 ) + x 9 + 1 1+ x 210 for example, wherein feature x 6 is not involved, x 9 is linearly transformed, and x 10 is transformed by an even function.

Original: All of them are correctly learned as shown in Figure 4 (right).
Revised: All of them are correctly learned as shown in Figure 4 (right).

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: x5 1.51.00.5 0.00.51.01.5 0. x 7 1.5 1.0 0.50.0 0.5 1.0 1. x 9 1.51.00.5 0.00.51.01.50.0050.0000.
Revised: x 7 1.5 1.0 0.50.0 0.5 1.0 1. x 9 1.51.00.50.00.51.01.50.0050.0000.

Original: sinh( x 1 + x 2 ) cos( x 4 + x 5 ) sec( x 7 x 9 ) 1.6 0.8 0.0 0.8 1.6 0.095 0.
Revised: sinh( x 1 + x 2 ) cos( x 4 + x 5 ) sec( x 7 x 9 ) 1.6 0.8 0.0 0.8 1.6 0.095 0.

Original: s 6 ( x 6 )
Revised: s 6 ( x 6 )

Original: s 9 ( x 9 )
Revised: s 9 ( x 9 )

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: For real applications, we ﬁrst take the Cal housing dataset for further analysis as less expert knowledge is required.
Revised: For real applications, we ﬁrst take the Cal housing dataset for further analysis as less expert knowledge is required.

Original: From Figure 5, we can easily recognize that { longitude, latitude } and { totalRooms,totalBedrooms } show strong interactions.
Revised: From Figure 5, we can easily recognize that { longitude, latitude } and { totalRooms,totalBedrooms } show strong interactions.

Original: This makes sense, because { longitude, latitude } indicates the location, and { totalRooms,totalBedrooms } indicates the fraction of bedrooms.
Revised: This makes sense, because { longitude, latitude } indicates the location, and { totalRooms,totalBedrooms } indicates the fraction of bedrooms.

Original: Also in the Parkinsons dataset, we ﬁnd { Age, Sex } are interacted, which is consistent with the result that “age at onset (of Parkinson’s disease) was 2.1 years later in women than in men” in Haaxma et al.
Revised: Also in the Parkinsons dataset, we ﬁnd { Age, Sex } are interacted, which is consistent with the result that “age at onset (of Parkinson’s disease) was 2.1 years later in women than in men” in Haaxma et al.

Original: Interaction detection may also have a huge impact on drug combination discovery (Julkunen et al., 2020).
Revised: Interaction detection may also have a signiﬁcant impact on drug combination discovery (Julkunen et al., 2020).

Original: We applied our interaction detection method on an RF model trained with the Drug combination dataset.
Revised: We applied our interaction detection method on an RF model trained with the Drug combination dataset.

Original: Among the top 34 detected drug-drug interactions, 15 pairwise interactions have been veriﬁed in the DrugBank database (Wishart et al., 2018)
Revised: Among the top 34 detected drug-drug interactions, 15 pairwise interactions have been veriﬁed in the DrugBank database (Wishart et al., 2018)

Original: (see supplement J).
Revised: (see supplement J).

Original: These resultsnot only conﬁrmed the effectiveness of our interaction detection method but also strongly call for careful investigation of the remaining interaction pairs, potentially of great value
Revised: These results conﬁrmed the effectiveness of our interaction detection method and strongly call for careful investigation of the unexplored drug pairs, potentially of great value

Original: [Figure]
Revised: [Figure]

Original: combination data.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we proposed a fast generic and model-agnostic interaction detection method, which is able to extract interaction knowledge from a pre-trained Teacher model.
Revised: This paper proposed a fast, generic, and model-agnostic interaction detection method.

Original: Thorough analyses on the computational complexity of our  method were conducted.
Revised: A thorough analysis of the computational complexity of our adaptive method proved its superior efﬁcacy.

Original: The obtained interaction knowledge can be further encoded into designing a  Student model, called ParaACE, which is  more compact and equally accurate (mostly even better).
Revised: The detected interaction knowledge can be further exploited to build a novel Student model, called ParaACE, which is compact, more interpretable, and mostly more accurate than its Teacher and other competing methods.

Original: We also showed experimentally that our ParaACE model is sample efﬁcient.
Revised: We also showed experimentally that our ParaACE model is sample efﬁcient owing to its simple architecture.

Original: The model-agnostic property enables tremendous applications in different domains of ﬁnance,smart medicine, biology, wireless communication,etc., where a great number of wellperformed black box models have been built but not interpreted yet.
Revised: Its model-agnostic property enables tremendous applications in different domains of ﬁnance, medicine, biology, wireless communication, where a signiﬁcant number of well-performed black-box models have been built but not interpreted yet.

Original: Supplementary DocumentFast Generic Interaction Detection to Interpretability A P ROBLEMS OF A NALYTICAL E VALUATION
Revised: Supplementary Document

Revised: Fast Generic Interaction Detection for Model Interpretability and Compression A P ROBLEMS WITH A NALYTICAL E VALUATION

Original: In this section, we ﬁrst illustrate the conceptual difference between local and global interaction, and then state the problems of analytical evaluation.
Revised: In this section, we ﬁrst illustrate the conceptual difference between local and global interaction and then state the problems of analytical evaluation.

Original: Illustration of Local vs. Global Interaction.
Revised: Illustration of Local vs. Global Interaction.

Original: A quick example showing the difference of local/global interaction is the MATLAB logo (Figure 6(a)).
Revised: A quick example showing the difference of local/global interaction is the MATLAB symbol 1 (Figure 6(a)).

Original: There is no local interaction inside the ﬂat region, but two input variables globally interact.
Revised: There is no local interaction inside the ﬂat region, but two input variables globally interact.

Original: Analytical evaluation works for neural networks with differentiable activation function, e.g.  sigmoid , tanh , etc.
Revised: Analytical evaluation works for neural networks with differentiable activation functions, e.g. , sigmoid , tanh , etc.

Original: However, it is problematic when we use neural networks with piece-wise linear activation functions (PLNN), such as ReLU , Leaky ReLU , and so on.
Revised: However, it is problematic when we use neural networks with piece-wise linear activation functions (PLNN), such as ReLU , Leaky ReLU , and so on.

Original: The Hessian will be a zero matrix at every second-order differentiable point, and it will no longer providethe information about local interaction.
Revised: The Hessian will be a zero matrix at every second-order differentiable point, and it will no longer provide information about local interaction.

Original: Imagine the function landscape is joint with many facets,  the interaction information is hidden in the boundary of those facets (the set of non-differentiable points).
Revised: Imagine the function landscape is joint with many facets, and the interaction information is hidden in the boundary of those facets (the set of non-differentiable points).

Original: A simple example is checkerboard-like function, e.g.  XOR function F ( x 1 , x 2 )
Revised: A simple example is checkerboard-like function, e.g. , XOR function F ( x 1 , x 2 )

Original: =
Revised: =

Original: 1 { x 1 x 2 > 0 } , where x 1 , x 2 are uniformly drawn from [ − 1 , 1] .
Revised: 1 { x 1 x 2 > 0 } , where x 1 , x 2 are uniformly drawn from [ − 1 , 1] .

Original: In this case, x 1 and x 2 are clearly interacted, however, the second-order derivatives ∂F ∂x 1 ∂x 2 will be zero on almost all sampled points.
Revised: In this case, x 1 and x 2 are clearly interacted, however, the second-order derivatives [Figure] 2 will be zero for almost all sampled points.

Original: The interaction information is hidden at the line x 1 = 0 and x 2 = 0 .
Revised: The interaction information is hidden at the line x 1 = 0 and x 2 = 0 .

Original: [Figure]
Revised: [Figure]

Revised: The function is approximated by a PLNN, and the landscape is spliced by ﬂat facets.

Original:  almost every point (no local interaction), however, x 1 and x 2 still interact globally.
Revised: The Hessian matrix is a zero matrix on almost every point (no local interaction), but x 1 and x 2 still interact globally.

Original: B P ROOF OF T HEOREM 3.1 AND D ISCUSSION ON THE C HOICE OF h
Revised: B P ROOF OF T HEOREM 3.1 AND D ISCUSSION ON THE C HOICE OF h

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: For any x and y , function F shows no interaction between x and y , i.e. it can be decomposed as F ( x, y )
Revised: For any x and y , function F shows no interaction between them, i.e., it can be decomposed as F ( x, y )

Original: = a ( x ) + b ( y ) if and only if , ∀ h, k > 0 , F ( x + h, y + k ) − F ( x + h, y − k ) − F ( x − h, y + k ) + F ( x − h, y − k )
Revised: = a ( x ) + b ( y ) if and only if , ∀ h, k > 0 , F ( x + h, y + k ) − F ( x + h, y − k ) − F ( x − h, y + k ) + F ( x − h, y − k )

Original: = 0 .
Revised: = 0 .

Original: Proof.
Revised: Proof.

Original: ( ⇒ ) is trivial.
Revised: ( ⇒ ) is trivial.

Original: ( ⇐ ) We will directly have, ∀ h, k > 0 and ∀ x, y ,
Revised: ( ⇐ ) We will directly have, ∀ h, k > 0 and ∀ x, y ,

Original: [Equation]
Revised: [Equation]

Original: Let k → 0 , we will have ∂F∂y ( x + h, y )
Revised: Let k → 0 , we will have ∂F∂y ( x + h, y )

Original: =
Revised: =

Original: ∂F∂y ( x − h, y ) for any positive h ; let h → 0 , we will have ∂F∂x ( x, y + k ) =
Revised: ∂F∂y ( x − h, y ) for any positive h ; let h → 0 , we will have ∂F∂x ( x, y + k ) =

Original: ∂F∂x ( x, y − k ) for any positive k .
Revised: ∂F∂x ( x, y − k ) for any positive k .

Original: Thus, ∂F∂y is irrespective of x and ∂F∂x is irrespective of y .
Revised: Thus, ∂F∂y is irrespective of x and ∂F∂x is irrespective of y .

Original: Remark B.1.
Revised: Remark B.1.

Original: Theorem 3.1 can be extended to a higher-order interactioncase with the same proof sketch.
Revised: Theorem 3.1 can be extended for higher-order interaction with the same machinery.

Original: [Figure]
Revised: [Figure]

Original: A guide for the choice of h i ( j )  We empirically test the sensitivity ofthe choice of h i ( j ) , see Figure 7.
Revised: A guide for the choice of h i ( j ) . We empirically test the sensitivity of h i ( j ) , see Figure 7.

Original: For standardized features , we suggest using h i ( j ) ∈ [0 . 6 , 0 . 9] as  rule-of-thumb.
Revised: For standardized features , we suggest using h i ( j ) ∈ [0 . 6 , 0 . 9] as the rule-of-thumb.

Original: Further discussion on the choice of h i ( j )  People may argue that for the interaction detection task, a small number of evaluations for each feature pair are already enough to get pretty good results, and the beneﬁts of  the UCB algorithm would be trivial, which is not always true.
Revised: Further discussions on the choice of h i ( j ) . People may argue that for the interaction detection task, a small number of evaluations for each feature pair are already enough to get pretty good results, and the beneﬁts of using the UCB algorithm is trivial. This is not always true.

Original: We address this from two perspectives.
Revised: We address this from two perspectives in the following.

Original: First, in many scenarios, the function can only be evaluated sequentially, and the evaluation may be very expensive.
Revised: First, in many scenarios, the function can only be evaluated sequentially, and the evaluation may be very expensive.

Original: The beneﬁts of the UCB algorithm are revealed then.
Revised: The beneﬁts of the UCB algorithm are revealed then.

Original: Second, we empirically study the performance of evaluating each arm evenly under different conﬁgurations (perturbation size h i ( j ) , and the number of evaluations for each arm), see Figure 8.
Revised: Second, we empirically study the performance of evaluating each arm evenly under different conﬁgurations (perturbation size h i ( j ) , and the number of evaluations for each arm), see Figure 8.

Original: The AUC scores were obtained from the OverparaFC model trained as described in Section 5.2.
Revised: The ROC-AUC scores were obtained from the OverparaFC model trained as described in Section 5.2.

Original: We expect that given the ﬁxed perturbation size h , the AUC scores will increase as the number of evaluations increase.
Revised: We expect that given the ﬁxed perturbation size h , the ROC-AUC scores will rise as the number of evaluations increases.

Original: This is consistent with the observation that the bricks are brighter from top to bottom.
Revised: This is consistent with the observation that the bricks are brighter from top to bottom.

Original: Also, we observed that with a properly chosen h ∈ [0 . 6 , 0 . 9] , the interactions can be correctly detected with a relatively small number of evaluations (the bricks are brighter in the middle, see dataset 3, 4, and 5).
Revised: Also, we observed that with a properly chosen h ∈ [0 . 6 , 0 . 9] , the interactions can be correctly detected with a relatively small number of evaluations (the bricks are brighter in the middle, see dataset 3, 4, and 5).

Original: The corresponding reasons can be that (a) if h is too small, four evaluated points lie in the same ﬂat region with high probability, which is harmful to interaction detection; (b) if h is too large, some evaluated points may be out of the distribution of the training data, which makes the detected interaction strength less representative.
Revised: The corresponding reasons can be that (a) if h is too small, four evaluated points lie in the same ﬂat region with high probability, which is harmful to interaction detection; (b) if h is too large, some evaluated points may be out of the distribution of the training data, which makes the detected interaction strength less representative.

Original: Even though the suggestion of using h i ( j ) ∈ [0 . 6 , 0 . 9] is purely empirical, the plots here help us better understand the essence of statistical interactions and their detection process.
Revised: Even though the suggestion of using h i ( j ) ∈ [0 . 6 , 0 . 9] is purely empirical, the plots here help us better understand the essence of statistical interactions and their detection process.

Original: C P ROOF OF T HEOREM 3. 
Revised: C P ROOF OF T HEOREM 3.

Original: Recall that we construct the 1 − δ conﬁdence intervals of ˆ f (cid:96) ( r ) as,
Revised: Recall that we construct the 1 − δ conﬁdence intervals of ˆ f (cid:96) ( r ) as,

Original: [Equation]
Revised: [Equation]

Original: This is derived from concentration inequality (Inequality 2.9 in Wainwright (2019)).
Revised: This is derived from concentration inequality (Inequality 2.9 in Wainwright (2019)).

Original: We set δ = 2 n 3 m , which is a small number and easy to analyze.
Revised: We set δ = 2 n 3 m , which is a small number and easy to analyze.

Original: We have,
Revised: We have,

Original: Lemma C.1 (high probability event/clean event) .
Revised: Lemma C.1 (high probability event/clean event) .

Original: With probability  1 − 2 n 2 , all true values µ r lie in the their conﬁdence intervals during the run of the algorithm.Proof.
Revised: With probability (w.p.) 1 − 2 n 2 , all true values µ r lie in the their conﬁdence intervals during the run of the algorithm.

Original: It’s equivalent to proof:

Original: [Equation]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: t

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: n s

Original: [Figure]

Original: [Figure]

Original: # e v a l u a t
Revised: # e v a l u a t

Original: i o n s
Revised: i o n s

Original: [Figure]
Revised: [Figure]

Original: dataset

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: n
Revised: n

Original: [Figure]
Revised: [Figure]

Original: The opposite of this event is ∃ t and r, s.t. , | µ r
Revised: Proof. It’s equivalent to proof:

Original: [Equation]
Revised: [Equation]

Original:  r [Equation] t 
Revised: The opposite of this event is ∃ t and r, s.t. , | µ r − ˆ µ r ( t ) | > C r ( t ) . Since we have,

Original: [Equation]
Revised: [Equation]

Original: | > C r [Equation] Since we have,
Revised: and we evaluate at most nm times ( n arms, each arm is pulled m times),

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Revised: and r, s.t. , | µ

Original: [Equation]
Revised: [Equation]

Original: and we evaluate at most nm times ( n arms, each arm is pulled m times),
Revised: − ˆ

Original: [Equation]
Revised: [Equation]

Original: (union bound)

Original: [Equation]
Revised: [Equation]

Original: (clean event)
Revised: ) } ) ≤ [Equation] [Table]

Original: Using Lemma C.1, we restate Theorem 3.2 as follows.
Revised: Using Lemma C.1, we restate Theorem 3.2 as follows.

Original: Theorem C.2 (Restating Theorem 3.2) .
Revised: Theorem C.2 (Restating Theorem 3.2) .

Original: With probability 1 − Θ( 1 n 2 ) , Algorithm 1 returns the kstrongest interactions with at most
Revised: With probability 1 − Θ( 1 n 2 ) , Algorithm 1 returns the kstrongest interactions with at most

Original: [Equation]
Revised: [Equation]

Original: local interaction strength evaluations.
Revised: local interaction strength evaluations.

Original: This takes O (cid:16)(cid:80) ni
Revised: This takes O (cid:16)(cid:80) ni

Original: =1 log [Equation] n 
Revised: =1 log( n ) (cid:16)(cid:16)

Original: [Equation]
Revised: [Equation]

Original: (cid:16)(cid:16)
Revised: log

Original: [Equation]
Revised: [Equation]

Original: ∆

Original: [Equation] m [Equation] time.
Revised: (cid:17) ∧ m (cid:17)(cid:17) time.

Original: Proof.
Revised: Proof.

Original: We ﬁrst analyze the running of algorithm till the ﬁrst strongest interaction comes out.
Revised: We ﬁrst analyze the running of algorithm till the ﬁrst strongest interaction comes out.

Original: If we choose to update arm i (cid:54)
Revised: If we choose to update arm i (cid:54)

Original: = i ∗ 1 at time t , then we have
Revised: = i ∗ 1 at time t , then we have

Original: [Equation]
Revised: [Equation]

Original: For equation 6 to occur, at least one of the following three events must occur:
Revised: For equation 6 to occur, at least one of the following three events must occur:

Original: [Equation]
Revised: [Equation]

Original: To see this, note that if none of E 1 , E 2 , E 3 occurs, we have
Revised: To see this, note that if none of E 1 , E 2 , E 3 occurs, we have

Original: [Equation]
Revised: [Equation]

Original: From Lemma C.1, E 1 and E 2 do not occur during the run of the algorithm with probability 1 − 2 n 2 , because [Equation] p
Revised: From Lemma C.1, E 1 and E 2 do not occur during the run of the algorithm with probability 1 − 2 n 2 , because w.p.

Original: [Equation]
Revised: [Equation]

Original: It also implies w.p. 1 − Θ( 1 n 2 ) , the algorithm does not stop pulling arm i until event E 3 stops occurring.
Revised: It also implies w.p. 1 − Θ( 1 n 2 ) , the algorithm does not stop pulling arm i until event E 3 stops occurring.

Original: Let ζ ( w ) i index the iteration in which Algorithm 1 evaluates arm A
Revised: Let ζ ( w ) i index the iteration in which Algorithm 1 evaluates arm A

Original: i for the last time before declaring it to be the w -th strongest interaction.
Revised: i for the last time before declaring it to be the w -th strongest interaction.

Original: Let’s ﬁrst consider how many pulls are needed for each arm to pick out the ﬁrst strongest interaction.
Revised: Let’s ﬁrst consider how many pulls are needed for each arm to pick out the ﬁrst strongest interaction.

Original: If C i ( ζ (1) i ) ≤ ∆ (1) i 2 for arm
Revised: If C i ( ζ (1) i ) ≤ ∆ (1) i 2 for arm

Original: A i , then we can stop evaluating arm
Revised: A i , then we can stop evaluating arm

Original: A i , that is,
Revised: A i , that is,

Original: [Equation]
Revised: [Equation]

Original: Note that T i ( t ) denotes for the number of pulls of arm A i at iteration t .
Revised: Note that T i ( t ) denotes for the number of pulls of arm A i at iteration t .

Original: We will evaluate arm A i at most
Revised: We will evaluate arm A i at most

Original: [Equation]
Revised: [Equation]

Original: times.
Revised: times.

Original: To pick out the ﬁrst strongest interaction, we need at most M evaluations satisfying
Revised: To pick out the ﬁrst strongest interaction, we need at most M evaluations satisfying

Original: [Equation]
Revised: [Equation]

Original: This result can be extend to ﬁnd all top k strongest interactions concretely,
Revised: This result can be extend to ﬁnd all top k strongest interactions concretely,

Original: [Equation]
Revised: [Equation]

Original: The Algorithm 1 takes O (cid:16)(cid:80)ni
Revised: The Algorithm 1 takes O (cid:16)(cid:80)

Original: [Equation]
Revised: [Equation]

Original: log [Equation] (cid:17)
Revised: ∧ m

Original: [Equation]
Revised: [Equation]

Original: m (cid:17)(cid:17) time, where O (log( n )) is the time for maintaining a priority queue(to ﬁnd the minimal LCB or maximal UCB) in each iteration.
Revised: time, where O (log( n )) is the time for maintaining a priority queue(to ﬁnd the minimal LCB or maximal UCB) in each iteration.

Original: Theorem C.3 (Remark 3.1) .
Revised: Theorem C.3 (Remark 3.1) .

Original: If we further assume that ∆ i ∼ N ( γ, 1) , for some γ , and m = cn , for c ∈ [0 , 1] , then the expected pulls of arms (over randomness in ∆ i ) satisﬁes
Revised: If we further assume that ∆ i ∼ N ( γ, 1) , for some γ , and m = cn , for c ∈ [0 , 1] , then the expected pulls of arms (over randomness in ∆ i ) satisﬁes

Original: E [ M ] ≤ O ( n log( nm ) + km ) with probability 1 − Θ( 1 n 2 ) over randomness in Algorithm 1.
Revised: E [ M ] ≤ O ( n log( nm ) + km ) with probability 1 − Θ( 1 n 2 ) over randomness in Algorithm 1.

Original: Thus, the expected running time is well bounded by O ( n log( nm ) log( n ) + km log( n )) .
Revised: Thus, the expected running time is well bounded by O ( n log( nm ) log( n ) + km log( n )) .

Original: Proof.
Revised: Proof.

Original: This follows directly from the Appendix 2 of (Bagaria et al., 2018a).
Revised: This follows directly from the Appendix 2 of (Bagaria et al., 2018a).

Original: D D ATASETS P RE - PROCESSING D. S YNTHETIC T EST S UITE
Revised: D D ATASETS P RE - PROCESSING D. S YNTHETIC T EST S UITE

Original: To make our experimental results convincing, we follow the test suite ever used in (Tsang et al., 2018a) with the details given in Table 4.
Revised: To make our experimental results convincing, we follow the test suite ever used in (Tsang et al., 2018a) with the details given in Table 4.

Original: Among others, F 1 is a widely used test function for interaction detection, which is generated as described in (Sorokina et al., 2008).
Revised: Among others, F 1 is a widely used test function for interaction detection, which can be generated as described in (Sorokina et al., 2008).

Original: For all the other functions, the input dimension is  10, and x 1 , . . .
Revised: For all the other functions, the input dimension is set to 10, and x 1 , . . .

Original: , x 10 i.i.d.
Revised: , x 10 i.i.d.

Original: ∼ U ( − 1 , 1).
Revised: ∼ U ( − 1 , 1)

Original: There also exist redundant variables (missing in the ground truth function) in a few datasets.
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: D. R EAL D ATASETS
Revised: D. R EAL D ATASETS

Original: All the real datasets are publicly available.
Revised: All the real datasets are publicly available.

Original: We preprocessed the datasets as follows.
Revised: We preprocessed the datasets as follows.

Original: For Parkinsons data, we remove the column motor UPDRS, and predictfor the total UPDRS .
Revised: For Parkinsons data, we remove the column motor UPDRS and predict the total UPDRS .

Original: For SkillCraft data, the target is set to be LeagueIndex .
Revised: For SkillCraft data, the target is set to be LeagueIndex .

Original: For the Bike sharing data, the feature weather is converted into a one-hot representation.
Revised: For the Bike sharing data, the feature weather is converted into a onehot representation.

Original: Before feeding the data into neural networks, we performed data normalization ﬁrst for all datasets.
Revised: Before feeding the data into neural networks, we performed data normalization ﬁrst for all datasets.

Original: The Drug combination dataset is processed as follows.
Revised: The Drug combination dataset is processed as follows.

Original: There are 110 features extracted,  the ﬁrst 50 features are the concentration of 50 unique FDA-approved drugs, and the last 60 features are the one-hot encodings for the cell lines.
Revised: There are 110 features extracted, among which the ﬁrst 50 features are the concentration of 50 unique FDA-approved drugs, and the lastfeatures are the one-hot encodings for the cell lines.

Original: E AUC FOR P AIRWISE I NTERACTION D ETECTION
Revised: E ROC-AUC FOR P AIRWISE I NTERACTION D ETECTION

Revised: We calculate the ROC-AUC scores for the synthetic datasets, where the ground truth interaction pairs are known.

Original: To obtain the AUC value, two vectors are needed,
Revised: To obtain the ROC-AUC value, two vectors are needed,

Original: namely the pairwise interaction score ∈
Revised: namely the pairwise interaction score ∈

Original: R 45+ and ground truth ∈ { 0 , 1 } 45 , both of them are of dimension p ( p − 1)= 45 .
Revised: R 45+ and ground truth ∈ { 0 , 1 } 45 , both of them are of dimension p ( p − 1)= 45 .

Original: By setting different thresholds for the pairwise interaction score ranking, different classiﬁers with False Positive (FP) rate and True Positive (TP) rate can be obtained.
Revised: By setting different thresholds for the pairwise interaction score ranking, different classiﬁers with False Positive (FP) rate and True Positive (TP) rate can be obtained.

Original: The AUC value is approximated from those (FP, TP) pairs by the trapezoidal rule.
Revised: The ROC-AUC value is approximated from those (FP, TP) pairs by the trapezoidal rule.

Original: F P AIRWISE I NTERACTION D ETECTION ON H IGH - D IMENSIONAL D ATASETS
Revised: F P AIRWISE I NTERACTION D ETECTION ON H IGH - D IMENSIONAL D ATASETS

Original: In this section, we created  new high-dimensional datasets by simply combining the 10 synthetic datasets considered in the paper.
Revised: We created two new high-dimensional datasets by simply combining the 10 synthetic datasets considered in the paper.

Original: The new datasets have  50 (using F1-F5) and 100 features (using F1-F10) respectively.
Revised: The new datasets have either 50 (using F1-F5) or 100 features (using F1-F10).

Original: The labels of  two datasets are the sum of the original labels in each dataset of input dimension p = 10 .
Revised: The labels of the two datasets are the sum of the original labels in each dataset of input dimension p = 10 .

Original: The training sample size is increased to 500,000, to mimic big data.
Revised: The training sample size is increased to 500,000 to mimic big data.

Original: Several models are constructed as in the NID paper.

Original: Our method consistently outperforms NID on different network structures (trained with different schemes), see the table of results below.
Revised: We compared 5 different model architectures and our method consistently outperforms NID, see the Table 5 below for the results. [Table] network 1 with main effect [Table]

Revised: L1reg

Original: [Table]
Revised: [Table]

Revised: Small network 2 with main effect + L1reg

Revised: [Table]

Revised: Small network without main effect + L1reg

Revised: [Table]

Revised: Standard big network without main effect

Revised: [Table]

Revised: Here we also report the computational complexity of our UCB-based interaction detection method.

Revised: Due to the randomness of the UCB algorithm, the number of pulls ﬂuctuates.

Revised: Note that we select top k = 200 interaction pairs for 100 -dimensional case, and the total number of pulls is around 40000 , while the naive approach needs 100 × 100 × 99 / 2 = 495000 pulls.

Revised: We select top k =interaction pairs for 50 -dimensional case, and the total number of pulls is around 21000 , while the naive approach needs 100 × 50 × 49 / 2 = 122500 pulls.

Original: Discussion on High Dimensional Data: For high dimensional data, the Hessian matrix will be too huge to handle.
Revised: Discussion on High Dimensional Data: For high dimensional data, the Hessian matrix will be too huge to handle.

Original: One possible solution is to focus on those important features.
Revised: One possible solution is to focus on the important features.

Original: We may ﬁrst take advantage of DeepLIFT(Shrikumar et al., 2017) or SHAP (Lundberg & Lee, 2017) to screen out the  important features, and then apply our interaction detection method to them.
Revised: We may ﬁrst take advantage of DeepLIFT(Shrikumar et al., 2017) or SHAP (Lundberg & Lee, 2017) to screen out the most important features, and then apply our interaction detection method.

Original: If the number of selected features is still too large to handle, we may use ANOVA and F-test to select an affordable number of interaction candidates to pull log( n ) times, and explore the remaining pairs with smaller probability, say 5%.
Revised: If the number of important features is still too large to handle, we may use ANOVA and F-test to select an affordable number of interaction candidates to pull log( n ) times, and explore the remaining pairs with a small probability, say 5%, like the common strategy used in reinforcement learning..

Original: G E XTENSION TO H IGHER -O RDER I NTERACTIONS
Revised: G E XTENSION TO H IGHER -O RDER I NTERACTIONS

Original: We deﬁne three-way interaction as follows, and higher-order interactions can be deﬁned similarly.
Revised: We deﬁne three-way interaction as follows, and higher-order interactions can be deﬁned similarly.

Original: Deﬁnition G.1 (three-way interaction) .
Revised: Deﬁnition G.1 (three-way interaction) .

Original: A function F : R p
Revised: A function F : R p

Original: → R is said to exhibit an interaction among three of its variables x i , x j and x k if,
Revised: → R is said to exhibit an interaction among three of its variables x i , x j and x k if,

Original: [Equation]
Revised: [Equation]

Original: The naive method to detect higher-order interactions is quite similar to FANOVA graph Muehlenstaedt et al.
Revised: The naive method to detect higher-order interactions is quite similar to FANOVA graph Muehlenstaedt et al.

Original: We give three examples of detecting higher-order interactions from the synthetic data.
Revised: We give three examples of detecting higher-order interactions from the synthetic data.

Original: We detect the pairwise interaction strength ﬁrst, then build a weighted graph (the edge weight corresponds to the interaction strength).
Revised: We detect the pairwise interaction strength ﬁrst, then build a weighted graph (the edge weight corresponds to the interaction strength).

Original: By setting a proper number of clusters, higher-order interactions can be discovered correctly, see Figure 9.
Revised: By setting a proper number of clusters, higher-order interactions can be discovered correctly, see Figure 9.

Original: (a) y =
Revised: [Equation]

Original: x 0 x 1 + x 2 x 3 x 4 + x 5 x 6 x 7 + x 8 x(b)
Revised: [Figure]

Original: y =
Revised: Require:

Original: e | x 0 − x 1 | ∗ x 9 + | x 2 ∗ x 3 | − x 2 | x 5 |+ ( x 6 x 7 ) 2 + x(c)
Revised: The target number of i -way interactions { k ( i ) } pi =2 . 1: Detect top- k (2) pairwise interaction via UCB algorithm.

Original: y
Revised: 2: Construct an undirected graph G based on the detected pairwise interactions.

Original:  =x 0 x 1 + | x 1 + x 2 x 3 | + x 4 x 5 − ( x 2 5 ) x 6 − e x 7 + x 8 x [Figure]
Revised: 3: Enumerate all the cliques C in the graph G . 4: for i = 3 , 4 , · · · , p do 5: if C ( i ) = ∅ then

Original: One drawback of the above naive clustering method is that it doesn’t allow for the overlap of nodes, i.e. one variable may appear in several interactions ( F 8 in the test suite).
Revised: One drawback of the above naive clustering method is that it doesn’t allow for the overlap of nodes, i.e. one variable may appear in several interactions ( F 8 in the test suite).

Original: To address the above problem, we propose the Algorithm 2, which detects the higher-order interactions hierarchically.
Revised: To address the above problem, we propose the Algorithm 2, which detects the higher-order interactions hierarchically.

Original: We ﬁrst construct an undirected graph G from the detected pairwise interactions.
Revised: We ﬁrst construct an undirected graph G from the detected pairwise interactions.

Original: To shrink our search space for higher-order interactions, we restrict ourselves to the set of all cliques C of the graph G , where a clique C ∈ C in a graph is a subset of vertices that are all joined by edges.
Revised: To shrink our search space for higher-order interactions, we restrict ourselves to the set of all cliques C of the graph G , where a clique C ∈ C in a graph is a subset of vertices that are all joined by edges.

Original: We use C ( i ) ⊆
Revised: We use C ( i ) ⊆

Original: C to denote the collection of the i -cliques (the cliques with i vertices).
Revised: C to denote the collection of the i -cliques (the cliques with i vertices).

Original: For example, to detect 3-way interactions, one only needs to search in the C (3) , which contains the cliques with three vertices.
Revised: For example, to detect 3-way interactions, one only needs to search in the C (3) , which contains the cliques with three vertices.

Original: Furthermore, we can shrink the search space to C ( i ) reﬁne for i -way interactions, based on the detected ( i − 1 )-way
Revised: Furthermore, we can shrink the search space to C ( i ) reﬁne for i -way interactions, based on the detected ( i − 1 )-way

Original: interactions (see line 8 in Algorithm 2).
Revised: interactions (see line 8 in Algorithm 2).

Original: For example, while detecting 4-way interactions, we put 4-clique { 2 , 3 , 4 , 6 } into consideration only if all the 3-way interactions { 3 , 4 , 6 } , { 2 , 4 , 6 } , { 2 , 3 , 6 } , { 3 , 4 , 6 } exist.
Revised: For example, while detecting 4-way interactions, we put 4-clique { 2 , 3 , 4 , 6 } into consideration only if all the 3-way interactions { 3 , 4 , 6 } , { 2 , 4 , 6 } , { 2 , 3 , 6 } , { 3 , 4 , 6 } exist.

Original: We then use the UCB algorithm to verify if the cliques in C ( i ) reﬁne admit the interaction relationship.
Revised: We then use the UCB algorithm to verify if the cliques in C ( i ) reﬁne admit the interaction relationship.

Original: In this way, we detect the i -way interactions from the detected ( i − 1 )-way interactions information.
Revised: In this way, we detect the i -way interactions from the detected ( i − 1 )-way interactions information.

Original: The 3-way interaction strength can be calculated from Equation 8.
Revised: The 3-way interaction strength can be calculated from Equation 8.

Original: However, with the order of interaction increasing, the number of function evaluations for one gradient computation increases geometrically.
Revised: However, with the order of interaction increasing, the number of function evaluations for one gradient computation increases geometrically.

Original: We do not recommend detecting the interaction whose order is higher than four, which is uneconomical to compute and hard to interpret.
Revised: We do not recommend detecting the interaction whose order is higher than four, which is uneconomical to compute and hard to interpret.

Original: We found our proposed Algorithm 2 can successfully detect the higher-order interactions for F 8 in the test suite 2 .[Equation]
Revised: We found our proposed Algorithm 2 can successfully detect the higher-order interactions for F 8 in the test suite 2 .

Original: [Equation]
Revised: [Equation]

Original: H M ORE ON THE P ARAMETRIC ACE
Revised: H M ORE ON THE P ARAMETRIC ACE

Original: In section 4, we introduced a parametric ACE model based on the following generalized linear additive model with main effects and pairwise feature interactions, namely, 
Revised: In section 4, we introduced a parametric ACE model based on the following generalized linear additive model with main effects and pairwise feature interactions, namely, [Equation]

Original: [Equation]
Revised: [Equation]

Original: The consideration of interactions improves the model performance signiﬁcantly, compare to Generalized Additive Neural Networks (GANN)(Potts, 1999; Agarwal et al., 2020) 3 , see Table 6. 
Revised: The consideration of interactions improves the model performance signiﬁcantly, compare to Generalized Additive Neural Networks (GANN)(Potts, 1999; Agarwal et al., 2020) 3 , see Table 6. were averaged over 5 folds.)

Original: [Table]
Revised: [Table]

Original: The idea behind the original nonparametric ACE algorithm (Breiman & Friedman, 1985) is to alternate between an inner process for ﬁnding the optimal transformation functions of the inputs and an outer process for ﬁnding the optimal transformation function of the output.
Revised: The idea behind the original nonparametric ACE algorithm (Breiman & Friedman, 1985) is to alternate between an inner process for ﬁnding the optimal transformation functions of the inputs and an outer process for ﬁnding the optimal transformation function of the output.

Original: Here, we made several modiﬁcations.
Revised: Here, we made several modiﬁcations.

Original: First, all the transformation functions are represented by small subnetworks.
Revised: First, all the transformation functions are represented by small subnetworks.

Original: Second, we reformulate the above regression equation approximately as
Revised: Second, we reformulate the above regression equation approximately as

Original: [Equation]
Revised: [Equation]

Original: Therefore, the ﬁx-up layer shown in Figure 3 can be understood as the inverse optimal transformation of the output θ − 1 ( · ) .
Revised: Therefore, the ﬁx-up layer shown in Figure 3 can be understood as the inverse optimal transformation of the output θ − 1 ( · ) .

Original: To be more general, θ − 1 ( · ) takes all input transformations as individuals instead of their high-level summary (weighted sum).
Revised: To be more general, θ − 1 ( · ) takes all input transformations as individuals instead of their high-level summary (weighted sum).

Original: Our parametric ACE then alternately tunes the ﬁx-up layer conditioned on the current optimal transformation layer (outer iteration) and tunes the optimal transformation layer conditioned on the current ﬁx-up layer until some convergence condition is met.
Revised: Our parametric ACE then alternately tunes the ﬁx-up layer conditioned on the current optimal transformation layer (outer iteration) and tunes the optimal transformation layer conditioned on the current ﬁx-up layer until some convergence condition is met.

Original: Another important function of the ﬁx-up layer is to alleviate the adversarial impacts of wrongly detected pairwise interactions and/or undetected higher-order interactions altogether on the output.
Revised: Another important function of the ﬁx-up layer is to alleviate the negative impacts of wrongly detected pairwise interactions and/or undetected higher-order interactions altogether on the output.

Original: For this purpose, we could make this ﬁx-up layer a network of small subnetworks, so that the whole architecture is a network of small subnetworks.
Revised: For this purpose, we could make this ﬁx-up layer a network of small subnetworks, so that the whole architecture is a network of small subnetworks.

Original: Each subnetwork can be regarded as a meta-neuron that is expected to be more competent than any SOTA activation function.
Revised: Each subnetwork can be regarded as a meta-neuron that is expected to be more competent than any SOTA activation function.

Original: With such a nice structure, we have divided the hyper-parameters naturally into blocks and the whole network can hopefully be made adaptive to new tasks more rapidly by tuning just a few blocks.
Revised: With such a nice structure, we have divided the hyper-parameters naturally into blocks and the whole network can hopefully be made adaptive to new tasks more rapidly by tuning just a few blocks.

Original: I E XTENSION TO C LASSIFICATION T ASK
Revised: I E XTENSION TO C LASSIFICATION T ASK

Original: We generated the datasets for binary classiﬁcation from the synthetic regression datasets ( samples with injected noise).
Revised: We generated the datasets for binary classiﬁcation from the synthetic regression datasets ( samples with injected noise).

Original: We choose the median of the targetin the synthetic regression dataset as a threshold to separate  the dataset into two classes.
Revised: We choose the median of the target as a threshold to separate each synthetic regression the dataset into two classes.

Original: The comparison between the baseline OverparaFC and our proposed ParaACE  is shown in Table 7.
Revised: The comparison between the baseline OverparaFC and our proposed ParaACE in terms of classiﬁcation accuracy is shown in Table 7.

Original: For real world datasets, we picked Higgs Boson (Baldi et al., 2014)
Revised: For real-world datasets, we picked Higgs Boson (Baldi et al., 2014), Spambase Dua & Graff (2017) [Table]

Original: Spambase Dua & Graff (2017), and Diabetes (Turney, 1994)for comparison. The accuracy and  compression ratio are reported in Table 8.
Revised: and Diabetes (Turney, 1994).Both the classiﬁcation accuracy and the compression ratio are reported in Table [Table] [Table]

Revised: [Table]

Original: J D ETECTED P AIRWISE I NTERACTIONS FOR S YNTHETIC AND R EAL D ATASynthetic
Revised: J D ETECTED P AIRWISE I NTERACTIONS FOR S YNTHETIC AND R EAL D ATA

Original:  data: Figure 10 shows the pairwise interaction strength produced by our proposed method.
Revised: Synthetic data: Figure 10 shows the pairwise interaction strength produced by our proposed method.

Original: [Table] [Table]
Revised: [Figure] [Table]

Original:  Real data: We provide the detected interactions for ﬁve real datasets in Figure 12.
Revised: [Figure] Real data: We provide the detected interactions for ﬁve real datasets in Figure 11.

Original: For the Cal housing dataset, we terminate the UCB algorithm when k = 20 interactions stand out; for the other four datasets, we terminate at k = 50 .
Revised: For the Cal housing dataset, we terminate the UCB algorithm when k = 20 interactions stood out; for the other four datasets, we terminate at k = 50 .

Original: The green, orange, and blue points represent the UCB, estimated mean, and LCB respectively.
Revised: The green, orange, and blue points represent the UCB, estimated mean, and LCB respectively.

Original: For the drug combination dataset, the results are shown in
Revised: For the drug combination dataset, the results are shown in

Original: Figure 11.
Revised: Figure 12.

Revised: [Table]

Revised: [Figure]

Revised: d A g e

Revised: H o u r

Revised: s

Original: K E MPIRICAL R ESULT FOR S AMPLE E FFICIENCY
Revised: P e r W ee k T o t a l H o u r s A P M

Revised: S e l e c t B y H

Revised: o t

Revised: k e y s

Revised: A ss i g n T

Revised: o H o t

Revised: k e y s

Revised: U n i q u e H

Revised: o t

Revised: k e y s M i n i m a p

Revised: A tt a c k s

Revised: M i n i m a p R

Revised: i g h t C li c k s

Revised: N u m b e r

Revised: O f P

Revised: A C s

Revised: G a p B e t w ee n P A

Revised: C s

Revised: A c t i o n L a t e n c y

Revised: A c t i o n s I n P A C

Revised: T o t a l M a p E x p l o

Revised: [Figure]

Revised: e

Revised: [Figure]

Revised: o r k e

Revised: [Figure]

Revised: s M a d

Revised: [Figure]

Revised: U

Revised: [Figure]

Revised: i q u

Revised: [Figure]

Revised: U n

Revised: [Figure]

Revised: t s

Revised: [Figure]

Revised: a d e

Revised: [Figure]

Revised: l e x U n i t s M a

Revised: [Figure]

Revised: [Table]

Revised: SelectByHotkeysAssignToHotkeysUniqueHotkeysMinimapAttacksMinimapRightClicksNumberOfPACsGapBetweenPACsActionLatencyActionsInPACTotalMapExploredWorkersMadeUniqueUnitsMadeComplexUnitsMadeComplexAbilitiesUsed

Revised: [Figure]

Revised: [Figure]

Revised: K E MPIRICAL R ESULT FOR S AMPLE E FFICIENCY

Original: Figure 13 shows our proposed framework is sample efﬁcient.
Revised: Figure 13 shows our proposed framework is sample efﬁcient.

Original: The data are from the test suite (Table 4), and noise η is injected.
Revised: The data are from the test suite (Table 4), and noise η is injected.

Original: With theincrease of training samples, both over-parameterized neural nets and our proposed ParaACE perform better.
Revised: Increasing the training sample size, both over-parameterized neural nets and our proposed ParaACE perform better.

Original: But ParaACE is less data-hungry since fewer training samples are needed for ParaACE to achieve a similar performance of the over-parameterized neural nets.
Revised: But ParaACE is less data-hungry since fewer training samples are needed for ParaACE to achieve a similar performance of the over-parameterized neural nets.

Original: The experiments show that ParaACE still performs well, even if with small training samples.
Revised: The experiments show that ParaACE still performs well, even if with small training samples.

Original: L D ETAILS ON THE C OMPARISON WITH KD AND LTH L. C OMPARISON WITH KD
Revised: L D ETAILS ON THE C OMPARISON WITH KD, LTH, AND S YN F LOW L. C OMPARISON WITH KD

Original: KD is widely used for multi-class classiﬁcation problems, which extracts knowledge from the ”soft label” by controlling the temperature, butmuch fewer KD methods are there for regression problems.
Revised: KD is widely used for multi-class classiﬁcation problems, which extracts knowledge from the ”soft label” by controlling the temperature, but fewer KD methods are there for regression problems.

Original: Currently, people mainly use the Teacher bounded regression loss Chen et al.
Revised: Currently, people mainly use the Teacher bounded regression loss Chen et al.

Original: (2017) and the hint loss to deal with the objective detection problem.
Revised: (2017) and the hint loss to deal with the objective detection problem.

Original: Passing unlabeled data to the Teacher model to produce pseudo labels can also help.
Revised: Passing unlabeled data to the Teacher model to produce pseudo labels can also help.

Original: In these ways, the Student is expected to have a similar performance to the Teacher.
Revised: In these ways, the Student is expected to have a similar performance to the Teacher.

Original: We let an over-parameterized FC (10-5000-900-400-100-30-1) be the Teacher, and a ReLU network with architecture (10-70-70-70-70-30-1) be the Student.
Revised: We let an over-parameterized FC (10-5000-900-400-100-30-1) be the Teacher, and a ReLU network with architecture (10-70-70-70-70-30-1) be the Student.

Original: The Student is trained with 800 original data and 4000 pseudo data (labeled by the Teacher).
Revised: The Student is trained with 800 original data and 4000 pseudo data (labeled by the Teacher).

Original: The loss function we adopted is L 2 + Hint loss, where the Hint loss is used on the layer with 30 neurons.
Revised: The loss function we adopted is L 2 + Hint loss, where the Hint loss is used on the layer with 30 neurons.

Original: It shows that KD trained with pseudo data and carefully designed loss function achieves similar performance as the Teacher, while ParaACE trained with original data and simple L 2 loss achieves signiﬁcantly better performance.
Revised: It shows that KD trained with pseudo data and carefully designed loss function achieves similar performance as the Teacher, while ParaACE trained with original data and simple L 2 loss achieves signiﬁcantly better performance.

Original: L. C OMPARISON WITH LTH
Revised: L. C OMPARISON WITH LTH

Original: We used the pruning technique proposed in the lottery tickets hypothesis (LTH) paper(Frankle &
Revised: We used the pruning technique proposed in the lottery tickets hypothesis (LTH) paper(Frankle &

Original: Carbin, 2019).
Revised: Carbin, 2019).

Original: We pruned the fully connected neural network after every 20 epochs.
Revised: We pruned the fully connected neural network after every 20 epochs.

Original: Every time% weights were pruned in each layer except for the last layer.
Revised: Every time% weights were pruned in each layer except for the last layer.

Original: We trained 500 epochs, and set the batch size to be 400.
Revised: We trained 500 epochs, and set the batch size to be 400.

Original: We report the best test performance in each round of pruning in Figure 14.
Revised: We report the best test performance in each round of pruning in Figure 14.

Original: [Table]

Original: [Figure] [Figure] Figure 14 shows that, in most cases pruning the network properly can improve the model performance. 
Revised: Figure 14 shows that, in most cases pruning the network properly can improve the model performance. But if the

Revised: L. C OMPARISON WITH S YN F LOW

Original: But if the  network was over-pruned, the performance may drop.
Revised: We implemented the single shot SynFlow for regression tasks based on the ofﬁcial code repository https://github.com/ganguli-lab/Synaptic-Flow (Tanaka et al., 2020). The sparsity is set to 10 − 2 .for synthetic datasets, thus the neural network is compressed 280 times. For real-world datasets, we compressed the model by 120 times.

Original: M A CCURATE A PPROXIMATION F UNCTION B ENEFITS I NTERACTION D ETECTION
Revised: And we retrained the neural network after the single-shot prune (post-training). Note that we did not prune the biases here. We set the hyper parameters for the post-training process as follows. The batch size is set to be 500, and the number of epochs is 100.

Revised: We use Adam as the optimizer with lr = 0 .

Revised: 001 , betas = (0 . 9 , 0 . 99) .

Revised: M A CCURATE A PPROXIMATION F UNCTION B ENEFITS I NTERACTION D ETECTION

Original: Suppose the underlying ground truth function is f ( x ) , and our approximation function is g θ ( x ) , we omit the θ in the following.
Revised: Suppose the underlying ground truth function is f ( x ) , and our approximation function is g θ ( x ) , we omit the θ in the following.

Original: We show that the estimation error for Hessian | E x (cid:104) ∂ 2 g ( x ) ∂x i ∂x j (cid:105) −
Revised: We show that the estimation error for Hessian (cid:12)(cid:12)(cid:12) E x [Equation] g [Equation]

Original: E x (cid:104) ∂
Revised: x [Equation] (cid:105)

Original: 2 f ( x ) ∂x i ∂x j (cid:105) | is bounded by o ( (cid:15) ) , when | f ( x ) − g ( x ) | < (cid:15) for all x .
Revised: [Equation] E x (cid:104) [Equation] f ( x ) ∂x i ∂x j (cid:105)(cid:12)(cid:12)(cid:12) is bounded by o ( (cid:15) ) , when | f ( x ) − g ( x ) | < (cid:15) for all x .

Original: This implies more accurate pre-trained model leads to better interaction detection accuracy.
Revised: This implies more accurate pre-trained model leads to better interaction detection accuracy.

Original: According to the following theorem, deep neural networks are good surrogate functions due to the universal function approximation capability (Hornik, 1991).
Revised: According to the following theorem, deep neural networks are good surrogate functions due to the universal function approximation capability (Hornik, 1991).

Original: Theorem M.1.
Revised: Theorem M.1.

Original: Assume x
Revised: Assume x

Original: i are uniformly drawn from [ − 1 , 1] independently, if there exist an (cid:15) > 0 , such that | f ( x ) − g ( x ) | ≤ (cid:15) for all x ∈ R p , then
Revised: i are uniformly drawn from [ − 1 , 1] independently, if there exist an (cid:15) > 0 , such that | f ( x ) − g ( x ) | ≤ (cid:15) for all x ∈ R p , then

Original: E x (cid:20) ∂ 2 g ( x ) ∂x i ∂x j (cid:21)
Revised: E x (cid:20) ∂ 2 g ( x ) ∂x i ∂x j (cid:21)

Original: − E x (cid:20) ∂ 2 f ( x ) ∂x i ∂x j (cid:21) ≤ (cid:15).
Revised: − E x (cid:20) ∂ 2 f ( x ) ∂x i ∂x j (cid:21) ≤ (cid:15).

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: E x (cid:20) ∂ 2 g ( x ) ∂x i ∂x j (cid:21) = (cid:90) ∂ 2 g ( x ) ∂x i ∂x j p ( x ) d x = (cid:90) ∂ 2 g ( x ) ∂x i ∂x

Original: [ where p ( x i , x j ) = 1 / 4]
Revised: j p ( x i , x j ) p ( x \ ij )

Original: [Equation]
Revised: d x = (cid:90) p ( x \ ij ) (cid:90)−(cid:90)−∂ 2 g ( x ) ∂x i ∂x j p ( x i , x j ) dx i dx j d x \ ij [ where p ( x i , x j ) = 1 / 4] =  (cid:90) p ( x \ ij ) (cid:90)−(cid:90)−∂ 2 g ( x ) ∂x i ∂x j dx i dx j d x \ ij , where (cid:90)−(cid:90)−∂ 2 g ( x ) ∂x i ∂x j dx i dx

Original: where
Revised: j =

Original: [Equation]
Revised: (cid:90)−∂g ( x \ ij , x i = 1 , x j ) − ∂g ( x \ ij , x i = − 1 , x j ) ∂x j dx j = g ( x \ ij , x i = 1 , x j = 1) − g ( x \ ij , x i = 1 , x j = − 1) − ( g ( x \ ij , x i = − 1 , x j = 1) − g ( x \ ij , x i = − 1 , x j = − 1)) .

Original: We can derive E x (cid:104) ∂ 2 f ( x ) ∂x i ∂x j (cid:105) in a similar fashion.
Revised: We can derive E x (cid:104) ∂ 2 f ( x ) ∂x i ∂x j (cid:105) in a similar fashion.

Original: Thus,
Revised: Thus,

Revised: [Equation]

Revised: Theorem M.2.

Revised: Assume x

Revised: i are uniformly drawn from [ − 1 , 1] independently, and further assume (cid:12) (cid:12) (cid:12) ∂ 2 f ( x ) ∂x i ∂x j (cid:12) (cid:12) (cid:12) , (cid:12) (cid:12) (cid:12) ∂ 2 g ( x ) ∂x i ∂x j (cid:12) (cid:12) (cid:12) ≤ M, ∀ ( x i , x j ) .

Revised: If there exist an (cid:15) > 0 , such that | f ( x ) − g ( x ) | ≤ (cid:15) for all x ∈ R p , then

Original: [Equation]
Revised: [Equation]

Revised: 2 (cid:35) − E x (cid:34)(cid:12)(cid:12)(cid:12)(cid:12) ∂ 2 f ( x ) ∂x i ∂x j (cid:12)(cid:12)(cid:12)(cid:12) 2 (cid:35) ≤ 2 M(cid:15).

Revised: Proof.

Revised: E x (cid:34)(cid:12)(cid:12)(cid:12)(cid:12) ∂ 2 g ( x ) ∂x i ∂x j (cid:12)(cid:12)(cid:12)(cid:12) 2 (cid:35) −

Revised: [Equation]

Revised: (cid:34)(cid:12)(cid:12)(cid:12)(cid:12) ∂

Revised: [Equation]

Revised: f ( x ) ∂x i ∂x j (cid:12)(cid:12)(cid:12)(cid:12) 2 (cid:35) =

Revised: E x (cid:34)(cid:12)(cid:12)(cid:12)(cid:12) ∂ 2 g ( x ) ∂x i ∂x j (cid:12)(cid:12)(cid:12)(cid:12) − (cid:12)(cid:12)(cid:12)(cid:12)

Revised: [Equation]

Revised: f ( x ) ∂x i ∂x j (cid:12)(cid:12)(cid:12)(cid:12)

Revised: [Equation]

Original: ∂x j (cid:21) = 
Revised: (cid:35) = E x (cid:20)(cid:18) ∂ [Equation] g ( x ) ∂x i ∂x

Revised: j − ∂

Revised: [Equation]

Revised: ( x )

Revised: [Equation]

Original: 4 E x \ ij [ g ( x \ ij , x i = 1 , x j = 1) − f ( x \ ij , x i = 1 , x j = 1) − g ( x \ ij , x i = − 1 , x j = 1) + f ( x \ ij , x i = − 1 , x j = 1) − g ( x \ ij , x i = 1 , x j = − 1) + f ( x \ ij , x i = 1 , x j = − 1)
Revised: j (cid:19) (cid:18) ∂ [Equation] ( x ) ∂x i ∂x [Equation]

Original: + g ( x \ ij , x i = − 1 , x j = − 1) − f ( x \ ij , x i = − 1 , x j = − 1)]
Revised: + ∂ [Equation] f ( x ) ∂x [Equation]

Original: ≤4 E x \ ij [4 (cid:15) ] = (cid:15).
Revised: ∂x j (cid:19)(cid:21) ≤ (cid:12)(cid:12)(cid:12)(cid:12) E x (cid:20)(cid:18) ∂ 2 g ( x ) ∂x i ∂x j − ∂ 2 f ( x )

Original: [Table]
Revised: [Table]

Original: To show how much performance gain we are able to obtain from the interaction detection procedure.
Revised: Note that M can be interpreted as the magnitude of the strongest local interaction.

Original: We conduct an experiment that input random pairwise interactions to ParaACE to compare with the one with detected interactions on synthetic datasets, see Table [Table] [Table] [Figure]
Revised: For the functions with strong interactions, higher approximation quality is desired

Original: d A g e

Original: H o u r

Original: s

Original: P e r W ee k T o t a l H o u r s A P M
Revised: [Table]

Original: S e l e c t B y H o t k e y s A ss i g n T
Revised: [Table]

Original: o H o t

Original: k e y s

Original: U n i q u e H o t k e y s M i n i m a p A tt a c k s
Revised: N A BLATION S TUDY FOR THE EFFECTIVENESS OF I NTERACTION D ETECTION

Original: M i n i m a p R i g h t C li c k s N u m b e r O f P
Revised: To show how much performance gain we are able to obtain from the interaction detection procedure.

Original: A C s

Original: G a p B e t w ee n P A C s A c t i o n L a t e n c y A c t i o n s I n P A C
Revised: We conduct an experiment that input random pairwise interactions to ParaACE to compare with the one with detected interactions on synthetic datasets, see Table

Original: T o t a l M a p E x p l o

Original: [Figure]
Revised: [Table]

Original: e

Original: [Figure]

Original: o r k e

Original: [Figure]

Original: s M a d

Original: [Figure]

Original: U

Original: [Figure]

Original: i q u

Original: [Figure]

Original: U n

Original: [Figure]

Original: t s

Original: [Figure]

Original: a d e

Original: [Figure]
Revised: [Figure]

Original: l e x U n i t s M a

Original: [Figure][Table]
Revised: [Figure]

Original: SelectByHotkeysAssignToHotkeysUniqueHotkeysMinimapAttacksMinimapRightClicksNumberOfPACsGapBetweenPACsActionLatencyActionsInPACTotalMapExploredWorkersMadeUniqueUnitsMadeComplexUnitsMadeComplexAbilitiesUsed

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]