Original: Scalable Interpretability via Polynomials
Revised: Scalable Interpretability via Polynomials

Original: AI
Revised: AI

Original: Dhruv Mahajan
Revised: Dhruv Mahajan

Original: Meta AI dhruvm fb
Revised: Meta AI dhruvm fb

Original: Abstract
Revised: Abstract

Original: Generalized Additive Models (GAMs) have quickly become the leading choice for interpretable machine learning.
Revised: Generalized Additive Models (GAMs) have quickly become the leading choice for interpretable machine learning.

Original: However, unlike uninterpretable methods such as
Revised: However, unlike uninterpretable methods such as

Original: DNNs, they lack expressive power and easy scalability, and are hence not a feasible alternative for real-world tasks.
Revised: DNNs, they lack expressive power and easy scalability, and are hence not a feasible alternative for real-world tasks.

Original: We present a new class of GAMs that use tensor rank decompositions of polynomials to learn powerful, inherently-interpretable models.
Revised: We present a new class of GAMs that use tensor rank decompositions of polynomials to learn powerful, inherently-interpretable models.

Original: Our approach, titled Scalable Polynomial Additive Models (SPAM) is effortlessly scalable and models all higher-order feature interactions without a combinatorial parameter explosion.
Revised: Our approach, titled Scalable Polynomial Additive Models (SPAM) is effortlessly scalable and models all higher-order feature interactions without a combinatorial parameter explosion.

Original: SPAM outperforms all current interpretable approaches, and matches DNN/XGBoost performance on a series of real-world benchmarks with up to hundreds of thousands of features.
Revised: SPAM outperforms all current interpretable approaches, and matches DNN/XGBoost performance on a series of real-world benchmarks with up to hundreds of thousands of features.

Original: We demonstrate by human subject evaluations that SPAMs are demonstrably more interpretable in practice, and are hence an effortless replacement for DNNs for creating interpretable and high-performance systems suitable for large-scale machine learning.
Revised: We demonstrate by human subject evaluations that SPAMs are demonstrably more interpretable in practice, and are hence an effortless replacement for DNNs for creating interpretable and high-performance systems suitable for large-scale machine learning.

Original: Source code is available at github.com/facebookresearch/nbm-spam .
Revised: Source code is available at github.com/facebookresearch/nbm-spam .

Original:  Introduction
Revised:  Introduction

Original: Interpretable machine learning systems suffer from a tradeoff between approximation and interpretability: high-performing models used in practice have millions of parameters and are highly non-linear, making them difficult to interpret.
Revised: Interpretable machine learning systems suffer from a tradeoff between approximation and interpretability: high-performing models used in practice have millions of parameters and are highly non-linear, making them difficult to interpret.

Original: Post-hoc explainability attempts to remove this tradeoff by explaining black-box predictions with interpretable instance-specific approximations (e g.,
Revised: Post-hoc explainability attempts to remove this tradeoff by explaining black-box predictions with interpretable instance-specific approximations (e g.,

Original: LIME [Ribeiro et al., 2016], SHAP [Lundberg and Lee, 2017]), however, they are notoriously unstable [Ghorbani et al., 2019a, Lakkaraju and Bastani, 2020], expensive to compute [Slack et al., 2021], and in many cases, inaccurate [Lipton, 2018].
Revised: LIME [Ribeiro et al., 2016], SHAP [Lundberg and Lee, 2017]), however, they are notoriously unstable [Ghorbani et al., 2019a, Lakkaraju and Bastani, 2020], expensive to compute [Slack et al., 2021], and in many cases, inaccurate [Lipton, 2018].

Original: It is therefore desirable to learn models that are instead inherently-interpretable (glass-box), i e., do not require post-hoc interpretability, but provide better performance than inherently-interpretable classical approaches such as linear or logistic regression.
Revised: It is therefore desirable to learn models that are instead inherently-interpretable (glass-box), i e., do not require post-hoc interpretability, but provide better performance than inherently-interpretable classical approaches such as linear or logistic regression.

Original: This has led to a flurry of research interest in Generalized Additive Models [Hastie and Tibshirani, 2017], i.e., methods that non-linearly transform each input feature separately, e.g., tree-based GAM and GA 2 Ms
Revised: This has led to a flurry of research interest in Generalized Additive Models [Hastie and Tibshirani, 2017], i.e., methods that non-linearly transform each input feature separately, e.g., tree-based GAM and GA 2 Ms

Original: [Lou et al., 2013], or NAMs [Agarwal et al., 2021].
Revised: [Lou et al., 2013], or NAMs [Agarwal et al., 2021].

Original: While these are an improvement over linear methods, due to their highly complex training procedures, they are not a simple dropin replacement for black-box methods.
Revised: While these are an improvement over linear methods, due to their highly complex training procedures, they are not a simple dropin replacement for black-box methods.

Original: Furthermore, it is not straightforward to model feature interactions in these models: the combinatorial explosion in higher-order interactions makes this infeasible without a computationally expensive feature selection [Lou et al., 2013, Chang et al., 2021].
Revised: Furthermore, it is not straightforward to model feature interactions in these models: the combinatorial explosion in higher-order interactions makes this infeasible without a computationally expensive feature selection [Lou et al., 2013, Chang et al., 2021].

Original: Contrary to these approaches, we revisit an entirely different solution: polynomials .
Revised: Contrary to these approaches, we revisit an entirely different solution: polynomials .

Original: It is folklore in machine learning that since polynomials are universal approximators [Stone, 1948, Weierstrass, 1885], hypothetically, modeling all possible feature interactions will suffice in creating powerful learners and eliminate the need for intricate non-linear transformations of data.
Revised: It is folklore in machine learning that since polynomials are universal approximators [Stone, 1948, Weierstrass, 1885], hypothetically, modeling all possible feature interactions will suffice in creating powerful learners and eliminate the need for intricate non-linear transformations of data.

Original: Furthermore, polynomials can also provide a rubric for interpretability: lower-degree polynomials, that have interactions between a few features are interpretable but likely less accurate, whereas polynomials of higher degree have larger predictive power at the cost of interpretability.
Revised: Furthermore, polynomials can also provide a rubric for interpretability: lower-degree polynomials, that have interactions between a few features are interpretable but likely less accurate, whereas polynomials of higher degree have larger predictive power at the cost of interpretability.

Original: Thus, not only are polynomial models capable of capturing all possible interactions between features, they also give practitioners the ability to select a suitable model (order of interactions, i.e., degree) based precisely on their requirement in the  interpretability-performance tradeoff.
Revised: Thus, not only are polynomial models capable of capturing all possible interactions between features, they also give practitioners the ability to select a suitable model (order of interactions, i.e., degree) based precisely on their requirement in the 36th Conference interpretability-performance tradeoff.

Original: However, such an approach has till date remained elusive, as learning interpretable feature interactions efficiently and at scale remains an open problem.
Revised: However, such an approach has till date remained elusive, as learning interpretable feature interactions efficiently and at scale remains an open problem.

Original: In this paper we introduce a highly scalable approach to modeling feature interactions for inherently interpretable classifiers based on rank decomposed polynomials .
Revised: In this paper we introduce a highly scalable approach to modeling feature interactions for inherently interpretable classifiers based on rank decomposed polynomials .

Original: Our contributions are:
Revised: Our contributions are:

Original: First, we present an algorithm titled Scalable Polynomial Additive Models ( SPAM ) to learn inherently-interpretable classifiers that can learn all possible feature interactions by leveraging low-rank tensor decompositions of polynomials [Nie, 2017b].
Revised: First, we present an algorithm titled Scalable Polynomial Additive Models ( SPAM ) to learn inherently-interpretable classifiers that can learn all possible feature interactions by leveraging low-rank tensor decompositions of polynomials [Nie, 2017b].

Original: To the best of our knowledge, SPAM is the among the first Generalized Additive Models with feature interactions scalable to 100K features.
Revised: To the best of our knowledge, SPAM is the among the first Generalized Additive Models with feature interactions scalable to 100K features.

Original: SPAM can be trained end-to-end with SGD (backpropagation) and GPU acceleration, but has orders of magnitude fewer parameters than comparable interpretable models.
Revised: SPAM can be trained end-to-end with SGD (backpropagation) and GPU acceleration, but has orders of magnitude fewer parameters than comparable interpretable models.

Original: We demonstrate that under a coarse regularity assumption (Assumption 1), SPAM converges to the optimal polynomial as the number of samples n → ∞ .
Revised: We demonstrate that under a coarse regularity assumption (Assumption 1), SPAM converges to the optimal polynomial as the number of samples n → ∞ .

Original: Furthermore, we establish novel non-asymptotic excess risk bounds that match classic bounds for linear or full-rank polynomials.
Revised: Furthermore, we establish novel non-asymptotic excess risk bounds that match classic bounds for linear or full-rank polynomials.

Original: SPAM outperforms all current interpretable baselines on several machine learning tasks.
Revised: SPAM outperforms all current interpretable baselines on several machine learning tasks.

Original: To the best of our knowledge, our experimental benchmark considers problems of size (number of samples, and dimensionality) orders of magnitude larger than prior work, and we show that our simple approach scales easily to all problems, whereas prior approaches fail.
Revised: To the best of our knowledge, our experimental benchmark considers problems of size (number of samples, and dimensionality) orders of magnitude larger than prior work, and we show that our simple approach scales easily to all problems, whereas prior approaches fail.

Original: Moreover, we demonstrate that the need for modeling feature interactions is as important as non-linear feature transformations: on most real-world tabular datasets, pairwise interactions suffice to match DNN performance.
Revised: Moreover, we demonstrate that the need for modeling feature interactions is as important as non-linear feature transformations: on most real-world tabular datasets, pairwise interactions suffice to match DNN performance.

Original: We conduct a detailed human subject evaluation to corroborate our performance with practical interpretability.
Revised: We conduct a detailed human subject evaluation to corroborate our performance with practical interpretability.

Original: We demonstrate that SPAM explanations are more faithful compared to posthoc methods, making them a drop-in replacement for black-box methods such as DNNs for interpretable machine learning on tabular and concept-bottleneck [Koh et al., 2020] problems.
Revised: We demonstrate that SPAM explanations are more faithful compared to posthoc methods, making them a drop-in replacement for black-box methods such as DNNs for interpretable machine learning on tabular and concept-bottleneck [Koh et al., 2020] problems.

Original:  Related Work
Revised:  Related Work

Original: Here we survey the literature most relevant to our work, i e., interpretable ML with feature interactions, and polynomials for machine learning.
Revised: Here we survey the literature most relevant to our work, i e., interpretable ML with feature interactions, and polynomials for machine learning.

Original: Please see Appendix Section B for a detailed survey.
Revised: Please see Appendix Section B for a detailed survey.

Original: Transparent and Interpretable Machine Learning .
Revised: Transparent and Interpretable Machine Learning .

Original: Early work has focused on greedy or ensemble approaches to modeling interactions [Friedman, 2001, Friedman and Popescu, 2008] that enumerate pairwise interactions and learn additive interaction effects.
Revised: Early work has focused on greedy or ensemble approaches to modeling interactions [Friedman, 2001, Friedman and Popescu, 2008] that enumerate pairwise interactions and learn additive interaction effects.

Original: Such approaches often pick spurious interactions when data is sparse [Lou et al., 2013] and are impossible to scale to modern-sized datasets due to enumeration of individual combinations.
Revised: Such approaches often pick spurious interactions when data is sparse [Lou et al., 2013] and are impossible to scale to modern-sized datasets due to enumeration of individual combinations.

Original: As an improvement, Lou et al.
Revised: As an improvement, Lou et al.

Original: [2013] proposed GA 2 M that uses a statistical test to select only “true” interactions.
Revised: [2013] proposed GA 2 M that uses a statistical test to select only “true” interactions.

Original: However GA 2 M fails to scale to large datasets as it requires constant re-training of the model and ad-hoc operations such as discretization of features which may require tuning for datasets with a large dimensionality.
Revised: However GA 2 M fails to scale to large datasets as it requires constant re-training of the model and ad-hoc operations such as discretization of features which may require tuning for datasets with a large dimensionality.

Original: Other generalized additive models require expensive training of millions of decision trees, kernel machines or splines [Hastie and Tibshirani, 2017], making them unattractive compared to black-box models.
Revised: Other generalized additive models require expensive training of millions of decision trees, kernel machines or splines [Hastie and Tibshirani, 2017], making them unattractive compared to black-box models.

Original: An alternate approach is is to learn interpretable neural network transformations.
Revised: An alternate approach is is to learn interpretable neural network transformations.

Original: Neural Additive
Revised: Neural Additive

Original: Models (NAMs, Agarwal et al.
Revised: Models (NAMs, Agarwal et al.

Original: [2021]) learn a DNN per feature.
Revised: [2021]) learn a DNN per feature.

Original: TabNet [Arık and Pfister, 2021] and NIT [Tsang et al., 2018] alternatively modify NN architectures to increase their interpretability.
Revised: TabNet [Arık and Pfister, 2021] and NIT [Tsang et al., 2018] alternatively modify NN architectures to increase their interpretability.

Original: NODE-GAM
Revised: NODE-GAM

Original: [Chang et al., 2021] improves NAMs with oblivious decision trees for better performance while maintaining interpretability.
Revised: [Chang et al., 2021] improves NAMs with oblivious decision trees for better performance while maintaining interpretability.

Original: Our approach is notably distinct from these prior works: we do not require iterative re-training; we can learn all pairwise interactions regardless of dimensionality; we can train SPAM via backpropagation; and we scale effortlessly to very large-scale datasets.
Revised: Our approach is notably distinct from these prior works: we do not require iterative re-training; we can learn all pairwise interactions regardless of dimensionality; we can train SPAM via backpropagation; and we scale effortlessly to very large-scale datasets.

Original: Learning Polynomials .
Revised: Learning Polynomials .

Original: The idea of decomposing polynomials was of interest prior to the deep learning era.
Revised: The idea of decomposing polynomials was of interest prior to the deep learning era.

Original: Specifically, the work of Ivakhnenko [1971], Oh et al.
Revised: Specifically, the work of Ivakhnenko [1971], Oh et al.

Original: [2003], Shin and Ghosh [1991] study learning neural networks with polynomial interactions, also known as ridge polynomial networks (RPNs).
Revised: [2003], Shin and Ghosh [1991] study learning neural networks with polynomial interactions, also known as ridge polynomial networks (RPNs).

Original: However, RPNs are typically not interpretable: they learn interactions of a very high order, and include non-linear transformation.
Revised: However, RPNs are typically not interpretable: they learn interactions of a very high order, and include non-linear transformation.

Original: Similar rank decompositions have been studied in the context of matrix completion [Recht, 2011], and are also a subject of interest in tensor decompositions
Revised: Similar rank decompositions have been studied in the context of matrix completion [Recht, 2011], and are also a subject of interest in tensor decompositions

Original: [Nie, 2017a, Brachat et al., 2010], where, contrary to our work, the objective is to decompose existing tensors rather than directly learn decompositions from gradient descent.
Revised: [Nie, 2017a, Brachat et al., 2010], where, contrary to our work, the objective is to decompose existing tensors rather than directly learn decompositions from gradient descent.

Original: Recently, Chrysos et al.
Revised: Recently, Chrysos et al.

Original: [2019, 2020] use tensor decompositions to learn higher-order polynomial relationships in intermediate layers of generative models.
Revised: [2019, 2020] use tensor decompositions to learn higher-order polynomial relationships in intermediate layers of generative models.

Original: However, their work uses a recursive formulation and learns high-degree polynomials directly from uninterpretable input data (e.g., images), and hence is non-interpretable.
Revised: However, their work uses a recursive formulation and learns high-degree polynomials directly from uninterpretable input data (e.g., images), and hence is non-interpretable.

Original:  Scalable Polynomial Additive Models
Revised:  Scalable Polynomial Additive Models

Original: Setup and Notation .
Revised: Setup and Notation .

Original: Matrices are represented by uppercase boldface letters, e.g., X and vectors by boldface lowercase, i.e., x .
Revised: Matrices are represented by uppercase boldface letters, e.g., X and vectors by boldface lowercase, i.e., x .

Original: We assume that the covariates lie within the set X ⊆ R d , and the labels lie within the finite set Y .
Revised: We assume that the covariates lie within the set X ⊆ R d , and the labels lie within the finite set Y .

Original: Data ( x , y ) ∈
Revised: Data ( x , y ) ∈

Original: X × Y are drawn following some unknown (but fixed) distribution P .
Revised: X × Y are drawn following some unknown (but fixed) distribution P .

Original: We assume we are provided with n i.i.d.
Revised: We assume we are provided with n i.i.d.

Original: samples { ( x i , y i ) } ni =1 as the training set.
Revised: samples { ( x i , y i ) } ni =1 as the training set.

Original: Motivation .
Revised: Motivation .

Original: Generalized Additive Models [Hastie and Tibshirani, 2017] are an excellent design choice for interpretable models, as they learn transformations of individual features, allowing us to model exactly the contribution of any feature.
Revised: Generalized Additive Models [Hastie and Tibshirani, 2017] are an excellent design choice for interpretable models, as they learn transformations of individual features, allowing us to model exactly the contribution of any feature.

Original: A typical GAM is as follows.
Revised: A typical GAM is as follows.

Original: [Equation]
Revised: [Equation]

Original: Where f i , etc., are possibly non-linear transformations.
Revised: Where f i , etc., are possibly non-linear transformations.

Original: It is evident [Lou et al., 2013] that as the order of interaction increases, e.g., beyond pairwise interactions, these models are no longer interpretable, albeit at some benefit to performance.
Revised: It is evident [Lou et al., 2013] that as the order of interaction increases, e.g., beyond pairwise interactions, these models are no longer interpretable, albeit at some benefit to performance.

Original: While some approaches (e.g., Agarwal et al.
Revised: While some approaches (e.g., Agarwal et al.

Original: [2021], Chang et al.
Revised: [2021], Chang et al.

Original: [2021]) learn f i via neural networks, we want to learn the simplest GAMs, i.e., polynomials.
Revised: [2021]) learn f i via neural networks, we want to learn the simplest GAMs, i.e., polynomials.

Original: Specifically, we want to learn a polynomial P ( x ) , of order (degree) k ≤ d of the form:
Revised: Specifically, we want to learn a polynomial P ( x ) , of order (degree) k ≤ d of the form:

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: Here, the weights W ( l ) = { w ( l ) i 1 ...
Revised: Here, the weights W ( l ) = { w ( l ) i 1 ...

Original: i l } , 1 ≤ l ≤ k capture the l − ary interactions between subsets of l features.
Revised: i l } , 1 ≤ l ≤ k capture the l − ary interactions between subsets of l features.

Original: For small values of d and k , one can potentially enumerate all possible interactions and learn a linear model of O ( d k ) dimensionality, however this approach quickly becomes infeasible for large d and k .
Revised: For small values of d and k , one can potentially enumerate all possible interactions and learn a linear model of O ( d k ) dimensionality, however this approach quickly becomes infeasible for large d and k .

Original: Furthermore, the large number of parameters in P ( x ) make regularization essential for performance, and the computation of each interaction can be expensive.
Revised: Furthermore, the large number of parameters in P ( x ) make regularization essential for performance, and the computation of each interaction can be expensive.

Original: Alternatively, observe that any polynomial that models k − ary interactions can be written as follows, for weights { W ( l ) } kl =1 ,
Revised: Alternatively, observe that any polynomial that models k − ary interactions can be written as follows, for weights { W ( l ) } kl =1 ,

Original: [Equation]
Revised: [Equation]

Original: Here, the weights W ( l ) ∈ R d l are written as a tensor of order l , and the operation ⊙ l refers to a repeated inner product along l dimensions, e.g., W ∈ R d 2 , x
Revised: Here, the weights W ( l ) ∈ R d l are written as a tensor of order l , and the operation ⊙ l refers to a repeated inner product along l dimensions, e.g., W ∈ R d 2 , x

Original: ⊤ W x = W ⊙ 2 x =
Revised: ⊤ W x = W ⊙ 2 x =

Original: ( W ⊙ x ) ⊙ x where ⊙ denotes the inner product.
Revised: ( W ⊙ x ) ⊙ x where ⊙ denotes the inner product.

Original: More generally, any symmetric order l tensor admits an equivalent polynomial representation having only degree − l terms.
Revised: More generally, any symmetric order l tensor admits an equivalent polynomial representation having only degree − l terms.

Original: Now, this representation is still plagued with the earlier curse of dimensionality for arbitrary weight tensors { W ( l ) } kl =1 , and we will now demonstrate how to circumvent this issue by exploiting rank decompositions.
Revised: Now, this representation is still plagued with the earlier curse of dimensionality for arbitrary weight tensors { W ( l ) } kl =1 , and we will now demonstrate how to circumvent this issue by exploiting rank decompositions.

Original: 3.1 Learning Low-Rank Decompositions of Polynomials
Revised: 3.1 Learning Low-Rank Decompositions of Polynomials

Original: The primary insight of our approach is to observe that any symmetric tensor W ( l ) of order l also admits an additional rank decomposition [Nie, 2017a, Brachat et al., 2010] ( ⊗ being the outer product):
Revised: The primary insight of our approach is to observe that any symmetric tensor W ( l ) of order l also admits an additional rank decomposition [Nie, 2017a, Brachat et al., 2010] ( ⊗ being the outer product):

Original: [Equation]
Revised: [Equation]

Original: Where { u i } ri =1 are the (possibly orthonormal)
Revised: Where { u i } ri =1 are the (possibly orthonormal)

Original: d dimensional basis vectors, r ∈ Z + denotes the rank of the tensor, and the scalars { λ i } ri =1 ,
Revised: d dimensional basis vectors, r ∈ Z + denotes the rank of the tensor, and the scalars { λ i } ri =1 ,

Original: λ i ∈ R denote the singular values.
Revised: λ i ∈ R denote the singular values.

Original: Our objective is to directly learn the rank decomposition of an order l tensor, and therefore learn the required polynomial function.
Revised: Our objective is to directly learn the rank decomposition of an order l tensor, and therefore learn the required polynomial function.

Original: This gives us a total of O ( rd ) learnable parameters per tensor, down from a previous dependence of O ( d l ) .
Revised: This gives us a total of O ( rd ) learnable parameters per tensor, down from a previous dependence of O ( d l ) .

Original: This low-rank formulation additionally enables us to compute the polynomial more efficiently.
Revised: This low-rank formulation additionally enables us to compute the polynomial more efficiently.

Original: Specifically, for a degree k polynomial with ranks r = { 1 , r 2 , ..., r k } , we have:
Revised: Specifically, for a degree k polynomial with ranks r = { 1 , r 2 , ..., r k } , we have:

Original: [Equation]
Revised: [Equation]

Original: Where { u li } r l i =1 , { λ li } r l i =1 denote the corresponding bases and eigenvalues for W ( l ) .
Revised: Where { u li } r l i =1 , { λ li } r l i =1 denote the corresponding bases and eigenvalues for W ( l ) .

Original: The above term can now be easily computed via a simple inner product, and has a time as well as space complexity of O ( d ∥ r ∥ 1 ) , instead of the earlier O ( d k ) .
Revised: The above term can now be easily computed via a simple inner product, and has a time as well as space complexity of O ( d ∥ r ∥ 1 ) , instead of the earlier O ( d k ) .

Original: Here, ∥ r ∥ 1 simply represents the sum of ranks of all the k tensors.
Revised: Here, ∥ r ∥ 1 simply represents the sum of ranks of all the k tensors.

Original: Note that low-rank decompositions do not enjoy a free lunch: the rank r of each weight tensor can, in the worst case, be polynomial in d .
Revised: Note that low-rank decompositions do not enjoy a free lunch: the rank r of each weight tensor can, in the worst case, be polynomial in d .

Original: However, it is reasonable to assume that correlations in the data X will ensure that a small r will suffice for real-world problems (see, e.g., Assumption 1).
Revised: However, it is reasonable to assume that correlations in the data X will ensure that a small r will suffice for real-world problems (see, e.g., Assumption 1).

Original: Optimization .
Revised: Optimization .

Original: For notational simplicity, let us parameterize any order k polynomial by the weight set θ = { b, u 1 , { λ 2 i , u 2 i } r 2 i =1 , ..., { λ ki , u ki } r
Revised: For notational simplicity, let us parameterize any order k polynomial by the weight set θ = { b, u 1 , { λ 2 i , u 2 i } r 2 i =1 , ..., { λ ki , u ki } r

Original: k
Revised: k

Original: i =1 } ∈ R r ( d +1) where from now onwards r = ∥
Revised: i =1 } ∈ R r ( d +1) where from now onwards r = ∥

Original: r ∥ 1 = 1 + r 2 + r 3 + ...
Revised: r ∥ 1 = 1 + r 2 + r 3 + ...

Original: + r k denotes the cumulative rank across all the tensors.
Revised: + r k denotes the cumulative rank across all the tensors.

Original: We write the polynomial parametrized by θ as P ( · ; θ ) , and the learning problem for any loss ℓ is:
Revised: We write the polynomial parametrized by θ as P ( · ; θ ) , and the learning problem for any loss ℓ is:

Original: [Equation]
Revised: [Equation]

Original: Here, Θ denotes the feasible set for θ , and R ( · ) denotes an appropriate regularization term scaled by β > 0 .
Revised: Here, Θ denotes the feasible set for θ , and R ( · ) denotes an appropriate regularization term scaled by β > 0 .

Original: We can show that the above problem is well-behaved for certain data distributions X .
Revised: We can show that the above problem is well-behaved for certain data distributions X .

Original: Proposition 1.
Revised: Proposition 1.

Original: If the regularization R and loss function ℓ : Y × Y → [0 , 1] are convex, λ > 0 and
Revised: If the regularization R and loss function ℓ : Y × Y → [0 , 1] are convex, λ > 0 and

Original: X ⊂ R d
Revised: X ⊂ R d

Original: + then the optimization problem in Equation 2 is convex in θ for Θ ⊂ R r
Revised: + then the optimization problem in Equation 2 is convex in θ for Θ ⊂ R r

Original: ( d +1)
Revised: ( d +1)

Original: + .
Revised: + .

Original: Proposition 1 suggests that if the training data is positive (achieved by renormalization), typical loss functions (e.g., cross-entropy or mean squared error) and R such as L p norms are well-suited for optimization.
Revised: Proposition 1 suggests that if the training data is positive (achieved by renormalization), typical loss functions (e.g., cross-entropy or mean squared error) and R such as L p norms are well-suited for optimization.

Original: To ensure convergence, one can use proximal SGD with a small learning rate
Revised: To ensure convergence, one can use proximal SGD with a small learning rate

Original: [Nitanda, 2014].
Revised: [Nitanda, 2014].

Original: In practice, we find that unconstrained SGD also provides comparable performance.
Revised: In practice, we find that unconstrained SGD also provides comparable performance.

Original: 3.2 Improving Polynomials for Learning
Revised: 3.2 Improving Polynomials for Learning

Original: Geometric rescaling .
Revised: Geometric rescaling .

Original: Learning higher-order interactions is tricky as the order of interactions increases: the product of two features is an order of magnitude different from the original, and consequently, higher-order products are progressively disproportionate.
Revised: Learning higher-order interactions is tricky as the order of interactions increases: the product of two features is an order of magnitude different from the original, and consequently, higher-order products are progressively disproportionate.

Original: To mitigate this, we rescale features such that (a) the scale is preserved across terms, and (b) the variance in interactions is captured.
Revised: To mitigate this, we rescale features such that (a) the scale is preserved across terms, and (b) the variance in interactions is captured.

Original: We replace the input x with ˜ x l = sign( x ) · | x | 1 /l for an interaction of order l .
Revised: We replace the input x with ˜ x l = sign( x ) · | x | 1 /l for an interaction of order l .

Original: Specifically,
Revised: Specifically,

Original: P ( x )
Revised: P ( x )

Original: [Equation]
Revised: [Equation]

Original: , x
Revised: , x

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: We denote this model as SPAM-L INEAR .
Revised: We denote this model as SPAM-L INEAR .

Original: We argue that this rescaling for unit-bounded features ensures higher interpretability as well as better learning (since all features are of similar variance).
Revised: We argue that this rescaling for unit-bounded features ensures higher interpretability as well as better learning (since all features are of similar variance).

Original: Note that for order 1, ˜ x 1 = x .
Revised: Note that for order 1, ˜ x 1 = x .

Original: For k ≥ 2 , consider a pairwise interaction between two features x i = 0 .
Revised: For k ≥ 2 , consider a pairwise interaction between two features x i = 0 .

Original: 5 and x j = 0 .
Revised: 5 and x j = 0 .

Original: Naively multiplying the two will give a feature value of x i x j = 0 .(smaller than both x i and x j ), whereas an intuitive value of the feature would be √ x i x j = 0 .
Revised: Naively multiplying the two will give a feature value of x i x j = 0 .(smaller than both x i and x j ), whereas an intuitive value of the feature would be √ x i x j = 0 .

Original: 54 , i.e., the geometric mean.
Revised: 54 , i.e., the geometric mean.

Original: Regarding the reduced variance, observe that unconstrained, the variance
Revised: Regarding the reduced variance, observe that unconstrained, the variance

Original: V ( x i x j ) ≤ V ( x i ) · V ( x j ) and V ( √ x i x j ) ≤ V ( √ x i ) V ( √ x j ) ≤ (cid:112) V ( x i ) · V ( x j ) .
Revised: V ( x i x j ) ≤ V ( x i ) · V ( x j ) and V ( √ x i x j ) ≤ V ( √ x i ) V ( √ x j ) ≤ (cid:112) V ( x i ) · V ( x j ) .

Original: If the features have small variance, e.g., V ( x i ) =
Revised: If the features have small variance, e.g., V ( x i ) =

Original: V ( x j ) =
Revised: V ( x j ) =

Original: 10 − 2 and are uncorrelated, then the first case would provide a much smaller variance for the interaction, whereas the rescaling would preserve variance.
Revised: 10 − 2 and are uncorrelated, then the first case would provide a much smaller variance for the interaction, whereas the rescaling would preserve variance.

Original: Shared bases for multi-class problems .
Revised: Shared bases for multi-class problems .

Original: When we are learning a multi-class classifier, it is observed that learning a unique polynomial for each class creates a large model (with O (2 drC ) weights) when the number of classes C is large, which leads to overfitting and issues with regularization.
Revised: When we are learning a multi-class classifier, it is observed that learning a unique polynomial for each class creates a large model (with O (2 drC ) weights) when the number of classes C is large, which leads to overfitting and issues with regularization.

Original: Instead, we propose sharing bases ( u ) across classes for all higher order terms, and learning classspecific singular values ( λ ) for all higher order terms ( ≥ 2 ) per class.
Revised: Instead, we propose sharing bases ( u ) across classes for all higher order terms, and learning classspecific singular values ( λ ) for all higher order terms ( ≥ 2 ) per class.

Original: For any input x , P ( x ; θ ) =
Revised: For any input x , P ( x ; θ ) =

Original: S OFT M AX { P c ( x ; θ ) }
Revised: S OFT M AX { P c ( x ; θ ) }

Original: Cc =1 , where, for any class c ∈ { 1 , ..., C } ,
Revised: Cc =1 , where, for any class c ∈ { 1 , ..., C } ,

Original: [Equation]
Revised: [Equation]

Original: The terms in green denote weights unique to each class, and terms in red denote weights shared across classes.
Revised: The terms in green denote weights unique to each class, and terms in red denote weights shared across classes.

Original: This reduces the model size to O (( d + r ) C + rd ) from O ( rdC ) .
Revised: This reduces the model size to O (( d + r ) C + rd ) from O ( rdC ) .

Original: We set ℓ as the cross-entropy of the softmax of { P c ( · , θ ) } Cc =1 , as both these operations preserve convexity.
Revised: We set ℓ as the cross-entropy of the softmax of { P c ( · , θ ) } Cc =1 , as both these operations preserve convexity.

Original: Exploring nonlinear input transformations .
Revised: Exploring nonlinear input transformations .

Original: Motivated by non-linear GAMs (e.g., GA 2 M [Lou et al., 2013] and NAM [Agarwal et al., 2021]), we modify our low-rank decomposed algorithm by replacing ˜ x l for each l with feature-wise non-linearities.
Revised: Motivated by non-linear GAMs (e.g., GA 2 M [Lou et al., 2013] and NAM [Agarwal et al., 2021]), we modify our low-rank decomposed algorithm by replacing ˜ x l for each l with feature-wise non-linearities.

Original: We learn a non-linear SPAM P ( · ; θ ) as,
Revised: We learn a non-linear SPAM P ( · ; θ ) as,

Original: P ( x ; θ ) =
Revised: P ( x ; θ ) =

Original: b + ⟨ u 1 , F 1 ( x ) ⟩ +
Revised: b + ⟨ u 1 , F 1 ( x ) ⟩ +

Original: [Equation]
Revised: [Equation]

Original: 2 (cid:88)
Revised: 2 (cid:88)

Original: i
Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: ⟩ 2 + ...
Revised: ⟩ 2 + ...

Original: [Equation]
Revised: [Equation]

Original: Here, the function F i ( x ) =
Revised: Here, the function F i ( x ) =

Original: [ f i 1 ( x 1 ) , f i 2 ( x 2 ) , ..., f id ( x d )] is a Neural Additive Model (NAM)
Revised: [ f i 1 ( x 1 ) , f i 2 ( x 2 ) , ..., f id ( x d )] is a Neural Additive Model (NAM)

Original: [Agarwal et al., 2021].
Revised: [Agarwal et al., 2021].

Original: We denote this model as SPAM-N EURAL .
Revised: We denote this model as SPAM-N EURAL .

Original: For more details on the NAM we use, please refer to the Appendix Section C. Note that the resulting model is still interpretable, as the interaction between features x i and x j , e.g., will be given by ( (cid:80) r 2 k =1 λ 2 k u 2 ki u 2 kj ) · f 2 i ( x i ) f 2 j ( x j ) instead of the previous ( (cid:80) r 2 k =1 λ 2 k u 2 ki u 2 kj ) · √ x i · x j for typical SPAM.
Revised: For more details on the NAM we use, please refer to the Appendix Section C. Note that the resulting model is still interpretable, as the interaction between features x i and x j , e.g., will be given by ( (cid:80) r 2 k =1 λ 2 k u 2 ki u 2 kj ) · f 2 i ( x i ) f 2 j ( x j ) instead of the previous ( (cid:80) r 2 k =1 λ 2 k u 2 ki u 2 kj ) · √ x i · x j for typical SPAM.

Original: Dropout for bases via λ .
Revised: Dropout for bases via λ .

Original: To capture non-overlapping feature correlations, we introduce a basis dropout by setting the contribution from any particular basis to zero at random via the singular values λ .
Revised: To capture non-overlapping feature correlations, we introduce a basis dropout by setting the contribution from any particular basis to zero at random via the singular values λ .

Original: Specifically, observe that the contribution of any basis-singular value pair ( u , λ ) at order l is λ ⟨ u , x ⟩ l .
Revised: Specifically, observe that the contribution of any basis-singular value pair ( u , λ ) at order l is λ ⟨ u , x ⟩ l .

Original: We apply dropout to λ to ensure that the network learns robust basis directions.
Revised: We apply dropout to λ to ensure that the network learns robust basis directions.

Original: 3.3 Approximation and Learning-Theoretic Guarantees
Revised: 3.3 Approximation and Learning-Theoretic Guarantees

Original: We present some learning-theoretic guarantees for SPAM.
Revised: We present some learning-theoretic guarantees for SPAM.

Original: Proofs are deferred to Appendix Section A.
Revised: Proofs are deferred to Appendix Section A.

Original: Proposition 2.
Revised: Proposition 2.

Original: Let f be a continuous real-valued function over a compact set X .
Revised: Let f be a continuous real-valued function over a compact set X .

Original: For any threshold ϵ > 0 , there exists a low-rank polynomial P : X → R such that sup x ∈X | f ( x ) − P ( x ) | < ϵ .
Revised: For any threshold ϵ > 0 , there exists a low-rank polynomial P : X → R such that sup x ∈X | f ( x ) − P ( x ) | < ϵ .

Original: The above result demonstrates that asymptotically, decomposed polynomials are universal function approximators.
Revised: The above result demonstrates that asymptotically, decomposed polynomials are universal function approximators.

Original: Next, we present an assumption on the data-generating distribution as well as a novel excess risk bound that characterizes learning with polynomials more precisely (non-asymptotic).
Revised: Next, we present an assumption on the data-generating distribution as well as a novel excess risk bound that characterizes learning with polynomials more precisely (non-asymptotic).

Original: Assumption 1 (Exponential Spectral Decay of Polynomial Approximators) .
Revised: Assumption 1 (Exponential Spectral Decay of Polynomial Approximators) .

Original: Let P k denote the family of all polynomials of degree at most k , and let P ⋆,k denote the optimal polynomial in P k , i.e.,
Revised: Let P k denote the family of all polynomials of degree at most k , and let P ⋆,k denote the optimal polynomial in P k , i.e.,

Original: P ⋆,k = arg min
Revised: P ⋆,k = arg min

Original: [Equation]
Revised: [Equation]

Original: E (
Revised: E (

Original: [Equation]
Revised: [Equation]

Original: ,y ) ∼ P [ ℓ ( P ( x ) , y )] .
Revised: ,y ) ∼ P [ ℓ ( P ( x ) , y )] .

Original: We assume that ∀ k , P ⋆,k admits a decomposition as described in Equation 1 such that, for all 1 ≤ l ≤ k , there exist absolute constants C 1 < 1 and
Revised: We assume that ∀ k , P ⋆,k admits a decomposition as described in Equation 1 such that, for all 1 ≤ l ≤ k , there exist absolute constants C 1 < 1 and

Original: C 

Revised: C

Original: = O (1) such that | λ lj | ≤ C 1 · exp ( − C 2 · j γ ) for each j ≥ 1 and l ∈ [1 , k ] .
Revised: = O (1) such that | λ lj | ≤ C 1 · exp ( − C 2 · j γ ) for each j ≥ 1 and l ∈ [1 , k ] .

Original: This condition is essentially bounding the “smoothness” of the nearest polynomial approximator of f , i.e., implying that only a few degrees of freedom suffice to approximate f accurately.
Revised: This condition is essentially bounding the “smoothness” of the nearest polynomial approximator of f , i.e., implying that only a few degrees of freedom suffice to approximate f accurately.

Original: Assumptionprovides a soft threshold for singular value decay.
Revised: Assumptionprovides a soft threshold for singular value decay.

Original: One can also replace the exponential with a slower “polynomial” decay with similar results, and we discuss this in the Appendix Section A. We can now present our error bound for L 2 regularized models (see Appendix for L 1 -regularized models).
Revised: One can also replace the exponential with a slower “polynomial” decay with similar results, and we discuss this in the Appendix Section A. We can now present our error bound for L 2 regularized models (see Appendix for L 1 -regularized models).

Original: Theorem 1.
Revised: Theorem 1.

Original: Let ℓ be 1-Lipschitz, δ ∈ (0 , 1] , the generalization error for the optimal degree- k polynomial P ⋆,k be E ( P ⋆ , k ) and the generalization error for the empirical risk minimizer SPAM (cid:98) P r ,k , i.e., (cid:98) P r ,k = arg min P (cid:80) ni
Revised: Let ℓ be 1-Lipschitz, δ ∈ (0 , 1] , the generalization error for the optimal degree- k polynomial P ⋆,k be E ( P ⋆ , k ) and the generalization error for the empirical risk minimizer SPAM (cid:98) P r ,k , i.e., (cid:98) P r ,k = arg min P (cid:80) ni

Original: =1 ℓ ( P ( x i ) , y i ) with ranks r = [1 , r 2 , ..., r k ] be E ( (cid:98) P r ,k ) .
Revised: =1 ℓ ( P ( x i ) , y i ) with ranks r = [1 , r 2 , ..., r k ] be E ( (cid:98) P r ,k ) .

Original: Then, if
Revised: Then, if

Original: P satisfies Assumption 1 with constants C 1 , C 2 and γ , we have for L 2 − regularized ERM, where ∥ u ij ∥ 2 ≤ B u, 2 ∀ i ∈
Revised: P satisfies Assumption 1 with constants C 1 , C 2 and γ , we have for L 2 − regularized ERM, where ∥ u ij ∥ 2 ≤ B u, 2 ∀ i ∈

Original: [ k ] , j ∈ [ r i ] , and ∥ λ ∥ 2 ≤ B λ, 2 where λ = {{ λ ij } r i j =1 } ki =1 , with probability at least 1 − δ ,
Revised: [ k ] , j ∈ [ r i ] , and ∥ λ ∥ 2 ≤ B λ, 2 where λ = {{ λ ij } r i j =1 } ki =1 , with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: The above result demonstrates an expected scaling of the generalization error, matching the bounds obtained for L 1 (see Appendix Section A) and L 2 − regularized linear classifiers
Revised: The above result demonstrates an expected scaling of the generalization error, matching the bounds obtained for L 1 (see Appendix Section A) and L 2 − regularized linear classifiers

Original: [Wainwright, 2019].
Revised: [Wainwright, 2019].

Original: We that the rank r = ∥
Revised: We that the rank r = ∥

Original: r ∥ 1 has an O ( r ) dependence on the Rademacher complexity.
Revised: r ∥ 1 has an O ( r ) dependence on the Rademacher complexity.

Original: This highlights a trade-off between optimization and generalization: a larger r gives a lower approximation error (smaller second term), but a larger model complexity (larger first term).
Revised: This highlights a trade-off between optimization and generalization: a larger r gives a lower approximation error (smaller second term), but a larger model complexity (larger first term).

Original: Observe, however, that even when r = Ω((log d ) p ) for some p ≥ 1 , the approximation error (second term) diminishes as o ( 1 d pγ ) .
Revised: Observe, however, that even when r = Ω((log d ) p ) for some p ≥ 1 , the approximation error (second term) diminishes as o ( 1 d pγ ) .

Original: This suggests that in practice, one only needs a cumulative rank (poly)logarithmic in d .
Revised: This suggests that in practice, one only needs a cumulative rank (poly)logarithmic in d .

Original: The above result also provides a non-asymptotic variant of Proposition 2; as n, r → ∞ ,
Revised: The above result also provides a non-asymptotic variant of Proposition 2; as n, r → ∞ ,

Original: E ( (cid:98) P r ,k )
Revised: E ( (cid:98) P r ,k )

Original: → E ( P ⋆,k ) + .
Revised: → E ( P ⋆,k ) + .

Original:  Offline Experiments
Revised:  Offline Experiments

Original: Here we evaluate SPAM against a set of benchmark algorithms on both classification and regression problems.
Revised: Here we evaluate SPAM against a set of benchmark algorithms on both classification and regression problems.

Original: The baseline approaches we consider are (see Appendix Section C for precise details):
Revised: The baseline approaches we consider are (see Appendix Section C for precise details):

Original: • Deep Neural Networks (DNN) :
Revised: • Deep Neural Networks (DNN) :

Original: These are standard multi-layer fully-connected neural networks included to demonstrate an upper-bound on performance.
Revised: These are standard multi-layer fully-connected neural networks included to demonstrate an upper-bound on performance.

Original: These are explained via the perturbationbased LIME [Ribeiro et al., 2016] and SHAP [Lundberg and Lee, 2017] methods in Section 5.
Revised: These are explained via the perturbationbased LIME [Ribeiro et al., 2016] and SHAP [Lundberg and Lee, 2017] methods in Section 5.

Original: • Linear / Logistic Regression : These serve as our baseline interpretable models.
Revised: • Linear / Logistic Regression : These serve as our baseline interpretable models.

Original: We learn both Land L 2 − regularized models on unary and pairwise features ( (cid:0) d 2 (cid:1) features) with minibatch SGD.
Revised: We learn both Land L 2 − regularized models on unary and pairwise features ( (cid:0) d 2 (cid:1) features) with minibatch SGD.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: • Gradient Boosted Trees (XGBoost) : We use the library xgboost .
Revised: • Gradient Boosted Trees (XGBoost) : We use the library xgboost .

Original: This baseline is mainly to compare accuracy, as the number of trees required are typically large and are hence uninterpretable.
Revised: This baseline is mainly to compare accuracy, as the number of trees required are typically large and are hence uninterpretable.

Original: • Explainable Boosting Machines (EBMs)
Revised: • Explainable Boosting Machines (EBMs)

Original: [Lou et al., 2013]: EBMs use millions of shallow bagged trees operating on each feature at a time.
Revised: [Lou et al., 2013]: EBMs use millions of shallow bagged trees operating on each feature at a time.

Original: Note that this approach is not scalable to datasets with many features or multi-class problems.
Revised: Note that this approach is not scalable to datasets with many features or multi-class problems.

Original: We report scores on the datasets where we successfully trained EBMs without sophisticated engineering, using the interpretml library [Nori et al., 2019].
Revised: We report scores on the datasets where we successfully trained EBMs without sophisticated engineering, using the interpretml library [Nori et al., 2019].

Original: • Neural Additive Models (NAMs)
Revised: • Neural Additive Models (NAMs)

Original: [Agarwal et al., 2021]: These models are neural network extensions of prior EBMs.
Revised: [Agarwal et al., 2021]: These models are neural network extensions of prior EBMs.

Original: Note that this method also does not scale to some datasets.
Revised: Note that this method also does not scale to some datasets.

Original: Training Setup .
Revised: Training Setup .

Original: SPAM is implemented by learning L 1 /L 2 -regularized variants by minibatch SGD implemented in PyTorch.
Revised: SPAM is implemented by learning L 1 /L 2 -regularized variants by minibatch SGD implemented in PyTorch.

Original: For regression tasks, we measure the root mean squared error (RMSE).
Revised: For regression tasks, we measure the root mean squared error (RMSE).

Original: For binary classification, we report the area under the ROC (AUROC), for multi-class classification, we report the top-1 accuracy (Acc@1), and finally, for object detection, we report mean Average
Revised: For binary classification, we report the area under the ROC (AUROC), for multi-class classification, we report the top-1 accuracy (Acc@1), and finally, for object detection, we report mean Average

Original: Precision (mAP).
Revised: Precision (mAP).

Original: We tune hyperparameters via random sampling approach over a grid.
Revised: We tune hyperparameters via random sampling approach over a grid.

Original: Note that for all experiments, both NAM and SPAM-N EURAL have identical MLP structures to ensure identical approximation power.
Revised: Note that for all experiments, both NAM and SPAM-N EURAL have identical MLP structures to ensure identical approximation power.

Original: For definitions of metrics and hyperparameter ranges, see Appendix Section C.
Revised: For definitions of metrics and hyperparameter ranges, see Appendix Section C.

Original: 4.1 Measuring Benchmark Performance
Revised: 4.1 Measuring Benchmark Performance

Original: We select tasks to explore a variety of settings from regression to multi-class classification, and also explore different dataset scales, from a few hundred samples and tens of features to 100K-scale datasets (both in the number of samples and data dimensionality), while ensuring that the features are interpretable.
Revised: We select tasks to explore a variety of settings from regression to multi-class classification, and also explore different dataset scales, from a few hundred samples and tens of features to 100K-scale datasets (both in the number of samples and data dimensionality), while ensuring that the features are interpretable.

Original: Our datasets are summarized in Table 2.
Revised: Our datasets are summarized in Table 2.

Original: Please see Appendix Section C.1 for details.
Revised: Please see Appendix Section C.1 for details.

Original: For all datasets with no defined train-val-test split, we use a fixed random sample of 70% of the data for training, 10% for validation and 20% for testing.
Revised: For all datasets with no defined train-val-test split, we use a fixed random sample of 70% of the data for training, 10% for validation and 20% for testing.

Original: For the 20 Newsgroups dataset, we split the pre-defined training split 7:1 for training and validation, respectively.
Revised: For the 20 Newsgroups dataset, we split the pre-defined training split 7:1 for training and validation, respectively.

Original: In an effort to scale interpretable approaches beyond tabular datasets, we consider benchmark problems using the “Independent Concept Bottleneck” framework of Koh et al.
Revised: In an effort to scale interpretable approaches beyond tabular datasets, we consider benchmark problems using the “Independent Concept Bottleneck” framework of Koh et al.

Original: [2020] for image
Revised: [2020] for image

Original: [Figure]
Revised: [Figure]

Original: classification and object detection.
Revised: classification and object detection.

Original: We use a convolutional neural network (CNN) backbone ResNet

Revised: We use a convolutional neural network (CNN) backbone ResNet

Original: [He et al., 2016] that is trained to predict interpretable concepts (e
Revised: [He et al., 2016] that is trained to predict interpretable concepts (e

Original: [Figure]
Revised: [Figure]

Original: g., parts of object) from images.
Revised: g., parts of object) from images.

Original: After training the backbone , we extract predicted concepts for all inputs
Revised: After training the backbone , we extract predicted concepts for all inputs

Original: [Figure]
Revised: [Figure]

Original: and learn an interpretable classifier head from these concepts.
Revised: and learn an interpretable classifier head from these concepts.

Original: In experiments, the backbone remains identical for all comparisons
Revised: In experiments, the backbone remains identical for all comparisons

Original: [Figure]
Revised: [Figure]

Original: and we only compare the head classifiers (see Appendix Section C.3 for more details).
Revised: and we only compare the head classifiers (see Appendix Section C.3 for more details).

Original: We select three datasets for evaluation in the concept bottleneck setting:
Revised: We select three datasets for evaluation in the concept bottleneck setting:

Original: Caltech-UCSD Birds (CUB-200)
Revised: Caltech-UCSD Birds (CUB-200)

Original: [Wah et al., 2011]: This is a fine-grained visual categorization dataset where one has to classify between 200 species of birds.
Revised: [Wah et al., 2011]: This is a fine-grained visual categorization dataset where one has to classify between 200 species of birds.

Original: The interpretable concepts are 278 binary bird attributes, e.g., the shape of the beak, the color of the wings, etc.
Revised: The interpretable concepts are 278 binary bird attributes, e.g., the shape of the beak, the color of the wings, etc.

Original: The dataset has 5,994 training and 5,794 validation images.
Revised: The dataset has 5,994 training and 5,794 validation images.

Original: See Appendix Section C.3.1 for more details.
Revised: See Appendix Section C.3.1 for more details.

Original: iNaturalist “Birds” [Van Horn et al., 2018, 2021]: iNaturalist can be thought of as a larger version of the previous dataset.
Revised: iNaturalist “Birds” [Van Horn et al., 2018, 2021]: iNaturalist can be thought of as a larger version of the previous dataset.

Original: We only select the “Birds” super-category of samples, which has 414, training and 14,860 validation instances across 1,486 classes.
Revised: We only select the “Birds” super-category of samples, which has 414, training and 14,860 validation instances across 1,486 classes.

Original: Since the iNaturalist dataset does not contain dense part and attribute annotations, we use the predicted concepts from the CUBbackbone model extracted on all samples.
Revised: Since the iNaturalist dataset does not contain dense part and attribute annotations, we use the predicted concepts from the CUBbackbone model extracted on all samples.

Original: Common Objects Dataset (CO114, Proprietary): Here we consider, for the first time, a concept bottleneck model for object detection.
Revised: Common Objects Dataset (CO114, Proprietary): Here we consider, for the first time, a concept bottleneck model for object detection.

Original: We construct a dataset involving common household objects (e.g., bed, saucer, tables, etc.) with their bounding box annotations.
Revised: We construct a dataset involving common household objects (e.g., bed, saucer, tables, etc.) with their bounding box annotations.

Original: For each bounding box, we collect 2,618 interpretable annotations (e.g., parts, color, etc.).
Revised: For each bounding box, we collect 2,618 interpretable annotations (e.g., parts, color, etc.).

Original: The dataset has 2,645,488 training and 58,525 validation samples across 115 classes with 2,618 interpretable concepts.
Revised: The dataset has 2,645,488 training and 58,525 validation samples across 115 classes with 2,618 interpretable concepts.

Original: We report the mean Average Precision (mAP) metric.
Revised: We report the mean Average Precision (mAP) metric.

Original: For more details please refer to Appendix Section C.3.2.
Revised: For more details please refer to Appendix Section C.3.2.

Original: Our results are summarized in Table 1.
Revised: Our results are summarized in Table 1.

Original: There are four main takeaways from the results.
Revised: There are four main takeaways from the results.

Original: First, observe that both SPAM-L INEAR and SPAM-N EURAL comfortably outperform their prior interpretable counterparts on all datasets (e.g., Order 2 SPAM-L INEAR outperforms all linear methods – even the full rank pairwise model), and SPAM-N EURAL comfortably outperforms all non-linear baselines.
Revised: First, observe that both SPAM-L INEAR and SPAM-N EURAL comfortably outperform their prior interpretable counterparts on all datasets (e.g., Order 2 SPAM-L INEAR outperforms all linear methods – even the full rank pairwise model), and SPAM-N EURAL comfortably outperforms all non-linear baselines.

Original: Next, observe that in all datasets but CoverType, second degree interactions suffice to match or even outperform DNN performance.
Revised: Next, observe that in all datasets but CoverType, second degree interactions suffice to match or even outperform DNN performance.

Original: We will discuss CoverType in detail in the next paragraph.
Revised: We will discuss CoverType in detail in the next paragraph.

Original: Thirdly, observe that SPAM models are as scalable as the black-box approaches, whereas prior work (e.g.,
Revised: Thirdly, observe that SPAM models are as scalable as the black-box approaches, whereas prior work (e.g.,

Original: NAM and EBMs) are not.
Revised: NAM and EBMs) are not.

Original: In the case of NAM, we were unable to scale due to the sheer number of parameters required to model large datasets, e.g., Newsgroups ( ≈ 900M parameters), and for EBMs, the training time and memory requirement increased dramatically for large datasets.
Revised: In the case of NAM, we were unable to scale due to the sheer number of parameters required to model large datasets, e.g., Newsgroups ( ≈ 900M parameters), and for EBMs, the training time and memory requirement increased dramatically for large datasets.

Original: Furthermore,
Revised: Furthermore,

Original: EB 2 Ms do not even support multi-class problems.
Revised: EB 2 Ms do not even support multi-class problems.

Original: Finally, observe that for many problems, we do not require non-linear feature transformations once we model feature interactions: e.g., iNat and CUB.
Revised: Finally, observe that for many problems, we do not require non-linear feature transformations once we model feature interactions: e.g., iNat and CUB.

Original: CoverType Dataset.
Revised: CoverType Dataset.

Original: XGBoost and DNNs perform substantially better on CoverType compared to linear or neural SPAM.
Revised: XGBoost and DNNs perform substantially better on CoverType compared to linear or neural SPAM.

Original: On analysis, we found that existing SPAM models underfit on CoverType, and hence we increased the total parameters for SPAM-N EURAL via subnets , identical to NAMs [Agarwal et al., 2021], where each feature is mapped to s non-linear features (as opposed to 1 originally).
Revised: On analysis, we found that existing SPAM models underfit on CoverType, and hence we increased the total parameters for SPAM-N EURAL via subnets , identical to NAMs [Agarwal et al., 2021], where each feature is mapped to s non-linear features (as opposed to 1 originally).

Original: With s = 8 subnets, the performance of SPAM-N EURAL (order 3) improves to 0 .
Revised: With s = 8 subnets, the performance of SPAM-N EURAL (order 3) improves to 0 .

Original: In comparison,
Revised: In comparison,

Original: NAMs with 8 subnets provides a lower accuracy of 0 . 7551 .
Revised: NAMs with 8 subnets provides a lower accuracy of 0 . 7551 .

Original: 4.2 Ablation Studies
Revised: 4.2 Ablation Studies

Original: Rank and Degree of Interactions .
Revised: Rank and Degree of Interactions .

Original: By Proposition 2, it is natural to expect the approximation quality to increase with the degree k and rank r , i.e., as we introduce more higher-order interactions
Revised: By Proposition 2, it is natural to expect the approximation quality to increase with the degree k and rank r , i.e., as we introduce more higher-order interactions

Original: [Table]
Revised: [Table]

Original: between features.
Revised: between features.

Original: We ablate the degree k on the tabular benchmark CoverType and concept bottleneck benchmark iNaturalist (Birds) , as summarized in Figure 1A. We observe, as expected, that increasing the degree k leads to moderate improvements beyond k ≥ 3 , but with diminishing returns.
Revised: We ablate the degree k on the tabular benchmark CoverType and concept bottleneck benchmark iNaturalist (Birds) , as summarized in Figure 1A. We observe, as expected, that increasing the degree k leads to moderate improvements beyond k ≥ 3 , but with diminishing returns.

Original: Similarly, we examine the effect of the cumulative rank r on performance on the Newsgroups and iNaturalist (Birds) datasets (Figure 1B); we observe that performance is sufficiently insensitive to r , and plateaus after a while.
Revised: Similarly, we examine the effect of the cumulative rank r on performance on the Newsgroups and iNaturalist (Birds) datasets (Figure 1B); we observe that performance is sufficiently insensitive to r , and plateaus after a while.

Original: We imagine that as r increases, model complexity will dominate and performance will likely begin to decrease, matching the full-rank performance at r
Revised: We imagine that as r increases, model complexity will dominate and performance will likely begin to decrease, matching the full-rank performance at r

Original: = O
Revised: = O

Original: ( d 2 ) for pairwise models.
Revised: ( d 2 ) for pairwise models.

Original: Sparsity in Higher-Order Relationships.
Revised: Sparsity in Higher-Order Relationships.

Original: A requirement for interpretability is to ensure that the learned models can be explained with a few interpretable concepts.
Revised: A requirement for interpretability is to ensure that the learned models can be explained with a few interpretable concepts.

Original: Since the number of higher-order combinations increases with k , we examine sparsity to limit the number of active feature assignments.
Revised: Since the number of higher-order combinations increases with k , we examine sparsity to limit the number of active feature assignments.

Original: We penalize the objective in Equation 2 with a regularization term that inhibits dense feature selection.
Revised: We penalize the objective in Equation 2 with a regularization term that inhibits dense feature selection.

Original: If U = {{ u li } r l i =1 } kl =1 denotes all the basis vectors in matrix form, we add the penalty R ( θ ) ≜
Revised: If U = {{ u li } r l i =1 } kl =1 denotes all the basis vectors in matrix form, we add the penalty R ( θ ) ≜

Original: ∥ U ∥ 1 .
Revised: ∥ U ∥ 1 .

Original: This ensures that every basis u only captures limited interactions between features, and hence the overall complexity of the model is limited.
Revised: This ensures that every basis u only captures limited interactions between features, and hence the overall complexity of the model is limited.

Original: We examine accuracy as a function of the fraction of non-zero pairwise interactions for a degree 2, rank 800 SPAM on CUB-200 in Figure 1C, and find that only 6% of the possible interactions suffice to obtain competitive performance.
Revised: We examine accuracy as a function of the fraction of non-zero pairwise interactions for a degree 2, rank 800 SPAM on CUB-200 in Figure 1C, and find that only 6% of the possible interactions suffice to obtain competitive performance.

Original: Examining Assumption 1 (Spectral Decay).
Revised: Examining Assumption 1 (Spectral Decay).

Original: The spectral decay assumption presented in Assumption 1 is crucial to obtain reasonable generalization bounds (Theorem 1).
Revised: The spectral decay assumption presented in Assumption 1 is crucial to obtain reasonable generalization bounds (Theorem 1).

Original: To test this assumption, we examine the spectra of a SPAM-L INEAR (Order 2) model for all 200 of CUB-200 classes.
Revised: To test this assumption, we examine the spectra of a SPAM-L INEAR (Order 2) model for all 200 of CUB-200 classes.

Original: These are depicted in Figure 2A.
Revised: These are depicted in Figure 2A.

Original: Furthermore, we fit an exponential model on the decay itself, and obtain (in line with Assumption 1) γ = 3 , C 1 = 0 .
Revised: Furthermore, we fit an exponential model on the decay itself, and obtain (in line with Assumption 1) γ = 3 , C 1 = 0 .

Original: 54 and C 2 = 0 .
Revised: 54 and C 2 = 0 .

Original: 006 , all in accordance with the assumption.
Revised: 006 , all in accordance with the assumption.

Original: 4.3 Additional Experiments 4.3.1 Comparisons on Common Benchmarks
Revised: 4.3 Additional Experiments 4.3.1 Comparisons on Common Benchmarks

Original: In addition to the 7 datasets we considered earlier, we evaluate on 9 further benchmark datasets commonly used in the interpretability literature.
Revised: In addition to the 7 datasets we considered earlier, we evaluate on 9 further benchmark datasets commonly used in the interpretability literature.

Original: We evaluate on the MIMIC2, Credit and COMPAS datasets as presented in Agarwal et al.
Revised: We evaluate on the MIMIC2, Credit and COMPAS datasets as presented in Agarwal et al.

Original: [2021], and the Click, Epsilon, Higgs, Microsoft, Yahoo and
Revised: [2021], and the Click, Epsilon, Higgs, Microsoft, Yahoo and

Original: Year datasets from Chang et al.
Revised: Year datasets from Chang et al.

Original: The result of this is summarized in Table 4.
Revised: The result of this is summarized in Table 4.

Original: We observe that
Revised: We observe that

Original: SPAM models are competitive with prior work, and outperforming prior work on a number of tasks (the best interpretable model is in bold ).
Revised: SPAM models are competitive with prior work, and outperforming prior work on a number of tasks (the best interpretable model is in bold ).

Original: 4.3.2 Runtime Evaluation
Revised: 4.3.2 Runtime Evaluation

Original: We provide a comparison of SPAM runtimes on 4 different datasets to establish their scalability.
Revised: We provide a comparison of SPAM runtimes on 4 different datasets to establish their scalability.

Original: We consider the California Housing (CH, Pace and Barry [1997], 8 features), FICO HELOC (FICO, FICO [2018], 23 features), CoverType (CovType, Blackard and Dean [1999], 54 features) and 20 Newsgroups (News, Lang [1995], 146,016 features), and evaluate the throughput of the optimal Linear and Neural SPAM models against Neural Additive Models (NAM, Agarwal et al.
Revised: We consider the California Housing (CH, Pace and Barry [1997], 8 features), FICO HELOC (FICO, FICO [2018], 23 features), CoverType (CovType, Blackard and Dean [1999], 54 features) and 20 Newsgroups (News, Lang [1995], 146,016 features), and evaluate the throughput of the optimal Linear and Neural SPAM models against Neural Additive Models (NAM, Agarwal et al.

Original: [2021]) and MLPs.
Revised: [2021]) and MLPs.

Original: Table 3 describes the throughput, where we observe that Linear SPAM is more efficient than MLPs, and NeuralSPAM is orders of magnitude faster than NAM, which does not scale to higher orders.
Revised: Table 3 describes the throughput, where we observe that Linear SPAM is more efficient than MLPs, and NeuralSPAM is orders of magnitude faster than NAM, which does not scale to higher orders.

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  Human Subject Evaluations
Revised:  Human Subject Evaluations

Original: Experiment Methodology .
Revised: Experiment Methodology .

Original: We now evaluate how well explanations from SPAM fare in a practical setting with non-interpretable benchmarks such as black-box models equipped with post-hoc explanations.
Revised: We now evaluate how well explanations from SPAM fare in a practical setting with non-interpretable benchmarks such as black-box models equipped with post-hoc explanations.

Original: Our objective is to mimic a practical setting, where the model user must decide between a fully-interpretable model vs. a black-box classifier with post-hoc explainability.
Revised: Our objective is to mimic a practical setting, where the model user must decide between a fully-interpretable model vs. a black-box classifier with post-hoc explainability.

Original: Our experiment design is a Prediction Task [Hoffman et al., 2018, Muramatsu and Pratt, 2001], where the objective for the users is to guess what the model predicted given an explanation.
Revised: Our experiment design is a Prediction Task [Hoffman et al., 2018, Muramatsu and Pratt, 2001], where the objective for the users is to guess what the model predicted given an explanation.

Original: The motivation of such a design is to ascertain both the faithfulness and interpretability of the model explanations.
Revised: The motivation of such a design is to ascertain both the faithfulness and interpretability of the model explanations.

Original: Computing Explanations .
Revised: Computing Explanations .

Original: We first compute the feature importances for any class c , as the contribution of any specific feature in the logit for class c (for binary classification or regression, there is only one class).
Revised: We first compute the feature importances for any class c , as the contribution of any specific feature in the logit for class c (for binary classification or regression, there is only one class).

Original: For the Logisitc Regression model, we simply use the contribution of each feature to the prediction.
Revised: For the Logisitc Regression model, we simply use the contribution of each feature to the prediction.

Original: For SPAM-Linear (Order 2), we still use each feature’s contribution, but features now can be unary ( x i ) or pairwise ( x i and x j ).
Revised: For SPAM-Linear (Order 2), we still use each feature’s contribution, but features now can be unary ( x i ) or pairwise ( x i and x j ).

Original: As black-box post-hoc baselines, we consider DNNs with
Revised: As black-box post-hoc baselines, we consider DNNs with

Original: LIME [Ribeiro et al., 2016] and KernelSHAP [Lundberg and Lee, 2017] to generate post-hoc linear explanations.
Revised: LIME [Ribeiro et al., 2016] and KernelSHAP [Lundberg and Lee, 2017] to generate post-hoc linear explanations.

Original: See Table 5 for the formulation of importance for the models we use in experiments.
Revised: See Table 5 for the formulation of importance for the models we use in experiments.

Original: Experiment Design We conduct a separate experiment for each model M with explanations of length E by selecting E most important features.
Revised: Experiment Design We conduct a separate experiment for each model M with explanations of length E by selecting E most important features.

Original: Each such experiment runs in two phasestraining and testing .
Revised: Each such experiment runs in two phasestraining and testing .

Original: In the training phase, participants are shown 8 images of birds and their corresponding explanations to develop their mental model.
Revised: In the training phase, participants are shown 8 images of birds and their corresponding explanations to develop their mental model.

Original: Four of these images have been predicted as class A by the model, and the remaining are predicted as class B .
Revised: Four of these images have been predicted as class A by the model, and the remaining are predicted as class B .

Original: They then move on to the testing phase, where they are successively shown the explanations for 5 unseen images (which the model could have predicted as either A or B ), and the users must answer “given the explanation, which class (of A or B ) did the model predict?” .
Revised: They then move on to the testing phase, where they are successively shown the explanations for 5 unseen images (which the model could have predicted as either A or B ), and the users must answer “given the explanation, which class (of A or B ) did the model predict?” .

Original: If they desire, the users can move back to the training phase at any time to revise model explanations.
Revised: If they desire, the users can move back to the training phase at any time to revise model explanations.

Original: We do not show the corresponding images in the test phase, as we want the user feedback to rely solely on the faithfulness of explanations.
Revised: We do not show the corresponding images in the test phase, as we want the user feedback to rely solely on the faithfulness of explanations.

Original: We measure the mean user accuracy on predicting model decisions.
Revised: We measure the mean user accuracy on predicting model decisions.

Original: For more details on the interface, please see Appendix Section D.1.
Revised: For more details on the interface, please see Appendix Section D.1.

Original: For each experiment, we follow an independentsubjects design, and gathered a total of 150 participants per experiment.
Revised: For each experiment, we follow an independentsubjects design, and gathered a total of 150 participants per experiment.

Original: Each task (corresponding to one model-explanation pair) lasted 3 minutes.
Revised: Each task (corresponding to one model-explanation pair) lasted 3 minutes.

Original: The participants were compensated with $ 0.
Revised: The participants were compensated with $ 0.

Original: USD ( $ 15/hour), and all experiments were run on Amazon Mechanical Turk (AMT, Buhrmester et al.
Revised: USD ( $ 15/hour), and all experiments were run on Amazon Mechanical Turk (AMT, Buhrmester et al.

Original: [2016]) with a total cost of $ 3000.
Revised: [2016]) with a total cost of $ 3000.

Original: To remove poorly performing participants, we only select those that get the first decision correct, and compute the mean user accuracy using the remaining 4 images.
Revised: To remove poorly performing participants, we only select those that get the first decision correct, and compute the mean user accuracy using the remaining 4 images.

Original: Study #1: Comparing Black-Box and Transparent Explanations .
Revised: Study #1: Comparing Black-Box and Transparent Explanations .

Original: We compare different interpretable models with a fixed explanation length of E = 7 .
Revised: We compare different interpretable models with a fixed explanation length of E = 7 .

Original: Our objective is to assess the interpretability of pairwise interactions compared to black-box and linear explanations, and to re-answer whether it was even necessary to have fully-interpretable models compared to black-box models with post-hoc explanations.
Revised: Our objective is to assess the interpretability of pairwise interactions compared to black-box and linear explanations, and to re-answer whether it was even necessary to have fully-interpretable models compared to black-box models with post-hoc explanations.

Original: We used samples from the 5 pairs of classes, that is A and B ), from CUB-200 dataset (see Section D.1 for images).
Revised: We used samples from the 5 pairs of classes, that is A and B ), from CUB-200 dataset (see Section D.1 for images).

Original: The results from this experiment are summarized in Figure 2B. Observe that with SPAM, the mean user accuracy (mUA) is substantially higher (0.71) compared to both linear (0.67) and post-hoc LIME (0.65).
Revised: The results from this experiment are summarized in Figure 2B. Observe that with SPAM, the mean user accuracy (mUA) is substantially higher (0.71) compared to both linear (0.67) and post-hoc LIME (0.65).

Original: We would like to remark that even for E = 7 , SPAM has an
Revised: We would like to remark that even for E = 7 , SPAM has an

Original: [Figure]
Revised: [Figure]

Original: |
Revised: |

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: average of 3 . 1 pairwise terms in each explanation, and therefore the higher-order terms are substantial.
Revised: average of 3 . 1 pairwise terms in each explanation, and therefore the higher-order terms are substantial.

Original: A one-sided t-test [Freund and Simon, 1967] provides a significance ( p − value) of 2 .
Revised: A one-sided t-test [Freund and Simon, 1967] provides a significance ( p − value) of 2 .

Original: 29 × 10 − 4 and a test statistic of 3 . 52 for the SPAM mUA being higher than LIME, and correspondingly for Linear, we get a significance ( p − value) of 0 . 03801 and a test statistic of 1 . 666 .
Revised: 29 × 10 − 4 and a test statistic of 3 . 52 for the SPAM mUA being higher than LIME, and correspondingly for Linear, we get a significance ( p − value) of 0 . 03801 and a test statistic of 1 . 666 .

Original: Hence SPAM’s improvements in interpretability are statistically significant over both linear and post-hoc approaches.
Revised: Hence SPAM’s improvements in interpretability are statistically significant over both linear and post-hoc approaches.

Original: Study #2: Varying Explanation Length .
Revised: Study #2: Varying Explanation Length .

Original: One can argue that the increased interpretability of polynomial approaches is due to the fact that each higher-order “term” involves multiple features (e.g., pairwise involves two features), and hence the increased mean user accuracy (mUA) is due to the larger number of features shown (either individually or as pairs).
Revised: One can argue that the increased interpretability of polynomial approaches is due to the fact that each higher-order “term” involves multiple features (e.g., pairwise involves two features), and hence the increased mean user accuracy (mUA) is due to the larger number of features shown (either individually or as pairs).

Original: We examine this hypothesis by varying the explanation length E for both SPAM-Linear (Order 2) and LIME, which consequently increases the number of terms seen for both approaches.
Revised: We examine this hypothesis by varying the explanation length E for both SPAM-Linear (Order 2) and LIME, which consequently increases the number of terms seen for both approaches.

Original: We observe in Figure 2C that regardless of explanation method, mUA is maximum at E = 7 , and introducing more terms decreases interpretability.
Revised: We observe in Figure 2C that regardless of explanation method, mUA is maximum at E = 7 , and introducing more terms decreases interpretability.

Original:  Discussion and Conclusion
Revised:  Discussion and Conclusion

Original: We presented a simple and scalable approach for modeling higher-order interactions in interpretable machine learning.
Revised: We presented a simple and scalable approach for modeling higher-order interactions in interpretable machine learning.

Original: Our approach alleviates several of the concerns with existing algorithms and focuses on presenting a viable alternative for practical, large-scale machine learning systems.
Revised: Our approach alleviates several of the concerns with existing algorithms and focuses on presenting a viable alternative for practical, large-scale machine learning systems.

Original: In addition to offline experimental guarantees, we demonstrate by an extensive user study that these models are indeed more interpretable in practice, and therefore can readily substitute post-hoc interpretability without any loss in performance.
Revised: In addition to offline experimental guarantees, we demonstrate by an extensive user study that these models are indeed more interpretable in practice, and therefore can readily substitute post-hoc interpretability without any loss in performance.

Original: Moreover, to the best of our knowledge, our work provides the first incremental analysis of interpretability and performance: we show how progressively increasing the model complexity (with higher order interactions) brings better performance while compromising interpretability in practice, and the “sweet spot” appears to rest at pairwise interactions.
Revised: Moreover, to the best of our knowledge, our work provides the first incremental analysis of interpretability and performance: we show how progressively increasing the model complexity (with higher order interactions) brings better performance while compromising interpretability in practice, and the “sweet spot” appears to rest at pairwise interactions.

Original: We have several follow-up directions.
Revised: We have several follow-up directions.

Original: First, given that our model provides precise guarantees on generalization, it is a feasible starting point to understanding tradeoffs between privacy and explainability from a rigorous perspective.
Revised: First, given that our model provides precise guarantees on generalization, it is a feasible starting point to understanding tradeoffs between privacy and explainability from a rigorous perspective.

Original: Next, one can consider utilizing SPAM to understand failure modes and spurious correlations directly.
Revised: Next, one can consider utilizing SPAM to understand failure modes and spurious correlations directly.

Original: Furthermore, SPAM-esque decompositions can also be useful in other domains beyond interpretability, e g., language and vision, where modeling higher-order interactions is challenging due to the curse of dimensionality.
Revised: Furthermore, SPAM-esque decompositions can also be useful in other domains beyond interpretability, e g., language and vision, where modeling higher-order interactions is challenging due to the curse of dimensionality.

Original: A Deferred Proofs A.1 Proof of Proposition
Revised: A Deferred Proofs A.1 Proof of Proposition

Original: This follows from noting that the Hessian of the objective function is positive definite for all λ >when the constraints in the proposition hold.
Revised: This follows from noting that the Hessian of the objective function is positive definite for all λ >when the constraints in the proposition hold.

Original: A.2 Proof of Proposition
Revised: A.2 Proof of Proposition

Original: This follows directly from the Stone-Weierstrass Theorem [Stone, 1948, Weierstrass, 1885] and
Revised: This follows directly from the Stone-Weierstrass Theorem [Stone, 1948, Weierstrass, 1885] and

Original: Theorem 1 of Chui and Li [1993].
Revised: Theorem 1 of Chui and Li [1993].

Original: A.3 Proof of Theorem
Revised: A.3 Proof of Theorem

Original: We state Theorem 1 in full here.
Revised: We state Theorem 1 in full here.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let ℓ be a 1-Lipschitz loss, δ ∈ (0 , 1] and the generalization error for the optimal degree k polynomial P ⋆,k be given by E ( P ⋆ , k ) and the training error for the ERM SPAM (cid:98) P r ,k , i.e., (cid:98) P r ,k = arg
Revised: Let ℓ be a 1-Lipschitz loss, δ ∈ (0 , 1] and the generalization error for the optimal degree k polynomial P ⋆,k be given by E ( P ⋆ , k ) and the training error for the ERM SPAM (cid:98) P r ,k , i.e., (cid:98) P r ,k = arg

Original: min P (cid:80) ni
Revised: min P (cid:80) ni

Original: =1 ℓ ( P ( x i , y i ) with ranks r = [1 , r 2 , ..., r k ] be given by (cid:98) E n ( (cid:98) P r ,k ) .
Revised: =1 ℓ ( P ( x i , y i ) with ranks r = [1 , r 2 , ..., r k ] be given by (cid:98) E n ( (cid:98) P r ,k ) .

Original: Then, for
Revised: Then, for

Original: L 2 − regularized models, where ∥ u ij ∥ 2 ≤ B u, 2 for all i ∈ [ k ] , j ∈ [ r i ] , and ∥ λ ∥ 2 ≤ B λ, 2 where λ
Revised: L 2 − regularized models, where ∥ u ij ∥ 2 ≤ B u, 2 for all i ∈ [ k ] , j ∈ [ r i ] , and ∥ λ ∥ 2 ≤ B λ, 2 where λ

Original: = {{ λ ij } r i j =1 } ki
Revised: = {{ λ ij } r i j =1 } ki

Original: =1 , we have that with probability at least 1 − δ there exists an absolute constant C such that,
Revised: =1 , we have that with probability at least 1 − δ there exists an absolute constant C such that,

Original: [Equation]
Revised: [Equation]

Original: For L 1 − regularization, ∥ u ij ∥ 1 ≤ B u, 1 for all i ∈ [ k ] , j ∈ [ r i ] , and ∥ λ ∥ 1 ≤ B λ, 1 where λ = {{ λ ij } r i j =1 } ki =1 , we have with probability at least 1 − δ there exists an absolute constant C such that,
Revised: For L 1 − regularization, ∥ u ij ∥ 1 ≤ B u, 1 for all i ∈ [ k ] , j ∈ [ r i ] , and ∥ λ ∥ 1 ≤ B λ, 1 where λ = {{ λ ij } r i j =1 } ki =1 , we have with probability at least 1 − δ there exists an absolute constant C such that,

Original: [Equation]
Revised: [Equation]

Original: While we will eventually prove Theorem 1, the approach is to first establish the necessary mathematical background and outline the proof approach.
Revised: While we will eventually prove Theorem 1, the approach is to first establish the necessary mathematical background and outline the proof approach.

Original: We will provide a general result under arbitrary
Revised: We will provide a general result under arbitrary

Original: Lipschitz loss functions via a metric entropy bound for low-rank polynomial approximations.
Revised: Lipschitz loss functions via a metric entropy bound for low-rank polynomial approximations.

Original: We will denote, for any k > 1 , the space of all low-rank polynomials of order k with cumulative rank r = [1 , r 1 , r 2 , ..., r k ] ∈ R k as Θ r ⊂ R d
Revised: We will denote, for any k > 1 , the space of all low-rank polynomials of order k with cumulative rank r = [1 , r 1 , r 2 , ..., r k ] ∈ R k as Θ r ⊂ R d

Original: ( r +1)
Revised: ( r +1)

Original: + where r = ∥ r ∥ 1 .
Revised: + where r = ∥ r ∥ 1 .

Original: Recall from Section 3 that any θ ∈ Θ is composed of the components, { b, {{ λ ij } r i j =1 } ki =1 , {{ u ij } r i j =1 } ki =1 } and we use θ as a shorthand.
Revised: Recall from Section 3 that any θ ∈ Θ is composed of the components, { b, {{ λ ij } r i j =1 } ki =1 , {{ u ij } r i j =1 } ki =1 } and we use θ as a shorthand.

Original: Now, for any function f : X → Y and any bounded L − Lipschitz loss function ℓ : Y × Y → [0 , 1] (we assume, without loss of generality, that the range of ℓ is the closed interval [0 , 1] ), the training error over n samples as,
Revised: Now, for any function f : X → Y and any bounded L − Lipschitz loss function ℓ : Y × Y → [0 , 1] (we assume, without loss of generality, that the range of ℓ is the closed interval [0 , 1] ), the training error over n samples as,

Original: [Equation]
Revised: [Equation]

Original: Similarly, we can define the expected risk over the sample distribution P : X × Y → [0 , 1) as,
Revised: Similarly, we can define the expected risk over the sample distribution P : X × Y → [0 , 1) as,

Original: [Equation]
Revised: [Equation]

Original: At a high level, our objective is to bound the expected risk of the ERM low-rank decomposed polynomial (LRDP) E
Revised: At a high level, our objective is to bound the expected risk of the ERM low-rank decomposed polynomial (LRDP) E

Original: ( (cid:98) P r ,k ) of some degree k and (cumulative) rank r with risk of the optimal polynomial of degree k , i.e., E ( P ⋆,k ) .
Revised: ( (cid:98) P r ,k ) of some degree k and (cumulative) rank r with risk of the optimal polynomial of degree k , i.e., E ( P ⋆,k ) .

Original: Note that P ⋆,k is not necessarily low-rank and can have cumulative rank of order O ( d k ) .
Revised: Note that P ⋆,k is not necessarily low-rank and can have cumulative rank of order O ( d k ) .

Original: Let us denote the maximum possible rank of any polynomial of degree k as ¯ r and the corresponding set of ranks as ¯ r .
Revised: Let us denote the maximum possible rank of any polynomial of degree k as ¯ r and the corresponding set of ranks as ¯ r .

Original: Observe that any polynomial of degree k has a (non-unique) representation in Θ ¯ r , and furthermore,
Revised: Observe that any polynomial of degree k has a (non-unique) representation in Θ ¯ r , and furthermore,

Original: [Equation]
Revised: [Equation]

Original: Similarly, we have that the empirical risk minimizer LRDP (cid:98) P r ,k of degree k and ranks r satisfies,
Revised: Similarly, we have that the empirical risk minimizer LRDP (cid:98) P r ,k of degree k and ranks r satisfies,

Original: [Equation]
Revised: [Equation]

Original: We can now begin the proof.
Revised: We can now begin the proof.

Original: For the optimal polynomial P ⋆,k , let us denote the corresponding singular values as {{ λ ⋆ij } ¯ r i j =1 } ki =1 , bases as {{ u ⋆ij } ¯ r i j =1 } ki =1 and bias as b ⋆,k .
Revised: For the optimal polynomial P ⋆,k , let us denote the corresponding singular values as {{ λ ⋆ij } ¯ r i j =1 } ki =1 , bases as {{ u ⋆ij } ¯ r i j =1 } ki =1 and bias as b ⋆,k .

Original: We will bound the excess risk E ( (cid:98) P r ,k ) − E ( P ⋆,k ) by introducing a third polynomial (cid:101) P r ,k ∈ Θ r which is defined as a truncated version of P ⋆,k up to the ranks r .
Revised: We will bound the excess risk E ( (cid:98) P r ,k ) − E ( P ⋆,k ) by introducing a third polynomial (cid:101) P r ,k ∈ Θ r which is defined as a truncated version of P ⋆,k up to the ranks r .

Original: Specifically, we set the singular values {{ (cid:101) λ ij } r i j =1 } ki =1 , bases as {{ (cid:101)
Revised: Specifically, we set the singular values {{ (cid:101) λ ij } r i j =1 } ki =1 , bases as {{ (cid:101)

Original: u ij } r i j =1 } ki =1 and bias as (cid:101) b k as,
Revised: u ij } r i j =1 } ki =1 and bias as (cid:101) b k as,

Original: [Equation]
Revised: [Equation]

Original: The polynomial (cid:101) P r ,k can therefore be thought of as a “truncated” version of the optimal polynomial
Revised: The polynomial (cid:101) P r ,k can therefore be thought of as a “truncated” version of the optimal polynomial

Original: P ⋆,k up to the ranks r .
Revised: P ⋆,k up to the ranks r .

Original: Observe that we can then write the excess risk as,
Revised: Observe that we can then write the excess risk as,

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: P
Revised: P

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: The term (cid:98) E n ( (cid:98) P r ,k ) − (cid:98) E n ( (cid:101) P r ,k ) ≤ 0 since (cid:98) P r ,k minimizes the empirical risk within Θ r (Equation 5).
Revised: The term (cid:98) E n ( (cid:98) P r ,k ) − (cid:98) E n ( (cid:101) P r ,k ) ≤ 0 since (cid:98) P r ,k minimizes the empirical risk within Θ r (Equation 5).

Original: Hence bounding terms A and B will provide us the bound.
Revised: Hence bounding terms A and B will provide us the bound.

Original: We will first bound term B via
Revised: We will first bound term B via

Original: Lemma 1.
Revised: Lemma 1.

Original: We have that with probability at least 1 − δ/ 2 for any δ ∈ (0 , 1] ,
Revised: We have that with probability at least 1 − δ/ 2 for any δ ∈ (0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: To bound term A , we proceed via bounding the Rademacher complexity, a classic approach in statistical learning [Wainwright, 2019].
Revised: To bound term A , we proceed via bounding the Rademacher complexity, a classic approach in statistical learning [Wainwright, 2019].

Original: We will first establish a bound on the Rademacher complexity of Θ r under L 1 and L 2 − regularization.
Revised: We will first establish a bound on the Rademacher complexity of Θ r under L 1 and L 2 − regularization.

Original: Note that similar bounds via directly bounding the metric entropy or Dudley’s entropy can also be obtained, but we present this approach for simplicity.
Revised: Note that similar bounds via directly bounding the metric entropy or Dudley’s entropy can also be obtained, but we present this approach for simplicity.

Original: We anticipate that unless alternative assumptions are made about the polynomials, the bounds will remain identical via these approaches.
Revised: We anticipate that unless alternative assumptions are made about the polynomials, the bounds will remain identical via these approaches.

Original: Observe first, that since the loss function is Lipschitz and bounded, we have with probability at least 1 − δ/ 2 , δ ∈ (0 , 1] , from Theorem 4 and Theorem 8 of Bartlett and
Revised: Observe first, that since the loss function is Lipschitz and bounded, we have with probability at least 1 − δ/ 2 , δ ∈ (0 , 1] , from Theorem 4 and Theorem 8 of Bartlett and

Original: Mendelson [2002],
Revised: Mendelson [2002],

Original: [Equation]
Revised: [Equation]

Original: Where R n denotes the empirical Rademacher complexity at n samples [Bartlett and Mendelson, 2002], and P ( Θ r ) denotes the set of all polynomials represented by the parameterization Θ r .
Revised: Where R n denotes the empirical Rademacher complexity at n samples [Bartlett and Mendelson, 2002], and P ( Θ r ) denotes the set of all polynomials represented by the parameterization Θ r .

Original: Now, observe that any element in P ( Θ r ) comprises k + 1 polynomials of a fixed degree (one for each degree from 0 to k ).
Revised: Now, observe that any element in P ( Θ r ) comprises k + 1 polynomials of a fixed degree (one for each degree from 0 to k ).

Original: With some abuse of notation, let us denote the family of all polynomials with a degree exactly l and rank exactly r as P l,r .
Revised: With some abuse of notation, let us denote the family of all polynomials with a degree exactly l and rank exactly r as P l,r .

Original: Observe that by Theorem 4,
Revised: Observe that by Theorem 4,

Original: [Equation]
Revised: [Equation]

Original: The last equality follows from the fact that P 0 , · is always a constant function with a Rademacher complexity of 0.
Revised: The last equality follows from the fact that P 0 , · is always a constant function with a Rademacher complexity of 0.

Original: Furthermore, since ℓ is Lipschitz, we have once again by Theorem 4 that R n ( ℓ ⊙
Revised: Furthermore, since ℓ is Lipschitz, we have once again by Theorem 4 that R n ( ℓ ⊙

Original: P ( Θ r )) ≤ 2 L · R n ( P ( Θ r )) .
Revised: P ( Θ r )) ≤ 2 L · R n ( P ( Θ r )) .

Original: Putting this together, we have that with probability at least 1 − δ/ 2 , δ ∈ (0 , 1] ,
Revised: Putting this together, we have that with probability at least 1 − δ/ 2 , δ ∈ (0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: Now, to bound the remaining term, we consider two regularization constraints for Θ r .
Revised: Now, to bound the remaining term, we consider two regularization constraints for Θ r .

Original: The first is L 2 − regularization, where we assume that the parameter λ, u are bounded in L 2 .
Revised: The first is L 2 − regularization, where we assume that the parameter λ, u are bounded in L 2 .

Original: Let λ = {{ λ ij } r i j =1 } ki =1 denote the singular values across all ranks.
Revised: Let λ = {{ λ ij } r i j =1 } ki =1 denote the singular values across all ranks.

Original: Then, we assume for all i ∈ [1 , ..., k ] and j ∈ [1 , ..., r i ] that ∥ u ij ∥ 2 ≤ B u, 2 and ∥ λ ∥ 2 ≤ B λ, 2 .
Revised: Then, we assume for all i ∈ [1 , ..., k ] and j ∈ [1 , ..., r i ] that ∥ u ij ∥ 2 ≤ B u, 2 and ∥ λ ∥ 2 ≤ B λ, 2 .

Original: Under this constraint, we can bound the empirical Rademacher complexity of each polynomial via Lemma 2.
Revised: Under this constraint, we can bound the empirical Rademacher complexity of each polynomial via Lemma 2.

Original: Replacing this result, we obtain with probability at least 1 − δ/ 2 , δ ∈ (0 , 1] ,
Revised: Replacing this result, we obtain with probability at least 1 − δ/ 2 , δ ∈ (0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: Where B x 2 = sup x ∈X ∥ x ∥ 2 .
Revised: Where B x 2 = sup x ∈X ∥ x ∥ 2 .

Original: Without loss of generality we can assume B x 2 = √ d .
Revised: Without loss of generality we can assume B x 2 = √ d .

Original: Now, for
Revised: Now, for

Original: L 1 − regularized models, we assume that for all i ∈ [1 , ..., k ] and j ∈ [1 , ..., r i ] that ∥ u ij ∥ 1 ≤ B u,and ∥ λ ∥ 1 ≤ B λ, 1 .
Revised: L 1 − regularized models, we assume that for all i ∈ [1 , ..., k ] and j ∈ [1 , ..., r i ] that ∥ u ij ∥ 1 ≤ B u,and ∥ λ ∥ 1 ≤ B λ, 1 .

Original: This gives us the following bound via Lemma 3:
Revised: This gives us the following bound via Lemma 3:

Original: [Equation]
Revised: [Equation]

Original: Where B x ∞ = sup x ∈X ∥ x ∥ ∞ .
Revised: Where B x ∞ = sup x ∈X ∥ x ∥ ∞ .

Original: Without loss of generality we can assume B x ∞ = 1 .
Revised: Without loss of generality we can assume B x ∞ = 1 .

Original: Using the bound for term B and applying a union bound provides us the final result.
Revised: Using the bound for term B and applying a union bound provides us the final result.

Original: A.4 Polynomial Decay of Spectrum
Revised: A.4 Polynomial Decay of Spectrum

Original: As discussed earlier, one can replace the “exponential decay” assumption in Assumption 1 to a “softer” decay criterion with fatter tails, e.g., like a polynomial decay.
Revised: As discussed earlier, one can replace the “exponential decay” assumption in Assumption 1 to a “softer” decay criterion with fatter tails, e.g., like a polynomial decay.

Original: We present extensions of earlier results to this polynomial decay condition.
Revised: We present extensions of earlier results to this polynomial decay condition.

Original: First we state the polynomial decay criterion precisely.
Revised: First we state the polynomial decay criterion precisely.

Original: Assumption 2.
Revised: Assumption 2.

Original: Let P k denote the family of all polynomials of degree at most k , and let P ⋆,k denote the optimal polynomial in P k , i.e., P ⋆,k = arg min P ∈P k E ( x ,y ) ∼ P [ ℓ ( P ( x ) , y )] .
Revised: Let P k denote the family of all polynomials of degree at most k , and let P ⋆,k denote the optimal polynomial in P k , i.e., P ⋆,k = arg min P ∈P k E ( x ,y ) ∼ P [ ℓ ( P ( x ) , y )] .

Original: We assume that ∀ k ,
Revised: We assume that ∀ k ,

Original: P ⋆,k admits a decomposition as described in Equation 1 such that, for all 1 ≤ l ≤ k , there exist constants C 1 < 1 and C 

Revised: P ⋆,k admits a decomposition as described in Equation 1 such that, for all 1 ≤ l ≤ k , there exist constants C 1 < 1 and C

Original: = O (1) such that | λ lj | ≤ C 1 · j − γ − 1 for each j ≥ 1 and l ∈ [1 , k ] .
Revised: = O (1) such that | λ lj | ≤ C 1 · j − γ − 1 for each j ≥ 1 and l ∈ [1 , k ] .

Original: Theorem 3.
Revised: Theorem 3.

Original: Let ℓ be a 1-Lipschitz loss, δ ∈ (0 , 1] and the generalization error for the optimal degree k polynomial P ⋆,k be given by E ( P ⋆ , k ) and the training error for the ERM SPAM (cid:98) P r ,k , i.e., (cid:98) P r ,k = arg
Revised: Let ℓ be a 1-Lipschitz loss, δ ∈ (0 , 1] and the generalization error for the optimal degree k polynomial P ⋆,k be given by E ( P ⋆ , k ) and the training error for the ERM SPAM (cid:98) P r ,k , i.e., (cid:98) P r ,k = arg

Original: min P (cid:80) ni
Revised: min P (cid:80) ni

Original: =1 ℓ ( P ( x i , y i ) with ranks r = [1 , r 2 , ..., r k ] be given by (cid:98) E n ( (cid:98) P r ,k ) .
Revised: =1 ℓ ( P ( x i , y i ) with ranks r = [1 , r 2 , ..., r k ] be given by (cid:98) E n ( (cid:98) P r ,k ) .

Original: Then, for
Revised: Then, for

Original: L 2 − regularized models, where ∥ u ij ∥ 2 ≤ B u, 2 for all i ∈ [ k ] , j ∈ [ r i ] , and ∥ λ ∥ 2 ≤ B λ, 2 where λ
Revised: L 2 − regularized models, where ∥ u ij ∥ 2 ≤ B u, 2 for all i ∈ [ k ] , j ∈ [ r i ] , and ∥ λ ∥ 2 ≤ B λ, 2 where λ

Original: = {{ λ ij } r i j =1 } ki
Revised: = {{ λ ij } r i j =1 } ki

Original: =1 , we have that with probability at least 1 − δ there exists an absolute constant C such that,
Revised: =1 , we have that with probability at least 1 − δ there exists an absolute constant C such that,

Original: [Equation]
Revised: [Equation]

Original: For L 1 − regularization, ∥ u ij ∥ 1 ≤ B u, 1 for all i ∈ [ k ] , j ∈ [ r i ] , and ∥ λ ∥ 1 ≤ B λ, 1 where λ = {{ λ ij } r i j =1 } ki =1 , we have with probability at least 1 − δ there exists an absolute constant C such that,
Revised: For L 1 − regularization, ∥ u ij ∥ 1 ≤ B u, 1 for all i ∈ [ k ] , j ∈ [ r i ] , and ∥ λ ∥ 1 ≤ B λ, 1 where λ = {{ λ ij } r i j =1 } ki =1 , we have with probability at least 1 − δ there exists an absolute constant C such that,

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is identical to the proof of Theorem 1, except for the application of Lemma 1.
Revised: The proof is identical to the proof of Theorem 1, except for the application of Lemma 1.

Original: Note that following Lemma 1, the sum of the singular values can be bound using Assumption 2 as, k
Revised: Note that following Lemma 1, the sum of the singular values can be bound using Assumption 2 as, k

Original: [Equation]
Revised: [Equation]

Original: Replacing this result completes the proof.
Revised: Replacing this result completes the proof.

Original: Discussion .
Revised: Discussion .

Original: Observe that in contrast to the result for exponential decay (where one required r to be polylogarithmic in d ), we now require a larger rank r to achieve competitive performance.
Revised: Observe that in contrast to the result for exponential decay (where one required r to be polylogarithmic in d ), we now require a larger rank r to achieve competitive performance.

Original: Specifically, one can observe from Theorem 3 that setting r i = r/k =
Revised: Specifically, one can observe from Theorem 3 that setting r i = r/k =

Original: d γ/m for some m >guarantees that the error term decays at the rate O (cid:16) k 1 − γm d − 1 m (cid:17) which diminishes quickly for small m and large d .
Revised: d γ/m for some m >guarantees that the error term decays at the rate O (cid:16) k 1 − γm d − 1 m (cid:17) which diminishes quickly for small m and large d .

Original: However, this degrades the rate of growth of the Rademacher complexity (first term) from a O ( √ d ) dependence to O ( d 1 p ( 1+ γm )) for L p regularization, p ∈ { 1 , 2 } .
Revised: However, this degrades the rate of growth of the Rademacher complexity (first term) from a O ( √ d ) dependence to O ( d 1 p ( 1+ γm )) for L p regularization, p ∈ { 1 , 2 } .

Original: A.5 Omitted Results
Revised: A.5 Omitted Results

Original: Lemma 1.
Revised: Lemma 1.

Original: For the polynomials (cid:101) P r ,k and P ⋆,k as defined in Equations 4 and 6, the following holds with probability at least 1 − δ, δ ∈ (0 , 1] , for some absolute constant C ≪ 1 ,
Revised: For the polynomials (cid:101) P r ,k and P ⋆,k as defined in Equations 4 and 6, the following holds with probability at least 1 − δ, δ ∈ (0 , 1] , for some absolute constant C ≪ 1 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Observe
Revised: Observe

Original: [Equation]
Revised: [Equation]

Original: To bound B 1 , observe that for any point ( x , y ) within the training set, E [ ℓ ( (cid:101) P r ,k ( x ) , y )]
Revised: To bound B 1 , observe that for any point ( x , y ) within the training set, E [ ℓ ( (cid:101) P r ,k ( x ) , y )]

Original: =
Revised: =

Original: E ( (cid:101) P r ,k ) .
Revised: E ( (cid:101) P r ,k ) .

Original: Furthermore, 0 ≤ ℓ ( · , · ) ≤ 1 .
Revised: Furthermore, 0 ≤ ℓ ( · , · ) ≤ 1 .

Original: We can therefore apply the Azuma-Hoeffding inequality [Bercu et al., 2015] and obtain with probability at least 1 − δ, δ ∈ (0 , 1] ,
Revised: We can therefore apply the Azuma-Hoeffding inequality [Bercu et al., 2015] and obtain with probability at least 1 − δ, δ ∈ (0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: n .
Revised: n .

Original: To bound B 2 , observe that since ℓ is L − Lipschitz, for some x 1 , x 2 , y ∈ Y ,
Revised: To bound B 2 , observe that since ℓ is L − Lipschitz, for some x 1 , x 2 , y ∈ Y ,

Original: [Equation]
Revised: [Equation]

Original: Using this
Revised: Using this

Original: [Equation]
Revised: [Equation]

Original: we have that,
Revised: we have that,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Observe now that for any x ∈ X ,
Revised: Observe now that for any x ∈ X ,

Original: [Equation]
Revised: [Equation]

Original: | λ ⋆ij | .
Revised: | λ ⋆ij | .

Original: Recall that Assumption 1 can control (cid:80) ki
Revised: Recall that Assumption 1 can control (cid:80) ki

Original: =1 (cid:80) ¯ r
Revised: =1 (cid:80) ¯ r

Original: i j
Revised: i j

Original: = r
Revised: = r

Original: i | λ ⋆ij | .
Revised: i | λ ⋆ij | .

Original: If the spectrum obeys the D − finite condition and r i = D ∀ i ∈ [1 , k ] we have that (cid:80) k
Revised: If the spectrum obeys the D − finite condition and r i = D ∀ i ∈ [1 , k ] we have that (cid:80) k

Original: i =1 (cid:80) ¯ r
Revised: i =1 (cid:80) ¯ r

Original: i j
Revised: i j

Original: = r
Revised: = r

Original: i | λ ⋆ ij | = 0 .
Revised: i | λ ⋆ ij | = 0 .

Original: Next, if the spectrum obeys the γ − exponential decay, we have that λ ij = C 1 exp( − C 2 j γ ) .
Revised: Next, if the spectrum obeys the γ − exponential decay, we have that λ ij = C 1 exp( − C 2 j γ ) .

Original: This gives us,
Revised: This gives us,

Original: [Equation]
Revised: [Equation]

Original: By Equation E.16 from Yang et al.
Revised: By Equation E.16 from Yang et al.

Original: [2020] we can bound the R.H.S. since γ ≥ 1 , (cid:90) ∞
Revised: [2020] we can bound the R.H.S. since γ ≥ 1 , (cid:90) ∞

Original: [Equation]
Revised: [Equation]

Original: Replacing this result above completes the proof.
Revised: Replacing this result above completes the proof.

Original: Lemma 2 (Empirical Rademacher Complexity under L 2 regularization) .
Revised: Lemma 2 (Empirical Rademacher Complexity under L 2 regularization) .

Original: Let P l,r denote the set of polynomials that have rank exactly r and degree exactly l such that they admit a rank-decomposed representation of (cid:80) rj =

Revised: Let P l,r denote the set of polynomials that have rank exactly r and degree exactly l such that they admit a rank-decomposed representation of (cid:80) rj =

Original: λ j · u j ⊗ ... ⊗ u j , where λ =
Revised: λ j · u j ⊗ ... ⊗ u j , where λ =

Original: [ λ 1 , ..., λ r ] s.t.
Revised: [ λ 1 , ..., λ r ] s.t.

Original: ∥ λ ∥ 2 ≤ B λ, 2 and ∥ u j ∥ 2 ≤ B u,for all j ∈ 1 , ..., l .
Revised: ∥ λ ∥ 2 ≤ B λ, 2 and ∥ u j ∥ 2 ≤ B u,for all j ∈ 1 , ..., l .

Original: Let the rescaled data distribution as defined in Equation 3 for degree l be given by (cid:101)
Revised: Let the rescaled data distribution as defined in Equation 3 for degree l be given by (cid:101)

Original: X
Revised: X

Original: l = { sign ( x ) · | x | 1 /l | x ∈ X} and we sample n points i.i.d.
Revised: l = { sign ( x ) · | x | 1 /l | x ∈ X} and we sample n points i.i.d.

Original: from (cid:101) X l .
Revised: from (cid:101) X l .

Original: Then the empirical
Revised: Then the empirical

Original: Rademacher complexity R n of P l,r obeys,
Revised: Rademacher complexity R n of P l,r obeys,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: n (cid:118)(cid:117)(cid:117)(cid:116) r (cid:88)
Revised: n (cid:118)(cid:117)(cid:117)(cid:116) r (cid:88)

Original: j =
Revised: j =

Original: E (cid:34) sup ∥ u j ∥ 2 ≤ B u,n (cid:88) i
Revised: E (cid:34) sup ∥ u j ∥ 2 ≤ B u,n (cid:88) i

Original: =
Revised: =

Original: ϵ 2 n · ⟨ u j , ˜ x li ⟩ 2 l (cid:35) ≤ B λ,n (cid:118) (cid:117)
Revised: ϵ 2 n · ⟨ u j , ˜ x li ⟩ 2 l (cid:35) ≤ B λ,n (cid:118) (cid:117)

Original: (cid:117)(cid:116) r
Revised: (cid:117)(cid:116) r

Original: (cid:88)
Revised: (cid:88)

Original: j =
Revised: j =

Original: E (cid:34) sup ∥ u j ∥ 2 ≤ B u,n (cid:88) i =
Revised: E (cid:34) sup ∥ u j ∥ 2 ≤ B u,n (cid:88) i =

Original: ∥
Revised: ∥

Original: u j ∥ 2 l 2 ∥ ˜ x li ∥ 2 l (cid:35) ≤ B λ, 2 ( B u, 2 ) l B x(cid:114)
Revised: u j ∥ 2 l 2 ∥ ˜ x li ∥ 2 l (cid:35) ≤ B λ, 2 ( B u, 2 ) l B x(cid:114)

Original: r n.
Revised: r n.

Original: Lemma 3 (Empirical Rademacher Complexity under L 1 regularization) .
Revised: Lemma 3 (Empirical Rademacher Complexity under L 1 regularization) .

Original: Let P l,r denote the set of polynomials that have rank exactly r and degree exactly l such that they admit a rank-decomposed representation of (cid:80) rj =

Revised: Let P l,r denote the set of polynomials that have rank exactly r and degree exactly l such that they admit a rank-decomposed representation of (cid:80) rj =

Original: λ j · u j ⊗ ... ⊗ u j , where λ =
Revised: λ j · u j ⊗ ... ⊗ u j , where λ =

Original: [ λ 1 , ..., λ r ] s.t.
Revised: [ λ 1 , ..., λ r ] s.t.

Original: ∥ λ ∥ 1 ≤ B λ, 1 and ∥ u j ∥ 1 ≤ B u,for all j ∈ 1 , ..., l .
Revised: ∥ λ ∥ 1 ≤ B λ, 1 and ∥ u j ∥ 1 ≤ B u,for all j ∈ 1 , ..., l .

Original: Let the rescaled data distribution as defined in Equation 3 for degree l be given by (cid:101)
Revised: Let the rescaled data distribution as defined in Equation 3 for degree l be given by (cid:101)

Original: X
Revised: X

Original: l = { sign ( x ) · | x | 1 /l | x ∈ X} and we sample n points i.i.d.
Revised: l = { sign ( x ) · | x | 1 /l | x ∈ X} and we sample n points i.i.d.

Original: from (cid:101) X l .
Revised: from (cid:101) X l .

Original: Then the empirical
Revised: Then the empirical

Original: Rademacher complexity R n of P l,r obeys,
Revised: Rademacher complexity R n of P l,r obeys,

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: The last line follows from Massart’s Finite Lemma (Massart [2000], Lemma 5.2).
Revised: The last line follows from Massart’s Finite Lemma (Massart [2000], Lemma 5.2).

Original: Theorem 4 (Structural Results for Rademacher Complexity, Theorem 12 of Bartlett and Mendelson [2002]) .
Revised: Theorem 4 (Structural Results for Rademacher Complexity, Theorem 12 of Bartlett and Mendelson [2002]) .

Original: Let F , F 1 , ..., F k and H be classes of functions.
Revised: Let F , F 1 , ..., F k and H be classes of functions.

Original: Then,
Revised: Then,

Original: If F ⊆ H , R n ( F ) ≤ R n ( H ) .
Revised: If F ⊆ H , R n ( F ) ≤ R n ( H ) .

Original: R n ( (cid:80) ki =1 F i ) ≤ (cid:80) ki
Revised: R n ( (cid:80) ki =1 F i ) ≤ (cid:80) ki

Original: =1 R n ( F i ) .
Revised: =1 R n ( F i ) .

Original: For any L -Lipschitz function h , R n ( F ⊙ h ) ≤ 2 L · R n ( F ) .
Revised: For any L -Lipschitz function h , R n ( F ⊙ h ) ≤ 2 L · R n ( F ) .

Original: B Related Work
Revised: B Related Work

Original: Post-hoc explainability .
Revised: Post-hoc explainability .

Original: In contrast to designing fully-interpretable models such as GAMs (Generalized Additive Models), a dominant line of work in interpretable machine learning is of post-hoc explainability.
Revised: In contrast to designing fully-interpretable models such as GAMs (Generalized Additive Models), a dominant line of work in interpretable machine learning is of post-hoc explainability.

Original: Post-hoc methods refers to delivering explanations of model predictions after the prediction has been made.
Revised: Post-hoc methods refers to delivering explanations of model predictions after the prediction has been made.

Original: The most popular line of work in this domain is that of instance-based explanations, i.e., explaining each prediction via a local input-specific explanation.
Revised: The most popular line of work in this domain is that of instance-based explanations, i.e., explaining each prediction via a local input-specific explanation.

Original: For example, the work of Ribeiro et al.
Revised: For example, the work of Ribeiro et al.

Original: [2016] introduces LIME (Local Interpretable Model Explanations) that fit a weighted linear regression model, for any input x , using random samples generated from
Revised: [2016] introduces LIME (Local Interpretable Model Explanations) that fit a weighted linear regression model, for any input x , using random samples generated from

Original: B 2 ( x ) , where the weights are computed via the distance of the random point from the true sample.
Revised: B 2 ( x ) , where the weights are computed via the distance of the random point from the true sample.

Original: SHAP [Lundberg and Lee, 2017] computes a similar linear explanation based on Shapley values using influence functions derived from cooperative game theory.
Revised: SHAP [Lundberg and Lee, 2017] computes a similar linear explanation based on Shapley values using influence functions derived from cooperative game theory.

Original: It has been shown to unify several prior approaches, e.g., DeepLIFT [Shrikumar et al., 2017, 2016], LIME [Ribeiro et al., 2016], and
Revised: It has been shown to unify several prior approaches, e.g., DeepLIFT [Shrikumar et al., 2017, 2016], LIME [Ribeiro et al., 2016], and

Original: Layerwise Relevance Propagation (LRP) Bach et al.
Revised: Layerwise Relevance Propagation (LRP) Bach et al.

Original: Several follow-up works have addressed shortcomings in this line of work, please see Madsen et al.
Revised: Several follow-up works have addressed shortcomings in this line of work, please see Madsen et al.

Original: [2021], Chakraborty et al.
Revised: [2021], Chakraborty et al.

Original: [2017], Du et al.
Revised: [2017], Du et al.

Original: [2019], Carvalho et al.
Revised: [2019], Carvalho et al.

Original: [2019] for excellent surveys on recent developments in this area.
Revised: [2019] for excellent surveys on recent developments in this area.

Original: More importantly, however, these approaches are not effective for high-stakes decision-making.
Revised: More importantly, however, these approaches are not effective for high-stakes decision-making.

Original: They are notoriously unstable [Ghorbani et al., 2019a, Lakkaraju and Bastani, 2020], expensive to compute [Slack et al., 2021], unjustified [Laugel et al., 2019] and in many cases, inaccurate [Lipton, 2018].
Revised: They are notoriously unstable [Ghorbani et al., 2019a, Lakkaraju and Bastani, 2020], expensive to compute [Slack et al., 2021], unjustified [Laugel et al., 2019] and in many cases, inaccurate [Lipton, 2018].

Original: Rudin [2019] outlines several of these shortcomings in incredible detail, and hence makes the case to replace post-hoc interpretable approaches with methods that are inherently explainable.
Revised: Rudin [2019] outlines several of these shortcomings in incredible detail, and hence makes the case to replace post-hoc interpretable approaches with methods that are inherently explainable.

Original: Transparent and Interpretable Machine Learning .
Revised: Transparent and Interpretable Machine Learning .

Original: Early work has focused on greedy or ensemble approaches to modeling interactions [Friedman, 2001, Friedman and Popescu, 2008] that enumerate pairwise interactions and learn additive interaction effects.
Revised: Early work has focused on greedy or ensemble approaches to modeling interactions [Friedman, 2001, Friedman and Popescu, 2008] that enumerate pairwise interactions and learn additive interaction effects.

Original: Such approaches often pick up spurious interactions when data is sparse [Lou et al., 2013] and are impossible to scale to modern-sized datasets due to enumeration of individual combinations.
Revised: Such approaches often pick up spurious interactions when data is sparse [Lou et al., 2013] and are impossible to scale to modern-sized datasets due to enumeration of individual combinations.

Original: As an improvement, Lou et al.
Revised: As an improvement, Lou et al.

Original: [2013] proposed
Revised: [2013] proposed

Original: GA 2 M that uses a statistical test to filter out only “true” interactions.
Revised: GA 2 M that uses a statistical test to filter out only “true” interactions.

Original: However GA 2 M fails to scale to large datasets as it requires constant re-training of the model and ad-hoc operations such as discretization of features which may require tuning for datasets with a large dimensionality.
Revised: However GA 2 M fails to scale to large datasets as it requires constant re-training of the model and ad-hoc operations such as discretization of features which may require tuning for datasets with a large dimensionality.

Original: Other generalized additive models require expensive training of numerous decision trees, kernel machines or splines [Hastie and Tibshirani, 2017], which make them unattractive compared to black-box models.
Revised: Other generalized additive models require expensive training of numerous decision trees, kernel machines or splines [Hastie and Tibshirani, 2017], which make them unattractive compared to black-box models.

Original: An alternate approach is is to learn interpretable neural network transformations.
Revised: An alternate approach is is to learn interpretable neural network transformations.

Original: Neural Additive
Revised: Neural Additive

Original: Models (NAMs, Agarwal et al.
Revised: Models (NAMs, Agarwal et al.

Original: [2021]) learn a DNN per feature.
Revised: [2021]) learn a DNN per feature.

Original: TabNet [Arık and Pfister, 2021] and NIT [Tsang et al., 2018] alternatively modify NN architectures to increase their interpretability.
Revised: TabNet [Arık and Pfister, 2021] and NIT [Tsang et al., 2018] alternatively modify NN architectures to increase their interpretability.

Original: NODE-GAM
Revised: NODE-GAM

Original: [Chang et al., 2021] improves NAMs with oblivious decision trees for better performance while maintaining interpretability.
Revised: [Chang et al., 2021] improves NAMs with oblivious decision trees for better performance while maintaining interpretability.

Original: Our approach is notably distinct from these prior works: we do not require iterative re-training; we can learn all pairwise interactions regardless of dimensionality; we can train SPAM via backpropagation; and we scale effortlessly to very large-scale datasets.
Revised: Our approach is notably distinct from these prior works: we do not require iterative re-training; we can learn all pairwise interactions regardless of dimensionality; we can train SPAM via backpropagation; and we scale effortlessly to very large-scale datasets.

Original: Learning Polynomials .
Revised: Learning Polynomials .

Original: The idea of decomposing polynomials was of interest prior to the deep learning era.
Revised: The idea of decomposing polynomials was of interest prior to the deep learning era.

Original: Specifically, the work of Ivakhnenko [1971], Oh et al.
Revised: Specifically, the work of Ivakhnenko [1971], Oh et al.

Original: [2003], Shin and Ghosh [1991] study learning neural networks with polynomial interactions, also known as ridge polynomial networks (RPNs).
Revised: [2003], Shin and Ghosh [1991] study learning neural networks with polynomial interactions, also known as ridge polynomial networks (RPNs).

Original: However, RPNs are typically not interpretable: they learn interactions of a very high order, and include non-linear transformation.
Revised: However, RPNs are typically not interpretable: they learn interactions of a very high order, and include non-linear transformation.

Original: Similar rank decompositions have been studied in the context of matrix completion [Recht, 2011], and are also a subject of interest in tensor decompositions
Revised: Similar rank decompositions have been studied in the context of matrix completion [Recht, 2011], and are also a subject of interest in tensor decompositions

Original: [Nie, 2017a, Brachat et al., 2010], where, contrary to our work, the objective is to decompose existing tensors rather than directly learn decompositions from gradient descent.
Revised: [Nie, 2017a, Brachat et al., 2010], where, contrary to our work, the objective is to decompose existing tensors rather than directly learn decompositions from gradient descent.

Original: Recently, Chrysos et al.
Revised: Recently, Chrysos et al.

Original: [2019, 2020] use tensor decompositions to learn higher-order polynomial relationships in intermediate layers of generative models.
Revised: [2019, 2020] use tensor decompositions to learn higher-order polynomial relationships in intermediate layers of generative models.

Original: However, their work uses a recursive formulation and learns high-degree polynomials directly from uninterpretable input data (e.g., images), and hence is non-interpretable.
Revised: However, their work uses a recursive formulation and learns high-degree polynomials directly from uninterpretable input data (e.g., images), and hence is non-interpretable.

Original: C Experimental Details
Revised: C Experimental Details

Original: All our benchmarks are implemented in PyTorch and run on a cluster, where each machine was equipped with 8 × V100 NVIDIA
Revised: All our benchmarks are implemented in PyTorch and run on a cluster, where each machine was equipped with 8 × V100 NVIDIA

Original: GPU machines with 32 GB VRAM.
Revised: GPU machines with 32 GB VRAM.

Original: We set the batch size of at most 1024 per GPU, and in case a model is too large to fit a batch size of 1024 , we perform a binary search to find the largest batch size that it will accommodate.
Revised: We set the batch size of at most 1024 per GPU, and in case a model is too large to fit a batch size of 1024 , we perform a binary search to find the largest batch size that it will accommodate.

Original: All datasets have features renormalized between the range [0 , 1] for uniformity and simplicity.
Revised: All datasets have features renormalized between the range [0 , 1] for uniformity and simplicity.

Original: For all experiments we use the Adam with decoupled weight decay (AdamW) optimizer
Revised: For all experiments we use the Adam with decoupled weight decay (AdamW) optimizer

Original: [Loshchilov and Hutter, 2017].
Revised: [Loshchilov and Hutter, 2017].

Original: C.1 Tabular Dataset Details
Revised: C.1 Tabular Dataset Details

Original: We use the following tabular datasets.
Revised: We use the following tabular datasets.

Original: California Housing (CH, Pace and Barry [1997]): This dataset was derived from the 1990 U.S. census, and is a regression task, fitting the median house value for California districts using demographic information.
Revised: California Housing (CH, Pace and Barry [1997]): This dataset was derived from the 1990 U.S. census, and is a regression task, fitting the median house value for California districts using demographic information.

Original: The dataset contains 20,640 instances and 8 numeric features.
Revised: The dataset contains 20,640 instances and 8 numeric features.

Original: FICO HELOC (FICO, FICO [2018]):
Revised: FICO HELOC (FICO, FICO [2018]):

Original: This is a binary task, part of the FICO Explainability challenge.
Revised: This is a binary task, part of the FICO Explainability challenge.

Original: The target variable denotes the “risk” of home equity line of credit (HELOC) applicants and features correspond to credit report information.
Revised: The target variable denotes the “risk” of home equity line of credit (HELOC) applicants and features correspond to credit report information.

Original: It has 10,459 instances and 23 features.
Revised: It has 10,459 instances and 23 features.

Original: Cover Type (CovType, Blackard and Dean [1999]): This multi-class dataset is part of the UCI ML repository [Asuncion and Newman, 2007].
Revised: Cover Type (CovType, Blackard and Dean [1999]): This multi-class dataset is part of the UCI ML repository [Asuncion and Newman, 2007].

Original: The task is classification of forest cover type from cartographic variables.
Revised: The task is classification of forest cover type from cartographic variables.

Original: It contains 581,012 instances and 54 attributes over 6 classes.
Revised: It contains 581,012 instances and 54 attributes over 6 classes.

Original: 20 Newsgroups (Newsgroups, Lang [1995]):
Revised: 20 Newsgroups (Newsgroups, Lang [1995]):

Original: This is a popular benchmark dataset where the problem is multi-class classification of email text into 20 subject categories.
Revised: This is a popular benchmark dataset where the problem is multi-class classification of email text into 20 subject categories.

Original: Following standard convention, we extract TF-IDF features on the training set, which gives us 18,828 instances and 146,016 features over 20 classes (see Appendix Section C.2 for details).
Revised: Following standard convention, we extract TF-IDF features on the training set, which gives us 18,828 instances and 146,016 features over 20 classes (see Appendix Section C.2 for details).

Original: C.2 20 Newsgroups Feature Extraction
Revised: C.2 20 Newsgroups Feature Extraction

Original: We use the following feature extraction pipeline from the sklearn repository.
Revised: We use the following feature extraction pipeline from the sklearn repository.

Original: We first use the
Revised: We first use the

Original: TfIdfVectorizer to compute the TF-IDF features on only the training set.
Revised: TfIdfVectorizer to compute the TF-IDF features on only the training set.

Original: Next, we transform the testing and validation sets using the learnt vectorizer, and rescale all the features within the range [0 , 1] .
Revised: Next, we transform the testing and validation sets using the learnt vectorizer, and rescale all the features within the range [0 , 1] .

Original: This gives us 18,828 instances and 146,016 features over 20 classes.
Revised: This gives us 18,828 instances and 146,016 features over 20 classes.

Original: C.3 Concept Bottleneck Implementation
Revised: C.3 Concept Bottleneck Implementation

Original: In an effort to scale interpretable approaches to problems beyond tabular datasets, we consider benchmark problems using the “Sequential Concept Bottleneck” framework of Koh et al.
Revised: In an effort to scale interpretable approaches to problems beyond tabular datasets, we consider benchmark problems using the “Sequential Concept Bottleneck” framework of Koh et al.

Original: The general idea of concept bottleneck (CB) models are to enable feature extraction necessary for problems in domains such as computer vision by learning a black-box model that can first predict human interpretable “concepts”, and then learn a simple, typically fully-transparent model on these concepts to predict the final category.
Revised: The general idea of concept bottleneck (CB) models are to enable feature extraction necessary for problems in domains such as computer vision by learning a black-box model that can first predict human interpretable “concepts”, and then learn a simple, typically fully-transparent model on these concepts to predict the final category.

Original: For example, in the problem of bird species classification from images [Wah et al., 2011], one can consider visible parts and attributes of the birds as intermediary “concepts” on which one can learn a linear classifier in order to provide reasonable classification performance while at the same time providing interpretable decisions.
Revised: For example, in the problem of bird species classification from images [Wah et al., 2011], one can consider visible parts and attributes of the birds as intermediary “concepts” on which one can learn a linear classifier in order to provide reasonable classification performance while at the same time providing interpretable decisions.

Original: Similar approaches have also been explored in the work of, e.g., Chen et al.
Revised: Similar approaches have also been explored in the work of, e.g., Chen et al.

Original: [2019], Ghorbani et al.
Revised: [2019], Ghorbani et al.

Original: [2019b] and Zhang et al.
Revised: [2019b] and Zhang et al.

Original: We consider concept bottleneck models for computer vision tasks.
Revised: We consider concept bottleneck models for computer vision tasks.

Original: C.3.1 CUB-200 Concept Bottleneck
Revised: C.3.1 CUB-200 Concept Bottleneck

Original: The Caltech-UCSD Birds (CUB)-200 [Wah et al., 2011] dataset is an image classification dataset that has images belonging to 200 different species of birds, a problem that is an instance of the broader problem of fine-grained visual classification .
Revised: The Caltech-UCSD Birds (CUB)-200 [Wah et al., 2011] dataset is an image classification dataset that has images belonging to 200 different species of birds, a problem that is an instance of the broader problem of fine-grained visual classification .

Original: The dataset is equipped with keypoint annotations of 15 bird parts, e.g., crown, wings, etc., and each part has ≥ 1 part-attribute labels, e.g., brown leg , buff neck , etc.
Revised: The dataset is equipped with keypoint annotations of 15 bird parts, e.g., crown, wings, etc., and each part has ≥ 1 part-attribute labels, e.g., brown leg , buff neck , etc.

Original: For the concept bottleneck model, we train a convolutional neural network architecture that predicts these intermediary concepts from the images, and then used this trained CNN model (with weights fixed) to extract features that are then trained using one of the several benchmark approaches (including SPAM).
Revised: For the concept bottleneck model, we train a convolutional neural network architecture that predicts these intermediary concepts from the images, and then used this trained CNN model (with weights fixed) to extract features that are then trained using one of the several benchmark approaches (including SPAM).

Original: One key difference between prior implementations and ours in this concept bottleneck
Revised: One key difference between prior implementations and ours in this concept bottleneck

Original: CNN architecture .
Revised: CNN architecture .

Original: We train a ResNet-

Revised: We train a ResNet-

Original: [He et al., 2016] on images resized to the size 448 ×resolution until the last pooling layer of the network ( pool5 ).
Revised: [He et al., 2016] on images resized to the size 448 ×resolution until the last pooling layer of the network ( pool5 ).

Original: The features are 2048-dimensional but ignoring the spatial pooling, are evaluated on each 14 × 14 patch within the image.
Revised: The features are 2048-dimensional but ignoring the spatial pooling, are evaluated on each 14 × 14 patch within the image.

Original: We train part-attribute linear classifiers on these spatially-arranged extracted features, where the part locations are used from the training annotations.
Revised: We train part-attribute linear classifiers on these spatially-arranged extracted features, where the part locations are used from the training annotations.

Original: Once the network has been trained, we run max-pooling over the 14 × 14 grid to extract features that are used in the second stage training.
Revised: Once the network has been trained, we run max-pooling over the 14 × 14 grid to extract features that are used in the second stage training.

Original: Note that this setup is different from the original Concept Bottleneck [Koh et al., 2020] architecture, as the original setup predicted the keypoints from the entire image, instead of a spatially-supervised version.
Revised: Note that this setup is different from the original Concept Bottleneck [Koh et al., 2020] architecture, as the original setup predicted the keypoints from the entire image, instead of a spatially-supervised version.

Original: We use this architecture as the original setup overfit very easily.
Revised: We use this architecture as the original setup overfit very easily.

Original: C.3.2 Common Objects Dataset Details
Revised: C.3.2 Common Objects Dataset Details

Original: We construct a dataset by collecting public images from Instagram 1 , involving common household objects (e.g., bed, stove, saucer, tables, etc.) with their bounding box annotations.
Revised: We construct a dataset by collecting public images from Instagram 1 , involving common household objects (e.g., bed, stove, saucer, tables, etc.) with their bounding box annotations.

Original: For each bounding box, we collect 2,618 interpretable annotations, namely, parts (e.g., leg, handle, top, etc.), attributes (e.g., colors, textures, shapes, etc.), and part-attribute compositions.
Revised: For each bounding box, we collect 2,618 interpretable annotations, namely, parts (e.g., leg, handle, top, etc.), attributes (e.g., colors, textures, shapes, etc.), and part-attribute compositions.

Original: The dataset has 2,645, training and 58,525 validation samples across 115 classes with 2,618 interpretable concepts.
Revised: The dataset has 2,645, training and 58,525 validation samples across 115 classes with 2,618 interpretable concepts.

Original: C.3.3 Hyperparameters
Revised: C.3.3 Hyperparameters

Original: Gradient-descent based methods are trained with the Adam with decoupled weight decay (AdamW) optimizer Loshchilov and Hutter [2017].
Revised: Gradient-descent based methods are trained with the Adam with decoupled weight decay (AdamW) optimizer Loshchilov and Hutter [2017].

Original: All methods are trained for 1000 epochs on the California
Revised: All methods are trained for 1000 epochs on the California

Original: Housing and FICO HELOC datasets, 100 epochs on iNaturalist Birds and Common Objects datasets, and 500 epochs on the CoverType, Newsgroups and CUB datasets.
Revised: Housing and FICO HELOC datasets, 100 epochs on iNaturalist Birds and Common Objects datasets, and 500 epochs on the CoverType, Newsgroups and CUB datasets.

Original: We use a cosine annealing learning rate across all gradient-descent based approaches and do a random hyperparameter search within the following specified grids.
Revised: We use a cosine annealing learning rate across all gradient-descent based approaches and do a random hyperparameter search within the following specified grids.

Original: Linear (Order 1 and Order 2).
Revised: Linear (Order 1 and Order 2).

Original: The learning rate range is [1e − 5 , 100] , and the weight decay interval is [1e − 13 , 1 . 0) .
Revised: The learning rate range is [1e − 5 , 100] , and the weight decay interval is [1e − 13 , 1 . 0) .

Original: We use the same interval for L 1 − regularized models as well.
Revised: We use the same interval for L 1 − regularized models as well.

Original: Deep Neural Networks.
Revised: Deep Neural Networks.

Original: We experiment with 3 neural network architectures that are all comprised of successive fully-connected layers.
Revised: We experiment with 3 neural network architectures that are all comprised of successive fully-connected layers.

Original: The first has 5 hidden layers of the shapes [128 , 128 , 64 , 64 , 64] units; the second is three hidden layers deep with [1024 , 512 , 512] units, and finally we have a shallow but wide architecture having 2 hidden layers with [2048 , 1024] units.
Revised: The first has 5 hidden layers of the shapes [128 , 128 , 64 , 64 , 64] units; the second is three hidden layers deep with [1024 , 512 , 512] units, and finally we have a shallow but wide architecture having 2 hidden layers with [2048 , 1024] units.

Original: We report the best accuracies achieved across all three architectures.
Revised: We report the best accuracies achieved across all three architectures.

Original: Moreover, we observed that increasing the depth with more layers did not improve accuracy.
Revised: Moreover, we observed that increasing the depth with more layers did not improve accuracy.

Original: We search the initial learning rate in the interval [10 − 6 , 10 . 0) , weight decay in [1e − 9 , 1 . 0) , dropout in the set { 0 , 0 . 05 , 0 . 1 , 0 . 2 , 0 .
Revised: We search the initial learning rate in the interval [10 − 6 , 10 . 0) , weight decay in [1e − 9 , 1 . 0) , dropout in the set { 0 , 0 . 05 , 0 . 1 , 0 . 2 , 0 .

Original: Neural Additive Models.
Revised: Neural Additive Models.

Original: We proceed with the standard proposed architectures in the original implementation of Agarwal et al.
Revised: We proceed with the standard proposed architectures in the original implementation of Agarwal et al.

Original: We experiment with two architectures: the first is an
Revised: We experiment with two architectures: the first is an

Original: MLP with 3 hidden layers and [64 , 64 , 32] units, and the second is an MLP with 1 hidden layer with 1 , 024 units and an ExU activation.
Revised: MLP with 3 hidden layers and [64 , 64 , 32] units, and the second is an MLP with 1 hidden layer with 1 , 024 units and an ExU activation.

Original: We tune the learning rate in the interval [1e − 5 , 1 . 0) , weight decay in [1e − 9 , 1 . 0) , output penalty weight in [1e − 6 , 100) , dropout and feature dropout in the set { 0 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } .
Revised: We tune the learning rate in the interval [1e − 5 , 1 . 0) , weight decay in [1e − 9 , 1 . 0) , output penalty weight in [1e − 6 , 100) , dropout and feature dropout in the set { 0 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } .

Original: EBMs.
Revised: EBMs.

Original: We describe the hyperparameter followed by the range of values we search from.
Revised: We describe the hyperparameter followed by the range of values we search from.

Original: • Maximum bins: { 8 , 16 , 32 , 64 , 128 , 256 , 512 } • Number of interactions: { 0 , 2 , 4 , 8 , 16 , 32 , 64 , 128 , 256 , 512 } ( 0 for EBMs, ≥ 0 for EB 2 Ms) • learning rate: [1e − 6 , 100) • maximum rounds: { 1000 , 2000 , 4000 , 8000 , 16000 } • minimum samples in a leaf node: { 1 , 2 , 4 , 8 , 10 , 15 , 20 , 25 , 50 } • maximum leaves: { 1 , 2 , 4 , 8 , 10 , 15 , 20 , 25 , 50 } • binning: {“quantile”, “uniform”, “quantile_humanized”} • inner/outer bags: { 1 , 2 , 4 , 8 , 16 , 32 , 64 , 128 } .
Revised: • Maximum bins: { 8 , 16 , 32 , 64 , 128 , 256 , 512 } • Number of interactions: { 0 , 2 , 4 , 8 , 16 , 32 , 64 , 128 , 256 , 512 } ( 0 for EBMs, ≥ 0 for EB 2 Ms) • learning rate: [1e − 6 , 100) • maximum rounds: { 1000 , 2000 , 4000 , 8000 , 16000 } • minimum samples in a leaf node: { 1 , 2 , 4 , 8 , 10 , 15 , 20 , 25 , 50 } • maximum leaves: { 1 , 2 , 4 , 8 , 10 , 15 , 20 , 25 , 50 } • binning: {“quantile”, “uniform”, “quantile_humanized”} • inner/outer bags: { 1 , 2 , 4 , 8 , 16 , 32 , 64 , 128 } .

Original: XGBoost.
Revised: XGBoost.

Original: We describe the hyperparameter followed by the range of values we search from.
Revised: We describe the hyperparameter followed by the range of values we search from.

Original: • number of estimators: { 1 , 2 , 4 , 8 , 10 , 20 , 50 , 100 , 200 , 250 , 500 , 1000 } • max-depth: {∞ , 2 , 5 , 10 , 20 , 25 , 50 , 100 , 2000 } • η : [0 . 0 , 1 . 0)
Revised: • number of estimators: { 1 , 2 , 4 , 8 , 10 , 20 , 50 , 100 , 200 , 250 , 500 , 1000 } • max-depth: {∞ , 2 , 5 , 10 , 20 , 25 , 50 , 100 , 2000 } • η : [0 . 0 , 1 . 0)

Original: • subsample : [0 . 0 , 1 . 0)
Revised: • subsample : [0 . 0 , 1 . 0)

Original: • colsample_bytree : [0 . 0 , 1 . 0)
Revised: • colsample_bytree : [0 . 0 , 1 . 0)

Original: CART.
Revised: CART.

Original: We describe the hyperparameter followed by the range of values we search from.
Revised: We describe the hyperparameter followed by the range of values we search from.

Original: • criterion: {“absolute error”, “friedman mse”, “squared error”, “poisson”} • splitter: {“best”, “random”} • minimum samples leaf: { 4 , 8 , 16 , 32 , 64 , 128 , 256 } • minimum samples split: { 4 , 8 , 16 , 32 , 64 , 128 , 256 }
Revised: • criterion: {“absolute error”, “friedman mse”, “squared error”, “poisson”} • splitter: {“best”, “random”} • minimum samples leaf: { 4 , 8 , 16 , 32 , 64 , 128 , 256 } • minimum samples split: { 4 , 8 , 16 , 32 , 64 , 128 , 256 }

Original: SPAM-Linear .
Revised: SPAM-Linear .

Original: For SPAM-Linear, there are 4 sets of hyperparameters: initial learning rate (tuned logarithmically from the set [10 − 7 , 10 2 ] ), weight decay (tuned logarithmically from the set [10 − 13 , 10 1 ] ), dropout for the singular values λ , which is tuned from the range [0 , 1] , and the vector of ranks r which consequently specifies the degree of the polynomial.
Revised: For SPAM-Linear, there are 4 sets of hyperparameters: initial learning rate (tuned logarithmically from the set [10 − 7 , 10 2 ] ), weight decay (tuned logarithmically from the set [10 − 13 , 10 1 ] ), dropout for the singular values λ , which is tuned from the range [0 , 1] , and the vector of ranks r which consequently specifies the degree of the polynomial.

Original: For degree 2 SPAM, we search ranks from the set S
Revised: For degree 2 SPAM, we search ranks from the set S

Original: = { [25] , [50] , [100] , [200] , [250] , [400] , [500] , [750] , [800] , [1000] , [1200] , [1400] , [1600] } and we for order 3, we search from the set S × S (i.e., product set of original range).
Revised: = { [25] , [50] , [100] , [200] , [250] , [400] , [500] , [750] , [800] , [1000] , [1200] , [1400] , [1600] } and we for order 3, we search from the set S × S (i.e., product set of original range).

Original: SPAM-Neural .
Revised: SPAM-Neural .

Original: For SPAM-Neural, in addition to the 4 original hyperparameters, we search from the NAM space identical to the range employed for NAM (please see above).
Revised: For SPAM-Neural, in addition to the 4 original hyperparameters, we search from the NAM space identical to the range employed for NAM (please see above).

Original: Regarding the original hyperparameters, we search: initial learning rate (tuned logarithmically from the set [10 − 7 , 10 2 ] ), weight decay (tuned logarithmically from the set [10 − 13 , 10 1 ] ), dropout for the singular values λ , which is tuned from the range [0 , 1] , and the vector of ranks r which consequently specifies the degree of the polynomial.
Revised: Regarding the original hyperparameters, we search: initial learning rate (tuned logarithmically from the set [10 − 7 , 10 2 ] ), weight decay (tuned logarithmically from the set [10 − 13 , 10 1 ] ), dropout for the singular values λ , which is tuned from the range [0 , 1] , and the vector of ranks r which consequently specifies the degree of the polynomial.

Original: For degree 2 SPAM, we search ranks from the set
Revised: For degree 2 SPAM, we search ranks from the set

Original: S = { [25] , [50] , [100] , [200] , [250] , [400] , [500] , [750] , [800] , [1000] , [1200] , [1400] , [1600] } and we for order 3, we search from the set S × S (i.e., product set of original range).
Revised: S = { [25] , [50] , [100] , [200] , [250] , [400] , [500] , [750] , [800] , [1000] , [1200] , [1400] , [1600] } and we for order 3, we search from the set S × S (i.e., product set of original range).

Original: D Human Subject Evaluation Details D.1 CUB-200 Dataset Details
Revised: D Human Subject Evaluation Details D.1 CUB-200 Dataset Details

Original: Figure 3 denotes a random sample of images that were shown to the participants.
Revised: Figure 3 denotes a random sample of images that were shown to the participants.

Original: We randomly sampled images from a pre-selected set of a pair of classes.
Revised: We randomly sampled images from a pre-selected set of a pair of classes.

Original: There were a total of 5 pairs, corresponding to the following classes:
Revised: There were a total of 5 pairs, corresponding to the following classes:

Original: “122.Harris_Sparrow", “060.Glaucous_winged_Gull" 2. “057.Rose_breasted_Grosbeak", “034.Gray_crowned_Rosy_Finch" 3.
Revised: “122.Harris_Sparrow", “060.Glaucous_winged_Gull" 2. “057.Rose_breasted_Grosbeak", “034.Gray_crowned_Rosy_Finch" 3.

Original: “066.Western_Gull", “035.Purple_Finch" 4. “064.Ring_billed_Gull", “008.Rhinoceros_Auklet" 5.
Revised: “066.Western_Gull", “035.Purple_Finch" 4. “064.Ring_billed_Gull", “008.Rhinoceros_Auklet" 5.

Original: “095.Baltimore_Oriole", “086.Pacific_Loon"
Revised: “095.Baltimore_Oriole", “086.Pacific_Loon"

Original: All images were sampled from the validation set.
Revised: All images were sampled from the validation set.

Original: D.2 Experiment Interface
Revised: D.2 Experiment Interface

Original: The experiment was implemented using the Mephisto [Miller et al., 2017] library in react.js .
Revised: The experiment was implemented using the Mephisto [Miller et al., 2017] library in react.js .

Original: Please see the subsequent pages for screenshots of the exact interface.
Revised: Please see the subsequent pages for screenshots of the exact interface.