Original: Neural Basis Models for Interpretability
Revised: Neural Basis Models for Interpretability

Original: Abhimanyu Dubey
Revised: Abhimanyu Dubey

Original: Meta AI
Revised: Meta AI

Original: Dhruv Mahajan
Revised: Dhruv Mahajan

Original: Meta AI
Revised: Meta AI

Original: Abstract
Revised: Abstract

Original: Due to the widespread use of complex machine learning models in real-world applications, it is becoming critical to explain model predictions.
Revised: Due to the widespread use of complex machine learning models in real-world applications, it is becoming critical to explain model predictions.

Original: However, these models are typically black-box deep neural networks, explained post-hoc via methods with known faithfulness limitations.
Revised: However, these models are typically black-box deep neural networks, explained post-hoc via methods with known faithfulness limitations.

Original: Generalized Additive Models (GAMs) are an inherently interpretable class of models that address this limitation by learning a non-linear shape function for each feature separately, followed by a linear model on top.
Revised: Generalized Additive Models (GAMs) are an inherently interpretable class of models that address this limitation by learning a non-linear shape function for each feature separately, followed by a linear model on top.

Original: However, these models are typically difficult to train, require numerous parameters, and are difficult to scale.
Revised: However, these models are typically difﬁcult to train, require numerous parameters, and are difﬁcult to scale.

Original: We propose an entirely new subfamily of
Revised: We propose an entirely new subfamily of

Original: GAMs that utilizes basis decomposition of shape functions.
Revised: GAMs that utilizes basis decomposition of shape functions.

Original: A small number of basis functions are shared among all features, and are learned jointly for a given task, thus making our model scale much better to large-scale data with high-dimensional features, especially when features are sparse.
Revised: A small number of basis functions are shared among all features, and are learned jointly for a given task, thus making our model scale much better to large-scale data with high-dimensional features, especially when features are sparse.

Original: We propose an architecture denoted as the Neural Basis Model (NBM) which uses a single neural network to learn these bases.
Revised: We propose an architecture denoted as the Neural Basis Model (NBM) which uses a single neural network to learn these bases.

Original: On a variety of tabular and image datasets, we demonstrate that for interpretable machine learning, NBMs are the state-of-the-art in accuracy, model size, and, throughput and can easily model all higher-order feature interactions.
Revised: On a variety of tabular and image datasets, we demonstrate that for interpretable machine learning, NBMs are the state-of-the-art in accuracy, model size, and, throughput and can easily model all higher-order feature interactions.

Original: Source code is available at github.com/facebookresearch/nbm-spam .
Revised: Source code is available at github.com/facebookresearch/nbm-spam .

Original:  Introduction
Revised:  Introduction

Original: Real world machine learning models [14, 61] are mostly used as a black-box , i.e. , it is very difficult to analyze and understand why a specific prediction was made.
Revised: Real world machine learning models [14, 61] are mostly used as a black-box , i.e. , it is very difﬁcult to analyze and understand why a speciﬁc prediction was made.

Original: In order to explain such black-box models, an instance-specific local interpretable model is often learned [38, 50].
Revised: In order to explain such black-box models, an instance-speciﬁc local interpretable model is often learned [38, 50].

Original: However, these approaches tend to be unstable and unfaithful [1, 52], i e. , they often misrepresent the model’s behavior.
Revised: However, these approaches tend to be unstable and unfaithful [1, 52], i.e. , they often misrepresent the model’s behavior.

Original: On the other hand, a family of models known as generalized additive models (GAMs) [24] have been used for decades as an inherently interpretable alternative to black-box models.
Revised: On the other hand, a family of models known as generalized additive models (GAMs) [24] have been used for decades as an inherently interpretable alternative to black-box models.

Original: GAMs learn a shape function for each feature independently and outputs of such functions are added (with a bias term) to obtain the final model prediction.
Revised: GAMs learn a shape function for each feature independently, and outputs of such functions are added (with a bias term) to obtain the ﬁnal model prediction.

Original: All models from this family share an important trait: the impact of any specific feature on the prediction does not rely on the other features, and can be understood by visualizing its corresponding shape function.
Revised: All models from this family share an important trait: the impact of any speciﬁc feature on the prediction does not rely on the other features, and can be understood by visualizing its corresponding shape function.

Original: Original GAMs [24] were fitted using splines, which have since been improved in explainable boosting machines (EBMs) [36] by fitting boosted decision trees, or very recently in neural additive models (NAMs) [2] by fitting deep neural networks (DNNs).
Revised: Original GAMs [24] were ﬁtted using splines, which have since been improved in explainable boosting machines (EBMs) [36] by ﬁtting boosted decision trees, or very recently in neural additive models (NAMs) [2] by ﬁtting deep neural networks (DNNs).

Original: A drawback for all the aforementioned approaches is that for each shape function, they require either millions of decision trees [36], or a DNN with tens of thousands of parameters [2], making them prohibitively expensive for learning datasets with a large number of features.
Revised: A drawback for all the aforementioned approaches is that for each shape function, they require either millions of decision trees [36], or a DNN with tens of thousands of parameters [2], making them prohibitively expensive for learning datasets with a large number of features.

Original: In this work, we propose a novel subfamily of GAMs, which, unlike previous approaches, learn to decompose each feature’s shape function into a small set of basis functions shared across all features.
Revised: In this work, we propose a novel subfamily of GAMs, which, unlike previous approaches, learn to decompose each feature’s shape function into a small set of basis functions shared across all features.

Original: The shape functions are fitted as the feature-specific linear combination of these shared bases, see Figure 1.
Revised: The shape functions are ﬁtted as the feature-speciﬁc linear combination of these shared bases, see Figure 1.

Original: At an abstract level, our approach is motivated by signal decomposition using traditional basis functions like the Fourier basis [9] or Legendre polynomials [45], where a weighted  combination of a few basis functions suffice to reconstruct complex signal shapes.
Revised: At an abstract level, our approach is motivated by signal decomposition using traditional basis functions like the Fourier basis [9] or Legendre polynomials [45], where a weighted 36th Conference on Neural Information Processing Systems (NeurIPS 2022). combination of a few basis functions sufﬁce to reconstruct complex signal shapes.

Original: However, in contrast to these approaches, our basis decomposition is not fixed a priori .
Revised: However, in contrast to these approaches, our basis decomposition is not ﬁxed a priori .

Original: In fact, it is learnt specifically for the prediction task.
Revised: In fact, it is learnt speciﬁcally for the prediction task.

Original: Consequently, we maintain the most important feature of GAMs, i.e. , their interpretability, as the contribution of single feature does not depend on the other features.
Revised: Consequently, we maintain the most important feature of GAMs, i.e. , their interpretability, as the contribution of single feature does not depend on the other features.

Original: At the same time, we gain scalability, as the number of basis functions needed in practice is much smaller than the number of input features.
Revised: At the same time, we gain scalability, as the number of basis functions needed in practice is much smaller than the number of input features.

Original: Moreover, we show that the usage of basis functions can increase computational efficiency by several orders of magnitude when the input features are sparse.
Revised: Moreover, we show that the usage of basis functions can increase computational efﬁciency by several orders of magnitude when the input features are sparse.

Original: Additionally, we propose an approach to learning the basis functions using a single DNN.
Revised: Additionally, we propose an approach to learning the basis functions using a single DNN.

Original: We call this solution the Neural Basis Model (NBM).
Revised: We call this solution the Neural Basis Model (NBM).

Original: Using neural networks allows for even higher scalability, as training and inference are performed on GPUs or other specialized hardware, and can be easily implemented in any deep learning framework using standard, already developed, building blocks.
Revised: Using neural networks allows for even higher scalability, as training and inference are performed on GPUs or other specialized hardware, and can be easily implemented in any deep learning framework using standard, already developed, building blocks.

Original: Our contributions are as follows: (i)
Revised: Our contributions are as follows: (i)

Original: We propose a novel subfamily of GAMs whose shape functions are composed of shared basis functions, and propose an approach to learn basis functions via DNNs, denoted as Neural Basis Model (NBM).
Revised: We propose a novel subfamily of GAMs whose shape functions are composed of shared basis functions, and propose an approach to learn basis functions via DNNs, denoted as Neural Basis Model (NBM).

Original: This architecture is suitable for mini-batch gradient descent training on GPUs and easy to plug-in into any deep learning framework.
Revised: This architecture is suitable for mini-batch gradient descent training on GPUs and easy to plug-in into any deep learning framework.

Original: (ii) We demonstrate that
Revised: (ii) We demonstrate that

Original: NBMs can be easily extended to incorporate pairwise functions, similar to GA 2 Ms [37], by learning another set of bases to model the higher order interactions.
Revised: NBMs can be easily extended to incorporate pairwise functions, similar to GA 2 Ms [37], by learning another set of bases to model the higher order interactions.

Original: This approach effectively only linearly increases the parameters, while other models such as EB 2 Ms [36, 37] and NA 2 Ms [2] suffer from quadratic growth of parameters, and often require heuristics and repeated training to select the most important interactions before learning [37].
Revised: This approach effectively only linearly increases the parameters, while other models such as EB 2 Ms [36, 37] and NA 2 Ms [2] suffer from quadratic growth of parameters, and often require heuristics and repeated training to select the most important interactions before learning [37].

Original: (iii) Through extensive evaluation of regression, binary classification, and multi-class classification, with both tabular and computer vision datasets, we show that NBMs and NB 2 Ms outperform state-of-the-art GAMs and GA 2 Ms, while scaling significantly better, i.e. , fitting much fewer parameters and having higher throughput.
Revised: (iii) Through extensive evaluation of regression, binary classiﬁcation, and multi-class classiﬁcation, with both tabular and computer vision datasets, we show that NBMs and NB 2 Ms outperform state-of-the-art GAMs and GA 2 Ms, while scaling signiﬁcantly better, i.e. , ﬁtting much fewer parameters and having higher throughput.

Original: For datasets with more than ten features, using NBMs result in around 5 × –50 × reduction in parameters over NAMs [2], and 4 × –7 × better throughput.
Revised: For datasets with more than ten features, using NBMs result in around 5 ⇥ –50 ⇥ reduction in parameters over NAMs [2], and 4 ⇥ –7 ⇥ better throughput.

Original: (iv) We propose an efficient extension of NBMs to sparse datasets with more than a hundred thousand features, where other GAMs do not scale at all.
Revised: (iv) We propose an efﬁcient extension of NBMs to sparse datasets with more than a hundred thousand features, where other GAMs do not scale at all.

Original:  Related work
Revised:  Related work

Original: Shape functions in GAMs [24, 64] have many different representations in the literature, including: splines [24], trees or random forests [36], deep neural networks [2], neural oblivious decision trees [12].
Revised: Shape functions in GAMs [24, 64] have many different representations in the literature, including: splines [24], trees or random forests [36], deep neural networks [2], neural oblivious decision trees [12].

Original: Popular methods of fitting GAMs are: backfitting [24], gradient boosting [36], or minibatch gradient descent [2, 12].
Revised: Popular methods of ﬁtting GAMs are: backﬁtting [24], gradient boosting [36], or minibatch gradient descent [2, 12].

Original: Our work falls under the GAM umbrella, however, it differs from these approaches by not learning shape functions independently, but rather, learning a set of shared basis functions that are used to compose each shape function.
Revised: Our work falls under the GAM umbrella, however, it differs from these approaches by not learning shape functions independently, but rather, learning a set of shared basis functions that are used to compose each shape function.

Original: The bases themselves can be complex non-linear functions that operate on one feature at a time, e.g. , decision trees or splines, however, to keep the scope of the work concise and to ensure straightforward scalability, we represent them with deep neural networks and use mini-batch stochastic gradient descent to learn the bases.
Revised: The bases themselves can be complex non-linear functions that operate on one feature at a time, e.g. , decision trees or splines, however, to keep the scope of the work concise and to ensure straightforward scalability, we represent them with deep neural networks and use mini-batch stochastic gradient descent to learn the bases.

Original: This makes our work most closely related to neural additive models (NAMs) [2], however, crucially, our method learns far fewer parameters by sharing bases compared to NAMs.
Revised: This makes our work most closely related to neural additive models (NAMs) [2], however, crucially, our method learns far fewer parameters by sharing bases compared to NAMs.

Original: Methods have been proposed to model pairwise interactions [12, 37, 58], and they are commonly denoted as GA 

Revised: Methods have been proposed to model pairwise interactions [12, 37, 58], and they are commonly denoted as GA

Original: Ms. However, they usually require sophisticated feature selection using: heuristics [37], back-propagation [12], or, two-stage iterative training [58].
Revised: Ms. However, they usually require sophisticated feature selection using: heuristics [37], back-propagation [12], or, two-stage iterative training [58].

Original: We note that one of the main goals of our work is to analyze the scalability of the newly proposed NBM architecture in extreme scenarios, hence, we do not apply feature selection algorithms.
Revised: We note that one of the main goals of our work is to analyze the scalability of the newly proposed NBM architecture in extreme scenarios, hence, we do not apply feature selection algorithms.

Original: That being said, before-mentioned algorithms are complementary and can be applied to NB 2 Ms as well.
Revised: That being said, before-mentioned algorithms are complementary and can be applied to NB 2 Ms as well.

Original: Finally, GAMs are a popular choice in high-risk applications for healthcare [11, 56], finance [6], forensics [55], etc.
Revised: Finally, GAMs are a popular choice in high-risk applications for healthcare [11, 56], ﬁnance [6], forensics [55], etc.

Original: Another line of work applicable to these domains are interpretable surrogate models such as LIME [50], SHAP [38], tree-based surrogates [5, 57], that give a post-hoc explanation of a high-complexity model.
Revised: Another line of work applicable to these domains are interpretable surrogate models such as LIME [50], SHAP [38], tree-based surrogates [5, 57], that give a post-hoc explanation of a high-complexity model.

Original: However, there are almost no theoretical guarantees that the simple surrogate model is highly representative of the more complex model [1].
Revised: However, there are almost no theoretical guarantees that the simple surrogate model is highly representative of the more complex model [1].

Original: This issue is completely resolved when using inherently transparent models such as NBMs.
Revised: This issue is completely resolved when using inherently transparent models such as NBMs.

Original: We hope that our approach sparks wider usage of GAMs in mission-critical applications with large-scale data, where the ability to interpret, understand, and correct the model is of utmost importance.
Revised: We hope that our approach sparks wider usage of GAMs in mission-critical applications with large-scale data, where the ability to interpret, understand, and correct the model is of utmost importance.

Original:  Method 3.1 Background
Revised:  Method 3.1 Background

Original: Generalized Additive Model (GAM) [24].
Revised: Generalized Additive Model (GAM) [24].

Original: Given a D -dimensional interpretable input x = { x i } Di =1 , x ∈ R D , a target label y , a link function g ( e.g. , logistic function), a univariate shape function f i , corresponding to the input feature x i , a bivariate shape function f ij , corresponding to the feature interaction, and a bias term f 0 , the prediction function in GAM and GA 2 M is expressed as 
Revised: Given a D -dimensional interpretable input x = { x i } Di =1 , x 2 R D , a target label y , a link function g ( e.g. , logistic function), a univariate shape function f i , corresponding to the input feature x i , a bivariate shape function f ij , corresponding to the feature interaction, and a bias term f 0 , the prediction function in GAM and GA 2 M is expressed as GAM [Equation] GA

Original: [Equation]
Revised: [Equation]

Revised: X

Revised: [Equation]

Revised: f

Revised: [Equation]

Revised: X

Revised: [Equation]

Revised: D

Revised: X

Original:  Interpreting GAMs is straightforward as the impact of a feature on the prediction does not rely on the other features, and can be understood by visualizing its corresponding shape function, e.g. , plotting x i on the x -axis
Revised: [Equation] >i f [Equation] Interpreting GAMs is straightforward as the impact of a feature on the prediction does not rely on the other features, and can be understood by visualizing its corresponding shape function, e.g. , plotting x i on the x -axis

Original: and f i ( x i ) on the y -axis.
Revised: and f i ( x i ) on the y -axis.

Original: A certain level of interpretability is sacrificed for accuracy when modeling interactions, as f ij shape functions are harder to visualize.
Revised: A certain level of interpretability is sacriﬁced for accuracy when modeling interactions, as f ij shape functions are harder to visualize.

Original: Shape function visualization through heatmaps [12, 37] is commonly used towards that purpose.
Revised: Shape function visualization through heatmaps [12, 37] is commonly used towards that purpose.

Original: Note that, the graphs visualizations of GAMs are an exact description of how GAMs compute a prediction.
Revised: Note that, the graphs visualizations of GAMs are an exact description of how GAMs compute a prediction.

Original: 3.2 Our model architecture
Revised: 3.2 Our model architecture

Original: We observe that, typically, input features of high-dimensional data are correlated with each other.
Revised: We observe that, typically, input features of high-dimensional data are correlated with each other.

Original: As a result, it should be possible to decompose each shape function f
Revised: As a result, it should be possible to decompose each shape function f

Original: i into a small set of basis functions shared among all the features.
Revised: i into a small set of basis functions shared among all the features.

Original: This is the core idea behind our approach.
Revised: This is the core idea behind our approach.

Original: Neural Basis Model (NBM).
Revised: Neural Basis Model (NBM).

Original: We propose to represent shape functions
Revised: We propose to represent shape functions

Original: f i as
Revised: f i as

Original: [Equation]
Revised: [Equation]

Original: where { h 1 , h 2 , ..., h B } represents a set of B shared basis functions that are independent of feature indices, and coefficients a ik are the projection of each feature to the shared bases.
Revised: where { h 1 , h 2 , ..., h B } represents a set of B shared basis functions that are independent of feature indices, and coefﬁcients a ik are the projection of each feature to the shared bases.

Original: We additionally propose to learn basis functions using a DNN, i.e. , a single one -input B -output multi-layer perceptron (MLP) for all { h k ; k = 1 , . . .
Revised: We additionally propose to learn basis functions using a DNN, i.e. , a single one -input B -output multi-layer perceptron (MLP) for all { h k ; k = 1 , . . .

Original: , B } .
Revised: , B } .

Original: The resulting architecture is shown in Figure 1.
Revised: The resulting architecture is shown in Figure 1.

Original: Multi-class / multi-task architecture.
Revised: Multi-class / multi-task architecture.

Original: Let l correspond to the target class y l in the multi-class setting.
Revised: Let l correspond to the target class y l in the multi-class setting.

Original: Similar to Equation 1, the prediction function g l for class y l in GAMs can be written as:
Revised: Similar to Equation 1, the prediction function g l for class y l in GAMs can be written as:

Original: [Equation]
Revised: [Equation]

Original: where feature shape functions f i ( x i ) are shared among the classes and are linearly combined using class specific weights w il .
Revised: where feature shape functions f i ( x i ) are shared among the classes and are linearly combined using class speciﬁc weights w il .

Original: Combining Equations 2 and 3, multi-class NBM can be represented as:
Revised: Combining Equations 2 and 3, multi-class NBM can be represented as:

Original: [Equation]
Revised: [Equation]

Original: Extension to NB 2 M. Similar to NBM, we represent GA 2 M shape functions f ij in Equation 1 as:
Revised: Extension to NB 2 M. Similar to NBM, we represent GA 2 M shape functions f ij in Equation 1 as:

Original: [Equation]
Revised: [Equation]

Original: where { u 1 , u 2 , ..., u B } represents a set of B shared bi-variate basis functions that are independent of feature indices and coefficients b ijk are the projection of pair-wise features to the shared bases.
Revised: where { u 1 , u 2 , ..., u B } represents a set of B shared bi-variate basis functions that are independent of feature indices and coefﬁcients b ijk are the projection of pair-wise features to the shared bases.

Original: We learn an additional two -input B -output MLP for all { u k ; k = 1 , . . .
Revised: We learn an additional two -input B -output MLP for all { u k ; k = 1 , . . .

Original: , B } to learn the bases.
Revised: , B } to learn the bases.

Original: Extension to multi-class setting can be done in the same way as for NBMs.
Revised: Extension to multi-class setting can be done in the same way as for NBMs.

Original: Sparse architecture.
Revised: Sparse architecture.

Original: Typically, datasets with high-dimensional features are sparse in nature.
Revised: Typically, datasets with high-dimensional features are sparse in nature.

Original: For example, in the Newsgroups dataset [32], news articles are represented by tf-idf features, and, for a given instance, most of the features are absent due to the vocabulary being of the order of 100K words.
Revised: For example, in the Newsgroups dataset [32], news articles are represented by tf-idf features, and, for a given instance, most of the features are absent due to the vocabulary being of the order of 100K words.

Original: Since NBM uses a single DNN to learn all the bases, we can simply append the single value representing the absent feature to the batch, to compute the corresponding basis function values.
Revised: Since NBM uses a single DNN to learn all the bases, we can simply append the single value representing the absent feature to the batch, to compute the corresponding basis function values.

Original: The subsequent linear projection to feature indices via a ik is a computationally inexpensive operation.
Revised: The subsequent linear projection to feature indices via a ik is a computationally inexpensive operation.

Original: In contrast, typical GAMs ( e.g. , Neural Additive Model (NAM) [42]) need to pass the absent value through every shape function f
Revised: In contrast, typical GAMs ( e.g. , Neural Additive Model (NAM) [42]) need to pass the absent value through every shape function f

Original: i which makes it compute-intensive as well as difficult to implement.
Revised: i which makes it compute-intensive as well as difﬁcult to implement.

Original: Training and regularization.
Revised: Training and regularization.

Original: We use mean squared error (MSE) for regression, and cross-entropy loss for classification.
Revised: We use mean squared error (MSE) for regression, and cross-entropy loss for classiﬁcation.

Original: To avoid overfitting, we use the following regularization techniques: (i)
Revised: To avoid overﬁtting, we use the following regularization techniques: (i)

Original: L 2 -normalization (weight decay) [31] of parameters; (ii) batch-norm [28] and dropout [54] on hidden layers of the basis functions network; (iii) an L 2 -normalization penalty on the outputs f i to incentivize fewer strong feature contributions, as done in [2]; (iv) basis dropout to randomly drop individual basis functions in order to decorrelate them.
Revised: L 2 -normalization (weight decay) [31] of parameters; (ii) batch-norm [28] and dropout [54] on hidden layers of the basis functions network; (iii) an L 2 -normalization penalty on the outputs f i to incentivize fewer strong feature contributions, as done in [2]; (iv) basis dropout to randomly drop individual basis functions in order to decorrelate them.

Original: Similar techniques have been used for other GAMs [2, 12].
Revised: Similar techniques have been used for other GAMs [2, 12].

Original: Selecting the number of bases.
Revised: Selecting the number of bases.

Original: One can use the theory of Reproducing Hilbert Kernel Spaces (RKHS, [7]) to devise a heuristic for selecting the number of bases B .
Revised: One can use the theory of Reproducing Hilbert Kernel Spaces (RKHS, [7]) to devise a heuristic for selecting the number of bases B .

Original: Specifically, we demonstrate that any NBM model lies on a subspace within the space spanned by a complete GAM if the
Revised: Speciﬁcally, we demonstrate that any NBM model lies on a subspace within the space spanned by a complete GAM if the

Original: GAM shape functions reside within a ball in an RKHS.
Revised: GAM shape functions reside within a ball in an RKHS.

Original: Assuming a regularity property in the data distribution, one can then demonstrate that B
Revised: Assuming a regularity property in the data distribution, one can then demonstrate that B

Original: = O (log D ) bases are sufficient to obtain competitive performance.
Revised: = O (log D ) bases are sufﬁcient to obtain competitive performance.

Original: We present this formally in Appendix Sec.
Revised: We present this formally in Appendix Sec.

Original: A.5.
Revised: A.5.

Original: This provides the alternate interpretation of NBM as learning a “principal components” decomposition in the L 2 − space of functions, as we learn a set of (preferably orthogonal) basis functions to approximate the decision boundary.
Revised: This provides the alternate interpretation of NBM as learning a “principal components” decomposition in the L 2  space of functions, as we learn a set of (preferably orthogonal) basis functions to approximate the decision boundary.

Original: 3.3 Discussion
Revised: 3.3 Discussion

Original: In this section, we contrast NBMs with closely related GAMs: Neural Additive Models (NAMs) [2].
Revised: In this section, we contrast NBMs with closely related GAMs: Neural Additive Models (NAMs) [2].

Original: Neural Additive Model (NAM) [2].
Revised: Neural Additive Model (NAM) [2].

Original: NAMs learn a linear combination of networks that each attend to a single input feature: each f i in (1) is parametrized by a deep neural network, i.e. , a one -input one -output multi-layer perceptron (MLP).
Revised: NAMs learn a linear combination of networks that each attend to a single input feature: each f i in (1) is parametrized by a deep neural network, i.e. , a one -input one -output multi-layer perceptron (MLP).

Original: These MLPs are trained jointly using backpropagation and can learn arbitrarily complex shape functions.
Revised: These MLPs are trained jointly using backpropagation and can learn arbitrarily complex shape functions.

Original: [Figure]| NA
Revised: [Figure]

Original: M | / |
Revised: [Equation] [Figure]

Original: N B
Revised: [Equation] [Figure] [Equation]

Original: M |
Revised: [Figure]

Original: Number of parameters.
Revised: Number of parameters.

Original: We compare number of weight parameters needed to learn NAM vs.
Revised: We compare number of weight parameters needed to learn NAM vs.

Original: NBM for the binary-classification task.
Revised: NBM for the binary-classiﬁcation task.

Original: See Appendix Section A.3 for multi-class analysis.
Revised: See Appendix Section A.3 for multi-class analysis.

Original: Let us denote with M the number of parameters in
Revised: Let us denote with M the number of parameters in

Original: MLP for each feature in NAM, and with N the number of parameters in MLP for bases in NBM.
Revised: MLP for each feature in NAM, and with N the number of parameters in MLP for bases in NBM.

Original: In most experiments the optimal NAM has 3 hidden layers with 64, 64 and 32 units ( M = 6401 ), and, NBM has 3 hidden layers with 256, 128, 128 units ( N = 62820 ) and B = 100 basis functions.
Revised: In most experiments the optimal NAM has 3 hidden layers with 64, 64 and 32 units ( M = 6401 ), and, NBM has 3 hidden layers with 256, 128, 128 units ( N = 62820 ) and B = 100 basis functions.

Original: Then the ratio of number of parameters in
Revised: Then the ratio of number of parameters in

Original: NAM vs. NBM is given by,
Revised: NAM vs. NBM is given by,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Figure 2 shows this ratio for different values of feature dimensionality D .
Revised: Figure 2 shows this ratio for different values of feature dimensionality D .

Original: For D = 10 , NBMs and NAMs have roughly equal number of parameters.
Revised: For D = 10 , NBMs and NAMs have roughly equal number of parameters.

Original: For most textual and vision datasets, feature dimensionality is significantly higher, thus leading to 10 × –50 × reduction in parameters.
Revised: For most textual and vision datasets, feature dimensionality is signiﬁcantly higher, thus leading to 10 ⇥ –50 ⇥ reduction in parameters.

Original: We also observe that as a result, for many datasets, specialized regularization discussed in the previous section gives incremental gains for NBMs, while they are very crucial for NAMs to give good performance.
Revised: We also observe that as a result, for many datasets, specialized regularization discussed in the previous section gives incremental gains for NBMs, while they are very crucial for NAMs to give good performance.

Original: Additional analysis on number of parameters is given in Section 4.4 and Table 2.
Revised: Additional analysis on number of parameters is given in Section 4.4 and Table 2.

Original: Throughput.
Revised: Throughput.

Original: One of the main challenges in approach like NAMs is the low throughput rate, that is the number of data instances processed per second, which directly affects the training speed.
Revised: One of the main challenges in approach like NAMs is the low throughput rate, that is the number of data instances processed per second, which directly affects the training speed.

Original: Since
Revised: Since

Original: NAMs use separate MLP per dimension, it is much more challenging to implement efficiently.
Revised: NAMs use separate MLP per dimension, it is much more challenging to implement efﬁciently.

Original: NBMs on the other hand are much more efficient since feature specific linear layer on the top of bases is very fast.
Revised: NBMs on the other hand are much more efﬁcient since feature speciﬁc linear layer on the top of bases is very fast.

Original: For example, for datasets with around hundred features, the original NAM implementation [2] is around 20 × slower in training compared to NBMs.
Revised: For example, for datasets with around hundred features, the original NAM implementation [2] is around 20 ⇥ slower in training compared to NBMs.

Original: We optimized the speed of NAMs using group convolutions (see Appendix Section A.3), but even this optimized version is around 5 × slower.
Revised: We optimized the speed of NAMs using group convolutions (see Appendix Section A.3), but even this optimized version is around 5 ⇥ slower.

Original: Stability.
Revised: Stability.

Original: The interpretability of models and their explanations is tightly coupled to their stability.
Revised: The interpretability of models and their explanations is tightly coupled to their stability.

Original: For example, post-hoc explanations of black-box models are known to be unstable, and produce different explanations for small input changes [53].
Revised: For example, post-hoc explanations of black-box models are known to be unstable, and produce different explanations for small input changes [53].

Original: GAMs are exactly explained by visualizing shape functions that represent the model, however, a desirable property is to have similar shape functions when repeatedly training the model on the same data while varying random initialization.
Revised: GAMs are exactly explained by visualizing shape functions that represent the model, however, a desirable property is to have similar shape functions when repeatedly training the model on the same data while varying random initialization.

Original: Because of the over-parameterization in NAMs, we observe that shape functions of different runs are often unstable ( i.e. they often diverge), especially for feature regions where the data density is low.
Revised: Because of the over-parameterization in NAMs, we observe that shape functions of different runs are often unstable ( i.e. they often diverge), especially for feature regions where the data density is low.

Original: On the other hand, NBMs train a single set of shared bases for the entire dataset, which makes shape functions significantly more stable, see Section 4.6 and Figure 3.
Revised: On the other hand, NBMs train a single set of shared bases for the entire dataset, which makes shape functions signiﬁcantly more stable, see Section 4.6 and Figure 3.

Original: NA 2 M vs. NB 2 M. NAMs can trivially be extended to NA 2 Ms by learning additional two -input one -output MLPs for each pairwise feature interaction f ij .
Revised: NA 2 M vs. NB 2 M. NAMs can trivially be extended to NA 2 Ms by learning additional two -input one -output MLPs for each pairwise feature interaction f ij .

Original: Since the numbers of parameters grow quadratically, this setting further exaggerates the parameters and throughput discrepancies.
Revised: Since the numbers of parameters grow quadratically, this setting further exaggerates the parameters and throughput discrepancies.

Original: In fact, as we show in Section 4.4, for high dimensional datasets the NA 2 M approach does not scale at all.
Revised: In fact, as we show in Section 4.4, for high dimensional datasets the NA 2 M approach does not scale at all.

Original:  Experiments 4.1 Datasets
Revised:  Experiments 4.1 Datasets

Original: Tabular datasets.
Revised: Tabular datasets.

Original: We report performance on CA Housing [10, 46], FICO [22], CoverType [8, 16, 20], and Newsgroups [32, 43] tabular datasets.
Revised: We report performance on CA Housing [10, 46], FICO [22], CoverType [8, 16, 20], and Newsgroups [32, 43] tabular datasets.

Original: We perform one-hot encoding for categorical features, and min-max scaling of features to [0 , 1] range.
Revised: We perform one-hot encoding for categorical features, and min-max scaling of features to [0 , 1] range.

Original: Data is split to have 70 / 10 / 20 ratio for training, validation, and, testing, respectively; except for Newsgroups where test split is fixed, so we only split the train part to 85 / 15 ratio for train and validation.
Revised: Data is split to have 70 / 10 / 20 ratio for training, validation, and, testing, respectively; except for Newsgroups where test split is ﬁxed, so we only split the train part to 85 / 15 ratio for train and validation.

Original: We also report performance on MIMIC-II [41, 51], Credit [17, 19], Click [15], Epsilon [21],
Revised: We also report performance on MIMIC-II [41, 51], Credit [17, 19], Click [15], Epsilon [21],

Original: Higgs [3, 26], Microsoft [40, 49], Yahoo [60], and Year [66] tabular datasets.
Revised: Higgs [3, 26], Microsoft [40, 49], Yahoo [60], and Year [66] tabular datasets.

Original: For these datasets, we follow [12, 47] to use the same training, validation, and, testing splits, and to perform the same feature normalization: target encoding for categorical features, quantile transformation withbins for all features to Gaussian distribution.
Revised: For these datasets, we follow [12, 47] to use the same training, validation, and, testing splits, and to perform the same feature normalization: target encoding for categorical features, quantile transformation withbins for all features to Gaussian distribution.

Original: Additional details for all tabular datasets are given in Table 1 and Appendix Section A.1.
Revised: Additional details for all tabular datasets are given in Table 1 and Appendix Section A.1.

Original: Image datasets (classification).
Revised: Image datasets (classiﬁcation).

Original: We experiment with two bird classification datasets: CUB [18, 62] and iNaturalist Birds [27, 59].
Revised: We experiment with two bird classiﬁcation datasets: CUB [18, 62] and iNaturalist Birds [27, 59].

Original: CUB images are annotated with keypoint locations of 15 bird parts, and each location is associated with one or more part-attribute labels.
Revised: CUB images are annotated with keypoint locations of 15 bird parts, and each location is associated with one or more part-attribute labels.

Original: iNaturalist Birds contains significantly more bird classes and more challenging scenes compared to CUB, but lacks keypoint annotations.
Revised: iNaturalist Birds contains signiﬁcantly more bird classes and more challenging scenes compared to CUB, but lacks keypoint annotations.

Original: We perform the concept-bottleneck-style [29] interpretable pre-processing: (i) Images are randomly cropped and resized to 448 × 448 size, and passed through a ResNet50 model [25] until the last pooling layer to extract 2048-D features on the 14 × 14 spatial grid.
Revised: We perform the concept-bottleneck-style [29] interpretable pre-processing: (i) Images are randomly cropped and resized to 448 ⇥ 448 size, and passed through a ResNet50 model [25] until the last pooling layer to extract 2048-D features on the 14 ⇥ 14 spatial grid.

Original: (ii) Part-attribute linear classifiers are trained on the extracted features using part locations and part-attribute labels from CUB training split.
Revised: (ii) Part-attribute linear classiﬁers are trained on the extracted features using part locations and part-attribute labels from CUB training split.

Original: (iii) Max-pooling of part-attribute scores over spatial locations is performed to extract 278 interpretable features ( e.g. , orange legs, striped wings, needle-shaped beak) for each image in both CUB and iNaturalist, using the part-attribute classifiers trained on CUB only.
Revised: (iii) Max-pooling of part-attribute scores over spatial locations is performed to extract 278 interpretable features ( e.g. , orange legs, striped wings, needle-shaped beak) for each image in both CUB and iNaturalist, using the part-attribute classiﬁers trained on CUB only.

Original: Splits are preset and results are reported on the validation split, which is a common practice in the computer vision community.
Revised: Splits are preset and results are reported on the validation split, which is a common practice in the computer vision community.

Original: Additional details are given in Table 1 and Appendix Section A.1.
Revised: Additional details are given in Table 1 and Appendix Section A.1.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Image dataset (object detection).
Revised: Image dataset (object detection).

Original: For this task we use a proprietary object detection dataset, denoted as Common Objects , that contains 114 common objects, (plus a background class) with bounding box locations, 200 parts and 54 attributes.
Revised: For this task we use a proprietary object detection dataset, denoted as Common Objects , that contains 114 common objects, (plus a background class) with bounding box locations, 200 parts and 54 attributes.

Original: Each box for each image is pre-processed using compositions of parts and attributes to extract 2,618 interpretable features and 100k pairwise feature interactions.
Revised: Each box for each image is pre-processed using compositions of parts and attributes to extract 2,618 interpretable features and 100k pairwise feature interactions.

Original: For the purpose of evaluating models in this paper, one data instance in this dataset is equivalent to one bounding box.
Revised: For the purpose of evaluating models in this paper, one data instance in this dataset is equivalent to one bounding box.

Original: As with previous computer vision datasets, results are reported on the validation split.
Revised: As with previous computer vision datasets, results are reported on the validation split.

Original: Additional details are given in Table 1 and Appendix Section A.1.
Revised: Additional details are given in Table 1 and Appendix Section A.1.

Original: 4.2 Baselines
Revised: 4.2 Baselines

Original: We implement the following baselines in PyTorch [48], and train using mini-batch gradient descent:
Revised: We implement the following baselines in PyTorch [48], and train using mini-batch gradient descent:

Original: Linear.
Revised: Linear.

Original: Linear / logistic regression are interpretable models that make a prediction decision based on the value of a linear combination of the features.
Revised: Linear / logistic regression are interpretable models that make a prediction decision based on the value of a linear combination of the features.

Original: NAM [2].
Revised: NAM [2].

Original: We experiment with two proposed NAM architectures [2]: (i) MLPs containing 3 hidden layers with 64, 64 and 32 units and ReLU [23] activation, and (ii) single hidden layer MLPs with 1,024 ExU units and ReLU-1 activation.
Revised: We experiment with two proposed NAM architectures [2]: (i) MLPs containing 3 hidden layers with 64, 64 and 32 units and ReLU [23] activation, and (ii) single hidden layer MLPs with 1,024 ExU units and ReLU-1 activation.

Original: We reimplement the original NAM implementation [42] (details in Appendix Section A.3) achieving around × 2– × 10 speedup at training, depending on the dataset.
Revised: We reimplement the original NAM implementation [42] (details in Appendix Section A.3) achieving around ⇥ 2– ⇥ 10 speedup at training, depending on the dataset.

Original: Finally, we extend the implementation to NA 2 Ms, as well.
Revised: Finally, we extend the implementation to NA 2 Ms, as well.

Original: We released our NAM implementation together with the rest of our code at github.com/facebookresearch/nbm-spam .
Revised: We released our NAM implementation together with the rest of our code at github.com/facebookresearch/nbm-spam .

Original: MLP.
Revised: MLP.

Original: Multi-layer perceptron (MLP) is a non-interpretable black-box model capturing high-order interaction between the features: g ( x )=
Revised: Multi-layer perceptron (MLP) is a non-interpretable black-box model capturing high-order interaction between the features: g ( x )=

Original: f ( x 1 , x 2 , . . .
Revised: f ( x 1 , x 2 , . . .

Original: , x D ) .
Revised: , x D ) .

Original: For most datasets, MLP sets the upper bound on the performance, and gives an idea of the trade-off between accuracy and interpretability.
Revised: For most datasets, MLP sets the upper bound on the performance, and gives an idea of the trade-off between accuracy and interpretability.

Original: We experiment with the following architectures: (i) 5 hidden layers with 128, 128, 64, 64, 64 units; (ii) 3 hidden layers with 1024, 512, 512 units; (iii) 2 hidden layers with 2048, 1024 units.
Revised: We experiment with the following architectures: (i) 5 hidden layers with 128, 128, 64, 64, 64 units; (ii) 3 hidden layers with 1024, 512, 512 units; (iii) 2 hidden layers with 2048, 1024 units.

Original: We have observed that increasing the depth by adding more layers for any of the three architectures has no additional accuracy gain.
Revised: We have observed that increasing the depth by adding more layers for any of the three architectures has no additional accuracy gain.

Original: For all datasets, we report the best performing architecture across the three.
Revised: For all datasets, we report the best performing architecture across the three.

Original: For the following baselines, we use the available implementations:
Revised: For the following baselines, we use the available implementations:

Original: EBM [36, 37].
Revised: EBM [36, 37].

Original: Explainable Boosting Machines (EBMs) are another state-of-the-art GAM which use gradient boosting of millions of shallow trees to learn a shape function for each input feature.
Revised: Explainable Boosting Machines (EBMs) are another state-of-the-art GAM which use gradient boosting of millions of shallow trees to learn a shape function for each input feature.

Original: These models support automatic pairwise interactions through their EB 2 M implementation, but only for regression and binary classification tasks.
Revised: These models support automatic pairwise interactions through their EB 2 M implementation, but only for regression and binary classiﬁcation tasks.

Original: We use the interpretml library [44].
Revised: We use the interpretml library [44].

Original: XGBoost [13].
Revised: XGBoost [13].

Original: EXtreme Gradient Boosted trees (XGBoost) are another non-interpretable black-box model that learn high-order feature interactions.
Revised: EXtreme Gradient Boosted trees (XGBoost) are another non-interpretable black-box model that learn high-order feature interactions.

Original: We use the xgboost library [13].
Revised: We use the xgboost library [13].

Original: [Table]
Revised: [Table]

Original: 4.3 Implementation details
Revised: 4.3 Implementation details

Original: NBM.
Revised: NBM.

Original: We use the following architecture for NBMs and NB 2 Ms: MLP containing 3 hidden layers with 256, 128, and 128 units, ReLU [23], B = 100 basis outputs for NBMs and B = 200 for NB 2 Ms.
Revised: We use the following architecture for NBMs and NB 2 Ms: MLP containing 3 hidden layers with 256, 128, and 128 units, ReLU [23], B = 100 basis outputs for NBMs and B = 200 for NB 2 Ms.

Original: Source code is available at github.com/facebookresearch/nbm-spam .
Revised: Source code is available at github.com/facebookresearch/nbm-spam .

Original: Training details.
Revised: Training details.

Original: Linear, NAM, NBM
Revised: Linear, NAM, NBM

Original: [Table]
Revised: [Table]

Original: and MLP models are trained using the Adam with decoupled weight decay (AdamW) optimizer [35], on 8 × V100 GPU machines with 32 GB memory
Revised: and MLP models are trained using the Adam with decoupled weight decay (AdamW) optimizer [35], on 8 ⇥ V100 GPU machines with 32 GB memory

Original: [Table]
Revised: [Table]

Original: and a batch size of at most 1024 per GPU (divided by 2 every time a batch cannot fit in the memory).
Revised: and a batch size of at most 1024 per GPU (divided by 2 every time a batch cannot ﬁt in the memory).

Original: We train for 1,000, 500, 100, or, 50 epochs, depending on the size and feature dimensionality of the dataset.
Revised: We train for 1,000, 500, 100, or, 50 epochs, depending on the size and feature dimensionality of the dataset.

Original: The learning rate is decayed with cosine annealing [34] from the starting value until zero.
Revised: The learning rate is decayed with cosine annealing [34] from the starting value until zero.

Original: For
Revised: For

Original: NBMs on all datasets, we tune the starting learning rate in the continuous interval [1e − 5 , 1 . 0) , weight decay in the interval [1e − 10 , 1 . 0) , output penalty coefficient in the interval [1e − 7 , 100) , dropout and basis dropout coefficients in the discrete set { 0 , 0 .
Revised: NBMs on all datasets, we tune the starting learning rate in the continuous interval [1e  5 , 1 . 0) , weight decay in the interval [1e  10 , 1 . 0) , output penalty coefﬁcient in the interval [1e  7 , 100) , dropout and basis dropout coefﬁcients in the discrete set { 0 , 0 .

Original: We find optimal hyper-parameters using validation set and random search.
Revised: We ﬁnd optimal hyper-parameters using validation set and random search.

Original: Similar hyper-parameter search is performed for Linear, NAM, and MLP baselines.
Revised: Similar hyper-parameter search is performed for Linear, NAM, and MLP baselines.

Original: Finally, for EBMs and XGBoost, CPU machines are used, with hyper-parameter search using the guidelines in the original works.
Revised: Finally, for EBMs and XGBoost, CPU machines are used, with hyper-parameter search using the guidelines in the original works.

Original: See
Revised: See

Original: Appendix Section A.2 for more training and hyper-parameter search details.
Revised: Appendix Section A.2 for more training and hyper-parameter search details.

Original: Evaluation details.
Revised: Evaluation details.

Original: Performance metrics: (i) for regression we report mean-squared error (MSE) or root MSE (RMSE); (ii) for binary classification we report area under the ROC curve (AUROC) or error rate (Error); (iii) for multi-class classification in tabular and image domains we report accuracy@ (acc@1); and, (iv) for object detection in image domain we report mean average precision (mAP) averaged over IoU thresholds from 0.5 to 0.9 as per MS-COCO [33] definitions.
Revised: Performance metrics: (i) for regression we report mean-squared error (MSE) or root MSE (RMSE); (ii) for binary classiﬁcation we report area under the ROC curve (AUROC) or error rate (Error); (iii) for multi-class classiﬁcation in tabular and image domains we report accuracy@ (acc@1); and, (iv) for object detection in image domain we report mean average precision (mAP) averaged over IoU thresholds from 0.5 to 0.9 as per MS-COCO [33] deﬁnitions.

Original: We report average performance and standard deviation over 10 runs with different random seeds.
Revised: We report average performance and standard deviation over 10 runs with different random seeds.

Original: 4.4 Comparison with baselines
Revised: 4.4 Comparison with baselines

Original: In this section we compare NBM, as well as the pairwise interactions NB 2 M counterpart, with popular and widely used GAMs and non-interpretable black-box models.
Revised: In this section we compare NBM, as well as the pairwise interactions NB 2 M counterpart, with popular and widely used GAMs and non-interpretable black-box models.

Original: First, we perform extensive comparison on the number of parameters and throughput against the most similar architecture, i.e. Neural Additive Model (NAM) [2].
Revised: First, we perform extensive comparison on the number of parameters and throughput against the most similar architecture, i.e. Neural Additive Model (NAM) [2].

Original: Both NAMs and NBMs utilize deep networks and are able to run on GPUs, which helps to scale the models on large datasets.
Revised: Both NAMs and NBMs utilize deep networks and are able to run on GPUs, which helps to scale the models on large datasets.

Original: Results on representative datasets are presented in Table 2.
Revised: Results on representative datasets are presented in Table 2.

Original: The throughput is measured as the number of input instances that we can process per second ( x / sec) on one 32 GB V100 GPU, in inference mode.
Revised: The throughput is measured as the number of input instances that we can process per second ( x / sec) on one 32 GB V100 GPU, in inference mode.

Original: For each model, we take the largest batch size (up to 8,192) that fits on the GPU and calculate the average time over 100 runs to process that batch.
Revised: For each model, we take the largest batch size (up to 8,192) that ﬁts on the GPU and calculate the average time over 100 runs to process that batch.

Original: With that, we calculate the number of input instances processed per second.
Revised: With that, we calculate the number of input instances processed per second.

Original: Throughput can vary depending on the implementation, library used, etc.
Revised: Throughput can vary depending on the implementation, library used, etc.

Original: Hence, in order to be as fair as possible, we compare both models with our own optimized implementation in the same deep learning library, i.e. , PyTorch [48].
Revised: Hence, in order to be as fair as possible, we compare both models with our own optimized implementation in the same deep learning library, i.e. , PyTorch [48].

Original: The only dataset where NAM narrowly beats NBM in the number of parameters is CA Housing, which has the smallest number of input features D = 8 .
Revised: The only dataset where NAM narrowly beats NBM in the number of parameters is CA Housing, which has the smallest number of input features D = 8 .

Original: On other datasets, NBM has around 5 × –50 × fewer parameters than NAM, while having 4 × –7 × smaller runtime.
Revised: On other datasets, NBM has around 5 ⇥ –50 ⇥ fewer parameters than NAM, while having 4 ⇥ –7 ⇥ smaller runtime.

Original: Finally, only the sparse version of NBM model can efficiently run on Newsgroups and Common Objects (with pairwise interactions) datasets, where standard
Revised: Finally, only the sparse version of NBM model can efﬁciently run on Newsgroups and Common Objects (with pairwise interactions) datasets, where standard

Original: NAMs (and other GAMs) have throughput too low for any practical application.
Revised: NAMs (and other GAMs) have throughput too low for any practical application.

Original: Note that these comparisons are with our optimized implementation of NAM.
Revised: Note that these comparisons are with our optimized implementation of NAM.

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Next, we compare performance with GAM baselines, and non-interpretable black-box models in
Revised: Next, we compare performance with GAM baselines, and non-interpretable black-box models in

Original: Table 3.
Revised: Table 3.

Original: We notice that NBMs achieve the best performance among all GAMs, even outperforming full complexity MLPs on several datasets.
Revised: We notice that NBMs achieve the best performance among all GAMs, even outperforming full complexity MLPs on several datasets.

Original: The difference in performance is more pronounced on larger datasets, such as iNaturalist and Common Objects, where NBMs generalize better.
Revised: The difference in performance is more pronounced on larger datasets, such as iNaturalist and Common Objects, where NBMs generalize better.

Original: EBMs have a big downside as they do not support pairwise interactions on multi-class tasks, and do not scale at all to very large datasets.
Revised: EBMs have a big downside as they do not support pairwise interactions on multi-class tasks, and do not scale at all to very large datasets.

Original: Finally, on datasets with large number of features, i.e. Newsgroups and
Revised: Finally, on datasets with large number of features, i.e. Newsgroups and

Original: Common Objects (with pairwise interactions), NBMs are the only GAMs that scale, as we did not manage to complete a successful training with NAMs or EBMs.
Revised: Common Objects (with pairwise interactions), NBMs are the only GAMs that scale, as we did not manage to complete a successful training with NAMs or EBMs.

Original: We finally observe that pairwise interactions are enough to match or beat black-box models on 6 out of 7 datasets.
Revised: We ﬁnally observe that pairwise interactions are enough to match or beat black-box models on 6 out of 7 datasets.

Original: The only exception is CoverType, where MLP has 0.9694 acc@1 vs. NB 2 M with 0.8908.
Revised: The only exception is CoverType, where MLP has 0.9694 acc@1 vs. NB 2 M with 0.8908.

Original: However, we scale NBMs further and train NB 3 M ( i.e. , including triplet interactions) that gets 0.
Revised: However, we scale NBMs further and train NB 3 M ( i.e. , including triplet interactions) that gets 0.

Original: acc@1.
Revised: acc@1.

Original: Even though this is an interesting result that helps us understand the data behavior, we argue that triplet feature interactions are very hard to visualize and hence not interpretable.
Revised: Even though this is an interesting result that helps us understand the data behavior, we argue that triplet feature interactions are very hard to visualize and hence not interpretable.

Original: 4.5 Comparison with state of the art
Revised: 4.5 Comparison with state of the art

Original: We finally compare NBM, as well as the pairwise interactions NB 2 M counterpart, with neural-based state-of-the-art GAMs.
Revised: We ﬁnally compare NBM, as well as the pairwise interactions NB 2 M counterpart, with neural-based state-of-the-art GAMs.

Original: Namely, we compare with Neural Additive Model (NAM) [2] and Neural
Revised: Namely, we compare with Neural Additive Model (NAM) [2] and Neural

Original: Oblivious Decision Trees GAM (NODE-GAM) [12], and their pairwise interactions versions NA 2 M and NODE-GA 2 M. Results are presented in Table 4.
Revised: Oblivious Decision Trees GAM (NODE-GAM) [12], and their pairwise interactions versions NA 2 M and NODE-GA 2 M. Results are presented in Table 4.

Original: We compute NAM and NBM results using our codebase, while NODE-GAM results are reproduced from [12].
Revised: We compute NAM and NBM results using our codebase, while NODE-GAM results are reproduced from [12].

Original: For a fair comparison on these datasets, we use training, validation, and, testing splits from NODE-GAM [12], and perform the same feature normalization, see Section 4.1 for details.
Revised: For a fair comparison on these datasets, we use training, validation, and, testing splits from NODE-GAM [12], and perform the same feature normalization, see Section 4.1 for details.

Original: Without any additional model selection (architecture tuning, stochastic weight averaging, etc. )
Revised: Without any additional model selection (architecture tuning, stochastic weight averaging, etc. )

Original: NBMs outperform NODE-GAMs on 7 out of 8 presented datasets (albeit marginally), and outperform NAM on all datasets.
Revised: NBMs outperform NODE-GAMs on 7 out of 8 presented datasets (albeit marginally), and outperform NAM on all datasets.

Original: Note that, the idea of NBM is perpendicular to that of NODE-GAM and it is possible that bigger gains can be achieved by combining them, especially for Epsilon and Yahoo datasets, where NB 2 Ms do not scale due to a high dimensionality and non-sparse nature of features. 
Revised: Note that, the idea of NBM is perpendicular to that of NODE-GAM and it is possible that bigger gains can be achieved by combining them, especially for Epsilon and Yahoo datasets, where NB 2 Ms do not scale due to a high dimensionality and non-sparse nature of features. [Table] [Table]

Original: [Table]
Revised: [Table]

Revised: ± 0.

Original: [Figure]
Revised: ± 0. NB 2 M 0. ± 0. ± 0.

Original: H o u s
Revised: ± 0. ± 0. ± 0. ±

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: H o u s e

Original: P r i ce C o n tr i
Revised: P r i ce C o n tr i

Original: [Figure]
Revised: [Figure]

Original: t i o n
Revised: o n

Original: [Figure]

Original: P r i

Original: [Figure]
Revised: [Figure]

Original: C o n tr i
Revised: n tr i

Original: [Figure]
Revised: [Figure]

Original: t i [Figure] n
Revised: t i o n

Original: [Figure]
Revised: [Figure]

Original: 4.6 Stability and interpretability
Revised: 4.6 Stability and interpretability

Original: The interpretability of GAMs comes from the fact that the learned shape functions can be easily visualized.
Revised: The interpretability of GAMs comes from the fact that the learned shape functions can be easily visualized.

Original: In the same manner as the other GAM approaches, each feature’s importance in an NBM can be represented by a unique shape function that exactly describes how the NBM computes a prediction.
Revised: In the same manner as the other GAM approaches, each feature’s importance in an NBM can be represented by a unique shape function that exactly describes how the NBM computes a prediction.

Original: We demonstrate this on the CA Housing dataset because it has a small number of input features ( D = 8 ) and the full model can conveniently be visualized in a single row, see Figure 3.
Revised: We demonstrate this on the CA Housing dataset because it has a small number of input features ( D = 8 ) and the full model can conveniently be visualized in a single row, see Figure 3.

Original: Towards this purpose, we train an ensemble of 100 models by running different random seeds with optimal hyper-parameters, in order to analyze when the models learn the same shape and when they diverge.
Revised: Towards this purpose, we train an ensemble of 100 models by running different random seeds with optimal hyper-parameters, in order to analyze when the models learn the same shape and when they diverge.

Original: Following Agarwal et al.
Revised: Following Agarwal et al.

Original: [2], we set the average score for each shape function to be zero by subtracting the respective mean feature score.
Revised: [2], we set the average score for each shape function to be zero by subtracting the respective mean feature score.

Original: Next, we plot each shape function as f i ( x i ) vs. x i for each model in the ensemble using a semi-transparent line, and an average ensemble shape function using a thick line.
Revised: Next, we plot each shape function as f i ( x i ) vs. x i for each model in the ensemble using a semi-transparent line, and an average ensemble shape function using a thick line.

Original: Finally, the x -axis is divided by bars depicting the normalized data density, i.e. , darker areas contain more data points.
Revised: Finally, the x -axis is divided by bars depicting the normalized data density, i.e. , darker areas contain more data points.

Original: Figure 3 depicts an ensemble of NAMs [2] (upper row) and
Revised: Figure 3 depicts an ensemble of NAMs [2] (upper row) and

Original: NBMs (bottom row).
Revised: NBMs (bottom row).

Original: Although the shape functions are correlated for most features, we observe that
Revised: Although the shape functions are correlated for most features, we observe that

Original: NBMs diverge much less in the cases where there are only few data points (light / white areas in the graphs).
Revised: NBMs diverge much less in the cases where there are only few data points (light / white areas in the graphs).

Original: This is due to the fact that the basis network in NBM is trained jointly with only linear composing weights being trained for each feature separately.
Revised: This is due to the fact that the basis network in NBM is trained jointly with only linear composing weights being trained for each feature separately.

Original: In contrast, each feature in NAM has its own network, which becomes unstable and diverges in cases of few training data points.
Revised: In contrast, each feature in NAM has its own network, which becomes unstable and diverges in cases of few training data points.

Original: Finally, to quantify stability, we compute the standard deviation for each visualized shape function over 100 models, and report the mean standard deviation over all features.
Revised: Finally, to quantify stability, we compute the standard deviation for each visualized shape function over 100 models, and report the mean standard deviation over all features.

Original: For NAMs, this mean standard deviation is 0.9921, while for NBMs it is 0.1987.
Revised: For NAMs, this mean standard deviation is 0.9921, while for NBMs it is 0.1987.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: 4.7 Ablation study
Revised: 4.7 Ablation study

Original: Number of basis functions.
Revised: Number of basis functions.

Original: We evaluate the robustness of NBMs w.r.t.
Revised: We evaluate the robustness of NBMs w.r.t.

Original: the choice of the number of basis functions B .
Revised: the choice of the number of basis functions B .

Original: Results are presented in Figure 4 for Newsgroups, CoverType, and, CUB.
Revised: Results are presented in Figure 4 for Newsgroups, CoverType, and, CUB.

Original: We observe that NBMs are not overly sensitive to the choice of B .
Revised: We observe that NBMs are not overly sensitive to the choice of B .

Original: Rather than tuning this hyperparameter, we recommend setting B = 100 for NBMs and B = 200 for NB 2 Ms as it performs well across a large variety of datasets we experimented with.
Revised: Rather than tuning this hyperparameter, we recommend setting B = 100 for NBMs and B = 200 for NB 2 Ms as it performs well across a large variety of datasets we experimented with.

Original: Although, e.g. , for NB 2 Ms on CoverType, using a larger number of basis functions leads to some performance gains, it comes with a throughput trade-off, as computing the linear combination of bases starts becoming the bottleneck.
Revised: Although, e.g. , for NB 2 Ms on CoverType, using a larger number of basis functions leads to some performance gains, it comes with a throughput trade-off, as computing the linear combination of bases starts becoming the bottleneck.

Original: Thus, we suggest using the recommended values as a great trade-off between accuracy and throughput.
Revised: Thus, we suggest using the recommended values as a great trade-off between accuracy and throughput.

Original: Multiple subnets.
Revised: Multiple subnets.

Original: Agarwal et al.
Revised: Agarwal et al.

Original: [2] propose to extend NAMs to multi-task / multi-class setups by associating each feature with multiple subnets.
Revised: [2] propose to extend NAMs to multi-task / multi-class setups by associating each feature with multiple subnets.

Original: This method is directly applicable on NBMs as well, by using S different networks to learn S sets of basis functions.
Revised: This method is directly applicable on NBMs as well, by using S different networks to learn S sets of basis functions.

Original: We compare few options on the multi-class CoverType predictions with the number of subnets S =1 and S =5 in Table 5.
Revised: We compare few options on the multi-class CoverType predictions with the number of subnets S =1 and S =5 in Table 5.

Original: Both NAMs and NBMs have a similarly small relative accuracy improvement when using 5 subnets over thesubnet, however that comes at a 5 × increase in the number of parameters and 5 × lower throughput.
Revised: Both NAMs and NBMs have a similarly small relative accuracy improvement when using 5 subnets over thesubnet, however that comes at a 5 ⇥ increase in the number of parameters and 5 ⇥ lower throughput.

Original: With the accuracy-complexity trade-off in mind, we did not see much benefit of using S = 5 on our datasets, so we keep S = 1 across all other experiments.
Revised: With the accuracy-complexity trade-off in mind, we did not see much beneﬁt of using S = 5 on our datasets, so we keep S = 1 across all other experiments.

Original: Interestingly, NB 2 M with S =1 has a comparable throughput to NAM with S =5 , while achieving an impressive accuracy gain, see Table 5.
Revised: Interestingly, NB 2 M with S =1 has a comparable throughput to NAM with S =5 , while achieving an impressive accuracy gain, see Table 5.

Original:  Conclusion and future work
Revised:  Conclusion and future work

Original: This work describes novel Neural Basis Models (NBMs), which belong to a new subfamily of
Revised: This work describes novel Neural Basis Models (NBMs), which belong to a new subfamily of

Original: Generalized Additive Models (GAMs), and utilize deep learning techniques to scale to large datasets and high-dimensional features.
Revised: Generalized Additive Models (GAMs), and utilize deep learning techniques to scale to large datasets and high-dimensional features.

Original: Our approach addresses several scalability and performance issues associated with GAMs, while still preserving their interpretability compared to black-box deep neural networks (DNNs).
Revised: Our approach addresses several scalability and performance issues associated with GAMs, while still preserving their interpretability compared to black-box deep neural networks (DNNs).

Original: We show that our NBMs and NB 2 Ms achieve state-of-the-art performance on a large variety of datasets and tasks, while being much smaller and faster than other neural-based
Revised: We show that our NBMs and NB 2 Ms achieve state-of-the-art performance on a large variety of datasets and tasks, while being much smaller and faster than other neural-based

Original: GAMs.
Revised: GAMs.

Original: As a result, they can be used as a drop-in replacements for the black-box DNNs.
Revised: As a result, they can be used as a drop-in replacements for the black-box DNNs.

Original: We do recognize that our approach, though highly scalable, has limitations w.r.t.
Revised: We do recognize that our approach, though highly scalable, has limitations w.r.t.

Original: number of features.
Revised: number of features.

Original: Beyond 10,000 dense (or 1 million sparse) features, we would need to apply some form of feature selection [12, 37, 58] to scale further.
Revised: Beyond 10,000 dense (or 1 million sparse) features, we would need to apply some form of feature selection [12, 37, 58] to scale further.

Original: Scalability issue is even more pronounced when modeling pairwise interactions in NB
Revised: Scalability issue is even more pronounced when modeling pairwise interactions in NB

Original: 2 M.
Revised: 2 M.

Original: However, NBMs can still handle an order of magnitude more than what can be handled by NAM or other GAM approaches that do not perform feature selection.
Revised: However, NBMs can still handle an order of magnitude more than what can be handled by NAM or other GAM approaches that do not perform feature selection.

Original: There are many future directions for this line of work.
Revised: There are many future directions for this line of work.

Original: First, for the computer vision domain, we assume an intermediate, interpretable concept layer [29] on which NBMs and in general GAMs can be applied.
Revised: First, for the computer vision domain, we assume an intermediate, interpretable concept layer [29] on which NBMs and in general GAMs can be applied.

Original: It would be interesting to explore visual interpretability by either directly going to the pixel space with NBMs or learning visual features that can do recognition with lower order interactions in
Revised: It would be interesting to explore visual interpretability by either directly going to the pixel space with NBMs or learning visual features that can do recognition with lower order interactions in

Original: NBM framework (for example, NB 2 Ms).
Revised: NBM framework (for example, NB 2 Ms).

Original: Finally, the idea of using shared basis is generic.
Revised: Finally, the idea of using shared basis is generic.

Original: Although, we used neural networks to learn these basis, we can enhance the model interpretability further for higher order interactions, by using more interpretable learning functions such as polynomials.
Revised: Although, we used neural networks to learn these basis, we can enhance the model interpretability further for higher order interactions, by using more interpretable learning functions such as polynomials.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contribu- tions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Sec.
Revised: [Yes] See Sec.

Original: A.5.
Revised: A.5.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Sec.
Revised: [Yes] See Sec.

Original: A.5.
Revised: A.5.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: No new assets except code, see 3(a).
Revised: No new assets except code, see 3(a).

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: us- ing/curating?
Revised: us- ing/curating?

Original: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?
Revised: [Yes] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable informa- tion or offensive content?

Original: [Yes] 5.
Revised: [Yes] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?[N/A] A Appendix A.1 Dataset descriptions CA Housing [10, 46].
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: The target to regress is the median house value for California districts (1990 U.S. census), expressed in hundreds of thousands of dollars, from 8 household-related variables.

Original: FICO [22].

Original: Anonymized dataset of line of credit applications made by real homeowners.

Original: The customers in this dataset have requested a credit line in the range of $5,000 – $150,000.

Original: The task is to make a binary prediction whether applicants will repay their account within 2 years.

Original: CoverType [8, 16, 20].

Original: The samples in this dataset correspond to 30 × 30m patches of forest in the U.S., collected for the task of predicting each patch’s cover type, i.e. , the dominant species of tree.

Original: There are seven covertypes, making this a multi-class classification problem.

Original: Newsgroups [32, 43].

Original: This dataset is a collection of news documents, partitioned across 20 different newsgroups, making this a multi-class classification problem.

Original: Each article is represented by a tf-idf term for each word in the training split word vocabulary.

Original: Newsgroups dataset has sparsity 99.9%, i.e. , only around 150 words from a vocabulary of 150k words appears per a given article, on average.

Original: MIMIC-II [41, 51].

Original: The task is to make a binary prediction on mortality of Intensive Care Unit (ICU) patients.

Original: Contains physiologic signals and vital signs time series captured from patient monitors for tens of thousands of ICU patients.

Original: Credit [17, 19].

Original: This dataset contains anonymized credit card transactions labeled as fraudulent or genuine, and the task is to make a binary prediction between them.

Original: Click [15].

Original: Subset of data from the KDD Cup 2012.

Original: Namely, 500,000 objects of a positive class and 500, objects of a negative class were randomly sampled to create a binary prediction task.

Original: We would like to gratefully acknowledge the organizers of KDD Cup 2012 as well as Tencent Inc. for making the dataset available.

Original: Epsilon [21].

Original: Dataset for binary prediction from the PASCAL Large Scale Learning Challenge.

Original: Higgs [3, 26].

Original: The problem is to binary predict whether the given event produces Higgs bosons.

Original: Microsoft [40, 49].

Original: Ranking dataset, where features are extracted from query-url pairs.

Original: Each pair has relevance judgment labels to regress , which take values from 0 (irrelevant) to 4 (very relevant).

Original: Yahoo [60].

Original: Another ranking dataset with query-url pairs that have labels to regress from 0 to 4.

Original: Year [66].

Original: Subset of Million Song Dataset.

Original: The task is to regress the release year of the song by using the audio features.

Original: It contains tracks from 1922 to 2011.

Original: CUB [18, 62].

Original: This image classification dataset consists of images of 200 bird classes.

Original: All images are annotated with keypoint locations of 15 bird parts ( e.g. , beak, wing, crown) and each location is associated with one or more part-attribute labels ( e.g. , orange leg, striped wing).

Original: Some of the keypoint annotations distinguish between the left-right instances of parts, e.g. , ‘left wing’ / ‘right wing’, ‘left eye’ / ‘right eye’.

Original: We treat these as the same part, i.e. , ‘left wing’ and ‘right wing’ as ‘wing’.

Original: iNaturalist Birds [27, 59].

Original: Another image classification dataset that contains 1,486 bird classes.

Original: The full iNaturalist 2021 dataset consists of various super-categories ( e.g. , plants, insects, birds), covering 10K species in total.

Original: The Birds super-category contains 1,486 bird classes and more challenging scenes compared to CUB.

Original: Therefore, the iNaturalist dataset is a challenging testbed for any image classification method.

Original: However, note that this dataset lacks keypoint annotations.

Original: Common Objects.

Original: Proprietary object detection dataset created by collecting public images from Instagram 1 .

Original: Dataset contains 114 common household objects, ( e.g. , stove, bed, table), plus a background class, with bounding box locations, 200 parts and 54 attributes.

Original: Each bounding box for each image is pre-processed using compositions of parts ( e.g. , leg, handle, top) and attributes ( e.g. , colors, textures, shapes) to extract 2,618 interpretable features and 100k pairwise feature interactions.

Original: Common Objects dataset has sparsity 97%, i.e.

Original: [Table]
Revised: [N/A]

Original: Linear, MLP, NAM, and NBM are trained using the Adam with decoupled weight decay (AdamW) optimizer [35], on 8 × V100 GPU machines with 32 GB memory, and a batch size of at most 1024 per GPU.

Original: We train for 1,000, 500, 100, or, 50 epochs, depending on the size and feature dimensionality of the dataset.

Original: The learning rate is decayed with cosine annealing [34] from the starting value until zero.

Original: We find optimal hyper-parameters for all models using validation set and random search, following the detailed guidelines.

Original: Linear.

Original: We tune the starting learning rate in the continuous interval [1e − 5 , 100) , weight decay in the interval [1e − 10 , 1 . 0) .

Original: MLP.

Original: We tune the starting learning rate in the continuous interval [1e − 5 , 1 . 0) , weight decay in the interval [1e − 10 , 1 . 0) , dropout coefficients in the discrete set { 0 , 0 .

Original: NAM.

Original: We tune the starting learning rate in the continuous interval [1e − 5 , 1 . 0) , weight decay in the interval [1e − 10 , 1 . 0) , output penalty coefficient in the interval [1e − 7 , 100) , dropout and feature dropout coefficients in the discrete set { 0 , 0 .

Original: NBM.

Original: We tune the starting learning rate in the continuous interval [1e − 5 , 1 . 0) , weight decay in the interval [1e − 10 , 1 . 0) , output penalty coefficient in the interval [1e − 7 , 100) , dropout and basis dropout coefficients in the discrete set { 0 , 0 .

Original: Optimal hyper-parameters for NBMs and

Original: NB 2 Ms on all datasets are given in Table A.1.

Original: [Figure]

Original: | NA

Original: M | / |

Original: [Figure]

Original: D | NA 2 M | / | N

Original: B 2 M |

Original: A.3 Additional discussion w.r.t. NAM

Original: Number of parameters for multi-class and pairwise feature interactions.

Original: We compare number of weight parameters needed to learn NAM vs. NBM for the multi-class task.

Original: This discussion is an extension of the discussion in Section 3.3.

Original: Let us denote with M the number of parameters in MLP for each feature in NAM, and with N the number of parameters in MLP for bases in NBM.

Original: In most experiments the optimal NAM has 3 hidden layers with 64, 64 and 32 units ( M = 6401 ), and, NBM has 3 hidden layers with 256, 128, 128 units ( N = 62820 ) and B = 100 basis functions.

Original: Finally, let us denote with D the input feature dimensionality, and with C the number of classes in the multi-class task.

Original: Then the ratio of number of parameters in NAM vs. NBM is given by,

Original: [Equation]

Original: Similarly, in the case of pairwise feature interactions in NA 2 M vs. NB 2 M, this ratio is given by,

Original: [Equation]

Original: Figure A.1 shows both ratios for different values of feature dimensionality D and number of classes C .

Original: For unary features (NAM vs. NBM, Figure A.1 left), the conclusion is the same as in the case of binary classification, i.e. , for D = 10 , NBMs and NAMs have roughly equal number of parameters, for any given value of C .

Original: For higher number of classes, NBMs still provide significant gain over NAMs, however, that gain starts decreasing, due to the fact that the final linear classifier starts becoming the most memory hungry part of the model.

Original: Nevertheless, even with C = 1,486 and D = 278 in iNaturalist Birds, which has the most classes from the datasets we used, NBMs have around 5 × less parameters than NAMs, see Table 2.

Original: For pairwise feature interactions (NA 2 M vs. NB 2 M, Figure A.1 right), the ratio is much more pronounced, i.e. , already at D = 5 , NB 2 Ms and NA 2 Ms have equal number of parameters, and the growth of the ratio w.r.t.

Original: D is much more significant.

Original: Already after few hundred feature dimensions the ratio is at peak.

Original: Throughput optimization of NAMs.

Original: Neural Additive Models (NAMs) [2] learn an MLP network for each input feature, followed by a linear combination to make a prediction.

Original: Official implementation [42] runs a for loop over all networks, which results in a poor GPU utilization.

Original: More precisely, this implementation requires extremely large batches ( >> 1,024) per GPU to make the training efficient, which is impractical.

Original: We do recognize that efficiency was not of highest priority to the authors [2], but in our case we are scaling GAMs to multi-class datasets with order of million data points.

Original: Thus, to facilitate a fair comparison against our NBMs, we reimplement NAMs using grouped convolutions [30, 65], which are readily available in standard deep learning libraries.

Original: Namely, we stack corresponding hidden layers of all MLPs ( e.g. , first hidden layer of all MLPs) into a grouped 1-D convolution, where number of groups equals the number of features.

Original: The computation performed is identical to original NAMs, i.e. there is no feature interaction, while achieving around × 2– × 10 speedup, depending on the dataset.

Original: We perform the same implementation trick to NA 2 Ms, as well.

Original: A.4 Additional visualization

Original: The interpretability of GAMs comes from the fact that the learned shape functions can be easily visualized.

Original: In the same manner as the other GAM approaches, each feature’s importance in an NBM can be represented by a unique shape function that exactly describes how the NBM computes a prediction.

Original: For an example, see Figure 3 visualization on the CA Housing dataset.

Original: We additionally demonstrate this on the CUB image classification dataset that consists of 200 bird classes, where each image is represented by interpretable features, e.g. , a “bird” image can be represented with “striped wings”, “needle-shaped beak”, “long legs”, etc. , that are predicted from the image using a convolutional-neural-network (CNN) model.

Original: We present visualizations of shape functions with highest positive or negative contribution to 6 randomly selected bird classes, see Figures A.2 and A.3.

Original: Towards this purpose, we train an ensemble of 20 models by running different random seeds with optimal hyper- parameters, in order to analyze when the models learn the same shape and when they diverge.

Original: Following Agarwal et al.

Original: [2], we set the average score for each shape function to be zero by subtracting the respective mean feature score.

Original: Next, we plot each shape function as f i ( x i ) vs. x i for each model in the ensemble using a semi-transparent line, and an average ensemble shape function using a thick line.

Original: Finally, the x -axis is divided by bars depicting the normalized data density, i.e. , darker areas contain more data points.

Original: Figures A.2 and A.3 depict few image examples for the respective class (upper row), and an ensemble of NBMs (bottom row).

Original: We visually observe that NBMs provide a strong interpretable overview of the respective bird class, and that the shape functions do not diverge significantly even in the cases where there are only few data points (light / white areas in the graphs).

Original: [Figure]

Original: Crested Auklet

Original: [Figure]

Original: C o n tr i

Original: [Figure]

Original: t i o n

Original: [Figure]

Original: [Figure]

Original: Yellow-headed Blackbird

Original: [Figure]

Original: [Table]

Original: Towhee

Original: [Table]

Original: [Figure]

Original: A.5 Learning-Theoretic Guarantees for Basis Models in a RKHS

Original: As discussed briefly in the main paper, it is possible to develop a more rigorous argument for the use of a small set of basis functions instead of a complete generalized additive model.

Original: To elucidate we first require establishing some notation: We represent matrices by uppercase boldface, e.g. , X and vectors by lowercase boldface, i.e. , x .

Original: We assume that the covariates lie within the set X ⊆ R D , and the labels lie within the finite set Y .

Original: Data ( x , y ) ∈

Original: X × Y are drawn following some unknown (but fixed) distribution P .

Original: We assume we are provided with n i.i.d.

Original: samples { ( x i , y i ) } ni =1 as the train set.

Original: Consider a generalized additive model (GAM)

Original: g : X → Y : g ( x ) =

Original: D (cid:88) i =

Original: [Equation]

Original: Assume that the shape functions f 1 , . . .

Original: , f D ; f i : R → Y have a maximum norm B H > 0 in some Reproducing Kernel Hilbert Space (RKHS, [7])

Original: H endowed with a PSD kernel k ( · , · ) : R × R

Original: →

Original: R and feature ϕ : R → R d H , i.e. , ∥ f i ∥

Original: H ≤ B H , and w = { w i } Di =1 ∈ R D , k

Original: ( x, y )

Original: =

Original: ϕ ( x ) ⊤ ϕ

Original: ( y )

Original: such that ∥ w ∥ 2 ≤ B w for B w > 0 .

Original: This characterization corresponds to a family of functions H

Original: A , i.e. ,

Original: H A = { g | g ( x ) =

Original: D (cid:88) i =

Original: [Equation]

Original: [Equation]

Original: (A.3)

Original: The idea behind the basis decomposition approach highlighted in this paper is to only use a fixed number of bases, B , to model each f i .

Original: Observe that one can obtain rigorous guarantees for f i that lie within an RKHS using

Original: Mercer’s Theorem [39].

Original: We have that if the kernel k associated with the RKHS H is continuous, positive-definite and symmetric, there exist a set of eigenvalues { λ i } ∞ i =1 and eigenfunctions (basis functions) { ω i } ∞ i =1 that form an orthonormal basis for k , i.e. , for any x, y ∈ R ,

Original: [Equation]

Original: Where the bases are orthonormal

Original: [-, i.e. , (cid:82) x ∈ R ω i ( x ) ω j ( x ) dx = 
-]
Revised: [-, i.e. , (cid:82) x ∈ R ω i ( x ) ω j ( x ) dx = 
-]

Original: for i ̸

Original: =

Original: j and 1 otherwise.

Original: This representation naturally gives a form for ϕ ( · ) =

Original: [ √ λ i ω i ( · )]

Original: ∞ i

Original: =1 .

Original: Furthermore, we have that for each f ∈ H there exists f ∈ L 2 such that f ( x ) = ⟨ f , ϕ ( x ) ⟩ H ∀ x ∈ R .

Original: Note once again that the reproducing kernel Hilbert space H corresponds to the feature-wise functions f , whereas the space H A corresponds to the overall function g .

Original: Now, we can define, for the family H A a Generalized Basis Model of order B (denoted as H B ) as the following.

Original: Definition 1.

Original: A Generalized Basis Model of order B for any function class H A that satisfies the characterization in Equation A.3 for some ( H , B H , B w ) is given by the family H B :

Original: [Equation]

Original: Where orthogonality ( ⊥ ) is defined as h i ⊥ h

Original: j = ⇒ (cid:82)

Original: x ∈ R h i ( x ) · h j ( x ) dx = 0 .

Original: Next, note that by Mercer’s Theorem, for each function f ∈ H , there exists f = { f i } ∞ i =1 , f ∈ L 2 such that f ( x ) = ⟨ f , ϕ ( x ) ⟩ H .

Original: Combining this statement with the basis representation for ϕ gives us an alternate representation of any f ∈ H , as

Original: [Equation]

Original: Under this representation, we can relate the two spaces H A and H B as follows.

Original: Proposition 1.

Original: For any H , dimensionality D , and number of basis functions B > 0 , H B ⊆ H A .

Original: Proof.

Original: Follows from Mercer’s Theorem [39].

Original: Any g ∈ H B can be written as a linear combination of functions in H (and consequently H A ), each of which admit a basis representation via Mercer’s Theorem, where all but B components have coefficient 0 .

Original: In the limit B → ∞ , H B = H A .

Original: Since the basis functions in H B lie on a finite-dimensional subspace within H spanned by B basis vectors, we can without loss of generality, assume that these B basis vectors correspond to { ω i } B i =1 obtained from

Original: Equation A.4.

Original: Now, to prove generalization bounds on the best function learnable in H B and contrast that with H

Original: A , we require a “smoothing” assumption in { ω i } ∞ i =1 (and correspondingly on H ).

Original: The essence of this assumption is to ensure that the kernel H can be spanned without introducing much error by only with a few basis components, and is similar to smoothing kernel assumptions made in other areas as well, e.g. , in reinforcement learning.

Original: Assumption 1 ( γ -Exponential Spectral Decay of H ) .

Original: [-For the decomposition of H as outlined in Equation A.4, we assume that there exist absolute constants C 1 < 1 and C 
-]
Revised: [-For the decomposition of H as outlined in Equation A.4, we assume that there exist absolute constants C 1 < 1 and C 
-]

Original: = O (1) and parameter γ such that λ i ≤

Original: C 1 exp( − C 2 · i γ ) for each i ≥ 1 .

Original: At a high level, our approach is to bound the test error of the empirical risk minimizer in H A , with the optimal risk minimizer in H

Original: B to demonstrate that learning a generalized basis model does not incur significantly larger error compared to learning the full model.

Original: We first make these terms precise.

Original: Recall that the empirical risk for any function g is given by (cid:98) L n ( g ) =

Original: 1 n (cid:80) ni =1 ℓ ( g ( x i ) , y i ) .

Original: We denote ˆ g as the empirical risk minimizer within H

Original: B , i.e. ,

Original: [Equation]

Original: Similarly, the expected risk can be given, for any function g as L = E ( x ,y ) ∼ P [

Original: [Equation]

Original: ( g ( x ) , y )] .

Original: Then we can define the optimal expected risk minimizer g ⋆ in H A as,

Original: [Equation]

Original: We are now equipped to discuss our generalization bound.

Original: Theorem 1.

Original: Let ℓ be a 1-Lipschitz loss, δ ∈ (0 , 1] and Assumption 1 hold with constants C 1 , C 2 , γ .

Original: Then we have that with probability at least 1 − δ there exist absolute constants C 1 , C 2 such that,

Original: [Equation]

Original: Proof .

Original: We will denote the weights and singular values for f ⋆ as w ⋆ and λ ⋆ij , i.e. , g ⋆ ( x ) = (cid:80) Di =1 w ⋆i h

Original: ⋆i ( x i ) where h ⋆i ( x )

Original: = (cid:80) ∞

Original: [Equation]

Original: Note that this represnetation exists for some λ ⋆ij by Mercer’s Theorem, as discussed earlier.

Original: For any H B , H A , consider the function ˜ g ∈ H B

Original: that is a truncated version of g ⋆ up to b bases, i.e. ,

Original: ˜ g

Original: ( x ) = (cid:80) Di =1 w ⋆i (cid:101) h

Original: i ( x i ) where (cid:101) h i ( x ) =

Original: (cid:80) bj =

Original: 1 λ ⋆ij ω

Original: j

Original: ( x ) .

Original: Clearly, ˜ g ∈ H B .

Original: We can then rewrite the L.H.S. in the Theorem as,

Original: [Equation]

Original: Note that the middle term (cid:98) L n (ˆ g ) − (cid:98) L n (˜ g ) ≤ 0 since ˆ g is the empirical risk minimizer in H B .

Original: Hence, by bounding terms A and B , the proof will be complete.

Original: We can bound B via Lemma 1.

Original: We have that with probability at least 1 − δ/ 2 for any δ ∈ (0 , 1] ,

Original: [Equation]

Original: We bound A via bounding the Rademacher complexity [63].

Original: Since the loss function is Lipschitz and bounded, with probability at least 1 − δ/ 2 , δ ∈ (0 , 1] , we have that by Theorem 12 and Theorem 8 of Bartlett and

Original: Mendelson [4],

Original: [Equation]

Original: Where R n denotes the empirical Rademacher complexity at n samples [4].

Original: Observe that each element of H

Original: B is a linear combination of d elements that are represented by b basis vectors in H .

Original: Hence, there exist weights {{ α ij } Di =1 } Bj =1 such that any f ∈ H B can be written as (cid:80) i,j α ij ω j ( x i ) ,

Original: ∥ α ∥ 2 ≤ B H B w where α = {{ α ij } Di =1 } Bj =1 .

Original: Furthermore, we have that for any x , ϕ ( x ) ⊤ ϕ ( x ) =

Original: (cid:80) Bj =1 ω j

Original: ( x i ) 2 ≤ B .

Original: We therefore have, by Theorem 12 of [4] that with probability at least 1 − δ/ 2 , δ ∈ (0 , 1] ,

Original: [Equation]

Original: n

Original: The last inequality follows from Lemma 22 of [4].

Original: Replacing the above result for k , we have that with probability at least 1 − δ/ 2 , Using the bound for term B and applying a union bound provides us the final result.

Original: Lemma 1.

Original: The following holds with probability at least 1 − δ, δ ∈ (0 , 1] , for some absolute constant C ≪ 1 ,

Original: [Equation]

Original: Proof.

Original: [Equation]

Original: To bound 1 , we have that for any ( x , y ) within the training set, E [ ℓ (˜ g ( x ) , y )]

Original: =

Original: L (˜ g ) and 0 ≤ ℓ ( · , · ) ≤ 1 .

Original: By

Original: Azuma-Hoeffding, we obtain with probability at least 1 − δ, δ ∈ (0 ,

Original: [Equation]

Original: [Equation]

Original: For 2 , since ℓ is L − Lipschitz, we have for some x 1 , x 2 , y ∈ Y ,

Original: [Equation]

Original: [Equation]

Original: −

Original: [Equation]

Original: [Equation]

Original: Therefore:

Original: [Equation]

Original: Observe now that for any x ∈ X ,

Original: [Equation]

Original: Invoking Assumption 1, we have that

Original: D

Original: [Equation]

Original: Since γ ≥ 1 , we have,

Original: [Equation]

Original: A union bound for both parts finishes the proof.

Original: Discussion .

Original: The result holds when the target function class is a member of a Reproducing Kernel Hilbert Space (RKHS).

Original: While RKHSes include a variety of expressive machine learning function classes, e.g. , radial basis functions, polynomials, linear classifiers, it is not known whether arbitrarily initialized neural networks have a small norm in any RKHS with desirable properties.

Original: Most notably, however, it was shown recently that certain neural networks can be represented via the Neural Tangent Kernel (NTK), an example of where the theory can be applied as-is.

Original: More generally, however, this result demonstrates for arbitrary infinite-dimensional RKHS, we have an exponential dependence on the number of basis B required in the approximation error (second term).

Original: Observe that if we set B = O (log D ) , the second term is o (1) and goes to 0 as n → ∞ , which suggests that in practice, we only require a number of bases, B that grows logarithmically with the dimensionality D .