Original: Sparse tree-based Initialization for Neural Networks
Revised: Sparse tree-based Initialization for Neural Networks

Original: Anonymous authors Paper under double-blind review

Original: Abstract
Revised: Abstract

Original: Dedicated neural network (NN) architectures have been designed to handle specific data types (such as CNN for images or RNN for text), which ranks them among state-of-the-art methods for dealing with these data.
Revised: Dedicated neural network (NN) architectures have been designed to handle specific data types (such as CNN for images or RNN for text), which ranks them among state-of-the-art methods for dealing with these data.

Original: Unfortunately, no architecture has been found for dealing with tabular data yet, for which tree ensemble methods (tree boosting, random forests) usually show the best predictive performances.
Revised: Unfortunately, no architecture has been found for dealing with tabular data yet, for which tree ensemble methods (tree boosting, random forests) usually show the best predictive performances.

Original: In this work, we propose a new sparse initialization technique for (potentially deep) multilayer perceptrons (MLP): we first train a tree-based procedure to detect feature interactions and use the resulting information to initialize the network, which is subsequently trained via standard gradient descent (GD) strategies.
Revised: In this work, we propose a new sparse initialization technique for (potentially deep) multilayer perceptrons (MLP): we first train a tree-based procedure to detect feature interactions and use the resulting information to initialize the network, which is subsequently trained via standard gradient descent (GD) strategies.

Original: Numerical experiments on several tabular data sets show the benefits of this new, simple and easy-to-use method, both in terms of generalization capacity and computation time, compared to default MLP initialization and even to existing complex deep learning solutions.
Revised: Numerical experiments on several tabular data sets show the benefits of this new, simple and easy-to-use method, both in terms of generalization capacity and computation time, compared to default MLP initialization and even to existing complex deep learning solutions.

Original: In fact, this wise MLP initialization raises the performances of the resulting NN methods to that of gradient boosting on tabular data.
Revised: In fact, this wise MLP initialization raises the performances of the resulting NN methods to that of gradient boosting on tabular data.

Original: Besides, such initializations are able to preserve the sparsity of weights introduced in the first layers of the network throughout the training, which emphasizes that the first layers act as a sparse feature extractor (like convolutional layers in CNN).
Revised: Besides, such initializations are able to preserve the sparsity of weights introduced in the first layers of the network throughout the training, which emphasizes that the first layers act as a sparse feature extractor (like convolutional layers in CNN).

Original:  Introduction
Revised:  Introduction

Original: Neural networks are now widely used in many domains of machine learning, in particular when dealing with very structured data.
Revised: Neural networks are now widely used in many domains of machine learning, in particular when dealing with very structured data.

Original: They indeed provide state-of-the-art performances for applications with images or text.
Revised: They indeed provide state-of-the-art performances for applications with images or text.

Original: However, neural networks still perform poorly on tabular inputs, for which tree ensemble methods remain the gold standards (Grinsztajn et al., 2022).
Revised: However, neural networks still perform poorly on tabular inputs, for which tree ensemble methods remain the gold standards (Grinsztajn et al., 2022).

Original: The goal of this paper is to improve the performances of the former by using the strengths of the latter.
Revised: The goal of this paper is to improve the performances of the former by using the strengths of the latter

Original: Tree ensemble methods Tree-based methods are widely used in the ML community, especially for processing tabular data.
Revised: Tree ensemble methods Tree-based methods are widely used in the ML community, especially for processing tabular data.

Original: Two main approaches exist depending on whether the tree building process is parallel (e.g. Random Forest, RF, see Breiman, 2001b) or sequential (e.g. Gradient Boosting Decision Trees, GBDT, see Friedman, 2001).
Revised: Two main approaches exist depending on whether the tree building process is parallel (e.g. Random Forest, RF, see Breiman, 2001b) or sequential (e.g. Gradient Boosting Decision Trees, GBDT, see Friedman, 2001).

Original: In these tree ensemble procedures, the final prediction relies on averaging predictions of randomized decision trees, coding for particular partitions of the input space.
Revised: In these tree ensemble procedures, the final prediction relies on averaging predictions of randomized decision trees, coding for particular partitions of the input space.

Original: The two most successful and most widely used implementations of these methods are XGBoost and LightGBM (see Chen &
Revised: The two most successful and most widely used implementations of these methods are XGBoost and LightGBM (see Chen &

Original: Guestrin, 2016; Ke et al., 2017) which both rely on the sequential GBDT approach.
Revised: Guestrin, 2016; Ke et al., 2017) which both rely on the sequential GBDT approach.

Original: Neural networks Neural Networks (NN) are efficient methods to unveil the patterns of spatial or temporal data, such as images (Krizhevsky et al., 2012) or texts (Liu et al., 2016).
Revised: Neural networks Neural Networks (NN) are efficient methods to unveil the patterns of spatial or temporal data, such as images (Krizhevsky et al., 2012) or texts (Liu et al., 2016).

Original: Their performance results notably from the fact that several architectures directly encode relevant structures in the input: convolutional neural networks (CNN, LeCun et al., 1995) use convolutions to detect spatially-invariant patterns in images, and recurrent neural networks (RNN, Rumelhart et al., 1985) use a hidden temporal state to leverage the natural order of a text.
Revised: Their performance results notably from the fact that several architectures directly encode relevant structures in the input: convolutional neural networks (CNN, LeCun et al., 1995) use convolutions to detect spatially-invariant patterns in images, and recurrent neural networks (RNN, Rumelhart et al., 1985) use a hidden temporal state to leverage the natural order of a text.

Original: However, a dedicated natural architecture has yet to be introduced to deal with tabular data.
Revised: However, a dedicated natural architecture has yet to be introduced to deal with tabular data.

Original: Indeed, designing such an architecture would require to detect and leverage the structure of the relations between variables, which is much easier for images or text (spatial or temporal correlation) than for tabular data (unconstrained covariance structure).
Revised: Indeed, designing such an architecture would require to detect and leverage the structure of the relations between variables, which is much easier for images or text (spatial or temporal correlation) than for tabular data (unconstrained covariance structure).

Original: NN initialization and training In the absence of a suitable architecture for handling tabular data, the Multi-Layer Perceptron (MLP) architecture (Rumelhart et al., 1986) remains the obvious choice due to its generalist nature.
Revised: NN initialization and training In the absence of a suitable architecture for handling tabular data, the Multi-Layer Perceptron (MLP) architecture (Rumelhart et al., 1986) remains the obvious choice due to its generalist nature.

Original: Apart from the large number of parameters, one difficulty of MLP training arises from the non-convexity of the loss function (see, e.g.,
Revised: Apart from the large number of parameters, one difficulty of MLP training arises from the non-convexity of the loss function (see, e.g.,

Original: Sun, 2020).
Revised: Sun, 2020).

Original: In such situations, the initialization of the network parameters (weights and biases) are of the utmost importance, since it can influence both the optimization stability and the quality of the minimum found.
Revised: In such situations, the initialization of the network parameters (weights and biases) are of the utmost importance, since it can influence both the optimization stability and the quality of the minimum found.

Original: Typically, such initializations are drawn according to independent uniform distributions with a variance decreasing w.r.t.
Revised: Typically, such initializations are drawn according to independent uniform distributions with a variance decreasing w.r.t.

Original: the size of the layer (He et al., 2015).
Revised: the size of the layer (He et al., 2015).

Original: Therefore, one may wonder how to capitalize on methods that are inherently capable of recognizing patterns in tabular data (e.g., tree-based methods) to propose a new NN architecture suitable for tabular data and an initialization procedure that leads to faster convergence and better generalization performance.
Revised: Therefore, one may wonder how to capitalize on methods that are inherently capable of recognizing patterns in tabular data (e.g., tree-based methods) to propose a new NN architecture suitable for tabular data and an initialization procedure that leads to faster convergence and better generalization performance.

Original: 1.1 Related works
Revised: 1.1 Related works

Original: How MLP can be used to handle tabular data remains unclear, especially since a corresponding prior in the MLP architecture adapted to the correlations of the input is not obvious, to say the least.
Revised: How MLP can be used to handle tabular data remains unclear, especially since a corresponding prior in the MLP architecture adapted to the correlations of the input is not obvious, to say the least.

Original: Indeed, none of the existing NN architectures can consistently match the performance of state-of-the-art tree-based predictors on tabular data (Shwartz-Ziv &
Revised: Indeed, none of the existing NN architectures can consistently match the performance of state-of-the-art tree-based predictors on tabular data (Shwartz-Ziv &

Original: Armon, 2022; Gorishniy et al., 2021; and in particular Table 2 in Borisov et al., 2021).
Revised: Armon, 2022; Gorishniy et al., 2021; and in particular Table 2 in Borisov et al., 2021).

Original: Self-attention architectures Specific NN architectures have been proposed to deal with tabular data.
Revised: Self-attention architectures Specific NN architectures have been proposed to deal with tabular data.

Original: For example, TabNet (Arik & Pfister, 2021) uses a sequential self-attention structure to detect relevant features and then applies several networks for prediction.
Revised: For example, TabNet (Arik & Pfister, 2021) uses a sequential self-attention structure to detect relevant features and then applies several networks for prediction.

Original: SAINT (Somepalli et al., 2021), on the other hand, uses a two-dimensional attention structure (on both features and samples) organized in several layers to extract relevant information which is then fed to a classical MLP.
Revised: SAINT (Somepalli et al., 2021), on the other hand, uses a two-dimensional attention structure (on both features and samples) organized in several layers to extract relevant information which is then fed to a classical MLP.

Original: These methods typically require a large amount of data, since the self-attention layers and the output network involve numerous MLP.
Revised: These methods typically require a large amount of data, since the self-attention layers and the output network involve numerous MLP.

Original: Trees and neural networks
Revised: Trees and neural networks

Original: Several solutions have been proposed to leverage the correspondence between tree-based methods and NN, in order to develop more efficient models for processing tabular data.
Revised: Several solutions have been proposed to leverage the correspondence between tree-based methods and NN, in order to develop more efficient models for processing tabular data.

Original: For example, TabNN (Ke et al., 2018) first trains a GBDT on the available data, then extracts a group of features per individual tree, compresses the resulting groups, and uses a tailored Recursive Encoder based on the structure of these groups (with an initialization based on the tree leaves).
Revised: For example, TabNN (Ke et al., 2018) first trains a GBDT on the available data, then extracts a group of features per individual tree, compresses the resulting groups, and uses a tailored Recursive Encoder based on the structure of these groups (with an initialization based on the tree leaves).

Original: Therefore, TabNN employs pre-trained tree-based methods to design more efficient NN.
Revised: Therefore, TabNN employs pre-trained tree-based methods to design more efficient NN.

Original: Conversely, Sethi (1990) Brent (1991), and later Welbl (2014), Richmond et al.
Revised: Conversely, Sethi (1990) Brent (1991), and later Welbl (2014), Richmond et al.

Original: and Biau et al.
Revised: and Biau et al.

Original: (2019) propose to translate decision trees into very specific MLP (made of 3 layers) and use GD training to improve upon the original tree-based method.
Revised: (2019) propose to translate decision trees into very specific MLP (made of 3 layers) and use GD training to improve upon the original tree-based method.

Original: Such procedures can be seen as a way to relax and generalize the partition geometry produced by trees and their aggregation.
Revised: Such procedures can be seen as a way to relax and generalize the partition geometry produced by trees and their aggregation.

Original: To our knowledge, such translations have not been used to boost the training of general NN architectures.
Revised: To our knowledge, such translations have not been used to boost the training of general NN architectures.

Original: 1.2 Contributions
Revised: 1.2 Contributions

Original: In this work, we propose a new method to initialize a potentially deep MLP for learning tasks with tabular data.
Revised: In this work, we propose a new method to initialize a potentially deep MLP for learning tasks with tabular data.

Original: Our method consists in first training a tree-based predictor (RF,
Revised: Our method consists in first training a tree-based predictor (RF,

Original: GBDT or Deep Forest, see Section 2.1) and then using its translation into an MLP as initialization for the first two layers, the deeper ones being randomly initialized.
Revised: GBDT or Deep Forest, see Section 2.1) and then using its translation into an MLP as initialization for the first two layers, the deeper ones being randomly initialized.

Original: With subsequent standard GD training, this procedure is shown to outperform the widely used uniform initialization of MLP (default initialization in Pytorch Paszke et al., 2019) as follows.
Revised: With subsequent standard GD training, this procedure is shown to outperform the widely used uniform initialization of MLP (default initialization in Pytorch Paszke et al., 2019) as follows.

Original: Improved performances.
Revised: Improved performances.

Original: For tabular data, the predictive performances of the MLP after training are improved compared to MLP that use a random initialization.
Revised: For tabular data, the predictive performances of the MLP after training are improved compared to MLP that use a random initialization.

Original: Our procedure also outperforms more complex deep learning procedures based on selfattention and is on par with classical tree-based methods (such as XGBoost).
Revised: Our procedure also outperforms more complex deep learning procedures based on selfattention and is on par with classical tree-based methods (such as XGBoost).

Original: 2. Faster optimization.
Revised: 2. Faster optimization.

Original: The optimization following a tree-based initialization is boosted in the sense that it enjoys a faster convergence towards a (better) empirical minimum: a tree-based initialization results in faster training of the MLP.
Revised: The optimization following a tree-based initialization is boosted in the sense that it enjoys a faster convergence towards a (better) empirical minimum: a tree-based initialization results in faster training of the MLP.

Original: Initializing the first few layers of the MLP with the translation of the tree-based method and initializing randomly the deeper layers is the most successful initialization scheme that we experimented.
Revised: Initializing the first few layers of the MLP with the translation of the tree-based method and initializing randomly the deeper layers is the most successful initialization scheme that we experimented.

Original: This supports the idea that in our method, the (first) tree-based initialized layers act as relevant feature extractors that allow the MLP to detect correlations in the inputs.
Revised: This supports the idea that in our method, the (first) tree-based initialized layers act as relevant feature extractors that allow the MLP to detect correlations in the inputs.

Original: In this context, our approach is dedicated on improving the performance of standard
Revised: In this context, our approach is dedicated on improving the performance of standard

Original: MLP models; therefore it is conceptually different from pre-existing procedures also relying on the translation of tree-based models into NN: (Biau et al., 2019) aim at fine-tuning tree-based methods using a very specific neural network framework (made of only 3 layers).
Revised: MLP models; therefore it is conceptually different from pre-existing procedures also relying on the translation of tree-based models into NN: (Biau et al., 2019) aim at fine-tuning tree-based methods using a very specific neural network framework (made of only 3 layers).

Original: We, on the other hand, use tree-based methods to carefully initialize certain layers of a generic MLP, which is then substantially trained using standard GD strategies.
Revised: We, on the other hand, use tree-based methods to carefully initialize certain layers of a generic MLP, which is then substantially trained using standard GD strategies.

Original: Outline In Section 2, we introduce the predictors in play and describe how tree-based methods can be translated into MLP.
Revised: Outline In Section 2, we introduce the predictors in play and describe how tree-based methods can be translated into MLP.

Original: The core of our analysis is contained in Section 3, where we describe in detail the MLP initialization process and provide extensive numerical evaluations showing the benefits of this method.
Revised: The core of our analysis is contained in Section 3, where we describe in detail the MLP initialization process and provide extensive numerical evaluations showing the benefits of this method.

Original:  Equivalence between trees and MLP
Revised:  Equivalence between trees and MLP

Original: Consider the classical setting of supervised learning in which we are given a set of input/output samples { ( X i , Y i ) } ni =1 drawn i.i.d.
Revised: Consider the classical setting of supervised learning in which we are given a set of input/output samples { ( X i , Y i ) } ni =1 drawn i.i.d.

Original: from some unknown joint distribution.
Revised: from some unknown joint distribution.

Original: Our goal is to construct a (MLP) function to predict the output from the input.
Revised: Our goal is to construct a (MLP) function to predict the output from the input.

Original: To do so, we leverage the translation of tree-based methods into MLP.
Revised: To do so, we leverage the translation of tree-based methods into MLP.

Original: 2.1 Presentation of the predictors in play
Revised: 2.1 Presentation of the predictors in play

Original: Tree-based methods We consider three different tree ensemble methods: Random Forests (RF), Gradient Boosting Decision Trees (GBDT) and Deep Forests (DF).
Revised: Tree-based methods We consider three different tree ensemble methods: Random Forests (RF), Gradient Boosting Decision Trees (GBDT) and Deep Forests (DF).

Original: They all share the same base component: the Decision Tree (DT, for details see Breiman et al., 1984).
Revised: They all share the same base component: the Decision Tree (DT, for details see Breiman et al., 1984).

Original: We call its terminal nodes leaf nodes , which correspond to the cells of the final tree partition.
Revised: We call its terminal nodes leaf nodes , which correspond to the cells of the final tree partition.

Original: RF (Breiman, 2001a) is a predictor consisting of a collection of independently trained and randomized trees.
Revised: RF (Breiman, 2001a) is a predictor consisting of a collection of independently trained and randomized trees.

Original: Its final prediction is made by averaging the predictions of all its DT in regression or by a majority vote in classification.
Revised: Its final prediction is made by averaging the predictions of all its DT in regression or by a majority vote in classification.

Original: GBDT (Friedman, 2001) aims at minimizing a prediction loss function by successively aggregating DT that approximate the opposite gradient of that loss function (see Chen & Guestrin, 2016, for details on XGBoost).
Revised: GBDT (Friedman, 2001) aims at minimizing a prediction loss function by successively aggregating DT that approximate the opposite gradient of that loss function (see Chen & Guestrin, 2016, for details on XGBoost).

Original: DF (Zhou & Feng, 2019) is a hybrid learning procedure in which random forests are used as elementary components (neurons) of a neural-network-like architecture (see Figure 5 and
Revised: DF (Zhou & Feng, 2019) is a hybrid learning procedure in which random forests are used as elementary components (neurons) of a neural-network-like architecture (see Figure 5 and

Original: Appendix A for details).
Revised: Appendix A for details).

Original: Multilayer Perceptron (MLP)
Revised: Multilayer Perceptron (MLP)

Original: The multilayer perceptron is a predictor consisting of a composition of multiple affine functions, with (potentially different) nonlinear activation functions between them.
Revised: The multilayer perceptron is a predictor consisting of a composition of multiple affine functions, with (potentially different) nonlinear activation functions between them.

Original: Standard activation functions include, for instance, the rectified linear unit or the hyperbolic tangent.
Revised: Standard activation functions include, for instance, the rectified linear unit or the hyperbolic tangent.

Original: Deep MLP are a much richer class of predictors than tree-based methods which build simple partitions of the space and output piecewise constant predictions.
Revised: Deep MLP are a much richer class of predictors than tree-based methods which build simple partitions of the space and output piecewise constant predictions.

Original: Therefore, any of the tree-based models presented above can be approximated and in fact exactly rewritten as an MLP as follows.
Revised: Therefore, any of the tree-based models presented above can be approximated and in fact exactly rewritten as an MLP as follows.

Original: 2.2 An exact translation of tree-based methods into MLP
Revised: 2.2 An exact translation of tree-based methods into MLP

Original: From decision tree to 3-layer MLP Recall that a decision tree codes for a partition of the input space in as many parts as there are leaf nodes in the tree.
Revised: From decision tree to 3-layer MLP Recall that a decision tree codes for a partition of the input space in as many parts as there are leaf nodes in the tree.

Original: Given an input x , we can identify the leaf where x falls by examining for each hyperplane of the partition whether x falls on the right or left side of the hyperplane.
Revised: Given an input x , we can identify the leaf where x falls by examining for each hyperplane of the partition whether x falls on the right or left side of the hyperplane.

Original: The prediction is then made by averaging the outputs of all the training samples falling into the leaf of x .
Revised: The prediction is then made by averaging the outputs of all the training samples falling into the leaf of x .

Original: A DT can be thus translated into a highly sparse 3-layer MLP:
Revised: A DT can be thus translated into a highly sparse 3-layer MLP:

Original: 10           xx
Revised: 10           xx

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 10           xx
Revised: 10           xx

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 10           xx
Revised: 10           xx

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: The first layer contains a number of neurons equal to the number of hyperplanes in the partition, each neuron encoding by ± 1 whether x falls on the 
Revised: The first layer contains a number of neurons equal to the number of hyperplanes in the partition, each neuron encoding by ± 1 whether x falls on the left

Revised: side of the hyperplane

Original: The second layer contains a number of neurons equal to the number of leaves in the DT.
Revised: The second layer contains a number of neurons equal to the number of leaves in the DT.

Original: Based on the first layer, it identifies in which leaf x falls and outputs a vector with a single 1 at the leaf position and − 1 everywhere else.
Revised: Based on the first layer, it identifies in which leaf x falls and outputs a vector with a single 1 at the leaf position and − 1 everywhere else.

Original: The last layer contains a single output neuron that returns the tree prediction.
Revised: The last layer contains a single output neuron that returns the tree prediction.

Original: Its weights encode the average output of all training samples for each leaf of the tree.
Revised: Its weights encode the average output of all training samples for each leaf of the tree.

Original: This procedure is explained in detail and formally in Biau et al.
Revised: This procedure is explained in detail and formally in Biau et al.

Original: (2019) and in Appendix B.
Revised: (2019) and in Appendix B.

Original: From RF/GBDT to 3-layer MLP Although RF and GBDT are constructed in different ways, they both average multiple DT predictions to give the final result.
Revised: From RF/GBDT to 3-layer MLP Although RF and GBDT are constructed in different ways, they both average multiple DT predictions to give the final result.

Original: Thus, to translate a RF or a GBDT into an MLP, we simply turn each tree into a 3-layer MLP as described above, and concatenate all the obtained networks to form a wider 3-layer MLP.
Revised: Thus, to translate a RF or a GBDT into an MLP, we simply turn each tree into a 3-layer MLP as described above, and concatenate all the obtained networks to form a wider 3-layer MLP.

Original: When concatenating, we set all weights between the MLP translations of the different trees to 0, since the trees do not interact with each other in predicting the target value for a new feature vector.
Revised: When concatenating, we set all weights between the MLP translations of the different trees to 0, since the trees do not interact with each other in predicting the target value for a new feature vector.

Original: The step in which the responses of the different trees are averaged can be combined with the third layer of the individual tree translations, resulting in a final MLP translation with a total of three layers.
Revised: The step in which the responses of the different trees are averaged can be combined with the third layer of the individual tree translations, resulting in a final MLP translation with a total of three layers.

Original: From Deep Forests to deeper MLP A Deep Forest is a cascade of Random Forests.
Revised: From Deep Forests to deeper MLP A Deep Forest is a cascade of Random Forests.

Original: As such, it can be translated into an MLP containing the MLP translations of the different RF in cascade, resulting in a deeper and wider MLP (note that the obtained MLP has a number of layers that is a multiple of 3).Furthermore, in the Deep Forest architecture, the input vector is concatenated to the output of each intermediate layer.
Revised: As such, it can be translated into an MLP containing the MLP translations of the different RF in cascade, resulting in a deeper and wider MLP (note that the obtained MLP has a number of layers that is a multiple of 3).Furthermore, in the Deep Forest architecture, the input vector is concatenated to the output of each intermediate layer.

Original: To mimic these skip connections in the MLP, we add additional neurons to each layer, except for the last three, which encode an identity mapping.
Revised: To mimic these skip connections in the MLP, we add additional neurons to each layer, except for the last three, which encode an identity mapping.

Original: Appendix A gives more insights into DF and their MLP translations.
Revised: Appendix A gives more insights into DF and their MLP translations.

Original: In particular, perfect translation of a DF suffers from numerical instabilities due to the replication of catastrophic cancellations (the deeper the DF, the greater their amplitude, cf Appendix D).
Revised: In particular, perfect translation of a DF suffers from numerical instabilities due to the replication of catastrophic cancellations (the deeper the DF, the greater their amplitude, cf Appendix D).

Original: This does not impact the sequel of the study, which relies on MLP approximations introduced in Section 2.2.
Revised: This does not impact the sequel of the study, which relies on MLP approximations introduced in Section 2.2.

Original: 2.3 Relaxing tree-based translation to allow gradient descent training
Revised: 2.3 Relaxing tree-based translation to allow gradient descent training

Original: As shown in the previous section, one can construct an MLP that exactly reproduces a tree-based predictor.
Revised: As shown in the previous section, one can construct an MLP that exactly reproduces a tree-based predictor.

Original: However this translation involves (i) piecewise constant activation functions (sign) and (ii) different activation functions in a same layer (sign and identity when translating DF).
Revised: However this translation involves (i) piecewise constant activation functions (sign) and (ii) different activation functions in a same layer (sign and identity when translating DF).

Original: These constraints can hinder the MLP training, which relies on GD strategies (requiring differentiability), as well as efficient implementation tricks, given that automatic differentiation libraries only support one activation function per layer.
Revised: These constraints can hinder the MLP training, which relies on GD strategies (requiring differentiability), as well as efficient implementation tricks, given that automatic differentiation libraries only support one activation function per layer.

Original: Therefore, given a pre-trained tree-based predictor (RF, GBDT or DF), we aim at relaxing its translation into a MLP, mimicking its behavior as closely as possible but in a compatible way with standard NN training.
Revised: Therefore, given a pre-trained tree-based predictor (RF, GBDT or DF), we aim at relaxing its translation into a MLP, mimicking its behavior as closely as possible but in a compatible way with standard NN training.

Original: From tree-based methods to differentiable MLP To do so, Welbl (2014);
Revised: From tree-based methods to differentiable MLP To do so, Welbl (2014);

Original: Biau et al.
Revised: Biau et al.

Original: (2019) consider the differentiable tanh activation, well suited for approximating both the sign and identity functions.
Revised: (2019) consider the differentiable tanh activation, well suited for approximating both the sign and identity functions.

Original: Indeed, this can be achieved by multiplying or dividing the output of a neuron by a large constant before applying the function tanh and rescaling the result accordingly if necessary, i.e. for large enough a, c > 0 , sign( x ) ≈ tanh( ax ) and x ≈ c tanh (cid:0) xc (cid:1) .
Revised: Indeed, this can be achieved by multiplying or dividing the output of a neuron by a large constant before applying the function tanh and rescaling the result accordingly if necessary, i.e. for large enough a, c > 0 , sign( x ) ≈ tanh( ax ) and x ≈ c tanh (cid:0) xc (cid:1) .

Original: However, we cannot choose a arbitrarily large as this would make gradients vanish during the network optimization (the function being flat on most of the space), and hinder training.
Revised: However, we cannot choose a arbitrarily large as this would make gradients vanish during the network optimization (the function being flat on most of the space), and hinder training.

Original: We therefore introduce 4 hyper-parameters for the MLP encoding of any tree-based method that regulate the degree of approximation for the activation functions after the first, second and third layers of a decision tree translation, as well as for the identity mapping, respectively denoted by strength01 , strength12 , strength23 and strength id .
Revised: We therefore introduce 4 hyper-parameters for the MLP encoding of any tree-based method that regulate the degree of approximation for the activation functions after the first, second and third layers of a decision tree translation, as well as for the identity mapping, respectively denoted by strength01 , strength12 , strength23 and strength id .

Original: Hyperparameter choice The use of the tanh activation function involves extra hyperparameters.
Revised: Hyperparameter choice The use of the tanh activation function involves extra hyperparameters.

Original: We study the influence of each one, by making them vary in some range (keeping the others fixed to 10 10 , resulting in an almost perfect approximation of the sign and identity functions), see Appendix D.1 for details.
Revised: We study the influence of each one, by making them vary in some range (keeping the others fixed to 10 10 , resulting in an almost perfect approximation of the sign and identity functions), see Appendix D.1 for details.

Original: Our analysis shows that increasing the hyperparameters beyond some limit value is no longer beneficial (as the activation functions are already perfectly approximated) and, across multiple data sets, these limit values are similar.
Revised: Our analysis shows that increasing the hyperparameters beyond some limit value is no longer beneficial (as the activation functions are already perfectly approximated) and, across multiple data sets, these limit values are similar.

Original: We also exhibit relevant search spaces that will allow us to find optimal HP values for each application.
Revised: We also exhibit relevant search spaces that will allow us to find optimal HP values for each application.

Original:  A new initialization method for MLP training
Revised:  A new initialization method for MLP training

Original: In this section, we study the impact of tree-based initialization methods for MLP training when dealing with tabular data.
Revised: In this section, we study the impact of tree-based initialization methods for MLP training when dealing with tabular data.

Original: The latter empirically proves to be always preferable to standard random initialization and makes MLP a competitive predictor for tabular data.
Revised: The latter empirically proves to be always preferable to standard random initialization and makes MLP a competitive predictor for tabular data.

Original: 3.1 Our proposal
Revised: 3.1 Our proposal

Original: Random initialization is the most common technique for initializing MLP prior to stochastic gradient training.
Revised: Random initialization is the most common technique for initializing MLP prior to stochastic gradient training.

Original: It consists in setting all layer parameters to random values of small magnitude centered at 0.
Revised: It consists in setting all layer parameters to random values of small magnitude centered at 0.

Original: More precisely, all parameter values of the j -th layer are uniformly drawn in [ − 1 / (cid:112)
Revised: More precisely, all parameter values of the j -th layer are uniformly drawn in [ − 1 / (cid:112)

Original: d j , 1 / (cid:112) d
Revised: d j , 1 / (cid:112) d

Original: j ] where d j is the layer input dimension; this is the default behaviour of most MLP implementations such as nn.
Revised: j ] where d j is the layer input dimension; this is the default behaviour of most MLP implementations such as nn.

Original: Linear in PyTorch (Paszke et al., 2019).
Revised: Linear in PyTorch (Paszke et al., 2019).

Original: We introduce new ways of initializing an MLP for learning with tabular data, by leveraging the recasting of tree-based methods in a neural network fashion:
Revised: We introduce new ways of initializing an MLP for learning with tabular data, by leveraging the recasting of tree-based methods in a neural network fashion:

Original: • RF/GBDT initialization.
Revised: • RF/GBDT initialization.

Original: First, a RF/GBDT is fitted to the training data and transformed into a 3 -layer neural network, following the procedure described in Section 2.
Revised: First, a RF/GBDT is fitted to the training data and transformed into a 3 -layer neural network, following the procedure described in Section 2.

Original: The first two layers of this network are used to initialize the first two layers of the network of interest.
Revised: The first two layers of this network are used to initialize the first two layers of the network of interest.

Original: Thus, upon initialization, these first two layers encode the RF/GBDT partition.
Revised: Thus, upon initialization, these first two layers encode the RF/GBDT partition.

Original: The parameters of the third and all subsequent layers are randomly initialized as described above.
Revised: The parameters of the third and all subsequent layers are randomly initialized as described above.

Original: See Figure 7 in Appendix C for an illustration.
Revised: See Figure 7 in Appendix C for an illustration.

Original: • DF initialization.
Revised: • DF initialization.

Original: Similarly as above, a Deep Forest (DF) using ℓ forest layers is first fitted to the training data.
Revised: Similarly as above, a Deep Forest (DF) using ℓ forest layers is first fitted to the training data.

Original: The first 3 ℓ − 1 layers of the MLP are then initialized using the first 3 ℓ − 1 layers of the MLP encoding of this pre-trained DF.
Revised: The first 3 ℓ − 1 layers of the MLP are then initialized using the first 3 ℓ − 1 layers of the MLP encoding of this pre-trained DF.

Original: The parameters of the 3 ℓ -th and all subsequent layers are randomly initialized as explained above.
Revised: The parameters of the 3 ℓ -th and all subsequent layers are randomly initialized as explained above.

Original: These tree-based initialization techniques may seem far-fetched at first glance, but they are actually consistent with recent approaches to adapting Deep Learning models for tabular data.
Revised: These tree-based initialization techniques may seem far-fetched at first glance, but they are actually consistent with recent approaches to adapting Deep Learning models for tabular data.

Original: The key to interpreting them is to think of the first (tree-based initialized) layers of the MLP as a feature extractor that produces an abstract representation of the input data (in fact, this is a vector encoding the tree-based predictor’s space partition in which the observation lies).
Revised: The key to interpreting them is to think of the first (tree-based initialized) layers of the MLP as a feature extractor that produces an abstract representation of the input data (in fact, this is a vector encoding the tree-based predictor’s space partition in which the observation lies).

Original: The subsequent randomly initialized layers, once trained, then perform the prediction task based on this abstract representation.
Revised: The subsequent randomly initialized layers, once trained, then perform the prediction task based on this abstract representation.

Original: 3.2 Experimental setup
Revised: 3.2 Experimental setup

Original: Datasets & learning tasks We compare prediction performances on a total of 10 datasets: 3 regression datasets (Airbnb, Diamonds and Housing), 5 binary classification datasets (Adult, Bank, Blastchar, Heloc, Higgs) and 2 multi-class classification datasets (Covertype and Volkert).
Revised: Datasets & learning tasks We compare prediction performances on a total of 10 datasets: 3 regression datasets (Airbnb, Diamonds and Housing), 5 binary classification datasets (Adult, Bank, Blastchar, Heloc, Higgs) and 2 multi-class classification datasets (Covertype and Volkert).

Original: We mostly chose data sets that are used for benchmarking in relevent literature: Adult, Heloc, Housing, Higgs and Covertype are used by Borisov et al.
Revised: We mostly chose data sets that are used for benchmarking in relevent literature: Adult, Heloc, Housing, Higgs and Covertype are used by Borisov et al.

Original: (2021) and
Revised: (2021) and

Original: Bank, Blastchar and Volkert are used by Somepalli et al.
Revised: Bank, Blastchar and Volkert are used by Somepalli et al.

Original: Moreover, we add Airbnb and Diamonds to balance the different types of prediction tasks.
Revised: Moreover, we add Airbnb and Diamonds to balance the different types of prediction tasks.

Original: The considered datasets are all medium-sized (10–60k observations) except for Covertype and Higgs (approx.
Revised: The considered datasets are all medium-sized (10–60k observations) except for Covertype and Higgs (approx.

Original: 500k observations).
Revised: 500k observations).

Original: Details about the datasets are given in Appendix E.1.
Revised: Details about the datasets are given in Appendix E.1.

Original: Predictors We consider the following tree-based predictors: Random Forest (RF), Deep
Revised: Predictors We consider the following tree-based predictors: Random Forest (RF), Deep

Original: Forest (DF, Zhou & Feng, 2017) and XGBoost (denoted by GBDT, Chen & Guestrin, 2016).
Revised: Forest (DF, Zhou & Feng, 2017) and XGBoost (denoted by GBDT, Chen & Guestrin, 2016).

Original: The latter usually achieves state-of-the-art performances on tabular data sets (see, e.g.,
Revised: The latter usually achieves state-of-the-art performances on tabular data sets (see, e.g.,

Original: Shwartz-Ziv & Armon, 2022; Gorishniy et al., 2021; Borisov et al., 2021).
Revised: Shwartz-Ziv & Armon, 2022; Gorishniy et al., 2021; Borisov et al., 2021).

Original: We also consider deep learning approaches: MLP with default uniform initialization (MLP rand.
Revised: We also consider deep learning approaches: MLP with default uniform initialization (MLP rand.

Original: init.) or tree-based initialization (resp.
Revised: init.) or tree-based initialization (resp.

Original: MLP RF init.
Revised: MLP RF init.

Original: , MLP GBDT init.
Revised: , MLP GBDT init.

Original: and MLP DF init.); and a transformer architecture SAINT Somepalli et al.
Revised: and MLP DF init.); and a transformer architecture SAINT Somepalli et al.

Original: This complex architecture is specifically designed for applications on tabular data and includes self-attention and inter-sample attention layers that extract feature correlations that are then passed on to an MLP.
Revised: This complex architecture is specifically designed for applications on tabular data and includes self-attention and inter-sample attention layers that extract feature correlations that are then passed on to an MLP.

Original: For regression and classification tasks, we use the mean-squared error (MSE) and cross-entropy loss for NN training, respectively.
Revised: For regression and classification tasks, we use the mean-squared error (MSE) and cross-entropy loss for NN training, respectively.

Original: We choose SAINT as a baseline model as it is reported to outperform all other NN predictors on most of our data sets (all except Airbnb and Diamonds, see Borisov et al., 2021; Somepalli et al., 2021).
Revised: We choose SAINT as a baseline model as it is reported to outperform all other NN predictors on most of our data sets (all except Airbnb and Diamonds, see Borisov et al., 2021; Somepalli et al., 2021).

Original: [Figure]
Revised: [Figure]

Original:  optimization All  are trained using the Adam optimizer (Kingma & Ba, 2014).
Revised: Parameter optimization All NN are trained using the Adam optimizer (Kingma & Ba, 2014).

Original: All hyper-parameters (HP) are determined empirically using the optuna library (Akiba et al., 2019) for Bayesian optimization.
Revised: All hyper-parameters (HP) are determined empirically using the optuna library (Akiba et al., 2019) for Bayesian optimization.

Original: For most HP, we use the default search spaces of Borisov et al.
Revised: For most HP, we use the default search spaces of Borisov et al.

Original: For all HP tuning the tree-to-MLP translation, we have identified relevant search spaces (see Appendix D.1).
Revised: For all HP tuning the tree-to-MLP translation, we have identified relevant search spaces (see Appendix D.1).

Original: An overview of all search spaces used for each method and the HP selected for experimental protocol P2 can be found in Appendix E.5.
Revised: An overview of all search spaces used for each method and the HP selected for experimental protocol P2 can be found in Appendix E.5.

Original: The quantity minimized during HP tuning is the model’s validation loss, and the smallest validation loss that occurred during training for MLP-based models.
Revised: The quantity minimized during HP tuning is the model’s validation loss, and the smallest validation loss that occurred during training for MLP-based models.

Original: 3.3 A betterMLP initialization for a better optimization
Revised: 3.3 A better

Revised: MLP initialization for a better optimization

Original: In this subsection, the optimization of standard
Revised: In this subsection, the optimization of standard

Original: MLP is shown to benefit from the proposed initialization technique.
Revised: MLP is shown to benefit from the proposed initialization technique.

Original: Experiments have been conducted on 6 out of the 10 data sets.
Revised: Experiments have been conducted on 6 out of the 10 data sets.

Original: Experimental protocol 1 (P1) To obtain com- parable optimization processes, we ensure that all MLP-related hyper-parameters (width, depth, learning rate), are identical for all the
Revised: Experimental protocol 1 (P1) To obtain com- parable optimization processes, we ensure that all MLP-related hyper-parameters (width, depth, learning rate), are identical for all the

Original: MLP regardless of the initialization scheme.
Revised: MLP regardless of the initialization scheme.

Original: These HP are chosen to maximize the predictive performance of the standard randomly initialized MLP.
Revised: These HP are chosen to maximize the predictive performance of the standard randomly initialized MLP.

Original: All HP related to the ini- tialization technique (HP for the tree-based predictors and their translation) are optimized independently for each tree-based initializations.
Revised: All HP related to the ini- tialization technique (HP for the tree-based predictors and their translation) are optimized independently for each tree-based initializations.

Original: Results Figure 2 shows that for most data sets, the use of tree-based initialization methods for MLP training provides a faster convergence towards a better minimum (in terms of generalization) than random initialization.
Revised: Results Figure 2 shows that for most data sets, the use of tree-based initialization methods for MLP training provides a faster convergence towards a better minimum (in terms of generalization) than random initialization.

Original: This is all the more remarkable since Protocol P1 has been calibrated in favor of random initialization.
Revised: This is all the more remarkable since Protocol P1 has been calibrated in favor of random initialization.

Original: Among tree-based initializers,
Revised: Among tree-based initializers,

Original: GBDT initializations outperform or are on par with RF initializations in terms of the optimization behavior on all regression and binary classification problems.
Revised: GBDT initializations outperform or are on par with RF initializations in terms of the optimization behavior on all regression and binary classification problems.

Original: However, for multiclass classification problems, the advantages of tree-based initialization seem to be limited.
Revised: However, for multiclass classification problems, the advantages of tree-based initialization seem to be limited.

Original: This is probably due to the fact that the MLP architecture at play is tailored for random initialization, being thus too restrictive for tree-based initializers.
Revised: This is probably due to the fact that the MLP architecture at play is tailored for random initialization, being thus too restrictive for tree-based initializers.

Original: Experiments presented in Appendix E.3 with fixed arbitrary widths corroborate this idea: in this case, the RF initialization is beneficial for the optimization process.
Revised: Experiments presented in Appendix E.3 with fixed arbitrary widths corroborate this idea: in this case, the RF initialization is beneficial for the optimization process.

Original: For the Adult, Bank, and Volkert data sets, Figure 2 also shows the performance of each method at initialization.
Revised: For the Adult, Bank, and Volkert data sets, Figure 2 also shows the performance of each method at initialization.

Original: None of these procedures leads to a better MLP performance at initialization (due to both the non-exact translation from trees to MLP and to the additional randomly initialized layers), but rather help guiding the MLP in its learning process.
Revised: None of these procedures leads to a better MLP performance at initialization (due to both the non-exact translation from trees to MLP and to the additional randomly initialized layers), but rather help guiding the MLP in its learning process.

Original: 3.4 A better MLP initialization for a better generalization
Revised: 3.4 A better MLP initialization for a better generalization

Original: In this subsection, tree-based initialization methods are shown to systematically improve the predictive power of neural networks compared to random initialization.
Revised: In this subsection, tree-based initialization methods are shown to systematically improve the predictive power of neural networks compared to random initialization.

Original: We compare our procedure to the predictors described in Section 3.2, but also to 3 other NN techniques: one close to the default uniform initialization (Xavier init.
Revised: We compare our procedure to the predictors described in Section 3.2, but also to 3 other NN techniques: one close to the default uniform initialization (Xavier init.

Original: , see Glorot & Bengio, 2010),
Revised: , see Glorot & Bengio, 2010),

Original: one using random orthonal matrices (LUSV init.
Revised: one using random orthonal matrices (LUSV init.

Original: , see Mishkin & Matas, 2015) and the winning ticket lottery strategy (WT prun.
Revised: , see Mishkin & Matas, 2015) and the winning ticket lottery strategy (WT prun.

Original: , see Frankle & Carbin, 2018), which is a pruning method used during training to end up with a sparse NN.
Revised: , see Frankle & Carbin, 2018), which is a pruning method used during training to end up with a sparse NN.

Original: The reader may refer to Appendix E.4.
Revised: The reader may refer to Appendix E.4.

Original: for more details about these three techniques.
Revised: for more details about these three techniques.

Original: Experimental protocol 2 (P2)
Revised: Experimental protocol 2 (P2)

Original: Each MLP is trained on 100 epochs, but with HP tuned depending on the initialization technique.
Revised: Each MLP is trained on 100 epochs, but with HP tuned depending on the initialization technique.

Original: For maximum comparability, the optimization budget is strictly the same for all methods (100 “optuna” iterations each, where one optuna iteration consists of one hold-out validation).
Revised: For maximum comparability, the optimization budget is strictly the same for all methods (100 “optuna” iterations each, where one optuna iteration consists of one hold-out validation).

Original: In particular, when using a tree-based initializer, we use 25 HP optimization iterations to find optimal HP for the tree-based predictor, fix these HPs, and then use the remaining 75 iterations to determine optimal HP for the MLP.
Revised: In particular, when using a tree-based initializer, we use 25 HP optimization iterations to find optimal HP for the tree-based predictor, fix these HPs, and then use the remaining 75 iterations to determine optimal HP for the MLP.

Original: For all NN approaches, the model with the best performance on the validation set during training is kept (using the classical early-stopping procedure).
Revised: For all NN approaches, the model with the best performance on the validation set during training is kept (using the classical early-stopping procedure).

Original: Performances are measured via the MSE for regression, the AUROC score (AUC) for binary classification and the accuracy (Acc.) for multi-class classification, averaging 5 runs of 5-fold cross-validation.
Revised: Performances are measured via the MSE for regression, the AUROC score (AUC) for binary classification and the accuracy (Acc.) for multi-class classification, averaging 5 runs of 5-fold cross-validation.

Original: Results Table 1 shows that RF or GBDT initialization strictly outperform random initialization, in terms of final generalization performance, for all data sets except Covertype (for which performances are similar).
Revised: Results Table 1 shows that RF or GBDT initialization strictly outperform random initialization, in terms of final generalization performance, for all data sets except Covertype (for which performances are similar).

Original: They also systematically achieve better results than the
Revised: They also systematically achieve better results than the

Original: LUSV and Xavier init.
Revised: LUSV and Xavier init.

Original: and are better on all but 2 datasets than the WT pruning procedure which is a more refined procedure.
Revised: and are better on all but 2 datasets than the WT pruning procedure which is a more refined procedure.

Original: Additionally, the MLP using both RF and GBDT initialization techniques outperform SAINT on all medium-sized data sets and fall short on large data sets (Higgs and Covertype).
Revised: Additionally, the MLP using both RF and GBDT initialization techniques outperform SAINT on all medium-sized data sets and fall short on large data sets (Higgs and Covertype).

Original: Despite its simplicity , the proposed method (based on RF or GBDT) is on par with GBDT on half of the data sets, ranking MLP as relevant predictors for tabular data.
Revised: Despite its simplicity , the proposed method (based on RF or GBDT) is on par with GBDT on half of the data sets, ranking MLP as relevant predictors for tabular data.

Original: Note that the GBDT used for initialization of the MLP is way less powerful than the best one found here (see details in Tables 10 and 11).
Revised: Note that the GBDT used for initialization of the MLP is way less powerful than the best one found here (see details in Tables 10 and 11).

Original: This shows that our procedure produces, with a relatively low initialization cost, powerful MLP relevant for tabular data.
Revised: This shows that our procedure produces, with a relatively low initialization cost, powerful MLP relevant for tabular data.

Original: Among the tree-based initializers, RF is on par with or outperforms GBDT initialization on all data sets but Housing.
Revised: Among the tree-based initializers, RF is on par with or outperforms GBDT initialization on all data sets but Housing.

Original: DF initialization, for its part, cannot compete with RF and GBDT initialization, despite showing some improvement over the random one (except for Covertype and Volkert).
Revised: DF initialization, for its part, cannot compete with RF and GBDT initialization, despite showing some improvement over the random one (except for Covertype and Volkert).

Original: This underlines that injecting prior information via tree-based methods into the first layers of a
Revised: This underlines that injecting prior information via tree-based methods into the first layers of a

Original: MLP is, among all the aforementioned methods, the best way to improve its performance.
Revised: MLP is, among all the aforementioned methods, the best way to improve its performance.

Original: [Table]
Revised: [Table]

Original: The interested reader may find a comparison of the optimization procedures of all MLP methods and SAINT (Figure 13) and tables summarizing all HP (Tables 10 and 11) in Appendix E.5.2.
Revised: The interested reader may find a comparison of the optimization procedures of all MLP methods and SAINT (Figure 13) and tables summarizing all HP (Tables 10 and 11) in Appendix E.5.2.

Original: We remark that tree-based initializers generally bring into play wider networks with similar depths (fixed width of 2048 and adaptive depth between 4 and 10) compared to MLP with default initialization.
Revised: We remark that tree-based initializers generally bring into play wider networks with similar depths (fixed width of 2048 and adaptive depth between 4 and 10) compared to MLP with default initialization.

Original: Yet, for most data sets, the overall procedure is computationally more efficient than state-of-the-art deep learning architectures like SAINT, both in terms of number of parameters and training time (see Tables 6-8 in Appendix E.4).
Revised: Yet, for most data sets, the overall procedure is computationally more efficient than state-of-the-art deep learning architectures like SAINT, both in terms of number of parameters and training time (see Tables 6-8 in Appendix E.4).

Original: 3.5 Analyzing key elements of the new initialization methods
Revised: 3.5 Analyzing key elements of the new initialization methods

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: Influence of the MLP width We mainly use standard search spaces from (Borisov et al., 2021) to determine the optimal hyper-parameters for each model.
Revised: Influence of the MLP width We mainly use standard search spaces from (Borisov et al., 2021) to determine the optimal hyper-parameters for each model.

Original: However, the MLP width is an exception to this.
Revised: However, the MLP width is an exception to this.

Original: The standard search spaces used in the literature usually involve MLP with a few hundred neurons per layer (e.g. up to 100 neurons in Borisov et al., 2021); yet, in this work, we consider MLP with a width up to 2048 neurons.
Revised: The standard search spaces used in the literature usually involve MLP with a few hundred neurons per layer (e.g. up to 100 neurons in Borisov et al., 2021); yet, in this work, we consider MLP with a width up to 2048 neurons.

Original: Large MLP are actually very beneficial for tree-based initialization methods as they allow the use of more expressive tree-based models in the initialization step.
Revised: Large MLP are actually very beneficial for tree-based initialization methods as they allow the use of more expressive tree-based models in the initialization step.

Original: Figure 3 compares the performance of an MLP with random/GDBT initializations and various widths.
Revised: Figure 3 compares the performance of an MLP with random/GDBT initializations and various widths.

Original: There is no gain in prediction by using wider (thus more complex) NN, when randomly initialized.
Revised: There is no gain in prediction by using wider (thus more complex) NN, when randomly initialized.

Original: This is corroborated by the results of Table 4: for all regression and binary classification data sets, the performance of our (potentially much wider) MLP with random initialisation is consistently close to the literature values, and only increases for multi-class classification tasks.
Revised: This is corroborated by the results of Table 4: for all regression and binary classification data sets, the performance of our (potentially much wider) MLP with random initialisation is consistently close to the literature values, and only increases for multi-class classification tasks.

Original: However, an MLP initialized with GBDT significantly benefits from enlarging the NN width (justifying a fixed width of 2048 for tree-based initialized MLP).
Revised: However, an MLP initialized with GBDT significantly benefits from enlarging the NN width (justifying a fixed width of 2048 for tree-based initialized MLP).

Original: This confirms the idea that tree-based initialization helps revealing relevant features to the MLP, all the more as the width increases, and by doing so, boosts the MLP performance after training.
Revised: This confirms the idea that tree-based initialization helps revealing relevant features to the MLP, all the more as the width increases, and by doing so, boosts the MLP performance after training.

Original: Performance of the initializer Another interesting step in unraveling the essence of the new initialization method is to understand which characteristics of a tree-based model are relevant to its success as an initializer.
Revised: Performance of the initializer Another interesting step in unraveling the essence of the new initialization method is to understand which characteristics of a tree-based model are relevant to its success as an initializer.

Original: Undoubtedly, its predictive accuracy plays an important role, but does this aspect alone suffice to characterize the success of the new initialization method?
Revised: Undoubtedly, its predictive accuracy plays an important role, but does this aspect alone suffice to characterize the success of the new initialization method?

Original: Figure 14 compares the predictive performance of different RF/GBDT initializers and the performance of the respective MLP after training.
Revised: Figure 14 compares the predictive performance of different RF/GBDT initializers and the performance of the respective MLP after training.

Original: As the figure illustrates, a better performance of the tree-based predictor used for initialization does not always lead to a better performance of the MLP after training (see Airbnb and Volkert).
Revised: As the figure illustrates, a better performance of the tree-based predictor used for initialization does not always lead to a better performance of the MLP after training (see Airbnb and Volkert).

Original: This observation suggests that other aspects, such as the expressiveness of the feature interactions captured by the initializer, the structure it induces on the MLP or the weight distributions of the initializer, must also play a significant role in the initialization method’s success.
Revised: This observation suggests that other aspects, such as the expressiveness of the feature interactions captured by the initializer, the structure it induces on the MLP or the weight distributions of the initializer, must also play a significant role in the initialization method’s success.

Original: MLP sparsity Finally, we investigate the structure that tree-based initialization induces on the MLP after training.
Revised: MLP sparsity Finally, we investigate the structure that tree-based initialization induces on the MLP after training.

Original: Figure 4 shows the weight distributions of the three first and  last layers before and after MLP training, for random, RF and GBDT initializations on Housing (see Appendix E.7 for more data sets).
Revised: Figure 4 shows the weight distributions of the three first and the last layers before and after MLP training, for random, RF and GBDT initializations on Housing (see Appendix E.7 for more data sets).

Original: It indicates that the weight distribution on the first two layers change significantly during training when the MLP is randomly initialized: the weights are uniformly distributed at epoch 0 but appear to be Gaussian after training.
Revised: It indicates that the weight distribution on the first two layers change significantly during training when the MLP is randomly initialized: the weights are uniformly distributed at epoch 0 but appear to be Gaussian after training.

Original: When RF or GBDT initializers are used instead, the weights of the first two layers are sparsely distributed at epoch 0 by construction, and their distribution is preserved during training (notice the logarithmic y-axis for these plots in Figure 4).
Revised: When RF or GBDT initializers are used instead, the weights of the first two layers are sparsely distributed at epoch 0 by construction, and their distribution is preserved during training (notice the logarithmic y-axis for these plots in Figure 4).

Original: Note that the (uniform) distribution of the weights in other layers is also preserved through training (third and last lines of Figure 4).
Revised: Note that the (uniform) distribution of the weights in other layers is also preserved through training (third and last lines of Figure 4).

Original: This means that our initialization technique, in combination with SGD optimization strategies, introduces an implicit regularization of the NN optimization: the sparse structure of the initialisation (on first layers) is maintained.
Revised: This means that our initialization technique, in combination with SGD optimization strategies, introduces an implicit regularization of the NN optimization: the sparse structure of the initialisation (on first layers) is maintained.

Original: This is very similar to the CNN architecture (constrained by design), a very successful class of NN designed for image processing.
Revised: This is very similar to the CNN architecture (constrained by design), a very successful class of NN designed for image processing.

Original: Besides, the weight distributions are not squeezed towards zero during learning when sparse initialization is used, preventing poor generalization performances according to previous works (Neal, 2012; Blundell et al., 2015).
Revised: Besides, the weight distributions are not squeezed towards zero during learning when sparse initialization is used, preventing poor generalization performances according to previous works (Neal, 2012; Blundell et al., 2015).

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Looking at Figure 4, one may get the impression that the weights in the first layers remain unchanged during GD training, and that ultimately no learning takes place in these layers.
Revised: Looking at Figure 4, one may get the impression that the weights in the first layers remain unchanged during GD training, and that ultimately no learning takes place in these layers.

Original: However, numerical experiments (see Appendix E.4.3) show that the weights of all layers are modified during learning; the first two layers actually undergo the greatest changes.
Revised: However, numerical experiments (see Appendix E.4.3) show that the weights of all layers are modified during learning; the first two layers actually undergo the greatest changes.

Original: As such, training the RF/GBDT-initialized layers via GD strategies is essential for the success of the new initialization method.
Revised: As such, training the RF/GBDT-initialized layers via GD strategies is essential for the success of the new initialization method.

Original:  Conclusion and Future work
Revised:  Conclusion and Future work

Original: This work builds upon the permeability that exists between tree methods and neural networks, in particular how the former can help the latter during training, with tabular inputs.
Revised: This work builds upon the permeability that exists between tree methods and neural networks, in particular how the former can help the latter during training, with tabular inputs.

Original: We proposed new methods for smartly initializing the first layers of standard MLP using pre-trained tree-based methods.
Revised: We proposed new methods for smartly initializing the first layers of standard MLP using pre-trained tree-based methods.

Original: The sparsity of this initialization is preserved during training, which shows that it encodes relevant correlations between the data features.
Revised: The sparsity of this initialization is preserved during training, which shows that it encodes relevant correlations between the data features.

Original: Among deep learning methods, such initializations of MLP always improve the performance compared to the widely used random initialization, and provide an easy-to-use and more efficient alternative to SAINT (attention-based method) for tabular data.
Revised: Among deep learning methods, such initializations of MLP always improve the performance compared to the widely used random initialization, and provide an easy-to-use and more efficient alternative to SAINT (attention-based method) for tabular data.

Original: The performance of this wiselyinitialized MLP is remarkably approaching that of XGBoost, which so far reigns supreme for learning tasks on tabular data.
Revised: The performance of this wiselyinitialized MLP is remarkably approaching that of XGBoost, which so far reigns supreme for learning tasks on tabular data.

Original: Limitations & future work While our procedure is quite generic, some restrictions are noticeable.
Revised: Limitations & future work While our procedure is quite generic, some restrictions are noticeable.

Original: First, our analysis only allows to initialize neural networks with tanh activation functions; removing this limitation by considering ReLU is a good avenue for future work.
Revised: First, our analysis only allows to initialize neural networks with tanh activation functions; removing this limitation by considering ReLU is a good avenue for future work.

Original: Besides, while quite reasonable, our initialization is more time-consuming than the random (default) one.
Revised: Besides, while quite reasonable, our initialization is more time-consuming than the random (default) one.

Original: Moreover, we need to further investigate the benefits of our initialization method on very large data sets.
Revised: Moreover, we need to further investigate the benefits of our initialization method on very large data sets.

Original: Finally, another interesting direction could be using the efficient hyperparameter search in tree-based methods to automatically determine a good default NN architecture.
Revised: Finally, another interesting direction could be using the efficient hyperparameter search in tree-based methods to automatically determine a good default NN architecture.

Original: A Details on Deep Forest (DF) and its translation
Revised: A Details on Deep Forest (DF) and its translation

Original: The layers of DF are composed of an assortment of Breiman’s Random Forests and
Revised: The layers of DF are composed of an assortment of Breiman’s Random Forests and

Original: Completely-Random Forests (CRF, Fan et al. (2003)) and are trained one after another in a cascade manner.
Revised: Completely-Random Forests (CRF, Fan et al. (2003)) and are trained one after another in a cascade manner.

Original: At a given layer, the outputs of all forests are concatenated, together with the raw input data.
Revised: At a given layer, the outputs of all forests are concatenated, together with the raw input data.

Original: This new vector serves as input for the next DF layer.
Revised: This new vector serves as input for the next DF layer.

Original: This process is repeated for each layer and the final output is obtained by averaging the forest outputs of the best layer (without raw data).
Revised: This process is repeated for each layer and the final output is obtained by averaging the forest outputs of the best layer (without raw data).

Original: neural networks.
Revised: neural networks.

Original: We believe that in order to tackle complicated learning tasks, it is likely that learning models have to go deep.
Revised: We believe that in order to tackle complicated learning tasks, it is likely that learning models have to go deep.

Original: Current deep models, however, are always neural networks, multiple layers of parameterized differentiable nonlinear modules that can be trained by backpropagation.
Revised: Current deep models, however, are always neural networks, multiple layers of parameterized differentiable nonlinear modules that can be trained by backpropagation.

Original: It is interesting to consider whether deep learning can be realized with other modules, because they have their own advantages and may exhibit great potentials if being able to go deep.
Revised: It is interesting to consider whether deep learning can be realized with other modules, because they have their own advantages and may exhibit great potentials if being able to go deep.

Original: This paper devotes to addressing this fundamental question and illustrates how to construct deep forest; this may open a door towards alternative to deep neural networks for many tasks.
Revised: This paper devotes to addressing this fundamental question and illustrates how to construct deep forest; this may open a door towards alternative to deep neural networks for many tasks.

Original: In the next sections we will introduce gcForest and report on experiments, followed by related work and conclusion.
Revised: In the next sections we will introduce gcForest and report on experiments, followed by related work and conclusion.

Original:  The Proposed Approach
Revised:  The Proposed Approach

Original: In this section we will ﬁrst introduce the cascade forest structure, and then the multi-grained scanning, followed by the overall architecture and remarks on hyper-parameters.
Revised: In this section we will ﬁrst introduce the cascade forest structure, and then the multi-grained scanning, followed by the overall architecture and remarks on hyper-parameters.

Original: 2. Cascade Forest Structure
Revised: 2. Cascade Forest Structure

Original: Representation learning in deep neural networks mostly relies on the layer-by-layer processing of raw features.
Revised: Representation learning in deep neural networks mostly relies on the layer-by-layer processing of raw features.

Original: Inspired by this recognition, gcForest employs a cascade structure, as illustrated in Figure 1, where each level of cascade receives feature information processed by its preceding level, and outputs its processing result to the next level.
Revised: Inspired by this recognition, gcForest employs a cascade structure, as illustrated in Figure 1, where each level of cascade receives feature information processed by its preceding level, and outputs its processing result to the next level.

Original: Each level is an ensemble of decision tree forests, i.e., an ensemble of ensembles.
Revised: Each level is an ensemble of decision tree forests, i.e., an ensemble of ensembles.

Original: Here, we include different types of forests to encourage the diversity , as it is well known that diversity is crucial for ensemble construction [ Zhou, 2012].
Revised: Here, we include different types of forests to encourage the diversity , as it is well known that diversity is crucial for ensemble construction [ Zhou, 2012].

Original: For simplicity, suppose that we use two completely- random tree forests and two random forests [Breiman, 2001].
Revised: For simplicity, suppose that we use two completely- random tree forests and two random forests [Breiman, 2001].

Original: Each completely-random tree forest contains 500 completely- random trees [Liu et al. , 2008], generated by randomly select-
Revised: Each completely-random tree forest contains 500 completely- random trees [Liu et al. , 2008], generated by randomly select-

Original: ing a feature for split at each node of the tree, and growing tree until each leaf node contains only the same class of in- stances.
Revised: ing a feature for split at each node of the tree, and growing tree until each leaf node contains only the same class of in- stances.

Original: Similarly, each random forest contains 500 trees, by randomly selecting p d number of features as candidate ( d is the number of input features) and choosing the one with the best gini value for split.
Revised: Similarly, each random forest contains 500 trees, by randomly selecting p d number of features as candidate ( d is the number of input features) and choosing the one with the best gini value for split.

Original: The number of trees in each forest is a hyper-parameter, which will be discussed in Section 2.3.
Revised: The number of trees in each forest is a hyper-parameter, which will be discussed in Section 2.3.

Original: Given an instance, each forest will produce an estimate of class distribution, by counting the percentage of different classes of training examples at the leaf node where the concerned instance falls, and then averaging across all trees in the same forest, as illustrated in Figure 2, where red color highlights paths along which the instance traverses to leaf nodes.
Revised: Given an instance, each forest will produce an estimate of class distribution, by counting the percentage of different classes of training examples at the leaf node where the concerned instance falls, and then averaging across all trees in the same forest, as illustrated in Figure 2, where red color highlights paths along which the instance traverses to leaf nodes.

Original: The estimated class distribution forms a class vector, which is then concatenated with the original feature vector to be input to the next level of cascade.
Revised: The estimated class distribution forms a class vector, which is then concatenated with the original feature vector to be input to the next level of cascade.

Original: For example, suppose there are three classes, then each of the four forests will produce a three-dimensional class vector; thus, the next level of cascade will receive 12 (= 3 (cid:2) 4) augmented features.
Revised: For example, suppose there are three classes, then each of the four forests will produce a three-dimensional class vector; thus, the next level of cascade will receive 12 (= 3 (cid:2) 4) augmented features.

Original: To reduce the risk of overﬁtting, class vector produced by each forest is generated by k -fold cross validation.
Revised: To reduce the risk of overﬁtting, class vector produced by each forest is generated by k -fold cross validation.

Original: In detail, each instance will be used as training data for k (cid:0) 1 times, resulting in k (cid:0) 1 class vectors, which are then averaged to produce the ﬁnal class vector as augmented features for the next level of cascade.
Revised: In detail, each instance will be used as training data for k (cid:0) 1 times, resulting in k (cid:0) 1 class vectors, which are then averaged to produce the ﬁnal class vector as augmented features for the next level of cascade.

Original: After expanding a new level, the performance of the whole cascade will be estimated on validation set, and the training procedure will terminate if there is no signiﬁcant performance gain; thus, the number of cascade levels is automatically determined.
Revised: After expanding a new level, the performance of the whole cascade will be estimated on validation set, and the training procedure will terminate if there is no signiﬁcant performance gain; thus, the number of cascade levels is automatically determined.

Original: In contrast to most deep neural networks whose model complexity is ﬁxed, gcForest adaptively decides its model complexity by terminating training when adequate.
Revised: In contrast to most deep neural networks whose model complexity is ﬁxed, gcForest adaptively decides its model complexity by terminating training when adequate.

Original: This enables it to be applicable to different scales of training data, not limited to large-scale ones.
Revised: This enables it to be applicable to different scales of training data, not limited to large-scale ones.

Original: 2. Multi-Grained Scanning
Revised: 2. Multi-Grained Scanning

Original: Deep neural networks are powerful in handling feature relationships, e.g., convolutional neural networks are effective on image data where spatial relationships among the raw pixels are critical [LeCun et al. , 1998; Krizhenvsky et al. , 2012]; recurrent neural networks are effective on sequence data where sequential relationships are critical [Graves et al. , 2013;
Revised: Deep neural networks are powerful in handling feature relationships, e.g., convolutional neural networks are effective on image data where spatial relationships among the raw pixels are critical [LeCun et al. , 1998; Krizhenvsky et al. , 2012]; recurrent neural networks are effective on sequence data where sequential relationships are critical [Graves et al. , 2013;

Original: Cho et al. , 2014 ] .
Revised: Cho et al. , 2014 ] .

Original: Inspired by this recognition, we enhance cascade forest with a procedure of multi-grained scanning.
Revised: Inspired by this recognition, we enhance cascade forest with a procedure of multi-grained scanning.

Original: As Figure 3 illustrates, sliding windows are used to scan the raw features.
Revised: As Figure 3 illustrates, sliding windows are used to scan the raw features.

Original: Suppose there are 400 raw features and a window size of 100 features is used.
Revised: Suppose there are 400 raw features and a window size of 100 features is used.

Original: For sequence data, a 100-dimensional feature vector will be generated by sliding the window for one feature; in total 301 feature vectors are produced.
Revised: For sequence data, a 100-dimensional feature vector will be generated by sliding the window for one feature; in total 301 feature vectors are produced.

Original: If the raw features are with spacial relationships, such as a 20 (cid:2) 20 panel of 400 image pixels, then a 10 (cid:2) 10 window will produce 121 feature vectors (i.e., 121 10 (cid:2)
Revised: If the raw features are with spacial relationships, such as a 20 (cid:2) 20 panel of 400 image pixels, then a 10 (cid:2) 10 window will produce 121 feature vectors (i.e., 121 10 (cid:2)

Original: Proceedings of the Twenty-Sixth International Joint Conference on Artiﬁcial Intelligence (IJCAI-17)
Revised: Proceedings of the Twenty-Sixth International Joint Conference on Artiﬁcial Intelligence (IJCAI-17)

Original: [Figure]
Revised: [Figure]

Original: B Details of the translation of a decision tree into an MLP
Revised: B Details of the translation of a decision tree into an MLP

Original: Recall that a decision tree codes for a partition of the input space in as many parts as there are leaf nodes in the tree.
Revised: Recall that a decision tree codes for a partition of the input space in as many parts as there are leaf nodes in the tree.

Original: To know in which partition cell an input feature vector x ∈ R d falls into, we move in the tree from the root to the corresponding leaf using simple rules: at each m -th inner node, x is passed onto the left child node if its i m -th coordinate is less than or equal to some threshold t m , and to the right child node otherwise.
Revised: To know in which partition cell an input feature vector x ∈ R d falls into, we move in the tree from the root to the corresponding leaf using simple rules: at each m -th inner node, x is passed onto the left child node if its i m -th coordinate is less than or equal to some threshold t m , and to the right child node otherwise.

Original: The decision rule at each inner node of the tree introduces a split of the feature space into two subsets
Revised: The decision rule at each inner node of the tree introduces a split of the feature space into two subsets

Original: H − m = { x ∈ R d | x ( i m ) ≤ t m } and H + m = { x ∈ R d | x ( i m ) > t m } .
Revised: H − m = { x ∈ R d | x ( i m ) ≤ t m } and H + m = { x ∈ R d | x ( i m ) > t m } .

Original: Consistent with how the MLP translation works, we intentionally define
Revised: Consistent with how the MLP translation works, we intentionally define

Original: H − m and H
Revised: H − m and H

Original: + m such that at each inner node m , H − m ∪ H + m =
Revised: + m such that at each inner node m , H − m ∪ H + m =

Original: R d .
Revised: R d .

Original: Let N be the number of inner nodes of the decision tree; note that the decision tree has exactly
Revised: Let N be the number of inner nodes of the decision tree; note that the decision tree has exactly

Original: N + 1 leaf nodes, since it is by definition a complete binary tree, see Figure 1 for an illustration.
Revised: N + 1 leaf nodes, since it is by definition a complete binary tree, see Figure 1 for an illustration.

Original: For a leaf node ℓ ∈ { 1 , . . .
Revised: For a leaf node ℓ ∈ { 1 , . . .

Original: , N + 1 } of the tree, let
Revised: , N + 1 } of the tree, let

Original: P − ℓ ⊂ { 1 , . . . , N } (respectively P + ℓ ) be the set of all inner nodes whose left (respectivelyright
Revised: P − ℓ ⊂ { 1 , . . . , N } (respectively P + ℓ ) be the set of all inner nodes whose left (respectively

Original: ) subtree contains ℓ , that is, P + ℓ ∪ P − ℓ is the set of all parent nodes of ℓ .
Revised: right) subtree contains ℓ , that is, P + ℓ ∪ P − ℓ is the set of all parent nodes of ℓ .

Original: Then, the decision tree sorts an observation x ∈ R d into its leaf R ℓ if and only if
Revised: Then, the decision tree sorts an observation x ∈ R d into its leaf R ℓ if and only if

Original: [Equation]
Revised: [Equation]

Original: In fact, {R ℓ } ℓ ∈L is the feature space partition coded by the tree, see Figure 1 for an example.
Revised: In fact, {R ℓ } ℓ ∈L is the feature space partition coded by the tree, see Figure 1 for an example.

Original: Finally, the tree returns the average response of all training samples that fall into the same leaf as the input data; let us call a ℓ the average response of all training samples in R ℓ .
Revised: Finally, the tree returns the average response of all training samples that fall into the same leaf as the input data; let us call a ℓ the average response of all training samples in R ℓ .

Original: The final prediction of the decision tree g can therefore be expressed as
Revised: The final prediction of the decision tree g can therefore be expressed as

Original: [Equation]
Revised: [Equation]

Original: Let us now explore how an MLP can be designed to reproduce the prediction of a decision tree.
Revised: Let us now explore how an MLP can be designed to reproduce the prediction of a decision tree.

Original: Consider an MLP of depth 3 with N neurons on the first layer.
Revised: Consider an MLP of depth 3 with N neurons on the first layer.

Original: For each inner node m ∈ { 1 , . . . , N } , the m -th neuron of the first layer indicates on which side of the split introduced by this inner node a given feature vector lies: it equals − 1 if the feature vector lies in H − m and +1 if it lies in H + m .
Revised: For each inner node m ∈ { 1 , . . . , N } , the m -th neuron of the first layer indicates on which side of the split introduced by this inner node a given feature vector lies: it equals − 1 if the feature vector lies in H − m and +1 if it lies in H + m .

Original: This can be achieved applying the following affine transformation and a sign activation function to the feature vector,
Revised: This can be achieved applying the following affine transformation and a sign activation function to the feature vector,

Original: [Equation]
Revised: [Equation]

Original: The second layer of the 3-layer MLP has N +1 neurons.
Revised: The second layer of the 3-layer MLP has N +1 neurons.

Original: For each leaf node ℓ ∈ { 1 , . . .
Revised: For each leaf node ℓ ∈ { 1 , . . .

Original: , N +1 } , the ℓ -th neuron of the second layer indicates whether a given feature vector x ∈ R d lies in
Revised: , N +1 } , the ℓ -th neuron of the second layer indicates whether a given feature vector x ∈ R d lies in

Original: R ℓ or not: it equals +1 if x ∈ R ℓ and − 

Revised: R ℓ or not: it equals +1 if x ∈ R ℓ and −

Original: if x / ∈ R ℓ .
Revised: if x / ∈ R ℓ .

Original: Using equation (1), this can be achieved by applying the following affine transformation and a sign activation function to the output of the first layer,
Revised: Using equation (1), this can be achieved by applying the following affine transformation and a sign activation function to the output of the first layer,

Original: [Equation]
Revised: [Equation]

Original: The last layer of the MLP contains a single output neuron that returns the tree prediction.
Revised: The last layer of the MLP contains a single output neuron that returns the tree prediction.

Original: Using the output of the second layer, this can be achieved by applying the following affine transformation and an identity activation function,
Revised: Using the output of the second layer, this can be achieved by applying the following affine transformation and an identity activation function,

Original: [Equation]
Revised: [Equation]

Original: where a ℓ is the average response of all training samples in R ℓ .
Revised: where a ℓ is the average response of all training samples in R ℓ .

Original: Note that { a ℓ } N +1 ℓ =1 is a set of real numbers in regression problems and a set of probability vectors representing class distributions in classification problems.
Revised: Note that { a ℓ } N +1 ℓ =1 is a set of real numbers in regression problems and a set of probability vectors representing class distributions in classification problems.

Original: An illustration of the MLP translation of a decision tree is shown in Figure 1.
Revised: An illustration of the MLP translation of a decision tree is shown in Figure 1.

Original: This translation procedure is explained, for example, in Biau et al.
Revised: This translation procedure is explained, for example, in Biau et al.

Original: (2019) with more details.
Revised: (2019) with more details.

Original: C Illustration of our initialisation method
Revised: C Illustration of our initialisation method

Original: We provide below an illustration (Figure 7) showing how the whole MLP is initialised using both the tree-based method for the first layers and the random initialisation for the deeper layers.
Revised: We provide below an illustration (Figure 7) showing how the whole MLP is initialised using both the tree-based method for the first layers and the random initialisation for the deeper layers.

Original: D Detail on the MLP translation accuracy
Revised: D Detail on the MLP translation accuracy

Original: Recasting a Deep Forest into a deep MLP using our method may suffer from numerical instabilities altering the predictive behaviour.
Revised: Recasting a Deep Forest into a deep MLP using our method may suffer from numerical instabilities altering the predictive behaviour.

Original: This is due to a phenomenon of catastrophic cancellation, more likely to occur with deep MLP translations.
Revised: This is due to a phenomenon of catastrophic cancellation, more likely to occur with deep MLP translations.

Original: This is explained in the following section.
Revised: This is explained in the following section.

Original: X i
Revised: X i

Original: X i1 x ≥1 x ≥1 x ≥1 x ≥1 x ≥1 x ≥1 x ≥1 x ≥1 x ≥1 x ≥
Revised: X i1 x ≥1 x ≥1 x ≥1 x ≥1 x ≥1 x ≥1 x ≥1 x ≥1 x ≥1 x ≥

Original: Tree Tree
Revised: Tree Tree

Original: X i
Revised: X i

Original: X itanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh . . .
Revised: X itanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh tanh . . .

Original: [Figure]
Revised: [Figure]

Original: D.1 On the choice of hyper-parameters
Revised: D.1 On the choice of hyper-parameters

Original: In Section 2.3, four hyper-parameters were introduced to approximate the sign and identity functions through the layers of an elementary MLP.
Revised: In Section 2.3, four hyper-parameters were introduced to approximate the sign and identity functions through the layers of an elementary MLP.

Original: We address here the choice of the HPs and propose an optimal range for these parameters in the sense that they are as small as possible while guaranteeing a faithful MLP translation.
Revised: We address here the choice of the HPs and propose an optimal range for these parameters in the sense that they are as small as possible while guaranteeing a faithful MLP translation.

Original: We focus on the analysis of deep forest translation, as the structure of all other tree-based methods can be seen as a truncated variant of a deep forest.
Revised: We focus on the analysis of deep forest translation, as the structure of all other tree-based methods can be seen as a truncated variant of a deep forest.

Original: The deep forest is trained and translated into an MLP on each data set (see Section 2) for different values of the HPs.
Revised: The deep forest is trained and translated into an MLP on each data set (see Section 2) for different values of the HPs.

Original: To identify the influence of each HP, we make them vary in some range while the other three
Revised: To identify the influence of each HP, we make them vary in some range while the other three

Original: HPs are fixed to 10 10 , resulting in an almost perfect approximation of the respective sign and identity functions.
Revised: HPs are fixed to 10 10 , resulting in an almost perfect approximation of the respective sign and identity functions.

Original: Figure 8 shows the predictive performance of a deep forest and its
Revised: Figure 8 shows the predictive performance of a deep forest and its

Original: MLP translation playing with different HPs.
Revised: MLP translation playing with different HPs.

Original: Figure 8 shows in particular that 
Revised: Figure 8 shows in particular that (

Original: (i) increasing the HPs beyond some limit value is no longer beneficial as the activation functions are already perfectly approximated; (ii) across multiple data sets, these limit values are similar.
Revised: i) increasing the HPs beyond some limit value is no longer beneficial as the activation functions are already perfectly approximated; (ii) across multiple data sets, these limit values are similar.

Original: One could note that the coefficients in the first layer of a decision tree translation should be of a larger order of magnitude than those corresponding to the other activation functions to achieve an accurate translation.
Revised: One could note that the coefficients in the first layer of a decision tree translation should be of a larger order of magnitude than those corresponding to the other activation functions to achieve an accurate translation.

Original: To give some insight into why this is the case, recall that the m -th neuron of the first layer determines whether the input vector belongs to H − m or H + m , and note that its outputs can be of arbitrarily small size because the vector can be arbitrarily close to the decision boundaries.
Revised: To give some insight into why this is the case, recall that the m -th neuron of the first layer determines whether the input vector belongs to H − m or H + m , and note that its outputs can be of arbitrarily small size because the vector can be arbitrarily close to the decision boundaries.

Original: Note also that an MLP translation would better compromise on translation accuracy to ensure sufficient gradient flow.
Revised: Note also that an MLP translation would better compromise on translation accuracy to ensure sufficient gradient flow.

Original: Based on these observations, we remark that choosing the HP of the following orders allows for maximum gradient flow while still providing an accurate translation: strength01 ∈ [1 , 10 4 ] , strength12 ∈ [10 − 2 , 10 2 ] , strength23 ∈ [10 − 2 , 10 2 ] and strength id ∈ [10 − 2 , 10 2 ] .
Revised: Based on these observations, we remark that choosing the HP of the following orders allows for maximum gradient flow while still providing an accurate translation: strength01 ∈ [1 , 10 4 ] , strength12 ∈ [10 − 2 , 10 2 ] , strength23 ∈ [10 − 2 , 10 2 ] and strength id ∈ [10 − 2 , 10 2 ] .

Original: This will actually help us later on to calibrate the search spaces when empirically tuning these HPs for each data set.
Revised: This will actually help us later on to calibrate the search spaces when empirically tuning these HPs for each data set.

Original: M S E L o ss H
Revised: M S E L o ss H

Original: o u s i n g
Revised: o u s i n g

Original: MLPDeep Forest 0.5 0.
Revised: MLPDeep Forest 0.5 0.

Original: A u r o c S c
Revised: A u r o c S c

Original: o r e B
Revised: o r e B

Original: i n a r y
Revised: i n a r y

Original: A d u l t 0.5 0.
Revised: A d u l t 0.5 0.

Original: A u r o c S c
Revised: A u r o c S c

Original: o r e B
Revised: o r e B

Original: i n a r y
Revised: i n a r y

Original: B a n k 0.4 0.
Revised: B a n k 0.4 0.

Original: A cc u r a c y
Revised: A cc u r a c y

Original: C o v e r t y p e 10 0 10 1 10 2 10 3 10 4 strength01 0.{+
+}
Revised: C o v e r t y p e 10 0 10 1 10 2 10 3 {+
+}

Revised: [Figure]

Original: A cc u r a c y V
Revised: A cc u r a c y V

Original: o
Revised: o

Original: l k e r t10 1 10 0 10 1 10 2 10 3 strength 

Revised: l k e r t

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: D.2 A fundamental numerical instability of the neural network encoding
Revised: D.2 A fundamental numerical instability of the neural network encoding

Original: The encoding of a decision tree by a neural network proposed in Section 2.3 is numerically unstable, i.e., it does not necessarily yield the same result as the tree itself, even when using the original, non-approximated activation functions.
Revised: The encoding of a decision tree by a neural network proposed in Section 2.3 is numerically unstable, i.e., it does not necessarily yield the same result as the tree itself, even when using the original, non-approximated activation functions.

Original: This is the result of a catastrophic cancellation that occurs within the MLP translation.
Revised: This is the result of a catastrophic cancellation that occurs within the MLP translation.

Original: The term catastrophic cancellation describes the remarkable loss of precision that occurs when two nearly equal numbers are numerically subtracted.
Revised: The term catastrophic cancellation describes the remarkable loss of precision that occurs when two nearly equal numbers are numerically subtracted.

Original: For example, take the numbers a = 1 and b = 10 − 10 , and perform the computation ( a + b ) − a on a machine with limited precision, say to 8 significant digits.
Revised: For example, take the numbers a = 1 and b = 10 − 10 , and perform the computation ( a + b ) − a on a machine with limited precision, say to 8 significant digits.

Original: The machine will return ( a + b ) − a = 1 − 1 = 0 , although this result is clearly not correct.
Revised: The machine will return ( a + b ) − a = 1 − 1 = 0 , although this result is clearly not correct.

Original: This phenomenon occurs in the third layer of the MLP encoding, see equation (2).
Revised: This phenomenon occurs in the third layer of the MLP encoding, see equation (2).

Original: The two sums calculated in this layer are almost equal in magnitude but have opposite signs, resulting in a catastrophic cancellation that has a greater impact the more partitions of the input space the decision tree uses, i.e. the deeper it is.
Revised: The two sums calculated in this layer are almost equal in magnitude but have opposite signs, resulting in a catastrophic cancellation that has a greater impact the more partitions of the input space the decision tree uses, i.e. the deeper it is.

Original: Figure 9 illustrates the effect of this phenomenon, comparing the mean approximation error between a simple decision tree and its neural network encoding on the airbnb data set.
Revised: Figure 9 illustrates the effect of this phenomenon, comparing the mean approximation error between a simple decision tree and its neural network encoding on the airbnb data set.

Original: In
Revised: In

Original: Figure 9a, the result at the output layer of the tree was replaced by the exact training mean of the corresponding decision tree partition, compensating for the catastrophic cancellation.
Revised: Figure 9a, the result at the output layer of the tree was replaced by the exact training mean of the corresponding decision tree partition, compensating for the catastrophic cancellation.

Original: No such compensation was done for Figure 9b.
Revised: No such compensation was done for Figure 9b.

Original: This shows the grave implications of this instability: the mean error grows exponentially with the depth of an individual tree.
Revised: This shows the grave implications of this instability: the mean error grows exponentially with the depth of an individual tree.

Original: Although the errors introduced by this phenomenon may not be large for a given decision tree, they might accumulate when several such trees are composed, for example in Random or Deep Forests.
Revised: Although the errors introduced by this phenomenon may not be large for a given decision tree, they might accumulate when several such trees are composed, for example in Random or Deep Forests.

Original: Figure 10 compares the mean approximation error between Random/Deep
Revised: Figure 10 compares the mean approximation error between Random/Deep

Original: Forests of different complexities and their corresponding neural net encoding on the Airbnb data set.
Revised: Forests of different complexities and their corresponding neural net encoding on the Airbnb data set.

Original: It shows that the composition of several trees in a cascade manner, as performed by the Deep Forest, leads to a stronger amplification of their individual inaccuracies than the parallel composition of trees, as performed by the Random Forest.
Revised: It shows that the composition of several trees in a cascade manner, as performed by the Deep Forest, leads to a stronger amplification of their individual inaccuracies than the parallel composition of trees, as performed by the Random Forest.

Original: This result is to be expected because decision trees composed in parallel do not influence each other’s predictions, whereas in a cascade architecture the results of the first layer of decision trees affect the input of the subsequent layers and inaccuracies can thus develop stronger effects.
Revised: This result is to be expected because decision trees composed in parallel do not influence each other’s predictions, whereas in a cascade architecture the results of the first layer of decision trees affect the input of the subsequent layers and inaccuracies can thus develop stronger effects.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: We note that this catastrophic cancellation can be easily circumvented by introducing an additional layer.
Revised: We note that this catastrophic cancellation can be easily circumvented by introducing an additional layer.

Original: If this maps the output of the second layer from {− 1 , 1 } to { 0 , 1 } , the last layer could then simply multiply each of these outputs by the average response of a partition set.
Revised: If this maps the output of the second layer from {− 1 , 1 } to { 0 , 1 } , the last layer could then simply multiply each of these outputs by the average response of a partition set.

Original: However, Figure 10 also shows that the error introduced by the catastrophic cancellation remains relatively small, except for deep forests with many layers.
Revised: However, Figure 10 also shows that the error introduced by the catastrophic cancellation remains relatively small, except for deep forests with many layers.

Original: Therefore, we did not immediately address this issue and planned to fall back on this analysis if the MLP coding did not produce the expected results later in our analysis.
Revised: Therefore, we did not immediately address this issue and planned to fall back on this analysis if the MLP coding did not produce the expected results later in our analysis.

Original: However, this somewhat imprecise MLP coding worked well for all our purposes.
Revised: However, this somewhat imprecise MLP coding worked well for all our purposes.

Original: E Supplements to numerical evaluations E.1 Data sets
Revised: E Supplements to numerical evaluations E.1 Data sets

Original: Data sets description In the sequel, we run numerical experiments on 10 real-world, heterogeneous, tabular data sets, all but two of which have already been used to benchmark deep learning methods, see Borisov et al.
Revised: Data sets description In the sequel, we run numerical experiments on 10 real-world, heterogeneous, tabular data sets, all but two of which have already been used to benchmark deep learning methods, see Borisov et al.

Original: ; Somepalli et al.
Revised: ; Somepalli et al.

Original: The chosen data sets represent a variety of different learning tasks and sample sizes.
Revised: The chosen data sets represent a variety of different learning tasks and sample sizes.

Original: Tables 2 & 3 respectively give links to the platforms storing the data sets (four of them are available on the UCI
Revised: Tables 2 & 3 respectively give links to the platforms storing the data sets (four of them are available on the UCI

Original: Machine Learning Repository, Dua & Graff, 2017) and an overview of their main properties.
Revised: Machine Learning Repository, Dua & Graff, 2017) and an overview of their main properties.

Original: The Housing data set contains U.S. Census household attributes and the associated learning task is to predict the median house value for California districts (Kelley Pace & Barry, 1997).
Revised: The Housing data set contains U.S. Census household attributes and the associated learning task is to predict the median house value for California districts (Kelley Pace & Barry, 1997).

Original: The Airbnb data set is provided by the company itself and holds attributes on different
Revised: The Airbnb data set is provided by the company itself and holds attributes on different

Original: Airbnb listings in Berlin, such as the location of the apartment, the number of reviews, etc.
Revised: Airbnb listings in Berlin, such as the location of the apartment, the number of reviews, etc.

Original: The goal is to predict the price of each listing.
Revised: The goal is to predict the price of each listing.

Original: Similarly, the diamond data set contains characteristics of different diamonds (e.g., carat weight or cut quality), and the goal is to predict the price of a diamond.
Revised: Similarly, the diamond data set contains characteristics of different diamonds (e.g., carat weight or cut quality), and the goal is to predict the price of a diamond.

Original: The Adult data set contains Census information on adults (over 16-year olds) and its prediction task is to determine whether a person earns over
Revised: The Adult data set contains Census information on adults (over 16-year olds) and its prediction task is to determine whether a person earns over

Original: [Table] [Table]
Revised: [Table] [Table]

Original: $50k a year.
Revised: $50k a year.

Original: The Bank data set is related with direct marketing campaigns (phone calls) of a Portuguese banking institution, the classification goal is to predict whether the client will subscribe a term deposit.
Revised: The Bank data set is related with direct marketing campaigns (phone calls) of a Portuguese banking institution, the classification goal is to predict whether the client will subscribe a term deposit.

Original: The Blastchar data set features information on customers of a fictional company that provides phone and internet services.
Revised: The Blastchar data set features information on customers of a fictional company that provides phone and internet services.

Original: The classification goal is to predict whether a customer cancels their contract in the upcoming month.
Revised: The classification goal is to predict whether a customer cancels their contract in the upcoming month.

Original: The Heloc data set contains personal and credit record information on people that recently took on a line of credit, the classification task being to predict whether they will repay this credit withinyears.
Revised: The Heloc data set contains personal and credit record information on people that recently took on a line of credit, the classification task being to predict whether they will repay this credit withinyears.

Original: On the Higgs data set (Baldi et al., 2014), the classification problem is to distinguish between signal processes that produce Higgs bosons and background processes that do not.
Revised: On the Higgs data set (Baldi et al., 2014), the classification problem is to distinguish between signal processes that produce Higgs bosons and background processes that do not.

Original: For this purpose, it contains kinematic properties measured by the particle detectors in the accelerator that have been produced using Monte Carlo simulations.
Revised: For this purpose, it contains kinematic properties measured by the particle detectors in the accelerator that have been produced using Monte Carlo simulations.

Original: The Covertype data set contains cartographic variables on forest cells and it’s task is to predict the forest cover type.
Revised: The Covertype data set contains cartographic variables on forest cells and it’s task is to predict the forest cover type.

Original: Finally, for the Volkert data set, different patches of the same size have been cut from images that belong to 10 different landscape scenes (coast, forest, mountain, plain, etc.).
Revised: Finally, for the Volkert data set, different patches of the same size have been cut from images that belong to 10 different landscape scenes (coast, forest, mountain, plain, etc.).

Original: Each observation contains visual descriptors of one patch, the goal of this classification problem is to find the landscape type of the original picture.
Revised: Each observation contains visual descriptors of one patch, the goal of this classification problem is to find the landscape type of the original picture.

Original: E.2 Implementation details
Revised: E.2 Implementation details

Original: RFs are implemented using sklearn ’s
Revised: RFs are implemented using sklearn ’s

Original: RandomForestRegressor and
Revised: RandomForestRegressor and

Original: RandomForestClassifier classes with default configuration for all parameters that are not mentioned explicitly.
Revised: RandomForestClassifier classes with default configuration for all parameters that are not mentioned explicitly.

Original: DFs are implemented using the ForestLayer library (Zhou & Feng, 2017) and GBDTs are implemented using the XGBoost library (Chen &
Revised: DFs are implemented using the ForestLayer library (Zhou & Feng, 2017) and GBDTs are implemented using the XGBoost library (Chen &

Original: Guestrin, 2016).
Revised: Guestrin, 2016).

Original: MLPs are implemented and trained with pytorch , using the meansquared error and the cross entropy as objective function for regression and classification problems respectively.
Revised: MLPs are implemented and trained with pytorch , using the meansquared error and the cross entropy as objective function for regression and classification problems respectively.

Original: The SAINT model is implemented using the library provided by
Revised: The SAINT model is implemented using the library provided by

Original: Somepalli et al.
Revised: Somepalli et al.

Original: All methods are trained on a 32 GB RAM machine using 12 Intel Core i7-8700K CPUs, and one NVIDIA GeForce RTX 2080 GPU when possible (only the GDBT and MLP implementations including SAINT use the GPU).
Revised: All methods are trained on a 32 GB RAM machine using 12 Intel Core i7-8700K CPUs, and one NVIDIA GeForce RTX 2080 GPU when possible (only the GDBT and MLP implementations including SAINT use the GPU).

Original: Hyper-parameter searches are parallelized on up to 4 of these machines.
Revised: Hyper-parameter searches are parallelized on up to 4 of these machines.

Original: Hyper-parameter optimization We tune all hyper-parameters using the optuna library (Akiba et al., 2019) with a fixed number of iterations for all models.
Revised: Hyper-parameter optimization We tune all hyper-parameters using the optuna library (Akiba et al., 2019) with a fixed number of iterations for all models.

Original: In this context, an iteration corresponds to a set of hyper-parameters whose performance is evaluated with
Revised: In this context, an iteration corresponds to a set of hyper-parameters whose performance is evaluated with

Original: [Figure]
Revised: [Figure]

Original: respect to a given method.
Revised: respect to a given method.

Original: The optuna library uses Bayesian optimization and, in particular, the tree-structured Parzen estimator model (Bergstra et al., 2011) to determine the parameters to be explored at each iteration of hyper-parameter optimization.
Revised: The optuna library uses Bayesian optimization and, in particular, the tree-structured Parzen estimator model (Bergstra et al., 2011) to determine the parameters to be explored at each iteration of hyper-parameter optimization.

Original: This approach has been reported to outperform random search for hyper-parameter optimization (Turner et al., 2021).
Revised: This approach has been reported to outperform random search for hyper-parameter optimization (Turner et al., 2021).

Original: Data pre-processing Machine learning pipelines often include pre-processing transformations of the input data before the training phase, a necessary step, especially when using neural networks (Bishop et al., 1995).
Revised: Data pre-processing Machine learning pipelines often include pre-processing transformations of the input data before the training phase, a necessary step, especially when using neural networks (Bishop et al., 1995).

Original: We follow the pre-processing that is used in Borisov et al.
Revised: We follow the pre-processing that is used in Borisov et al.

Original: (2021) and Somepalli et al.
Revised: (2021) and Somepalli et al.

Original: Hence, we normalize all continuous input features to zero mean and unit variance.
Revised: Hence, we normalize all continuous input features to zero mean and unit variance.

Original: This corresponds to linearly transform the input features as follows
Revised: This corresponds to linearly transform the input features as follows

Original: [Equation]
Revised: [Equation]

Original: where x : j is the j -th continuous feature of either train, validation or test observations, µ and σ are the mean and standard deviation calculated over the train set only.
Revised: where x : j is the j -th continuous feature of either train, validation or test observations, µ and σ are the mean and standard deviation calculated over the train set only.

Original: This way we assure that no information from the validation or test sets is used in the normalization step.
Revised: This way we assure that no information from the validation or test sets is used in the normalization step.

Original: Moreover, all categorical features are label encoded, i.e. each level of a categorical variable is replaced with an integer in { 1 , . . .
Revised: Moreover, all categorical features are label encoded, i.e. each level of a categorical variable is replaced with an integer in { 1 , . . .

Original: , # levels } .
Revised: , # levels } .

Original: E.3 Working with an arbitrary width in P1 (optimization behaviour)
Revised: E.3 Working with an arbitrary width in P1 (optimization behaviour)

Original: Figure 11 shows the optimization behaviour of the randomly, RF and GBDT initialized MLP on the multi-class classification problems.
Revised: Figure 11 shows the optimization behaviour of the randomly, RF and GBDT initialized MLP on the multi-class classification problems.

Original: Note that in contrast to Figure 2 in this setting, which is less restrictive for RF initialization, this method does indeed lead to a faster convergence and a better minimum (in terms of generalization).
Revised: Note that in contrast to Figure 2 in this setting, which is less restrictive for RF initialization, this method does indeed lead to a faster convergence and a better minimum (in terms of generalization).

Original: However, for these multi-class classification problems, the GBDT initialization tends to deteriorate the optimization compared to RF or random initialization methods.
Revised: However, for these multi-class classification problems, the GBDT initialization tends to deteriorate the optimization compared to RF or random initialization methods.

Original: Indeed, RF are genuinely multiclassification predictors whose splits are built using all output classes simultaneously whereas splits in GBDT are only built following a one-vs-all strategy.
Revised: Indeed, RF are genuinely multiclassification predictors whose splits are built using all output classes simultaneously whereas splits in GBDT are only built following a one-vs-all strategy.

Original: This implies that, with a fixed budget of splits (and therefore of neurons), RF are likely to be more versatile than GBDT.
Revised: This implies that, with a fixed budget of splits (and therefore of neurons), RF are likely to be more versatile than GBDT.

Original: E.4 Additional material for Protocol P2 (generalization behaviour) E.4.1 Details about additional NN training techniques
Revised: E.4 Additional material for Protocol P2 (generalization behaviour) E.4.1 Details about additional NN training techniques

Original: In Protocol 2, we assess the performances of generalization of the proposed methods, of the predictors described in Section 3.2, but also on three additional NN techniques:
Revised: In Protocol 2, we assess the performances of generalization of the proposed methods, of the predictors described in Section 3.2, but also on three additional NN techniques:

Original: the Xavier initialization (Glorot & Bengio, 2010) corresponds to a rescaled uniform initialization U ∼ U (cid:20) ± √√ n j +1 + n j (cid:21) , where n j are the number of neurons in layer j .
Revised: the Xavier initialization (Glorot & Bengio, 2010) corresponds to a rescaled uniform initialization U ∼ U (cid:20) ± √√ n j +1 + n j (cid:21) , where n j are the number of neurons in layer j .

Original: This random initialization is very close to the one used by default in this paper and simply denoted “random init”; 2.
Revised: This random initialization is very close to the one used by default in this paper and simply denoted “random init”; 2.

Original: the layer-sequential unit-variance orthogonal initialization (LSUV) (Mishkin & Matas, 2015) consists in a simple initialization that combines elements of (Glorot & Bengio, 2010) and (Saxe et al., 2013).
Revised: the layer-sequential unit-variance orthogonal initialization (LSUV) (Mishkin & Matas, 2015) consists in a simple initialization that combines elements of (Glorot & Bengio, 2010) and (Saxe et al., 2013).

Original: In a first step, the weights of each layer are initialized as random orthogonal matrices.
Revised: In a first step, the weights of each layer are initialized as random orthogonal matrices.

Original: Then, the variance in the outputs on each layer on the training data is scaled close to 1 by repeatedly dividing the layer’s weights by the empirically determined standard deviation.
Revised: Then, the variance in the outputs on each layer on the training data is scaled close to 1 by repeatedly dividing the layer’s weights by the empirically determined standard deviation.

Original: Although targeted to Computer Vision applications, this approach seems easily adaptable for our case; 3.
Revised: Although targeted to Computer Vision applications, this approach seems easily adaptable for our case; 3.

Original: the winning ticket network pruning (Frankle & Carbin, 2018) is more a simplification approach of the NN architecture during training than an initialization technique.
Revised: the winning ticket network pruning (Frankle & Carbin, 2018) is more a simplification approach of the NN architecture during training than an initialization technique.

Original: That being said, it remains interesting to compare this strategy to the one developed in the paper, as the winning ticket network pruning enforces NN sparsity during training.
Revised: That being said, it remains interesting to compare this strategy to the one developed in the paper, as the winning ticket network pruning enforces NN sparsity during training.

Original: This can be indeed put in parallel to the sparsity of the first layers introduced by the proposed initialization and preserved during training.The principle is to
Revised: This can be indeed put in parallel to the sparsity of the first layers introduced by the proposed initialization and preserved during training.

Original:  train a randomly initialized network, pruning 
Revised: The principle is to train a randomly initialized network, pruning it to obtain a sparse NN

Original: it to
Revised: with

Original: obtain a sparse NN with similar performance and then re-train the sparse network a second time using the same instance of random initialization 
Revised: similar performance and then re train the sparse network a second time using the same instance of random initialization as before.

Original: as

Original: before

Original: These

Original:  steps are repeated a certain number of times.
Revised: These steps are repeated a certain number of times.

Original: The winning ticket network pruning is therefore compu - tationally very intense and has 
Revised: The winning ticket network pruning is therefore computationally very intense and has to the best of our knowledge only been studied on medium-sized data sets.

Original: to the best of our knowledge

Original: only

Original: been

Original: studied

Original: on

Original: medium

Original: -sized data sets.

Original: We thus use a slightly different procedure than (Frankle & Carbin, 2018) to determine winning tickets.
Revised: We thus use a slightly different procedure than (Frankle & Carbin, 2018) to determine winning tickets.

Original: First of all, we allocate at most N training epochs to determining a winning ticket where N is the number of epochs during the final model training itself
Revised: First of all, we allocate at most N training epochs to determining a winning ticket where N is the number of epochs during the final model training itself.

Original: This fixed number of training epochs is then distributed among n pruning rounds, each of which consists in training the model (for N/n epochs), pruning it, and resetting all non-pruned weights to their initial (random) coefficients.
Revised: This fixed number of training epochs is then distributed among n pruning rounds, each of which consists in training the model (for N/n epochs), pruning it, and resetting all non-pruned weights to their initial (random) coefficients.

Original: This approach takes the same time as one-shot pruning but proves to be more efficient.
Revised: This approach takes the same time as one-shot pruning but proves to be more efficient.

Original: E.4.2 Extension of Table 1 (best performances)
Revised: E.4.2 Extension of Table 1 (best performances)

Original: Table 4 provides a comparison of the performances obtained by ourselves and the literature (where available) for each model.
Revised: Table 4 provides a comparison of the performances obtained by ourselves and the literature (where available) for each model.

Original: Notice that our results are broadly consistent with those in the literature, with two exceptions.
Revised: Notice that our results are broadly consistent with those in the literature, with two exceptions.

Original: First, our random initialized MLP tends to perform better than in the literature, which can be explained by the fact that we use a much larger search space than usual for the MLP width (see Section 3.5 for a discussion on this).
Revised: First, our random initialized MLP tends to perform better than in the literature, which can be explained by the fact that we use a much larger search space than usual for the MLP width (see Section 3.5 for a discussion on this).

Original: Second, our performance on Higgs is significantly lower than in the literature.
Revised: Second, our performance on Higgs is significantly lower than in the literature.

Original: This can be explained by the fact that we only include 5% of the original data set’s observations in our analysis due to hardware limitations that do not allow us to train large MLP on 11M samples.
Revised: This can be explained by the fact that we only include 5% of the original data set’s observations in our analysis due to hardware limitations that do not allow us to train large MLP on 11M samples.

Original: E.4.3 Benefits of training the feature extractor via gradient descent
Revised: E.4.3 Benefits of training the feature extractor via gradient descent

Original: In Section 3, we demonstrated ways in which our initialization method can be beneficial for
Revised: In Section 3, we demonstrated ways in which our initialization method can be beneficial for

Original: MLP training, resulting in faster convergence towards better minima (in the sens of generalization).
Revised: MLP training, resulting in faster convergence towards better minima (in the sens of generalization).

Original: A natural question that might arise in this context is whether translating the tree-based method into a MLP framework is actually beneficial.
Revised: A natural question that might arise in this context is whether translating the tree-based method into a MLP framework is actually beneficial.

Original: After all, one could be tempted to directly use the tree-based method as a feature pre-processing (without translating it into an MLP) and feed the resulting features into an MLP.
Revised: After all, one could be tempted to directly use the tree-based method as a feature pre-processing (without translating it into an MLP) and feed the resulting features into an MLP.

Original: In this case, the MLP would be trained via gradient descent without the feature extraction.
Revised: In this case, the MLP would be trained via gradient descent without the feature extraction.

Original: However, it turns out that (i) the weights on the sparse feature extraction layer are indeed modified during gradient descent optimization and (ii) training the feature extractor via gradient descent largely contributes to the competitive generalization performance of our initialization method.
Revised: However, it turns out that (i) the weights on the sparse feature extraction layer are indeed modified during gradient descent optimization and (ii) training the feature extractor via gradient descent largely contributes to the competitive generalization performance of our initialization method.

Original: Figure 12 shows the histograms of the differences between all MLP parameters at initialization (RF strategy) and after training.
Revised: Figure 12 shows the histograms of the differences between all MLP parameters at initialization (RF strategy) and after training.

Original: As the histograms indicate, the weights in all layer
Revised: As the histograms indicate, the weights in all layer

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: throughout the  are  training.
Revised: throughout the MLP are modified during training.

Original: In particular, the  first  are  to 
Revised: In particular, the weights of the first two (RF initialized) layers are not stationary but change to a large extent.

Original: Table 5 shows the generalization performance of MLP initialized with the RF strategy, and compares the two scenarios in which the parameters of the first two layers (that is, the feature extraction layers built using the RF) are modified or frozen during MLP training.
Revised: Table 5 shows the generalization performance of MLP initialized with the RF strategy, and compares the two scenarios in which the parameters of the first two layers (that is, the feature extraction layers built using the RF) are modified or frozen during MLP training.

Original: These results show that training the feature extraction layers is essential for the success of our initialization method.
Revised: These results show that training the feature extraction layers is essential for the success of our initialization method.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: E.4.4 Number of parameters of best neural networks
Revised: E.4.4 Number of parameters of best neural networks

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we compare the number of parameters of each NN method.
Revised: we compare the number of parameters of each NN method.

Original: Although the treebased initialised MLP contain more parameters than the randomly initialized ones, the former are mostly sparse and the execution times are close (see Table 7).
Revised: Although the treebased initialised MLP contain more parameters than the randomly initialized ones, the former are mostly sparse and the execution times are close (see Table 7).

Original: Finally note that the number of parameters of the RF/GBDT init.
Revised: Finally note that the number of parameters of the RF/GBDT init.

Original: MLP is globally on par with that of SAINT (sometimes more, sometimes less) but for a smaller execution times (Table 7) and mostly better performances (Table 4).
Revised: MLP is globally on par with that of SAINT (sometimes more, sometimes less) but for a smaller execution times (Table 7) and mostly better performances (Table 4).

Original: E.4.5 Comparison of the execution times of the best neural networks
Revised: E.4.5 Comparison of the execution times of the best neural networks

Original: Table 7 presents a comparison of the execution times of the training of different NN methods using the hyper-parameters determined by the protocol P2.
Revised: Table 7 presents a comparison of the execution times of the training of different NN methods using the hyper-parameters determined by the protocol P2.

Original: For each model, the total training time (initialization + gradient descent optimization) is given, measured up to the point where the best validation loss is reached (“early stopping”).
Revised: For each model, the total training time (initialization + gradient descent optimization) is given, measured up to the point where the best validation loss is reached (“early stopping”).

Original: It shows that RF/GBDT initialized MLP train faster than SAINT and a bit slower than randomly initialized MLP.
Revised: It shows that RF/GBDT initialized MLP train faster than SAINT and a bit slower than randomly initialized MLP.

Original: For completeness, Table 8 gives the execution time for the initialization and training step separately.
Revised: For completeness, Table 8 gives the execution time for the initialization and training step separately.

Original: E.4.6 Optimization behaviour
Revised: E.4.6 Optimization behaviour

Original: For completeness, Figure 13 shows the optimization behaviour of the randomly, RF and
Revised: For completeness, Figure 13 shows the optimization behaviour of the randomly, RF and

Original: GBDT initialized MLP as well as SAINT under the Protocol P2.
Revised: GBDT initialized MLP as well as SAINT under the Protocol P2.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Housing
Revised: Housing

Original: [Table]
Revised: Airbnb Diamonds

Original: Adult
Revised: Adult

Original: [Figure]
Revised: [Figure]

Original: Bank
Revised: Bank

Original: Blastchar
Revised: Blastchar

Original: Heloc
Revised: Heloc

Original: Higgs*
Revised: Higgs

Original: [Figure]
Revised: [Figure]

Revised: t r o

Original: [Figure]
Revised: [Figure]

Revised: y

Original: [Figure]
Revised: [Figure]

Revised: o

Revised: [Figure]

Revised: [Figure]

Original: E.5 Hyper-parameter setting E.5.1 Search spaces
Revised: E.5 Hyper-parameter setting E.5.1 Search spaces

Original: Table 9 shows the HP search spaces that were used to determine an optimal HP setting.
Revised: Table 9 shows the HP search spaces that were used to determine an optimal HP setting.

Original: The same search spaces were used for the experimental protocols P1 and P2.
Revised: The same search spaces were used for the experimental protocols P1 and P2.

Original: Note that, in
Revised: Note that, in

Original: Table 9, n classes corresponds to the number of classes for classification problems and is 1 for regression problems.
Revised: Table 9, n classes corresponds to the number of classes for classification problems and is 1 for regression problems.

Original: Furthermore, the different search spaces given for SAINT were used for smaller/larger data sets, where a data set qualifies as smaller if it has less thatexplanatory variables.
Revised: Furthermore, the different search spaces given for SAINT were used for smaller/larger data sets, where a data set qualifies as smaller if it has less thatexplanatory variables.

Original: E.5.2 Experimental protocol P 
Revised: E.5.2 Experimental protocol P

Original: Tables 10 and 11 show the HP setting used for the experimental protocol P2.
Revised: Tables 10 and 11 show the HP setting used for the experimental protocol P2.

Original: For the search spaces and descriptions of the function of each HP see Table 9.
Revised: For the search spaces and descriptions of the function of each HP see Table 9.

Original: E.6 Performances of tree-based methods used for initialisation of MLP
Revised: E.6 Performances of tree-based methods used for initialisation of MLP

Original: Figure 14 compares the performance of RF and GBDT models and the performance of optimized MLP, initialized with RF and GBDT respectively.
Revised: Figure 14 compares the performance of RF and GBDT models and the performance of optimized MLP, initialized with RF and GBDT respectively.

Original: We can notice that the difference in performance between GBDT and RF does not systematically turn into the same difference in performance for the corresponding trained networks.
Revised: We can notice that the difference in performance between GBDT and RF does not systematically turn into the same difference in performance for the corresponding trained networks.

Original: This suggests that beyond their respective performances, the very structures of RF and GBDT predictors play an important role in the final MLP performances.
Revised: This suggests that beyond their respective performances, the very structures of RF and GBDT predictors play an important role in the final MLP performances.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: MLP DF

Original: [Table]
Revised: [Table]

Revised: init

Original: [Table]
Revised: [Table]

Original: experimental protocol P2.
Revised: protocol P2.

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Adult
Revised: Adult

Original: Bank
Revised: Bank

Original: [Figure]
Revised: [Figure]

Original: Blastchar
Revised: Blastchar

Original: [Figure]
Revised: [Figure]

Original: E.7 Additional Figures to Section 3.5 (Analyzing key elements of the new initialization methods)
Revised: E.7 Additional Figures to Section 3.5 (Analyzing key elements of the new initialization methods)

Original: Figures 15 and 16 show the same histograms as Figure 4 evaluated on the other data set considered in protocol P2.
Revised: Figures 15 and 16 show the same histograms as Figure 4 evaluated on the other data set considered in protocol P2.

Original: Note the logarithmic y-axis for the first two RF and GBDT initialized layers.
Revised: Note the logarithmic y-axis for the first two RF and GBDT initialized layers.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Covertype
Revised: Covertype

Original: Volkert
Revised: Volkert

Original: [Figure]
Revised: [Figure]

Revised: [Figure]