A N EURAL T ANGENT K ERNEL P ERSPECTIVE OF I NFINITE T REE E NSEMBLES
Anonymous authors
Paper under double-blind review
A BSTRACT
In practical situations, the tree ensemble is one of the most popular models along with neural networks.
A soft tree is a variant of a decision tree.
Instead of using a greedy method for searching splitting rules, the soft tree is trained using a gradient method in which the whole splitting operation is formulated in a differentiable form.
Although ensembles of such soft trees have been increasingly used in recent years, little theoretical work has been done for understanding their behavior.
In this paper, by considering an ensemble of inﬁnite soft trees, we introduce and study the Tree Neural Tangent Kernel (TNTK), which provides new insights into the behavior of the inﬁnite ensemble of soft trees.
Using the TNTK, we succeed in theoretically ﬁnding several non-trivial properties, such as global convergence of the training, the equivalence of the oblivious tree structure, and the degeneracy of the TNTK induced by the deepening of the trees.
I NTRODUCTION
Tree ensembles and neural networks are powerful machine learning models used in various realworld applications.
A soft tree ensemble is one of the variants of tree ensemble models that inherits characteristics of neural networks.
Instead of using a greedy method (Quinlan, 1986; Breiman et al., 1984) for searching splitting rules, the soft tree makes the splitting rules soft and updates the entire model
’s parameters simultaneously using the gradient method.
Soft tree ensemble models are known to have high empirical performance (Kontschieder et al., 2015; Popov et al., 2020; Hazimeh et al., 2020), especially for tabular datasets.
Besides accuracy, there are many additional reasons why one should formulate trees in a soft manner.
For example, unlike hard decision trees, the model can be updated sequentially (Ke et al., 2019) and trained in combination with pre-training (Arik & Pﬁster, 2019), resulting in favorable characteristics in terms of real-world continuous service deployment.
Their model interpretability induced by the hierarchical splitting structure has also attracted much attention (Frosst & Hinton, 2017; Wan et al., 2021; Tanno et al., 2019).
In addition, the idea of the soft tree is implicitly used in many different places; for example, the process of allocating data to the appropriate leaves can be interpreted as a special case of Mixture-of-Experts (Jordan & Jacobs, 1993;
Shazeer et al., 2017;
Lepikhin et al., 2021), a technique for balancing computational complexity and prediction performance.
Although various techniques have been proposed to train trees, the theoretical validity of such techniques is not well understood at sufﬁcient depth.
Examples of the practical technique include constraints on individual trees using parameter sharing (Popov et al., 2020), adjusting the hardness of the splitting operation (Frosst & Hinton, 2017; Hazimeh et al., 2020), and the use of overparameterization (Belkin et al., 2019; Karthikeyan et al., 2021).
To better understand the training of tree ensemble models, we focus on the Neural Tangent Kernel (NTK)
(Jacot et al., 2018), a powerful tool that has been successfully applied to various neural network models with inﬁnite hidden layer nodes.
Every model architecture is known to produce a distinct NTK.
Not only for the multi-layer perceptron (MLP), many studies have been performed across various models, such as for Convolutional Neural
Networks (CNTK) (Arora et al., 2019; Li et al., 2019), Graph Neural Networks (GNTK) (Du et al., 2019b), and Recurrent Neural Networks (RNTK) (Alemohammad et al., 2021).
The NTK theory is often used in the context of overparameterization of neural networks.
In response to recent trends, overparameterization is also a subject of interest for tree ensembles (Belkin et al., 2019; Karthikeyan et al., 2021).
Although a number of ﬁndings have been obtained using the NTK, they are mainly for typical neural networks, and it is still not obvious how to apply the NTK theory to the tree models.
In this paper by considering the limit of inﬁnitely many trees we introduce and study the neural tangent kernel for tree ensembles, called the Tree Neural Tangent Kernel (TNTK), which provides new insights into the behavior of the ensemble of soft trees.
The goal of this research is to derive the kernel that characterizes the training behavior of soft tree ensembles, and to obtain theoretical support for the empirical techniques.
Our contributions are summarized as follows:
• First extension of the NTK concept to the tree ensemble models.
We derive the analytical form for the TNTK at initialization induced by inﬁnitely many complete binary trees with arbitrary depth.
(Section 4.1.1).
We also prove that the TNTK remains constant during the training of inﬁnite soft trees.
This property allows us to analyze the behavior by kernel regression and discuss global convergence of training using the positive deﬁniteness of the TNTK (Section 4.1.2, 4.1.3).
• Equivalence of the oblivious tree ensemble models.
We show the TNTK induced by the oblivious tree structure used in practical open-source libraries such as CatBoost (Prokhorenkova et al., 2018) and NODE (Popov et al., 2020) converges to the same TNTK induced by a non-oblivious one in the limit of inﬁnite trees.
This observation implicitly supports the good empirical performance of oblivious trees with parameter sharing between tree nodes (Section 4.2.1).
• Nonlinearity by adjusting the tree splitting operation.
Practically, various functions have been proposed to represent the tree splitting operation.
The most basic function is sigmoid .
We show that the TNTK is almost a linear kernel in the basic case, and when we adjust the splitting function hard, the TNTK becomes nonlinear (Section 4.2.2).
• Degeneracy of the TNTK with deep trees.
The TNTK associated with deep trees exhibits degeneracy: the TNTK values are almost identical for deep trees even if the inner products of inputs are different.
As a result, poor performance in numerical experiments is observed with the TNTK induced by inﬁnitely many deep trees.
This result supports the fact that the depth of trees is usually not so large in practical situations (Section 4.2.3).
• Comparison to the NTK induced by the MLP.
We investigate the generalization performance of inﬁnite tree ensembles by kernel regression with the TNTK on 90 real-world datasets.
Although the MLP with inﬁnite width has better prediction accuracy on average, the inﬁnite tree ensemble performs better than the inﬁnite width MLP in more than 30% of the datasets.
We also showed that the TNTK is superior to the MLP-induced NTK in computational speed (Section 5).
B ACKGROUND AND RELATED WORK
Our main focus in this paper is the soft tree and the neural tangent kernel.
We brieﬂy introduce and review them.
2. S OFT TREE
Based on Kontschieder et al.
, we formulate a regression by soft trees.
Figure 1 is a schematic image of an ensemble of M soft trees.
We deﬁne a data matrix x ∈ R N 0 × N for N training samples { x 1 , . . .
, x N } with N 0 features and deﬁne tree-wise parameter matrices for internal nodes w m ∈ R N 0 ×N and leaf nodes π m ∈ R 1 ×L for each tree m ∈
[ M ] = { 1 , . .
, M } as x = (cid:32) | . . .
| x. . .
x N | . . .
| (cid:33) , w m = (cid:32) | . . .
| w m,. . .
w m, N | . . .
| (cid:33) , π m =
( π m, 1 , . . .
, π m, L ) , where internal nodes (blue nodes in Figure 1) and leaf nodes (green nodes in Figure 1) are indexed from 1 to N and 1 to L , respectively.
N and L may change across trees in general, while we assume that they are always ﬁxed for simplicity throughout the paper.
We also write horizontal concatenation of (column) vectors as x = ( x 1 , . . .
, x N ) ∈ R N 0 × N and w m = ( w m, 1 , . . .
, w m, N ) ∈ R N 0 ×N .
Unlike hard decision trees, we consider a model in which every single leaf node (cid:96)
∈
[ L ] = { 1 , . . .
, L} of a tree m holds the probability that data will reach to it.
Therefore, splitting operation at an intermediate node n ∈ [ N ] = { 1 , . . .
, N} does not deﬁnitively decide splitting to the left or right.
To provide an explicit form of the probabilistic tree splitting operation, we introduce the following binary relations that depend on the tree’s structure: (cid:96) (cid:46) n (resp.
n (cid:38) (cid:96) ), which is true if a leaf (cid:96) belongs to the left (resp.
right) subtree of a node n and false otherwise.
We can now exploit µ m,(cid:96) ( x i , w m ) :
R N 0 × R N 0 ×N → [0 , 1] , a function that returns the probability that a sample x i will reach a leaf (cid:96) of the tree m , as follows:
[Equation]
where 1 Q is an indicator function conditioned on the argument Q , i.e., 1 true = 1 and 1 false = 0 , and g m,n : R N 0 × R N
→ [0 , 1] is a decision function at each internal node n of a tree m .
To approximate decision tree splitting, the output of the decision function g m,n should be between 0 . 0 and 1 . 0 .
If the output of a decision function takes only 0 . 0 or 1 . 0 , the splitting operation is equivalent to hard splitting used in typical decision trees.
We will deﬁne an explicit form of the decision function g m,n in Equation (5) in the next section.
The prediction for each x i from a tree m with nodes parameterized by w m and π m is given by
[Equation]
where f m : R N 0 × R N 0 ×N ×
R 1 ×L → R , and π m,(cid:96) denotes the response of a leaf (cid:96) of the tree m .
This formulation means that the prediction output is the average of the leaf values π m,(cid:96) weighted by µ m,(cid:96)
( x i , w m ) , probability of assigning the sample x i to the leaf (cid:96) .
If µ m,(cid:96) ( x i , w m ) takes only 1 .for one leaf and 0 .
for the other leaves, the behavior is equivalent to a typical decision tree prediction.
In this model, w m and π m are updated during training with a gradient method.
While many empirical successes have been reported, theoretical analysis for soft tree ensemble models has not been sufﬁciently developed.
2. N EURAL TANGENT KERNEL
Given N samples x ∈ R N 0 × N , the NTK induced by any model architecture at a training time τ is formulated as a matrix (cid:99) H ∗ τ ∈ R N × N , in which each ( i, j ) ∈ [ N ] × [ N ] component is deﬁned as
[Equation]
where (cid:104)· , ·(cid:105) denotes the inner product and θ τ ∈ R P is a concatenated vector of all the P trainable model parameters at τ .
An asterisk “ ∗ ” indicates that the model is arbitrary.
The model function f arbitrary : R N 0 × R P
→ R used in Equation (3) is expected to be applicable to a variety of model structures.
For the soft tree ensembles introduced in Section 2.1, the NTK is formulated as
[Equation]
In the limit of inﬁnite width with a proper parameter scaling, a variety of properties have been discovered from the NTK induced by the MLP.
For example, Jacot et al.
(2018) showed the convergence of (cid:98) Θ MLP( x i , x j ) , which can vary with respect to parameters, to the unique limiting kernel Θ( x i , x j ) at initialization in probability.
Moreover, they also showed that the limiting kernel does not change during training in probability:
[Equation]
This property helps in the analytical understanding of the model behavior.
For example, with the squared loss and inﬁnitesimal step size with learning rate η , the training dynamics of gradient ﬂow in function space coincides with kernel ridge-less regression with the limiting NTK.
Such a property gives us a data-dependent generalization bound (Bartlett & Mendelson, 2003) related to the NTK and the prediction targets.
In addition, if the NTK is positive deﬁnite, the training can achieve global convergence (Du et al., 2019a; Jacot et al., 2018).
Although a number of ﬁndings have been obtained using the NTK, they are mainly for typical neural networks such as MLP and ResNet, and the NTK theory has not been applied to tree models yet.
The
NTK theory is often used in the context of overparameterization, a subject of interest not only for the neural networks but also for the tree models (Belkin et al., 2019; Karthikeyan et al., 2021; Tang et al., 2018).
S ETUP
[Figure]
A magenta We train model parameters w and π to minimize the squared loss using the gradient method, where w = ( w 1 , . . . , w m ) and
π = ( π 1 , . . . , π m ) .
The tree structure is ﬁxed during training.
In order to use a known closed-form solution of the
NTK (Williams, 1996; Lee et al., 2019), we use a scaled error function σ : R → (0 , 1) , resulting in the following decision function:
[Equation]
where erf( p ) = 2 √ π (cid:82) p
e − t 2 d t for p ∈ R .
This scaled error function approximates commonly used sigmoid function.
Since the bias term for the input of σ can be expressed inside of w by adding an element that takes a ﬁxed constant value for all input of the soft trees x , we do not consider the bias for simplicity.
The scaling factor α is introduced by Frosst &
Hinton (2017) to avoid too soft splitting.
Figure 2 shows that the decision function becomes harder as α increases (from blue to red), and in the limit α → ∞ it coincides with the hard splitting used in typical decision trees.
When aggregating the output of multiple trees, we divide the sum of the tree outputs by the square root of the number of trees
[Equation]
This 1 / √ M scaling is known to be essential in the existing NTK literature to use the weak law of the large numbers (Jacot et al., 2018).
On top of Equation (6), we initialize each of model parameters w m,n and π m,(cid:96) with zero-mean i.i.d.
Gaussians with unit variances.
We refer such a parameterization as NTK initialization .
In this paper, we consider a model such that all M trees have the same complete binary tree structure, a common setting for soft tree ensembles (Popov et al., 2020; Kontschieder et al., 2015; Hazimeh et al., 2020).
T HEORETICAL RESULTS 4. B ASIC PROPERTIES OF THE TNTK
We call the NTK in Equation (3) induced by the soft tree ensembles the TNTK and denote by (cid:98) Θ ( d ) 0 ( x i , x j ) the TNTK at initialization induced by the ensemble of trees with depth d .
In this section,
1.0 Inner product of the inputs 0.
0 ( x i , x j ) d =3 , =2.0 M =
M =64 M =256 M =1024 M =
M = 1010 3 M 101010|| H
H || F || H
|| F
Change with fixed d = trend: M 1/2 d =3, = 0.5 d =3, = 1.0 d =3, = 2.
d =3, = 4.0 d =3, = 8.
d =3, = 16.0 d =3, = 32.
d =3, = 64.
1010 3 M 1010|| H
H || F || H
|| F
Change d with fixed =2.
trend: M 1/2 d =1, =2.0 d =2, =2.0 d =3, =2.
d =4, =2.0 d =5, =2.0 d =6, =2. only for showing the slope.
The error-bars show the standard deviations of 10 executions.
we show the properties of the TNTK that are important for understanding the training behavior of the soft tree ensembles.
4.1. TNTK FOR INFINITE TREE ENSEMBLES
First, we show the formula of the TNTK at initialization, which converges when considering the limit of inﬁnite trees ( M → ∞ ) .
Theorem 1.
Let u ∈ R N 0 × 1 be any column vector sampled from zero-mean i.i.d.
Gaussians with unit variance.
The TNTK for an ensemble of soft complete binary trees with tree depth d converges in probability to the following deterministic kernel as M → ∞ ,
[Equation]
, (7) where Σ( x i , x j ) :
= x (cid:62) i x j , T ( x i , x j ) :=
E [ σ ( u (cid:62) x i ) σ ( u (cid:62) x j )] , and ˙ T ( x i , x j ) :=
E [ ˙ σ ( u (cid:62) x i ) ˙
σ ( u (cid:62) x j )] .
Moreover, T ( x i , x j ) and ˙ T ( x i , x j ) are analytically obtained in the closedform as
[Equation]
The dot used in ˙ σ ( u (cid:62) x i ) means the ﬁrst derivative: αe − ( α u (cid:62) x i ) 2 / √ π , and E [ · ] means the expectation.
The scalar π in Equation (8) and Equation (9) is the circular constant, and u corresponds to w m,n at an arbitrary internal node.
The proof is given by induction.
We can derive the formula of the limiting TNTK by treating the number of trees in a tree ensemble like the width of the hidden layer in MLP, although the MLP and the soft tree ensemble are apparently different models.
Due to the space limit, detailed proofs are given in the supplementary material.
We demonstrate convergence of the TNTK in Figure 3.
We empirically observe that the TNTK induced by sufﬁciently many soft trees converges to the limiting TNTK given in Equation (7).
The kernel values induced by an ﬁnite ensemble are already close to the limiting TNTK if the number of trees is larger than several hundreds, which is a typical order of the number of trees in practical applications 1 .
It is therefore reasonable to analyze soft tree ensembles via the TNTK.
By comparing Equation (7) and the limiting NTK induced by a two-layer perceptron (shown in the supplementary material), we can immediately derive the following when the tree depth is 1 .
Corollary 1.
If the splitting function at the tree internal node is the same as the activation function of the neural network, the limiting TNTK obtained from a soft tree ensemble of depth 1 is equivalent to the limiting NTK generated by a two-layer perceptron up to constant multiple.
For any tree depth larger than 1 , the limiting NTK induced by the MLP with any number of layers (Arora et al., 2019, shown in the supplementary material) and the limiting TNTK do not match.
It implies that the hierarchical splitting structure is a distinctive feature of soft tree ensembles.
4.1. P OSITIVE DEFINITENESS OF THE LIMITING TNTK
Since the loss surface of a large model is expected to be highly non-convex, understanding the good empirical trainability of overparameterized models is still an open problem (Dauphin et al., 2014).
The positive deﬁniteness of the limiting kernel is one of the most important conditions for achieving global convergence (Du et al., 2019a; Jacot et al., 2018).
Jacot et al.
(2018) have shown that the conditions (cid:107) x i (cid:107) 2 = 1 for all i ∈
[ N ] and x i (cid:54) =
x j ( i (cid:54) =
j ) are necessary for the positive deﬁniteness of the NTK induced by the MLP for an input set.
As for the TNTK, since the formulation (Equation (7)) is different from that of typical neural networks such as an MLP, it is not obvious whether or not the limiting TNTK is positive deﬁnite.
We prove that the TNTK induced by inﬁnite trees is also positive deﬁnite under the same condition for the MLP.
Proposition 1.
For inﬁnitely many soft trees with any depth and the NTK initialization, the limiting
TNTK is positive deﬁnite if (cid:107)
x i (cid:107) 2 = 1 for all i ∈ [ N ] and x i (cid:54) =
x j ( i (cid:54) = j ) .
The proof is in the supplementary material.
Similar to the discussion for the MLP (Du et al., 2019a;
Jacot et al., 2018), if the limiting TNTK is constant during training, the positive deﬁniteness of the limiting TNTK at initialization indicates that training of the inﬁnite trees with a gradient method can converge to the global minimum.
We show the constantness of the limiting TNTK during training in the next section.
4.1. C HANGE OF THE TNTK DURING TRAINING
We prove that the TNTK hardly changes from its initial value during training when considering an ensemble of inﬁnite trees with ﬁnite α (used in Equation (5)).
Theorem 2.
Let λ min and λ max be the minimum and maximum eigenvalues of the limiting TNTK.
Assume that the limiting TNTK is positive deﬁnite for input sets.
For soft tree ensemble models with the NTK initialization and a positive ﬁnite scaling factor α trained under gradient ﬂow with a learning rate η < 2 / ( λ min + λ max ) , we have with high probability
[Equation]
The complete proof is in the supplementary material.
Figure 4 shows that the training trajectory analytically obtained (Jacot et al., 2018; Lee et al., 2019) from the limiting TNTK and the trajectory during gradient descent training become similar as the number of trees increases, demonstrating the validity of using the TNTK framework to analyze the training behavior.
Remarks.
In the limit of inﬁnitely large α , which corresponds to a hard decision tree splitting (Figure 2), it should be noted that this theorem does not hold because of the lack of local Lipschitzness (Lee et al., 2019), which is the fundamental property for this proof.
Therefore the change in the
TNTK during training is no longer necessarily asymptotic to zero, even if the number of trees is inﬁnite.
This means that understanding the hard decision tree’s behavior using the TNTK is not straightforward.
4. I MPLICATIONS FOR PRACTICAL TECHNIQUES
In this section, from the viewpoint of the TNTK, we discuss the training techniques that have been used in practice.
[Figure]
4.2. I NFLUENCE OF THE OBLIVIOUS TREE STRUCTURE
An oblivious tree is a practical tree model architecture, where the rules across decision tree splitting are shared across the same depth as illustrated in Figure 5.
Since the number of the splitting decision calculation can be reduced from O (2 d ) to O ( d ) , the oblivious tree structure is used in various open-source libraries such as CatBoost (Prokhorenkova et al., 2018) and NODE (Popov et al., 2020).
However, the reason for the good empirical performance of oblivious trees is non-trivial despite weakening the expressive power due to parameter sharing.
We ﬁnd that the oblivious tree structure does not change the limiting TNTK from the non-oblivious one.
This happens because, even with parameter sharing at splitting nodes, leaf parameters π are not shared, resulting in independence between outputs of left and right subtrees.
Theorem 3.
The TNTK with the complete binary tree ensemble and the TNTK of its corresponding oblivious tree ensemble obtained via parameter sharing converge to the same kernel in probability in the limit of inﬁnite trees ( M → ∞ ) .
The complete proof is in the supplementary material.
Note that leaf values π do not have to be the same for oblivious and non-oblivious trees.
This theorem supports the recent success of tree ensemble models with the oblivious tree structures.
4.2. E FFECT OF THE DECISION FUNCTION MODIFICATION
Practically, based on the commonly used sigmoid function, a variety of functions have been proposed for the splitting operation.
By considering a large scaling factor α in Equation (5), we can envisage situations in which practically-used hard functions such as two-class sparsemax , σ ( x ) = sparsemax([ x, 0])
(Martins & Astudillo, 2016), and two-class entmax , σ ( x ) = entmax([ x, 0])
(Peters et al., 2019).
Figure 6 shows α dependencies of TNTK parameters.
Equation (7) means that the
TNTK is formulated by multiplying T ( x i , x j ) and ˙
T ( x i , x j ) to the linear kernel Σ( x i , x j ) .
On the one hand, when we look at T ( x i , x j ) and ˙ T ( x i , x j ) with small α , they are almost constant even for
1 inner product of the inputs 0.
Normalized (2) ( x i , x j )  1 Inner product of the inputs 0.
1.00 Normalized ( d ) ( x i , x j ), =  1 inner product of the inputs 0.
( x i , x j )  1 inner product of the inputs 0.
( x i , x j )      5 10 15 20 25 d different input inner products, resulting in the almost linear TNTK.
On the other hand, we can see that the nonlinearity increases as α increases.
When we look at Figure 2, the original sigmoid function corresponds to a scaled error function for α ∼ 0 . 5 , which is almost the linear kernel.
Although a closed-form TNTK using sigmoid as a decision function has not been obtained, its kernel is expected to be almost linear.
Therefore, from the viewpoint of the TNTK, an adjustment of the decision function (Frosst & Hinton, 2017; Popov et al., 2020; Hazimeh et al., 2020) can be interpreted as an escape from the linear kernel behavior.
4.2. D EGENERACY CAUSED BY DEEP TREES
As the depth increases, T ( x i , x j ) deﬁned in Equation (8), which consists of the arcsine function, is multiplied multiple times to calculate the limiting TNTK in Equation (7).
Therefore, when we increase the depth too much, the resulting TNTK exhibits degeneracy: its output values are almost the same with each other even though the input’s inner products are different.
The rightmost panel of Figure 6 shows such degeneracy behavior.
In terms of kernel regression, models using a kernel that gives almost the same inner product to all data except those that are quite close to each other are expected to have poor generalization performance.
Such behavior is observed in our numerical experiments (Section 5).
In practical applications, one does not usually use too deep soft or hard decision trees as too deep trees show poor performance (Luo et al., 2021), which is supported by the degeneracy of the TNTK.
N UMERICAL EXPERIMENTS
Setup We present our experimental results on 90 classiﬁcation tasks in the UCI database (Dua &
Graff, 2017), with less than 5000 data points as in Arora et al.
We performed kernel regression using the limiting TNTK deﬁned in Equation (7) with varying parameters, the tree depth ( d ) and the scaling ( α ) of the decision function.
The limiting TNTK does not change during training for an inﬁnite ensemble of soft trees (Theorem 2), hence predictions from that model are equivalent to kernel regression using the limiting TNTK
(Jacot et al., 2018).
For considering the ridge-less situation, regularization strength is set to be 1 .
× 10 − 8 , a very small constant.
For a comparison, performances of the kernel regression with the MLP-induced NTK (Jacot et al., 2018) and the RBF kernel are also reported.
For the MLP-induced NTK, we use ReLU for the activation function.
We follow the procedures of Arora et al.
and Fernández-Delgado et al.
We report 4 -fold cross-validation performance with random data splitting.
To tune parameters, all available training samples are randomly split into one training and one validation set, while imposing that each class has the same number of training and validation samples.
Then the parameter with the best validation accuracy is selected.
Other details are in the supplementary material
Comparison to the MLP The left panel of Figure 7 shows the averaged performance as a function of the depth.
Although the TNTK with properly tuned parameters tend to be better than those obtained with the RBF kernel, they are often inferior to the MLP-induced NTK.
The results support the good performance of the MLP-induced NTK (Arora et al., 2020).
However, it should be noted that when we look at each dataset one by one, the TNTK is superior to the MLP-induced NTK for more than 30% [Figure] [Table] of the dataset, as shown in Table 1.
This is a case where the characteristics of data and the inductive bias of the model ﬁt well together.
In addition, although the computational cost of the MLP-induced
NTK is linear with repsect to the depth of the model because of the recursive computation (Jacot et al., 2018), that of the TNTK does not depend on the depth as shown in Equation (7), resulting in much faster computation than the MLP-induced NTK when the depth increases, illustrated in the right panel of Figure 7.
Even if the MLP-induced NTK is better in prediction accuracy, the TNTK may be used in practical cases as a trade-off for computational complexity.
Arora et al. (2019) proposes the use of the NTK for a neural architecture search (Elsken et al., 2019;
Chen et al., 2021).
In such applications, the fast computation of the kernel in various architectures can be a beneﬁt.
We leave extensions of this idea to tree models as future work.
Consistency with implications from the TNTK theory When we increase the tree depth, we observe an improvement in performance at the beginning.
After that, the performance gradually decreases.
This behavior is consistent with the performance deterioration due to degeneracy (Section 4.2.3), similar to that reported for neural networks without skip-connection (Huang et al., 2020) shown with the dotted yellow line.
The performance improvement by adjusting α in the decision function (Frosst & Hinton, 2017) is also observed.
Performances with hard ( α > 0 .
5 ) decision functions are always better than the sigmoid -like function ( α = 0 .
5 , as shown in Figure 2).
C ONCLUSION
In this paper, we have introduced and studied the Tree Neural Tangent Kernel (TNTK) by considering the ensemble of inﬁnitely many soft trees.
The TNTK provides new insights into the behavior of the inﬁnite ensemble of soft trees, such as the effect of the oblivious tree structure and the degeneracy of the TNTK induced by the deepening of the trees.
In numerical experiments, we have observed the degeneracy phenomena induced by the deepening of the soft tree model, which is suggested by our theoretical results.
To date, the NTK theory has been mostly applied to neural networks, and our study is the ﬁrst to apply it to the tree model.
Therefore our study will be a milestone of further development of the NTK theory.
E THICS STATEMENT
We believe that the TNTK’s theoretical analysis will not have a negative impact on society as our work does not directly lead to harmful applications.
R EPRODUCIBILITY STATEMENT
Proofs of all theoretical results are given in the supplementary material.
For the numerical experiments and ﬁgures, we share our reproducible source code in the supplementary material
A P ROOF OF T HEOREM
Proof.
The model output from a certain depth tree ensemble f ( d ) can be written alternatively using an incremental formula as
[Equation]
where indices ( l ) and ( r ) used with model parameters w m and π m mean the parameters at the (l)eft subtree and the (r)ight subtree, respectively, and t used in w m,t denotes the node at (t)op of the tree.
For example, for trees of depth 3 as shown in Figure 1, w ( l ) m = ( w m, 2 , w m, 4 , w m, 5 ) , w ( r ) m = ( w m, 3 , w m, 6 , w m, 7 ) , and w m,t = w m, 1 .
We prove the theorem by induction.
When d = 1 ,
[Equation]
Derivatives are
[Equation]
Therefore, since there is only one internal node per a single tree, from the deﬁnition of the NTK, the
TNTK is obtained as
[Equation]
∂f ( x i , w , π ) ∂π m,, ∂f ( x j , w , π ) ∂π m,(cid:29) + (cid:28) ∂f ( x i , w , π ) ∂π m,, ∂f ( x j , w , π ) ∂π m,(cid:29) (cid:33) =
M
M (cid:88) m = (cid:16) ( π m, 1 − π m, 2 ) 2 x (cid:62) i x j ˙ σ ( w (cid:62) m, 1 x i ) ˙
σ ( w (cid:62) m, 1 x j ) + 2 σ ( w (cid:62) m, 1 x i ) σ ( w (cid:62) m, 1 x j ) (cid:17) .
(A.6)
Since we are considering the inﬁnite number of trees ( M → ∞ ) , the average in Equation (A.6) can be replaced by the expected value by applying the law of the large numbers:
[Equation]
which is consistent with Equation (7).
Here, E m (cid:104) ( π m, 1 − π m, 2 ) 2 (cid:105) = 2 because the variance of π m,(cid:96) is 1 . 0 , and w m, 1 corresponds to u in Theorem 1.
For d > 1 , we divide the TNTK into four components:
[Equation]
where the indices ( t ) , ( l ) and ( r ) mean the parameters of the (t)op of the tree, (l)eft subtree, and (r)ight subtree, respectively.
The index ( b ) implies the (b)ottom of the tree: tree leaves.
With Equation (A.7), we have
[Equation]
For each component, we show the following lemmas:
Lemma 1.
[Equation]
Lemma 2.
[Equation]
Lemma 3.
[Equation]
Combining them, we can derive Equation (7).
A. P ROOF OF L EMMA
Proof.
An incremental formula for the model output with a certain depth tree ensemble is
[Equation]
A
[Equation]
where t used in w m,t implies the node at the top of the tree.
With Equation (A.15), ∂f
[Equation]
Since f ( d ) m (cid:16) x i , w ( r ) m , π ( r ) m (cid:17) and f ( d ) m (cid:16) x j , w ( l ) m , π ( l ) m (cid:17) are independent to each other and have zero-mean
Gaussian distribution because of the initialization of π m with zero-mean i.i.d
Gaussians 2 ,
[Equation]
and
[Equation]
are zero.
Therefore, considering the inﬁnite number of trees ( M → ∞ ) ,
[Equation]
From Equation (A.18), what we need to prove is E m (cid:104) f
[Equation]
) m
[Equation]
x
[Equation]
w
[Equation]
π
[Equation]
) m
[Equation]
x
[Equation]
w
[Equation]
m ) (cid:105) =
(2 T ( x i , x j ))
d .
We prove it by induction.
In the base case with d = 1 ,
[Equation]
where we use the property
[Equation] [Equation]
[Equation] [Equation]
for any v generated from a zero-mean Gaussian distribution.
The subscript arrows ( → ) show what the expected value will be.
Next, when the depth is d + 1 ,
[Equation]
[Table]
  
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
 

[Table]
x j
[Equation]
w
[Equation]
r
[Equation]
m
[Equation]
π
[Equation]
r
[Equation]
m
[Equation]
21) where the last equality sign is just a simpliﬁcation to separate components (A), (B), (C), and (D).
Since f ( d ) m ( x i , w ( r ) m , π ( r ) m ) and f ( d ) m ( x j , w ( l ) m , π ( l ) m ) are independent to each other and have zero-mean i.i.d Gaussian distribution, we obtain
[Equation]
Equation (A.23), Equation (A.24), and Equation (A.25) cancel each other out.
Therefore, we obtain
[Equation]
By induction hypothesis and
Equation (A.19), we have
E m (cid:104) f
( d ) m ( x i , w m , π m ) f
[Equation]
d ) m
[Equation]
x
[Equation]
m
[Equation]
π m )
[Equation]
=
[Equation]
T
[Equation]
x i , x j ))
d .
Therefore, the original lemma also follows.
A. P ROOF OF L EMMA
Proof.
When the depth is d + 1 , the derivatives of the parameters in the left subtree and the right subtree are
[Equation]
) where l and r used in w m,l and w m,r implies the node at the left subtree and right subtree, respectively.
Therefore, the limiting TNTK for the left subtree is
[Equation]
, x j )) .
(A.29)
Similarly, for the right subtree, since
[Equation]
we can also derive
[Equation]
Finally, we obtain the following by combining with Equation (A.29) and Equation (A.31),
[Equation]
[Equation] [Equation]
A. P ROOF OF L EMMA
Proof.
With Equation (1) and Equation (6),
[Equation]
[Table]
(A.33)
When we focus on a leaf (cid:96) , for a tree with depth of d , 1 n (cid:38) (cid:96) or 1 (cid:96) (cid:46) n equals to 1 d times.
Therefore, by Equation (A.30), we can say that there is a T ( x i , x j ) d contribution to the limiting kernel Θ ( d ) , ( b ) ( x i , x j ) per leaf index (cid:96) ∈
[ L ] .
Since there are 2 d leaf indices in the complete binary tree, Θ
[Equation]
[Equation]
[Equation]
In other words, since the number of leaves doubles with each additional depth, we can say
[Equation]
A. C LOSED - FORM FORMULA FOR THE SCALED ERROR FUNCTION
Since we are using the scaled error function as a decision function deﬁned as
[Equation]
T and ˙ T in Theorem 1 can be calculated analytically.
Closed-form solutions for the error function (Williams, 1996; Lee et al., 2019) are known to be
[Equation]
˙
[Equation]
[Equation]
Using the above equations
[Equation]
we can calculate T and ˙ T with the scaled error function as
[Equation]
B N EURAL T ANGENT K ERNEL FOR MULTI - LAYER PERCEPTRON B. E QUIVALENCE BETWEEN THE TWO - LAYER PERCEPTRON AND TREES OF DEPTH
By using the same symbols used in soft trees (Section 2.1) to make it easier to see the correspondences, we consider a two-layer perceptron as
[Equation]
where we use M as the number of the hidden layer nodes, σ as a nonlinear activation function, and w = ( w 1 , . . .
, w m ) ∈ R N 0 × M and a = ( a 1 , . .
. , a m ) ∈ R 1 × M as parameters at the ﬁrst and second layers initialized by zero-mean Gaussians with unit variances.
Since
[Equation]
we have
[Equation]
Considering the inﬁnite width limit ( M → ∞ ) [Equation] we have
[Equation]
which is the same as the limiting TNTK shown in Equation (7) with d = 1 up to constant multiple [Equation]
B. F ORMULA FOR THE MLP- INDUCED NTK
Based on Arora et al. [Equation] we deﬁne the L -hidden-layer perceptron 3 as f MLP (L) ( x i , W ) =
W ( L +1) ·
[Equation]
where W (1) ∈ R M 1 × N 0 , W ( h ) ∈ R M h
× M h − 1 , and W ( L +1) ∈
R 1 × M L are trainable parameters.
We initialize all the weights W = (cid:0) W (1) , . . .
, W ( L +1) (cid:1) to values independently drawn from the standard normal distribution.
Considering the limit of the inﬁnite width M 1 , M 2 , . .
, M L → ∞ , the formula for the limiting NTK of L -hidden-layer MLP is known to be
[Equation]
where
[Equation]
B.
[Equation]
We let ˙Σ ( L +1) ( x i , x j ) :
= 1 for convenience.
See Arora et al.
for derivation.
There is a correspondence between Σ (1) ( x i , x j ) and Σ ( x i , x j ) in Theorem 1, Σ (1) ( x i , x j ) and T ( x i , x j ) in
Theorem 1, and ˙Σ (1) ( x i , x j ) and ˙ T ( x i , x j ) in Theorem 1, respectively.
Since the recursive calculation is needed in Equation (B.7), the computational cost increases as the layers get deeper.
It can be seen that the effect of increasing depth is different from that of the limiting
TNTK, in which the depth of the tree affects only the value of the exponential power as shown in
Equation (7).
Therefore, for any tree depth larger than 1 , the limiting NTK induced by the MLP with any number of layers and the limiting TNTK do not match.
C P ROOF OF P ROPOSITION
Proof.
As shown in Section 4.1.1, there is a close correspondence between the soft tree ensemble of depth 1 and the two-layer perceptron.
On the one hand, from Equation (A.7), the limiting TNTK induced by inﬁnite trees with the depth of 1 is 2(Σ( x i , x j ) ˙
T ( x i , x j ) + T ( x i , x j )) .
On the other hand, if the activation function used in the two-layer perceptron is same as σ deﬁned in Equation (5), the NTK induced by the inﬁnite width two-layer MLP is Σ( x i , x j ) ˙
T ( x i , x j ) + T ( x i , x j ) (Jacot et al., 2018; Lee et al., 2019).
Hence these are exactly the same kernel up to constant multiple.
The conditions under which the MLP-induced NTK are positively deﬁnite have already been studied.
Lemma 4 (Jacot et al.
For a non-polynomial Lipschitz nonlinearity σ , for any input dimension N 0 , the NTK induced by the inﬁnite width MLP is positive deﬁnite if (cid:107) x
i (cid:107) 2 = 1 for all i ∈ [ N ] and x i (cid:54) =
x j ( i (cid:54) = j ) .
Note that σ deﬁned in Equation (5) has the non-polynomial Lipschitz nonlinearity.
Since the positive deﬁnite kernel multiplied by a constant is a positive deﬁnite kernel, it follows that the limiting TNTK Θ (1) ( x i , x j ) for the depth 1 is also positive deﬁnite.
As shown in Equation (C.1), as the trees get deeper, T ( x i , x j ) deﬁned in Equation (8) is multiplied multiple times in the limiting TNTK: [Equation]
( x i , x j ) (cid:124) (cid:123)(cid:122) (cid:125) contribution from inner nodes + (
[Equation]
x i
[Equation]
j
[Equation]
) d (cid:124) (cid:123)(cid:122) (cid:125) contribution from leaves = 2(
[Equation]
x i , x j ))
[Equation]
[Equation]
x i
[Equation]
j )
[Equation]
x i , x j ) + T ( x i , x j )) (cid:124) (cid:123)(cid:122) (cid:125) NTK induced by two-layer perceptron (if d = 1 ) .
(C.1)
The positive deﬁniteness of T ( x i , x j ) has already been proven.
Lemma 5 (Jacot et al.
For a non-polynomial Lipschitz nonlinearity σ , for any input dimension N 0 , the T ( x i , x j ) :
= E [ σ ( u (cid:62) x i ) σ ( u (cid:62) x j )] deﬁned in Theorem 1 is positive deﬁnite if (cid:107) x
i (cid:107) 2 = 1 for all i ∈ [ N ] and x i (cid:54) =
x j ( i (cid:54) = j ) .
Note that d Σ( x i , x j ) ˙
T ( x i , x j ) + T ( x i , x j ) for d ∈ N is positive deﬁnite.
Since the product of the positive deﬁnite kernel is positive deﬁnite, for inﬁnite trees of arbitrary depth, the positive deﬁniteness of Θ ( d ) ( x i , x j ) holds under the same conditions as in MLP.
D P ROOF OF T HEOREM
Proof.
We use the following lemmas in the proof.
Lemma 6.
Let a ∈ R n be a random vector whose entries are independent standard normal random variables.
For every v ≥ 0 , with probability at least 1 − 2 n e ( − v 2 n/ 2) we have:
[Equation]
Lemma 7.
Let a i ∈ R ≥ 0 .
We have
[Equation]
In addition, our proof is based on the strategy used in Lee et al.
(2019) which relies on the the local
Lipschitzness of the model Jacobian at initialization J ( x , θ ) , whose ( i, j ) entry is ∂f ( x i , θ )
∂θ j where θ j is a j -th component of θ :
Theorem 4 (Lee et al.
Assume that the limiting NTK induced by any model architecture is positive deﬁnite for input sets x such that minimum eigenvalue of the NTK λ min > 0 .
For models with [Figure] local gradient a rate we have with high
[Equation]
It is not obvious whether or not the soft tree ensemble’s
Jacobian is local Lipschitz.
Therefore, we prove Lemma 8 to prove Theorem 2.
Lemma 8.
For soft tree ensemble models with the NTK initialization and a positive ﬁnite scaling factor α , there is K > 0 such that for every C > 0 , with high probability, the following holds:
[Equation]
where
[Equation]
By proving that the soft tree ensemble’s Jacobian under the NTK initialization is the local Lipschitz with high probability, we extend the Theorem 2 for the TNTK.
D. P ROOF OF L EMMA
Proof.
By use of the Chebyshev’s inequality, for some constant c , we obtain
[Equation]
where P means a probability.
Since erf( γ/ √ 2) ≤ 1 , when we use γ = c/n , we get
[Equation]
Lemma 6 can be obtained by assigning vn to c .
Figure 8 shows the right hand side of the Equation (D.7) with c = 5 n .
when n = 1 , probability is 7 . 45 × 10 − 6 .
As n becomes larger, the probability becomes even smaller.
D. P ROOF OF L EMMA
Proof.
By use of Cauchy-Schwarz inequality, for p, q, x, y ∈ R ≥ 0 , we have
[Equation]
With Equation (D.8), we prove the lemma by induction.
In the base case, √
[Equation]
which is consistent to the lemma.
Next, when we assume √
[Equation]
we have
[Equation]
D. P ROOF OF L EMMA
Proof.
Consider the contribution of the leaf parameters at ﬁrst:
[Equation]
[Equation]
Next, contribution from the leaf parameters is
[Equation]
where
[Equation]
14) and & is a logical conjunction.
For any real scalar p and q , the scaled error function σ deﬁned in
Equation (5) is bounded as follows:
[Equation]
Therefore, the absolute value of S n,(cid:96) does not exceed 1 .
With Equation (D.15) [Equation] we can obtain
[Equation]
with high probability.
Therefore, with Lemma 6, in probability,
[Equation]
2 F =
N (cid:88) i = (cid:0) (cid:107) J
( x i , w ) (cid:107) 2 F + (cid:107) J
( x i , π ) (cid:107) 2 F (cid:1) = M N (cid:88) i = M (cid:88) m =  
N (cid:88) n =   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
L (cid:88) (cid:96) =
π m,(cid:96) ∂µ m,(cid:96)
( x i , w m ) ∂ w m,n (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  
 +
L (cid:88) (cid:96) = (cid:0) µ m,(cid:96) ( x i , w m ) 2 (cid:1) ≤
M
N (cid:88) i =
{+M (cid:88) m = (cid:32) N (cid:88) n = v 2 L 2 α 2 (cid:107) x i (cid:107) 
+}
+
L (cid:88) (cid:96) =  (cid:33) =
N (cid:88) i =
L ( v 2 L α 2 N(cid:107) x i (cid:107) 22 +1) .
(D.17)
Next, we will consider the Jacobian difference.
Since µ m,(cid:96) ( x i , w m ) is a multiple multiplication of the decision function, by use of
[Equation]
we obtain
[Equation]
where it should be noted that ( (cid:96) (cid:46) n ) & ( n (cid:38) (cid:96) ) must be false.
S n,(cid:96) ( x i , w m ) − S n,(cid:96) ( x i , ˜ w m ) can be bound in the same way as Equation (D.19).
Therefore, we also obtain
[Equation]
To link Equation (D.19) and Equation (D.20) to the (cid:107) θ − ˜ θ (cid:107) 2 , we use Lemma 7 to obtain the following inequalities:
[Equation]
With Equation (D.1), Equation (D.16), Equation (D.19), Equation (D.20), Equation (D.21), and Equation (D.22), [Equation] [Equation]
[Equation]
=
[Equation]
(cid:88)
[Equation]
n
[Equation]
N
[Equation]
n
[Equation]
L (cid:88) (cid:96)
[Equation]
(cid:18)
[Equation]
≤
M
[Equation]
i =
[Equation]
≤
M N (cid:88) i = M (cid:88) m = (cid:32)
[Equation]
By considering the square root of both sides in Equation (D.17) and Equation (D.23)
[Equation]
we concludes the proof for Lemma
[Equation]
E P ROOF OF T HEOREM
Proof.
We can use the same approach with the proof of Theorem 1.
Using an incremental formula
[Equation]
the output from the oblivious tree ensembles can be written as follows:
[Equation]
where ( s ) of w ( s ) m means (s)hared parameters at subtrees.
Intuitively, the fundamental of Theorem 3 is that the outputs of the left subtree and right subtree are still independent with oblivious tree structure.
Even with parameter sharing at the same depth, since the leaf parameters π are not shared, the outputs of the left subtree and right subtree are independent.
We will see that Lemma 1, 2 and 3 are also valid for oblivious tree ensembles.
Correspondence to Lemma 1 To show the correspondence to Lemma 1, it is sufﬁcient to show that
Equation (A.22), Equation (A.23), Equation (A.24), and Equation (A.25) hold when
[Equation]
This equation corresponds to Equation (A.21).
Here, since the leaf parameters π are not shared, the outputs of the left subtree and right subtree are still independent even with the oblivious tree structure.
Therefore, we can obtain the correspondences to Equation (A.22), Equation (A.23), Equation (A.24), and Equation (A.25) with the same procedures.
Correspondence to Lemma 2 For the depth d + 1 , since
[Equation]
the corresponding limiting TNTK is
[Equation]
∂
[Equation]

 
 =
d (cid:88) s =
E m   
[Equation]
(cid:124) (cid:123)(cid:122)
[Equation]

[Equation]
 
[Equation]
(cid:124) (cid:123)(cid:122) (cid:125) (C)
[Equation]
( d )
[Equation]
(cid:16)
[Equation]
j
[Equation]
[Equation]
[Equation]
π ( r
[Equation] ∂ [Equation] (cid:124) (cid:123)(cid:122) (cid:125)
(D)   
 .
(E.4)
Since ∂f ( d ) m ( x i , w ( s ) m , π ( r ) m ) ∂ w m,s and ∂f ( d ) m ( x j , w ( s ) m , π ( l ) m ) ∂ w m,s for s = { 2 , 3 , . . .
, d } are independent to each other and have zero-mean Gaussian distribution 4 , similar calculation used for Equation (A.22), Equa- tion (A.23), Equation (A.24), and Equation (A.25) gives
E m [ (A) × (C) ]
= T ( x i , x j ) E m
[Equation]
E m [ (B) × (C) ]
= − 0 .
[Equation]

[Equation]
∂
[Equation]

[Equation]

[Equation]
[Equation]
[Equation]
E m [ (A) × (D) ]
= − 0 .
5 E m      ∂f ( d ) m (cid:16) x i , w ( s ) m , π ( r ) m (cid:17) ∂ w m,s 
 (cid:62)   ∂f ( d ) m (cid:16) x j , w ( s ) m
[Equation]
π ( r ) m (cid:17) ∂ w m,s 
[Equation]
E m [ (B) ×
(D) ]
=
E m
[Equation]
As in the previous calculations, Equation (E.6), Equation (E.7), and Equation (E.8) cancel each other out.
As a result, we obtain
[Equation]
Correspondence to Lemma 3 Considering Equation (A.33), once we focus on a leaf (cid:96) , it is not possible for both 1 1 (cid:38) (cid:96) and 1 (cid:96) (cid:46) 1 to be 1 .
This means that a leaf cannot belong to both the right subtree and the left subtree.
Therefore, even with oblivious tree structure, there is no inﬂuences.
Therefore, we get exactly the same result to the Lemma 3.
F D ETAILS OF NUMERICAL EXPERIMENTS
F. S ETUP F.1. D ATASET ACQUISITION
We use the UCI datasets (Dua & Graff, 2017) preprocessed by Fernández-Delgado et al.
(2014), which are publicly available at http://persoal.citius.usc.es/manuel.fernandez.
delgado/papers/jmlr/data.tar.gz .
Since the size of the kernel is the square of the dataset size and too many data make training impractical, we use preprocessed UCI datasets with the number of samples smaller than 5000 .
Arora et al.
(2020) reported the bug in the preprocess when the explicit training/test split is given.
Therefore, we do not use that dataset with explicit training/test split.
As a consequence, 90 different datasets are available.
F.1. K ERNEL SPECIFICATIONS
TNTK See Theorem 1 for the detailed deﬁnitions.
We change the tree depth from 1 to 29 and change α in { 0 . 5 , 1 . 0 , 2 . 0 , 4 . 0 , 8 . 0 , 16 . 0 , 32 . 0 , 64 . 0 } .
MLP-induced NTK
We assume the MLP activation function as ReLU .
Our implementation is based on the publicly available code 5 used in Arora et al.
For the detailed deﬁnitions, see Arora et al.
The hyperparameter of this kernel is the model depth.
We change the depth from 1 to 29 .
Here, depth = 1 means there is no hidden layer in the MLP. [Figure] [Figure]
RBF kernel We use scikit-learn implementation 6 .
The hyperparameter of this kernel is γ , inverse of the standard deviation of the RBF kernel (Gaussian function).
For Figure 3, we tune γ in { 0 . 01 , 0 .
9 , 1 . 0 , 2 . 0 , 3 . 0 , 4 . 0 , 5 . 0 , 6 . 0 , 7 . 0 , 8 . 0 , 9 . 0 , 10 . 0 , 20 . 0 , 30 . 0 } , resulting in 30 candidates in total.
In our experiments, γ = 2 . 0 performs the best on average (Figure 9).
F.1. M ODEL SPECIFICATIONS
We used kernel regression implemented in scikit-learn 7 .
To consider ridge-less situation, regularization strength is set to be 1 .
× 10 − 8 , a very small constant.
F.1. C OMPUTATIONAL COSTS
Since the training and inference algorithms of the kernel regression are common across different kernels, we analyze the computational cost of computing a single value in a gram matrix of the corresponding kernels in the following.
The time complexity of the MLP-induced NTK is linear with respect to the layer depth, while that of the TNTK remains to be constant.
Such a trend can be seen in the right panel of Figure 7.
For the RBF kernel, the computational cost remains the same with respect to changes in hyperparameters, thus its trend is similar to the TNTK.
In terms of the space complexity, when considering a multi-layered MLP, since it is not necessary to store all past calculation results in memory during the recursive computation, the MLP-induced NTK computation consumes a certain amount of memory regardless of the depth of the layers.
Therefore, the memory usage is almost the same across the RBF kernel, TNTK, and MLP-induced NTK.
F.1. C OMPUTATIONAL RESOURCE
We used Ubuntu Linux (version: 4.15.0-117-generic) and ran all experiments on 2.20 GHz Intel Xeon
E5-2698 CPU and 252 GB of memory.
F. R ESULTS F.2. S TATISTICAL SIGNIFICANCE OF THE PARAMETER DEPENDENCY
Wilcoxon signed rank test is carried out to check the statistical signiﬁcance of the differences between different α .
Figure 10 shows the p-values for the depth of 3 and 20 .
As shown in Figure 7, when the tree is shallow, the accuracy started to deteriorate after around α = 8 . 0 , but as the tree becomes deeper, the deterioration became less apparent.
Therefore, statistically signiﬁcantly different pairs for deep tress and shallow trees are different.
When the tree is deep, large α shows a signiﬁcant difference over those with small α .
However, when the tree is shallow, the best performance is achieved with α of about 8 . 0 , and if α is too large, the performance deteriorates predominantly. [Figure] [Figure] [Figure]
[Figure]
F.2. D ATASET - WISE RESULTS
For each α , scatter-plots are shown in Figures 11 and 12.
As shown in Figure 13, the correlation coefﬁcients with the TNTK are likely to be higher for the MLP-induced NTK compared to the
RBF kernel.
Tables 2, 3 and 4 are dataset-wise results of the comparison between the TNTK, the
MLP-induced NTK, and the RBF kernel.
For each α , depth is tuned for each dataset.
In terms of the depth, the best performers from 1 to 29 are compared with the TNTK and the MLP-induced NTK.
For the RBF kernel, γ is tuned in each dataset from 30 candidate values as described in Section F.1.2.
Therefore, the number of tunable parameters is the same across all methods.
All parameter-wise results are visualized in Figures 14 and 15.
n a m e s i ze α = 0 .
α = 1 .
α
= 2 .
α
= 4 .
α = 8 .
α = 16 .α
= 32 .α
= 64 .
M L P - N T
K
RB F t r a i n s 87 .
b a ll oon s 87 .
l e n s e s 87 .
l ung - ca n ce r 56 .
po
s top e r a ti v e 63 .
p itt s bu r gb r
i dg e sSP
AN 55 .
f e r tilit y 84 .
z oo 100 .
p itt s bu r gb r
i dg e s - T - O R - D 81 .
p itt s bu r gb r
i dg e s - R
EL - L 67 .
p itt s bu r gb r
i dg e s - T
Y P E 57 .
m o l ec - b i
o l - p r o m o t
e r 90 .
p itt s bu r gb r
i dg e s - M
A TE
R I A
L 93 .
b r ea s t - ti ss u e 65 .
α c u t e - n e ph r iti s 100 .
α c u t e - i n ﬂ a mm a ti on 100 .
h ea r t - s w it
ze r l a nd 37 .
ec ho ca r d i og r a m 81 .
l y m phog
r a phy 88 .
i r i s 95 .
t eac h i ng 56 .
h e p a titi s 83 .
w i n e 98 .
p l a nn i ng 62 .
ﬂ a g s 52 .
p a r k i n s on s 93 .
b r ea s t - ca n
ce rw
i s cp r og 82 .
h ea r t - v a 31 .
c onn - b e n c h - s on a rm i n e s -r o c k s 86 .
s ee
d s 90 .
T a b l e 2 : C
o m p a r
i s on b e t w ee n
T N T
K a nd
M L P - i ndu ce d
N T K f o r a h a l f o f t
h e d a t a s e t ( 1 /) .
n a m e s i ze α = 0 .
α = 1 .
α
= 2 .
α
= 4 .
α = 8 .
α = 16 .α
= 32 .α
= 64 .
M L P - N T
K
RB F g l a ss 60 .
s t a tl og - h ea r t 83 .
b r ea s t - ca n ce r 64 .
h ea r thung a r i a n 83 .
h
ea r t - c
l e v e l a nd 55 .
h a b e r m a n - s u r v
i v a l 59 .
v e r t e b r a lc o l u m nc l a s e s 69 .
v e r t e b r a lc o l u m nc l a s e s 71 .
p r i m a r y - t u m o r 47 .
ec o li 71 .
i ono
s ph e r e 90 .
li b r a s 82 .
d e r m a t
o l ogy 97 .
c ong r e ss i on a l - vo ti ng 61 .
α rr hy t h m i a 69 .
m u s k89 .
c y li nd e rb a nd s 79 .
l o w -r e ss p ec t 91 .
b r ea s t - ca n
ce rw i s cd i a g 96 .
il pd - i nd i a nli v e r 64 .
s yn t
h e ti c - c on t r o l 99 .
b a l a n ce - s ca l e 81 .
s t a tl og - a u s t r a li a nc r e d it 59 .
c r e d it - a pp r ov a l 82 .
b r ea s t - ca n
ce rw i s c 96 .
b l ood 67 .
e n e r gy - y  89 .
p i m a 68 .
e n e r gy - y  93 .
s t a tl og - v e h
i c l e 78 .
T a b l e 3 : C
o m p a r
i s on b e t w ee n
T N T
K a nd
M L P - i ndu ce d
N T K f o r a h a l f o f t
h e d a t a s e t ( 2 /) .
n a m e s i ze α = 0 .
α = 1 .
α
= 2 .
α
= 4 .
α = 8 .
α = 16 .α
= 32 .α
= 64 .
M L P - N T
K
RB F oo c y t
e s _ t
r
i s op t e r u s _nu c l e u s _2 f 9182 .
[Table] [Table]
[Table]
oo c y t e s _ t r
i s op t e r u s _ s t a t e s _ 5b 9192 .
[Table]
[Table] [Table]
[Table]
ti c - t ac - t
o e
[Table] [Table]
[Table] [Table]
[Table]
m a mm og r a ph i c 9672 .
[Table]
[Table] [Table]
[Table] [Table]
[Table]
s t a tl o g - g e r m a n - c r e d it 1075 .
[Table]
[Table] [Table]
l e dd i s p l a y 172 .
oo c y t e s _ m e r l u cc i u
s _nu c l e u s _4d 181 .
oo c y t e s _ m e r l u cc i u
s _ s t a t e s _ f 192 .
c on t r ac 140 .
y ea s t 142 .
s e m e i on 193 .
w i n e - qu a lit y -r e d 1563 .
p l a n t - t e x t u r e 183 .
p l a n t - m a r g i n 184 .
p l a n t - s h a p e 164 .
ca r 1797 .
s t ee l - p
l a t e s 176 .
ca r d i o t
o c og r a phyc l a s e s 292 .
ca r d i o t
o c og r a phy10 c l a s e s 280 .
tit a n i c 278 .
s t a tl o g - i m a g e 296 .
o z on e 297 .
m o l ec - b i o l - s p li ce 386 .
c h e ss - k r vkp 399 .
α b a l o n e 450 .
b a nk 488 .
s p a m b a s e 4691 .
w i n e - qu a lit y - w h it e 463 .
w a v e f o r m -
no i s e 586 .
w a v e f
o
r
m 5085 .
T a b l e 4 : C
o m p a r
i s on b e t w ee n
T N T
K a nd
M L P - i ndu ce d
N T K f o r a h a l f o f t
h e d a t a s e t ( 3 /) .
[Figure] [Figure]