Original: Depth is More Powerful than Width  in Deep ForestAnonymous Author(s)
Revised: Depth is More Powerful than Width with Feature Concatenation in Deep Forest

Original: Abstract
Revised: Abstract

Original: Random Forest (RF) is an ensemble learning algorithm proposed by Breiman[2001] that constructs a large number of randomized decision trees individuallyand aggregates their predictions by naive averaging.
Revised: Random Forest (RF) is an ensemble learning algorithm proposed by Breiman [1] that constructs a large number of randomized decision trees individually and aggregates their predictions by naive averaging.

Original: Zhou and Feng [2019] furtherpropose Deep Forest (DF) algorithm with multi-layer feature transformation, whichsignificantly outperformssingle-layer random forest in various application fields.
Revised: Zhou and Feng [2] further propose Deep Forest (DF) algorithm with multi-layer feature transformation, which significantly outperforms random forest in various application fields.

Original: Despite its great successes, little is  known about the mathematical properties of thecascade structure.
Revised: The prediction concatenation (PreConc) operation is crucial for the multi-layer feature transformation in deep forest, though little has been known about its theoretical property.

Original: In this paper, we analyze the influence of depth and width on theconsistency of cascade forest.
Revised: In this paper, we analyze the influence of Preconc on the consistency of deep forest.

Original: Especially when the individual tree is inconsistent (asin practice, the individual tree is often set to be fully grown, i.e., there is only onesample at each leaf node), we find that the convergence rate of two-layer DF w.r.t.the number of trees M can reach O (1 /M 2 ) under some mild conditions, whilethe convergence rate ofsingle-layer RF is O (1 /M ) .
Revised: Especially when the individual tree is inconsistent (as in practice, the individual tree is often set to be fully grown, i.e., there is only one sample at each leaf node), we find that the convergence rate of two-layer DF w.r.t. the number of trees M can reach O (1 /M 2 ) under some mild conditions, while the convergence rate of RF is O (1 /M ) .

Original: Therefore, learning decisiontrees in the “deep” layer will be more powerful thanlearning in the “shallow” layer.
Revised: Therefore, with the help of PreConc, DF with deeper layer will be more powerful than the shallower layer.

Original: Experimentsfurther confirmthe theoretical advantages.
Revised: Experiments confirm theoretical advantages.

Original:  Introduction
Revised:  Introduction

Original: Random forest (RF) [Breiman, 2001] is a state-of-art classification and regression algorithm thatconstructs a number of randomized individual decision trees during a parallel training phase andpredicts by naive averaging the results.
Revised: Random forest (RF) [1] is a state-of-art classification and regression algorithm that constructs a number of randomized individual decision trees during a parallel training phase and predicts by naive averaging the results.

Original: Recently, Zhou and Feng [2019] propose Deep Forest(DF) algorithm with multi-layer feature transformation to investigate the possibility of tree-basedrepresentation learning.
Revised: Recently, Zhou and Feng [2] propose Deep Forest (DF) algorithm with multi-layer feature transformation to investigate the possibility of tree-based representation learning.

Original: Benefiting from the layer-by-layer feature transformation in  cascade structure(depth), DF outperforms various single-layer forest algorithms [Breiman, 2001, Friedman, 2001, Geurts et al., 2006, Chen and Guestrin, 2016] in empirical study and have been involved in realapplications such as medicine [Su et al., 2019], computer vision [Zhou et al., 2019], remote sensing[Boualleg et al., 2019], financial fraud detection [Zhang et al., 2019] etc.
Revised: Benefiting from the multi-layer feature transformation in the cascade structure with prediction concatenation (PreConc), DF outperforms various tree-based algorithms [1, 3–5] in empirical study, and have been involved in real applications such as medicine [6], computer vision [7], remote sensing [8], financial fraud detection [9] etc.

Original: Numerous variants havebeen extended to various tasks and meet with remarkable success in Utkin and Ryabinin [2018, 2019], Yang et al.
Revised: Numerous variants have been extended to various tasks and meet with remarkable success in [10–13].

Original: [2020], Wang et al. There are also variants  aiming at improving performance andreducing computational cost [Pang et al., 2020, Chen et al., 2021].
Revised: There are also variants [14, 15] aiming at improving performance and reducing computational cost.

Original: Empirical successes have attracted attention to the theoretical analysis of DF.
Revised: Empirical successes have attracted attention to the theoretical analysis of DF.

Original: Since its cascadestructure transforms features layer by layer, and each layer consists of complex non-parametricrandom forest estimators, analyzing the impact of the new features extracted from these estimatorson generalization performance is important.
Revised: Since its PreConc operation transforms features layer by layer, and each layer consists of complex non-parametric random forest estimators, analyzing the impact of the new features extracted from these estimators on generalization performance is important.

Original: To start theoretical analysis on DF, Lyu et al.
Revised: To start theoretical analysis on DF, Lyu et al.

Original: [2019]prove a margin-based generalization bound for additive new features in DF.
Revised: [16] prove a margin-based generalization bound for additive new features in DF.

Original: Then, Pang et al.
Revised: Then, Pang et al.

Original: [2020]prove an upper bound on the sample efficiency and inspire an efficient improvement for reducingcomputational cost of DF.
Revised: [14] prove an upper bound on the sample efficiency and inspire an efficient improvement for reducing computational cost of DF.

Original: In the direction of consistency, Arnould et al.
Revised: In the direction of consistency, Arnould et al.

Original: [2021] provide tight lowerand upper bounds on the excess risk of a shallow centered random tree network, which leveragesthe new features to improve the performance of a single centered random tree.
Revised: [17] provide tight lower and upper bounds on the excess risk of a shallow centered random tree network, which leverages the new features to improve the performance of a single centered random tree.

Original: Assuming that theoriginal and new features are separated and independently used in two stages, Lyu et al.
Revised: Assuming that the original and new features are separated and independently used in two stages, Lyu et al.

Original: [2022] prove that the new features based on prediction are easy to cause overfitting risk, and propose to use theinteraction of decision rules to alleviate it.
Revised: [18] prove that the new features based on prediction are easy to cause overfitting risk, and propose to use the interaction of decision rules to alleviate it.

Original: However, previous theoretical studies often choose to ignore the impact of the new features on theforest generation.
Revised: However, previous theoretical studies often choose to ignore the impact of the new features on generating forests.

Original: For example, Lyu et al. [2019, 2022], Pang et al. [2020] assume that the forestestimator in each layer is a black-box model and Arnould et al.
Revised: For example, recent works [16, 18, 14] assume that the forest estimator in each layer is a black-box model and Arnould et al.

Original: [2021] simplify that there is only asingle centered random tree in each layer.
Revised: [17] assume that there is only a single centered random tree in each layer.

Original: Such strong assumptions are in favor of theoretical analysis,but widen the gap between the theoretical results and practical deep forest architecture.
Revised: Such strong assumptions are in favor of theoretical analysis, but widen the gap between the theoretical results and practical deep forest architecture.

Original: The difficultyin properly analyzing DF can be explained by thesubtle combination of different techniques. Amongthe complex architecture in DF, both prediction concatenation (PreConc) [Lyu et al., 2019] and theclassification and regression trees (CART)-split criterion [Breiman et al., 1984] playa critical role.
Revised: To properly analyze DF, we must notice that, among the complex architecture in DF, both prediction concatenation (PreConc) [16] and the classification and regression trees (CART)-split criterion [19] play critical roles.

Original: Therefore, in order to open the black box of DF, we need to  analyze the properties of the new featureand study how CART generates and utilizes it.
Revised: Therefore, in order to open the black box of DF, we need to study the subtle combination of different techniques, i.e., analyze the properties of the new feature and study how CART generates and utilizes it.

Original: Contributions.
Revised: Contributions.

Original: In this paper, we try to narrow the gap between theoretical and empirical studies on

Original: DF. We compare the advantages of depth (cascade structure with PreConc in deep forest) over width(bagging of trees in random forest) in the scope of L 22 -consistency, based on the assumptions of theadditive regression functions and uniform distribution over input space.
Revised: We compare the advantages of depth (cascade structure with PreConc in deep forest) over width (bagging of trees in random forest) in the scope of L 22 -consistency, based on the assumptions of the additive regression functions and uniform distribution over input space.

Original: The main contributions canbe summarized as follows:
Revised: The main contributions can be summarized as follows:

Original: • We are the first to establish a theoretical analysis of the priority of the new feature under theCART-split criterion in DF, which means the trees in the second layer and beyond tend to primarilychoose the new feature to split. We further prove that the degree of this priority depends on theerror of the first-layer forest estimator.
Revised: • We are the first to establish a consistency analysis of the prediction concatenation (PreConc) operation which is crucial for multi-layer feature transformation in deep forest, though based on a simplified version.

Original: • We prove the universal consistency ofCART, random forest and deep forest when the total number of leaves of individual trees is chosen properly.
Revised: • We prove the universal consistency of deep forest when the total number of leaves of individual trees is chosen properly.

Original: Although in this setting, neither depth nor widthwill affect the consistency, experiments show that such trees have poor performance in practice.•

Original:  In the practical setting, when the individual trees are fully grown, we prove that the convergencerate of two-layer deep forest reach O (1 /M 2 ) w.r.t.
Revised: • In the practical setting, when the individual trees are fully grown, we prove that the convergence rate of two-layer deep forest reach O (1 /M 2 ) w.r.t.

Original: the increasing number of trees M , while that ofsingle-layer random forest is O (1 /M ) .
Revised: the increasing number of trees M , while that of random forest is O (1 /M ) .

Original: This result reflects that depth is more powerful than width.
Revised: This result reflects that deep forest with deeper layer will be more powerful than shallower layer.

Original: Agenda.
Revised: Organization.

Original: The rest of this work is organized as follows: Section 3 shows the setting and notationsrelated to tree-based estimators in this work.
Revised: The rest of this work is organized as follows: Section 3 shows the setting and notations related to tree-based estimators in this work.

Original: Section 4 recall the original deep forest architecture andsimplify it into a two-layer deep forest.
Revised: Section 4 recalls the original deep forest architecture and simplifies it into a two-layer deep forest.

Original: Section 5 contains the properties of cascade structure withprediction concatenation.
Revised: Section 5 contains the properties of cascade structure with prediction concatenation.

Original: Section 6 proves the main result that depth is more powerful than width indeep forest architecture in the scope of consistency.
Revised: Section 6 proves the main result that depth is more powerful than width in deep forest architecture in the scope of consistency.

Original: Section 7 is devoted to the empirical studies ofdeep forest by verifying the theoretical results above.
Revised: Section 7 is devoted to the empirical studies of deep forest by verifying the theoretical results above.

Original: Section 8 concludes with future work.
Revised: Section 8 concludes with future work.

Original: Detailedproofs for theorems and propositions are given in the supplementary material due to page limitation.
Revised: More experimental results and detailed proofs for theorems and propositions are given in the supplementary material due to page limitation.

Original:  Related work
Revised:  Related work

Original: Despite the widespread use and remarkable success of random forest in real world applications, thetheoretical properties of it are still not fully understood [Biau, 2012, Denil et al., 2014].
Revised: Despite the widespread use and remarkable success of random forest in real world applications, the theoretical properties of it are still not fully understood [20, 21].

Original: Breiman[2001] offers an upper bound on the generalization error of random forest in terms of correlation andaccuracy of the individual trees.
Revised: Breiman [1] offers an upper bound on the generalization error of random forest in terms of correlation and accuracy of the individual trees.

Original: Lin and Jeon [2006] establish a connection between random forestand a particular class of nearest neighbor predictors, which are further studied by Biau and Devroye[2010].
Revised: Lin and Jeon [22] establish a connection between random forest and a particular class of nearest neighbor predictors, which are further studied by Biau and Devroye [23].

Original: Meinshausen [2006] studies the consistency of the quantile random forest for regression.
Revised: Meinshausen [24] studies the consistency of the quantile random forest for regression.

Original: Inrecent years, various theoretical works [Biau, 2012, Biau et al., 2008, Ishwaran and Kogalur, 2010, Genuer, 2012, Zhu et al., 2015] have been performed, analyzing the consistency of various simplifiedforests, and moving ever closer to practice.
Revised: In recent years, various theoretical works [20, 25–28] have been performed, analyzing the consistency of various simplified forests, and moving ever closer to practice.

Original: Denil et al.
Revised: Denil et al.

Original: [2014] narrows the gap between theoryand practice of random forests for regression.
Revised: [21] narrows the gap between theory and practice of random forests for regression.

Original: Scornet et al.
Revised: Scornet et al.

Original: [2015] prove the first L 22 -consistencyof Breiman’s original random forest with CART-split criterion based on the assumptions of theadditive regression functions and uniform distribution over input space.
Revised: [29] prove the first L 22 -consistency of Breiman’s original random forest with CART-split criterion based on the assumptions of the additive regression functions and uniform distribution over input space.

Original: Scornet [2016] prove thatinfinite forest consistency implies finite forest consistency.
Revised: Scornet [30] prove that infinite forest consistency implies finite forest consistency.

Original: Gao and Zhou [2020] then present theconvergence rate of purely randomized trees and a simplified variant of Breiman’s original CARTtrees.
Revised: Gao and Zhou [31] then present the convergence rate of purely randomized trees and a simplified variant of Breiman’s original CART trees. Gao et al.

Original:  In addition, another research route is the theory analysis of feature importance [Strobl et al. ,2007, Louppe et al., 2013, Arnould et al., 2022].
Revised: [32] further expand it to multi-class setting. In addition, another research route is the theory analysis of feature importance [33–35].

Original: Recently, Li et al.
Revised: Recently, Li et al.

Original: [2019] derive non-asymptoticlower and upper bounds on the expected bias of MDI importance for random forests.
Revised: [36] derive non-asymptotic lower and upper bounds on the expected bias of MDI importance for random forests.

Original: Sutera et al.[2021] establish a connection between MDI importance of pure random forest and Shapley values.
Revised: Sutera et al. [37] establish a connection between MDI importance of pure random forest and Shapley values.

Revised: However, while deep forests further improve generalization performance, there is no theory to prove the advantages brought by depth.

Revised: Therefore, studying the influence of depth is the theoretical cornerstone for distinguishing deep forests from random forests.

Revised: For example, in the well-known deep neural networks (DNNs), there are a lot of theoretical works to study the effect of depth and width on its representation ability and generalization performance, which show the theoretical advantages of deep neural networks over shallow neural networks [38–42].

Revised: These works all contribute to the understanding of deep learning and provide insight for designing algorithms.

Original:  Setting and notations
Revised:  Setting and notations

Original: We first describe the setting and notations related to tree-based estimators in this work.
Revised: We first describe the setting and notations related to tree-based estimators in this work.

Original: For the sakeof conciseness, we consider the regression setting.
Revised: For the sake of conciseness, we consider the regression setting.

Original: Setting.
Revised: Setting.

Original: We consider a regression framework, where the training set S n = { ( x 1 , y 1 ) , . . .
Revised: We consider a regression framework, where the training set S n = { ( x 1 , y 1 ) , . . .

Original: , ( x n , y n ) }of [0 , 1] d × R -valued independent random variables distributed as the prototype sample ( x , y ) with
Revised: , ( x n , y n ) } of [0 , 1] d × R -valued independent random variables distributed as the prototype sample ( x , y ) with

Original: E [ y 2 ] < ∞ .
Revised: E [ y 2 ] < ∞ .

Original: This underlying distribution, characterized by the marginal distribution D X on [0 , 1] dand by the conditional distribution D Y|X , can be written as
Revised: This underlying distribution, characterized by the marginal distribution D X on [0 , 1] d and by the conditional distribution D Y|X , can be written as

Original: [Equation]
Revised: [Equation]

Original: where f ( x ) =
Revised: where f ( x ) =

Original: E [ y | x ] is the conditional expectation of y given x , and ϵ is a noise satisfying E
Revised: E [ y | x ] is the conditional expectation of y given x , and ϵ is a noise satisfying E

Original: [ ϵ ] = 

Revised: [ ϵ ] =

Original: and Var[ ϵ ] < σ 2 .
Revised: and Var[ ϵ ] < σ 2 .

Original: The task considered in this paper is to output a randomizedestimator h n ( · , Θ , S n ): [0 , 1] d → R where Θ is a random variable that accounts for the randomization procedure and independent of the training set S n .
Revised: The task considered in this paper is to output a randomized estimator h n ( · , Θ , S n ): [0 , 1] d → R where Θ is a random variable that accounts for the randomization procedure and independent of the training set S n .

Original: To simplify notation, we denoteh n ( x , Θ) =
Revised: To simplify notation, we denote h n ( x , Θ) =

Original: h n ( x , Θ , S n ) .
Revised: h n ( x , Θ , S n ) .

Original: The quality of a randomized estimator h n is measured by its L 2 risk
Revised: The quality of a randomized estimator h n is measured by its L 2 risk

Original: [Equation]
Revised: [Equation]

Original:  where the expectation is taken with respect to ( x , Θ) , conditionally on S n .
Revised: ) where the expectation is taken with respect to ( x , Θ) , conditionally on S n .

Original: As the training data size nincreases, we get a sequence of estimators { h 1 , h 2 , . .
Revised: As the training data size n increases, we get a sequence of estimators { h 1 , h 2 , . .

Original: h n , . . .
Revised: h n , . . .

Original: A sequence of estimators { h n } ∞ n =1is said to be consistent if R ( h n ) → 0 as n → ∞ .
Revised: A sequence of estimators { h n } ∞ n = is said to be consistent if R ( h n ) → 0 as n → ∞ .

Original: Trees and forests.
Revised: Trees and forests.

Original: A random forest estimator h M,n ( x , Θ) outputs the average prediction over Mindividual randomized trees h n ( x , Θ j ) , ∀ j ∈
Revised: A random forest estimator h M,n ( x , Θ) outputs the average prediction over M individual randomized trees h n ( x , Θ j ) , ∀ j ∈

Original: [ M ] .
Revised: [ M ] .

Original: Here, [ M ] = { 1 , 2 , . . .
Revised: Here, [ M ] = { 1 , 2 , . . .

Original: , M } denotes the indexesof all individual randomized trees, where Θ 1 , . . .
Revised: , M } denotes the indexes of all individual randomized trees, where Θ 1 , . . .

Original: , Θ M are distributed identically and independentlyand denoted by a generic random variable Θ .
Revised: , Θ M are distributed identically and independently and denoted by a generic random variable Θ .

Original: The random variable Θ can be used to sample thetraining set or select the candidate dimensions or positions for splitting.
Revised: The random variable Θ can be used to sample the training set or select the candidate dimensions or positions for splitting.

Original: Specifically, a recursivepartition Π of [0 , 1] d is built by performing successive axis-aligned splits according to Θ :
Revised: Specifically, a recursive partition Π of [0 , 1] d is built by performing successive axis-aligned splits according to Θ :

Original: [Equation]
Revised: [Equation]

Original: where C Π ,n ( x ) is the cell of the tree partition containing x and N n ( C Π ,n ( x )) is the number oftraining samples falling into C Π ,n ( x ) with convention that the estimation equals to zero if the cell
Revised: where C Π ,n ( x ) is the cell of the tree partition containing x and N n ( C Π ,n ( x )) is the number of training samples falling into C Π ,n ( x ) with convention that the estimation equals to zero if the cell

Original: C Π ,n ( x ) is empty.
Revised: C Π ,n ( x ) is empty.

Original: These trees are combined to form a finite forest estimation:
Revised: These trees are combined to form a finite forest estimation:

Original: [Equation]
Revised: [Equation]

Original: By the law of large numbers, for any fixed x , conditionally on D n , the finite forest estimationconverges to the infinite forest estimation:
Revised: By the law of large numbers, for any fixed x , conditionally on D n , the finite forest estimation converges to the infinite forest estimation:

Original: [Equation]
Revised: [Equation]

Original: When the number of samples tends to ∞ , we denote by h ∞ ( x , Θ) the randomized tree with infinitesamples and h M, ∞ ( x , Θ) the random forest with M trees and infinite samples.
Revised: When the number of samples tends to ∞ , we denote by h ∞ ( x , Θ) the randomized tree with infinite samples and h M, ∞ ( x , Θ) the random forest with M trees and infinite samples.

Original:  Deep forest
Revised:  Deep forest

Original: We recall the original deep forest algorithm in Section 4.1, and describe the simplified two-layer deepforest algorithm in Section 4.2.4.1 Original deep forest
Revised: We recall the original deep forest algorithm in Section 4.1, and describe the simplified two-layer deep forest algorithm in Section 4.2.

Original: Deep forest [Zhou and Feng, 2019] is a tree-based deep model made up of non-differentiableforest modules without back-propagation.

Original: Each layer of DF is composed of two Breiman’s randomforest (RF) and two Completely-Random Forest (CRF).

Original: The RF is composed of CARTs and CRF

Original: I npu t F e a t u r e
Revised: I npu t F e a t u r e

Original: s concatenate
Revised: s concatenate

Original: RF RF
Revised: RF RF

Original: CRF CRF
Revised: CRF CRF

Original: Layer
Revised: Layer

Original: Layer∙∙∙
Revised: Layer∙∙∙

Original: Layer 𝑁
Revised: Layer 𝑁

Original: ~ predictions
Revised: ~ predictions

Original: F i n a l
Revised: F i n a l

Original: P r e d i c t i o n
Revised: P r e d i c t i o n

Original: RF RF
Revised: RF RF

Original: CRF CRF
Revised: CRF CRF

Original: RF RF
Revised: RF RF

Original: CRF CRF
Revised: CRF CRF

Original: N e w
Revised: N e w

Original: F e a t u r e s  is composed of pure random trees.
Revised: F e a t u r e s 4.1 Original deep forest Deep forest [2] is a tree-based deep model made up of non-differentiable forest modules without back-propagation. Each layer of DF is composed of two Breiman’s random forest (RF) and two

Revised: Completely-Random Forest (CRF).

Revised: The RF is composed of CARTs and CRF is composed of pure random trees.

Original: In the regression setting, each forest of each layer outputs aprediction for any query point x , corresponding to the average response in the leaf node (cell)containing x .
Revised: In the regression setting, each forest of each layer outputs a prediction for any query point x , corresponding to the average response in the leaf node (cell) containing x .

Original: At a given layer, the predictions of all forests of this layer are concatenated togetherwith raw features.
Revised: At a given layer, the predictions of all forests of this layer are concatenated together with raw features.

Original: This prediction concatenation process is called PreConc, which is repeated foreach layer until the best layer and construct a deep forest.
Revised: This prediction concatenation process is called PreConc, which is repeated for each layer until the best layer and construct a deep forest.

Original: For an overview of the architecture of deepforest, we refer readers to the work 
Revised: For an overview of the architecture of deep forest, we refer readers to the work [43, 2].

Original: [Zhou and Feng, 2017, 2019].

Original: 4.2 Simplified two-layer deep forest
Revised: 4.2 Simplified two-layer deep forest

Original: In order to narrow the gap between theoretical and empirical studies of deep forest, we retain CARTsplit criterion and PreConc when simplifying the model.
Revised: In order to narrow the gap between theoretical and empirical studies of deep forest, we retain CARTsplit criterion and PreConc when simplifying the model.

Original: We define a simplified two-layer deep forest,each layer of it is composed of one Breiman’s random forest.
Revised: We define a simplified two-layer deep forest, each layer of it is composed of one Breiman’s random forest.

Original: We denote by h (1) M,n ( x , Θ , S n ) thefirst-layer forest estimator and h (2) M,n ([ x , h ] , Θ , S n ) the second-layer forest estimator, where [ x , h ] isthe concatenation of the raw features x and the new feature h .
Revised: We denote by h (1) M,n ( x , Θ , S n ) the first-layer forest estimator and h (2) M,n ([ x , h ] , Θ , S n ) the second-layer forest estimator, where [ x , h ] is the concatenation of the raw features x and the new feature h .

Original: Then we can define the two-layer deepforest as follows:
Revised: Then we can define the two-layer deep forest as follows:

Original: [Equation]
Revised: [Equation]

Original: The complete algorithm is shown in Algorithm 1.
Revised: The complete algorithm is shown in Algorithm 1.

Original: This algorithm has three parameters: m try ∈{ 1 , . . .
Revised: This algorithm has three parameters: m try ∈ { 1 , . . .

Original: , d } is the number of pre-selected directions for splitting, a n ∈ { 1 , . . .
Revised: , d } is the number of pre-selected directions for splitting, a n ∈ { 1 , . . .

Original: , n } is the number ofsamples in each tree, t n ∈ { 1 , . .
Revised: , n } is the number of samples in each tree, t n ∈ { 1 , . .

Original: . , a n } is the number of leaves in each tree.
Revised: . , a n } is the number of leaves in each tree.

Original: In the default procedure,the parameters are set as follows: m try is set to d .
Revised: In the default procedure, the parameters are set as follows: m try is set to d .

Original: a n is set to n/k , where k is the k -fold crossvalidation in the Zhou and Feng [2019]’s deep forest.
Revised: a n is set to n/k , where k is the k -fold cross-validation in the Zhou and Feng [2]’s deep forest.

Original: t n = a n means we use fully grown CARTwhich is inconsistent in practice.
Revised: t n = a n means we use fully grown CART which is inconsistent in practice.

Original: Notice that k can control the subsampling rate a n /n =
Revised: Notice that k can control the subsampling rate a n /n =

Original: 1 /k , whichis proved by Scornet et al.
Revised: 1 /k , which is proved by Scornet et al.

Original: [2015] to be the key component for imposing tree diversity.
Revised: [29] to be the key component for imposing tree diversity.

Original: Given the above parameters, the most basic part of Algorithm 1 is the training process of each CART.
Revised: Given the above parameters, the most basic part of Algorithm 1 is the training process of each CART.

Original: Let C be a cell and N n ( C ) be the number of data points falling in C .
Revised: Let C be a cell and N n ( C ) be the number of data points falling in C .

Original: A split in C is a pair ( j, z ) ,where j is a dimension in { 1 , . . .
Revised: A split in C is a pair ( j, z ) , where j is a dimension in { 1 , . . .

Original: , d } and z is the position of the split along the j -th dimension in cell
Revised: , d } and z is the position of the split along the j -th dimension in cell

Original: C .
Revised: C .

Original: Let S C be the set of all possible splits in C .
Revised: Let S C be the set of all possible splits in C .

Original: The CART-split criterion [Breiman et al., 1984] takesthe form 
Revised: The CART-split criterion [19] takes the form ˆ

Original: [Equation]
Revised: [Equation]

Original: where C L = { x ∈ C : x ( j ) <
Revised: where C L = { x ∈ C : x ( j ) <

Original: z } , C R = { x ∈ C : x ( j ) ≥ z } , and µ n ( C ) =1 N n ( C ) (cid:80) i : x ∈
Revised: z } , C R = { x ∈ C : x ( j ) ≥ z } , and µ n ( C ) =

Original: [-A y idenotes the average response in any cell C (resp. µ n ( C L ) , µ n ( C R ) ), with the convention 0 / 
 = 0 . At each cell C , the best split ( j ∗ n , z ∗ n ) is selected by maximizing ˆ L-] n (j, z ) over M try and S C, that is,
Revised: [-A y idenotes the average response in any cell C (resp. µ n ( C L ) , µ n ( C R ) ), with the convention 0 / 
 = 0 . At each cell C , the best split ( j ∗ n , z ∗ n ) is selected by maximizing ˆ L-]1 N n ( C

Original: [Equation]
Revised: [Equation]

Revised: (cid:80) i : x ∈

Revised: A y i denotes the average response in

Original: Require:
Revised: Require:

Original: A training set S n = { ( x 1 , y 1 ) , . . .
Revised: A training set S n = { ( x 1 , y 1 ) , . . .

Original: , ( x n , y n ) } , the number of trees 2 M , m try ∈ { 1 , . . .
Revised: , ( x n , y n ) } , the number of trees 2 M , m try ∈ { 1 , . . .

Original: , d } , a n ∈ { 1 , . . . , n } , t n ∈ { 1 , . .
Revised: , d } , a n ∈ { 1 , . . . , n } , t n ∈ { 1 , . .

Original: . , a n } and the query point x ∈ [0 , 1] d .
Revised: . , a n } and the query point x ∈ [0 , 1] d .

Original: Ensure: Two-layer deep forest ¯ h 2 M,n ( · ) =
Revised: Ensure: Two-layer deep forest ¯ h 2 M,n ( · ) =

Original: h (2) M,n ◦ h (1) M,n ( · ) and the prediction of x .
Revised: h (2) M,n ◦ h (1) M,n ( · ) and the prediction of x .

Original: 1: for layer ℓ in { 1 , 2 } do 2: if ℓ = 2 then 3:
Revised: 1: for layer ℓ in { 1 , 2 } do 2: if ℓ = 2 then 3:

Original: S n ← { ([ x 1 , h (1) M,n ( x 1 )] , y 1 ) , . . .
Revised: S n ← { ([ x 1 , h (1) M,n ( x 1 )] , y 1 ) , . . .

Original: , ([ x n , h (1) M,n ( x n )] , y n ) } .
Revised: , ([ x n , h (1) M,n ( x n )] , y n ) } .

Original: ▷ Prediction concatenation.
Revised: ▷ Prediction concatenation.

Original: 4: end if 5: for tree j ∈ { 1 , 2 , . .
Revised: 4: end if 5: for tree j ∈ { 1 , 2 , . .

Original: , M } do 6:
Revised: , M } do 6:

Original: Select a n data points, without replacement, uniformly in S n .
Revised: Select a n data points, without replacement, uniformly in S n .

Original: Set Π 0 = { [0 , 1] d } the partition associated with the root of the tree.
Revised: Set Π 0 = { [0 , 1] d } the partition associated with the root of the tree.

Original: For all 1 ≤ i ≤ a n , set Π i = ∅ .
Revised: For all 1 ≤ i ≤ a n , set Π i = ∅ .

Original: Set n nodes = 1 and level = 0 .
Revised: Set n nodes = 1 and level = 0 .

Original: 10: while n nodes <
Revised: 10: while n nodes <

Original: t n do 11: if Π level = ∅ then 12: level = level + 1 .
Revised: t n do 11: if Π level = ∅ then 12: level = level + 1 .

Original: 13: else 14:
Revised: 13: else 14:

Original: Let C be the first element in Π level .
Revised: Let C be the first element in Π level .

Original: 15: if C contains exactly one point then 16: Π level ← Π level \ { C } .
Revised: 15: if C contains exactly one point then 16: Π level ← Π level \ { C } .

Original: 17: Π level +1 ← Π level +1 ∪ { C } .
Revised: 17: Π level +1 ← Π level +1 ∪ { C } .

Original: 18: else 19:
Revised: 18: else 19:

Original: Select the best split ( j ∗ n , z ∗ n ) in C by optimizing the CART-split criterion along the dimension D in { 1 , . . .
Revised: Select the best split ( j ∗ n , z ∗ n ) in C by optimizing the CART-split criterion along the dimension D in { 1 , . . .

Original: , d } .
Revised: , d } .

Original: ▷ See details in Eq.
Revised: ▷ See details in Eq.

Original: Split cell C along D according to the best split.
Revised: Split cell C along D according to the best split.

Original: Call C L and C R .
Revised: Call C L and C R .

Original: 21: Π level ← Π level \ { C} .
Revised: 21: Π level ← Π level \ [Equation] C

Original:  22: Π level +1 ←Π level +1 ∪ { C L } ∪ { C R } .
Revised: [Equation] 22: Π [Equation] ←

Revised: [Equation]

Original: 23: n nodes ← n nodes + 1 .
Revised: 23: n nodes ← n nodes + 1 .

Original: 24: end if 25: end if 26: end while 27:
Revised: 24: end if 25: end if 26: end while 27:

Original: Compute the predicted value h ( ℓ ) n ( x , Θ j , S n ) at the query point x equal to the average of the Y i ’s falling in the cell of x in partition Π level ∪ Π level +1 .
Revised: Compute the predicted value h ( ℓ ) n ( x , Θ j , S n ) at the query point x equal to the average of the Y i ’s falling in the cell of x in partition Π level ∪ Π level +1 .

Original: 28: end for 29:
Revised: 28: end for 29:

Original: Compute the random forest estimation h ( ℓ ) M,n ( x , Θ , S n ) at the query point x according to
Revised: Compute the random forest estimation h ( ℓ ) M,n ( x , Θ , S n ) at the query point x according to

Original: Eq.
Revised: Eq.

Original: 30: end for 31: Compute the two-layer deep forest estimation ¯ h 2 M,n ( x , Θ , S n ) at the query point x according to Eq.
Revised: 30: end for 31: Compute the two-layer deep forest estimation ¯ h 2 M,n ( x , Θ , S n ) at the query point x according to Eq.

Revised: At each cell C , the best split ( j ∗ n , z ∗ n ) is selected by maximizing ˆ L n ( j, z ) over M try and S C , that is,

Revised: [Equation]

Original:  Properties of cascade structure
Revised:  Properties of prediction concatenation

Original: In this section we show that the properties of the simplified deep forest enable us to analyze theinfluence of the concatenated new feature in deep forest and the local variation related to the empirical
Revised: In this section we show that the properties of the simplified deep forest enable us to analyze the influence of the concatenated new feature in deep forest and the local variation related to the empirical

Original: CART-split criterion.
Revised: CART-split criterion.

Original: We consider an additive regression model satisfying the following assumption:
Revised: We consider an additive regression model satisfying the following assumption:

Original: Assumption 1.
Revised: Assumption 1.

Original: The response y follows
Revised: The response y follows

Original: [Equation]
Revised: [Equation]

Original: where x = (cid:0) x (1) , . . .
Revised: where x = (cid:0) x (1) , . . .

Original: , x ( d ) (cid:1) is uniformly distributed over [0 , 1] d , ϵ is an independent centered
Revised: , x ( d ) (cid:1) is uniformly distributed over [0 , 1] d , ϵ is an independent centered

Original: Gaussian noise with finite variance σ 2 > 

Revised: Gaussian noise with finite variance σ 2 >

Original: and each component f j is continuous.
Revised: and each component f j is continuous.

Original: Stone [1985], Hastie and Tibshirani [1987] popularize these models, which decompose the regressionfunction as a sum of univariate functions.
Revised: Stone [44], Hastie and Tibshirani [45] popularize these models, which decompose the regression function as a sum of univariate functions.

Original: Especially, Scornet et al.
Revised: Especially, Scornet et al.

Original: [2015] prove the consistency of
Revised: [29] prove the consistency of

Original: Breiman’s original random forest under this assumption.
Revised: Breiman’s original random forest under this assumption.

Original: On this basis, we study the impact of thenew feature generated in the deep forest.
Revised: On this basis, we study the impact of the new feature generated in the deep forest.

Original: To start with, we analyze the priority of the new features generated by the previous layer in theselection of splitting features under the CART-split criterion, in both infinite sample regime and finitesample regime respectively.
Revised: To start with, we analyze the priority of the new features generated by the previous layer in the selection of splitting features under the CART-split criterion, in both infinite sample regime and finite sample regime respectively.

Original: Proposition 1 (Priority of the new feature) .
Revised: Proposition 1 (Priority of the new feature) .

Original: Assume the data set follows Assumption 1 and thefirst-layer forest is consistent.
Revised: Assume the data set follows Assumption 1 and the first-layer forest is consistent.

Original: The following results hold for any CART in the second-layer forest.1.
Revised: The following results hold for any CART in the second-layer forest.

Original: In the infinite sample regime ( n = ∞ ), we consider a single second-layer CART h (2) ∞ ( · , Θ) .
Revised: In the infinite sample regime ( n = ∞ ), we consider a single second-layer CART h (2) ∞ ( · , Θ) .

Original: Allsplits in this CART are performed along the new feature only.2.
Revised: All splits in this CART are performed along the new feature only.

Original: In the finite sample regime ( n < ∞ ), we consider a single second-layer CART h (2) n ( · , Θ) .
Revised: In the finite sample regime ( n < ∞ ), we consider a single second-layer CART h (2) n ( · , Θ) .

Original: Fixk ∈ N ∗ and ξ, ρ > 0 .
Revised: Fix k ∈ N ∗ and ξ, ρ > 0 .

Original: Then, with probability ≥ 1 − ρ , for all n large enough, we have, the errorof the first-layer forest is bound by ξ .
Revised: Then, with probability ≥ 1 − ρ , for all n large enough, we have, the error of the first-layer forest is bound by ξ .

Original: As a consequence, the first k splits ( j q,n ( x ) , 1 ≤ q ≤ k ) inthis CART are performed along the new feature only.
Revised: As a consequence, the first k splits ( j q,n ( x ) , 1 ≤ q ≤ k ) in this CART are performed along the new feature only.

Original: Proof sketch.
Revised: Proof sketch.

Original: (P1.1).
Revised: (P1.1).

Original: In the infinite sample regime, the random forest estimation of the first layer haszero error.
Revised: In the infinite sample regime, the random forest estimation of the first layer has zero error.

Original: Therefore, the new feature of the second layer is the target function h (1) M, ∞ ( x ) =
Revised: Therefore, the new feature of the second layer is the target function h (1) M, ∞ ( x ) =

Original: f
Revised: f

Original: ( x ) .
Revised: ( x ) .

Original: Obviously, the new feature is the most informative dimension.
Revised: Obviously, the new feature is the most informative dimension.

Original: Therefore, when splitting in any cell,
Revised: Therefore, when splitting in any cell,

Original: CART algorithm will select the new feature as the splitting dimension.
Revised: CART algorithm will select the new feature as the splitting dimension.

Original: (P1.2).
Revised: (P1.2).

Original: In the finite sampleregime, the random forest estimation of the first layer is not precise.
Revised: In the finite sample regime, the random forest estimation of the first layer is not precise.

Original: Therefore, there is an errorrelated to n between the new feature and the target function.
Revised: Therefore, there is an error related to n between the new feature and the target function.

Original: Firstly, we prove that the distancebetween the theoretical ( n = ∞ ) and empirical ( n < ∞ ) first k splits of the CART algorithm isbounded by cξ with probability ≥ 1 − ρ , when n is large enough.
Revised: Firstly, we prove that the distance between the theoretical ( n = ∞ ) and empirical ( n < ∞ ) first k splits of the CART algorithm is bounded by cξ with probability ≥ 1 − ρ , when n is large enough.

Original: Connecting with the result oftheoretical split above, the proof is completed.
Revised: Connecting with the result of theoretical split above, the proof is completed.

Original: Remark 1.
Revised: Remark 1.

Original: Proposition 1 shows that the trees in the second layer primarily choose the new feature tosplit and the degree of this priority depends on the error of the first-layer forest estimator.
Revised: Proposition 1 shows that the trees in the second layer primarily choose the new feature to split and the degree of this priority depends on the error of the first-layer forest estimator.

Original: This alsoreveals that the advantages of deep forest depend on the performance of the first layer.
Revised: This also reveals that the advantages of deep forest depend on the performance of the first layer.

Original: If the forestof the first layer does not return an estimation with noise reduction, the performance of deep forestcannot be further improved through cascade structure.
Revised: If the forest of the first layer does not return an estimation with noise reduction, the performance of deep forest cannot be further improved through PreConc operation.

Original: This is consistent with the empirical results inprevious work [Zhou and Feng, 2019, Arnould et al., 2021].
Revised: This is consistent with the empirical results in previous work [2, 17].

Original: In order to control the risk of deep forest, we need study the local variation property of the empirical
Revised: In order to control the risk of deep forest, we need study the local variation property of the empirical

Original: CART-split criterion.
Revised: CART-split criterion.

Original: For any cell C , the variation of regression function f ( x ) within C is defined as
Revised: For any cell C , the variation of regression function f ( x ) within C is defined as

Original: [Equation]
Revised: [Equation]

Original: Proposition 2 (Variation of f in the empirical cell) .
Revised: Proposition 2 (Variation of f in the empirical cell) .

Original: Assume that Assumption 1 holds and thefirst-layer forest is consistent.
Revised: Assume that Assumption 1 holds and the first-layer forest is consistent.

Original: The following results hold for any CART in the second-layer foresth (2) n ( · , Θ) .
Revised: The following results hold for any CART in the second-layer forest h (2) n ( · , Θ) .

Original: After splitting along the new feature, the CART will estimate the residual of the first-layerforest estimation.
Revised: After splitting along the new feature, the CART will estimate the residual of the first-layer forest estimation.

Original: Then for all ρ, ξ > 0 , there exists N ∈ N ∗ such that, for all n > N ,
Revised: Then for all ρ, ξ > 0 , there exists N ∈ N ∗ such that, for all n > N ,

Original: [Equation]
Revised: [Equation]

Original: Proof sketch.
Revised: Proof sketch.

Original: Firstly, we prove the variation of f ( x ) within the cell obtained by the theoretical
Revised: Firstly, we prove the variation of f ( x ) within the cell obtained by the theoretical

Original: CART-split criterion converges to zero.
Revised: CART-split criterion converges to zero.

Original: Secondly, we prove that the distance between the theoreticaland empirical first k splits of the CART convergence to zero.
Revised: Secondly, we prove that the distance between the theoretical and empirical first k splits of the CART convergence to zero.

Original: Finally, we prove that the variation off ( x ) within the empirical cell is close to the theoretical cell.
Revised: Finally, we prove that the variation of f ( x ) within the empirical cell is close to the theoretical cell.

Original: Proposition 2 shows that the variation of the regression function f ( x ) within a cell of a randomtree C Π ,n is small provided n is large enough, thereby forcing the approximation error of DF toasymptotically approach zero.
Revised: Proposition 2 shows that the variation of the regression function f ( x ) within a cell of a random tree C Π ,n is small provided n is large enough, thereby forcing the approximation error of DF to asymptotically approach zero.

Original:  Depth is more powerful than width
Revised:  Depth is more powerful than width

Original: Our first result considers the t n < a n regime, where the number of samples in each leaf node tendsto ∞ as t n → ∞ and a n → ∞ .
Revised: Our first result considers the t n < a n regime, where the number of samples in each leaf node tends to ∞ as t n → ∞ and a n → ∞ .

Original: We prove that controlling the depth of the trees through the number of leaves t n is sufficient to achieve consistency of deep forest
Revised: We prove that controlling the depth of the trees through the number of leaves t n is sufficient to achieve consistency of deep forest.

Original: Theorem 3 (Universal consistency) .
Revised: Theorem 3 (Universal consistency) .

Original: Let M ≥ 1 .
Revised: Let M ≥ 1 .

Original: Consider two-layer deep forest ¯ h 2 M,n givenby Eq.
Revised: Consider two-layer deep forest ¯ h 2 M,n given by Eq.

Original: (6) and Breiman’s random forest h M,n given by Eq.
Revised: (6) and Breiman’s random forest h M,n given by Eq.

Original: (4) for the random CARTs satisfyinga n → ∞ , t n → ∞ and t n (log a n ) 9 /a n → 0 .
Revised: (4) for the random CARTs satisfying a n → ∞ , t n → ∞ and t n (log a n ) 9 /a n → 0 .

Original: Then under the setting described in Section 3 andassume the data set follows Assumption 1,1.
Revised: Then under the setting described in Section 3 and assume the data set follows Assumption 1,

Original: [Scornet et al., 2015, Theorem 1] the Breiman’s random forest h 2 M,n is consistent for any M ≥ 1 ,2.
Revised: [29, Theorem 1] the Breiman’s random forest h 2 M,n is consistent for any M ≥ 1 , 2.

Original: the two-layer deep forest ¯ h M,n is consistent for any M ≥1 .
Revised: the two-layer deep forest ¯ h M,n is consistent for any M ≥

Original: Proof sketch.
Revised: Proof sketch.

Original: (T3.1).
Revised: (T3.1).

Original: The universal consistency of Breiman’s random forest is proved by Scornetet al.
Revised: The universal consistency of Breiman’s random forest is proved by Scornet et al.

Original: (T3.2).
Revised: (T3.2).

Original: Firstly, we already know that the variation of the target regression function f ( x )within a cell of a randomized CART in the second-layer forest is small when n is large enough via
Revised: Firstly, we already know that the variation of the target regression function f ( x ) within a cell of a randomized CART in the second-layer forest is small when n is large enough via

Original: Proposition 2.
Revised: Proposition 2.

Original: Similar as Scornet et al.
Revised: Similar as Scornet et al.

Original: [2015], we utilize Proposition 2 to control the approximationerror of the two-layer deep forest.
Revised: [29], we utilize Proposition 2 to control the approximation error of the two-layer deep forest.

Original: And the parameter t n allows us to control the size of the leaves of
Revised: And the parameter t n allows us to control the size of the leaves of

Original: CART, which allows us to have enough samples in each leaf node to smooth the impact of noise, soas to control the estimation error.
Revised: CART, which allows us to have enough samples in each leaf node to smooth the impact of noise, so as to control the estimation error.

Original: Connecting the approximation and estimation error, the consistencyof a CART of second-layer deep forest is proved.
Revised: Connecting the approximation and estimation error, the consistency of a CART of second-layer deep forest is proved.

Original: The universal consistency can be proved via Biauet al.
Revised: The universal consistency can be proved via [25,

Original: [2008, Proposition 1], which guarantees that the error of forest estimator is no more than twicethat of individual randomized CART.
Revised: Proposition 1], which guarantees that the error of forest estimator is no more than twice that of individual randomized CART.

Original: Remark 2.
Revised: Remark 2.

Original: Notice that under this setting, random forest and even deep forest have no obviousadvantages over single CART in theory.
Revised: Notice that under this setting, random forest and even deep forest have no obvious advantages over single CART in theory.

Original: When we use forests in practical, we do not choose to controlthe depth of the trees.
Revised: When we use forests in practical, we do not choose to control the depth of the trees.

Original: Empirical studies in Section 7.2 show that the forest with t n = a n alwaysoutperforms the forest with t n < a n .
Revised: Empirical studies in Section 7.2 show that the forest with t n = a n always outperforms the forest with t n < a n .

Original: Actually, the fully grown trees t n = a n is the setting close topractical forest algorithm.
Revised: Actually, the fully grown trees t n = a n is the setting close to practical forest algorithm.

Original: In order to deal with the t n = a n regime, we need to introduce an assumption first proposed by
Revised: In order to deal with the t n = a n regime [Equation] we need to introduce an assumption first proposed by

Original: Scornet et al.
Revised: Scornet et al.

Original: We denote by Z i = 1 ( x i ∈ C Π ,n ( x ))
Revised: We denote by Z i = 1 ( x i ∈ C Π ,n ( x ))

Original: the indicator that x i falls into the samecell as x in the random tree designed with D n and the random parameter Θ .
Revised: the indicator that x i falls into the same cell as x in the random tree designed with D n and the random parameter Θ .

Original: Z ′
Revised: Z ′

Original: j = 1 ( x i ∈
Revised: j = 1 ( x i ∈

Original: C Π ′ ( x ))is another independent indicator.
Revised: C Π ′ ( x )) is another independent indicator.

Original: Then, we define the correlation between these two indicatorsconditionally on y i , y j or not, respectivelyϕ i,j [Equation] i
Revised: Then, we define the correlation between these two indicators conditionally on y i , y j or not, respectively

Original: [Equation]
Revised: [Equation]

Original: Assumption 2.
Revised: Assumption 2.

Original: Let Z i,j = ( Z i , Z j ) .
Revised: Let Z i,j = ( Z i , Z j ) .

Original: Then one of the following two conditions holds:
Revised: Then one of the following two conditions holds:

Original: One has
Revised: One has

Original: [Equation]
Revised: [Equation]

Original: There exists a constant C > 0 and a sequence ( γ n ) n → 0 such that, almost surely,
Revised: There exists a constant C > 0 and a sequence ( γ n ) n → 0 such that, almost surely,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: (A2.1.) means that the influence of two labels y i , y j on the probability of connection of two couples ofrandom points converge to zero as n → ∞ .
Revised: (A2.1.) means that the influence of two labels y i , y j on the probability of connection of two couples of random points converge to zero as n → ∞ .

Original: (A2.2.) means that the correlation between the noise andthe probability of connection of two couples of random points vanishes quickly enough, as n → ∞ .
Revised: (A2.2.) means that the correlation between the noise and the probability of connection of two couples of random points vanishes quickly enough, as n → ∞ .

Original: However, this assumption is too strong for the Breiman’s original random forest [Breiman, 2001].
Revised: However, this assumption is too strong for the Breiman’s original random forest [1].

Original: Scornet et al.
Revised: Scornet et al.

Original: [2015] emphasize that they cannot know whether or not Assumption 2 is satisfied inrandom forest.
Revised: [29] emphasize that they cannot know whether or not Assumption 2 is satisfied in random forest.

Original: In this paper, we recall this assumption and state that this assumption is mild for thesecond-layer forest estimation in deep forest algorithm.
Revised: In this paper, we recall this assumption and state that this assumption is mild for the second-layer forest estimation in deep forest algorithm.

Original: Since deep forest concatenates the prediction with the raw features as the input for the next layer,the label information is encoded into the new feature of second layer.
Revised: Since deep forest concatenates the prediction with the raw features as the input for the next layer, the label information is encoded into the new feature of second layer.

Original: In this way, the influence oftwo labels on the connection probability of this pair of samples tends to zero, so (A2.1.)
Revised: In this way, the influence of two labels on the connection probability of this pair of samples tends to zero, so (A2.1.)

Original: is mild forthe second-layer forest estimation.
Revised: is mild for the second-layer forest estimation.

Original: From another point of view, Proposition 1, the priority of newfeature also shows that the partition can be independent of labels, and the information of new featureis enough to obtain appropriate partition results, so (A2.2.) is mild too.
Revised: From another point of view, Proposition 1, the priority of new feature also shows that the partition can be independent of labels, and the information of new feature is enough to obtain appropriate partition results, so (A2.2.) is mild too.

Original: Theorem 4 (Depth is more powerful than width) .
Revised: Theorem 4 (Depth is more powerful than width) .

Original: Let M ≥ 1 .
Revised: Let M ≥ 1 .

Original: Consider two-layer deep forest ¯ h 2 M,ngiven by Eq.
Revised: Consider two-layer deep forest ¯ h 2 M,n given by Eq.

Original: (6) and Breiman’s random forest h M,n given by Eq.
Revised: (6) and Breiman’s random forest h M,n given by Eq.

Original: (4) for the random CARTs satisfyinga n → ∞ , t n → ∞ , t n = a n and a n log n/n → 0 .
Revised: (4) for the random CARTs satisfying a n → ∞ , t n → ∞ , t n = a n and a n log n/n → 0 .

Original: Then under the setting described in Section 3 andassume the data set follows Assumption 1 and 2, the following results hold1.
Revised: Then under the setting described in Section 3 and assume the data set follows Assumption 1 and 2, the following results hold

Original: [Scornet et al., 2015, Theorem2] [Scornet, 2016, Theorem 3]
Revised: Theorem Theorem 3]

Original: The Breiman’s random foresth ∞ ,n is consistent, and for all M, n ∈ N ,
Revised: The Breiman’s random forest h ∞ ,n is consistent and for all M, n ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: The two-layer deep forest ¯ h ∞ ,n is consistent, and for all M, n ∈ N , if ∆( f, C Π ,n ( x , Θ)) issmall enough, then
Revised: The two-layer deep forest ¯ h ∞ ,n is consistent, and for all M, n ∈ N , if ∆( f, C Π [Equation] n ( x , Θ)) is small enough, then

Original: [Equation]
Revised: [Equation]

Original: Proof sketch.
Revised: Proof sketch.

Original: (T4.1).
Revised: (T4.1).

Original: The consistency of the infinite Breiman’s random forest is proved by Scornetet al.
Revised: The consistency of the infinite Breiman’s random forest is proved by Scornet et al.

Original: And the convergence rate of the finite random forest with the number of trees M isproved by Scornet [2016].
Revised: And the convergence rate of the finite random forest with the number of trees M is proved by Scornet [30].

Original: (T4.2).
Revised: (T4.2).

Original: Similar as Scornet et al.
Revised: Similar as Scornet et al.

Original: [2015], we recall Proposition 2 to controlthe approximation error of the two-layer deep forest.
Revised: [29], we recall Proposition 2 to control the approximation error of the two-layer deep forest.

Original: Then the estimation error is controlled byforcing the subsampling rate a n /n to be o (1 / log n ) .
Revised: Then the estimation error is controlled by forcing the subsampling rate a n /n to be o (1 / log n ) .

Original: Different from the bagging-style mechanism inrandom forest, the residual-style mechanism shown in Proposition 2 makes the second-layer forest in DF can reuse the first-layer estimation and focus on the residual learning.
Revised: Different from the bagging-style mechanism in random forest, the residual-style mechanism shown in Proposition 2 makes the second-layer forest in DF can reuse the first-layer estimation and focus on the residual learning.

Original: Remark 3.
Revised: Remark 3.

Original: In the t n = a n setting, Scornet et al.
Revised: In the t n = a n setting, Scornet et al.

Original: [2015] show that the sub-sampling rate a n /n is thekey component in random forest.
Revised: [29] show that the sub-sampling rate a n /n is the key component in random forest.

Original: Because the small rate ensures that query point x is connected withenough different data points through different trees, the convergence rate of RF is O (1 /M ) , w.r.t.
Revised: Because the small rate ensures that query point x is connected with enough different data points through different trees, the convergence rate of RF is O (1 /M ) , w.r.t.

Original: thenumber of trees M .
Revised: the number of trees M .

Original: Theorem 4 proves that, if the first layer forest can encode the regression functionf ( x ) into the new feature with noise reduction, the cascade structure  in DF can further accelerate theconvergence.
Revised: Theorem 4 proves that, if the first layer forest can encode the regression function f ( x ) into the new feature with noise reduction, the cascade structure with PreConc in DF can further accelerate the convergence.

Original: Because the second-layer forest estimates the residual of the first layer, the trees in eachlayer of forest are more different.
Revised: Because the second-layer forest estimates the residual of the first layer, the trees in each layer of forest are more different.

Original: As a result, the convergence rate of deep forest will be improved to O (1 /M 2 ) .
Revised: As a result, the convergence rate of deep forest will be improved to O (1 /M 2 ) .

Original: This reflects that depth is more powerful than width.
Revised: This reflects that depth is more powerful than width.

Original:  Simulation experiments 
Revised:  Simulation experiments 7.1 Priority of the new feature

Original: 7.1 Priority of the new feature

Original: This experiment aims to verify the priority of the new feature in choosing which feature to split assuggested in Proposition 1.
Revised: This experiment aims to verify the priority of the new feature in choosing which feature to split as suggested in Proposition 1.

Original: We focus on a second layer decision tree built upon the first layer randomforest.
Revised: We focus on a second layer decision tree built upon the first layer random forest.

Original: Since a regression forest has only one output dimension, there’s only one new feature for thesecond layer tree.
Revised: Since a regression forest has only one output dimension, there’s only one new feature for the second layer tree.

Original: More specifically, we count the maximum consecutive levels from root node thatuses the new feature to split, which we call effective depth for short.
Revised: More specifically, we count the maximum consecutive levels from root node that uses the new feature to split, which we call effective depth for short.

Original: The synthetic data set is generated as y = f ( x ) + ϵ , where
Revised: The synthetic data set is generated as y = f ( x ) + ϵ , where

Original: [Equation]
Revised: [Equation]

Original:  x is uniformly distributed over [0 , 1] 5 , ϵ ∼ N (cid:0) 0 , σ 2 (cid:1) where σ = 0 .
Revised: 18) x is uniformly distributed over [0 , 1] 5 , ϵ ∼ N (cid:0) 0 , σ 2 (cid:1) where σ = 0 .

Original: We vary the number oftraining samples and the number of trees in the first layer of forest, and report the average effectivedepth of 5 runs in Figure 2(a).
Revised: We vary the number of training samples and the number of trees in the first layer of forest, and report the average effective depth of 5 runs in Figure 2(a).

Original: It is easy to observe that no matter what the first layer’s setting is, theeffective depth of new feature is at least 2 . 4 .
Revised: It is easy to observe that no matter what the first layer’s setting is, the effective depth of new feature is at least 2 . 4 .

Original: That is to say, at the beginning of the growing of thesecond layer tree, CART will always choose the new feature to split.
Revised: That is to say, at the beginning of the growing of the second layer tree, CART will always choose the new feature to split.

Original: And we can see that with theincrease of trees in the first layer and training samples, the CART’s preference of new feature extendto deeper layers.
Revised: And we can see that with the increase of trees in the first layer and training samples, the CART’s preference of new feature extend [Figure] t l [Figure]

Original: Figure 2(b) further plots the effective depth against the predictive error of the firstlayer measured by root mean square error (RMSE), showing that the better the first layer’s predictiveperformance, the more the new features are preferred in splitting.
Revised: r [Figure] [Figure] [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: to deeper layers.

Revised: Figure 2(b) further plots the effective depth against the predictive error of the first layer measured by root mean square error (RMSE), showing that the better the first layer’s predictive performance, the more the new features are preferred in splitting.

Original: 7.2 Convergence rate w.r.t. the number of trees
Revised: 7.2 Convergence rate w.r.t. the number of trees

Original: To study the convergence rate of predictive error with respect to the number of trees in the wholemodel, we fix the number of training samples to be 50,000.
Revised: To study the convergence rate of predictive error with respect to the number of trees in the whole model, we fix the number of training samples to be 50,000.

Original: We compare a 2-layer deep forest (DF)with M trees each layer to a random forest (RF) with 2 M trees, and we adopt 3-fold split in trainingboth DF and RF to ensure fairness.
Revised: We compare a 2-layer deep forest (DF) with M trees each layer to a random forest (RF) with 2 M trees, and we adopt 3-fold split in training both DF and RF to ensure fairness.

Original: The performance is measured by RMSE with respect to thenoise-free version of data generating function.
Revised: The performance is measured by RMSE with respect to the noise-free version of data generating function.

Original: Since the training data are with noise, a fully growntree is inconsistent.
Revised: Since the training data are with noise, a fully grown tree is inconsistent.

Original: In Figure 3(a), we set the minimum leaf size of the trees to be √ n , i.e., 233 in thecase when training data size is 50,000.
Revised: In Figure 3(a), we set the minimum leaf size of the trees to be √ n , i.e., 233 in the case when training data size is 50,000.

Original: In Figure 3(b) the trees are fully grown with only one samplein each leaf.
Revised: In Figure 3(b) the trees are fully grown with only one sample in each leaf.

Original: We plot the average RMSE of 5 runs against the increasing number of trees, with thecolored band indicating the standard deviation.
Revised: We plot the average RMSE of 5 runs against the increasing number of trees, with the colored band indicating the standard deviation.

Original: Theorem 3 reveals that when the component trees areconsistent, random forest and deep forest are all consistent.
Revised: Theorem 3 reveals that when the component trees are consistent, RF and DF are both consistent.

Original: However, the consistency analysis resultcannot guarantee the finite sample performance in practice.
Revised: However, the consistency analysis result cannot guarantee the finite sample performance in practice.

Original: Comparing Figure 3(a) and Figure 3(b),we can see that even though the training set is as large as 50,000, the performance of RF and DFusing consistent trees is still much worse than using inconsistent trees.
Revised: Comparing Figure 3(a) to Figure 3(b), we can see that even though the training set is as large as 50,000, the performances of RF and DF using consistent trees are still much worse than using inconsistent trees.

Original: This observation confirmsthe efficacy of the default experimental setting of using fully grown trees in RF and DF.
Revised: This observation confirms the efficacy of the default experimental setting that uses fully grown trees in RF and DF.

Original: Figure 3(b)shows that DF enjoys a faster improvement in RMSE with the increasing of M .
Revised: Figure 3(b) shows that DF enjoys a faster improvement in RMSE with the increasing of M .

Original: More specifically, DF with M = 20 outperforms RF with M = 500 .
Revised: More specifically, DF with M = 20 outperforms RF with M = 500 .

Original: This experimental result matches our theoreticalanalysis in Theorem 4 that DF has a faster convergence rate w.r.t.
Revised: This experimental result matches our theoretical result in Theorem 4 that DF has a faster convergence rate w.r.t.

Original: the number of trees M .
Revised: the number of trees M .

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we analysis DF with its individual trees using CART-split criterion, narrowing thegap between theoretical and empirical studies on DF.

Original: Theoretically, when the individual trees areconsistent, the corresponding deep forest is universal consistent like RF, but practically, we often setthe trees to be fully grown, which leads to successful applications of deep forests.

Original: We further analyzedeep forest when the individual trees are inconsistent, and prove that a two-layer deep forest has afaster convergence rate w.r.t.
Revised: In this paper, we prove that a two-layer deep forest has a faster convergence rate w.r.t.

Original: the number of component trees M than random forest.
Revised: the number of component trees M than random forest.

Revised: This work provides the first theoretical analysis of the feature concatenation (PreConc) operation which is crucial for feature transformation in deep forests, although based on a very simplified structure where the concatenation of multiple random forests’ predictions and completely-random forests’ predictions in each layer of deep forest has not been taken into account.

Revised: On the one hand, this paper focuses on the asymptotic consistency of deep forests, so the result is strictly true only when the number of samples tends to infinity.

Revised: As the generalization analysis of deep forests with finite samples, we leave it to future work.

Revised: On the other hand, the two assumptions used in this paper have certain limitations.

Original: Our theoreticaland empirical results illustrate the effectiveness of the cascade forest structure in deep forest, showingthat depth is more powerful than width.
Revised: Experiments on simulation and real-world data sets show that our theoretical results are valid in many objective function classes other than Assumption 1. How to further relax the conditions in Assumption 1 will be an interesting problem. As for Assumption 2, it is still not strictly verified. However, quantifying the trade-off between label information and partition randomness will be a very important topic in future work.

Revised: Acknowledgements

Revised: This research was supported by the NSFC (61921006) and the Collaborative Innovation Center of

Revised: Novel Software Technology and Industrialization.

Revised: The authors would like to thank the anonymous reviewers for constructive suggestions, as well as Jin-Hui Wu, Qin-Cheng Zheng and Peng Tan for helpful discussions.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section 7.•
Revised: [Yes] See Section 7.

Original:  Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.•
Revised: [No] The code and the data are proprietary.

Original:  Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes](c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] This is atheoretical work.(d)
Revised: [No] This is a theoretical work.

Original:  Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes]3.
Revised: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyper-parameters, how theywere chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyper-parameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [No] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [No] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [No] 
Revised: [No] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [No](b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [No] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [No] (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
Revised: [No] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [No]
Revised: [No]