Original: Depth is More Powerful than Width in Deep Forest
Revised: Depth is More Powerful than Width in Deep Forest

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: Abstract
Revised: Abstract

Original: Random Forest (RF) is an ensemble learning algorithm proposed by Breiman[2001] that constructs a large number of randomized decision trees individuallyand aggregates their predictions by naive averaging.
Revised: Random Forest (RF) is an ensemble learning algorithm proposed by Breiman[2001] that constructs a large number of randomized decision trees individuallyand aggregates their predictions by naive averaging.

Original: Zhou and Feng [2019] furtherpropose Deep Forest (DF) algorithm with multi-layer feature transformation, whichsignificantly outperforms single-layer random forest in various application fields.
Revised: Zhou and Feng [2019] furtherpropose Deep Forest (DF) algorithm with multi-layer feature transformation, whichsignificantly outperforms single-layer random forest in various application fields.

Original: Despite its great successes, little is known about the mathematical properties of thecascade structure.
Revised: Despite its great successes, little is known about the mathematical properties of thecascade structure.

Original: In this paper, we analyze the influence of depth and width on theconsistency of cascade forest.
Revised: In this paper, we analyze the influence of depth and width on theconsistency of cascade forest.

Original: Especially when the individual tree is inconsistent (asin practice, the individual tree is often set to be fully grown, i.e., there is only onesample at each leaf node), we find that the convergence rate of two-layer DF w.r.t.the number of trees M can reach O (1 /M 2 ) under some mild conditions, whilethe convergence rate of single-layer RF is O (1 /M ) .
Revised: Especially when the individual tree is inconsistent (asin practice, the individual tree is often set to be fully grown, i.e., there is only onesample at each leaf node), we find that the convergence rate of two-layer DF w.r.t.the number of trees M can reach O (1 /M 2 ) under some mild conditions, whilethe convergence rate of single-layer RF is O (1 /M ) .

Original: Therefore, learning decisiontrees in the “deep” layer will be more powerful than learning in the “shallow” layer.
Revised: Therefore, learning decisiontrees in the “deep” layer will be more powerful than learning in the “shallow” layer.

Original: Experiments further confirm the theoretical advantages.
Revised: Experiments further confirm the theoretical advantages.

Original:  Introduction
Revised:  Introduction

Original: Random forest (RF) [Breiman, 2001] is a state-of-art classification and regression algorithm thatconstructs a number of randomized individual decision trees during a parallel training phase andpredicts by naive averaging the results.
Revised: Random forest (RF) [Breiman, 2001] is a state-of-art classification and regression algorithm thatconstructs a number of randomized individual decision trees during a parallel training phase andpredicts by naive averaging the results.

Original: Recently, Zhou and Feng [2019] propose Deep Forest(DF) algorithm with multi-layer feature transformation to investigate the possibility of tree-basedrepresentation learning.
Revised: Recently, Zhou and Feng [2019] propose Deep Forest(DF) algorithm with multi-layer feature transformation to investigate the possibility of tree-basedrepresentation learning.

Original: Benefiting from the layer-by-layer feature transformation in cascade structure(depth), DF outperforms various single-layer forest algorithms [Breiman, 2001, Friedman, 2001,
Revised: Benefiting from the layer-by-layer feature transformation in cascade structure(depth), DF outperforms various single-layer forest algorithms [Breiman, 2001, Friedman, 2001,

Original: Geurts et al., 2006, Chen and Guestrin, 2016] in empirical study and have been involved in realapplications such as medicine [Su et al., 2019], computer vision [Zhou et al., 2019], remote sensing[Boualleg et al., 2019], financial fraud detection [Zhang et al., 2019] etc.
Revised: Geurts et al., 2006, Chen and Guestrin, 2016] in empirical study and have been involved in realapplications such as medicine [Su et al., 2019], computer vision [Zhou et al., 2019], remote sensing[Boualleg et al., 2019], financial fraud detection [Zhang et al., 2019] etc.

Original: Numerous variants havebeen extended to various tasks and meet with remarkable success in Utkin and Ryabinin [2018, 2019],
Revised: Numerous variants havebeen extended to various tasks and meet with remarkable success in Utkin and Ryabinin [2018, 2019],

Original: Yang et al.
Revised: Yang et al.

Original: [2020], Wang et al.
Revised: [2020], Wang et al.

Original: There are also variants aiming at improving performance andreduce computational cost [Pang et al., 2020, Chen et al., 2021].
Revised: There are also variants aiming at improving performance andreducing computational cost [Pang et al., 2020, Chen et al., 2021].

Original: Empirical successes have attracted attention to the theoretical analysis of DF.
Revised: Empirical successes have attracted attention to the theoretical analysis of DF.

Original: Since its cascadestructure transforms features layer by layer, and each layer consists of complex non-parametricrandom forest estimators, analyzing the impact of the new features extracted from these estimatorson generalization performance is important.
Revised: Since its cascadestructure transforms features layer by layer, and each layer consists of complex non-parametricrandom forest estimators, analyzing the impact of the new features extracted from these estimatorson generalization performance is important.

Original: To start theoretical analysis on DF, Lyu et al.
Revised: To start theoretical analysis on DF, Lyu et al.

Original: [2019]prove a margin-based generalization bound for additive new features in DF.
Revised: [2019]prove a margin-based generalization bound for additive new features in DF.

Original: Then, Pang et al.
Revised: Then, Pang et al.

Original: [2020]prove an upper bound on the sample efficiency and inspire an efficient improvement for reducingcomputational cost of DF.
Revised: [2020]prove an upper bound on the sample efficiency and inspire an efficient improvement for reducingcomputational cost of DF.

Original: In the direction of consistency, Arnould et al.
Revised: In the direction of consistency, Arnould et al.

Original: [2021] provide tight lowerand upper bounds on the excess risk of a shallow centered random tree network, which leveragesthe new features to improve the performance of a single centered random tree.
Revised: [2021] provide tight lowerand upper bounds on the excess risk of a shallow centered random tree network, which leveragesthe new features to improve the performance of a single centered random tree.

Original: Assuming that theoriginal and new features are separated and independently used in two stages, Lyu et al.
Revised: Assuming that theoriginal and new features are separated and independently used in two stages, Lyu et al.

Original: [2022] prove that the new features based on prediction are easy to cause overfitting risk, and propose to use theinteraction of decision rules to alleviate it.
Revised: [2022] prove that the new features based on prediction are easy to cause overfitting risk, and propose to use theinteraction of decision rules to alleviate it.

Original: However, previous theoretical studies often choose to ignore the impact of the new features on theforest generation.
Revised: However, previous theoretical studies often choose to ignore the impact of the new features on theforest generation.

Original: For example, Lyu et al.
Revised: For example, Lyu et al.

Original: [2019, 2022], Pang et al.
Revised: [2019, 2022], Pang et al.

Original: [2020] assume that the forestestimator in each layer is a black-box model and Arnould et al.
Revised: [2020] assume that the forestestimator in each layer is a black-box model and Arnould et al.

Original: [2021] simplify that there is only asingle centered random tree in each layer.
Revised: [2021] simplify that there is only asingle centered random tree in each layer.

Original: Such strong assumptions are in favor of theoretical analysis,but widen the gap between the theoretical results and practical deep forest architecture.
Revised: Such strong assumptions are in favor of theoretical analysis,but widen the gap between the theoretical results and practical deep forest architecture.

Original: The difficultyin properly analyzing DF can be explained by the subtle combination of different techniques.
Revised: The difficultyin properly analyzing DF can be explained by the subtle combination of different techniques.

Original: Amongthe complex architecture in DF, both prediction concatenation (PreConc)
Revised: Amongthe complex architecture in DF, both prediction concatenation (PreConc)

Original: [Lyu et al., 2019] and theclassification and regression trees (CART)-split criterion [Breiman et al., 1984] play a critical role.
Revised: [Lyu et al., 2019] and theclassification and regression trees (CART)-split criterion [Breiman et al., 1984] play a critical role.

Original: Therefore, in order to open the black box of DF, we need to analyze the properties of the new featureand study how CART generates and utilizes it.
Revised: Therefore, in order to open the black box of DF, we need to analyze the properties of the new featureand study how CART generates and utilizes it.

Original: Contributions.
Revised: Contributions.

Original: In this paper, we try to narrow the gap between theoretical and empirical studies on
Revised: In this paper, we try to narrow the gap between theoretical and empirical studies on

Original: DF.
Revised: DF.

Original: We compare the advantages of depth (cascade structure with PreConc in deep forest) over width(bagging of trees in random forest) in the scope of L 22 -consistency, based on the assumptions of theadditive regression functions and uniform distribution over input space.
Revised: We compare the advantages of depth (cascade structure with PreConc in deep forest) over width(bagging of trees in random forest) in the scope of L 22 -consistency, based on the assumptions of theadditive regression functions and uniform distribution over input space.

Original: The main contributions canbe summarized as follows:
Revised: The main contributions canbe summarized as follows:

Original: • We are the first to establish a theoretical analysis of the priority of the new feature under theCART-split criterion in DF, which means the trees in the second layer and beyond tend to primarilychoose the new feature to split.
Revised: • We are the first to establish a theoretical analysis of the priority of the new feature under theCART-split criterion in DF, which means the trees in the second layer and beyond tend to primarilychoose the new feature to split.

Original: We further prove that the degree of this priority depends on theerror of the first-layer forest estimator.
Revised: We further prove that the degree of this priority depends on theerror of the first-layer forest estimator.

Original: • We prove the universal consistency of CART, random forest and deep forest when the total number
Revised: • We prove the universal consistency of CART, random forest and deep forest when the total number

Original: of leaves of individual trees is chosen properly.
Revised: of leaves of individual trees is chosen properly.

Original: Although in this setting, neither depth nor widthwill affect the consistency, experiments show that such trees have poor performance in practice.•
Revised: Although in this setting, neither depth nor widthwill affect the consistency, experiments show that such trees have poor performance in practice.•

Original: In the practical setting, when the individual trees are fully grown, we prove that
Revised: In the practical setting, when the individual trees are fully grown, we prove that

Original: the convergencerate of two-layer deep forest reach O (1 /M 2 ) w.r.t.
Revised: the convergencerate of two-layer deep forest reach O (1 /M 2 ) w.r.t.

Original: the increasing number of trees M , while thatof single-layer random forest is O (1 /M ) .
Revised: the increasing number of trees M , while that ofsingle-layer random forest is O (1 /M ) .

Original: This  reflects that depth is more powerful than width.
Revised: This result reflects that depth is more powerful than width.

Original: Agenda.
Revised: Agenda.

Original: The rest of this work is organized as follows: Section 3 shows the setting and notationsrelated to tree-based estimators in this work.
Revised: The rest of this work is organized as follows: Section 3 shows the setting and notationsrelated to tree-based estimators in this work.

Original: Section 4 recall the original deep forest architecture andsimplify it into a two-layer deep forest.
Revised: Section 4 recall the original deep forest architecture andsimplify it into a two-layer deep forest.

Original: Section 5 contains the properties of cascade structure withprediction concatenation.
Revised: Section 5 contains the properties of cascade structure withprediction concatenation.

Original: Section 6 proves the main result that depth is more powerful than width indeep forest architecture in the scope of consistency.
Revised: Section 6 proves the main result that depth is more powerful than width indeep forest architecture in the scope of consistency.

Original: Section 7 is devoted to the empirical studies ofdeep forest by verifying the theoretical results above.
Revised: Section 7 is devoted to the empirical studies ofdeep forest by verifying the theoretical results above.

Original: Section 8 concludes with future work.
Revised: Section 8 concludes with future work.

Original: Detailedproofs for theorems and propositions are given in the supplementary material due to page limitation.
Revised: Detailedproofs for theorems and propositions are given in the supplementary material due to page limitation.

Original:  Related work
Revised:  Related work

Original: Despite the widespread use and remarkable success of random forest in real world applications, thetheoretical properties of it are still not fully understood [Biau, 2012, Denil et al., 2014].
Revised: Despite the widespread use and remarkable success of random forest in real world applications, thetheoretical properties of it are still not fully understood [Biau, 2012, Denil et al., 2014].

Original: Breiman[2001] offers an upper bound on the generalization error of random forest in terms of correlation andaccuracy of the individual trees.
Revised: Breiman[2001] offers an upper bound on the generalization error of random forest in terms of correlation andaccuracy of the individual trees.

Original: Lin and Jeon [2006] establish a connection between random forestand a particular class of nearest neighbor predictors, which are further studied by Biau and Devroye[2010].
Revised: Lin and Jeon [2006] establish a connection between random forestand a particular class of nearest neighbor predictors, which are further studied by Biau and Devroye[2010].

Original: Meinshausen [2006] studies the consistency of the quantile random forest for regression.
Revised: Meinshausen [2006] studies the consistency of the quantile random forest for regression.

Original: Inrecent years, various theoretical works [Biau, 2012,
Revised: Inrecent years, various theoretical works [Biau, 2012,

Original: Biau et al., 2008, Ishwaran and Kogalur, 2010,
Revised: Biau et al., 2008, Ishwaran and Kogalur, 2010,

Original: Genuer, 2012, Zhu et al., 2015] have been performed, analyzing the consistency of various simplifiedforests, and moving ever closer to practice.
Revised: Genuer, 2012, Zhu et al., 2015] have been performed, analyzing the consistency of various simplifiedforests, and moving ever closer to practice.

Original: Denil et al.
Revised: Denil et al.

Original: [2014] narrows the gap between theoryand practice of random forests for regression.
Revised: [2014] narrows the gap between theoryand practice of random forests for regression.

Original: Scornet et al.
Revised: Scornet et al.

Original: [2015] prove the first L 22 -consistencyof Breiman’s original random forest with CART-split criterion based on the assumptions of theadditive regression functions and uniform distribution over input space.
Revised: [2015] prove the first L 22 -consistencyof Breiman’s original random forest with CART-split criterion based on the assumptions of theadditive regression functions and uniform distribution over input space.

Original: Scornet [2016] prove thatinfinite forest consistency implies finite forest consistency.
Revised: Scornet [2016] prove thatinfinite forest consistency implies finite forest consistency.

Original: Gao and Zhou [2020] then present theconvergence rate of purely randomized trees and a simplified variant of Breiman’s original CARTtrees.
Revised: Gao and Zhou [2020] then present theconvergence rate of purely randomized trees and a simplified variant of Breiman’s original CARTtrees.

Original: In addition, another research route is the theory analysis of feature importance [Strobl et al.
Revised: In addition, another research route is the theory analysis of feature importance [Strobl et al.

Original: ,2007, Louppe et al., 2013, Arnould et al., 2022].
Revised: ,2007, Louppe et al., 2013, Arnould et al., 2022].

Original: Recently, Li et al.
Revised: Recently, Li et al.

Original: [2019] derive non-asymptoticlower and upper bounds on the expected bias of MDI importance for random forests.
Revised: [2019] derive non-asymptoticlower and upper bounds on the expected bias of MDI importance for random forests.

Original: Sutera et al.[2021] establish a connection between MDI importance of pure random forest and Shapley values.
Revised: Sutera et al.[2021] establish a connection between MDI importance of pure random forest and Shapley values.

Original:  Setting and notations
Revised:  Setting and notations

Original: We first describe the setting and notations related to tree-based estimators in this work.
Revised: We first describe the setting and notations related to tree-based estimators in this work.

Original: For the sakeof conciseness, we consider the regression setting.
Revised: For the sakeof conciseness, we consider the regression setting.

Original: Setting.
Revised: Setting.

Original: We consider a regression framework, where the training set S n = { ( x 1 , y 1 ) , . . .
Revised: We consider a regression framework, where the training set S n = { ( x 1 , y 1 ) , . . .

Original: , ( x n , y n ) }of [0 , 1] d × R -valued independent random variables distributed as the prototype sample ( x , y ) with
Revised: , ( x n , y n ) }of [0 , 1] d × R -valued independent random variables distributed as the prototype sample ( x , y ) with

Original: E [ y 2 ] < ∞ .
Revised: E [ y 2 ] < ∞ .

Original: This underlying distribution, characterized by the marginal distribution D X on [0 , 1] dand by the conditional distribution D Y|X , can be written as
Revised: This underlying distribution, characterized by the marginal distribution D X on [0 , 1] dand by the conditional distribution D Y|X , can be written as

Original: [Equation]
Revised: [Equation]

Original: where f ( x ) =
Revised: where f ( x ) =

Original: E [ y | x ] is the conditional expectation of y given x , and ϵ is a noise satisfying E
Revised: E [ y | x ] is the conditional expectation of y given x , and ϵ is a noise satisfying E

Original: [ ϵ ] = 

Revised: [ ϵ ] =

Original: and Var[ ϵ ] < σ 2 .
Revised: and Var[ ϵ ] < σ 2 .

Original: The task considered in this paper is to output a randomizedestimator h n ( · , Θ , S n ): [0 , 1] d → R where Θ is a random variable that accounts for the randomization procedure and independent of the training set S n .
Revised: The task considered in this paper is to output a randomizedestimator h n ( · , Θ , S n ): [0 , 1] d → R where Θ is a random variable that accounts for the randomization procedure and independent of the training set S n .

Original: To simplify notation, we denoteh n ( x , Θ) =
Revised: To simplify notation, we denoteh n ( x , Θ) =

Original: h n ( x , Θ , S n ) .
Revised: h n ( x , Θ , S n ) .

Original: The quality of a randomized estimator h n is measured by its L 2 risk
Revised: The quality of a randomized estimator h n is measured by its L 2 risk

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken with respect to ( x , Θ) , conditionally on S n .
Revised: where the expectation is taken with respect to ( x , Θ) , conditionally on S n .

Original: As the training data size nincreases, we get a sequence of estimators { h 1 , h 2 , . .
Revised: As the training data size nincreases, we get a sequence of estimators { h 1 , h 2 , . .

Original: h n , . . .
Revised: h n , . . .

Original: A sequence of estimators { h n } ∞ n =1is said to be consistent if R ( h n ) → 0 as n → ∞ .
Revised: A sequence of estimators { h n } ∞ n =1is said to be consistent if R ( h n ) → 0 as n → ∞ .

Original: Trees and forests.
Revised: Trees and forests.

Original: A random forest estimator h M,n ( x , Θ) outputs the average prediction over Mindividual randomized trees h n ( x , Θ j ) , ∀ j ∈
Revised: A random forest estimator h M,n ( x , Θ) outputs the average prediction over Mindividual randomized trees h n ( x , Θ j ) , ∀ j ∈

Original: [ M ] .
Revised: [ M ] .

Original: Here, [ M ] = { 1 , 2 , . . .
Revised: Here, [ M ] = { 1 , 2 , . . .

Original: , M } denotes the indexesof all individual randomized trees, where Θ 1 , . . .
Revised: , M } denotes the indexesof all individual randomized trees, where Θ 1 , . . .

Original: , Θ M are distributed identically and independentlyand denoted by a generic random variable Θ .
Revised: , Θ M are distributed identically and independentlyand denoted by a generic random variable Θ .

Original: The random variable Θ can be used to sample thetraining set or select the candidate dimensions or positions for splitting.
Revised: The random variable Θ can be used to sample thetraining set or select the candidate dimensions or positions for splitting.

Original: Specifically, a recursivepartition Π of [0 , 1] d is built by performing successive axis-aligned splits according to Θ :
Revised: Specifically, a recursivepartition Π of [0 , 1] d is built by performing successive axis-aligned splits according to Θ :

Original: [Equation]
Revised: [Equation]

Original: where C Π ,n ( x ) is the cell of the tree partition containing x and N n ( C Π ,n ( x )) is the number oftraining samples falling into C Π ,n ( x ) with convention that the estimation equals to zero if the cell
Revised: where C Π ,n ( x ) is the cell of the tree partition containing x and N n ( C Π ,n ( x )) is the number oftraining samples falling into C Π ,n ( x ) with convention that the estimation equals to zero if the cell

Original: C Π ,n ( x ) is empty.
Revised: C Π ,n ( x ) is empty.

Original: These trees are combined to form a finite forest estimation:
Revised: These trees are combined to form a finite forest estimation:

Original: [Equation]
Revised: [Equation]

Original: By the law of large numbers, for any fixed x , conditionally on D n , the finite forest estimationconverges to the infinite forest estimation:
Revised: By the law of large numbers, for any fixed x , conditionally on D n , the finite forest estimationconverges to the infinite forest estimation:

Original: [Equation]
Revised: [Equation]

Original: When the number of samples tends to ∞ , we denote by h ∞ ( x , Θ) the randomized tree with infinitesamples and h M, ∞ ( x , Θ) the random forest with M trees and infinite samples.
Revised: When the number of samples tends to ∞ , we denote by h ∞ ( x , Θ) the randomized tree with infinitesamples and h M, ∞ ( x , Θ) the random forest with M trees and infinite samples.

Original:  Deep forest
Revised:  Deep forest

Original: We recall the original deep forest algorithm in Section 4.1, and describe the simplified two-layer deepforest algorithm in Section 4.2.
Revised: We recall the original deep forest algorithm in Section 4.1, and describe the simplified two-layer deepforest algorithm in Section 4.2.

Original: 4.1 Original deep forest
Revised: 4.1 Original deep forest

Original: Deep forest [Zhou and Feng, 2019] is a tree-based deep model made up of non-differentiableforest modules without back-propagation.
Revised: Deep forest [Zhou and Feng, 2019] is a tree-based deep model made up of non-differentiableforest modules without back-propagation.

Original: Each layer of DF is composed of two Breiman’s randomforest (RF) and two Completely-Random Forest (CRF).
Revised: Each layer of DF is composed of two Breiman’s randomforest (RF) and two Completely-Random Forest (CRF).

Original: The RF is composed of CARTs and CRF
Revised: The RF is composed of CARTs and CRF

Original: I npu t F e a t u r e
Revised: I npu t F e a t u r e

Original: s concatenate
Revised: s concatenate

Original: RF RF
Revised: RF RF

Original: CRF CRF
Revised: CRF CRF

Original: Layer
Revised: Layer

Original: Layer∙∙∙
Revised: Layer∙∙∙

Original: Layer 𝑁
Revised: Layer 𝑁

Original: ~ predictions
Revised: ~ predictions

Original: F i n a l
Revised: F i n a l

Original: P r e d i c t i o n
Revised: P r e d i c t i o n

Original: RF RF
Revised: RF RF

Original: CRF CRF
Revised: CRF CRF

Original: RF RF
Revised: RF RF

Original: CRF CRF
Revised: CRF CRF

Original: N e w
Revised: N e w

Original: F e a t u r e s is composed of pure random trees.
Revised: F e a t u r e s is composed of pure random trees.

Original: In the regression setting, each forest of each layer outputs aprediction for any query point x , corresponding to the average response in the leaf node (cell)containing x .
Revised: In the regression setting, each forest of each layer outputs aprediction for any query point x , corresponding to the average response in the leaf node (cell)containing x .

Original: At a given layer, the predictions of all forests of this layer are concatenated togetherwith raw features.
Revised: At a given layer, the predictions of all forests of this layer are concatenated togetherwith raw features.

Original: This prediction concatenation process is called PreConc, which is repeated foreach layer until the best layer and construct a deep forest.
Revised: This prediction concatenation process is called PreConc, which is repeated foreach layer until the best layer and construct a deep forest.

Original: For an overview of the architecture of deepforest, we refer readers to the work
Revised: For an overview of the architecture of deepforest, we refer readers to the work

Original: [Zhou and Feng, 2017, 2019].
Revised: [Zhou and Feng, 2017, 2019].

Original: 4.2 Simplified two-layer deep forest
Revised: 4.2 Simplified two-layer deep forest

Original: In order to narrow the gap between theoretical and empirical studies of deep forest, we retain CARTsplit criterion and PreConc when simplifying the model.
Revised: In order to narrow the gap between theoretical and empirical studies of deep forest, we retain CARTsplit criterion and PreConc when simplifying the model.

Original: We define a simplified two-layer deep forest,each layer of it is composed of one Breiman’s random forest.
Revised: We define a simplified two-layer deep forest,each layer of it is composed of one Breiman’s random forest.

Original: We denote by h (1) M,n ( x , Θ , S n ) thefirst-layer forest estimator and h (2) M,n ([ x , h ] , Θ , S n ) the second-layer forest estimator, where [ x , h ] isthe concatenation of the raw features x and the new feature h .
Revised: We denote by h (1) M,n ( x , Θ , S n ) thefirst-layer forest estimator and h (2) M,n ([ x , h ] , Θ , S n ) the second-layer forest estimator, where [ x , h ] isthe concatenation of the raw features x and the new feature h .

Original: Then we can define the two-layer deepforest as follows:
Revised: Then we can define the two-layer deepforest as follows:

Original: [Equation]
Revised: [Equation]

Original: The complete algorithm is shown in Algorithm 1.
Revised: The complete algorithm is shown in Algorithm 1.

Original: This algorithm has three parameters: m try ∈{ 1 , . . .
Revised: This algorithm has three parameters: m try ∈{ 1 , . . .

Original: , d } is the number of pre-selected directions for splitting, a n ∈ { 1 , . . .
Revised: , d } is the number of pre-selected directions for splitting, a n ∈ { 1 , . . .

Original: , n } is the number ofsamples in each tree, t n ∈ { 1 , . .
Revised: , n } is the number ofsamples in each tree, t n ∈ { 1 , . .

Original: . , a n } is the number of leaves in each tree.
Revised: . , a n } is the number of leaves in each tree.

Original: In the default procedure,the parameters are set as follows: m try is set to d .
Revised: In the default procedure,the parameters are set as follows: m try is set to d .

Original: a n is set to n/k , where k is the k -fold crossvalidation in the Zhou and Feng [2019]’s deep forest.
Revised: a n is set to n/k , where k is the k -fold crossvalidation in the Zhou and Feng [2019]’s deep forest.

Original: t n = a n means we use fully grown CARTwhich is inconsistent in practice.
Revised: t n = a n means we use fully grown CARTwhich is inconsistent in practice.

Original: Notice that k can control the subsampling rate a n /n =
Revised: Notice that k can control the subsampling rate a n /n =

Original: 1 /k , whichis proved by Scornet et al.
Revised: 1 /k , whichis proved by Scornet et al.

Original: [2015] to be the key component for imposing tree diversity.
Revised: [2015] to be the key component for imposing tree diversity.

Original: Given the above parameters, the most basic part of Algorithm 1 is the training process of each CART.
Revised: Given the above parameters, the most basic part of Algorithm 1 is the training process of each CART.

Original: Let C be a cell and N n ( C ) be the number of data points falling in C .
Revised: Let C be a cell and N n ( C ) be the number of data points falling in C .

Original: A split in C is a pair ( j, z ) ,where j is a dimension in { 1 , . . .
Revised: A split in C is a pair ( j, z ) ,where j is a dimension in { 1 , . . .

Original: , d } and z is the position of the split along the j -th dimension in cell
Revised: , d } and z is the position of the split along the j -th dimension in cell

Original: C .
Revised: C .

Original: Let S C be the set of all possible splits in C .
Revised: Let S C be the set of all possible splits in C .

Original: The CART-split criterion [Breiman et al., 1984] takesthe form
Revised: The CART-split criterion [Breiman et al., 1984] takesthe form

Original: [Equation]
Revised: [Equation]

Original: where C L = { x ∈ C : x ( j ) <
Revised: where C L = { x ∈ C : x ( j ) <

Original: z } , C R = { x ∈ C : x ( j ) ≥ z } , and µ n ( C ) =
Revised: z } , C R = { x ∈ C : x ( j ) ≥ z } , and µ n ( C ) =

Original: 1 N n ( C ) (cid:80) i : x ∈
Revised: 1 N n ( C ) (cid:80) i : x ∈

Original: A y idenotes the average response in any cell C (resp.
Revised: A y idenotes the average response in any cell C (resp.

Original: µ n ( C L ) , µ n ( C R ) ), with the convention 0 / 

Revised: µ n ( C L ) , µ n ( C R ) ), with the convention 0 /

Original: = 0 .
Revised: = 0 .

Original: At each cell C , the best split ( j ∗ n , z ∗ n ) is selected by maximizing ˆ L n ( j, z ) over M try and S C , that is,
Revised: At each cell C , the best split ( j ∗ n , z ∗ n ) is selected by maximizing ˆ L n ( j, z ) over M try and S C , that is,

Original: [Equation]
Revised: [Equation]

Original: Require:
Revised: Require:

Original: A training set S n = { ( x 1 , y 1 ) , . . .
Revised: A training set S n = { ( x 1 , y 1 ) , . . .

Original: , ( x n , y n ) } , the number of trees 2 M , m try ∈ { 1 , . . .
Revised: , ( x n , y n ) } , the number of trees 2 M , m try ∈ { 1 , . . .

Original: , d } , a n ∈ { 1 , . . . , n } , t n ∈ { 1 , . .
Revised: , d } , a n ∈ { 1 , . . . , n } , t n ∈ { 1 , . .

Original: . , a n } and the query point x ∈ [0 , 1] d .
Revised: . , a n } and the query point x ∈ [0 , 1] d .

Original: Ensure: Two-layer deep forest ¯ h 2 M,n ( · ) =
Revised: Ensure: Two-layer deep forest ¯ h 2 M,n ( · ) =

Original: h (2) M,n ◦ h (1) M,n ( · ) and the prediction of x .
Revised: h (2) M,n ◦ h (1) M,n ( · ) and the prediction of x .

Original: 1: for layer ℓ in { 1 , 2 } do 2: if ℓ = 2 then 3:
Revised: 1: for layer ℓ in { 1 , 2 } do 2: if ℓ = 2 then 3:

Original: S n ← { ([ x 1 , h (1) M,n ( x 1 )] , y 1 ) , . . .
Revised: S n ← { ([ x 1 , h (1) M,n ( x 1 )] , y 1 ) , . . .

Original: , ([ x n , h (1) M,n ( x n )] , y n ) } .
Revised: , ([ x n , h (1) M,n ( x n )] , y n ) } .

Original: ▷ Prediction concatenation.
Revised: ▷ Prediction concatenation.

Original: 4: end if 5: for tree j ∈ { 1 , 2 , . .
Revised: 4: end if 5: for tree j ∈ { 1 , 2 , . .

Original: , M } do 6:
Revised: , M } do 6:

Original: Select a n data points, without replacement, uniformly in S n .
Revised: Select a n data points, without replacement, uniformly in S n .

Original: Set Π 0 = { [0 , 1] d } the partition associated with the root of the tree.
Revised: Set Π 0 = { [0 , 1] d } the partition associated with the root of the tree.

Original: For all 1 ≤ i ≤ a n , set Π i = ∅ .
Revised: For all 1 ≤ i ≤ a n , set Π i = ∅ .

Original: Set n nodes = 1 and level = 0 .
Revised: Set n nodes = 1 and level = 0 .

Original: 10: while n nodes <
Revised: 10: while n nodes <

Original: t n do 11: if Π level = ∅ then 12: level = level + 1 .
Revised: t n do 11: if Π level = ∅ then 12: level = level + 1 .

Original: 13: else 14:
Revised: 13: else 14:

Original: Let C be the first element in Π level .
Revised: Let C be the first element in Π level .

Original: 15: if C contains exactly one point then 16: Π level ← Π level \ { C } .
Revised: 15: if C contains exactly one point then 16: Π level ← Π level \ { C } .

Original: 17: Π level +1 ← Π level +1 ∪ { C } .
Revised: 17: Π level +1 ← Π level +1 ∪ { C } .

Original: 18: else 19:
Revised: 18: else 19:

Original: Select the best split ( j ∗ n , z ∗ n ) in C by optimizing the CART-split criterion along the dimension D in { 1 , . . .
Revised: Select the best split ( j ∗ n , z ∗ n ) in C by optimizing the CART-split criterion along the dimension D in { 1 , . . .

Original: , d } .
Revised: , d } .

Original: ▷ See details in Eq.
Revised: ▷ See details in Eq.

Original: Split cell C along D according to the best split.
Revised: Split cell C along D according to the best split.

Original: Call C L and C R .
Revised: Call C L and C R .

Original: 21: Π level ← Π level \ { C } .
Revised: 21: Π level ← Π level \ { C } .

Original: 22: Π level +1 ← Π level +1 ∪ { C L } ∪ { C R } .
Revised: 22: Π level +1 ← Π level +1 ∪ { C L } ∪ { C R } .

Original: 23: n nodes ← n nodes + 1 .
Revised: 23: n nodes ← n nodes + 1 .

Original: 24: end if 25: end if 26: end while 27:
Revised: 24: end if 25: end if 26: end while 27:

Original: Compute the predicted value h ( ℓ ) n ( x , Θ j , S n ) at the query point x equal to the average of the Y i ’s falling in the cell of x in partition Π level ∪ Π level +1 .
Revised: Compute the predicted value h ( ℓ ) n ( x , Θ j , S n ) at the query point x equal to the average of the Y i ’s falling in the cell of x in partition Π level ∪ Π level +1 .

Original: 28: end for 29:
Revised: 28: end for 29:

Original: Compute the random forest estimation h ( ℓ ) M,n ( x , Θ , S n ) at the query point x according to
Revised: Compute the random forest estimation h ( ℓ ) M,n ( x , Θ , S n ) at the query point x according to

Original: Eq.
Revised: Eq.

Original: 30: end for 31: Compute the two-layer deep forest estimation ¯ h 2 M,n ( x , Θ , S n ) at the query point x according to Eq.
Revised: 30: end for 31: Compute the two-layer deep forest estimation ¯ h 2 M,n ( x , Θ , S n ) at the query point x according to Eq.

Original:  Properties of cascade structure
Revised:  Properties of cascade structure

Original: In this section we show that the properties of the simplified deep forest enable us to analyze theinfluence of the concatenated new feature in deep forest and the local variation related to the empirical
Revised: In this section we show that the properties of the simplified deep forest enable us to analyze theinfluence of the concatenated new feature in deep forest and the local variation related to the empirical

Original: CART-split criterion.
Revised: CART-split criterion.

Original: We consider an additive regression model satisfying the following assumption:
Revised: We consider an additive regression model satisfying the following assumption:

Original: Assumption 1.
Revised: Assumption 1.

Original: The response y follows
Revised: The response y follows

Original: [Equation]
Revised: [Equation]

Original: where x = (cid:0) x (1) , . . .
Revised: where x = (cid:0) x (1) , . . .

Original: , x ( d ) (cid:1) is uniformly distributed over [0 , 1] d , ϵ is an independent centered
Revised: , x ( d ) (cid:1) is uniformly distributed over [0 , 1] d , ϵ is an independent centered

Original: Gaussian noise with finite variance σ 2 > 

Revised: Gaussian noise with finite variance σ 2 >

Original: and each component f j is continuous.
Revised: and each component f j is continuous.

Original: Stone [1985], Hastie and Tibshirani [1987] popularize these models, which decompose the regressionfunction as a sum of univariate functions.
Revised: Stone [1985], Hastie and Tibshirani [1987] popularize these models, which decompose the regressionfunction as a sum of univariate functions.

Original: Especially, Scornet et al.
Revised: Especially, Scornet et al.

Original: [2015] prove the consistency of
Revised: [2015] prove the consistency of

Original: Breiman’s original random forest under this assumption.
Revised: Breiman’s original random forest under this assumption.

Original: On this basis, we study the impact of thenew feature generated in the deep forest.
Revised: On this basis, we study the impact of thenew feature generated in the deep forest.

Original: To start with, we analyze the priority of the new features generated by the previous layer in theselection of splitting features under the CART-split criterion, in both infinite sample regime and finitesample regime respectively.
Revised: To start with, we analyze the priority of the new features generated by the previous layer in theselection of splitting features under the CART-split criterion, in both infinite sample regime and finitesample regime respectively.

Original: Proposition 1 (Priority of the new feature) .
Revised: Proposition 1 (Priority of the new feature) .

Original: Assume the data set follows Assumption 1 and thefirst-layer forest is consistent.
Revised: Assume the data set follows Assumption 1 and thefirst-layer forest is consistent.

Original: The following results hold for any CART in the second-layer forest.1.
Revised: The following results hold for any CART in the second-layer forest.1.

Original: In the infinite sample regime ( n = ∞ ), we consider a single second-layer CART h (2) ∞ ( · , Θ) .
Revised: In the infinite sample regime ( n = ∞ ), we consider a single second-layer CART h (2) ∞ ( · , Θ) .

Original: Allsplits in this CART are performed along the new feature only.2.
Revised: Allsplits in this CART are performed along the new feature only.2.

Original: In the finite sample regime ( n < ∞ ), we consider a single second-layer CART h (2) n ( · , Θ) .
Revised: In the finite sample regime ( n < ∞ ), we consider a single second-layer CART h (2) n ( · , Θ) .

Original: Fixk ∈ N ∗ and ξ, ρ > 0 .
Revised: Fixk ∈ N ∗ and ξ, ρ > 0 .

Original: Then, with probability ≥ 1 − ρ , for all n large enough, we have, the errorof the first-layer forest is bound by ξ .
Revised: Then, with probability ≥ 1 − ρ , for all n large enough, we have, the errorof the first-layer forest is bound by ξ .

Original: As a consequence, the first k splits ( j q,n ( x ) , 1 ≤ q ≤ k ) inthis CART are performed along the new feature only.
Revised: As a consequence, the first k splits ( j q,n ( x ) , 1 ≤ q ≤ k ) inthis CART are performed along the new feature only.

Original: Proof sketch.
Revised: Proof sketch.

Original: (P1.1).
Revised: (P1.1).

Original: In the infinite sample regime, the random forest estimation of the first layer haszero error.
Revised: In the infinite sample regime, the random forest estimation of the first layer haszero error.

Original: Therefore, the new feature of the second layer is the target function h (1) M, ∞ ( x ) =
Revised: Therefore, the new feature of the second layer is the target function h (1) M, ∞ ( x ) =

Original: f
Revised: f

Original: ( x ) .
Revised: ( x ) .

Original: Obviously, the new feature is the most informative dimension.
Revised: Obviously, the new feature is the most informative dimension.

Original: Therefore, when splitting in any cell,
Revised: Therefore, when splitting in any cell,

Original: CART algorithm will select the new feature as the splitting dimension.
Revised: CART algorithm will select the new feature as the splitting dimension.

Original: (P1.2).
Revised: (P1.2).

Original: In the finite sampleregime, the random forest estimation of the first layer is not precise.
Revised: In the finite sampleregime, the random forest estimation of the first layer is not precise.

Original: Therefore, there is an errorrelated to n between the new feature and the target function.
Revised: Therefore, there is an errorrelated to n between the new feature and the target function.

Original: Firstly, we prove that the distancebetween the theoretical ( n = ∞ ) and empirical ( n < ∞ ) first k splits of the CART algorithm isbounded by cξ with probability ≥ 1 − ρ , when n is large enough.
Revised: Firstly, we prove that the distancebetween the theoretical ( n = ∞ ) and empirical ( n < ∞ ) first k splits of the CART algorithm isbounded by cξ with probability ≥ 1 − ρ , when n is large enough.

Original: Connecting with the result oftheoretical split above, the proof is completed.
Revised: Connecting with the result oftheoretical split above, the proof is completed.

Original: Remark 1.
Revised: Remark 1.

Original: Proposition 1 shows that the trees in the second layer primarily choose the new feature tosplit and the degree of this priority depends on the error of the first-layer forest estimator.
Revised: Proposition 1 shows that the trees in the second layer primarily choose the new feature tosplit and the degree of this priority depends on the error of the first-layer forest estimator.

Original: This alsoreveals that the advantages of deep forest depend on the performance of the first layer.
Revised: This alsoreveals that the advantages of deep forest depend on the performance of the first layer.

Original: If the forestof the first layer does not return an estimation with noise reduction, the performance of deep forestcannot be further improved through cascade structure.
Revised: If the forestof the first layer does not return an estimation with noise reduction, the performance of deep forestcannot be further improved through cascade structure.

Original: This is consistent with the empirical results inprevious work
Revised: This is consistent with the empirical results inprevious work

Original: [Zhou and Feng, 2019, Arnould et al., 2021].
Revised: [Zhou and Feng, 2019, Arnould et al., 2021].

Original: In order to control the risk of deep forest, we need study the local variation property of the empirical
Revised: In order to control the risk of deep forest, we need study the local variation property of the empirical

Original: CART-split criterion.
Revised: CART-split criterion.

Original: For any cell C , the variation of regression function f ( x ) within C is defined as
Revised: For any cell C , the variation of regression function f ( x ) within C is defined as

Original: [Equation]
Revised: [Equation]

Original: Proposition 2 (Variation of f in the empirical cell) .
Revised: Proposition 2 (Variation of f in the empirical cell) .

Original: Assume that Assumption 1 holds and thefirst-layer forest is consistent.
Revised: Assume that Assumption 1 holds and thefirst-layer forest is consistent.

Original: The following results hold for any CART in the second-layer foresth (2) n ( · , Θ) .
Revised: The following results hold for any CART in the second-layer foresth (2) n ( · , Θ) .

Original: After splitting along the new feature, the CART will estimate the residual of the first-layerforest estimation.
Revised: After splitting along the new feature, the CART will estimate the residual of the first-layerforest estimation.

Original: Then for all ρ, ξ > 0 , there exists N ∈ N ∗ such that, for all n > N ,
Revised: Then for all ρ, ξ > 0 , there exists N ∈ N ∗ such that, for all n > N ,

Original: [Equation]
Revised: [Equation]

Original: Proof sketch.
Revised: Proof sketch.

Original: Firstly, we prove the variation of f ( x ) within the cell obtained by the theoretical
Revised: Firstly, we prove the variation of f ( x ) within the cell obtained by the theoretical

Original: CART-split criterion converges to zero.
Revised: CART-split criterion converges to zero.

Original: Secondly, we prove that the distance between the theoreticaland empirical first k splits of the CART convergence to zero.
Revised: Secondly, we prove that the distance between the theoreticaland empirical first k splits of the CART convergence to zero.

Original: Finally, we prove that the variation off ( x ) within the empirical cell is close to the theoretical cell.
Revised: Finally, we prove that the variation off ( x ) within the empirical cell is close to the theoretical cell.

Original: Proposition 2 shows that the variation of the regression function f ( x ) within a cell of a randomtree C Π ,n is small provided n is large enough, thereby forcing the approximation error of DF toasymptotically approach zero.
Revised: Proposition 2 shows that the variation of the regression function f ( x ) within a cell of a randomtree C Π ,n is small provided n is large enough, thereby forcing the approximation error of DF toasymptotically approach zero.

Original:  Depth is more powerful than width
Revised:  Depth is more powerful than width

Original: Our first result considers the t n < a n regime, where the number of samples in each leaf node tendsto ∞ as t n →
Revised: Our first result considers the t n < a n regime, where the number of samples in each leaf node tendsto ∞ as t n →

Original: ∞ and a n → ∞ .
Revised: ∞ and a n → ∞ .

Original: We prove that controlling the depth of the trees through the number of leaves t n is sufficient to achieve consistency of deep forest
Revised: We prove that controlling the depth of the trees through the number of leaves t n is sufficient to achieve consistency of deep forest

Original: Theorem 3 (Universal consistency) .
Revised: Theorem 3 (Universal consistency) .

Original: Let M ≥ 1 .
Revised: Let M ≥ 1 .

Original: Consider two-layer deep forest ¯ h 2 M,n givenby Eq.
Revised: Consider two-layer deep forest ¯ h 2 M,n givenby Eq.

Original: (6) and Breiman’s random forest h M,n given by Eq.
Revised: (6) and Breiman’s random forest h M,n given by Eq.

Original: (4) for the random CARTs satisfyinga n → ∞ , t n → ∞ and t n (log a n ) 9 /a n → 0 .
Revised: (4) for the random CARTs satisfyinga n → ∞ , t n → ∞ and t n (log a n ) 9 /a n → 0 .

Original: Then under the setting described in Section 3 andassume the data set follows Assumption 1,1.
Revised: Then under the setting described in Section 3 andassume the data set follows Assumption 1,1.

Original: [Scornet et al., 2015, Theorem 1] the Breiman’s random forest h 2 M,n is consistent for any
Revised: [Scornet et al., 2015, Theorem 1] the Breiman’s random forest h 2 M,n is consistent for any

Original: M ≥ 1 ,2.
Revised: M ≥ 1 ,2.

Original: the two-layer deep forest ¯ h M,n is consistent for any M ≥ 1 .
Revised: the two-layer deep forest ¯ h M,n is consistent for any M ≥ 1 .

Original: Proof sketch.
Revised: Proof sketch.

Original: (T3.1).
Revised: (T3.1).

Original: The universal consistency of Breiman’s random forest is proved by Scornetet al.
Revised: The universal consistency of Breiman’s random forest is proved by Scornetet al.

Original: (T3.2).
Revised: (T3.2).

Original: Firstly, we already know that the variation of the target regression function f ( x )within a cell of a randomized CART in the second-layer forest is small when n is large enough via
Revised: Firstly, we already know that the variation of the target regression function f ( x )within a cell of a randomized CART in the second-layer forest is small when n is large enough via

Original: Proposition 2.
Revised: Proposition 2.

Original: Similar as Scornet et al.
Revised: Similar as Scornet et al.

Original: [2015], we utilize Proposition 2 to control the approximationerror of the two-layer deep forest.
Revised: [2015], we utilize Proposition 2 to control the approximationerror of the two-layer deep forest.

Original: And the parameter t n allows us to control the size of the leaves of
Revised: And the parameter t n allows us to control the size of the leaves of

Original: CART, which allows us to have enough samples in each leaf node to smooth the impact of noise, soas to control the estimation error.
Revised: CART, which allows us to have enough samples in each leaf node to smooth the impact of noise, soas to control the estimation error.

Original: Connecting the approximation and estimation error, the consistencyof a CART of second-layer deep forest is proved.
Revised: Connecting the approximation and estimation error, the consistencyof a CART of second-layer deep forest is proved.

Original: The universal consistency can be proved via Biauet al.
Revised: The universal consistency can be proved via Biauet al.

Original: [2008, Proposition 1], which guarantees that the error of forest estimator is no more than twicethat of individual randomized CART.
Revised: [2008, Proposition 1], which guarantees that the error of forest estimator is no more than twicethat of individual randomized CART.

Original: Remark 2.
Revised: Remark 2.

Original: Notice that under this setting, random forest and even deep forest have no obviousadvantages over single CART in theory.
Revised: Notice that under this setting, random forest and even deep forest have no obviousadvantages over single CART in theory.

Original: When we use forests in practical, we do not choose to controlthe depth of the trees.
Revised: When we use forests in practical, we do not choose to controlthe depth of the trees.

Original: Empirical studies in Section 7.2 show that the forest with t n = a n alwaysoutperforms the forest with t n < a n .
Revised: Empirical studies in Section 7.2 show that the forest with t n = a n alwaysoutperforms the forest with t n < a n .

Original: Actually, the fully grown trees t n = a n is the setting close topractical forest algorithm.
Revised: Actually, the fully grown trees t n = a n is the setting close topractical forest algorithm.

Original: In order to deal with the t n = a n regime, we need to introduce an assumption first proposed by
Revised: In order to deal with the t n = a n regime, we need to introduce an assumption first proposed by

Original: Scornet et al.
Revised: Scornet et al.

Original: We denote by Z i = 1 ( x i ∈ C Π ,n ( x ))
Revised: We denote by Z i = 1 ( x i ∈ C Π ,n ( x ))

Original: the indicator that x i falls into the samecell as x in the random tree designed with D n and the random parameter Θ .
Revised: the indicator that x i falls into the samecell as x in the random tree designed with D n and the random parameter Θ .

Original: Z ′
Revised: Z ′

Original: j = 1 ( x i ∈
Revised: j = 1 ( x i ∈

Original: C Π ′ ( x ))is another independent indicator.
Revised: C Π ′ ( x ))is another independent indicator.

Original: Then, we define the correlation between these two indicatorsconditionally on y i , y j or not, respectivelyϕ i,j
Revised: Then, we define the correlation between these two indicatorsconditionally on y i , y j or not, respectivelyϕ i,j

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Assumption 2.
Revised: Assumption 2.

Original: Let Z i,j = ( Z i , Z j ) .
Revised: Let Z i,j = ( Z i , Z j ) .

Original: Then one of the following two conditions holds:
Revised: Then one of the following two conditions holds:

Original: One has
Revised: One has

Original: [Equation]
Revised: [Equation]

Original: There exists a constant C > 0 and a sequence ( γ n ) n → 0 such that, almost surely,
Revised: There exists a constant C > 0 and a sequence ( γ n ) n → 0 such that, almost surely,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: (A2.1.) means that the influence of two labels y i , y j on the probability of connection of two couples ofrandom points converge to zero as n → ∞ .
Revised: (A2.1.) means that the influence of two labels y i , y j on the probability of connection of two couples ofrandom points converge to zero as n → ∞ .

Original: (A2.2.) means that the correlation between the noise andthe probability of connection of two couples of random points vanishes quickly enough, as n → ∞ .
Revised: (A2.2.) means that the correlation between the noise andthe probability of connection of two couples of random points vanishes quickly enough, as n → ∞ .

Original: However, this assumption is too strong for the Breiman’s original random forest [Breiman, 2001].
Revised: However, this assumption is too strong for the Breiman’s original random forest [Breiman, 2001].

Original: Scornet et al.
Revised: Scornet et al.

Original: [2015] emphasize that they cannot know whether or not Assumption 2 is satisfied inrandom forest.
Revised: [2015] emphasize that they cannot know whether or not Assumption 2 is satisfied inrandom forest.

Original: In this paper, we recall this assumption and state that this assumption is mild for thesecond-layer forest estimation in deep forest algorithm.
Revised: In this paper, we recall this assumption and state that this assumption is mild for thesecond-layer forest estimation in deep forest algorithm.

Original: Since deep forest concatenates the prediction with the raw features as the input for the next layer,the label information is encoded into the new feature of second layer.
Revised: Since deep forest concatenates the prediction with the raw features as the input for the next layer,the label information is encoded into the new feature of second layer.

Original: In this way, the influence oftwo labels on the connection probability of this pair of samples tends to zero, so (A2.1.)
Revised: In this way, the influence oftwo labels on the connection probability of this pair of samples tends to zero, so (A2.1.)

Original: is mild forthe second-layer forest estimation.
Revised: is mild forthe second-layer forest estimation.

Original: From another point of view, Proposition 1, the priority of newfeature also shows that the partition can be independent of labels, and the information of new featureis enough to obtain appropriate partition results, so (A2.2.) is mild too.
Revised: From another point of view, Proposition 1, the priority of newfeature also shows that the partition can be independent of labels, and the information of new featureis enough to obtain appropriate partition results, so (A2.2.) is mild too.

Original: Theorem 4 (Depth is more powerful than width) .
Revised: Theorem 4 (Depth is more powerful than width) .

Original: Let M ≥ 1 .
Revised: Let M ≥ 1 .

Original: Consider two-layer deep forest ¯ h 2 M,ngiven by Eq.
Revised: Consider two-layer deep forest ¯ h 2 M,ngiven by Eq.

Original: (6) and Breiman’s random forest h M,n given by Eq.
Revised: (6) and Breiman’s random forest h M,n given by Eq.

Original: (4) for the random CARTs satisfyinga n → ∞ , t n → ∞ , t n = a n and a n log n/n → 0 .
Revised: (4) for the random CARTs satisfyinga n → ∞ , t n → ∞ , t n = a n and a n log n/n → 0 .

Original: Then under the setting described in Section 3 andassume the data set follows Assumption 1 and 2, the following results hold1.
Revised: Then under the setting described in Section 3 andassume the data set follows Assumption 1 and 2, the following results hold1.

Original: [Scornet et al., 2015, Theorem 2] [Scornet, 2016, Theorem 3]
Revised: [Scornet et al., 2015, Theorem 2] [Scornet, 2016, Theorem 3]

Original: The Breiman’s random foresth ∞ ,n is consistent, and for all M, n ∈ N ,
Revised: The Breiman’s random foresth ∞ ,n is consistent, and for all M, n ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: The two-layer deep forest ¯ h ∞ ,n is consistent, and for all M, n ∈ N , if ∆( f, C Π ,n ( x , Θ)) is small enough then
Revised: The two-layer deep forest ¯ h ∞ ,n is consistent, and for all M, n ∈ N , if ∆( f, C Π ,n ( x , Θ)) issmall enough, then

Original: [Equation]
Revised: [Equation]

Original: Proof sketch.
Revised: Proof sketch.

Original: (T4.1).
Revised: (T4.1).

Original: The consistency of the infinite Breiman’s random forest is proved by Scornetet al.
Revised: The consistency of the infinite Breiman’s random forest is proved by Scornetet al.

Original: And the convergence rate of the finite random forest with the number of trees M isproved by Scornet [2016].
Revised: And the convergence rate of the finite random forest with the number of trees M isproved by Scornet [2016].

Original: (T4.2).
Revised: (T4.2).

Original: Similar as Scornet et al.
Revised: Similar as Scornet et al.

Original: [2015], we recall Proposition 2 to controlthe approximation error of the two-layer deep forest.
Revised: [2015], we recall Proposition 2 to controlthe approximation error of the two-layer deep forest.

Original: Then the estimation error is controlled byforcing the subsampling rate a n /n to be o (1 / log n ) .
Revised: Then the estimation error is controlled byforcing the subsampling rate a n /n to be o (1 / log n ) .

Original: Different from the bagging-style mechanism inrandom forest, the residual-style mechanism shown in Proposition 2 makes the second-layer forest in
Revised: Different from the bagging-style mechanism inrandom forest, the residual-style mechanism shown in Proposition 2 makes the second-layer forest in

Original: DF can reuse the first-layer estimation and focus on the residual learning.
Revised: DF can reuse the first-layer estimation and focus on the residual learning.

Original: Remark 3.
Revised: Remark 3.

Original: In the t n = a n setting, Scornet et al.
Revised: In the t n = a n setting, Scornet et al.

Original: [2015] show that the sub-sampling rate a n /n is thekey component in random forest.
Revised: [2015] show that the sub-sampling rate a n /n is thekey component in random forest.

Original: Because the small rate ensures that query point x is connected withenough different data points through different trees, the convergence rate of RF is O (1 /M ) , w.r.t.
Revised: Because the small rate ensures that query point x is connected withenough different data points through different trees, the convergence rate of RF is O (1 /M ) , w.r.t.

Original: thenumber of trees M .
Revised: thenumber of trees M .

Original: Theorem 4 proves that, if the first layer forest can encode the regression functionf ( x ) into the new feature with noise reduction, the cascade structure in DF can further accelerate theconvergence.
Revised: Theorem 4 proves that, if the first layer forest can encode the regression functionf ( x ) into the new feature with noise reduction, the cascade structure in DF can further accelerate theconvergence.

Original: Because the second-layer forest estimates the residual of the first layer, the trees in eachlayer of forest are more different.
Revised: Because the second-layer forest estimates the residual of the first layer, the trees in eachlayer of forest are more different.

Original: As a result, the convergence rate of deep forest will be improved to
Revised: As a result, the convergence rate of deep forest will be improved to

Original: O (1 /M 2 ) .
Revised: O (1 /M 2 ) .

Original: This reflects that depth is more powerful than width.
Revised: This reflects that depth is more powerful than width.

Original:  Simulation experiments
Revised:  Simulation experiments

Original: 7.1 Priority of the new feature
Revised: 7.1 Priority of the new feature

Original: This experiment aims to verify the priority of the new feature in choosing which feature to split assuggested in Proposition 1.
Revised: This experiment aims to verify the priority of the new feature in choosing which feature to split assuggested in Proposition 1.

Original: We focus on a second layer decision tree built upon the first layer randomforest.
Revised: We focus on a second layer decision tree built upon the first layer randomforest.

Original: Since a regression forest has only one output dimension, there’s only one new feature for thesecond layer tree.
Revised: Since a regression forest has only one output dimension, there’s only one new feature for thesecond layer tree.

Original: More specifically, we count the maximum consecutive levels from root node thatuses the new feature to split, which we call effective depth for short.
Revised: More specifically, we count the maximum consecutive levels from root node thatuses the new feature to split, which we call effective depth for short.

Original: The synthetic data set is generated as y = f ( x ) + ϵ , where f ( x ) = 15 (cid:80) 1 ≤ j ≤ 5 x j , x is uniformlydistributed over [0 , 1] 5 , ϵ ∼ N (cid:0) 0 , σ 2 (cid:1) where σ = 0 .
Revised: The synthetic data set is generated as y = f ( x ) + ϵ , where [Equation] x is uniformly distributed over [0 , 1] 5 , ϵ ∼ N (cid:0) 0 , σ 2 (cid:1) where σ = 0 .

Original: We vary the number of training samples andthe number of trees in the first layer of forest, and report the average effective depth of 5 runs in Figure 2(a).
Revised: We vary the number oftraining samples and the number of trees in the first layer of forest, and report the average effectivedepth of 5 runs in Figure 2(a).

Original: It is easy to observe that no matter what the first layer’s setting is, the effective depth ofnew feature is at least 2 . 4 .
Revised: It is easy to observe that no matter what the first layer’s setting is, theeffective depth of new feature is at least 2 . 4 .

Original: That is to say, at the beginning of the growing of the second layer tree, CART will always choose the new feature to split.
Revised: That is to say, at the beginning of the growing of thesecond layer tree, CART will always choose the new feature to split.

Original: And we can see that with the increase of trees inthe first layer and training samples, the CART’s preference of new feature extend to deeper layers.
Revised: And we can see that with theincrease of trees in the first layer and training samples, the CART’s preference of new feature extendto deeper layers.

Original: Figure 2(b) further plots the effective depth against the predictive error of the first layer measured byroot mean square error (RMSE), showing that the better the first layer’s predictive performance, themore the new features are preferred in splitting. 
Revised: Figure 2(b) further plots the effective depth against the predictive error of the firstlayer measured by root mean square error (RMSE), showing that the better the first layer’s predictiveperformance, the more the new features are preferred in splitting. [Figure] [Figure]

Original: 7.2 Convergence rate w.r.t. the number of trees
Revised: 7.2 Convergence rate w.r.t. the number of trees

Original: To study the convergence rate of predictive error with respect to the number of trees in the wholemodel, we fix the number of training samples to be 50,000.
Revised: To study the convergence rate of predictive error with respect to the number of trees in the wholemodel, we fix the number of training samples to be 50,000.

Original: We compare a 2-layer deep forest (DF)with M trees each layer to a random forest (RF) with 2 M trees, and we adopt 3-fold split in training [Figure] [Figure] [Figure]
Revised: We compare a 2-layer deep forest (DF)with M trees each layer to a random forest (RF) with 2 M trees, and we adopt 3-fold split in trainingboth DF and RF to ensure fairness.

Original: both DF and RF to ensure fairness.

Original: The performance is measured by RMSE with respect to thenoise-free version of data generating function.
Revised: The performance is measured by RMSE with respect to thenoise-free version of data generating function.

Original: Since the training data are with noise, a fully growntree is inconsistent.
Revised: Since the training data are with noise, a fully growntree is inconsistent.

Original: In Figure 3(a), we set the minimum leaf size of the trees to be √ n , i.e., 233 in thecase when training data size is 50,000.
Revised: In Figure 3(a), we set the minimum leaf size of the trees to be √ n , i.e., 233 in thecase when training data size is 50,000.

Original: In Figure 3(b) the trees are fully grown with only one samplein each leaf.
Revised: In Figure 3(b) the trees are fully grown with only one samplein each leaf.

Original: We plot the average RMSE of 5 runs against the increasing number of trees, with thecolored band indicating the standard deviation.
Revised: We plot the average RMSE of 5 runs against the increasing number of trees, with thecolored band indicating the standard deviation.

Original: Theorem 3 reveals that when the component trees are consistent, random forest and deep forest are allconsistent.
Revised: Theorem 3 reveals that when the component trees areconsistent, random forest and deep forest are all consistent.

Original: However, the consistency analysis result cannot guarantee the finite sample performancein practice.
Revised: However, the consistency analysis resultcannot guarantee the finite sample performance in practice.

Original: Comparing Figure 3(a) and Figure 3(b), we can see that even though the training set is aslarge as 50,000, the performance of RF and DF using consistent trees is still much worse than usinginconsistent trees.
Revised: Comparing Figure 3(a) and Figure 3(b),we can see that even though the training set is as large as 50,000, the performance of RF and DFusing consistent trees is still much worse than using inconsistent trees.

Original: This observation confirms the efficacy of the default experimental setting of usingfully grown trees in RF and DF.
Revised: This observation confirmsthe efficacy of the default experimental setting of using fully grown trees in RF and DF.

Original: Figure 3(b) shows that DF enjoys a faster improvement in RMSE with the increasing of M .
Revised: Figure 3(b)shows that DF enjoys a faster improvement in RMSE with the increasing of M .

Original: Morespecifically, DF with M = 20 outperforms RF with M = 500 .
Revised: More specifically, DF with M = 20 outperforms RF with M = 500 .

Original: This experimental result matches ourtheoretical analysis in Theorem 4 that DF has a faster convergence rate w.r.t.
Revised: This experimental result matches our theoreticalanalysis in Theorem 4 that DF has a faster convergence rate w.r.t.

Original: the number of trees M .
Revised: the number of trees M .

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we analysis DF with its individual trees using CART-split criterion, narrowing thegap between theoretical and empirical studies on DF.
Revised: In this paper, we analysis DF with its individual trees using CART-split criterion, narrowing thegap between theoretical and empirical studies on DF.

Original: Theoretically, when the individual trees areconsistent, the corresponding deep forest is universal consistent like RF, but practically, we often setthe trees to be fully grown, which leads to successful applications of deep forests.
Revised: Theoretically, when the individual trees areconsistent, the corresponding deep forest is universal consistent like RF, but practically, we often setthe trees to be fully grown, which leads to successful applications of deep forests.

Original: We further analyzedeep forest when the individual trees are inconsistent, and prove that a two-layer deep forest has afaster convergence rate w.r.t.
Revised: We further analyzedeep forest when the individual trees are inconsistent, and prove that a two-layer deep forest has afaster convergence rate w.r.t.

Original: the number of component trees M than random forest.
Revised: the number of component trees M than random forest.

Original: Our theoreticaland empirical results illustrate the effectiveness of the cascade forest structure in deep forest, showingthat depth is more powerful than width.
Revised: Our theoreticaland empirical results illustrate the effectiveness of the cascade forest structure in deep forest, showingthat depth is more powerful than width.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section 7.•
Revised: [Yes] See Section 7.•

Original: Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.•
Revised: [No] The code and the data areproprietary.•

Original: Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes](c)
Revised: [Yes](c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [No] This is atheoretical work.(d)
Revised: [No] This is atheoretical work.(d)

Original: Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]2.
Revised: [Yes]2.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes](b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes]3.
Revised: [Yes]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?

Original: [Yes](b) Did you specify all the training details (e.g., data splits, hyper-parameters, how theywere chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyper-parameters, how theywere chosen)?

Original: [Yes](c)
Revised: [Yes]

Original:  Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [No]
Revised: [No]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [No]
Revised: [No]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [No] (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: [No](b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [No]
Revised: [No]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [No]
Revised: [No]