Original: Depth is More Powerful than Width with Feature Concatenation in Deep Forest
Revised: Depth is More Powerful than Width with Feature Concatenation in Deep Forest

Original: Abstract
Revised: Abstract

Original: Random Forest (RF) is an ensemble learning algorithm proposed by Breiman [1] that constructs a large number of randomized decision trees individually and aggregates their predictions by naive averaging.
Revised: Random Forest (RF) is an ensemble learning algorithm proposed by Breiman [1] that constructs a large number of randomized decision trees individually and aggregates their predictions by naive averaging.

Original: Zhou and Feng [2] further propose Deep
Revised: Zhou and Feng [2] further propose Deep

Original: Forest (DF) algorithm with multi-layer feature transformation, which significantly outperforms random forest in various application fields.
Revised: Forest (DF) algorithm with multi-layer feature transformation, which significantly outperforms random forest in various application fields.

Original: The prediction concatenation (PreConc) operation is crucial for the multi-layer feature transformation in deep forest, though little has been known about its theoretical property.
Revised: The prediction concatenation (PreConc) operation is crucial for the multi-layer feature transformation in deep forest, though little has been known about its theoretical property.

Original: In this paper, we analyze the influence of Preconc on the consistency of deep forest.
Revised: In this paper, we analyze the influence of Preconc on the consistency of deep forest.

Original: Especially when the individual tree is inconsistent (as in practice, the individual tree is often set to be fully grown, i.e., there is only one sample at each leaf node), we find that the convergence rate of two-layer DF w.r.t.
Revised: Especially when the individual tree is inconsistent (as in practice, the individual tree is often set to be fully grown, i.e., there is only one sample at each leaf node), we find that the convergence rate of two-layer DF w.r.t.

Original: the number of trees M can reach O (1 /M 2 ) under some mild conditions, while the convergence rate of RF is
Revised: the number of trees M can reach O (1 /M 2 ) under some mild conditions, while the convergence rate of RF is

Original: O (1 /M ) .
Revised: O (1 /M ) .

Original: Therefore, with the help of PreConc, DF with deeper layer will be more powerful than the shallower layer.
Revised: Therefore, with the help of PreConc, DF with deeper layer will be more powerful than the shallower layer.

Original: Experiments confirm theoretical advantages.
Revised: Experiments confirm theoretical advantages.

Original:  Introduction
Revised:  Introduction

Original: Random forest (RF) [1] is a state-of-art classification and regression algorithm that constructs a number of randomized individual decision trees during a parallel training phase and predicts by naive averaging the results.
Revised: Random forest (RF) [1] is a state-of-art classification and regression algorithm that constructs a number of randomized individual decision trees during a parallel training phase and predicts by naive averaging the results.

Original: Recently, Zhou and Feng [2] propose Deep Forest (DF) algorithm with multi-layer feature transformation to investigate the possibility of tree-based representation learning.
Revised: Recently, Zhou and Feng [2] propose Deep Forest (DF) algorithm with multi-layer feature transformation to investigate the possibility of tree-based representation learning.

Original: Benefiting from the multi-layer feature transformation in the cascade structure with prediction concatenation (PreConc), DF outperforms various tree-based algorithms [1, 3–5] in empirical study, and have been involved in real applications such as medicine [6], computer vision [7], remote sensing [8], financial fraud detection [9] etc.
Revised: Benefiting from the multi-layer feature transformation in the cascade structure with prediction concatenation (PreConc), DF outperforms various tree-based algorithms [1, 3–5] in empirical study, and have been involved in real applications such as medicine [6], computer vision [7], remote sensing [8], financial fraud detection [9] etc.

Original: Numerous variants have been extended to various tasks and meet with remarkable success in [10–13].
Revised: Numerous variants have been extended to various tasks and meet with remarkable success in [10–13].

Original: There are also variants [14, 15] aiming at improving performance and reducing computational cost.
Revised: There are also variants [14, 15] aiming at improving performance and reducing computational cost.

Original: Empirical successes have attracted attention to the theoretical analysis of DF.
Revised: Empirical successes have attracted attention to the theoretical analysis of DF.

Original: Since its PreConc operation transforms features layer by layer, and each layer consists of complex non-parametric random forest estimators, analyzing the impact of the new features extracted from these estimators on generalization performance is important.
Revised: Since its PreConc operation transforms features layer by layer, and each layer consists of complex non-parametric random forest estimators, analyzing the impact of the new features extracted from these estimators on generalization performance is important.

Original: To start theoretical analysis on DF, Lyu et al.
Revised: To start theoretical analysis on DF, Lyu et al.

Original: [16] prove a margin-based generalization bound for additive new features in DF.
Revised: [16] prove a margin-based generalization bound for additive new features in DF.

Original: Then, Pang et al.
Revised: Then, Pang et al.

Original: [14] prove an upper bound on the sample efficiency and inspire an efficient improvement for reducing computational cost of DF.
Revised: [14] prove an upper bound on the sample efficiency and inspire an efficient improvement for reducing computational cost of DF.

Original: In the direction of consistency, Arnould et al.
Revised: In the direction of consistency, Arnould et al.

Original: [17] provide tight lower and upper bounds on the excess risk of a shallow centered random tree network, which leverages the new features to improve the performance of a single centered random tree.
Revised: [17] provide tight lower and upper bounds on the excess risk of a shallow centered random tree network, which leverages the new features to improve the performance of a single centered random tree.

Original: Assuming that the original and new features are separated and independently used in two stages, Lyu et al.
Revised: Assuming that the original and new features are separated and independently used in two stages, Lyu et al.

Original: [18] prove that the new features based on prediction are easy to cause overfitting risk, and propose to use the interaction of decision rules to alleviate it.
Revised: [18] prove that the new features based on prediction are easy to cause overfitting risk, and propose to use the interaction of decision rules to alleviate it.

Original: However, previous theoretical studies often choose to ignore the impact of the new features on generating forests.
Revised: However, previous theoretical studies often choose to ignore the impact of the new features on generating forests.

Original: For example, recent works [16, 18, 14] assume that the forest estimator in each layer is a black-box model and Arnould et al.
Revised: For example, recent works [16, 18, 14] assume that the forest estimator in each layer is a black-box model and Arnould et al.

Original: [17] assume that there is only a single centered random tree in each layer.
Revised: [17] assume that there is only a single centered random tree in each layer.

Original: Such strong assumptions are in favor of theoretical analysis, but widen the gap between the theoretical results and practical deep forest architecture.
Revised: Such strong assumptions are in favor of theoretical analysis, but widen the gap between the theoretical results and practical deep forest architecture.

Original: To properly analyze DF, we must notice that, among the complex architecture in DF, both prediction concatenation (PreConc) [16] and the classification and regression trees (CART)-split criterion [19] play critical roles.
Revised: To properly analyze DF, we must notice that, among the complex architecture in DF, both prediction concatenation (PreConc) [16] and the classification and regression trees (CART)-split criterion [19] play critical roles.

Original: Therefore, in order to open the black box of DF, we need to study the subtle combination of different techniques, i.e., analyze the properties of the new feature and study how CART generates and utilizes it.
Revised: Therefore, in order to open the black box of DF, we need to study the subtle combination of different techniques, i.e., analyze the properties of the new feature and study how CART generates and utilizes it.

Original: Contributions.
Revised: Contributions.

Original: We compare the advantages of depth (cascade structure with PreConc in deep forest) over width (bagging of trees in random forest) in the scope of L 22 -consistency, based on the assumptions of the additive regression functions and uniform distribution over input space.
Revised: We compare the advantages of depth (cascade structure with PreConc in deep forest) over width (bagging of trees in random forest) in the scope of L 22 -consistency, based on the assumptions of the additive regression functions and uniform distribution over input space.

Original: The main contributions can be summarized as follows:
Revised: The main contributions can be summarized as follows:

Original: • We are the first to establish a consistency analysis of the prediction concatenation (PreConc) operation which is crucial for multi-layer feature transformation in deep forest, though based on a simplified version.
Revised: • We are the first to establish a consistency analysis of the prediction concatenation (PreConc) operation which is crucial for multi-layer feature transformation in deep forest, though based on a simplified version.

Original: • We prove the universal consistency of deep forest when the total number of leaves of individual trees is chosen properly.
Revised: • We prove the universal consistency of deep forest when the total number of leaves of individual trees is chosen properly.

Original: • In the practical setting, when the individual trees are fully grown, we prove that the convergence rate of two-layer deep forest reach O (1 /M 2 ) w.r.t.
Revised: • In the practical setting, when the individual trees are fully grown, we prove that the convergence rate of two-layer deep forest reach O (1 /M 2 ) w.r.t.

Original: the increasing number of trees M , while that of random forest is O (1 /M ) .
Revised: the increasing number of trees M , while that of random forest is O (1 /M ) .

Original: This result reflects that deep forest with deeper layer will be more powerful than shallower layer.
Revised: This result reflects that deep forest with deeper layer will be more powerful than shallower layer.

Original: Organization.
Revised: Organization.

Original: The rest of this work is organized as follows: Section 3 shows the setting and notations related to tree-based estimators in this work.
Revised: The rest of this work is organized as follows: Section 3 shows the setting and notations related to tree-based estimators in this work.

Original: Section 4 recalls the original deep forest architecture and simplifies it into a two-layer deep forest.
Revised: Section 4 recalls the original deep forest architecture and simplifies it into a two-layer deep forest.

Original: Section 5 contains the properties of cascade structure with prediction concatenation.
Revised: Section 5 contains the properties of cascade structure with prediction concatenation.

Original: Section 6 proves the main result that depth is more powerful than width in deep forest architecture in the scope of consistency.
Revised: Section 6 proves the main result that depth is more powerful than width in deep forest architecture in the scope of consistency.

Original: Section 7 is devoted to the empirical studies of deep forest by verifying the theoretical results above.
Revised: Section 7 is devoted to the empirical studies of deep forest by verifying the theoretical results above.

Original: Section 8 concludes with future work.
Revised: Section 8 concludes with future work.

Original: More experimental results and detailed proofs for theorems and propositions are given in the supplementary material due to page limitation.
Revised: More experimental results and detailed proofs for theorems and propositions are given in the supplementary material due to page limitation.

Original:  Related work
Revised:  Related work

Original: Despite the widespread use and remarkable success of random forest in real world applications, the theoretical properties of it are still not fully understood [20, 21].
Revised: Despite the widespread use and remarkable success of random forest in real world applications, the theoretical properties of it are still not fully understood [20, 21].

Original: Breiman [1] offers an upper bound on the generalization error of random forest in terms of correlation and accuracy of the individual trees.
Revised: Breiman [1] offers an upper bound on the generalization error of random forest in terms of correlation and accuracy of the individual trees.

Original: Lin and Jeon [22] establish a connection between random forest and a particular class of nearest neighbor predictors, which are further studied by Biau and Devroye [23].
Revised: Lin and Jeon [22] establish a connection between random forest and a particular class of nearest neighbor predictors, which are further studied by Biau and Devroye [23].

Original: Meinshausen [24] studies the consistency of the quantile random forest for regression.
Revised: Meinshausen [24] studies the consistency of the quantile random forest for regression.

Original: In recent years, various theoretical works [20, 25–28] have been performed, analyzing the consistency of various simplified forests, and moving ever closer to practice.
Revised: In recent years, various theoretical works [20, 25–28] have been performed, analyzing the consistency of various simplified forests, and moving ever closer to practice.

Original: Denil et al.
Revised: Denil et al.

Original: [21] narrows the gap between theory and practice of random forests for regression.
Revised: [21] narrows the gap between theory and practice of random forests for regression.

Original: Scornet et al.
Revised: Scornet et al.

Original: [29] prove the first L 22 -consistency of Breiman’s original random forest with CART-split criterion based on the assumptions of the additive regression functions and uniform distribution over input space.
Revised: [29] prove the first L 22 -consistency of Breiman’s original random forest with CART-split criterion based on the assumptions of the additive regression functions and uniform distribution over input space.

Original: Scornet [30] prove that infinite forest consistency implies finite forest consistency.
Revised: Scornet [30] prove that infinite forest consistency implies finite forest consistency.

Original: Gao and Zhou [31] then present the convergence rate of purely randomized trees and a simplified variant of Breiman’s original CART trees.
Revised: Gao and Zhou [31] then present the convergence rate of purely randomized trees and a simplified variant of Breiman’s original CART trees.

Original: Gao et al.
Revised: Gao et al.

Original: [32] further expand it to multi-class setting.
Revised: [32] further expand it to multi-class setting.

Original: In addition, another research route is the theory analysis of feature importance [33–35].
Revised: In addition, another research route is the theory analysis of feature importance [33–35].

Original: Recently, Li et al.
Revised: Recently, Li et al.

Original: [36] derive non-asymptotic lower and upper bounds on the expected bias of MDI importance for random forests.
Revised: [36] derive non-asymptotic lower and upper bounds on the expected bias of MDI importance for random forests.

Original: Sutera et al.
Revised: Sutera et al.

Original: [37] establish a connection between MDI importance of pure random forest and Shapley values.
Revised: [37] establish a connection between MDI importance of pure random forest and Shapley values.

Original: However, while deep forests further improve generalization performance, there is no theory to prove the advantages brought by depth.
Revised: However, while deep forests further improve generalization performance, there is no theory to prove the advantages brought by depth.

Original: Therefore, studying the influence of depth is the theoretical cornerstone for distinguishing deep forests from random forests.
Revised: Therefore, studying the influence of depth is the theoretical cornerstone for distinguishing deep forests from random forests.

Original: For example, in the well-known deep neural networks (DNNs), there are a lot of theoretical works to study the effect of depth and width on its representation ability and generalization performance, which show the theoretical advantages of deep neural networks over shallow neural networks [38–42].
Revised: For example, in the well-known deep neural networks (DNNs), there are a lot of theoretical works to study the effect of depth and width on its representation ability and generalization performance, which show the theoretical advantages of deep neural networks over shallow neural networks [38–42].

Original: These works all contribute to the understanding of deep learning and provide insight for designing algorithms.
Revised: These works all contribute to the understanding of deep learning and provide insight for designing algorithms.

Original:  Setting and notations
Revised:  Setting and notations

Original: We first describe the setting and notations related to tree-based estimators in this work.
Revised: We first describe the setting and notations related to tree-based estimators in this work.

Original: For the sake of conciseness, we consider the regression setting.
Revised: For the sake of conciseness, we consider the regression setting.

Original: Setting.
Revised: Setting.

Original: We consider a regression framework, where the training set S n = { ( x 1 , y 1 ) , . . .
Revised: We consider a regression framework, where the training set S n = { ( x 1 , y 1 ) , . . .

Original: , ( x n , y n ) }  of [0 , 1] d × R -valued independent random variables distributed as the prototype sample ( x , y ) with E [ y 2 ] < ∞ .
Revised: , ( x n , y n ) } consists of [0 , 1] d × R -valued independent random variables distributed as the prototype sample ( x , y ) with E [ y 2 ] < ∞ .

Original: This underlying distribution, characterized by the marginal distribution D X on [0 , 1] d and by the conditional distribution D Y|X , can be written as
Revised: This underlying distribution, characterized by the marginal distribution D X on [0 , 1] d and by the conditional distribution D Y|X , can be written as

Original: [Equation]
Revised: [Equation]

Original: where f ( x ) =
Revised: where f ( x ) =

Original: E [ y | x ] is the conditional expectation of y given x , and ϵ is a noise satisfying E
Revised: E [ y | x ] is the conditional expectation of y given x , and ϵ is a noise satisfying E

Original: [ ϵ ] = 

Revised: [ ϵ ] =

Original: and Var[ ϵ ] < σ 2 .
Revised: and Var[ ϵ ] < σ 2 .

Original: The task considered in this paper is to output a randomized estimator h n ( · , Θ , S n ): [0 , 1] d → R where Θ is a random variable that accounts for the randomization procedure and independent of the training set S n .
Revised: The task considered in this paper is to output a randomized estimator h n ( · , Θ , S n ): [0 , 1] d → R where Θ is a random variable that accounts for the randomization procedure and independent of the training set S n .

Original: To simplify notation, we denote h n ( x , Θ) =
Revised: To simplify notation, we denote h n ( x , Θ) =

Original: h n ( x , Θ , S n ) .
Revised: h n ( x , Θ , S n ) .

Original: The quality of a randomized estimator h n is measured by its L 2 risk
Revised: The quality of a randomized estimator h n is measured by its L 2 risk

Original: [Equation]
Revised: [Equation]

Original: ) where the expectation is taken with respect to ( x , Θ) , conditionally on S n .
Revised: where the expectation is taken with respect to ( x , Θ) , conditionally on S n .

Original: As the training data size n increases, we get a sequence of estimators { h 1 , h 2 , . .
Revised: As the training data size n increases, we get a sequence of estimators { h 1 , h 2 , . .

Original: h n , . . .
Revised: h n , . . .

Original: A sequence of estimators { h n } ∞ n = is said to be consistent if R ( h n ) → 0 as n → ∞ .
Revised: A sequence of estimators { h n } ∞ n = is said to be consistent if R ( h n ) → 0 as n → ∞ .

Original: Trees and forests.
Revised: Trees and forests.

Original: A random forest estimator h M,n ( x , Θ) outputs the average prediction over M individual randomized trees h n ( x , Θ j ) , ∀ j ∈
Revised: A random forest estimator h M,n ( x , Θ) outputs the average prediction over M individual randomized trees h n ( x , Θ j ) , ∀ j ∈

Original: [ M ] .
Revised: [ M ] .

Original: Here, [ M ] = { 1 , 2 , . . .
Revised: Here, [ M ] = { 1 , 2 , . . .

Original: , M } denotes the indexes of all individual randomized trees, where Θ 1 , . . .
Revised: , M } denotes the indexes of all individual randomized trees, where Θ 1 , . . .

Original: , Θ M are distributed identically and independently and denoted by a generic random variable Θ .
Revised: , Θ M are distributed identically and independently and denoted by a generic random variable Θ .

Original: The random variable Θ can be used to sample the training set or select the candidate dimensions or positions for splitting.
Revised: The random variable Θ can be used to sample the training set and select the candidate dimensions and positions for splitting.

Original: Specifically, a recursive partition Π of [0 , 1] d is built by performing successive axis-aligned splits according to Θ :
Revised: Specifically, a recursive partition Π of [0 , 1] d is built by performing successive axis-aligned splits according to Θ :

Original: [Equation]
Revised: [Equation]

Original: where C Π ,n ( x ) is the cell of the tree partition containing x and N n ( C Π ,n ( x )) is the number of training samples falling into C Π ,n ( x ) with convention that the estimation equals to zero if the cell
Revised: where C Π ,n ( x ) is the cell of the tree partition containing x and N n ( C Π ,n ( x )) is the number of training samples falling into C Π ,n ( x ) with convention that the estimation equals to zero if the cell

Original: C Π ,n ( x ) is empty.
Revised: C Π ,n ( x ) is empty.

Original: These trees are combined to form a finite forest estimation:
Revised: These trees are combined to form a finite forest estimation:

Original: [Equation]
Revised: [Equation]

Original: By the law of large numbers, for any fixed x , conditionally on D n , the finite forest estimation converges to the infinite forest estimation:
Revised: By the law of large numbers, for any fixed x , conditionally on D n , the finite forest estimation converges to the infinite forest estimation:

Original: [Equation]
Revised: [Equation]

Original: When the number of samples tends to ∞ , we denote by h ∞ ( x , Θ) the randomized tree with infinite samples and h M, ∞ ( x , Θ) the random forest with M trees and infinite samples.
Revised: When the number of samples tends to ∞ , we denote by h ∞ ( x , Θ) the randomized tree with infinite samples and h M, ∞ ( x , Θ) the random forest with M trees and infinite samples.

Original:  Deep forest
Revised:  Deep forest

Original: We recall the original deep forest algorithm in Section 4.1, and describe the simplified two-layer deep forest algorithm in Section 4.2.
Revised: We recall the original deep forest algorithm in Section 4.1, and describe the simplified two-layer deep forest algorithm in Section 4.2.

Original: I npu t F e a t u r e
Revised: I npu t F e a t u r e

Original: s concatenate
Revised: s concatenate

Original: RF RF
Revised: RF RF

Original: CRF CRF
Revised: CRF CRF

Original: Layer
Revised: Layer

Original: Layer∙∙∙
Revised: Layer∙∙∙

Original: Layer 𝑁
Revised: Layer 𝑁

Original: ~ predictions
Revised: ~ predictions

Original: F i n a l
Revised: F i n a l

Original: P r e d i c t i o n
Revised: P r e d i c t i o n

Original: RF RF
Revised: RF RF

Original: CRF CRF
Revised: CRF CRF

Original: RF RF
Revised: RF RF

Original: CRF CRF
Revised: CRF CRF

Original: N e w
Revised: N e w

Original: F e a t u r e s
Revised: F e a t u r e s

Original: 4.1 Original deep forest
Revised: 4.1 Original deep forest

Original: Deep forest [2] is a tree-based deep model made up of non-differentiable forest modules without back-propagation.
Revised: Deep forest [2] is a tree-based deep model made up of non-differentiable forest modules without back-propagation.

Original: Each layer of DF is composed of two Breiman’s random forest (RF) and two
Revised: Each layer of DF is composed of two Breiman’s random forest (RF) and two

Original: Completely-Random Forest (CRF).
Revised: Completely-Random Forest (CRF).

Original: The RF is composed of CARTs and CRF is composed of pure random trees.
Revised: The RF is composed of CARTs and CRF is composed of pure random trees.

Original: In the regression setting, each forest of each layer outputs a prediction for any query point x , corresponding to the average response in the leaf node (cell) containing x .
Revised: In the regression setting, each forest of each layer outputs a prediction for any query point x , corresponding to the average response in the leaf node (cell) containing x .

Original: At a given layer, the predictions of all forests of this layer are concatenated together with raw features.
Revised: At a given layer, the predictions of all forests of this layer are concatenated together with raw features.

Original: This prediction concatenation process is called PreConc, which is repeated for each layer until the best layer and construct a deep forest.
Revised: This prediction concatenation process is called PreConc, which is repeated for each layer until the best layer and construct a deep forest.

Original: For an overview of the architecture of deep forest, we refer readers to the work [43, 2].
Revised: For an overview of the architecture of deep forest, we refer readers to the work [43, 2].

Original: 4.2 Simplified two-layer deep forest
Revised: 4.2 Simplified two-layer deep forest

Original: In order to narrow the gap between theoretical and empirical studies of deep forest, we retain CARTsplit criterion and PreConc when simplifying the model.
Revised: In order to narrow the gap between theoretical and empirical studies of deep forest, we retain CARTsplit criterion and PreConc when simplifying the model.

Original: We define a simplified two-layer deep forest,  each layerof it is composed of one Breiman’s random forest.
Revised: We define a simplified two-layer deep forest, whose each layer is composed of one Breiman’s random forest.

Original: We denote by h (1) M,n ( x , Θ , S n ) the first-layer forest estimator and h (2) M,n ([ x , h ] , Θ , S n ) the second-layer forest estimator, where [ x , h ] is the concatenation of the raw features x and the new feature h .
Revised: We denote by h (1) M,n ( x , Θ , S n ) the first-layer forest estimator and h (2) M,n ([ x , h ] , Θ , S n ) the second-layer forest estimator, where [ x , h ] is the concatenation of the raw features x and the new feature h .

Original: Then we can define the two-layer deep forest as follows:
Revised: Then we can define the two-layer deep forest as follows:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: The complete algorithm is shown in Algorithm 1.
Revised: The complete algorithm is shown in Algorithm 1.

Original: This algorithm has three parameters: m try ∈ { 1 , . . .
Revised: This algorithm has three parameters: m try ∈ { 1 , . . .

Original: , d } is the number of pre-selected directions for splitting, a n ∈ { 1 , . . .
Revised: , d } is the number of pre-selected directions for splitting, a n ∈ { 1 , . . .

Original: , n } is the number of samples in each tree, t n ∈ { 1 , . .
Revised: , n } is the number of samples in each tree, t n ∈ { 1 , . .

Original: . , a n } is the number of leaves in each tree.
Revised: . , a n } is the number of leaves in each tree.

Original: In the default procedure, the parameters are set as follows: m try is set to d .
Revised: In the default procedure, the parameters are set as follows: m try is set to d .

Original: a n is set to n/k , where k is the k -fold cross-validation inthe Zhou and Feng [2]’s deep forest.
Revised: a n is set to n/k , where k is the k -fold crossvalidation in Zhou and Feng [2]’s deep forest.

Original: t n = a n means we use fully grown CART which is inconsistent in practice.
Revised: t n = a n means we use fully grown CART.

Original: Notice that k can control the subsampling rate a n /n = 1 /k , which is proved by Scornet et al.
Revised: Notice that k controls the subsampling rate a n /n = 1 /k , which is proved by Scornet et al.

Original: [29] to be the key component for imposing tree diversity.
Revised: [29] to be the key component for imposing tree diversity.

Original: Given the above parameters, the most basic part of Algorithm 1 is the training process of each CART.
Revised: Given the above parameters, the most basic part of Algorithm 1 is the training process of each CART.

Original: Let C be a cell and N n ( C ) be the number of data points falling in C .
Revised: Let C be a cell and N n ( C ) be the number of data points falling in C .

Original: A split in C is a pair ( j, z ) , where j is a dimension in { 1 , . . .
Revised: A split in C is a pair ( j, z ) , where j is a dimension in { 1 , . . .

Original: , d } and z is the position of the split along the j -th dimension in cell
Revised: , d } and z is the position of the split along the j -th dimension in cell

Original: C .
Revised: C .

Original: Let S C be the set of all possible splits in C .
Revised: Let S C be the set of all possible splits in C .

Original: The CART-split criterion [19] takes the formˆ
Revised: The CART-split criterion [19] takes the form

Original: [Equation]
Revised: [Equation]

Original: where C L = { x ∈ C : x ( j ) <
Revised: where C L = { x ∈ C : x ( j ) <

Original: z } , C R = { x ∈ C : x ( j ) ≥ z } , and µ n ( C ) =1 N n ( C
Revised: z } , C R = { x ∈ C : x ( j ) ≥ z } , and µ n ( C ) =

Original: [Equation]
Revised: [Equation]

Original: (cid:80) i : x ∈
Revised: (cid:80) [Equation]

Original: A y i denotes the average response in 
Revised: y i denotes the average response in any cell C (resp. µ

Original: Require:
Revised: Require:

Original: A training set S n = { ( x 1 , y 1 ) , . . .
Revised: A training set S n = { ( x 1 , y 1 ) , . . .

Original: , ( x n , y n ) } , the number of trees 2 M , m try ∈ { 1 , . . .
Revised: , ( x n , y n ) } , the number of trees 2 M , m try ∈ { 1 , . . .

Original: , d } , a n ∈ { 1 , . . . , n } , t n ∈ { 1 , . .
Revised: , d } , a n ∈ { 1 , . . . , n } , t n ∈ { 1 , . .

Original: . , a n } and the query point x ∈ [0 , 1] d .
Revised: . , a n } and the query point x ∈ [0 , 1] d .

Original: Ensure: Two-layer deep forest ¯ h 2 M,n ( · ) =
Revised: Ensure: Two-layer deep forest ¯ h 2 M,n ( · ) =

Original: h (2) M,n ◦ h (1) M,n ( · ) and the prediction of x .
Revised: h (2) M,n ◦ h (1) M,n ( · ) and the prediction of x .

Original: 1: for layer ℓ in { 1 , 2 } do 2: if ℓ = 2 then 3:
Revised: 1: for layer ℓ in { 1 , 2 } do 2: if ℓ = 2 then 3:

Original: S n ← { ([ x 1 , h (1) M,n ( x 1 )] , y 1 ) , . . .
Revised: S n ← { ([ x 1 , h (1) M,n ( x 1 )] , y 1 ) , . . .

Original: , ([ x n , h (1) M,n ( x n )] , y n ) } .
Revised: , ([ x n , h (1) M,n ( x n )] , y n ) } .

Original: ▷ Prediction concatenation.
Revised: ▷ Prediction concatenation.

Original: 4: end if 5: for tree j ∈ { 1 , 2 , . .
Revised: 4: end if 5: for tree j ∈ { 1 , 2 , . .

Original: , M } do 6:
Revised: , M } do 6:

Original: Select a n data points, without replacement, uniformly in S n .
Revised: Select a n data points, without replacement, uniformly in S n .

Original: Set Π 0 = { [0 , 1] d } the partition associated with the root of the tree.
Revised: Set Π 0 = { [0 , 1] d } the partition associated with the root of the tree.

Original: For all 1 ≤ i ≤ a n , set Π i = ∅ .
Revised: For all 1 ≤ i ≤ a n , set Π i = ∅ .

Original: Set n nodes = 1 and level = 0 .
Revised: Set n nodes = 1 and level = 0 .

Original: 10: while n nodes <
Revised: 10: while n nodes <

Original: t n do 11: if Π level = ∅ then 12: level = level + 1 .
Revised: t n do 11: if Π level = ∅ then 12: level = level + 1 .

Original: 13: else 14:
Revised: 13: else 14:

Original: Let C be the first element in Π level .
Revised: Let C be the first element in Π level .

Original: 15: if C contains exactly one point then 16: Π level ← Π level \ { C } .
Revised: 15: if C contains exactly one point then 16: Π level ← Π level \ { C } .

Original: 17: Π level +1 ← Π level +1 ∪ { C } .
Revised: 17: Π level +1 ← Π level +1 ∪ { C } .

Original: 18: else 19:
Revised: 18: else 19:

Original: Select the best split ( j ∗ n , z ∗ n ) in C by optimizing the CART-split criterion along the dimension D in { 1 , . . .
Revised: Select the best split ( j ∗ n , z ∗ n ) in C by optimizing the CART-split criterion along the dimension D in { 1 , . . .

Original: , d } .
Revised: , d } .

Original: ▷ See details in Eq.
Revised: ▷ See details in Eq.

Original: Split cell C along D according to the best split.
Revised: Split cell C along D according to the best split ( j ∗ n , z ∗ n ) .

Original: Call C L and C R .
Revised: Call C L and C R .

Original: 21: Π level ← Π level \
Revised: 21: Π level ← Π [Equation]

Original: [Equation]
Revised: [Equation]

Original: C
Revised: C

Original: [Equation]
Revised: [Equation]

Original: 22: Π 
Revised: 22: Π level

Original: [Equation]
Revised: [Equation]

Original: ←
Revised: ←

Original: [Equation]
Revised: [Equation]

Original: 23: n nodes ← n nodes + 1 .
Revised: 23: n nodes ← n nodes + 1 .

Original: 24: end if 25: end if 26: end while 27:
Revised: 24: end if 25: end if 26: end while 27:

Original: Compute the predicted value h ( ℓ ) n ( x , Θ j , S n ) at the query point x equal to the average of the Y i ’s falling in the cell of x in partition Π level ∪ Π level +1 .
Revised: Compute the predicted value h ( ℓ ) n ( x , Θ j , S n ) at the query point x equaling the average of the Y i ’s falling in the cell of x in partition Π level ∪ Π level +1 .

Original: 28: end for 29:
Revised: 28: end for 29:

Original: Compute the random forest estimation h ( ℓ ) M,n ( x , Θ , S n ) at the query point x according to
Revised: Compute the random forest estimation h ( ℓ ) M,n ( x , Θ , S n ) at the query point x according to

Original: Eq.
Revised: Eq.

Original: 30: end for 31: Compute the two-layer deep forest estimation ¯ h 2 M,n ( x , Θ , S n ) at the query point x according to Eq.
Revised: 30: end for 31: Compute the two-layer deep forest estimation ¯ h 2 M,n ( x , Θ , S n ) at the query point x according to Eq.

Original: At each cell C , the best split ( j ∗ n , z ∗ n ) is selected by maximizing ˆ L n ( j, z ) over M try and S C , that is,
Revised: At each cell C , the best split ( j ∗ n , z ∗ n ) is selected by maximizing ˆ L n ( j, z ) over M try and S C , that is,

Original: [Equation]
Revised: [Equation]

Original:  Properties of prediction concatenation
Revised:  Properties of prediction concatenation

Original: In this section we show that the properties of the simplified deep forest enable us to analyze the influence of the concatenated new feature in deep forest and the local variation related to the empirical
Revised: In this section we show that the properties of the simplified deep forest enable us to analyze the influence of the concatenated new feature in deep forest and the local variation related to the empirical

Original: CART-split criterion.
Revised: CART-split criterion.

Original: We consider an additive regression model satisfying the following assumption:
Revised: We consider an additive regression model satisfying the following assumption:

Original: Assumption 1.
Revised: Assumption 1.

Original: The response y follows
Revised: The response y follows

Original: [Equation]
Revised: [Equation]

Original: where x = (cid:0) x (1) , . . .
Revised: where x = (cid:0) x (1) , . . .

Original: , x ( d ) (cid:1) is uniformly distributed over [0 , 1] d , ϵ is an independent centered
Revised: , x ( d ) (cid:1) is uniformly distributed over [0 , 1] d , ϵ is an independent centered

Original: Gaussian noise with finite variance σ 2 > 

Revised: Gaussian noise with finite variance σ 2 >

Original: and each component f j is continuous.
Revised: and each component f j is continuous.

Original: Stone [44], Hastie and Tibshirani [45] popularize these models, which decompose the regression function as a sum of univariate functions.
Revised: Stone [44], Hastie and Tibshirani [45] popularize these models, which decompose the regression function as a sum of univariate functions.

Original: Especially, Scornet et al.
Revised: Especially, Scornet et al.

Original: [29] prove the consistency of
Revised: [29] prove the consistency of

Original: Breiman’s original random forest under this assumption.
Revised: Breiman’s original random forest under this assumption.

Original: On this basis, we study the impact of the new feature generated in the deep forest.
Revised: On this basis, we study the impact of the new feature generated in the deep forest.

Original: To start with, we analyze the priority of the new features generated by the previous layer in the selection of splitting features under the CART-split criterion, in both infinite sample regime and finite sample regime respectively.
Revised: To start with, we analyze the priority of the new features generated by the previous layer in the selection of splitting features under the CART-split criterion, in both infinite sample regime and finite sample regime respectively.

Original: Proposition 1 (Priority of the new feature) .
Revised: Proposition 1 (Priority of the new feature) .

Original: Assume the data set follows Assumption 1 and the first-layer forest is consistent.
Revised: Assume the data set follows Assumption 1 and the first-layer forest is consistent.

Original: The following results hold for any CART in the second-layer forest.
Revised: The following results hold for any CART in the second-layer forest.

Original: In the infinite sample regime ( n = ∞ ), we consider a single second-layer CART h (2) ∞ ( · , Θ) .
Revised: In the infinite sample regime ( n = ∞ ), we consider a single second-layer CART h (2) ∞ ( · , Θ) .

Original: All splits in this CART are performed along the new feature only.
Revised: All splits in this CART are performed along the new feature only.

Original: In the finite sample regime ( n < ∞ ), we consider a single second-layer CART h (2) n ( · , Θ) .
Revised: In the finite sample regime ( n < ∞ ), we consider a single second-layer CART h (2) n ( · , Θ) .

Original: Fix k ∈ N ∗ and ξ, ρ > 0 .
Revised: Fix k ∈ N ∗ and ξ, ρ > 0 .

Original: Then, with probability ≥ 1 − ρ , for all n large enough, we have, the error of the first-layer forest is bound by ξ .
Revised: Then, with probability ≥ 1 − ρ , for all n large enough, we have, the error of the first-layer forest is bound by ξ .

Original: As a consequence, the first k splits ( j q,n ( x ) , 1 ≤ q ≤ k ) in this CART are performed along the new feature only.
Revised: As a consequence, the first k splits ( j q,n ( x ) , 1 ≤ q ≤ k ) in this CART are performed along the new feature only.

Original: Proof sketch.
Revised: Proof sketch.

Original: (P1.1).
Revised: (P1.1).

Original: In the infinite sample regime, the random forest estimation of the first layer has zero error.
Revised: In the infinite sample regime, the random forest estimation of the first layer has zero error.

Original: Therefore, the new feature of the second layer is the target function h (1) M, ∞ ( x ) =
Revised: Therefore, the new feature of the second layer is the target function h (1) M, ∞ ( x ) =

Original: f
Revised: f

Original: ( x ) .
Revised: ( x ) .

Original: Obviously, the new feature is the most informative dimension.
Revised: Obviously, the new feature is the most informative dimension.

Original: Therefore, when splitting in any cell,
Revised: Therefore, when splitting in any cell,

Original: CART algorithm will select the new feature as the splitting dimension.
Revised: CART algorithm will select the new feature as the splitting dimension.

Original: (P1.2).
Revised: (P1.2).

Original: In the finite sample regime, the random forest estimation of the first layer is not precise.
Revised: In the finite sample regime, the random forest estimation of the first layer is not precise.

Original: Therefore, there is an error related to n between the new feature and the target function.
Revised: Therefore, there is an error related to n between the new feature and the target function.

Original: Firstly, we prove that the distance between the theoretical ( n = ∞ ) and empirical ( n < ∞ ) first k splits of the CART algorithm is bounded by cξ with probability ≥ 1 − ρ , when n is large enough.
Revised: Firstly, we prove that the distance between the theoretical ( n = ∞ ) and empirical ( n < ∞ ) first k splits of the CART algorithm is bounded by cξ with probability ≥ 1 − ρ , when n is large enough.

Original: Connecting with the result of theoretical split above, the proof is completed.
Revised: Connecting with the result of theoretical split above, the proof is completed.

Original: Remark 1.
Revised: Remark 1.

Original: Proposition 1 shows that the trees in the second layer primarily choose the new feature to split and the degree of this priority depends on the error of the first-layer forest estimator.
Revised: Proposition 1 shows that the trees in the second layer primarily choose the new feature to split and the degree of this priority depends on the error of the first-layer forest estimator.

Original: This also reveals that the advantages of deep forest depend on the performance of the first layer.
Revised: This also reveals that the advantages of deep forest depend on the performance of the first layer.

Original: If the forest of the first layer does not return an estimation with noise reduction, the performance of deep forest cannot be further improved through PreConc operation.
Revised: If the forest of the first layer does not return an estimation with noise reduction, the performance of deep forest cannot be further improved through PreConc operation.

Original: This is consistent with the empirical results in previous work [2, 17].
Revised: This is consistent with the empirical results in previous work [2, 17].

Original: In order to control the risk of deep forest, we need study the local variation property of the empirical
Revised: In order to control the risk of deep forest, we need study the local variation property of the empirical

Original: CART-split criterion.
Revised: CART-split criterion.

Original: For any cell C , the variation of regression function f ( x ) within C is defined as
Revised: For any cell C , the variation of regression function f ( x ) within C is defined as

Original: [Equation]
Revised: [Equation]

Original: Proposition 2 (Variation of f in the empirical cell) .
Revised: Proposition 2 (Variation of f in the empirical cell) .

Original: Assume that Assumption 1 holds and the first-layer forest is consistent.
Revised: Assume that Assumption 1 holds and the first-layer forest is consistent.

Original: The following results hold for any CART in the second-layer forest h (2) n ( · , Θ) .
Revised: The following results hold for any CART in the second-layer forest h (2) n ( · , Θ) .

Original: After splitting along the new feature, the CART will estimate the residual of the first-layer forest estimation.
Revised: After splitting along the new feature, the CART will estimate the residual of the first-layer forest estimation.

Original: Then for all ρ, ξ > 0 , there exists N ∈ N ∗ such that, for all n > N ,
Revised: Then for all ρ, ξ > 0 , there exists N ∈ N ∗ such that, for all n > N ,

Original: [Equation]
Revised: [Equation]

Original: Proof sketch.
Revised: Proof sketch.

Original: Firstly, we prove the variation of f ( x ) within the cell obtained by the theoretical
Revised: Firstly, we prove the variation of f ( x ) within the cell obtained by the theoretical

Original: CART-split criterion converges to zero.
Revised: CART-split criterion converges to zero.

Original: Secondly, we prove that the distance between the theoretical and empirical first k splits of the CART convergence to zero.
Revised: Secondly, we prove that the distance between the theoretical and empirical first k splits of the CART convergence to zero.

Original: Finally, we prove that the variation of f ( x ) within the empirical cell is close to the theoretical cell.
Revised: Finally, we prove that the variation of f ( x ) within the empirical cell is close to the theoretical cell.

Original: Proposition 2 shows that the variation of the regression function f ( x ) within a cell of a random tree C Π ,n is small provided n is large enough, thereby forcing the approximation error of DF to asymptotically approach zero.
Revised: Proposition 2 shows that the variation of the regression function f ( x ) within a cell of a random tree C Π ,n is small provided n is large enough, thereby forcing the approximation error of DF to asymptotically approach zero.

Original:  Depth is more powerful than width
Revised:  Depth is more powerful than width

Original: Our first result considers the t n < a n regime, where the number of samples in each leaf node tends to ∞ as t n → ∞ and a n → ∞ .
Revised: Our first result considers the t n < a n regime, where the number of samples in each leaf node tends to ∞ as t n → ∞ and a n → ∞ .

Original: We prove that controlling the depth of the trees through the number of leaves t n is sufficient to achieve consistency of deep forest.
Revised: We prove that controlling the depth of the trees through the number of leaves t n is sufficient to achieve consistency of deep forest.

Original: Theorem 3 (Universal consistency) .
Revised: Theorem 3 (Universal consistency) .

Original: Let M ≥ 1 .
Revised: Let M ≥ 1 .

Original: Consider two-layer deep forest ¯ h 2 M,n given by Eq.
Revised: Consider two-layer deep forest ¯ h 2 M,n given by Eq.

Original: (6) and Breiman’s random forest h M,n given by Eq.
Revised: (6) and Breiman’s random forest h M,n given by Eq.

Original: (4) for the random CARTs satisfying a n → ∞ , t n → ∞ and t n (log a n ) 9 /a n → 0 .
Revised: (4) for the random CARTs satisfying a n → ∞ , t n → ∞ and t n (log a n ) 9 /a n → 0 .

Original: Then under the setting described in Section 3 and assume the data set follows Assumption 1,
Revised: Then under the setting described in Section 3 and assume the data set follows Assumption 1,

Original: [29, Theorem 1] the Breiman’s random forest h 2 M,n is consistent for any M ≥ 1 , 2.
Revised: [29, Theorem 1] the Breiman’s random forest h 2 M,n is consistent for any M ≥ 1 , 2.

Original: the two-layer deep forest ¯ h M,n is consistent for any M ≥
Revised: the two-layer deep forest ¯ h M,n is consistent for any M ≥

Original: Proof sketch.
Revised: Proof sketch.

Original: (T3.1).
Revised: (T3.1).

Original: The universal consistency of Breiman’s random forest is proved by Scornet et al.
Revised: The universal consistency of Breiman’s random forest is proved by Scornet et al.

Original: (T3.2).
Revised: (T3.2).

Original: Firstly, we already know that the variation of the target regression function f ( x ) within a cell of a randomized CART in the second-layer forest is small when n is large enough via
Revised: Firstly, we already know that the variation of the target regression function f ( x ) within a cell of a randomized CART in the second-layer forest is small when n is large enough via

Original: Proposition 2.
Revised: Proposition 2.

Original: Similar as Scornet et al.
Revised: Similar as Scornet et al.

Original: [29], we utilize Proposition 2 to control the approximation error of the two-layer deep forest.
Revised: [29], we utilize Proposition 2 to control the approximation error of the two-layer deep forest.

Original: And the parameter t n allows us to control the size of the leaves of
Revised: And the parameter t n allows us to control the size of the leaves of

Original: CART, which allows us to have enough samples in each leaf node to smooth the impact of noise, so as to control the estimation error.
Revised: CART, which allows us to have enough samples in each leaf node to smooth the impact of noise, so as to control the estimation error.

Original: Connecting the approximation and estimation error, the consistency of a CART of second-layer deep forest is proved.
Revised: Connecting the approximation and estimation error, the consistency of a CART of second-layer deep forest is proved.

Original: The universal consistency can be proved via [25,
Revised: The universal consistency can be proved via [25,

Original: Proposition 1], which guarantees that the error of forest estimator is no more than twice that of individual randomized CART.
Revised: Proposition 1], which guarantees that the error of forest estimator is no more than twice that of individual randomized CART.

Original: Remark 2.
Revised: Remark 2.

Original: Notice that under this setting, random forest and even deep forest have no obvious advantages over single CART in theory.
Revised: Notice that under this setting, random forest and even deep forest have no obvious advantages over single CART in theory.

Original: When we use forests in practical, we do not choose to control the depth of the trees.
Revised: When we use forests in practice, we do not choose to control the depth of the trees.

Original: Empirical studies in Section 7.2 show that the forest with t n = a n always outperforms the forest with t n < a n .
Revised: Empirical studies in Section 7.2 show that the forest with t n = a n always outperforms the forest with t n < a n .

Original: Actually, the fully grown trees t n = a n is the setting close to practical forest algorithm.
Revised: Actually, the fully grown trees t n = a n is the setting close to practical forest algorithm.

Original: In order to deal with the t n = a n regime
Revised: In order to deal with the t n = a n regime

Original: [Equation]
Revised: [Equation]

Original: we need to introduce an assumption first proposed by
Revised: we need to introduce an assumption first proposed by

Original: Scornet et al.
Revised: Scornet et al.

Original: We denote by Z i = 1 ( x i ∈ C Π ,n ( x ))
Revised: We denote by Z i = 1 ( x i ∈ C Π ,n ( x ))

Original: the indicator that x i falls into the same cell as x in the random tree designed with D n and the random parameter Θ .
Revised: the indicator that x i falls into the same cell as x in the random tree designed with D n and the random parameter Θ .

Original: Z ′
Revised: Z ′

Original: j = 1 ( x i ∈
Revised: j = 1 ( x i ∈

Original: C Π ′ ( x )) is another independent indicator.
Revised: C Π ′ ( x )) is another independent indicator.

Original: Then, we define the correlation between these two indicators conditionally on y
Revised: Then, we define the correlation between these two indicators conditionally on y

Original: i , y j or not, respectively
Revised: i , y j or not, respectively

Original: [Equation]
Revised: [Equation]

Original: Assumption 2.
Revised: Assumption 2.

Original: Let Z i,j = ( Z i , Z j ) .
Revised: Let Z i,j = ( Z i , Z j ) .

Original: Then one of the following two conditions holds:
Revised: Then one of the following two conditions holds:

Original: One has
Revised: One has

Original: [Equation]
Revised: [Equation]

Original: There exists a constant C > 0 and a sequence ( γ n ) n → 0 such that, almost surely,
Revised: There exists a constant C > 0 and a sequence ( γ n ) n → 0 such that, almost surely,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: (A2.1.) means that the influence of two labels y i , y j on the probability of connection of two couples of random points converge to zero as n → ∞ .
Revised: (A2.1.) means that the influence of two labels y i , y j on the probability of connection of two couples of random points converge to zero as n → ∞ .

Original: (A2.2.) means that the correlation between the noise and the probability of connection of two couples of random points vanishes quickly enough, as n → ∞ .
Revised: (A2.2.) means that the correlation between the noise and the probability of connection of two couples of random points vanishes quickly enough, as n → ∞ .

Original: However, this assumption is too strong for the Breiman’s original random forest [1].
Revised: However, this assumption is too strong for the Breiman’s original random forest [1].

Original: Scornet et al.
Revised: Scornet et al.

Original: [29] emphasize that they cannot know whether or not Assumption 2 is satisfied in random forest.
Revised: [29] emphasize that they cannot know whether or not Assumption 2 is satisfied in random forest.

Original: In this paper, we recall this assumption and state that this assumption is mild for the second-layer forest estimation in deep forest algorithm.
Revised: In this paper, we recall this assumption and state that this assumption is mild for the second-layer forest estimation in deep forest algorithm.

Original: Since deep forest concatenates the prediction with the raw features as the input for the next layer, the label information is encoded into the new feature of second layer.
Revised: Since deep forest concatenates the prediction with the raw features as the input for the next layer, the label information is encoded into the new feature of second layer.

Original: In this way, the influence of two labels on the connection probability of this pair of samples tends to zero, so (A2.1.)
Revised: In this way, the influence of two labels on the connection probability of this pair of samples tends to zero, so (A2.1.)

Original: is mild for the second-layer forest estimation.
Revised: is mild for the second-layer forest estimation.

Original: From another point of view, Proposition 1, the priority of new feature also shows that the partition can be independent of labels, and the information of new feature is enough to obtain appropriate partition results, so (A2.2.) is mild too.
Revised: From another point of view, Proposition 1, the priority of new feature also shows that the partition can be independent of labels, and the information of new feature is enough to obtain appropriate partition results, so (A2.2.) is mild too.

Original: Theorem 4 (Depth is more powerful than width) .
Revised: Theorem 4 (Depth is more powerful than width) .

Original: Let M ≥ 1 .
Revised: Let M ≥ 1 .

Original: Consider two-layer deep forest ¯ h 2 M,n given by Eq.
Revised: Consider two-layer deep forest ¯ h 2 M,n given by Eq.

Original: (6) and Breiman’s random forest h M,n given by Eq.
Revised: (6) and Breiman’s random forest h M,n given by Eq.

Original: (4) for the random CARTs satisfying a n → ∞ , t n → ∞ , t n = a n and a n log n/n → 0 .
Revised: (4) for the random CARTs satisfying a n → ∞ , t n → ∞ , t n = a n and a n log n/n → 0 .

Original: Then under the setting described in Section 3 and assume the data set follows Assumption 1 and 2, the following results hold
Revised: Then under the setting described in Section 3 and assume the data set follows Assumption 1 and 2, the following results hold

Original: Theorem
Revised: Theorem

Original: Theorem 3]
Revised: Theorem 3]

Original: The Breiman’s random forest h ∞ ,n is consistent
Revised: The Breiman’s random forest h ∞ ,n is consistent

Original: and for all M, n ∈ N,
Revised: and for all M, n ∈ N

Original: [Equation]
Revised: [Equation]

Original: The two-layer deep forest ¯ h ∞ ,n is consistent, and for all M, n ∈ N , if ∆( f, C Π
Revised: The two-layer deep forest ¯ h ∞ ,n is consistent, and for all M, n ∈ N , if ∆( f, C Π

Original: [Equation]
Revised: [Equation]

Original: n ( x , Θ)) is small enough, then
Revised: n ( x , Θ)) is small enough, then

Original: [Equation]
Revised: [Equation]

Original: Proof sketch.
Revised: Proof sketch.

Original: (T4.1).
Revised: (T4.1).

Original: The consistency of the infinite Breiman’s random forest is proved by Scornet et al.
Revised: The consistency of the infinite Breiman’s random forest is proved by Scornet et al.

Original: And the convergence rate of the finite random forest with the number of trees M is proved by Scornet [30].
Revised: And the convergence rate of the finite random forest with the number of trees M is proved by Scornet [30].

Original: (T4.2).
Revised: (T4.2).

Original: Similar as Scornet et al.
Revised: Similar to Scornet et al.

Original: [29], we recall Proposition 2 to control the approximation error of the two-layer deep forest.
Revised: [29], we recall Proposition 2 to control the approximation error of the two-layer deep forest.

Original: Then the estimation error is controlled by forcing the subsampling rate a n /n to be o (1 / log n ) .
Revised: Then the estimation error is controlled by forcing the subsampling rate a n /n to be o (1 / log n ) .

Original: Different from the bagging-style mechanism in random forest, the residual-style mechanism shown in Proposition 2 makes the second-layer forest in DF can reuse the first-layer estimation and focus on the residual learning.
Revised: Different from the bagging-style mechanism in random forest, the residual-style mechanism shown in Proposition 2 makes the second-layer forest in DF can reuse the first-layer estimation and focus on the residual learning.

Original: Remark 3.
Revised: Remark 3.

Original: In the t n = a n setting, Scornet et al.
Revised: In the t n = a n setting, Scornet et al.

Original: [29] show that the sub-sampling rate a n /n is the key component in random forest.
Revised: [29] show that the sub-sampling rate a n /n is the key component in random forest.

Original: Because the small rate ensures that query point x is connected with enough different data points through different trees, the convergence rate of RF is O (1 /M ), w.r.t.
Revised: Because the small rate ensures that query point x is connected with enough different data points through different trees, the convergence rate of RF is O (1 /M ) w.r.t.

Original: the number of trees M .
Revised: the number of trees M .

Original: Theorem 4 proves that, if the first layer forest can encode the regression function f ( x ) into the new feature with noise reduction, the cascade structure with PreConc in DF can further accelerate the convergence.
Revised: Theorem 4 proves that, if the first layer forest can encode the regression function f ( x ) into the new feature with noise reduction, the cascade structure with PreConc in DF can further accelerate the convergence.

Original: Because the second-layer forest estimates the residual of the first layer, the trees in each layer of forest are more different.
Revised: Because the second layer forest estimates the residual of the first layer, the trees in each layer of forest are more different.

Original: As a result, the convergence rate of deep forest will be improved to O (1 /M 2 ) .
Revised: As a result, the convergence rate of deep forest will be improved to O (1 /M 2 ) .

Original: This  reflects that depth is more powerful than width.
Revised: This result reflects that deep forest with deeper layer will be more powerful than shallower layer.

Original:  Simulation experiments 7.1 Priority of the new feature
Revised:  Simulation experiments 7.1 Priority of the new feature

Original: This experiment aims to verify the priority of the new feature in choosing which feature to split as suggested in Proposition 1.
Revised: This experiment aims to verify the priority of the new feature in choosing which feature to split as suggested in Proposition 1.

Original: We focus on a second layer decision tree built upon the first layer random forest.
Revised: We focus on a second layer decision tree built upon the first layer random forest.

Original: Since a regression forest has only one output dimension, there’s only one new feature for the second layer tree.
Revised: Since a regression forest has only one output dimension, there is only one new feature for the second layer tree.

Original: More specifically, we count the maximum consecutive levels from root node that uses the new feature to split, which we call effective depth for short.
Revised: More specifically, we count the maximum consecutive levels from root node that use the new feature to split, which we call effective depth for short.

Original: The synthetic data set is generated as y = f ( x ) + ϵ , where
Revised: The synthetic data set is generated as y = f ( x ) + ϵ , where

Original: [Equation]
Revised: [Equation]

Original: 18) x is uniformly distributed over [0 , 1] 5 , ϵ ∼ N (cid:0) 0 , σ 2 (cid:1)  where σ = 0 .
Revised: 18) x is uniformly distributed over [0 , 1] 5 , ϵ ∼ N (cid:0) 0 , σ 2 (cid:1) , where σ = 0 .

Original: We vary the number of training samples and the number of trees in the first layerof forest, and report the average effective depth of 5 runs in Figure 2(a).
Revised: We vary the number of training samples and the number of trees in the first layer forest, and report the average effective depth of 5 runs in Figure 2(a).

Original: It is easy to observe that no matter what the first layer’s setting is, the effective depth of new feature is at least 2 . 4 .
Revised: It is easy to observe that no matter what the first layer’s setting is, the effective depth of new feature is at least 2 . 4 .

Original: That is to say, at the beginning of the growing of the second layer tree, CART will always choose the new feature to split.
Revised: That is to say, at the beginning of the growing of the second layer tree, CART will always choose the new feature to split.

Original: And we can see that with the increase of trees in the first layer and training samples, the CART’s preference of new feature extend
Revised: And we can see that with [Figure] i [Figure]

Revised: h

Original: [Figure]
Revised: [Figure]

Original:  t l
Revised: s t l

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  to deeper layers.
Revised: the increase of trees in the first layer and training samples, the CART’s preference of new feature extend to deeper layers.

Original: Figure 2(b) further plots the effective depth against the predictive error of the first layer measured by root mean square error (RMSE), showing that the better the  first layer’s predictive performance, the more  the new features are preferred in splitting.
Revised: Figure 2(b) further plots the effective depth against the predictive error of the first layer measured by root mean square error (RMSE), showing that the better the prediction performance of the first layer, the more favored the new feature is for splitting.

Original: 7.2 Convergence rate w.r.t. the number of trees
Revised: 7.2 Convergence rate w.r.t. the number of trees

Original: To study the convergence rate of predictive error with respect to the number of trees in the whole model, we fix the number of training samples to be 50,000.
Revised: To study the convergence rate of predictive error with respect to the number of trees in the whole model, we fix the number of training samples to be 50,000.

Original: We compare a 2-layer deep forest (DF) with M trees each layer to a random forest (RF) with 2 M trees, and we adopt 3-fold split in training both DF and RF to ensure fairness.
Revised: We compare a 2-layer deep forest (DF) with M trees each layer to a random forest (RF) with 2 M trees, and we adopt 3-fold split in training both DF and RF to ensure fairness.

Original: The performance is measured by RMSE with respect to the noise-free version of data generating function.
Revised: The performance is measured by RMSE with respect to the noise-free version of data generating function.

Original: Since the training data are with noise, a fully grown tree is inconsistent.
Revised: Since the training data are with noise, a fully grown tree is inconsistent.

Original: In Figure 3(a), we set the minimum leaf size of the trees to be √ n , i.e., 233 in the case when training data size is 50,000.
Revised: In Figure 3(a), we set the minimum leaf size of the trees to be √ n , i.e., 233 in the case when training data size is 50,000.

Original: In Figure 3(b) the trees are fully grown with only one sample in each leaf.
Revised: In Figure 3(b) the trees are fully grown with only one sample in each leaf.

Original: We plot the average RMSE of 5 runs against the increasing number of trees, with the colored band indicating the standard deviation.
Revised: We plot the average RMSE of 5 runs against the increasing number of trees, with the colored band indicating the standard deviation.

Original: Theorem 3 reveals that when the component trees are consistent, RF and DF are both consistent.
Revised: Theorem 3 reveals that when the component trees are consistent, random forest and deep forest are both consistent.

Original: However, the consistency analysis result cannot guarantee the finite sample performance in practice.
Revised: However, the consistency analysis result cannot guarantee the finite sample performance in practice.

Original: Comparing Figure 3(a) to Figure 3(b), we can see that even though the training set is as large as 50,000, the performances of RF and DF using consistent trees are still much worse than using inconsistent trees.
Revised: Comparing Figure 3(a) to Figure 3(b), we can see that even though the training set is as large as 50,000, the performances of RF and DF using consistent trees are still much worse than using inconsistent trees.

Original: This observation confirms the efficacy of the default experimental setting that uses fully grown trees in RF and DF.
Revised: This observation confirms the efficacy of the default experimental setting that uses fully grown trees in RF and DF.

Original: Figure 3(b) shows that DF enjoys a faster improvement in RMSE with the increasing of M .
Revised: Figure 3(b) shows that DF enjoys a faster improvement in RMSE with the increasing of M .

Original: More specifically, DF with M = 20 outperforms RF with M = 500 .
Revised: More specifically, DF with M =outperforms RF with M = 500 .

Original: This experimental result matches our theoretical result in Theorem 4 that DF has a faster convergence rate w.r.t.
Revised: This experimental result matches our theoretical result in Theoremthat DF has a faster convergence rate w.r.t.

Original: the number of trees M .
Revised: the number of trees M .

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we prove that a two-layer deep forest has a faster convergence rate w.r.t.
Revised: In this paper, we prove that a two-layer deep forest has a faster convergence rate w.r.t.

Original: the number of component trees M than random forest.
Revised: the number of component trees M than random forest.

Original: This work provides the first theoretical analysis of the feature concatenation (PreConc) operation which is crucial for feature transformation in deep forests, although based on a very simplified structure where the concatenation of multiple random forests’ predictions and completely-random forests’ predictions in each layer of deep forest has not been taken into account.
Revised: This work provides the first theoretical analysis of the feature concatenation (PreConc) operation which is crucial for feature transformation in deep forests, although based on a very simplified structure where the concatenation of multiple random forests’ predictions and completely-random forests’ predictions in each layer of deep forest has not been taken into account.

Original: On the one hand, this paper focuses on the asymptotic consistency of deep forests, so the result is strictly true only when the number of samples tends to infinity.
Revised: On the one hand, this paper focuses on the asymptotic consistency of deep forests, so the result is strictly true only when the number of samples tends to infinity.

Original: As  the generalization analysis of deep forests with finite samples, we leave it to future work.
Revised: As for the generalization analysis of deep forests with finite samples, we leave it to future work.

Original: On the other hand, the two assumptions used in this paper have certain limitations.
Revised: On the other hand, the two assumptions used in this paper have certain limitations.

Original: Experiments on simulation and real-world data sets show that our theoretical results are valid in many objective function classes other than Assumption 1.
Revised: Experiments on simulation and real-world data sets show that our theoretical results are valid in many objective function classes other than Assumption 1.

Original: How to further relax the conditions in Assumption 1 will be an interesting problem.
Revised: How to further relax the conditions in Assumption 1 will be an interesting problem.

Original: As for Assumption 2, it is still not strictly verified.
Revised: As for Assumption 2, it is still not strictly verified.

Original: However, quantifying the trade-off between label information and partition randomness will be a very important topic in future work.
Revised: However, quantifying the trade-off between label information and partition randomness will be a very important topic in future work.

Original: Acknowledgements
Revised: Acknowledgements

Original: This research was supported by the NSFC (61921006) and the Collaborative Innovation Center of
Revised: This research was supported by the NSFC (61921006) and the Collaborative Innovation Center of

Original: Novel Software Technology and Industrialization.
Revised: Novel Software Technology and Industrialization.

Original: The authors would like to thank the anonymous reviewers for constructive suggestions, as well as Jin-Hui Wu, Qin-Cheng Zheng and Peng Tan for helpful discussions.
Revised: The authors would like to thank the anonymous reviewers for constructive suggestions, as well as Jin-Hui Wu, Qin-Cheng Zheng and Peng Tan for helpful discussions.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section 7.
Revised: [Yes] See Section 7.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] This is a theoretical work.
Revised: [No] This is a theoretical work.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: [Yes] 3.
Revised: [Yes] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] (b) Did you specify all the training details (e.g., data splits, hyper-parameters, how they were chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyper-parameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.
Revised: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether
Revised: [No] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [No] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [No] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [No] 5.
Revised: [No] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [No] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [No] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [No] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [No] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [No]
Revised: [No]