Paper:
When do random forests fail?
Random forests are learning algorithms that build large collections of random trees and make predictions by averaging the individual tree predictions. In this paper, we consider various tree constructions and examine how the choice of parameters affects the generalization error of the resulting random forests as the sample size goes to infinity. We show that subsampling of data points during the tree construction phase is important: Forests can become inconsistent with either no subsampling or too severe subsampling. As a consequence, even highly randomized trees can lead to inconsistent forests if no subsampling is used, which implies that some of the commonly used setups for random forests can be inconsistent. As a second consequence we can show that trees that have good performance in nearest-neighbor search can be a poor choice for random forests.

Review:
This paper studies necessary conditions for (infinite) ensembles of randomized regression trees to be inconsistent (in regression). The authors first establish that locality and diversity are necessary conditions for the consistency of forests. From this result, they show that several types of forests are inconsistent: forests of extremely randomized deep trees satisfying the nearest-neighbor preserving property (which includes two random projection tree algorithms as instances), forests of fully-grown trees with fast shrinking cells, and finally forests grown with too severe subsampling.   The paper contains interesting and, to the best of my knowledge, original theoretical results about (in)consistency of forests of randomized trees. The originality of the paper is to try to characterize conditions under which generic random forests models are inconsistent, while other papers focus on consistency proof of some simplified algorithms. The paper is also very well written and pleasant to read. The authors very well summarize their main theoretical results in the paper, while detailed proofs are given in the supplementary material.   Although the paper is well written, I find however that several statements or implications that are drawn from the theorems in the paper are too strong or not strictly correct. The authors overstate a bit the surprising nature of their results and how much they contradict statements from the literature. Just to give two examples of statements that I found not well argued:  1) About the importance of subsampling: "we show that subsampling of data points during the tree construction phase is critical: forests can become inconsistent with either no subsampling or too severe subsampling", "without subsampling, forests of deep trees can become inconsistent due to violation of diversity", "In summary, our results indicate that subsampling plays an important role in random forests and may need to be tuned more carefully than other parameters". I have two problems with these statements: First, I don't see where it is formally proven (in this paper or in another) that adding subsampling to the different tree models for which inconsistency is proven in the paper will lead to consistency. Actually, page 6, the authors say "we speculate that sufficient subsampling can make the forests consistent again...We leave this for future work". In the absence of a proof, I think the previous statements are premature. Second, even if subsampling would lead to consistency, I don't think subsampling is really crucial per se. Forests of deep trees are inconsistent because the trees are deep. Subsampling is thus clearly not the only or even the most obvious solution. Another solution to make them consistent could be for example to prune the trees (in a way that depends on learning sample size). As subsampling is not the only solution, it is not critical per se, what is critical is the effect that it has on diversity, not its added randomness. 2) About randomness and overfitting: "Theorem 2 implies that forests of extremely randomized trees can still overfit. This results disagrees with the conventional belief that injecting more "randomness" prevents tree from overfitting" and in the conclusion: "Another implication is that even extremely randomized tree can lead to overfitting forests, which disagrees with the conventional belief that injecting more "randomness" will prevent trees from overfitting". I don't understand this statement and the link with the theorem provided in the paper. The conventional belief is not that injecting more randomness implies NO overfitting but rather than injecting more randomness implies LESS overfitting. And I see no violation of this common belief in the paper: forests of extremely randomized deep trees have a low variance but not a zero variance (that's why they are not consistent), and adding subsampling is a way to add even more randomness that can lead to less overfitting and thus consistency, as suggested in the paper. So, I don't see any contradiction or surprising results here.  Overall, I think that the authors should be more cautious when discussing the implications of their theorems. This requires in my opinion a significant rewrite of some parts of the paper.  The paper topic does not fit also very well to the format of a conference paper. It does not focus on a single clear idea, as the contribution is a series of theorems that remains incremental with respect to previous results in this domain and also does not end any story. Their practical impact is also limited as they concern simplified models (UW trees) that are not really competitive in practice. The paper has also a 11-pages supplementary material that is difficult to check within the time allocated for reviews at conferences. A statistical journal might be a better target for this work.  Minor comments: * All theorems refer to a data model in Eq (2) and the authors mention that restrictions on the data model are more important for Lemma 2 than for Lemma 1. Actually, I could not find any Eq (2), neither in the paper nor in the appendix. What is this data model?  * The use of the term extremely randomized trees in the paper does not seem to match the use of this name in (Geurts et al., 2006) and is therefore a bit misleading. Extremely randomized trees in (Geurts et al., 2006) are not UW trees because the best split is still selected among K inputs selected at random, which is thus selected on the basis of the labels. Extremely randomized trees are UW trees only when K=1 and in this case, they are called Totally randomized trees in (Geurts et al., 2006). Furthermore, extremely randomized trees in (Geurts et al., 2006) are grown in a specific manner, while Definition 5 is generic. Why not calling them simply "deep UW trees"?  Update after the author's reponses  I thank the authors for their response. They do not totally address my concerns however: - Importance of subsampling: "The reviewer is also correct in that we did not formally prove but only speculate that with proper subsampling, our RF models shown inconsistent in our paper could become consistent again". Then, I reiterate my comment that you can not yet claim in the paper that subsampling is critical. - Randomness and overfitting: "We agree with the reviewer that the conventional belief is that “injecting more randomness implies less overfitting".This is exactly what we would like to argue against". "We argue (here and in paper) that our Theorem 2 suggests that this type of thinking is incorrect, since highly randomized trees without subsampling can still overfit." Again, to me adding subsampling means injecting more randomness, which implies less overfitting. I still don’t see how the result in theorem 2 really contradicts this statement. I agree however that different ways of injecting randomness are certainly not equivalent and that the RF variant of [Geurts et al., 2006] might overfit more than Breiman’s original Random Forests. But this is not strictly shown in the paper and I don’t really see how this could be proven as it is difficult to compare the levels of randomness of different randomization schemes. Note that the link between overfitting and consistency is not totally clear to me. At finite sample size, a model might overfit less than another, even if only the latter is consistent. Overall, I still believe that the discussion about randomness and overfitting in the paper is confusing and needs to be clarified.  Note that I acknowledge the value of the theoretical results in the paper. I still believe however that the authors should be more cautious in the conclusions they draw from these results.