ConsistencyforaSimpleModelofRandomForests,Leo.PDFVIP

CONSISTENCY FOR A SIMPLE MODEL OF RANDOM FORESTS Leo Breiman Technical Report 670 STATISTICS DEPARTMENT UNIVERSITY OF CALIFORNIA AT BERKELEY September 9, 2004 ^: 0) Introduction Random Forests is a classification algorithm with a simple structure--a forest of trees are grown as follows: 1) The training set is a bootstrap sample from the original training set. 2) An integer mtry is set by the user, where m mtry is less than the total number of variables. At each node, mtry variables are selected at random and the node is split on the best split among the selected mtry. The tree is grown to its maximal depth. 3) In regression, as a test vector x is put down each tree it is assigned the average values of the y-values at the node it stops at. The average of these over all trees in the forest is the predicted value for x. The predicted value for classification is the class getting the plurality of the forest votes. Random Forests is an accurate algorithm having the unusual ability to handle thousands of variables without deletion or deterioration of accuracy. The difficulty is that although the mechanism appears simple, it is difficult to analyze. A heuristic analysis is presented in this paper based on a simplified version of RF denoted RF0. The results from RF0 support the empirical results from RF. RF0 regression is consistent using a value of mtry that does not depend on the number of cases N The rate of convergence to the Bayes rule depends only on the number of strong variables and not on how many noise variables are also present.. This also implies consistency for the two class RF0 classification. The analysis also illuminates why RF is able to handle large numbers of input variables and what the role of mtry is. U