Module Expr Str Print NameLoad Import alias identifier Import alias identifier ImportFrom alias ImportFrom alias ImportFrom alias Assign NameStore Num Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Num Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr Num Num keyword NameLoad Assign NameStore BinOpDiv Num BinOpAdd Num Call AttributeLoad NameLoad attr UnaryOpUSub BinOpAdd BinOpSub Call AttributeLoad NameLoad attr BinOpMult Num NameLoad BinOpMult Num NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr Num NameLoad keyword NameLoad Assign NameStore SubscriptLoad AttributeLoad NameLoad attr Index TupleLoad NameLoad NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign TupleStore NameStore NameStore NameStore NameStore Call NameLoad NameLoad NameLoad keyword Num keyword Num Assign NameStore Dict Str Str Str Str Str Str Num Num Num Num Num Num Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Print Call AttributeLoad Str attr NameLoad Assign NameStore SubscriptLoad NameLoad Index Str Assign NameStore BinOpAdd Call AttributeLoad NameLoad attr NameLoad Num FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Assign NameStore Call AttributeLoad NameLoad attr TupleLoad NameLoad keyword AttributeLoad NameLoad attr For TupleStore NameStore NameStore Call NameLoad Call AttributeLoad NameLoad attr NameLoad body Assign SubscriptStore NameLoad Index NameLoad Call AttributeLoad NameLoad attr NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam defaults Num body Assign NameStore Call NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr TupleLoad NameLoad keyword AttributeLoad NameLoad attr For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad SubscriptLoad NameLoad Index NameLoad AugAssignAdd NameStore Call NameLoad NameLoad SubscriptLoad NameLoad Index NameLoad SubscriptLoad NameLoad Index NameLoad AugAssignDiv NameStore NameLoad Return NameLoad decorator_list Assign NameStore Call NameLoad Num Assign NameStore Call NameLoad NameLoad NameLoad NameLoad Assign NameStore UnaryOpUSub Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore SubscriptLoad NameLoad Index Call AttributeLoad NameLoad attr NameLoad AugAssignSub NameStore SubscriptLoad NameLoad Index Num Assign NameStore SubscriptLoad NameLoad Index Call AttributeLoad NameLoad attr NameLoad AugAssignSub NameStore SubscriptLoad NameLoad Index Num Assign NameStore SubscriptLoad NameLoad Index Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad Call NameLoad Lambda arguments args NameParam defaults BinOpDiv NameLoad Num TupleLoad Num Num Num Assign NameStore Call NameLoad Call NameLoad Lambda arguments args NameParam defaults BinOpDiv NameLoad Num TupleLoad Num Num Num Assign NameStore Call NameLoad Call NameLoad Lambda arguments args NameParam defaults BinOpDiv NameLoad Num TupleLoad Num Num Num Expr Call AttributeLoad NameLoad attr NameLoad NameLoad keyword Str keyword NameLoad Expr Call AttributeLoad NameLoad attr NameLoad NameLoad keyword Str keyword NameLoad Expr Call AttributeLoad NameLoad attr NameLoad NameLoad keyword Str keyword NameLoad Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad Expr Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr BinOpAdd Call AttributeLoad SubscriptLoad NameLoad Index Num attr ListLoad NameLoad NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr BinOpAdd Call NameLoad Call NameLoad Lambda arguments args NameParam defaults Call NameLoad NameLoad SubscriptLoad NameLoad Index Num ListLoad Str Str Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore SubscriptLoad NameLoad Index NameLoad Assign NameStore SubscriptLoad NameLoad Index NameLoad Expr Call AttributeLoad NameLoad attr NameLoad NameLoad Expr Call AttributeLoad NameLoad attr keyword Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr
N/A N/A 
======================================
Gradient Boosting Out-of-Bag estimates
======================================

Out-of-bag (OOB) estimates can be a useful heuristic to estimate
the "optimal" number of boosting iterations.
OOB estimates are almost identical to cross-validation estimates but
they can be computed on-the-fly without the need for repeated model
fitting.
OOB estimates are only available for Stochastic Gradient Boosting
(i.e. ``subsample < 1.0``), the estimates are derived from the improvement
in loss based on the examples not included in the bootstrap sample
(the so-called out-of-bag examples).
The OOB estimator is a pessimistic estimator of the true
test loss, but remains a fairly good approximation for a small number of trees.

The figure shows the cumulative sum of the negative OOB improvements
as a function of the boosting iteration. As you can see, it tracks the test
loss for the first hundred iterations but then diverges in a
pessimistic way.
The figure also shows the performance of 3-fold cross validation which
usually gives a better estimate of the test loss
but is computationally more demanding.
 N/A __doc__ N/A numpy np N/A matplotlib.pyplot plt sklearn ensemble sklearn.model_selection KFold sklearn.model_selection train_test_split N/A n_samples 1000 N/A random_state N/A N/A N/A np random RandomState 13 N/A x1 N/A N/A random_state uniform size n_samples N/A x2 N/A N/A random_state uniform size n_samples N/A x3 N/A N/A random_state randint 0 4 size n_samples N/A p N/A 1 N/A 1.0 N/A N/A np exp N/A N/A N/A N/A N/A np sin N/A 3 x1 N/A 4 x2 x3 N/A y N/A N/A random_state binomial 1 p size n_samples N/A X N/A N/A np c_ N/A N/A x1 x2 x3 N/A X N/A N/A X astype N/A np float32 N/A N/A X_train X_test y_train y_test N/A train_test_split X y test_size 0.5 random_state 9 N/A params N/A n_estimators max_depth subsample learning_rate min_samples_leaf random_state 1200 3 0.5 0.01 1 3 N/A clf N/A N/A ensemble GradientBoostingClassifier params N/A N/A N/A clf fit X_train y_train N/A acc N/A N/A clf score X_test y_test N/A N/A N/A Accuracy: {:.4f} format acc N/A n_estimators N/A params N/A n_estimators N/A x N/A N/A N/A np arange n_estimators 1 heldout_score N/A N/A clf X_test y_test N/A N/A N/A compute deviance scores on ``X_test`` and ``y_test``.  N/A score N/A N/A np zeros N/A n_estimators dtype N/A np float64 N/A N/A i y_pred N/A enumerate N/A N/A clf staged_decision_function X_test N/A N/A N/A score N/A i N/A N/A clf loss_ y_test y_pred N/A score N/A cv_estimate N/A N/A n_folds N/A 3 N/A N/A cv N/A KFold n_folds n_folds N/A cv_clf N/A N/A ensemble GradientBoostingClassifier params N/A val_scores N/A N/A np zeros N/A n_estimators dtype N/A np float64 N/A N/A train test N/A N/A cv split X_train y_train N/A N/A N/A N/A cv_clf fit N/A X_train N/A train N/A y_train N/A train N/A val_scores N/A heldout_score cv_clf N/A X_train N/A test N/A y_train N/A test N/A val_scores n_folds N/A val_scores N/A N/A cv_score N/A cv_estimate 3 N/A test_score N/A heldout_score clf X_test y_test N/A cumsum N/A N/A N/A np cumsum N/A clf oob_improvement_ N/A oob_best_iter N/A x N/A N/A N/A np argmin cumsum N/A test_score N/A test_score N/A 0 N/A test_best_iter N/A x N/A N/A N/A np argmin test_score N/A cv_score N/A cv_score N/A 0 N/A cv_best_iter N/A x N/A N/A N/A np argmin cv_score N/A oob_color N/A list N/A map N/A N/A N/A x N/A N/A x 256.0 N/A 190 174 212 N/A test_color N/A list N/A map N/A N/A N/A x N/A N/A x 256.0 N/A 127 201 127 N/A cv_color N/A list N/A map N/A N/A N/A x N/A N/A x 256.0 N/A 253 192 134 N/A N/A N/A plt plot x cumsum label OOB loss color oob_color N/A N/A N/A plt plot x test_score label Test loss color test_color N/A N/A N/A plt plot x cv_score label CV loss color cv_color N/A N/A N/A plt axvline x oob_best_iter color oob_color N/A N/A N/A plt axvline x test_best_iter color test_color N/A N/A N/A plt axvline x cv_best_iter color cv_color N/A xticks N/A N/A plt xticks N/A xticks_pos N/A N/A np array N/A N/A N/A N/A xticks N/A 0 tolist N/A oob_best_iter cv_best_iter test_best_iter N/A xticks_label N/A N/A np array N/A N/A list N/A map N/A N/A N/A t N/A N/A int t N/A xticks N/A 0 N/A OOB CV Test N/A ind N/A N/A np argsort xticks_pos N/A xticks_pos N/A xticks_pos N/A ind N/A xticks_label N/A xticks_label N/A ind N/A N/A N/A plt xticks xticks_pos xticks_label N/A N/A N/A plt legend loc upper right N/A N/A N/A plt ylabel normalized loss N/A N/A N/A plt xlabel number of iterations N/A N/A N/A plt show