Module Expr Str Print NameLoad Import alias identifier Import alias identifier ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias Assign NameStore Num Assign NameStore Num Assign NameStore ListLoad TupleLoad Num Num TupleLoad Num Num TupleLoad Num Num Assign TupleStore NameStore NameStore Call NameLoad keyword NameLoad keyword Num keyword Num keyword NameLoad keyword NameLoad keyword Num Assign SubscriptStore NameLoad Slice BinOpFloorDiv NameLoad Num Num Assign SubscriptStore NameLoad Slice BinOpFloorDiv NameLoad Num Num Assign NameStore Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr Num attr SubscriptLoad AttributeLoad NameLoad attr Index Num Assign TupleStore NameStore NameStore NameStore NameStore NameStore NameStore Call NameLoad NameLoad NameLoad NameLoad keyword Num keyword Num Assign NameStore Call NameLoad Expr Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr NameLoad ExtSlice Slice Index Num Assign NameStore Call NameLoad NameLoad keyword Num keyword Str Expr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr NameLoad ExtSlice Slice Index Num Assign NameStore Call NameLoad NameLoad keyword Num keyword Str Expr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr NameLoad ExtSlice Slice Index Num Print Str Assign NameStore Call NameLoad NameLoad NameLoad NameLoad Print BinOpMod Str NameLoad Assign NameStore Call NameLoad NameLoad NameLoad NameLoad Print BinOpMod Str NameLoad Assign NameStore Call NameLoad NameLoad NameLoad NameLoad Print BinOpMod Str NameLoad Expr Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Num Num AttributeLoad NameLoad attr For TupleStore NameStore NameStore Call NameLoad NameLoad NameLoad body Assign NameStore SubscriptLoad NameLoad Index CompareEq NameLoad NameLoad Assign NameStore SubscriptLoad NameLoad Index CompareEq NameLoad NameLoad Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad ExtSlice Slice Index Num SubscriptLoad NameLoad ExtSlice Slice Index Num keyword BinOpMult NameLoad Num keyword NameLoad keyword Num keyword BinOpMod Str NameLoad Expr Call AttributeLoad NameLoad attr keyword Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr TupleLoad NameLoad Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad Str keyword BinOpMod Str NameLoad Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad Str keyword Num keyword BinOpMod Str NameLoad Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad Str keyword Num keyword BinOpMod Str NameLoad Expr Call AttributeLoad NameLoad attr SubscriptLoad Call AttributeLoad NameLoad attr Num AttributeLoad NameLoad attr Num Slice Num Num Call AttributeLoad Call AttributeLoad SubscriptLoad NameLoad Index NameLoad attr Num Num attr Num Str keyword Num keyword Str Expr Call AttributeLoad NameLoad attr ListLoad Num Num Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr keyword Str Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr
N/A N/A 
======================================
Probability calibration of classifiers
======================================

When performing classification you often want to predict not only
the class label, but also the associated probability. This probability
gives you some kind of confidence on the prediction. However, not all
classifiers provide well-calibrated probabilities, some being over-confident
while others being under-confident. Thus, a separate calibration of predicted
probabilities is often desirable as a postprocessing. This example illustrates
two different methods for this calibration and evaluates the quality of the
returned probabilities using Brier's score
(see https://en.wikipedia.org/wiki/Brier_score).

Compared are the estimated probability using a Gaussian naive Bayes classifier
without calibration, with a sigmoid calibration, and with a non-parametric
isotonic calibration. One can observe that only the non-parametric model is able
to provide a probability calibration that returns probabilities close to the
expected 0.5 for most of the samples belonging to the middle cluster with
heterogeneous labels. This results in a significantly improved Brier score.
 N/A __doc__ N/A numpy np N/A matplotlib.pyplot plt matplotlib cm sklearn.datasets make_blobs sklearn.naive_bayes GaussianNB sklearn.metrics brier_score_loss sklearn.calibration CalibratedClassifierCV sklearn.model_selection train_test_split N/A n_samples 50000 N/A n_bins 3 N/A centers N/A N/A -5 -5 N/A 0 0 N/A 5 5 N/A N/A X y N/A make_blobs n_samples n_samples n_features 2 cluster_std 1.0 centers centers shuffle False random_state 42 N/A N/A y N/A N/A n_samples 2 0 N/A N/A y N/A N/A n_samples 2 1 N/A sample_weight N/A N/A N/A N/A N/A np random RandomState 42 rand N/A N/A y shape N/A 0 N/A N/A X_train X_test y_train y_test sw_train sw_test N/A train_test_split X y sample_weight test_size 0.9 random_state 42 N/A clf N/A GaussianNB N/A N/A N/A clf fit X_train y_train N/A prob_pos_clf N/A N/A N/A clf predict_proba X_test N/A N/A N/A 1 N/A clf_isotonic N/A CalibratedClassifierCV clf cv 2 method isotonic N/A N/A N/A clf_isotonic fit X_train y_train sw_train N/A prob_pos_isotonic N/A N/A N/A clf_isotonic predict_proba X_test N/A N/A N/A 1 N/A clf_sigmoid N/A CalibratedClassifierCV clf cv 2 method sigmoid N/A N/A N/A clf_sigmoid fit X_train y_train sw_train N/A prob_pos_sigmoid N/A N/A N/A clf_sigmoid predict_proba X_test N/A N/A N/A 1 N/A Brier scores: (the smaller the better) N/A clf_score N/A brier_score_loss y_test prob_pos_clf sw_test N/A N/A No calibration: %1.3f clf_score N/A clf_isotonic_score N/A brier_score_loss y_test prob_pos_isotonic sw_test N/A N/A With isotonic calibration: %1.3f clf_isotonic_score N/A clf_sigmoid_score N/A brier_score_loss y_test prob_pos_sigmoid sw_test N/A N/A With sigmoid calibration: %1.3f clf_sigmoid_score N/A N/A N/A plt figure N/A y_unique N/A N/A np unique y N/A colors N/A N/A cm rainbow N/A N/A np linspace 0.0 1.0 N/A y_unique size N/A N/A this_y color N/A zip y_unique colors N/A N/A this_X N/A X_train N/A N/A y_train this_y N/A this_sw N/A sw_train N/A N/A y_train this_y N/A N/A N/A plt scatter N/A this_X N/A N/A N/A 0 N/A this_X N/A N/A N/A 1 s N/A this_sw 50 c color alpha 0.5 label N/A Class %s this_y N/A N/A N/A plt legend loc best N/A N/A N/A plt title Data N/A N/A N/A plt figure N/A order N/A N/A np lexsort N/A prob_pos_clf N/A N/A N/A plt plot N/A prob_pos_clf N/A order r label N/A No calibration (%1.3f) clf_score N/A N/A N/A plt plot N/A prob_pos_isotonic N/A order g linewidth 3 label N/A Isotonic calibration (%1.3f) clf_isotonic_score N/A N/A N/A plt plot N/A prob_pos_sigmoid N/A order b linewidth 3 label N/A Sigmoid calibration (%1.3f) clf_sigmoid_score N/A N/A N/A plt plot N/A N/A N/A np linspace 0 N/A y_test size 51 N/A 1 2 N/A N/A N/A N/A N/A y_test N/A order reshape 25 -1 mean 1 k linewidth 3 label Empirical N/A N/A N/A plt ylim N/A -0.05 1.05 N/A N/A N/A plt xlabel Instances sorted according to predicted probability (uncalibrated GNB) N/A N/A N/A plt ylabel P(y=1) N/A N/A N/A plt legend loc upper left N/A N/A N/A plt title Gaussian naive Bayes probabilities N/A N/A N/A plt show