Module Expr Str ImportFrom alias Import alias Import alias identifier ImportFrom alias Import alias ImportFrom alias Import alias identifier ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias ImportFrom alias Expr Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr keyword Str Assign NameStore Call NameLoad Expr Call AttributeLoad NameLoad attr Str keyword Str keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str keyword Str keyword Str keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str keyword Str keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str keyword Str keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str keyword Str keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str keyword Str keyword Str Expr Call AttributeLoad NameLoad attr Str keyword Str keyword NameLoad keyword BinOpPow Num Num keyword Str Expr Call AttributeLoad NameLoad attr Str keyword Str keyword Str Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr If CompareGt Call NameLoad NameLoad Num body Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Num Expr Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Expr Call NameLoad If AttributeLoad NameLoad attr body Assign NameStore NameLoad orelse Assign NameStore ListLoad Str Str Str Str If AttributeLoad NameLoad attr body Assign NameStore TupleLoad Str Str Str orelse Assign NameStore TupleLoad Expr Call NameLoad Str Expr Call NameLoad IfExp NameLoad NameLoad Str Assign NameStore Call NameLoad keyword Str keyword NameLoad keyword NameLoad keyword Num keyword NameLoad Assign NameStore Call NameLoad keyword Str keyword NameLoad keyword NameLoad keyword Num keyword NameLoad Expr Call NameLoad Str Assign NameStore AttributeLoad NameLoad attr FunctionDef arguments args NameParam defaults body Return BinOpDiv Call NameLoad GeneratorExp Call NameLoad Call AttributeLoad NameLoad attr Str comprehension NameStore NameLoad Num decorator_list Assign NameStore Call NameLoad AttributeLoad NameLoad attr Assign NameStore Call NameLoad AttributeLoad NameLoad attr Expr Call NameLoad BinOpMod Str TupleLoad Call NameLoad AttributeLoad NameLoad attr NameLoad Expr Call NameLoad BinOpMod Str TupleLoad Call NameLoad AttributeLoad NameLoad attr NameLoad Expr Call NameLoad BinOpMod Str Call NameLoad NameLoad Expr Call NameLoad Assign TupleStore NameStore NameStore TupleLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call NameLoad Str Assign NameStore Call NameLoad If AttributeLoad NameLoad attr body Assign NameStore Call NameLoad keyword Str keyword NameLoad keyword AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr orelse Assign NameStore Call NameLoad keyword NameLoad keyword Num keyword Str Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore BinOpSub Call NameLoad NameLoad Expr Call NameLoad BinOpMod Str TupleLoad NameLoad BinOpDiv NameLoad NameLoad Expr Call NameLoad BinOpMod Str AttributeLoad NameLoad attr Expr Call NameLoad Expr Call NameLoad Str Assign NameStore Call NameLoad Assign NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Assign NameStore BinOpSub Call NameLoad NameLoad Expr Call NameLoad BinOpMod Str TupleLoad NameLoad BinOpDiv NameLoad NameLoad Expr Call NameLoad BinOpMod Str AttributeLoad NameLoad attr Expr Call NameLoad If AttributeLoad NameLoad attr body Assign NameStore NameLoad orelse Assign NameStore Call AttributeLoad NameLoad attr If AttributeLoad NameLoad attr body Expr Call NameLoad BinOpMod Str AttributeLoad NameLoad attr Assign NameStore Call NameLoad Assign NameStore Call NameLoad NameLoad keyword AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad If NameLoad body Assign NameStore ListComp SubscriptLoad NameLoad Index NameLoad comprehension NameStore Call AttributeLoad NameLoad attr keyword NameLoad Expr Call NameLoad BinOpMod Str BinOpSub Call NameLoad NameLoad Expr Call NameLoad If NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad FunctionDef arguments args NameParam defaults body Expr Str Return IfExp CompareLtE Call NameLoad NameLoad Num NameLoad BinOpAdd SubscriptLoad NameLoad Slice Num Str decorator_list FunctionDef arguments args NameParam defaults body Expr Call NameLoad BinOpMult Str Num Expr Call NameLoad Str Expr Call NameLoad NameLoad Assign NameStore Call NameLoad Expr Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore BinOpSub Call NameLoad NameLoad Expr Call NameLoad BinOpMod Str NameLoad Assign NameStore Call NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore BinOpSub Call NameLoad NameLoad Expr Call NameLoad BinOpMod Str NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad Expr Call NameLoad BinOpMod Str NameLoad If Call NameLoad NameLoad Str body Expr Call NameLoad BinOpMod Str SubscriptLoad AttributeLoad AttributeLoad NameLoad attr attr Index Num Expr Call NameLoad BinOpMod Str Call NameLoad AttributeLoad NameLoad attr If BoolOpAnd AttributeLoad NameLoad attr CompareIsNot NameLoad NameLoad body Expr Call NameLoad Str For TupleStore NameStore NameStore Call NameLoad NameLoad body Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr SubscriptLoad AttributeLoad NameLoad attr Index NameLoad Slice Num Expr Call NameLoad Call NameLoad BinOpMod Str TupleLoad NameLoad Call AttributeLoad Str attr SubscriptLoad NameLoad Index NameLoad Expr Call NameLoad If AttributeLoad NameLoad attr body Expr Call NameLoad Str Expr Call NameLoad Call AttributeLoad NameLoad attr NameLoad NameLoad keyword NameLoad If AttributeLoad NameLoad attr body Expr Call NameLoad Str Expr Call NameLoad Call AttributeLoad NameLoad attr NameLoad NameLoad Expr Call NameLoad Assign NameStore SubscriptLoad Call AttributeLoad Call NameLoad NameLoad attr Str Index Num Return TupleLoad NameLoad NameLoad NameLoad NameLoad decorator_list Assign NameStore ListLoad For TupleStore NameStore NameStore TupleLoad TupleLoad Call NameLoad keyword Num keyword Str Str TupleLoad Call NameLoad keyword Num Str TupleLoad Call NameLoad keyword Num Str TupleLoad Call NameLoad keyword Num Str TupleLoad Call NameLoad keyword Num Str body Expr Call NameLoad BinOpMult Str Num Expr Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Call NameLoad NameLoad For NameStore ListLoad Str Str body Expr Call NameLoad BinOpMult Str Num Expr Call NameLoad BinOpMod Str Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Call NameLoad Call NameLoad keyword Str keyword NameLoad keyword NameLoad keyword Num Expr Call AttributeLoad NameLoad attr Call NameLoad Call NameLoad keyword Num keyword Num keyword NameLoad Expr Call NameLoad BinOpMult Str Num Expr Call NameLoad Str Expr Call AttributeLoad NameLoad attr Call NameLoad Call NameLoad keyword Num keyword Num keyword Str Expr Call NameLoad BinOpMult Str Num Expr Call NameLoad Str Expr Call AttributeLoad NameLoad attr Call NameLoad Call NameLoad Expr Call NameLoad BinOpMult Str Num Expr Call NameLoad Str Expr Call AttributeLoad NameLoad attr Call NameLoad Call NameLoad keyword Num Expr Call AttributeLoad NameLoad attr Call NameLoad Call NameLoad keyword Num Expr Call NameLoad BinOpMult Str Num Expr Call NameLoad Str Expr Call AttributeLoad NameLoad attr Call NameLoad Call NameLoad ListLoad TupleLoad Str Call NameLoad keyword Str keyword NameLoad keyword Num TupleLoad Str Call NameLoad Assign NameStore Call AttributeLoad NameLoad attr Call NameLoad NameLoad Assign NameStore ListComp ListComp SubscriptLoad NameLoad Index NameLoad comprehension NameStore NameLoad comprehension NameStore Call NameLoad Num Assign TupleStore NameStore NameStore NameStore NameStore NameLoad Assign NameStore BinOpDiv Call AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr NameLoad Assign NameStore BinOpDiv Call AttributeLoad NameLoad attr NameLoad Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr keyword TupleLoad Num Num Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr NameLoad NameLoad Num keyword Str keyword Str Expr Call AttributeLoad NameLoad attr BinOpAdd NameLoad Num NameLoad Num keyword Str keyword Str Expr Call AttributeLoad NameLoad attr BinOpAdd NameLoad Num NameLoad Num keyword Str keyword Str Expr Call AttributeLoad NameLoad attr TupleLoad Expr Call AttributeLoad NameLoad attr keyword Str Expr Call AttributeLoad NameLoad attr keyword Num Expr Call AttributeLoad NameLoad attr keyword Num Expr Call AttributeLoad NameLoad attr keyword Num For TupleStore NameStore NameStore Call NameLoad NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr Num NameLoad NameLoad Expr Call AttributeLoad NameLoad attr
N/A N/A 
======================================================
Classification of text documents using sparse features
======================================================

This is an example showing how scikit-learn can be used to classify documents
by topics using a bag-of-words approach. This example uses a scipy.sparse
matrix to store the features and demonstrates various classifiers that can
efficiently handle sparse matrices.

The dataset used in this example is the 20 newsgroups dataset. It will be
automatically downloaded, then cached.

The bar plot indicates the accuracy, training time (normalized) and test time
(normalized) of each classifier.

 __future__ print_function N/A logging N/A numpy np optparse OptionParser N/A sys time time N/A matplotlib.pyplot plt sklearn.datasets fetch_20newsgroups sklearn.feature_extraction.text TfidfVectorizer sklearn.feature_extraction.text HashingVectorizer sklearn.feature_selection SelectKBest chi2 sklearn.linear_model RidgeClassifier sklearn.pipeline Pipeline sklearn.svm LinearSVC sklearn.linear_model SGDClassifier sklearn.linear_model Perceptron sklearn.linear_model PassiveAggressiveClassifier sklearn.naive_bayes BernoulliNB MultinomialNB sklearn.neighbors KNeighborsClassifier sklearn.neighbors NearestCentroid sklearn.ensemble RandomForestClassifier sklearn.utils.extmath density sklearn metrics N/A N/A N/A logging basicConfig level N/A logging INFO format %(asctime)s %(levelname)s %(message)s N/A op N/A OptionParser N/A N/A N/A op add_option --report action store_true dest print_report help Print a detailed classification report. N/A N/A N/A op add_option --chi2_select action store type int dest select_chi2 help Select some number of features using a chi-squared test N/A N/A N/A op add_option --confusion_matrix action store_true dest print_cm help Print the confusion matrix. N/A N/A N/A op add_option --top10 action store_true dest print_top10 help Print ten most discriminative terms per class for every classifier. N/A N/A N/A op add_option --all_categories action store_true dest all_categories help Whether to use all categories or not. N/A N/A N/A op add_option --use_hashing action store_true help Use a hashing vectorizer. N/A N/A N/A op add_option --n_features action store type int default N/A 2 16 help n_features when using the hashing vectorizer. N/A N/A N/A op add_option --filtered action store_true help Remove newsgroup information that is easily overfit: headers, signatures, and quoting. N/A N/A opts args N/A N/A op parse_args N/A N/A N/A len args 0 N/A N/A N/A N/A op error this script takes no arguments. N/A N/A N/A sys exit 1 N/A N/A print __doc__ N/A N/A N/A op print_help N/A N/A print N/A N/A opts all_categories N/A N/A categories None N/A N/A categories N/A alt.atheism talk.religion.misc comp.graphics sci.space N/A N/A opts filtered N/A N/A remove N/A headers footers quotes N/A N/A remove N/A N/A N/A print Loading 20 newsgroups dataset for categories: N/A N/A print N/A categories categories all N/A data_train N/A fetch_20newsgroups subset train categories categories shuffle True random_state 42 remove remove N/A data_test N/A fetch_20newsgroups subset test categories categories shuffle True random_state 42 remove remove N/A N/A print data loaded N/A categories N/A data_train target_names size_mb N/A N/A docs N/A N/A N/A N/A N/A sum N/A N/A len N/A N/A s encode utf-8 N/A s docs 1000000.0 N/A N/A data_train_size_mb N/A size_mb N/A data_train data N/A data_test_size_mb N/A size_mb N/A data_test data N/A N/A print N/A %d documents - %0.3fMB (training set) N/A N/A len N/A data_train data data_train_size_mb N/A N/A print N/A %d documents - %0.3fMB (test set) N/A N/A len N/A data_test data data_test_size_mb N/A N/A print N/A %d categories N/A len categories N/A N/A print N/A N/A y_train y_test N/A N/A data_train target N/A data_test target N/A N/A print Extracting features from the training data using a sparse vectorizer N/A t0 N/A time N/A N/A opts use_hashing N/A N/A vectorizer N/A HashingVectorizer stop_words english non_negative True n_features N/A opts n_features N/A X_train N/A N/A vectorizer transform N/A data_train data N/A N/A vectorizer N/A TfidfVectorizer sublinear_tf True max_df 0.5 stop_words english N/A X_train N/A N/A vectorizer fit_transform N/A data_train data N/A duration N/A N/A time t0 N/A N/A print N/A done in %fs at %0.3fMB/s N/A duration N/A data_train_size_mb duration N/A N/A print N/A n_samples: %d, n_features: %d N/A X_train shape N/A N/A print N/A N/A print Extracting features from the test data using the same vectorizer N/A t0 N/A time N/A X_test N/A N/A vectorizer transform N/A data_test data N/A duration N/A N/A time t0 N/A N/A print N/A done in %fs at %0.3fMB/s N/A duration N/A data_test_size_mb duration N/A N/A print N/A n_samples: %d, n_features: %d N/A X_test shape N/A N/A print N/A N/A opts use_hashing N/A N/A feature_names None N/A N/A feature_names N/A N/A vectorizer get_feature_names N/A N/A opts select_chi2 N/A N/A N/A print N/A Extracting %d best features by a chi-squared test N/A opts select_chi2 N/A t0 N/A time N/A ch2 N/A SelectKBest chi2 k N/A opts select_chi2 N/A X_train N/A N/A ch2 fit_transform X_train y_train N/A X_test N/A N/A ch2 transform X_test N/A feature_names N/A N/A feature_names N/A N/A feature_names N/A i N/A i N/A N/A ch2 get_support indices True N/A N/A print N/A done in %fs N/A N/A time t0 N/A N/A print N/A feature_names N/A N/A feature_names N/A N/A np asarray feature_names trim N/A N/A s N/A N/A N/A Trim string to fit on terminal (assuming 80-column display) N/A N/A N/A N/A len s 80 s N/A N/A s N/A 77 ... N/A benchmark N/A N/A clf N/A N/A N/A N/A print N/A _ 80 N/A N/A print Training:  N/A N/A print clf N/A t0 N/A time N/A N/A N/A clf fit X_train y_train N/A train_time N/A N/A time t0 N/A N/A print N/A train time: %0.3fs train_time N/A t0 N/A time N/A pred N/A N/A clf predict X_test N/A test_time N/A N/A time t0 N/A N/A print N/A test time:  %0.3fs test_time N/A score N/A N/A metrics accuracy_score y_test pred N/A N/A print N/A accuracy:   %0.3f score N/A N/A hasattr clf coef_ N/A N/A N/A print N/A dimensionality: %d N/A N/A N/A clf coef_ shape N/A 1 N/A N/A print N/A density: %f N/A density N/A clf coef_ N/A N/A N/A opts print_top10 N/A feature_names None N/A N/A N/A print top 10 keywords per class: N/A N/A i category N/A enumerate categories N/A N/A top10 N/A N/A N/A np argsort N/A N/A clf coef_ N/A i N/A -10 N/A N/A print N/A trim N/A %s: %s N/A category N/A N/A join N/A feature_names N/A top10 N/A N/A print N/A N/A opts print_report N/A N/A N/A print classification report: N/A N/A print N/A N/A metrics classification_report y_test pred target_names categories N/A N/A opts print_cm N/A N/A N/A print confusion matrix: N/A N/A print N/A N/A metrics confusion_matrix y_test pred N/A N/A print N/A clf_descr N/A N/A N/A N/A str clf split ( N/A 0 N/A N/A clf_descr score train_time test_time N/A N/A results N/A N/A N/A clf name N/A N/A N/A RidgeClassifier tol 0.01 solver lsqr Ridge Classifier N/A N/A Perceptron n_iter 50 Perceptron N/A N/A PassiveAggressiveClassifier n_iter 50 Passive-Aggressive N/A N/A KNeighborsClassifier n_neighbors 10 kNN N/A N/A RandomForestClassifier n_estimators 100 Random forest N/A N/A N/A print N/A = 80 N/A N/A print name N/A N/A N/A results append N/A benchmark clf N/A penalty N/A l2 l1 N/A N/A N/A print N/A = 80 N/A N/A print N/A %s penalty N/A N/A penalty upper N/A N/A N/A results append N/A benchmark N/A LinearSVC loss l2 penalty penalty dual False tol 0.001 N/A N/A N/A results append N/A benchmark N/A SGDClassifier alpha 0.0001 n_iter 50 penalty penalty N/A N/A print N/A = 80 N/A N/A print Elastic-Net penalty N/A N/A N/A results append N/A benchmark N/A SGDClassifier alpha 0.0001 n_iter 50 penalty elasticnet N/A N/A print N/A = 80 N/A N/A print NearestCentroid (aka Rocchio classifier) N/A N/A N/A results append N/A benchmark N/A NearestCentroid N/A N/A print N/A = 80 N/A N/A print Naive Bayes N/A N/A N/A results append N/A benchmark N/A MultinomialNB alpha 0.01 N/A N/A N/A results append N/A benchmark N/A BernoulliNB alpha 0.01 N/A N/A print N/A = 80 N/A N/A print LinearSVC with L1-based feature selection N/A N/A N/A results append N/A benchmark N/A Pipeline N/A N/A feature_selection N/A LinearSVC penalty l1 dual False tol 0.001 N/A classification N/A LinearSVC N/A indices N/A N/A np arange N/A len results N/A results N/A N/A N/A x N/A i N/A x results N/A i N/A range 4 N/A N/A clf_names score training_time test_time results N/A training_time N/A N/A N/A np array training_time N/A N/A np max training_time N/A test_time N/A N/A N/A np array test_time N/A N/A np max test_time N/A N/A N/A plt figure figsize N/A 12 8 N/A N/A N/A plt title Score N/A N/A N/A plt barh indices score 0.2 label score color navy N/A N/A N/A plt barh N/A indices 0.3 training_time 0.2 label training time color c N/A N/A N/A plt barh N/A indices 0.6 test_time 0.2 label test time color darkorange N/A N/A N/A plt yticks N/A N/A N/A N/A plt legend loc best N/A N/A N/A plt subplots_adjust left 0.25 N/A N/A N/A plt subplots_adjust top 0.95 N/A N/A N/A plt subplots_adjust bottom 0.05 N/A N/A i c N/A zip indices clf_names N/A N/A N/A N/A plt text -0.3 i c N/A N/A N/A plt show