Module Expr Str Import alias Import alias If BoolOpAnd CompareEq NameLoad Str CompareIs NameLoad NameLoad body Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Expr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Import alias identifier Import alias identifier ImportFrom alias ImportFrom alias Import alias ImportFrom alias alias ImportFrom alias alias alias alias alias ImportFrom alias ImportFrom alias alias ImportFrom alias Assign NameStore Call NameLoad FunctionDef arguments args NameParam NameParam defaults body Expr Str Assign NameStore Call NameLoad If CompareIsNot NameLoad NameLoad body For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body Assign SubscriptStore NameLoad Index NameLoad Call NameLoad Call NameLoad NameLoad Return Call Call NameLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam defaults body Expr Str Assign NameStore ListLoad For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad SubscriptLoad NameLoad ExtSlice Slice Index NameLoad Expr Call AttributeLoad NameLoad attr SubscriptLoad Call AttributeLoad NameLoad attr NameLoad ExtSlice Slice Index Num Return NameLoad decorator_list Assign NameStore Call AttributeLoad NameLoad attr Call NameLoad SubscriptLoad AttributeLoad NameLoad attr Index Num For TupleStore NameStore NameStore Call AttributeLoad SubscriptLoad NameLoad Index Str attr body Assign NameStore BinOpAdd BinOpAdd BinOpAdd Str NameLoad Str Call AttributeLoad Str attr NameLoad Exec NameLoad Assign NameStore SubscriptLoad SubscriptLoad NameLoad Index Str Index Str Assign NameStore SubscriptLoad SubscriptLoad NameLoad Index Str Index Str Assign NameStore SubscriptLoad SubscriptLoad NameLoad Index Str Index Str Assign NameStore SubscriptLoad SubscriptLoad NameLoad Index Str Index Str If CompareIn Str Call AttributeLoad SubscriptLoad NameLoad Index Str attr body Assign NameStore SubscriptLoad SubscriptLoad NameLoad Index Str Index Str orelse Assign NameStore Num If CompareIn Str SubscriptLoad NameLoad Index Str body Assign NameStore SubscriptLoad SubscriptLoad NameLoad Index Str Index Str orelse Assign NameStore ListLoad Assign NameStore ListLoad For NameStore SubscriptLoad NameLoad Index Str body For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr Call NameLoad NameLoad NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore NameLoad If CompareIn Str Call AttributeLoad NameLoad attr body Assign NameStore ListLoad For NameStore SubscriptLoad NameLoad Index Str body For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr Call NameLoad NameLoad NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore ListLoad For NameStore SubscriptLoad NameLoad Index Str body Expr Call AttributeLoad NameLoad attr BinOpMod Str NameLoad For TupleStore NameStore NameStore Call NameLoad NameLoad body Assign SubscriptStore NameLoad Index NameLoad Call NameLoad NameLoad Assign NameStore SubscriptLoad AttributeLoad NameLoad attr Index Num If CompareEq NameLoad Str body Assign NameStore NameLoad orelse If CompareEq NameLoad Str body Assign NameStore NameLoad orelse Raise Str If NameLoad body Assign NameStore Str Assign NameStore Str orelse Assign NameStore Str Assign NameStore Str Print BinOpMod Str TupleLoad NameLoad NameLoad Assign NameStore BinOpAdd NameLoad NameLoad If UnaryOpNot Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad Num Num Assign NameStore ListLoad Str Call NameLoad Str Call NameLoad keyword Call NameLoad Str Call NameLoad Str Assign NameStore Call NameLoad keyword NameLoad keyword Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr keyword ListLoad NameLoad Assign NameStore Call NameLoad For NameStore NameLoad body Print BinOpMod Str NameLoad Assign TupleStore NameStore NameStore NameStore NameStore Call NameLoad NameLoad NameLoad Assign NameStore NameLoad Assign NameStore NameLoad Assign NameStore BinOpMod Str TupleLoad NameLoad NameLoad Assign NameStore Call NameLoad NameLoad If CompareIsNot NameLoad NameLoad body Assign NameStore Call NameLoad NameLoad orelse Assign NameStore NameLoad Print Str If BoolOpAnd NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad orelse Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad If NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore NameLoad Assign NameStore NameLoad Print Str Assign NameStore BinOpMod Str TupleLoad NameLoad NameLoad For TupleStore NameStore NameStore AttributeLoad NameLoad attr body If Call NameLoad NameLoad Str body Expr Call AttributeLoad NameLoad attr NameLoad NameLoad If BoolOpAnd NameLoad Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad orelse Assign NameStore Call AttributeLoad NameLoad attr NameLoad If NameLoad body Expr Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore NameLoad Print Str If BoolOpAnd NameLoad CompareIs NameLoad NameLoad body Assign NameStore BinOpMod Str TupleLoad NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad For NameStore NameLoad body Assign NameStore BinOpMod Str TupleLoad NameLoad NameLoad NameLoad If Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad If CompareIs NameLoad NameLoad body Assign NameStore NameLoad orelse Assign NameStore SubscriptLoad AttributeLoad NameLoad attr Index TupleLoad NameLoad NameLoad Assign NameStore NameLoad orelse Raise Call NameLoad BinOpMod Str NameLoad For NameStore NameLoad body Assign NameStore BinOpMod Str TupleLoad NameLoad NameLoad NameLoad If Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad If CompareIs NameLoad NameLoad body Assign NameStore NameLoad orelse Assign NameStore SubscriptLoad AttributeLoad NameLoad attr Index TupleLoad NameLoad NameLoad Assign NameStore NameLoad orelse Raise Call NameLoad BinOpMod Str NameLoad Assign SubscriptStore NameLoad Index Call AttributeLoad NameLoad attr NameLoad Num Assign SubscriptStore NameLoad Index Call AttributeLoad NameLoad attr NameLoad Num If CompareIsNot NameLoad NameLoad body Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad For TupleStore NameStore NameStore AttributeLoad NameLoad attr body If Call NameLoad NameLoad Str body Expr Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Print Str Assign NameStore SubscriptLoad NameLoad Slice NameLoad If CompareEq NameLoad Num body Assign NameStore ListComp Call NameLoad NameLoad comprehension NameStore Call NameLoad Call NameLoad NameLoad orelse Assign NameStore Call NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Call NameLoad Call NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Print Str For NameStore Call NameLoad Call NameLoad NameLoad body Assign NameStore ListComp SubscriptLoad NameLoad Index NameLoad comprehension NameStore NameLoad Assign NameStore Call AttributeLoad Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad attr Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad NameLoad NameLoad If UnaryOpNot NameLoad body Assign NameStore Call AttributeLoad NameLoad attr ListComp Call NameLoad NameLoad NameLoad comprehension TupleStore NameStore NameStore Call NameLoad AttributeLoad SubscriptLoad NameLoad Slice NameLoad attr AttributeLoad NameLoad attr Print BinOpMod Str TupleLoad NameLoad NameLoad NameLoad Assign NameStore NameLoad Assign NameStore NameLoad Assign NameStore NameLoad Expr Call AttributeLoad NameLoad attr NameLoad If UnaryOpNot NameLoad body Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr Str ExtSlice Slice Slice Num Assign NameStore ListLoad For NameStore Call NameLoad Call NameLoad NameLoad body Expr Call AttributeLoad NameLoad attr ListLoad For NameStore NameLoad body Expr Call AttributeLoad SubscriptLoad NameLoad Index NameLoad attr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad NameLoad Assign SubscriptStore NameLoad Index NameLoad Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad If UnaryOpNot NameLoad body Assign NameStore ListComp Call NameLoad NameLoad NameLoad comprehension TupleStore NameStore NameStore Call NameLoad AttributeLoad SubscriptLoad NameLoad Slice NameLoad attr AttributeLoad SubscriptLoad NameLoad Index NameLoad attr Assign SubscriptStore NameLoad Index Str Call AttributeLoad NameLoad attr NameLoad Print Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr BinOpAdd BinOpAdd BinOpAdd NameLoad NameLoad NameLoad Str NameLoad Assign NameStore Call NameLoad Assign SubscriptStore NameLoad Index Str BinOpSub NameLoad NameLoad Expr Call AttributeLoad NameLoad attr BinOpMod Str TupleLoad NameLoad NameLoad Print NameLoad
N/A N/A 
Created on Wed Jul  8 21:56:55 2015.

@author: alex, rc

This script contain code to generate lvl1 model prediction.
usage : python genPreds.py model_name mode
with mode = val for validation and val = test for test.

This script will read the model description from the yaml file, load
dependencies, create preprocessing and classification pipeline and apply them
on raw data independently on each subjects.

This script support caching of preprocessed data, in order to allow reuse of
preprocessing pipeline across model.
 N/A os N/A sys N/A N/A N/A __name__ __main__ N/A __package__ None N/A N/A filePath N/A N/A N/A os path dirname N/A N/A N/A os path dirname N/A N/A N/A os path abspath __file__ N/A N/A N/A N/A sys path append filePath N/A numpy np N/A pandas pd time time copy deepcopy N/A yaml sklearn.pipeline make_pipeline Pipeline progressbar Bar ETA Percentage ProgressBar RotatingMarker sklearn.metrics roc_auc_score preprocessing.aux getEventNames load_raw_data multiprocessing Pool N/A cols N/A getEventNames _from_yaml_to_func N/A N/A method params N/A N/A N/A go from yaml to method.

    Need to be here for accesing local variables.
     N/A prm N/A dict N/A N/A params None N/A N/A N/A key val N/A N/A params iteritems N/A N/A N/A prm N/A key N/A eval N/A str val N/A N/A N/A eval method prm N/A doCols N/A N/A col N/A N/A N/A Train and Predict for one event. N/A p N/A N/A clf clfs N/A N/A N/A N/A clf fit trainPreprocessed N/A labels_train N/A N/A N/A col N/A N/A N/A p append N/A N/A N/A clf predict_proba testPreprocessed N/A N/A N/A 1 N/A p N/A N/A yml N/A N/A yaml load N/A open N/A N/A sys argv N/A 1 N/A N/A pkg functions N/A N/A N/A yml N/A imports iteritems N/A N/A stri N/A N/A N/A from  pkg  import  N/A N/A , join functions N/A stri N/A fileName N/A N/A yml N/A Meta N/A file N/A cores N/A N/A yml N/A Meta N/A cores N/A subsample N/A N/A yml N/A Meta N/A subsample N/A cache_preprocessed N/A N/A yml N/A Meta N/A cachePreprocessed N/A N/A subsample_test N/A N/A N/A yml N/A Meta keys N/A N/A subsample_test N/A N/A yml N/A Meta N/A subsample_test N/A N/A subsample_test 1 N/A N/A addPreprocessed N/A yml N/A Meta N/A N/A addPreprocessed N/A N/A yml N/A Meta N/A addPreprocessed N/A N/A addPreprocessed N/A N/A pipe N/A N/A item N/A yml N/A Preprocessing N/A N/A N/A method params N/A N/A item iteritems N/A N/A N/A N/A pipe append N/A _from_yaml_to_func method params N/A preprocess_base N/A make_pipeline pipe N/A postpreprocess_base None N/A N/A PostPreprocessing N/A N/A yml keys N/A N/A pipe N/A N/A item N/A yml N/A PostPreprocessing N/A N/A N/A method params N/A N/A item iteritems N/A N/A N/A N/A pipe append N/A _from_yaml_to_func method params N/A postpreprocess_base N/A make_pipeline pipe N/A clfs N/A N/A mdl N/A yml N/A Models N/A N/A N/A N/A clfs append N/A Pipeline([ %s ]) mdl N/A N/A i clf N/A enumerate clfs N/A N/A N/A clfs N/A i N/A eval clf N/A mode N/A N/A sys argv N/A 2 N/A N/A mode val N/A N/A test False N/A N/A N/A mode test N/A N/A test True N/A N/A Invalid mode. Please specify either val or test N/A test N/A N/A folder test/ N/A prefix test_ N/A N/A folder val/ N/A prefix val_ N/A N/A Running %s, to be saved in file %s N/A mode fileName N/A saveFolder N/A folder fileName N/A N/A N/A N/A N/A os path exists saveFolder N/A N/A N/A N/A os makedirs saveFolder N/A subjects N/A range 1 13 N/A widgets N/A Cross Val :  N/A Percentage N/A Bar marker N/A RotatingMarker N/A ETA N/A pbar N/A ProgressBar widgets widgets maxval N/A len subjects N/A N/A N/A pbar start N/A report N/A N/A pd DataFrame index N/A fileName N/A start_time N/A time N/A subject subjects N/A N/A N/A Loading data for subject %d... subject N/A N/A data_train labels_train data_test labels_test N/A load_raw_data subject test N/A trainPreprocessed None N/A testPreprocessed None N/A cacheFile N/A %s/train_sub%d.npy N/A saveFolder subject N/A preprocess N/A deepcopy preprocess_base N/A N/A postpreprocess_base None N/A N/A postpreprocess N/A deepcopy postpreprocess_base N/A N/A postpreprocess None N/A Preprocessing Training data... N/A N/A cache_preprocessed N/A N/A N/A os path isfile cacheFile N/A N/A trainPreprocessed N/A N/A np load cacheFile N/A N/A trainPreprocessed N/A N/A preprocess fit_transform data_train labels_train N/A cache_preprocessed N/A N/A N/A N/A np save cacheFile trainPreprocessed N/A trainPreprocessed None N/A data_train None N/A Preprocessing Test data... N/A cacheFile N/A %s/test_sub%d.npy N/A saveFolder subject N/A N/A name step N/A preprocess steps N/A N/A N/A hasattr step update_subsample N/A N/A N/A N/A step update_subsample subsample subsample_test N/A N/A cache_preprocessed N/A N/A N/A os path isfile cacheFile N/A N/A testPreprocessed N/A N/A np load cacheFile N/A N/A testPreprocessed N/A N/A preprocess transform data_test N/A cache_preprocessed N/A N/A N/A N/A np save cacheFile testPreprocessed N/A data_test None N/A Post Preprocessing data... N/A N/A cache_preprocessed N/A trainPreprocessed None N/A N/A cacheFile N/A %s/train_sub%d.npy N/A saveFolder subject N/A trainPreprocessed N/A N/A np load cacheFile N/A feat_name addPreprocessed N/A N/A featFile N/A %s/%s/train_sub%d.npy N/A folder feat_name subject N/A N/A N/A N/A os path isfile featFile N/A N/A feat N/A N/A np load featFile N/A N/A trainPreprocessed None N/A N/A trainPreprocessed feat N/A N/A trainPreprocessed N/A N/A np c_ N/A N/A trainPreprocessed feat N/A feat None N/A N/A N/A ValueError N/A File %s does not exist featFile N/A feat_name addPreprocessed N/A N/A featFile N/A %s/%s/test_sub%d.npy N/A folder feat_name subject N/A N/A N/A N/A os path isfile featFile N/A N/A feat N/A N/A np load featFile N/A N/A testPreprocessed None N/A N/A testPreprocessed feat N/A N/A testPreprocessed N/A N/A np c_ N/A N/A testPreprocessed feat N/A feat None N/A N/A N/A ValueError N/A File %s does not exist featFile N/A N/A trainPreprocessed N/A N/A N/A np isnan trainPreprocessed 0 N/A N/A testPreprocessed N/A N/A N/A np isnan testPreprocessed 0 N/A N/A postpreprocess None N/A N/A trainPreprocessed N/A N/A postpreprocess fit_transform trainPreprocessed labels_train N/A N/A name step N/A postpreprocess steps N/A N/A N/A hasattr step update_subsample N/A N/A N/A N/A step update_subsample subsample subsample_test N/A testPreprocessed N/A N/A postpreprocess transform testPreprocessed N/A Training models... N/A labels_train N/A labels_train N/A subsample N/A N/A cores 1 N/A N/A preds N/A N/A doCols i N/A i N/A range N/A len cols N/A N/A pool N/A Pool processes cores N/A preds N/A N/A pool map doCols N/A range N/A len cols N/A N/A N/A pool close N/A Aggregating results... N/A i N/A range N/A len clfs N/A N/A pred_i N/A N/A j N/A i N/A j preds N/A pred_i N/A N/A N/A N/A np array N/A N/A np vstack pred_i transpose N/A N/A N/A np save N/A %s/sub%d_clf%d.npy N/A saveFolder subject i pred_i N/A N/A test N/A N/A auc N/A N/A np mean N/A N/A roc_auc_score trueVals p N/A N/A trueVals p N/A zip N/A N/A labels_test N/A subsample_test T N/A pred_i T N/A N/A %d, clf %d: %.5f N/A subject i auc N/A preds None N/A trainPreprocessed None N/A testPreprocessed None N/A N/A N/A pbar update subject N/A N/A test N/A N/A labels N/A N/A N/A np load ../infos_val.npy N/A N/A N/A -2 N/A preds_tot N/A N/A i N/A range N/A len clfs N/A N/A N/A N/A preds_tot append N/A N/A subject subjects N/A N/A N/A N/A N/A preds_tot N/A i append N/A N/A np load N/A %s/sub%d_clf%d.npy N/A saveFolder subject i N/A N/A preds_tot N/A i N/A N/A np concatenate N/A preds_tot N/A i N/A N/A test N/A N/A auc N/A N/A roc_auc_score trueVals p N/A N/A trueVals p N/A zip N/A N/A labels N/A subsample_test T N/A N/A preds_tot N/A i T N/A N/A report N/A AUC N/A N/A np mean auc N/A N/A N/A np mean auc N/A N/A N/A np save N/A N/A N/A folder prefix fileName .npy preds_tot N/A end_time N/A time N/A N/A report N/A Time N/A end_time start_time N/A N/A N/A report to_csv N/A report/%s_%s.csv N/A prefix fileName N/A report