Module Expr Str ImportFrom alias alias Import alias Import alias Import alias Import alias ImportFrom alias ImportFrom alias ImportFrom alias alias Assign NameStore Str Assign NameStore ListLoad Str FunctionDef arguments args NameParam NameParam NameParam defaults body Expr Str Assign NameStore BinOpDiv NameLoad Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad AugAssignDiv NameStore Call AttributeLoad NameLoad attr keyword Num keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr keyword Call AttributeLoad NameLoad attr Return BinOpMult NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam NameParam defaults Num Num Num NameLoad AttributeLoad NameLoad attr body Expr Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad NameLoad keyword NameLoad AugAssignDiv NameStore Call AttributeLoad NameLoad attr NameLoad If CompareLtE NameLoad Num body Return TupleLoad NameLoad Call NameLoad NameLoad Assert CompareGt NameLoad Num Str Assign NameStore Call NameLoad keyword NameLoad keyword Str Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore SubscriptLoad Call AttributeLoad NameLoad attr NameLoad keyword NameLoad ExtSlice Slice Slice Num Assign NameStore Call AttributeLoad NameLoad attr Call NameLoad NameLoad keyword NameLoad For NameStore Call NameLoad NameLoad body Assign NameStore Call NameLoad NameLoad NameLoad NameLoad AugAssignMult NameStore BinOpPow BinOpDiv NameLoad NameLoad Num Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword NameLoad For TupleStore NameStore NameStore Call NameLoad NameLoad body Assign NameStore SubscriptLoad NameLoad Index NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad NameLoad AugAssignDiv NameStore Call AttributeLoad NameLoad attr NameLoad Assign SubscriptStore NameLoad Index NameLoad Call AttributeLoad NameLoad attr NameLoad keyword NameLoad Assign NameStore AttributeLoad NameLoad attr Assign NameStore BinOpMult Num BinOpPow Call AttributeLoad NameLoad attr NameLoad NameLoad Assign NameStore BinOpSub Num NameLoad Assign NameStore BinOpAdd BinOpMult SubscriptLoad NameLoad ExtSlice Slice Index AttributeLoad NameLoad attr SubscriptLoad NameLoad ExtSlice Index NameLoad Slice BinOpMult SubscriptLoad NameLoad ExtSlice Slice Index AttributeLoad NameLoad attr SubscriptLoad NameLoad ExtSlice Index NameLoad Slice Return TupleLoad Call AttributeLoad NameLoad attr NameLoad keyword AttributeLoad NameLoad attr Num decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam defaults body Expr Str Assign TupleStore NameStore NameStore NameLoad Assign TupleStore NameStore NameStore Call NameLoad SubscriptLoad NameLoad Index NameLoad Call NameLoad BinOpMult Call NameLoad NameLoad NameLoad keyword NameLoad For TupleStore NameStore NameStore Call NameLoad NameLoad body Assign SubscriptStore NameLoad Index NameLoad SubscriptLoad NameLoad Index NameLoad Return TupleLoad NameLoad NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults Num NameLoad NameLoad body Expr Str If CompareIs NameLoad NameLoad body Assign NameStore ListLoad Assign NameStore ListComp NameLoad comprehension NameStore AttributeLoad NameLoad attr CompareNotIn NameLoad NameLoad Assign NameStore BinOpDiv Call NameLoad NameLoad Call NameLoad NameLoad Assign NameStore Num If CompareEq Call NameLoad NameLoad Num body Assign TupleStore NameStore NameStore Call NameLoad NameLoad keyword NameLoad keyword NameLoad orelse Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call NameLoad Str Assign NameStore Call NameLoad NameLoad Assign NameStore Call NameLoad NameLoad NameLoad NameLoad BinOpMult ListLoad NameLoad NameLoad BinOpMult ListLoad NameLoad NameLoad BinOpMult ListLoad NameLoad NameLoad Assign TupleStore NameStore NameStore Call NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr CompareNotEq Call AttributeLoad NameLoad attr NameLoad Num Assign NameStore Call AttributeLoad NameLoad attr NameLoad If CompareGt NameLoad Num body Expr Call NameLoad BinOpMod Str TupleLoad NameLoad NameLoad Call NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults NameLoad body Expr Str Assert Call NameLoad NameLoad AttributeLoad NameLoad attr Str ImportFrom alias FunctionDef arguments args NameParam defaults body Expr Call NameLoad Call AttributeLoad Str attr NameLoad keyword NameLoad decorator_list Assign NameStore Call NameLoad NameLoad Call NameLoad NameLoad keyword NameLoad Expr Call AttributeLoad NameLoad attr keyword Num keyword NameLoad Assign NameStore Num Assign NameStore BinOpFloorDiv BinOpSub BinOpAdd Call NameLoad AttributeLoad NameLoad attr NameLoad Num NameLoad Expr Call AttributeLoad NameLoad attr keyword TupleLoad Num BinOpMult Num NameLoad For TupleStore NameStore NameStore Call NameLoad AttributeLoad NameLoad attr Num body Expr Call AttributeLoad NameLoad attr NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr ListLoad AttributeLoad SubscriptLoad NameLoad Index NameLoad attr AttributeLoad SubscriptLoad NameLoad Index NameLoad attr keyword Str keyword Num Expr Call AttributeLoad NameLoad attr Expr Call NameLoad Str Assign NameStore ListLoad Assign NameStore ListLoad For NameStore AttributeLoad NameLoad attr body Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad Assign NameStore Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad Assign NameStore Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad Assign NameStore Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad Expr Call AttributeLoad NameLoad attr ListLoad NameLoad NameLoad BinOpSub NameLoad NameLoad Call NameLoad BinOpDiv BinOpMult BinOpSub NameLoad NameLoad Num NameLoad NameLoad NameLoad BinOpSub NameLoad NameLoad Call NameLoad BinOpDiv BinOpMult BinOpSub NameLoad NameLoad Num NameLoad Expr Call NameLoad Call AttributeLoad NameLoad attr NameLoad keyword NameLoad keyword ListLoad Str Str Str Str Str Str Str Str Expr Call NameLoad Str Assign NameStore ListLoad Assign NameStore ListLoad For TupleStore NameStore NameStore Call NameLoad AttributeLoad NameLoad attr body For NameStore SubscriptLoad AttributeLoad NameLoad attr Slice BinOpAdd NameLoad Num body Expr Call AttributeLoad NameLoad attr TupleLoad NameLoad NameLoad Assign NameStore SubscriptLoad Call NameLoad SubscriptLoad NameLoad Index NameLoad SubscriptLoad NameLoad Index NameLoad Index Num Assign NameStore SubscriptLoad Call NameLoad SubscriptLoad NameLoad Index NameLoad SubscriptLoad NameLoad Index NameLoad Index Num Expr Call AttributeLoad NameLoad attr ListLoad NameLoad NameLoad BinOpSub NameLoad NameLoad Call NameLoad BinOpDiv BinOpMult BinOpSub NameLoad NameLoad Num NameLoad Expr Call NameLoad Call AttributeLoad NameLoad attr NameLoad keyword NameLoad keyword ListLoad Str Str Str Str For TupleStore NameStore NameStore Call NameLoad SubscriptLoad AttributeLoad NameLoad attr Slice Num body For NameStore SubscriptLoad AttributeLoad NameLoad attr Slice BinOpAdd NameLoad Num Num body Assign NameStore Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad Assign NameStore Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad Assign NameStore Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad Assign NameStore Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad Expr Call AttributeLoad NameLoad attr keyword TupleLoad Num Num Expr Call AttributeLoad NameLoad attr Num Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad SubscriptLoad NameLoad Index NameLoad Str keyword Num Expr TupleLoad Call AttributeLoad NameLoad attr NameLoad NameLoad Call AttributeLoad NameLoad attr NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Num Expr Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index NameLoad SubscriptLoad NameLoad Index NameLoad Str keyword Num Expr TupleLoad Call AttributeLoad NameLoad attr NameLoad NameLoad Call AttributeLoad NameLoad attr NameLoad NameLoad Expr Call AttributeLoad NameLoad attr Str Expr Call AttributeLoad NameLoad attr Call AttributeLoad Str attr NameLoad NameLoad Expr Call AttributeLoad NameLoad attr decorator_list
N/A N/A 
This module contains procedures to generate toy Monte-Carlo (toyMC)
by using modified SMOTE approach.

Mathematically, toyMC generation is like trying to generate data
from same distribution as you already have.

SMOTE approach relies on assumption that events close in euclidean metrics
in dataset are really close.

The more data -> the better toyMC.
 __future__ division print_function N/A math N/A numpy N/A pandas N/A pylab scipy.stats.stats pearsonr sklearn.neighbors NearestNeighbors commonutils map_on_cluster check_sample_weight N/A __author__ Alex Rogozhnikov N/A __all__ N/A generate_toymc_with_special_features _count_probabilities N/A N/A primary_weights secondary_weights knn N/A N/A N/A Computes probabilities of all points to be chosen as the second point in pair
    :type primary_weights: numpy.array, shape = [n_samples],
        the first event is generated according to these weights
    :type secondary_weights: numpy.array, shape = [n_samples],
        the second event is chosen between knn of first according to this weights
    :type knn: dict, {event_id: list of neighbors ids}.
    :rtype: numpy.array, shape = [n_samples], the probabilities
     N/A primary_probabilities N/A primary_weights N/A N/A numpy sum primary_weights N/A secondary_weights N/A N/A numpy array secondary_weights N/A knn_probabilities N/A N/A numpy take secondary_weights knn N/A knn_probabilities N/A N/A knn_probabilities sum axis 1 keepdims True N/A secondary_probabilities N/A N/A numpy bincount N/A N/A knn flatten weights N/A N/A knn_probabilities flatten N/A N/A primary_probabilities secondary_probabilities N/A generate_toymc N/A N/A data size knn power reweighting_iterations sample_weight random_state N/A 4 2.0 5 None N/A numpy random N/A N/A Generates toy Monte-Carlo, the dataset with distribution very close to the original one.

    :type data: numpy.array | pandas.DataFrame, the original distribution
    :type size: int, the number of events to generate
    :type knn: int | None, how many neighbours should we consider
    :type power: float, instead of uniform distribution, makes the point to tend to one of initial points
        (the greater, the closer to first point)
    :type reweighting_iterations: int, an iterative algorithm is used, which changes the probabilities
        so that all the points have equal probability to be chosen as neighbour
    :rtype: (pandas.DataFrame, int), returns the generated toymc and the number of events
        that were simply copied from original data set.
     N/A data N/A N/A pandas DataFrame data N/A input_length N/A len data N/A sample_weight N/A check_sample_weight data sample_weight sample_weight N/A sample_weight N/A N/A numpy sum sample_weight N/A N/A input_length 2 N/A N/A N/A data N/A len data N/A N/A knn 0 knn should be positive N/A neighbors_helper N/A NearestNeighbors n_neighbors knn algorithm ball_tree N/A N/A N/A neighbors_helper fit data N/A neighbours N/A N/A N/A neighbors_helper kneighbors data return_distance False N/A N/A N/A 1 N/A secondary_weights N/A N/A numpy ones N/A len neighbours dtype float N/A _ N/A range reweighting_iterations N/A N/A probabilities N/A _count_probabilities sample_weight secondary_weights neighbours N/A secondary_weights N/A N/A sample_weight probabilities 0.5 N/A k_1 N/A N/A random_state choice input_length p sample_weight size size N/A k_2 N/A N/A numpy zeros size dtype int N/A N/A i first_neighbour N/A enumerate k_1 N/A N/A neighs N/A neighbours N/A first_neighbour N/A neigh_weights N/A N/A numpy take secondary_weights neighs N/A neigh_weights N/A N/A numpy sum neigh_weights N/A N/A k_2 N/A i N/A N/A random_state choice neighs p neigh_weights N/A numpied_df N/A data values N/A t_1 N/A 0.6 N/A N/A N/A random_state random size power N/A t_2 N/A 1.0 t_1 N/A new_data N/A N/A N/A t_1 N/A N/A N/A N/A numpy newaxis N/A numpied_df N/A N/A k_1 N/A N/A N/A t_2 N/A N/A N/A N/A numpy newaxis N/A numpied_df N/A N/A k_2 N/A N/A N/A N/A N/A pandas DataFrame new_data columns N/A data columns 0 N/A prepare_toymc N/A N/A group clustering_features stayed_features size_factor N/A N/A N/A This procedure prepares one block of data,  written specially for parallel execution
    :type group: pandas.grouping = (group_key, group_data), the data used to generate monte-carlo
    :type clustering_features: the features that were used to split data
    :type stayed_features: the names of other features, needed to reconstruct
    :type size_factor: float, the size of generated toymc26 is about size_factor * len(group)
    :rtype: (pandas.DataFrame, int)
     N/A N/A group_values df group N/A N/A toymc_part n_copied N/A generate_toymc N/A df N/A stayed_features N/A int N/A N/A len df size_factor knn None N/A N/A i col N/A enumerate clustering_features N/A N/A N/A toymc_part N/A col N/A group_values N/A i N/A N/A toymc_part n_copied N/A generate_toymc_with_special_features N/A N/A data size knn clustering_features ipc_profile N/A 4 None None N/A N/A Generate the toymc.
    :type data: numpy.array | pandas.DataFrame, from which data is generated
    :type size: int, how many events to generate
    :type clustering_features: the events with different values of this feature can not be mixed together.
        For instance: is_signal, number of jets / muons.
    :type ipc_profile: toymc can be generated on the cluster,
        provided there is at least one clustering feature
    :rtype: pandas.DataFrame with result,
        all the columns should be the same as in input
     N/A N/A clustering_features None N/A N/A clustering_features N/A N/A stayed_features N/A col N/A col N/A data columns N/A col clustering_features N/A size_factor N/A N/A float size N/A len data N/A copied_groups 0 N/A N/A N/A len clustering_features 0 N/A N/A N/A result copied N/A generate_toymc data size size knn knn N/A N/A grouped N/A N/A data groupby clustering_features N/A N/A print Generating ... N/A n_groups N/A len grouped N/A results N/A map_on_cluster ipc_profile prepare_toymc grouped N/A N/A clustering_features n_groups N/A N/A stayed_features n_groups N/A N/A size_factor n_groups N/A N/A toymc_parts copied_list N/A zip results N/A copied N/A N/A numpy sum copied_list N/A copied_groups N/A N/A numpy sum N/A N/A N/A numpy array copied_list 0 N/A result N/A N/A pandas concat toymc_parts N/A N/A copied 0 N/A N/A N/A print N/A Copied %i events in %i groups from original file. Totally generated %i rows  N/A copied copied_groups N/A len result N/A result N/A compare_toymc N/A N/A data clustering_features N/A None N/A N/A Prints comparison of distributions: original one and generated toyMC.
    Should be used in IPython notebook.
    :param pandas.DataFrame data: the dataset we generate toyMC for.
    :type clustering_features: list | None, very close to integer ones, usually have some bool or integer values,
        but events with different values in these columns should not be mixed together
        example: 'isSignal', number of muons
    :type integer_features: list | None, features that have some discrete values, but events can be mixed together
        if they have different values in these columns, the result will be integer
        example: some weight of event, which should be integer due to technical restrictions, isolation vars.
     N/A N/A isinstance data N/A pandas DataFrame need a Pandas.DataFrame here IPython.display display_html print_title N/A N/A title N/A N/A N/A N/A display_html N/A N/A <h3>{}</h3> format title raw True N/A N/A toy_data N/A generate_toymc_with_special_features data N/A len data clustering_features clustering_features N/A N/A N/A numpy set_printoptions precision 4 suppress True N/A n_cols 3 N/A n_rows N/A N/A N/A N/A len N/A data columns n_cols 1 n_cols N/A N/A N/A pylab figure figsize N/A 18 N/A 5 n_rows N/A N/A i column N/A enumerate N/A data columns 1 N/A N/A N/A N/A pylab subplot n_rows n_cols i N/A N/A N/A pylab title column N/A N/A N/A pylab hist N/A N/A N/A data N/A column values N/A N/A toy_data N/A column values histtype step bins 20 N/A N/A N/A pylab show N/A N/A print_title Means and std N/A mean_index N/A N/A mean_rows N/A N/A column N/A data columns N/A N/A N/A N/A mean_index append column N/A mean1 N/A N/A numpy mean N/A data N/A column N/A mean2 N/A N/A numpy mean N/A toy_data N/A column N/A std1 N/A N/A numpy std N/A data N/A column N/A std2 N/A N/A numpy std N/A toy_data N/A column N/A N/A N/A mean_rows append N/A mean1 mean2 N/A mean2 mean1 N/A abs N/A N/A N/A mean1 mean2 100.0 mean1 std1 std2 N/A std2 std1 N/A abs N/A N/A N/A std2 std1 100.0 std1 N/A N/A display_html N/A N/A pandas DataFrame mean_rows index mean_index columns N/A mean orig mean toy difference error, % std orig std toy difference error, % N/A N/A print_title Covariance N/A cov_index N/A N/A cov_rows N/A N/A N/A i first_column N/A enumerate N/A data columns N/A N/A second_column N/A N/A data columns N/A N/A i 1 N/A N/A N/A N/A cov_index append N/A first_column second_column N/A data_cov N/A N/A pearsonr N/A data N/A first_column N/A data N/A second_column N/A 0 N/A toy_cov N/A N/A pearsonr N/A toy_data N/A first_column N/A toy_data N/A second_column N/A 0 N/A N/A N/A cov_rows append N/A data_cov toy_cov N/A toy_cov data_cov N/A abs N/A N/A N/A data_cov toy_cov 100.0 data_cov N/A N/A display_html N/A N/A pandas DataFrame cov_rows index cov_index columns N/A original toy difference error, % N/A N/A col first_column N/A enumerate N/A N/A data columns N/A 4 N/A N/A second_column N/A N/A data columns N/A N/A col 1 4 N/A N/A x_min N/A N/A numpy min N/A data N/A first_column N/A x_max N/A N/A numpy max N/A data N/A first_column N/A y_min N/A N/A numpy min N/A data N/A second_column N/A y_max N/A N/A numpy max N/A data N/A second_column N/A N/A N/A pylab figure figsize N/A 12 5 N/A N/A N/A pylab subplot 121 N/A N/A N/A pylab plot N/A data N/A first_column N/A data N/A second_column . alpha 0.1 N/A N/A N/A N/A pylab xlim x_min x_max N/A N/A pylab ylim y_min y_max N/A N/A N/A pylab title original data N/A N/A N/A pylab subplot 122 N/A N/A N/A pylab plot N/A toy_data N/A first_column N/A toy_data N/A second_column . alpha 0.1 N/A N/A N/A N/A pylab ylim y_min y_max N/A N/A pylab xlim x_min x_max N/A N/A N/A pylab title toy MC N/A N/A N/A pylab suptitle N/A N/A {} vs {} format first_column second_column N/A N/A N/A pylab show N/A