Module ImportFrom alias Import alias Import alias ImportFrom alias Import alias Import alias Import alias Import alias Import alias identifier Import alias ImportFrom alias ImportFrom alias alias alias alias ImportFrom alias ImportFrom alias Expr Str ClassDef bases NameLoad body FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam defaults NameLoad NameLoad Num body Assign AttributeStore NameLoad attr NameLoad Assign AttributeStore NameLoad attr NameLoad If CompareEq NameLoad Num body Assign NameStore Call NameLoad Assign AttributeStore NameLoad attr NameLoad If CompareIs NameLoad NameLoad body Assign NameStore NameLoad Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam defaults body Return Call AttributeLoad AttributeLoad NameLoad attr attr decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr Print TupleLoad Str Call AttributeLoad Call AttributeLoad NameLoad attr keyword NameLoad attr Str Assign NameStore ListLoad Assign NameStore ListLoad Assign NameStore ListLoad TryFinally body Print Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad BoolOpOr BinOpFloorDiv Call NameLoad NameLoad AttributeLoad NameLoad attr AttributeLoad NameLoad attr For NameStore NameLoad body If AttributeLoad NameLoad attr body Continue Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Call Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad Print Str Expr Call Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr GeneratorExp Call AttributeLoad NameLoad attr comprehension NameStore NameLoad Assign NameStore ListComp Call Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr NameLoad comprehension NameStore NameLoad Print TupleLoad Str Call NameLoad NameLoad Str Assign NameStore Call Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr NameLoad Print Str Assign NameStore Call NameLoad GeneratorExp Call AttributeLoad NameLoad attr comprehension NameStore NameLoad Print Str finalbody For NameStore NameLoad body Expr Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr Print TupleLoad Str BinOpSub NameLoad NameLoad Str Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam NameParam defaults body If CompareIsNot AttributeLoad NameLoad attr NameLoad body Assign NameStore Call NameLoad NameLoad AttributeLoad NameLoad attr NameLoad orelse Assign NameStore Call NameLoad NameLoad NameLoad Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Assign NameStore Call AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad AttributeLoad NameLoad attr FunctionDef arguments args defaults body For NameStore NameLoad body Expr Yield NameLoad Delete NameDel Expr Call AttributeLoad NameLoad attr decorator_list Expr Call AttributeLoad NameLoad attr Call NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Expr Call AttributeLoad NameLoad attr Assign NameStore BinOpDiv AttributeLoad Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr attr BinOpPow Num Num Assign NameStore Call AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr Assign NameStore BinOpDiv AttributeLoad Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr attr BinOpPow Num Num Assign NameStore Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Assign NameStore BinOpDiv AttributeLoad Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr attr BinOpPow Num Num Assign NameStore Call NameLoad Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr Assign NameStore BinOpDiv AttributeLoad Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr attr BinOpPow Num Num Print TupleLoad Str NameLoad Str NameLoad Str BinOpSub NameLoad NameLoad Print TupleLoad Str BinOpSub NameLoad NameLoad Str Print TupleLoad Str BinOpSub NameLoad NameLoad Str Print TupleLoad Str BinOpSub NameLoad NameLoad Str Return NameLoad decorator_list FunctionDef arguments args NameParam NameParam defaults body Print TupleLoad Str Call NameLoad NameLoad Str BinOpDiv Call AttributeLoad Call AttributeLoad NameLoad attr keyword NameLoad attr BinOpPow Num Num Str Assign NameStore ListLoad If CompareIs AttributeLoad NameLoad attr NameLoad body Return Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad For TupleStore NameStore NameStore Call AttributeLoad NameLoad attr AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad body If AttributeLoad NameLoad attr body Continue Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad Return NameLoad decorator_list decorator_list
N/A itertools chain N/A os N/A time multiprocessing cpu_count N/A sys N/A resource N/A gc N/A pandas N/A numpy np N/A joblib chatto_transform.transforms.transform_base Transform chatto_transform.lib.chunks from_chunks to_chunks to_group_chunks CHUNK_SIZE chatto_transform.lib temp_file chatto_transform.datastores.hdf_datastore HdfDataStore N/A Library for executing transforms in parallel.

Splits data into as many groups as there are cpus, and runs a transform on all groups simultaneously.

HDFStore is used as an intermediate storage target, to pass data between jobs. ParallelTransform N/A Transform N/A __init__ N/A N/A self transform group_index chunksize n_jobs N/A None None -1 N/A N/A N/A self transform_obj transform N/A N/A self group_index group_index N/A N/A n_jobs -1 N/A N/A n_jobs N/A cpu_count N/A N/A self n_jobs n_jobs N/A N/A chunksize None N/A N/A chunksize CHUNK_SIZE N/A N/A self chunksize chunksize N/A output_schema N/A N/A self N/A N/A N/A N/A N/A N/A self transform_obj output_schema N/A input_schema N/A N/A self N/A N/A N/A N/A N/A N/A self transform_obj input_schema N/A _transform N/A N/A self data N/A N/A N/A start N/A N/A time time N/A N/A transforming data of size N/A N/A N/A N/A data memory_usage index True sum bytes N/A store_chunks_jobs N/A N/A transform_jobs N/A N/A hdf_stores N/A N/A N/A N/A splitting data into large groups N/A group_iter N/A N/A self _group_iter data N/A N/A N/A len data N/A self n_jobs N/A self chunksize N/A group_data group_iter N/A N/A N/A group_data empty N/A N/A N/A f N/A N/A temp_file make_temporary_file N/A hdf_store N/A HdfDataStore N/A N/A self input_schema f N/A N/A N/A hdf_stores append hdf_store N/A N/A N/A hdf_store store group_data N/A N/A N/A store_chunks_jobs append N/A N/A N/A joblib delayed N/A self store_chunks_job hdf_store N/A breaking data into chunks in parallel N/A N/A N/A N/A joblib Parallel n_jobs N/A self n_jobs store_chunks_jobs N/A chunk_stores N/A N/A chain from_iterable N/A N/A N/A store chunk_stores N/A store hdf_stores N/A transform_jobs N/A N/A N/A N/A joblib delayed N/A self transform_job chunk_store N/A chunk_store chunk_stores N/A N/A running transforms in N/A len transform_jobs parallel jobs N/A result_hdf_stores N/A N/A N/A joblib Parallel n_jobs N/A self n_jobs transform_jobs N/A loading and merging the results N/A results N/A from_chunks N/A N/A N/A r_hdf_store load N/A r_hdf_store result_hdf_stores N/A finished merge N/A N/A hdf_store hdf_stores N/A N/A N/A N/A hdf_store delete_chunks N/A N/A N/A hdf_store delete N/A end N/A N/A time time N/A N/A took N/A end start seconds to transform all data in parallel N/A results N/A _group_iter N/A N/A self data chunksize N/A N/A N/A N/A N/A self group_index None N/A N/A group_iter N/A to_group_chunks data N/A self group_index chunksize N/A N/A group_iter N/A to_chunks data chunksize N/A group_iter N/A store_chunks_job N/A N/A self hdf_store N/A N/A N/A data N/A N/A hdf_store load N/A chunks N/A N/A self _group_iter data N/A self chunksize gc_chunks N/A N/A N/A N/A N/A chunk chunks N/A N/A N/A chunk N/A chunk N/A N/A N/A gc collect N/A N/A N/A N/A hdf_store store_chunks N/A gc_chunks N/A transform_job N/A N/A self chunk_store N/A N/A N/A N/A N/A gc collect N/A start N/A N/A N/A N/A resource getrusage N/A resource RUSAGE_SELF ru_maxrss N/A 10 6 N/A data N/A N/A chunk_store load N/A N/A N/A gc collect N/A finished_load N/A N/A N/A N/A resource getrusage N/A resource RUSAGE_SELF ru_maxrss N/A 10 6 N/A result N/A N/A self transform_chunk data N/A N/A N/A gc collect N/A finished_transform N/A N/A N/A N/A resource getrusage N/A resource RUSAGE_SELF ru_maxrss N/A 10 6 N/A t_hdf_store N/A HdfDataStore N/A N/A self output_schema N/A chunk_store hdf_file N/A N/A N/A t_hdf_store store result N/A N/A N/A gc collect N/A finished_store N/A N/A N/A N/A resource getrusage N/A resource RUSAGE_SELF ru_maxrss N/A 10 6 N/A N/A started with start mb, ended with finished_store difference = N/A finished_store start N/A N/A loading used N/A finished_load start mb N/A N/A transforming used N/A finished_transform finished_load mb N/A N/A storing used N/A finished_store finished_transform mb N/A t_hdf_store N/A transform_chunk N/A N/A self data N/A N/A N/A N/A transforming chunk of length N/A len data mem usage N/A N/A N/A N/A N/A data memory_usage index True sum N/A 10 6 mb N/A transformed_groups N/A N/A N/A N/A self group_index None N/A N/A N/A N/A N/A self transform_obj transform data N/A N/A group_id group_data N/A N/A data groupby N/A self group_index as_index False sort False N/A N/A N/A group_data empty N/A N/A N/A transformed_group N/A N/A N/A self transform_obj transform group_data N/A N/A N/A transformed_groups append transformed_group N/A transformed_data N/A N/A pandas concat transformed_groups N/A transformed_data N/A N/A