Module Expr Str Import alias identifier Import alias identifier Import alias identifier Import alias Import alias identifier ImportFrom alias ClassDef bases NameLoad body Expr Str FunctionDef arguments args NameParam NameParam defaults body Assign AttributeStore NameLoad attr NameLoad decorator_list FunctionDef arguments args NameParam defaults body Return Call NameLoad AttributeLoad NameLoad attr decorator_list decorator_list FunctionDef arguments args NameParam NameParam NameParam NameParam NameParam NameParam NameParam defaults Num Num NameLoad body Expr Str Assign NameStore Call NameLoad NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad keyword AttributeLoad NameLoad attr For TupleStore NameStore TupleStore NameStore NameStore Call NameLoad NameLoad body If UnaryOpNot Call AttributeLoad NameLoad attr CompareEq NameLoad SubscriptLoad NameLoad Slice BinOpMult NameLoad NameLoad BinOpMult BinOpAdd NameLoad Num NameLoad body Raise Call NameLoad Str If UnaryOpNot Call AttributeLoad NameLoad attr CompareEq NameLoad SubscriptLoad NameLoad Slice BinOpMult NameLoad NameLoad BinOpMult BinOpAdd NameLoad Num NameLoad body Raise Call NameLoad Str Assign NameStore Call NameLoad Call NameLoad NameLoad Expr Call AttributeLoad Call AttributeLoad AttributeLoad NameLoad attr attr Num attr NameLoad Assign NameStore Call NameLoad NameLoad keyword NameLoad Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword Num keyword NameLoad keyword AttributeLoad NameLoad attr For TupleStore NameStore TupleStore NameStore NameStore Call NameLoad NameLoad body If UnaryOpNot Call AttributeLoad NameLoad attr CompareEq NameLoad SubscriptLoad SubscriptLoad NameLoad Index NameLoad Slice BinOpMult NameLoad NameLoad BinOpMult BinOpAdd NameLoad Num NameLoad body Raise Call NameLoad Str If UnaryOpNot Call AttributeLoad NameLoad attr CompareEq NameLoad SubscriptLoad SubscriptLoad NameLoad Index NameLoad Slice BinOpMult NameLoad NameLoad BinOpMult BinOpAdd NameLoad Num NameLoad body Raise Call NameLoad Str Assign NameStore Call NameLoad NameLoad Assign NameStore NameStore Call AttributeLoad Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad keyword AttributeLoad NameLoad attr attr If CompareEq Call NameLoad NameLoad NameLoad body Raise Call NameLoad Str orelse If CompareNotEq SubscriptLoad AttributeLoad NameLoad attr Index Num NameLoad body Raise Call NameLoad Str Assign NameStore Dict Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Num Assign NameStore Call NameLoad NameLoad NameLoad Expr Call AttributeLoad NameLoad attr NameLoad NameLoad keyword Num Assign NameStore AttributeLoad NameLoad attr If Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr NameLoad NameLoad body Raise Call NameLoad Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword Call AttributeLoad NameLoad attr Num Assign NameStore BinOpDiv SubscriptLoad NameLoad Index Num Call NameLoad Call AttributeLoad NameLoad attr SubscriptLoad NameLoad Index Num Assign NameStore Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr BinOpSub Call AttributeLoad NameLoad attr NameLoad If CompareGt NameLoad BinOpMult Num Call NameLoad NameLoad body Raise Call NameLoad Str Assign NameStore SubscriptLoad NameLoad Slice NameLoad Assign NameStore SubscriptLoad NameLoad Slice NameLoad Assign NameStore Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad Assign NameStore BinOpSub NameLoad NameLoad Assign NameStore Call AttributeLoad NameLoad attr keyword NameLoad Assign NameStore BinOpSub NameLoad NameLoad AugAssignDiv NameStore NameLoad If CompareEq AttributeLoad NameLoad attr Str body Assign NameStore Call NameLoad NameLoad keyword NameLoad keyword NameLoad keyword NameLoad keyword Num orelse Assign NameStore Call NameLoad NameLoad keyword NameLoad keyword NameLoad keyword NameLoad Assign TupleStore NameStore NameStore Call AttributeLoad Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad keyword AttributeLoad NameLoad attr attr If UnaryOpNot Call AttributeLoad NameLoad attr CompareEq NameLoad NameLoad body Raise Call NameLoad Str Assign NameStore Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr Assign NameStore BinOpSub NameLoad Call AttributeLoad NameLoad attr keyword NameLoad Assign NameStore BinOpDiv NameLoad BinOpAdd Call AttributeLoad NameLoad attr keyword NameLoad Num Assign NameStore Call NameLoad NameLoad keyword NameLoad keyword NameLoad keyword NameLoad Assign TupleStore NameStore NameStore Call AttributeLoad Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad keyword AttributeLoad NameLoad attr attr If UnaryOpNot Call AttributeLoad NameLoad attr CompareEq NameLoad NameLoad body Raise Call NameLoad Str Assign NameStore ListLoad For NameStore NameLoad body Assign NameStore Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr keyword AttributeLoad NameLoad attr Assign NameStore BinOpSub NameLoad Call AttributeLoad NameLoad attr keyword NameLoad Assign NameStore BinOpDiv NameLoad BinOpAdd Call AttributeLoad NameLoad attr keyword NameLoad Num Expr Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword AttributeLoad NameLoad attr Assign NameStore Call NameLoad NameLoad keyword NameLoad keyword NameLoad keyword NameLoad Assign TupleStore NameStore NameStore Call AttributeLoad Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad keyword AttributeLoad NameLoad attr attr If UnaryOpNot Call AttributeLoad NameLoad attr CompareEq NameLoad NameLoad body Raise Call NameLoad Str If UnaryOpNot NameLoad body Return Assign NameStore Call NameLoad keyword Num keyword TupleLoad Num Num keyword Num keyword TupleLoad Num Num keyword NameLoad Assign NameStore Call NameLoad NameLoad NameLoad keyword NameLoad Assign TupleStore NameStore NameStore Call AttributeLoad Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad keyword AttributeLoad NameLoad attr attr If UnaryOpNot Call AttributeLoad NameLoad attr CompareEq NameLoad NameLoad body Raise Call NameLoad Str Assign NameStore Call NameLoad keyword TupleLoad BinOpDiv Num Num Num keyword TupleLoad Num Num keyword TupleLoad Num Num keyword TupleLoad Num Num keyword NameLoad keyword NameLoad Assign NameStore Call NameLoad NameLoad NameLoad keyword NameLoad Assign TupleStore NameStore NameStore Call AttributeLoad Call AttributeLoad NameLoad attr keyword NameLoad keyword NameLoad keyword NameLoad keyword AttributeLoad NameLoad attr attr If UnaryOpNot Call AttributeLoad NameLoad attr CompareEq NameLoad NameLoad body Raise Call NameLoad Str If BoolOpOr Call AttributeLoad NameLoad attr CompareEq NameLoad NameLoad Call AttributeLoad NameLoad attr CompareEq NameLoad NameLoad body Raise Call NameLoad Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad keyword Num keyword NameLoad Assign TupleStore NameStore NameStore Call AttributeLoad NameLoad attr Num Num keyword TupleLoad Num Num Expr Call AttributeLoad NameLoad attr BinOpMod Str AttributeLoad NameLoad attr keyword Num For TupleStore NameStore NameStore Call NameLoad SubscriptLoad NameLoad ExtSlice Slice Index Num body Assign NameStore SubscriptLoad NameLoad Index SubscriptLoad NameLoad Index NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr ListLoad Expr Call AttributeLoad NameLoad attr ListLoad If CompareEq NameLoad Num body Expr Call AttributeLoad NameLoad attr Str For TupleStore NameStore NameStore Call NameLoad SubscriptLoad NameLoad ExtSlice Slice Index Num body Assign NameStore SubscriptLoad NameLoad Index SubscriptLoad NameLoad Index NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr ListLoad Expr Call AttributeLoad NameLoad attr ListLoad If CompareEq NameLoad Num body Expr Call AttributeLoad NameLoad attr Str For TupleStore NameStore NameStore Call NameLoad SubscriptLoad NameLoad ExtSlice Slice Index Num body Assign NameStore SubscriptLoad NameLoad Index SubscriptLoad NameLoad Index NameLoad Expr Call AttributeLoad NameLoad attr NameLoad Expr Call AttributeLoad NameLoad attr ListLoad Expr Call AttributeLoad NameLoad attr ListLoad If CompareEq NameLoad Num body Expr Call AttributeLoad NameLoad attr Str decorator_list If CompareEq NameLoad Str body Assign NameStore Str Assign NameStore Str Assign NameStore Call AttributeLoad NameLoad attr NameLoad Assign NameStore AttributeLoad SubscriptLoad NameLoad Index Str attr Assign NameStore AttributeLoad SubscriptLoad NameLoad Index Str attr Assign NameStore ListComp Call AttributeLoad NameLoad attr Call AttributeLoad NameLoad attr Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad comprehension NameStore NameLoad Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword AttributeLoad NameLoad attr Assign NameStore Call AttributeLoad NameLoad attr NameLoad keyword NameLoad Assign NameStore Call AttributeLoad SubscriptLoad NameLoad Index Str attr Lambda arguments args NameParam defaults Call AttributeLoad AttributeLoad NameLoad attr attr NameLoad NameLoad Print Str Expr Call NameLoad AttributeLoad NameLoad attr NameLoad NameLoad NameLoad keyword Num keyword NameLoad Print Str Print Str Expr Call NameLoad AttributeLoad NameLoad attr NameLoad NameLoad NameLoad keyword Num keyword NameLoad Print Str
N/A N/A 
Test core generator functions
 N/A datumio.datagen dtd N/A matplotlib.pyplot plt N/A numpy np N/A os N/A pandas pd PIL Image TestGenError N/A Exception N/A N/A Raise error if one test fails. Each test usually requires the previous
    to finish without error __init__ N/A N/A self msg N/A N/A N/A N/A self msg msg N/A __str__ N/A N/A self N/A N/A N/A N/A repr N/A self msg N/A N/A main N/A N/A gen X y X_og BATCH_SIZE axis show_aug_test N/A 32 0 True N/A N/A Runs a test for each core functionality of `gen`.

    Parameters
    ----------
    gen: datumio.BaseGenerator
        Generator to test core functionality

    X: iterable
        X input to generator. For BatchGenerator, X = loaded data with shape
        of (dataset_length, height, width, channel). For DataGenerator,
        X = path to data files with shape of (dataset_length,).

    y: iterable
        y input to generator.

    X_og: ndarray
        Loaded data with shape of (dataset_length, height, width, channel)
        corresponding to `X`'s order. This is use for testing generators.

    BATCH_SIZE: int, optional
        Size of minibatches per call

    axis: None or int, optional
        Axis to test zmuv operations on.
        See numpy.mean for more options for axis

    show_aug_test: bool, optional
        If true, will plot the on-the-fly augmentations (static & random)
        with comparison to the original. This is for manual, visual
        test to see if augmentations are correct (or at least what we expect)

    Returns
    ----------
    None:
        If no exceptions were raised, all tests were ran successfully. However,
        you must check manually check the augmentation tests visually. See
        `show_aug_test`.

    Raises
    ----------
    TestGenError: Exception
        If any functionality fails to do what it's suppose to.
     N/A Batcher N/A gen X y y N/A batchgen N/A N/A Batcher get_batch batch_size BATCH_SIZE shuffle False chw_order False dtype N/A np uint8 N/A N/A idx N/A mb_x mb_y N/A enumerate batchgen N/A N/A N/A N/A N/A np all N/A mb_x N/A X_og N/A N/A idx BATCH_SIZE N/A N/A idx 1 BATCH_SIZE N/A N/A N/A TestGenError Not all batch in X match correctly from data N/A N/A N/A N/A np all N/A mb_y N/A y N/A N/A idx BATCH_SIZE N/A N/A idx 1 BATCH_SIZE N/A N/A N/A TestGenError Not all batch in y match correctly from data N/A idxs N/A range N/A len X N/A N/A N/A N/A N/A N/A np random RandomState 16 shuffle idxs N/A Batcher N/A gen X y y N/A batchgen N/A N/A Batcher get_batch batch_size BATCH_SIZE shuffle True rng_seed 16 chw_order False dtype N/A np uint8 N/A N/A idx N/A mb_x mb_y N/A enumerate batchgen N/A N/A N/A N/A N/A np all N/A mb_x N/A N/A X_og N/A idxs N/A N/A idx BATCH_SIZE N/A N/A idx 1 BATCH_SIZE N/A N/A N/A TestGenError Batches in X are not correctly shuffled N/A N/A N/A N/A np all N/A mb_y N/A N/A y N/A idxs N/A N/A idx BATCH_SIZE N/A N/A idx 1 BATCH_SIZE N/A N/A N/A TestGenError Batches in y are not correctly shuffled N/A Batcher N/A gen X N/A mb_x batchgen N/A N/A N/A N/A Batcher get_batch batch_size BATCH_SIZE shuffle False chw_order False dtype N/A np uint8 next N/A N/A N/A type mb_x tuple N/A N/A N/A TestGenError Getting batch is returning y when it shouldn't N/A N/A N/A N/A N/A mb_x shape N/A 0 BATCH_SIZE N/A N/A N/A TestGenError W/out y, but returning incorrect len of batch N/A class_weights N/A 0 1 2 3 4 5 6 7 8 9 0.25 0.0 0.0 0.15 0.1 0.25 0.0 0.0 0.15 0.1 N/A Batcher N/A gen X y N/A N/A N/A Batcher resample_dataset y class_weights sample_fraction 1.0 N/A resampled_y N/A Batcher y N/A N/A N/A np all N/A N/A np equal y resampled_y N/A N/A N/A TestGenError Resampled y should not equal to y N/A hist N/A N/A np histogram resampled_y bins N/A N/A np arange 11 N/A sample_weights N/A N/A hist N/A 0 N/A float N/A N/A np sum N/A hist N/A 0 N/A mae N/A N/A np mean N/A N/A np abs N/A N/A N/A class_weights values sample_weights N/A N/A mae N/A 0.01 N/A len sample_weights N/A N/A N/A TestGenError Resampled y has incorrect distribution N/A mb_x_og N/A X_og N/A BATCH_SIZE N/A mb_y_og N/A y N/A BATCH_SIZE N/A tmp_X N/A N/A np array N/A N/A X_og copy dtype N/A np float32 N/A mean N/A N/A tmp_X mean axis axis N/A tmp_X N/A tmp_X mean N/A std N/A N/A tmp_X std axis axis N/A zmuv_mb_x N/A mb_x_og mean N/A zmuv_mb_x std N/A N/A N/A gen __name__ DataGenerator N/A N/A Batcher N/A gen X y y dataset_zmuv True dataset_axis axis dataset_zmuv_bsize 500 N/A N/A Batcher N/A gen X y y dataset_zmuv True dataset_axis axis N/A N/A mb_x mb_y N/A N/A N/A N/A Batcher get_batch batch_size BATCH_SIZE shuffle False chw_order False dtype N/A np float32 next N/A N/A N/A N/A np all N/A mb_x zmuv_mb_x N/A N/A N/A TestGenError Error correctly generating batch w/ dataset-zmuv N/A mb_x_manual N/A N/A np array N/A N/A mb_x_og copy dtype N/A np float32 N/A mb_x_manual N/A mb_x_manual N/A N/A mb_x_manual mean axis axis N/A mb_x_manual N/A mb_x_manual N/A N/A N/A mb_x_manual std axis axis 1e-12 N/A Batcher N/A gen X y y batch_zmuv True batch_axis axis N/A N/A mb_x mb_y N/A N/A N/A N/A Batcher get_batch batch_size BATCH_SIZE shuffle False chw_order False dtype N/A np float32 next N/A N/A N/A N/A np all N/A mb_x mb_x_manual N/A N/A N/A TestGenError Error correctly generating batch w/ batch-zmuv N/A mb_x_manual N/A N/A x mb_x_og N/A N/A ex N/A N/A np array N/A N/A x copy dtype N/A np float32 N/A ex N/A ex N/A N/A ex mean axis axis N/A ex N/A ex N/A N/A N/A ex std axis axis 1e-12 N/A N/A N/A mb_x_manual append ex N/A mb_x_manual N/A N/A np array mb_x_manual dtype N/A np float32 N/A Batcher N/A gen X y y sample_zmuv True sample_axis axis N/A N/A mb_x mb_y N/A N/A N/A N/A Batcher get_batch batch_size BATCH_SIZE shuffle False chw_order False dtype N/A np float32 next N/A N/A N/A N/A np all N/A mb_x mb_x_manual N/A N/A N/A TestGenError Error correctly generating batch with sample zmuv N/A N/A show_aug_test N/A N/A N/A aug_params N/A dict rotation 15 zoom N/A 1.5 1.5 shear 7 translation N/A 5 -5 flip_lr True N/A Batcher N/A gen X y aug_params aug_params N/A N/A mb_x_aug mb_y_aug N/A N/A N/A N/A Batcher get_batch batch_size BATCH_SIZE shuffle False chw_order False dtype N/A np uint8 next N/A N/A N/A N/A np all N/A mb_y_og mb_y_aug N/A N/A N/A TestGenError Static aug created error in generation of truth N/A rng_aug_params N/A dict zoom_range N/A N/A 1 1.5 1.5 rotation_range N/A -15 15 shear_range N/A -7 7 translation_range N/A -5 5 do_flip_lr True allow_stretch False N/A Batcher N/A gen X y rng_aug_params rng_aug_params N/A N/A mb_x_rng mb_y_rng N/A N/A N/A N/A Batcher get_batch batch_size BATCH_SIZE shuffle False chw_order False dtype N/A np uint8 next N/A N/A N/A N/A np all N/A mb_y_og mb_y_rng N/A N/A N/A TestGenError Rng augs created error in generation of truth N/A N/A N/A N/A np all N/A mb_x_rng mb_x_aug N/A N/A np all N/A mb_x_rng mb_x_og N/A N/A N/A TestGenError Static or Random augs not correctly done N/A rng_idxs N/A N/A np arange BATCH_SIZE N/A rng_idxs N/A N/A N/A np random choice rng_idxs size 3 replace False N/A N/A fig axes N/A N/A plt subplots 3 3 figsize N/A 14 10 N/A N/A N/A fig suptitle N/A Generator: %s N/A gen __name__ fontsize 26 N/A N/A it ax N/A enumerate N/A axes N/A N/A N/A 0 N/A N/A img N/A mb_x_og N/A N/A rng_idxs N/A it N/A N/A N/A ax imshow img N/A N/A N/A ax set_xticks N/A N/A N/A N/A ax set_yticks N/A N/A N/A it 0 N/A N/A N/A N/A ax set_title Original Images N/A N/A it ax N/A enumerate N/A axes N/A N/A N/A 1 N/A N/A img N/A mb_x_aug N/A N/A rng_idxs N/A it N/A N/A N/A ax imshow img N/A N/A N/A ax set_xticks N/A N/A N/A N/A ax set_yticks N/A N/A N/A it 0 N/A N/A N/A N/A ax set_title Static Augmentations N/A N/A it ax N/A enumerate N/A axes N/A N/A N/A 2 N/A N/A img N/A mb_x_rng N/A N/A rng_idxs N/A it N/A N/A N/A ax imshow img N/A N/A N/A ax set_xticks N/A N/A N/A N/A ax set_yticks N/A N/A N/A it 0 N/A N/A N/A N/A ax set_title Random Augmentations N/A N/A N/A __name__ __main__ N/A N/A data_dir test_data/cifar-10/ N/A label_path test_data/cifar-10/labels.csv N/A label_df N/A N/A pd read_csv label_path N/A uids N/A N/A label_df N/A uid values N/A y N/A N/A label_df N/A label values N/A X N/A N/A N/A np array N/A N/A Image open N/A N/A N/A os path join data_dir uid N/A uid uids N/A X N/A N/A np array X dtype N/A np uint8 N/A y N/A N/A np array y dtype int N/A data_paths N/A N/A N/A label_df N/A uid apply N/A N/A N/A x N/A N/A N/A N/A os path join data_dir x N/A Performing tests on BatchGenerator N/A N/A main N/A dtd BatchGenerator X y X BATCH_SIZE 64 axis None N/A ... BatchGenerator passed all tests (see aug tests manually N/A Performing tests on DataGenerator N/A N/A main N/A dtd DataGenerator data_paths y X BATCH_SIZE 64 axis None N/A ... DataGenerator passed all tests (see aug tests manually