automunge 9.0
platform for preparing tabular data for machine learning
Requires Python: <4,>=3.11

# Automunge

![image](https://user-images.githubusercontent.com/44011748/76485571-4855b280-63f3-11ea-9574-c39a66e45e4e.png)

#  

## Table of Contents
* [Introduction](https://github.com/Automunge/AutoMunge#introduction)
* [Install, Initialize, and Basics](https://github.com/Automunge/AutoMunge#install-initialize-and-basics)
 ___ 
* [automunge(.)](https://github.com/Automunge/AutoMunge#automunge-1)
* [automunge(.) returned sets](https://github.com/Automunge/AutoMunge#automunge-returned-sets)
* [automunge(.) passed parameters](https://github.com/Automunge/AutoMunge#automunge-passed-parameters)
 ___ 
* [postmunge(.)](https://github.com/Automunge/AutoMunge#postmunge)
* [postmunge(.) returned sets](https://github.com/Automunge/AutoMunge#postmunge-returned-sets)
* [postmunge(.) passed parameters](https://github.com/Automunge/AutoMunge#postmunge-passed-parameters)
 ___ 
* [Default Transformations](https://github.com/Automunge/AutoMunge#default-transformations)
* [Library of Transformations](https://github.com/Automunge/AutoMunge#library-of-transformations)
* [Custom Transformation Functions](https://github.com/Automunge/AutoMunge#custom-transformation-functions)
* [Custom ML Infill Functions](https://github.com/Automunge/AutoMunge#custom-ml-infill-functions)
* [Final Model Training](https://github.com/Automunge/AutoMunge#final-model-training)
 ___ 
* [Conclusion](https://github.com/Automunge/AutoMunge#conclusion)
 ___ 

## Introduction
[Automunge](https://automunge.com) is an open source python library that has formalized and automated the data preparations for tabular learning in between the workflow boundaries of received “tidy data” (one column per feature and one row per sample) and returned dataframes suitable for the direct application of machine learning. Under automation numeric features are normalized, categoric features are binarized, and missing data is imputed. Data transformations are fit to properties of a training set for a consistent basis on any partitioned “validation data” or additional “test data”. When preparing training data, a compact python dictionary is returned recording the steps and parameters of transformation, which then may serve as a key for preparing additional data on a consistent basis.

> In other words, put simply:<br/>
>  - **automunge(.)** prepares tabular data for machine learning with encodings, missing data infill, and may channel stochastic perturbations into features<br/>
>  - **postmunge(.)** consistently prepares additional data very efficiently<br/>
>  
> We make machine learning easy.

In addition to data preparations under automation, Automunge may also serve as a platform for engineering data pipelines. An extensive internal library of univariate transformations includes options like numeric translations, bin aggregations, date-time encodings, noise injections, categoric encodings, and even “parsed categoric encodings” in which categoric strings are vectorized based on shared grammatical structure between entries. Feature transformations may be mixed and matched in sets that include generations and branches of derivations by use of our “family tree primitives”. Feature transformations fit to properties of a training set may be custom defined from a very simple template for incorporation into a pipeline. Dimensionality reductions may be applied, such as by principal component analysis, feature importance rankings, or categoric consolidations. Missing data receives “ML infill”, in which models are trained for a feature to impute missing entries based on properties of the surrounding features. Random sampling may be channeled into features as stochastic perturbations.

Be sure to check out our [Tutorial Notebooks](https://github.com/Automunge/AutoMunge/tree/master/Tutorials). If you are looking for something to cite, our paper [Tabular Engineering with Automunge](https://datacentricai.org/papers/15_CameraReady_TabularEngineering_102621_Final.pdf) was accepted to the Data-Centric AI workshop at NeurIPS 2021.

## Install, Initialize, and Basics

Automunge is now available for pip install:

```
pip install Automunge
```
Or to upgrade:
```
pip install Automunge --upgrade
```

Once installed, run this in a local session to initialize:

```
from Automunge import *
am = AutoMunge()
```

Where e.g. for train set processing  with default parameters run:

```
train, train_ID, labels, \
val, val_ID, val_labels, \
test, test_ID, test_labels, \
postprocess_dict = \
am.automunge(df_train)
```
Importantly, if the df_train set passed to automunge(.) includes a column
intended for use as labels, it should be designated with the labels_column
parameter.

Or for subsequent consistent processing of train or test data, using the
dictionary returned from original application of automunge(.), run:

```
test, test_ID, test_labels, \
postreports_dict = \
am.postmunge(postprocess_dict, df_test)
```

I find it helpful to pass these functions with the full range of arguments
included for reference, thus a user may simply copy and past this form.

```
#for automunge(.) function on original train and test data

train, train_ID, labels, \
val, val_ID, val_labels, \
test, test_ID, test_labels, \
postprocess_dict = \
am.automunge(df_train, df_test = False,
             labels_column = False, trainID_column = False, testID_column = False,
             valpercent=0.0, floatprecision = 32, cat_type = False, shuffletrain = True, noise_augment = 0,
             dupl_rows = False, TrainLabelFreqLevel = False, powertransform = False, binstransform = False,
             MLinfill = True, infilliterate=1, randomseed = False, eval_ratio = .5,
             numbercategoryheuristic = 255, pandasoutput = 'dataframe', NArw_marker = True,
             featureselection = False, featurethreshold = 0., inplace = False, orig_headers = False,
             Binary = False, PCAn_components = False, PCAexcl = [], excl_suffix = False,
             ML_cmnd = {'autoML_type':'randomforest',
                        'MLinfill_cmnd':{'RandomForestClassifier':{}, 'RandomForestRegressor':{}},
                        'PCA_type':'default',
                        'PCA_cmnd':{}},
             assigncat = {'1010':[], 'onht':[], 'ordl':[], 'bnry':[], 'hash':[], 'hsh2':[],
                          'DP10':[], 'DPoh':[], 'DPod':[], 'DPbn':[], 'DPhs':[], 'DPh2':[],
                          'nmbr':[], 'mnmx':[], 'retn':[], 'DPnb':[], 'DPmm':[], 'DPrt':[],
                          'bins':[], 'pwr2':[], 'bnep':[], 'bsor':[], 'por2':[], 'bneo':[],
                          'ntgr':[], 'srch':[], 'or19':[], 'tlbn':[], 'excl':[], 'exc2':[]},
             assignparam = {'global_assignparam'  : {'(parameter)': 42},
                            'default_assignparam' : {'(category)' : {'(parameter)' : 42}},
                                     '(category)' : {'(column)'   : {'(parameter)' : 42}}},
             assigninfill = {'stdrdinfill':[], 'MLinfill':[], 'zeroinfill':[], 'oneinfill':[],
                             'adjinfill':[], 'meaninfill':[], 'medianinfill':[], 'negzeroinfill':[],
                             'interpinfill':[], 'modeinfill':[], 'lcinfill':[], 'naninfill':[]},
             assignnan = {'categories':{}, 'columns':{}, 'global':[]},
             transformdict = {}, processdict = {}, evalcat = False, ppd_append = False,
             entropy_seeds = False, random_generator = False, sampling_dict = False,
             privacy_encode = False, encrypt_key = False, printstatus = 'summary', logger = {})
```

Please remember to save the automunge(.) returned object postprocess_dict 
such as using pickle library, which can then be later passed to the postmunge(.) 
function to consistently prepare subsequently available data.

```
#Sample pickle code:

#sample code to download postprocess_dict dictionary returned from automunge(.)
import pickle
with open('filename.pickle', 'wb') as handle:
  pickle.dump(postprocess_dict, handle, protocol=pickle.HIGHEST_PROTOCOL)

#to upload for later use in postmunge(.) in another notebook
import pickle
with open('filename.pickle', 'rb') as handle:
  postprocess_dict = pickle.load(handle)

#Please note that if you included externally initialized functions in an automunge(.) call
#like for custom_train transformation functions or customML inference functions
#they will need to be reinitialized prior to uploading the postprocess_dict with pickle.
```

We can then apply the postprocess_dict saved from a prior application of automunge
for consistent processing of additional data.
```
#for postmunge(.) function on additional available train or test data
#using the postprocess_dict object returned from original automunge(.) application

test, test_ID, test_labels, \
postreports_dict = \
am.postmunge(postprocess_dict, df_test,
             testID_column = False,
             pandasoutput = 'dataframe', printstatus = 'summary',
             dupl_rows = False, TrainLabelFreqLevel = False,
             featureeval = False, traindata = False, noise_augment = 0,
             driftreport = False, inversion = False,
             returnedsets = True, shuffletrain = False,
             entropy_seeds = False, random_generator = False, sampling_dict = False,
             randomseed = False, encrypt_key = False, logger = {})
```

The functions accept pandas dataframe or numpy array input and return encoded dataframes
with consistent order of columns between train and test data. 
(For input numpy arrays any label column should be positioned as final column in set.)
The functions return data with categoric features translated to numerical encodings 
and normalized numeric such as to make them suitable for direct application to a 
machine learning model in the framework of a user's choice, including sets for 
the various activities of a generic machine learning project such as training (train), 
validation (val), and inference (test). The automunge(.) function also returns a 
python dictionary (the "postprocess_dict") that can be used as a key to prepare additional
data with postmunge(.).

When left to automation, automunge(.) evaluates properties of a feature to select 
the type of encoding, for example whether a column is numeric, categoric, high cardinality, 
binary, date time, etc. Alternately, a user can 
assign specific processing functions to distinct columns (via assigncat parameter) - 
which may be pulled from the internal [Library of Transformations](https://github.com/Automunge/AutoMunge#library-of-transformations) or alternately [custom
defined](https://github.com/Automunge/AutoMunge#custom-transformation-functions).

The feature engineering transformations are recorded with suffixes 
appended to the column header title in the returned sets, for one example the 
application of z-score normalization returns a column with header origname + '\_nmbr'. 
As another example, for binary encoded sets the set of columns are returned with
header origname + '\_1010_#' where # is integer to distinguish columns in same set.
In most cases, the suffix appenders are derived from the transformation category 
identifier (which is by convention a 4 letter string). 

The default transforms applied under automation are detailed below in section 
[Default Transforms](https://github.com/Automunge/AutoMunge#default-transformations). 
Missing data receives ML infill (defaulting to random forest models) and missing marker aggregation. 
Other features of the library are detailed in the [tutorial notebooks](https://github.com/Automunge/AutoMunge/tree/master/Tutorials) 
and with their associated parameters below. 

Other options available in the library include feature importance (via featureselection parameter),
oversampling (via the TrainLabelFreqLevel parameter), dimensionality reductions (via PCAn_components, Binary, or featurethreshold parameters), and stochastic perturbations (by the DP family of transformations detailed in the library of transformations and tutorials). Further detail provided with parameter writeups below.

Note that there is a potential source of error if the returned column header 
title strings, which will include suffix appenders based on transformations applied, 
match any of the original column header titles passed to automunge. This is an edge 
case not expected to occur in common practice and will return error message at 
conclusion of printouts and a logged validation result as postprocess_dict['miscparameters_results']['suffixoverlap_aggregated_result']. This channel can
be eliminated by omitting the underscore character in received column headers.

Please note that we consider the postmunge(.) latency a key performance 
metric since it is the function that may be called under repetition in production.
The automunge(.) latency can be improved by manual assignment of root categories with the assigncat parameter 
or by deactivating ML infill with the MLinfill parameter.

## automunge(.)

The application of the automunge and postmunge functions requires the
assignment of the function to a series of named sets. We suggest using
consistent naming convention as follows:

```
#first you'll need to initialize
from Automunge import *
am = AutoMunge()

#then to run with default parameters
train, train_ID, labels, \
val, val_ID, val_labels, \
test, test_ID, test_labels, \
postprocess_dict = \
am.automunge(df_train)
```

The full set of parameters available to be passed are given here, with
explanations provided below: 

```
#first you'll need to initialize
from Automunge import *
am = AutoMunge()

#then if you want you can copy paste following to view all of parameter options
#where df_train is the target training data set to be prepared

train, train_ID, labels, \
val, val_ID, val_labels, \
test, test_ID, test_labels, \
postprocess_dict = \
am.automunge(df_train, df_test = False,
             labels_column = False, trainID_column = False, testID_column = False,
             valpercent=0.0, floatprecision = 32, cat_type = False, shuffletrain = True, noise_augment = 0,
             dupl_rows = False, TrainLabelFreqLevel = False, powertransform = False, binstransform = False,
             MLinfill = True, infilliterate=1, randomseed = False, eval_ratio = .5,
             numbercategoryheuristic = 255, pandasoutput = 'dataframe', NArw_marker = True,
             featureselection = False, featurethreshold = 0., inplace = False, orig_headers = False,
             Binary = False, PCAn_components = False, PCAexcl = [], excl_suffix = False,
             ML_cmnd = {'autoML_type':'randomforest',
                        'MLinfill_cmnd':{'RandomForestClassifier':{}, 'RandomForestRegressor':{}},
                        'PCA_type':'default',
                        'PCA_cmnd':{}},
             assigncat = {'1010':[], 'onht':[], 'ordl':[], 'bnry':[], 'hash':[], 'hsh2':[],
                          'DP10':[], 'DPoh':[], 'DPod':[], 'DPbn':[], 'DPhs':[], 'DPh2':[],
                          'nmbr':[], 'mnmx':[], 'retn':[], 'DPnb':[], 'DPmm':[], 'DPrt':[],
                          'bins':[], 'pwr2':[], 'bnep':[], 'bsor':[], 'por2':[], 'bneo':[],
                          'ntgr':[], 'srch':[], 'or19':[], 'tlbn':[], 'excl':[], 'exc2':[]},
             assignparam = {'global_assignparam'  : {'(parameter)': 42},
                            'default_assignparam' : {'(category)' : {'(parameter)' : 42}},
                                     '(category)' : {'(column)'   : {'(parameter)' : 42}}},
             assigninfill = {'stdrdinfill':[], 'MLinfill':[], 'zeroinfill':[], 'oneinfill':[],
                             'adjinfill':[], 'meaninfill':[], 'medianinfill':[],
                             'interpinfill':[], 'modeinfill':[], 'lcinfill':[], 'naninfill':[]},
             assignnan = {'categories':{}, 'columns':{}, 'global':[]},
             transformdict = {}, processdict = {}, evalcat = False, ppd_append = False,
             entropy_seeds = False, random_generator = False, sampling_dict = False,
             privacy_encode = False, encrypt_key = False, printstatus = 'summary', logger = {})
```

Or for the postmunge function:

```
#for postmunge(.) function on additional or subsequently available test (or train) data
#using the postprocess_dict object returned from original automunge(.) application

#first you'll need to initialize
from Automunge import *
am = AutoMunge()

#then to run with default parameters
test, test_ID, test_labels, \
postreports_dict = \
am.postmunge(postprocess_dict, df_test)
```

With the full set of arguments available to be passed as:

```
#first you'll need to initialize
from Automunge import *
am = AutoMunge()

#then if you want you can copy paste following to view all of parameter options
#here postprocess_dict was returned from corresponding automunge(.) call
#and df_test is the target data set to be prepared

test, test_ID, test_labels, \
postreports_dict = \
am.postmunge(postprocess_dict, df_test,
             testID_column = False,
             pandasoutput = 'dataframe', printstatus = 'summary', inplace = False,
             dupl_rows = False, TrainLabelFreqLevel = False,
             featureeval = False, traindata = False, noise_augment = 0,
             driftreport = False, inversion = False,
             returnedsets = True, shuffletrain = False,
             entropy_seeds = False, random_generator = False, sampling_dict = False,
             randomseed = False, encrypt_key = False, logger = {})
```

Note that the only required argument to the automunge function is the
train set dataframe, the other arguments all have default values if
nothing is passed. The postmunge function requires as minimum the
postprocess_dict object (a python dictionary returned from the application of
automunge) and a dataframe test set consistently formatted as those sets
that were originally applied to automunge. 

...

Here now are descriptions for the returned sets from automunge, which
will be followed by descriptions of the parameters which can be passed to
the function, followed by similar treatment for postmunge returned sets
and arguments. Further below is documentation for the library of transformations.

...

## automunge(.) returned sets:

Automunge defaults to returning data sets as pandas dataframes, or for
single column sets as pandas series. 

For dataframes, data types of returned columns are based on the transformation applied, 
for example columns with boolean integers are cast as int8, ordinal encoded
columns are given a conditional type based on the size of encoding space as either
uint8, uint16, or uint32. Continuous sets are cast as float16, float32, or float64
based on the automunge(.) floatprecision parameter. And direct passthrough columns
via excl transform retain the received data type.

* train: a numerically encoded set of data intended to be used to train a
downstream machine learning model in the framework of a user's choice

* train_ID: the set of ID values corresponding to the train set if a ID
column(s) was passed to the function. This set may be useful if the shuffle
option was applied. Note that an ID column may serve multiple purposes such
as row identifiers or for pairing tabular data rows with a corresponding
image file for instance. Also included in this set is a derived column
titled 'Automunge_index', this column serves as an index identifier for order
of rows as they were received in passed data, such as may be beneficial
when data is shuffled. If the received df_train had a non-ranged integer index,
it is extracted and returned in this set. For more information please refer to writeup for the 
trainID_column parameter.

* labels: a set of numerically encoded labels corresponding to the
train set if a label column was passed. Note that the function
assumes the label column is originally included in the train set. Note
that if the labels set is a single column a returned dataframe is flattened 
to a pandas Series or a returned Numpy array is also 
flattened (e.g. [[1,2,3]] converted to [1,2,3] ).

* val: a set of training data carved out from the train set
that is intended for use in hyperparameter tuning of a downstream model.

* val_ID: the set of ID values corresponding to the val
set. Comparable to columns returned in train_ID.

* val_labels: the set of labels corresponding to the val
set

* test: the set of features, consistently encoded and normalized as the
training data, that can be used to generate predictions from a
downstream model trained with train. Note that if no test data is
available during initial address this processing will take place in the
postmunge(.) function. 

* test_ID: the set of ID values corresponding to the test set. Comparable 
to columns returned in train_ID unless otherwise specified. For more 
information please refer to writeup for the testID_column parameter.

* test_labels: a set of numerically encoded labels corresponding to the
test set if a label column was passed.

* postprocess_dict: a returned python dictionary that includes
normalization parameters and trained ML infill models used to
generate consistent processing of additional train or test data such as 
may not have been available at initial application of automunge. It is 
recommended that this dictionary be externally saved on each application 
used to train a downstream model so that it may be passed to postmunge(.) 
to consistently process subsequently available test data, such as 
demonstrated with the pickle library above.

A few useful entries in the postprocess_dict include:
- postprocess_dict['finalcolumns_train']: list of returned column headers for train set including suffix appenders
- postprocess_dict['columntype_report']: a report classifying the returned column types, including lists of all categoric and all numeric returned columns
- postprocess_dict['column_map']:  a report mapping the input columns to their associated returned columns (excluding those consolidated as part of a dimensionality reduction). May be useful to inspect sets returned for a specific feature e.g. train[postprocess_dict['column_map']['input_column_header']]
- postprocess_dict['FS_sorted]: sorted results of feature importance evaluation if elected
- postprocess_dict['miscparameters_results']: reporting results of validation tests performed on parameters and passed data

...

## automunge(.) passed parameters

```
train, train_ID, labels, \
val, val_ID, val_labels, \
test, test_ID, test_labels, \
postprocess_dict = \
am.automunge(df_train, df_test = False,
             labels_column = False, trainID_column = False, testID_column = False,
             valpercent=0.0, floatprecision = 32, cat_type = False, shuffletrain = True, noise_augment = 0,
             dupl_rows = False, TrainLabelFreqLevel = False, powertransform = False, binstransform = False,
             MLinfill = True, infilliterate=1, randomseed = False, eval_ratio = .5,
             numbercategoryheuristic = 255, pandasoutput = 'dataframe', NArw_marker = True,
             featureselection = False, featurethreshold = 0., inplace = False, orig_headers = False,
             Binary = False, PCAn_components = False, PCAexcl = [], excl_suffix = False,
             ML_cmnd = {'autoML_type':'randomforest',
                        'MLinfill_cmnd':{'RandomForestClassifier':{}, 'RandomForestRegressor':{}},
                        'PCA_type':'default',
                        'PCA_cmnd':{}},
             assigncat = {'1010':[], 'onht':[], 'ordl':[], 'bnry':[], 'hash':[], 'hsh2':[],
                          'DP10':[], 'DPoh':[], 'DPod':[], 'DPbn':[], 'DPhs':[], 'DPh2':[],
                          'nmbr':[], 'mnmx':[], 'retn':[], 'DPnb':[], 'DPmm':[], 'DPrt':[],
                          'bins':[], 'pwr2':[], 'bnep':[], 'bsor':[], 'por2':[], 'bneo':[],
                          'ntgr':[], 'srch':[], 'or19':[], 'tlbn':[], 'excl':[], 'exc2':[]},
             assignparam = {'global_assignparam'  : {'(parameter)': 42},
                            'default_assignparam' : {'(category)' : {'(parameter)' : 42}},
                                     '(category)' : {'(column)'   : {'(parameter)' : 42}}},
             assigninfill = {'stdrdinfill':[], 'MLinfill':[], 'zeroinfill':[], 'oneinfill':[],
                             'adjinfill':[], 'meaninfill':[], 'medianinfill':[], 'negzeroinfill':[],
                             'interpinfill':[], 'modeinfill':[], 'lcinfill':[], 'naninfill':[]},
             assignnan = {'categories':{}, 'columns':{}, 'global':[]},
             transformdict = {}, processdict = {}, evalcat = False, ppd_append = False,
             entropy_seeds = False, random_generator = False, sampling_dict = False,
             privacy_encode = False, encrypt_key = False, printstatus = 'summary', logger = {})
```

* df_train: a pandas dataframe or numpy array containing a structured 
dataset intended for use to subsequently train a machine learning model. 
The set at a minimum should be 'tidy' meaning a single column per feature 
and a single row per observation, with all unique string column headers. If 
desired the set may include one are more
"ID" columns (intended to be carved out and consistently shuffled or partitioned
such as an index column) and zero or one column intended to be used as labels 
for a downstream training operation. The tool supports the inclusion of 
non-index-range column as index or multicolumn index (requires named index 
columns). Such index types are added to the returned "ID" sets which are 
consistently shuffled and partitioned as the train and test sets. For passed
numpy array any label column should be the final column.

* df_test: a pandas dataframe or numpy array containing a structured 
dataset intended for use to generate predictions from a downstream machine 
learning model trained from the automunge returned sets. The set must be 
consistently formatted as the train set with consistent column headers and
order of columns. (This set may optionally contain a labels column if one 
was included in the train set although its inclusion is not required). If 
desired the set may include one or more ID column(s) or column(s) intended 
for use as labels. A user may pass False if this set is not available. The tool 
supports the inclusion of non-index-range column as index or multicolumn index 
(requires named index columns). Such index types are added to the returned 
"ID" sets which are consistently shuffled and partitioned as the train and 
test sets.

* labels_column: a string of the column title for the column from the
df_train set intended for use as labels in training a downstream machine
learning model. The function defaults to False for cases where the
train set does not include a label column. An integer column index may 
also be passed such as if the source dataset was a numpy array. A user can 
also pass True in which case the label set will be taken from the final
column of the train set (including cases of single column in train set).
A label column for df_train data is partitioned and returned in the labels set.
Note that a designated labels column will automatically be checked for in
corresponding df_test data and partitioned to the returned test_labels set when 
included. Note that labels_column can also be passed as a list of multiple
label columns. Note that when labels_column is passed as a list, a first entry
set bracket specification comparable to as available for the Binary parameter
can be applied to designate that multiple categoric labels in the list may be consolidated to a
single categoric label, such as to train a single classification model for multiple classification targets,
which form may then be recovered in a postmunge inversion='labels' operation, such as to convert the
consolidated form after an inference operation back to the form of separate inferred labels.
When passing data as numpy arrays the label column needs to be the final column (on far right of dataframe).

* trainID_column:  defaults to False, user can pass a column header or list of column headers
for columns that are to be segregated from the df_train set for return in the train_ID
set (consistently shuffled and partitioned when applicable). For example this may 
be desired for an index column or any other column that the user wishes to exclude from
the ML infill basis. Defaults to False for cases where no ID columns are desired. Note 
that when designating ID columns for df_train if that set of ID columns is present in df_test
they will automatically be given comparable treatment unless otherwise specified. An integer 
column index or list of integer column indexes may also be passed such as if the source dataset 
was a numpy array. Note that the returned ID sets (such as train_ID, val_ID, and test_ID) are automatically
populated with an additional column with header 'Automunge_index' which may serve as an
index column in cases of shuffling, validation partitioning, or oversampling. In cases of unnamed
non-range integer indexes, they are automatically extracted and returned in the ID sets as 'Orig_index'.
If a user would like to include a column both in the features for encoding and the ID sets for original form
retention, they can pass trainID_column as a list of two lists, e.g. [list1, list2], where the first
list may include ID columns to be struck from the features and the second list may include ID columns
to be retained in the features.

* testID_column: defaults to False, user can pass a column header or list of column headers
for columns that are to be segregated from the df_test set for return in the test_ID
set (consistently shuffled and partitioned when applicable). For example this may 
be desired for an index column or any other column that the user wishes to exclude from
the ML infill basis. Defaults to False, which can be used for cases where the df_test
set does not contain any ID columns, or may also be passed as the default of False when 
the df_test ID columns match those passed in the trainID_column parameter, 
in which case they are automatically given comparable treatment. Thus, the primary intended use 
of the testID_column parameter is for cases where a df_test has ID columns 
different from those passed with df_train. Note that an integer column index 
or list of integer column indexes may also be passed such as if the source dataset was a numpy array. 
(When passing data as numpy arrays one should match ID partitioning between df_test and df_train.) In cases of unnamed
non-range integer indexes, they are automatically extracted and returned in the ID sets as 'Orig_index'.
If a user would like to include a column both in the features for encoding and the ID sets for original form
retention, they can pass testID_column as a list of two lists, e.g. [list1, list2], where the first
list may include ID columns to be struck from the features and the second list may include ID columns
to be retained in the features. (We recommend only using testID_column specification for cases where df_test
includes columns that aren't present in df_train, otehrwise it is automatic.)

* valpercent: a float value between 0 and 1 which designates the percent
of the training data which will be set aside for the validation
set (generally used for hyperparameter tuning of a downstream model).
This value defaults to 0 for no validation set returned. Note that when
shuffletrain parameter is activated (which is default for train sets) validation
sets will contain random rows.  If shuffletrain parameter is set to False then any 
validation set will be pulled from the bottom sequential rows of the df_train dataframe.

valpercent can also be passed as a two entry tuple in the form valpercent=(start, end),
where start is a float in the range 0<=start<1, end is a float in the range 0<end<=1, and start < end.
For example, if specified as valpercent=(0.2, 0.4), the returned training data would consist of the first 20% of rows and the last 60% of rows, while the validation set would consist of the remaining rows, and
where the train and validation sets may then be subsequently individually shuffled when activated by the shuffletrain parameter. The purpose of this valpercent tuple option is to support integration into a cross validation operation, for example for a cross validation with k=3, automunge(.) could be called three times with valpercent passed for each as (0,0.33), (0.33,0.66), (0.66,1) respectively. Please note that when using automunge(.) in a cross-validation operation, we recommend using the postprocess_dict['final_assigncat'] entry populated in the first automunge(.) call associated with the first train/validation split as the assigncat entry passed to the automunge(.) assigncat parameter in each subsequent automunge(.) call associated with the remaining train/validation splits, which will speed up the remaining calls by eliminating the automated evaluation of data properties as well as mitigate risk of (remote) edge case when category assignment to a column under automation may differ between different validation set partitions due to deviations in aggregate data properties associated with a column.
```
#example of preparing k folds in a cross validation:
k=3
for i in range(k):
  print('processing fold #', i)

  #valpercent accepts a tuple of float ratios to set boundaries of validation split
  valpercent = (i/k, (i+1)/k)

  if i == 0:
    #can also populate any manual assignments here
    assigncat = {}
  elif i > 0:
    #after first fold use the final assigncat from prior
    #to turn off automated category assignments
    #which will speed it up and eliminate an edge case
    assigncat = postprocess_dict['final_assigncat']

  train, train_ID, labels, \
  val, val_ID, val_labels, \
  test, test_ID, test_labels, \
  postprocess_dict = \
  am.automunge(df_train,
               labels_column = labels_column,
               valpercent = valpercent,
               assigncat = assigncat)
  
  #train and evaluate model with train/labels and val/val_labels

#note that in edge case number of columns may vary between folds
#which could arrise from e.g. 1010 binarization exposed to different range of entries in a feature
#if this becomes an obstacle can manually specify the range of activation targets in assignparam
#e.g. assignparam = {'1010' : {'<targetfeature>' : {'all_activations' : list_of_unique_values_for_targetfeature}}}

#or by just manually specifying ordinal encoding to categoric features in assigncat
#e.g. assigncat = {'ordl' : list_of_categoric_features}

#it is also possible to process folds for i>0 with train and validation data prepared seperately in postmunge
#this would run faster e.g. by eliminating redundant ML infill model training
#and ensure that each fold has same number of columns
#albeit with tradeoff of not strictly adhering to segregation of train/validation basis
#for avoidance of data leakage
```

* floatprecision: an integer with acceptable values of _16/32/64_ designating
the memory precision for returned float values. (A tradeoff between memory
usage and floating point precision, smaller for smaller footprint.) 
This currently defaults to 32 for 32-bit precision of float values. Note
that there may be energy efficiency benefits at scale to basing this to 16.
Note that integer data types are still retained with this option.

* cat_type: accepts boolean defaulting to False, when True returned integer encoded categoric
features are converted to pandas categorical data type based on the transform's MLinfill_type. 
In some cases this may actually slightly increase dataframe memory usage and 
is redundant with information stored in the postprocess_dict, however we expect there
are potential downstream workflows where a user may prefer categoric data type which 
is the reason for the option. Note that for cases where a categoric transform feature 
did not have full representation in the training data set (e.g. as could be the case for fixed width bins with bnwd/bnwo/variants), 
it is possible that this option will result in test data returned with missing values designated 
as NaN entries (which is partly why this is not the default). Note that this same basis is carried through to postmunge.

* shuffletrain: can be passed as one of _{True, False, 'traintest'}_ which 
indicates if the returned sets will have their rows shuffled. Defaults to True 
for shuffling the train data but not the test data. False deactivates. To shuffle 
both train and test data can pass as 'traintest'. Note that this impacts the
validation split if a valpercent was passed, where under the default of True
validation data will be randomly sampled and shuffled, or when shuffletrain is 
deactivated validation data will be based on a partition of sequential rows from 
the bottom of the train set. Note that row correspondence with shuffling is
maintained between train / ID / label sets. Note that we recommend deactivating 
shuffletrain for sequential (time-series) data.

* noise_augment: accepts type int or float(int) >=0, defaults to 0. Used to specify 
a count of additional duplicates of training data prepared and concatenated with the
original train set. Intended for use in conjunction with noise injection, such that
the increased size of training corpus can be a form of data augmentation. (Noise injection
still needs to be assigned, e.g. by assigning root categories in assigncat or could
turn on automated noise with powertransform = 'DP1'). Note that 
injected noise will be uniquely randomly sampled with each duplicate. When noise_augment
is received as a dtype of int, one of the duplicates will be prepared without noise. When
noise_augment is received as a dtype of float(int), all of the duplicates will be prepared 
with noise. When shuffletrain is activated the duplicates are collectively shuffled, and can distinguish
between duplicates by the original df_train.shape in comparison to the ID set's Automunge_index.
Please be aware that with large dataframes a large duplicate count may run into memory constraints,
in which case additional duplicates can be prepared separately in postmunge(.). Note that the entropy seed budget only accounts for preparing one set of data, for the noise_augment option with entropy seeding we recommend passing a custom extra_seed_generator with a sampling_type specification, which will result in internal samplings of additional entropy seeds for each additional noise_augment duplicate (or for the bulk_seeds case with external sampling can increase entropy_seed budget proportional to the number of additional duplicates with noise).

* dupl_rows: can be passed as _(True/False/'traintest'/'test')_ which indicates
if duplicate rows will be consolidated to single instance in returned sets. (In
other words, if same row included more than once, it will only be returned once.)
Defaults to False for not activated. True applies consolidation to train set only,
'test' applies consolidation to test set only, 'traintest' applies consolidation 
to both train and test sets separately. Note this is applied prior to 
TrainLabelFreqLevel if elected. As implemented this does not take into account
duplicate rows in train/test data which have different labels, only one version
of features/label pair is returned.

* TrainLabelFreqLevel: can be passed as _(True/False/'traintest'/'test')_ 
which indicates if the TrainLabelFreqLevel method will be applied to prepare for 
oversampling training data associated with underrepresented labels (aka class 
imbalance). The method adds multiples of training data rows for those labels with 
lower frequency resulting in an (approximately) levelized frequency. This defaults 
to False. Note that this feature may be applied to numerical label sets if 
the processing applied to the set includes aggregated bins, such as for example
by passing a label column to the 'exc3' category in assigncat for pass-through
force to numeric with inclusion of standard deviation bins or to 'exc4'