discovery-components-ds 3.2.7
Advanced data cleaning, data wrangling and feature extraction tools for ML engineers
License: BSD
Requires Python: >=3.6
Keywords: Wrangling ML Visualisation Dictionary Discovery Productize Classification Feature Engineering Cleansing

AI-STAC Discovery Transition and Feature Catalog
################################################

.. class:: no-web no-pdf

|pypi| |license| |wheel|


.. contents::

.. section-numbering::

What is AI-STAC
===============

“Exploratory data analysis can never be the whole story, but nothing else can serve as the foundation stone.”
— John Tukey

Augmented Intent - Single Task Accelerator components (AI-STAC) is a unique approach to data recovery, discovery, synthesis
and modeling that innovates the approach to data science and it's transition to production. it's origins came
from an incubator project that shadowed a team of Ph.D. data scientists in connection with the development and delivery
of machine learning initiatives to define measurable benefit propositions for customer success. From this, a number of
observable 'capabilities' were identified as unique and separate concerns. The challenges of the data scientist, and in
turn the production teams, were to effectively leveraging that separation of concern and distribute and loosely couple
the specialist capability needs to the appropriate skills set.

In addition the need to remove the opaque nature of the machine learning end-to-end required better transparency and
traceability, to better inform to the broadest of interested parties and be able to adapt without leaving being the
code 'sludge' of redundant ideas. AI-STAC is a disruptive innovation, changing the way we approach the challenges of
Machine Learning and Augmented Inelegance, introduces the ideas of 'Single Task Adaptive Component' around the
core concept of 'Parameterised Intent'

Main features
=============

* Machine Learning Capability Mapping
* Parametrised Intent
* Discovery Transitioning
* Feature Cataloguing
* Augmented Knowledge

Overview
========
AI-STAC is a change of approach in terms of improving productivity of the data
scientists. This approach deconstructs the machine learning discovery vertical into a set of capabilities, ideas and
knowledge.  It presents a completely novel approach to the traditional process automation and model wrapping that is
broadly offered as a solution to solve the considerable challenges that currently restrict the effectiveness of
machine learning in the enterprise business.

To achieve this, the project offers advanced and specialized programming methods that are unique in approach and novel
while maintaining familiarity within common tooling can be identified in four constructs.

1. Machine Learning Capability Mapping - Separation of capabilities, breaking the machine learning vertical into a set
of decoupled and targeted layers of discrete and refined actions that collectively present a human lead (ethical AI)
base truth to the next set of capabilities. This not only allows improved transparency of, what is, a messy and
sometimes confusing set of discovery orientated coded ideas but also loosely couples and targets activities that are,
generally, complex and specialized into identifiable and discrete capabilities that can be chained as separately
allocated activities.

2. Parametrized Intent - A unique technique extracting the ideas and thinking of the data scientist from their
discovery code and capturing it as intent with parameters that can be replayed against productionized code and data.
This decoupling and Separation of Concern between data, code and the intent of actions from that code on that data,
considerably improves time to market, code reuse, transparency of actions and the communication of ideas between data
scientists and product delivery specialists.

3. Discovery Transitioning - Discovery Transitioning - is a foundation of the sepatation of concerns between data
provisioning and feature selection. As part of the Accelerated ML discovery Vertical, Transitioning is a foundation
base truth facilitating a transparent transition of the raw canonical dataset to a fit-for-purpose canonical dataset
to enable the optimisation of discovery analysis and the identification of features-of-interest, for the data scientist
and created boundary separation of capabilities decoupling the Data Scientist for the Data Engineer. As output it also
provides 'intelligent Communication', not only to the Data Scientist through canonical fit-for-purpose datasets, but
more generally offers powerful visual discovery tools and artefact generation for production architects, data and
business SME's, Stakeholders and is the initiator of Augmented Knowledge for an enriched and transparent shared view of
the extended data knowledge.

4. Feature Cataloguing – With cross over skills within machine learning and advanced data heuristics,
investigation identified commonality and separation across customer engagements that particularly challenged our
Ph.D data scientists in their effective delivery of customer success. As a result the project designed and developed
Feature Cataloguing, a machine learning technique of extracting and engineering features and their characteristics
appropriately parameterized for model selection.  This technique implements a juxta view of how features are
characterized and presented to the modelling layer. Traditionally features are directly mapped as a representation
of the underlying data set. Feature Cataloguing treats each individual feature as its own individual set of
characteristics as its representation. The resulting outcome considerably improves experimentation, cross feature
association, even when unrelated in the original data sets, and the reuse of identified features-of-interest across
use case and business domains.

5. Augmented Knowledge - This the ability to capture information on data, activities and the rich stream of subject
matter expertise, injected into the machine learning discovery vertical to provide an Augmented n-view of the model
build. This includes security, sensitivity, data value scaling, dictionary, observations, performance, optimization,
bias, etc. This enriched view of data allows, amongst other things, improved knowledge share, AI explainability,
feature transparency, and accountability that feeds into AI ethics, and insight analysis.

Background
==========
Born out of the frustration of time constraints and the inability to show business value
within a business expectation, this project aims to provide a set of tools to quickly
produce visual and observational results. It also aims to improve the communication
outputs needed by ML delivery to talk to Pre-Sales, Stakholders, Business SME's, Data SME's
product coders and tooling engineers while still remaining within familiar code paragigms.

The package looks to build a set of outputs as part of standard data wrangling and ML exploration
that, by their nature, are familiar tools to the various reliant people and processes. For example
Data dictionaries for SME's, Visual representations for clients and stakeholders and configuration
contracts for architects, tool builders and data ingestion.

Discovery Transition
--------------------
Discovery Transition is first and key part of an end to end process of discovery, productization and tooling. It defines
the ‘intelligence’ and business differentiators of everything downstream.

To become effective in the Discovery Transition phase, the ability to be able to micro-iterate within distinct layers
enables the needed adaptive delivery and quicker returns on ML use case.

The building and discovery of an ML model can be broken down into three Separation of Concerns (SoC)
or Scope of Responsibility (SoR) for the ML engineer and ML model builder.

- Data Preparation
- Feature Engineering
- Model selection and optimisation

with a forth discipline of insight, interpretation and profiling as an outcome. these three SoC's can be perceived as
eight distinct disciplines

Conceptuasl Eight stages of Model preparation
---------------------------------------------
#. Connectivity (data sourcing and persisting, fit-for-purpose, quality, quantity, veracity, connectivity)
#. Data Discovery (filter, selection, typing, cleaning, valuing, validating)
#. Augmented Knowledge (observation, visualisation, knowledge, value scale)
#. Data Attribution (attribute mapping, quantitative attribute characterisation. predictor selection)
#. Feature Engineering (feature modelling, dirty clustering, time series, qualitative feature characterisation)
#. Feature Framing (hypothesis function, specialisation, custom model framing, model/feature selection)
#. Model Train (selection, optimisation, testing, training)
#. Model Predict (learning, feedback loops, opacity testing, insight, profiling, stabilization)

Though conceptual they do represent a set of needed disciplines and the complexity of the journey to quality output.

Layered approach and Capability Mapping
---------------------------------------
The idea behind the conceptual eight stages of Machine Learning is to layer the preparation and reuse of the activities
undertaken by the ML Data Engineer and ML Modeller. To provide a platform for micro iterations rather than a
constant repetition of repeatable tasks through the stack. It also facilitates contractual definitions between
the different disciplines that allows loose coupling and automated regeneration of the different stages of model
build. Finally it reduces the cross discipline commitments by creating a 'by-design' set of contracts targeted
at, and written in, the language of the consumer.

The concept of being able to quickly run over a single aspect of the ML discovery and then present a stable base for
the next layer to iterate against. this micro-iteration approach allows for quick to market adaptive delivery.

Getting Started
===============
The ``discovery-transition-ds`` package is a python/pandas implementation of the AI-STAC Transition component,
specifically aimed at Python, Numpy and Pandas based Data Science activities. It is build to be very light weight
in terms of package dependencies requiring nothing beyond what would be found in an basic Data Science environment.
Its designed to be used easily within multiple python based interfaces such as Jupyter, IDE or command-line python.

Installation
============

package install
---------------
The best way to install AI-STAC component packages is directly from the Python Package Index repository using pip.
All AI-STAC components are based on a pure python foundation package ``aistac-foundation``

.. code-block:: bash

    $ pip install aistac-foundation

The AI-STAC component package for the Transition is ``discovery-transition-ds`` and pip installed with:

.. code-block:: bash

    $ pip install discovery-transition-ds

if you want to upgrade your current version then using pip install upgrade with:

.. code-block:: bash

    $ pip install --upgrade discovery-transition-ds

First Time Env Setup
--------------------
In order to ease the startup of tasks a number of environment variables are available to pre-assign where and how
configuration and data can be collected. This can considerable improve the burden of setup and help in the migration
of the outcome contracts between environments.

In this section we will look at a couple of primary environment variables and demonstrate later how these are used
in the Component. In the following example we are assuming a local file reference but this is not the limit of how one
can use the environment variables to locate date from multiple different connection mediums. Examples of other
connectors include AWS S3, Hive, Redis, MongoDB, Azure Blob Storage, or specific connectors can be created very
quickly using the AS-STAC foundation abstracts.

If you are on linux or MacOS:

1. Open the current user's profile into a text editor.

.. code-block:: bash

    $> vi ~/.bash_profile.

2. Add the export command for each environment variable setting your preferred paths in this example I am setting
them to a demo projects folder

.. code-block:: bash

    # where to find the properties contracts
    export HADRON_PM_PATH=~/projects/demo/contracts

    # The default path for the source and the persisted data
    export HADRON_DEFAULT_PATH=~/projects/demo/data

3. In addition to the default environment variables you can set specific component environment variables. This is
particularly useful with the Transition component as source data tends to sit separate from our interim storage.
For Transition you replace the ``DEFAULT`` with ``TRANSITION``, and in this case specify this is the ``SOURCE`` path

.. code-block:: bash

    # specific to te transition component source path
    export HADRON_TRANSITION_SOURCE_PATH=/tmp/data/sftp

4. save your changes
5. re-run your bash_profile and check the variables have been set

.. code-block:: bash

    $> source ~/.bash_profile.
    $> env

Transition Task - Setup
=======================
The Transition Component is a 'Capability' component and a 'Separation of Concern' dealing specifically with the
transition of data from connectivity of data source to the persistence of 'data-of-interest' that has been prepared
appropriate for the language canonical, in this case 'Pandas DataFrame'.

In the following example we are assuming a local file reference and are using the default AI-STAC Connector Contracts
for Data Sourcing and Persisting, but this is not the limit of how one can use connect to data retrieval and storage.
Examples of other connectors include AWS S3, Hive, Redis, MongoDB, Azure Blob Storage, or specific connectors can be
created very quickly using the AS-STAC foundation abstracts.

Instantiation
-------------
The ``Transition`` class is the encapsulating class for the Transitioning Capability, providing a wrapper for
transitioning functionality. and imported as:

.. code-block:: python

    from ds_discovery import Transition

The easiest way to instantiate the ``Transition`` class is to use Factory Instantiation method ``.from_env(...)``
that takes advantage of our environment variables set up in the previous section. in order to differentiate each
instance of the Transition Component, we assign it a ``Task`` name that we can use going forward to retrieve
or re-create our Transition instance with all its 'Intent'

.. code-block:: python

    tr = Transition.from_env(task_name='demo')

Augmented Knowledge
-------------------
Once you have instantiated the Transition Task it is important to add a description of the task as a future remind,
for others using this task and when using the MasterLedger component (not covered in this tutorial) it allows for a
quick reference overview of all the tasks in the ledger.

.. code-block:: python

    tr.set_description("A Demo task used as an example for the Transitioning tutorial")

Note: the description should be a short summary of the task. If we need to be more verbose, and as good practice,
we can also add notes, that are timestamped and cataloged, to help augment knowledge about this
task that is carried as part of the Property Contract.

in the Transition Component notes are cataloged within five named sections:
* source - notes about the source data that help in what it is, where it came from and any SME knowledge of interest
* schema - data schemas to capture and report on the outcome data set
* observations - observations of interest or enhancement of the understanding of the task
* actions - actions needed, to be taken or have been taken within the task

each ``catalog`` can have multiple ``labels`` whick in tern can have multiple text entries, each text keyed by
timestamp. through the catalog set is fixed, ``labels`` can be any reference label

the following example adds a description to the source catalogue

.. code-block:: python

    tr.add_notes(catalog='source', label='describe', text="The source of this demo is a synthetic data set"

To retrieve the list of allowed ``catalog`` sections we use the property method:

.. code-block:: python

    tr.notes_catalog


We now have our Transition instance and had we previously set it up it will contain all the previously set
Property Contract

One-Time Connectors Settings
----------------------------
With each component task we need to set up its connectivity defining three ``Connector Contract`` which control the
loose coupling of where data is sourced and persisted to the code that uses it. Though we can define up each Connect
Contract, it is easier to take advantage of template connectors set up as part of the Factory initialisation method.

Though we can define as many Connector Contract as we like, by its nature, the Transition task has three key connectors
that need to be set up as a 'one-off' task. Once these are set they are stored in the Property Contract and thus do not
need to be set again.

Source Contract
~~~~~~~~~~~~~~~
Firstly we need to set up the 'Source Contract' that specifies the data to be sourced. Because we are taking advantage
of the environment variable ``HADRON_TRANSITION_SOURCE_PATH`` we only need to pass the source file name. In this
example we are also going to pass two 'optional' extra parameters that get passed directly to the Source reader,
``sep=`` and ``encoding=``

.. code-block:: python

    tr.set_source(uri_file='demo_data.txt', sep='\t', encoding='Latin1')


Persist Contract
~~~~~~~~~~~~~~~~
Secondly we need to specify where we are going to persist our data once we have transitioned it. Again we are going
to take advantage of what our Factory Initialisation method set up for us and allow the Transition task to define our
output based on constructed template Connector Contracts.

.. code-block:: python

    tr.set_persist()

Dictionary Contract
~~~~~~~~~~~~~~~~~~~
Finally, and optionally, we set up a Data Dictionary Connector that allows us to output a data dictionary of the source
or persist schema to a persisted state that can be shared with other parties of interest.
.. code-block:: python

    tr.set_dictionary()

Now we have set up the Connector Contracts we no longer need to reference this code again as the information as been
stored in the Property Contract. We will look later how we can report on these connectors and observe their settings

We are ready to go. The Transition task is ready to use.

Transition Task - Intent
========================

Instantiate the Task
--------------------

The easiest way to instantiate the ``Transition`` class is to use Factory Instantiation method ``.from_env(...)``
that takes advantage of our environment variables set up in the previous section. in order to differentiate each
instance of the Transition Component, we assign it a ``Task`` name that we can use going forward to retrieve
or re-create our Transition instance with all its 'Intent'

.. code-block:: python

    tr = Transition.from_env(task_name='demo')


Loading the Source Canonical
----------------------------

.. code-block:: python

    df = tr.load_source_canonical()


Canonical Reporting
-------------------

.. code-block:: python

    tr.canonical_report(df)

Parameterised Intent
--------------------
Parameterised intent is a core concept and represents the intended action and defining functions of the component.
Each method is known as a component intent and the parameters the task parameterisation of that intent. The intent
and its parameters are saved and can be replayed using the ``run_intent_pipeline(canonical)`` method

The following sections are a brief description of the intent and its parameters. To retrieve the list of available
intent methods in code run:

.. code-block:: python

    tr.intent_model.__dir__()

auto_clean_header
~~~~~~~~~~~~~~~~~
.. parsed-literal::

    def auto_clean_header(self, df, case=None, rename_map: dict=None, replace_spaces: str=None, inplace: bool=False,
                          save_intent: bool=None, intent_level: [int, str]=None):

        clean the headers of a pandas DataFrame replacing space with underscore

        :param df: the pandas.DataFrame to drop duplicates from
        :param rename_map: a from: to dictionary of headers to rename
        :param case: changes the headers to lower, upper, title, snake. if none of these then no change
        :param replace_spaces: character to replace spaces with. Default is '_' (underscore)
        :param inplace: if the passed pandas.DataFrame should be used or a deep copy
        :param save_intent: (optional) if the intent contract should be saved to the property manager
        :param intent_level: (optional) the level of the intent,
                        If None: default's 0 unless the global intent_next_available is true then -1
                        if -1: added to a level above any current instance of the intent section, level 0 if not found
                        if int: added to the level specified, overwriting any that already exist
        :return: if inplace, returns a formatted cleaner contract for this method, else a deep copy pandas.DataFrame.

auto_drop_correlated
~~~~~~~~~~~~~~~~~~~~
uses 'brute force' techniques to removes highly correlated columns based on the threshold,
        set by default to 0.998.

        :df: data: the Canonical data to drop duplicates from
        :threshold: (optional) threshold correlation between columns. default 0.998
        :inc_category: (optional) if category type columns should be converted to numeric representations
        :sample_percent: a sample percentage between 0.5 and 1 to avoid over-fitting. Default is 0.85
        :random_state: a random state should be applied to the test train split. Default is None
        :inplace: if the passed Canonical, should be used or a deep copy
        :save_intent: (optional) if the intent contract should be saved to the property manager
        :intent_level: (optional) the level of the intent,
                        If None: default's 0 unless the global intent_next_available is true then -1
                        if -1: added to a level above any current instance of the intent section, level 0 if not found
                        if int: added to the level specified, overwriting any that already exist
        :return: if inplace, returns a formatted cleaner contract for this method, else a deep copy Canonical,.

auto_remove_columns
~~~~~~~~~~~~~~~~~~~
auto removes columns that are np.NaN, a single value or have a predominant value greater than.

        :df: the pandas.DataFrame to auto remove
        :null_min: the minimum number of null values default to 0.998 (99.8%) nulls
        :predominant_max: the percentage max a single field predominates default is 0.998
        :nulls_list: can be boolean or a list:
                    if boolean and True then null_list equals ['NaN', 'nan', 'null', '', 'None', ' ']
                    if list then this is considered potential null values.
        :auto_contract: if the auto_category or to_category should be returned
        :test_size: a test percentage split from the df to avoid over-fitting. Default is 0 for no split
        :random_state: a random state should be applied to the test train split. Default is None
        :drop_empty_row: also drop any rows where all the values are empty
        :inplace: if to change the passed pandas.DataFrame or return a copy (see return)
        :save_intent: (optional) if the intent contract should be saved to the property manager
        :intent_level: (optional) the level of the intent,
                        If None: default's 0 unless the global intent_next_available is true then -1
                        if -1: added to a level above any current instance of the intent section, level 0 if not found
                        if int: added to the level specified, overwriting any that already exist
        :return: if inplace, returns a formatted cleaner contract for this method, else a deep copy pandas.DataFrame.

auto_to_category
~~~~~~~~~~~~~~~~
auto categorises columns that have a max number of uniqueness with a min number of nulls
        and are object dtype

        :df: the pandas.DataFrame to auto categorise
        :unique_max: the max number of unique values in the column. default to 20
        :null_max: maximum number of null in the column between 0 and 1. default to 0.7 (70% nulls allowed)
        :fill_nulls: a value to fill nulls that then can be identified as a category type
        :nulls_list:  potential null values to replace.
        :inplace: if the passed pandas.DataFrame should be used or a deep copy
        :save_intent: (optional) if the intent contract should be saved to the property manager
        :intent_level: (optional) the level of the intent,
                        If None: default's 0 unless the global intent_next_available is true then -1
                        if -1: added to a level above any current instance of the intent section, level 0 if not found
                        if int: added to the level specified, overwriting any that already exist
        :return: if inplace, returns a formatted cleaner contract for this method, else a deep copy pandas.DataFrame.

to_bool_type
~~~~~~~~~~~~
converts column to bool based on the map

        :df: the Pandas.DataFrame to get the column headers from
        :bool_map: a mapping of what to make True and False
        :headers: a list of headers to drop or filter on type
        :drop: to drop or not drop the headers
        :dtype: the column types to include or exclude. Default None else int, float, bool, object, 'number'
        :exclude: to exclude or include the dtypes
        :regex: a regular expression to search the headers
        :re_ignore_case: true if the regex should ignore case. Default is False
        :inplace: if the passed pandas.DataFrame should be used or a deep copy
        :save_intent: (optional) if the intent contract should be saved to the property manager
        :intent_level: (optional) the level of the intent,
                        If None: default's 0 unless the global intent_next_available is true then -1
                        if -1: added to a level above any current instance of the intent section, level 0 if not found
                        if int: added to the level specified, overwriting any that already exist
        :return: if inplace, returns a formatted cleaner contract for this method, else a deep copy pandas.DataFrame.

to_category_type
~~~~~~~~~~~~~~~~
converts columns to categories

        :df: the Pandas.DataFrame to get the column headers from
        :headers: a list of headers to drop or filter on type
        :drop: to drop or not drop the headers
        :dtype: the column types to include or exclude. Default None else int, float, bool, object, 'number'
        :exclude: to exclude or include the dtypes
        :regex: a regular expression to search the headers
        :re_ignore_case: true if the regex should ignore case. Default is False
        :as_num: if true returns the category as a category code
        :fill_nulls: a value to fill nulls that then can be identified as a category type
        :nulls_list:  potential null values to replace.
        :inplace: if the passed pandas.DataFrame should be used or a deep copy
        :save_intent: (optional) if the intent contract should be saved to the property manager
        :intent_level: (optional) the level of the intent,
                        If None: default's 0 unless the global intent_next_available is true then -1
                        if -1: added to a level above any current instance of the intent section, level 0 if not found
                        if int: added to the level specified, overwriting any that already exist
        :return: if inplace, returns a formatted cleaner contract for this method, else a deep copy pandas.DataFrame.

to_date_type
~~~~~~~~~~~~
converts columns to date types

        :df: the Pandas.DataFrame to get the column headers from
        :headers: a list of headers to drop or filter on type
        :drop: to drop or not drop the headers
        :dtype: the column types to include or exclude. Default None else int, float, bool, object, 'number'
        :exclude: to exclude or include the dtypes
        :regex: a regular expression to search the headers
        :re_ignore_case: true if the regex should ignore case. Default is False
        :inplace: if the passed pandas.DataFrame should be used or a deep copy
        :as_num: if true returns number of days since 0001-01-01 00:00:00 with fraction being hours/mins/secs
        :year_first: specifies if to parse with the year first
                If True parses dates with the year first, eg 10/11/12 is parsed as 2010-11-12.
                If both dayfirst and yearfirst are True, yearfirst is preceded (same as dateutil).
        :day_first: specifies if to parse with the day first
                If True, parses dates with the day first, eg %d-%m-%Y.
                If False default to the a prefered preference, normally %m-%d-%Y (but not strict)
        :date_format: if the date can't be inferred uses date format eg format='%Y%m%d'
        :save_intent: (optional) if the intent contract should be saved to the property manager
        :intent_level: (optional) the level of the intent,
                        If None: default's 0 unless the global intent_next_available is true then -1
                        if -1: added to a level above any current instance of the intent section, level 0 if not found
                        if int: added to the level specified, overwriting any that already exist
        :return: if inplace, returns a formatted cleaner contract for this method, else a deep copy pandas.DataFrame.

to_float_type
~~~~~~~~~~~~~
converts columns to float type

        :df: the Pandas.DataFrame to get the column headers from
        :headers: a list of headers to drop or filter on type
        :drop: to drop or not drop the headers
        :dtype: the column types to include or exclude. Default None else int, float, bool, object, 'number'
        :exclude: to exclude or include the dtypes
        :regex: a regular expression to search the headers
        :re_ignore_case: true if the regex should ignore case. Default is False
        :precision: how many decimal places to set the return values. if None then the number is unchanged
        :fillna: { num_value, 'mean', 'mode', 'median' }. Default to np.nan
                    - If num_value, then replaces NaN with this number value
                    - If 'mean', then replaces NaN with the mean of the column
                    - If 'mode', then replaces NaN with a mode of the column. random sample if more than 1
                    - If 'median', then replaces NaN with the median of the column
        :errors: {'ignore', 'raise', 'coerce'}, default 'coerce' }. Default to 'coerce'
                    - If 'raise', then invalid parsing will raise an exception
                    - If 'coerce', then invalid parsing will be set as NaN
                    - If 'ignore', then invalid parsing will return the input
        :inplace: if the passed pandas.DataFrame should be used or a deep copy
        :save_intent: (optional) if the intent contract should be saved to the property manager
        :intent_level: (optional) the level of the intent,
                        If None: default's 0 unless the global intent_next_available is true then -1
                        if -1: added to a level above any current instance of the intent section, level 0 if not found
                        if int: added to the level specified, overwriting any that already exist
        :return: if inplace, returns a formatted cleaner contract for this method, else a deep copy pandas.DataFrame.

to_int_type
~~~~~~~~~~~
converts columns to int type

        :df: the Pandas.DataFrame to get the column headers from
        :headers: a list of headers to drop or filter on type
        :drop: to drop or not drop the headers
        :dtype: the column types to include or exclude. Default None else int, float, bool, object, 'number'
        :exclude: to exclude or include the dtypes
        :regex: a regular expression to search the headers
        :re_ignore_case: true if the regex should ignore case. Default is False
        :fillna: { num_value, 'mean', 'mode', 'median' }. Default to 0
                    - If num_value, then replaces NaN with this number value
                    - If 'mean', then replaces NaN with the mean of the column
                    - If 'mode', then replaces NaN with a mode of the column. random sample if more than 1
                    - If 'median', then replaces NaN with the median of the column
        :errors: {'ignore', 'raise', 'coerce'}, default 'coerce'
                    - If 'raise', then invalid parsing will raise an exception
                    - If 'coerce', then invalid parsing will be set as NaN
                    - If 'ignore', then invalid parsing will return the input
        :inplace: if the passed pandas.DataFrame should be used or a deep copy
        :save_intent: (optional) if the intent contract should be saved to the property manager
        :intent_level: (optional) the level of the intent,
                        If None: default's 0 unless the global intent_next_available is true then -1
                        if -1: added to a level above any current instance of the intent section, level 0 if not found
                        if int: added to the level specified, overwriting any that already exist
        :return: if inplace, returns a formatted cleaner contract for this method, else a deep copy pandas.DataFrame.

to_normalised
~~~~~~~~~~~~~
converts columns to float type

        :df: the Pandas.DataFrame to get the column headers from
        :headers: a list of headers to drop or filter on type
        :drop: to drop or not drop the headers
        :dtype: the column types to include or exclude. Default None else int, float, bool, object, 'number'
        :exclude: to exclude or include the dtypes
        :regex: a regular expression to search the headers
        :re_ignore_case: true if the regex should ignore case. Default is False
        :precision: how many decimal places to set the return values. if None then the number is unchanged
        :inplace: if the passed pandas.DataFrame should be used or a deep copy
        :save_intent: (optional) if the intent contract should be saved to the property manager
        :intent_level: (optional) the level of the intent,
                        If None: default's 0 unless the global intent_next_available is true then -1
                        if -1: added to a level above any current instance of the intent section, level 0 if not found
                        if int: added to the level specified, overwriting any that already exist
        :return: if inplace, returns a formatted cleaner contract for this method, else a deep copy pandas.DataFrame.

to_numeric_type
~~~~~~~~~~~~~~~
converts columns to int type

        :df: the Pandas.DataFrame to get the column headers from
        :headers: a list of headers to drop or filter on type
        :drop: to drop or not drop the headers
        :dtype: the column types to include or exclude. Default None else int, float, bool, object, 'number'
        :exclude: to exclude or include the dtypes
        :regex: a regular expression to search the headers
        :re_ignore_case: true if the regex should ignore case. Default is False
        :precision: how many decimal places to set the return values. if None then the number is unchanged
        :fillna: { num_value, 'mean', 'mode', 'median' }. Default to np.nan
                    - If num_value, then replaces NaN with this number value. Must be a value not a string
                    - If 'mean', then replaces NaN with the mean of the column
                    - If 'mode', then replaces NaN with a mode of the column. random sample if more than 1
                    - If 'median', then replaces NaN with the median of the column
        :errors: {'ignore', 'raise', 'coerce'}, default 'coerce'
                    - If 'raise', then invalid parsing will raise an exception
                    - If 'coerce', then invalid parsing will be set as NaN
                    - If 'ignore', then invalid parsing will return the input
        :inplace: if the passed pandas.DataFrame should be used or a deep copy
        :save_intent: (optional) if the intent contract should be saved to the property manager
        :intent_level: (optional) the level of the intent,
                        If None: default's 0 unless the global intent_next_available is true then -1
                        if -1: added to a level above any current instance of the intent section, level 0 if not found
                        if int: added to the level specified, overwriting any that already exist
        :return: if inplace, returns a formatted cleaner contract for this method, else a deep copy pandas.DataFrame.

to_remove
~~~~~~~~~
remove columns from the pandas.DataFrame

        :df: the Pandas.DataFrame to get the column headers from
        :headers: a list of headers to drop or filter on type
        :drop: to drop or not drop the headers
        :dtype: the column types to include or exclude. Default None else int, float, bool, object, 'number'
        :exclude: to exclude or include the dtypes
        :regex: a regular expression to search the headers
        :re_ignore_case: true if the regex should ignore case. Default is False
        :inplace: if the passed pandas.DataFrame should be used or a deep copy
        :save_intent: (optional) if the intent contract should be saved to the property manager
        :intent_level: (optional) the level of the intent,
                        If None: default's 0 unless the global intent_next_available is true then -1
                        if -1: added to a level above any current instance of the intent section, level 0 if not found
                        if int: added to the level specified, overwriting any that already exist
        :return: if inplace, returns a formatted cleaner contract for this method, else a deep copy pandas.DataFrame.

to_select
~~~~~~~~~
selects columns from the pandas.DataFrame

        :df: the Pandas.DataFrame to get the column headers from
        :headers: a list of headers to drop or filter on type
        :drop: to drop or not drop the headers
        :dtype: the column types to include or exclude. Default None else int, float, bool, object, 'number'
        :exclude: to exclude or include the dtypes
        :regex: a regular expression to search the headers
        :re_ignore_case: true if the regex should ignore case. Default is False
        :inplace: if the passed pandas.DataFrame should be used or a deep copy
        :save_intent: (optional) if the intent contract should be saved to the property manager
        :intent_level: (optional) the level of the intent,
                        If None: default's 0 unless the global intent_next_available is true then -1
                        if -1: added to a level above any current instance of the intent section, level 0 if not found
                        if int: added to the level specified, overwriting any that already exist
        :return: if inplace, returns a formatted cleaner contract for this method, else a deep copy pandas.DataFrame.

to_str_type
~~~~~~~~~~~
converts columns to object type

        :df: the Pandas.DataFrame to get the column headers from
        :headers: a list of headers to drop or filter on type
        :drop: to drop or not drop the headers
        :dtype: the column types to include or exclude. Default None else int, float, bool, object, 'number'
        :exclude: to exclude or include the dtypes
        :regex: a regular expression to search the headers
        :re_ignore_case: true if the regex should ignore case. Default is False
        :use_string_type: if the dtype 'string' should be used or keep as object type
        :fill_nulls: a value to fill nulls that then can be identified as a category type
        :nulls_list:  potential null values to replace.
        :nulls_list: can be boolean or a list:
                    if boolean and True then null_list equals ['NaN', 'nan', 'null', '', 'None'. np.nan, None]
                    if list then this is considered potential null values.
        :inplace: if the passed pandas.DataFrame should be used or a deep copy
        :save_intent: (optional) if the intent contract should be saved to the property manager
        :intent_level: (optional) the level of the intent,
                        If None: default's 0 unless the global intent_next_available is true then -1
                        if -1: added to a level above any current instance of the intent section, level 0 if not found
                        if int: added to the level specified, overwriting any that already exist
        :return: if inplace, returns a formatted cleaner contract for this method, else a deep copy pandas.DataFrame.

Persist the Transitioned Canonical
----------------------------------


Save Clean Canonical
~~~~~~~~~~~~~~~~~~~~

.. code-block:: python

    tr.canonical_report(df_clean)

Save Data Dictionary
~~~~~~~~~~~~~~~~~~~~

.. code-block:: python

    tr.save_dictionary(tr.canonical_report(df, stylise=False))

Run Pipeline
------------

Locally
~~~~~~~

.. code-block:: python

    df_clean = tr.intent_model.run_intent_pipeline(df)

End-to-End
~~~~~~~~~~

.. code-block:: python

    tr.run_transition_pipeline()

Transparency and Traceability
=============================

Environ Report
------------------

.. code-block:: python

    tr.report_environ()

Connectors Report
-----------------

.. code-block:: python

    tr.report_connectors()

Intent Report
-------------

.. code-block:: python

    tr.report_Intent()

Run Book Report
---------------

.. code-block:: python

    tr.report_run_book()

Notes Report
------------

.. code-block:: python

    tr.report_Notes()

Schema Report
-------------


Reference
=========

Python version
--------------

Python 2.6,  2.7 or 3.5 are not supported. Although Python 3.6 is supported, it is recommended to install
``discovery-transition-ds`` against the latest Python 3.8.x whenever possible.

Pandas version
--------------

Pandas 0.25.x and above are supported but It is highly recommended to use the latest 1.0.x release as the first
major release of Pandas.

GitHub Project
--------------
discovery-transition-ds: `<https://github.com/Gigas64/discovery-transition-ds>`_.

Change log
----------

See `CHANGELOG <https://github.com/doatridge-cs/discovery-transition-ds/blob/master/CHANGELOG.rst>`_.


Licence
-------

BSD-3-Clause: `LICENSE <https://github.com/doatridge-cs/discovery-transition-ds/blob/master/LICENSE.txt>`_.


Authors
-------

`Gigas64`_  (`@gigas64`_) created discovery-transition-ds.


.. _pip: https://pip.pypa.io/en/stable/installing/
.. _Github API: http://developer.github.com/v3/issues/comments/#create-a-comment
.. _Gigas64: http://opengrass.io
.. _@gigas64: https://twitter.com/gigas64


.. |pypi| image:: https://img.shields.io/pypi/pyversions/Django.svg
    :alt: PyPI - Python Version

.. |rdt| image:: https://readthedocs.org/projects/discovery-transition-ds/badge/?version=latest
    :target: http://discovery-transition-ds.readthedocs.io/en/latest/?badge=latest
    :alt: Documentation Status

.. |license| image:: https://img.shields.io/pypi/l/Django.svg
    :target: https://github.com/Gigas64/discovery-transition-ds/blob/master/LICENSE.txt
    :alt: PyPI - License

.. |wheel| image:: https://img.shields.io/pypi/wheel/Django.svg
    :alt: PyPI - Wheel