kafe47 1.1
Feature Engineering Automation
License: MIT
Requires Python: >=3.7, <4

# Katonic Automatic Feature Engineering (KAFE)

KAFE is a python library used for automatic feature engineering. 


<br>

## Installation Steps

This following steps will help you setup KAFE on your system and import the package to use its modules within your python environment.


### Method 1 - Using .whl file

* python 3.7 should be installed.

* You can optionally create a virtual environment (venv) before proceeding. To learn how to setup follow this [link](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/)

* Download the .whl from sharepoint using this [link](https://netorgft6673619-my.sharepoint.com/personal/subhrajit_mohanty_katonic_ai/Documents/Microsoft%20Teams%20Chat%20Files/kafe-1.0-py3-none-any.whl)

* Run the following command to install the package through pip
    ```python
    >>   pip install kafe-1.0-py3-none-any.whl
    ```
* To test whether the package got successfully installed, import the package into python and check
   ```python
   >> import kafe
    ```
    
### Method 2 - Using Docker Custom File

* Install and run Docker and Docker Compose on your system from this [link](https://docs.docker.com/compose/install/)

* Download the latest stable version of KAFE’s Docker Custom File from this [link]()

* Run the following command to start running the container
   ```
    sudo docker-compose up -d
    ```
    
* You should now be able to connect to a local Jupyter notebook that contains KAFE at http://localhost:8000/tree. This may take a few minutes.
 
* Once Jupyter notebook opens up, test the successful installation of the package by running the following command
   ```python
   >> import kafe
    ```

<br>

## Usage Example

KAFE contains many modules which can be called for performing various feature engineering steps. Following are few examples of the same.

### Data Formating


```python
dfx = dtf.get_formatted_data(df)
dfx.toPandas().head(5)
```

<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>number</th>
      <th>knowledge</th>
      <th>active</th>
      <th>requested_by</th>
      <th>work_start</th>
      <th>reassignment_count</th>
      <th>escalation</th>
      <th>upon_approval</th>
      <th>sys_updated_by</th>
      <th>sys_created_on</th>
      <th>work_end</th>
      <th>close_notes</th>
      <th>sys_mod_count</th>
      <th>no_of_incidents</th>
      <th>category</th>
      <th>risk_factor</th>
      <th>risk</th>
      <th>risk_en</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>CHG0000094</td>
      <td>False</td>
      <td>True</td>
      <td>System Administrator</td>
      <td>23/08/2020 9:48</td>
      <td>0.0</td>
      <td>Normal</td>
      <td>Proceed to Next Task</td>
      <td>admin</td>
      <td>21/08/2020 8:21</td>
      <td>None</td>
      <td>None</td>
      <td>23</td>
      <td>95</td>
      <td>Network</td>
      <td>0.223438</td>
      <td>High</td>
      <td>1</td>
    </tr>
    <tr>
      <th>1</th>
      <td>CHG0000060</td>
      <td>False</td>
      <td>True</td>
      <td>System Administrator</td>
      <td>23/08/2020 9:48</td>
      <td>0.0</td>
      <td>Normal</td>
      <td>Proceed to Next Task</td>
      <td>admin</td>
      <td>21/08/2020 8:21</td>
      <td>None</td>
      <td>None</td>
      <td>25</td>
      <td>77</td>
      <td>Network</td>
      <td>0.311404</td>
      <td>Moderate</td>
      <td>0</td>
    </tr>
    <tr>
      <th>2</th>
      <td>CHG0002001</td>
      <td>False</td>
      <td>True</td>
      <td>User ITIL</td>
      <td>None</td>
      <td>0.0</td>
      <td>Normal</td>
      <td>Proceed to Next Task</td>
      <td>admin</td>
      <td>30/08/2018 8:02</td>
      <td>None</td>
      <td>None</td>
      <td>2</td>
      <td>92</td>
      <td>Other</td>
      <td>0.604531</td>
      <td>Moderate</td>
      <td>0</td>
    </tr>
    <tr>
      <th>3</th>
      <td>CHG0000070</td>
      <td>False</td>
      <td>True</td>
      <td>System Administrator</td>
      <td>23/08/2020 9:48</td>
      <td>0.0</td>
      <td>Normal</td>
      <td>Proceed to Next Task</td>
      <td>admin</td>
      <td>21/08/2020 8:21</td>
      <td>None</td>
      <td>None</td>
      <td>24</td>
      <td>81</td>
      <td>Network</td>
      <td>0.416384</td>
      <td>Moderate</td>
      <td>0</td>
    </tr>
    <tr>
      <th>4</th>
      <td>CHG0040005</td>
      <td>False</td>
      <td>True</td>
      <td>System Administrator</td>
      <td>None</td>
      <td>1.0</td>
      <td>Normal</td>
      <td>Proceed to Next Task</td>
      <td>admin</td>
      <td>10/08/2016 16:16</td>
      <td>None</td>
      <td>None</td>
      <td>1</td>
      <td>36</td>
      <td>Other</td>
      <td>0.130950</td>
      <td>Moderate</td>
      <td>0</td>
    </tr>
  </tbody>
</table>
</div>



### Summary Statistics


```python
sum_stats_df = summ.get_all_summary_stats(spark,dfx,"dfx")
sum_stats_df.head(5)
```

<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th>summary</th>
      <th>column_name</th>
      <th>dataframe_name</th>
      <th>transformation_step</th>
      <th>current_date_time</th>
      <th>is_categorical</th>
      <th>count</th>
      <th>mean</th>
      <th>stddev</th>
      <th>min</th>
      <th>mode</th>
      <th>...</th>
      <th>total_not_null_count</th>
      <th>null_value_count</th>
      <th>zero_value_count</th>
      <th>negative_value_count</th>
      <th>skewness</th>
      <th>kurtosis</th>
      <th>null_value_percentage</th>
      <th>zero_value_percentage</th>
      <th>negative_value_percentage</th>
      <th>countdistinct_percentage</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>number</th>
      <td>number</td>
      <td>dfx</td>
      <td>1</td>
      <td>2021-02-16 11:44:52.421620</td>
      <td>False</td>
      <td>50</td>
      <td>None</td>
      <td>None</td>
      <td>CHG0000002</td>
      <td>CHG0000039</td>
      <td>...</td>
      <td>50</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>None</td>
      <td>None</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>100.0</td>
    </tr>
    <tr>
      <th>knowledge</th>
      <td>knowledge</td>
      <td>dfx</td>
      <td>1</td>
      <td>2021-02-16 11:44:52.421620</td>
      <td>True</td>
      <td>50</td>
      <td>None</td>
      <td>None</td>
      <td>false</td>
      <td>false</td>
      <td>...</td>
      <td>50</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>None</td>
      <td>None</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>2.0</td>
    </tr>
    <tr>
      <th>active</th>
      <td>active</td>
      <td>dfx</td>
      <td>1</td>
      <td>2021-02-16 11:44:52.421620</td>
      <td>True</td>
      <td>50</td>
      <td>None</td>
      <td>None</td>
      <td>false</td>
      <td>true</td>
      <td>...</td>
      <td>50</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>None</td>
      <td>None</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>4.0</td>
    </tr>
    <tr>
      <th>requested_by</th>
      <td>requested_by</td>
      <td>dfx</td>
      <td>1</td>
      <td>2021-02-16 11:44:52.421620</td>
      <td>False</td>
      <td>50</td>
      <td>None</td>
      <td>None</td>
      <td>Don Goodliffe</td>
      <td>System Administrator</td>
      <td>...</td>
      <td>47</td>
      <td>3</td>
      <td>0</td>
      <td>0</td>
      <td>None</td>
      <td>None</td>
      <td>6.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>8.0</td>
    </tr>
    <tr>
      <th>work_start</th>
      <td>work_start</td>
      <td>dfx</td>
      <td>1</td>
      <td>2021-02-16 11:44:52.421620</td>
      <td>False</td>
      <td>50</td>
      <td>None</td>
      <td>None</td>
      <td>23/08/2020 9:48</td>
      <td>23/08/2020 9:48</td>
      <td>...</td>
      <td>39</td>
      <td>11</td>
      <td>0</td>
      <td>0</td>
      <td>None</td>
      <td>None</td>
      <td>22.0</td>
      <td>0.0</td>
      <td>0.0</td>
      <td>22.0</td>
    </tr>
  </tbody>
</table>
<p>5 rows × 27 columns</p>
</div>



### Variable Categorization


```python
df_vart, var_catt = variable_categorization.attribute_categorization(spark,dfx,sum_stats_df, id_column, label)
var_catt.head(5)
```

<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>column_name</th>
      <th>dataframe_name</th>
      <th>source_data_type</th>
      <th>category</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>risk_en</td>
      <td>dfx</td>
      <td>int</td>
      <td>LABEL</td>
    </tr>
    <tr>
      <th>1</th>
      <td>number</td>
      <td>dfx</td>
      <td>string</td>
      <td>ID</td>
    </tr>
    <tr>
      <th>2</th>
      <td>work_start</td>
      <td>dfx</td>
      <td>string</td>
      <td>DATE</td>
    </tr>
    <tr>
      <th>3</th>
      <td>risk</td>
      <td>dfx</td>
      <td>string</td>
      <td>CATEGORIC</td>
    </tr>
    <tr>
      <th>4</th>
      <td>upon_approval</td>
      <td>dfx</td>
      <td>string</td>
      <td>CATEGORIC</td>
    </tr>
  </tbody>
</table>
</div>

<br>

## Support

Project support can be found in following places :

* For bugs, issues, or feature requests start a [Github issue](https://github.com/katonic-dev/feature_engineering/issues).
* Reach out to our team for any other query at support@katonic.zendesk.com

<br>
<br>
<br>
<a href="https://katonic.ai"><img src="https://katonic.ai/wp-content/uploads/2020/09/katonic-logo.png" alt="logo" width="180" class="default-logo" align="right"></a>