semvec/pypi-packages huggingface dataset:
kafekatonic 1.0
Feature Engineering Automation
License: MIT
Requires Python: >=3.7, <4

# Scalable Katonic Feature Engineering



## Deployement Steps
* python 3.7 should be installed Please refer following links
    * To Install on Ubuntu use this [link](https://linuxize.com/post/how-to-install-python-3-7-on-ubuntu-18-04/) 
    * To Install on Centos use this [link](https://tecadmin.net/install-python-3-7-on-centos/)
    * To Install on Windows use this [link](https://medium.com/@itylergarrett.tag/how-to-install-python-3-7-on-windows-10-pc-the-non-developer-version-b063e1913b39)

* Download .whl from sharepoint using this [link](https://netorgft6673619.sharepoint.com/sites/KatonicPlatform/Shared%20Documents/General/25.%20Feature%20Engineering/release/v-1.0/katfeat-1.0.whl)

* Install package using the following command
    ```
    pip install katfeat-1.0.whl
    ```

## Steps for execution script creation 

### create spark session
```
from pyspark.sql import SparkSession
spark =  SparkSession.builder.master("local[*]").getOrCreate()
```

### import katfeat feature transformation
```
from katfeat.preprocess.feature_transformation import get_feature_transformation
```

### Create pyspark dataframe
```
input_path = "ChangeManagementServicenow.csv"
df = spark.read.csv(input_path,header=True, inferSchema=True)
```

### input id and label columns details of the data
```
label = "risk"
id_column_list = ["number"]
```

### Execute scalable auto feature transformation get auto featured transformation dataframe with summary stats metadata
```
df.cache()
stand_df, summ_meta = get_feature_transformation(spark, df, id_column_list = id_column_list, label_col = label)

stand_df.write.mode("overwrite").parquet("ChangeManagementServicenow")
summ_meta.to_csv("summary_meta.csv")
```

### Stop Spark Session
```
spark.stop()
```

### Save Script
```
save script as test.py
```

## Run using Spark-submit
```
spark-submit \
  --master local[*] \
  test.py
```