mle-training 1.0.0
TAMLEP Assignment 2.1 MLE-TRAINING
Requires Python: >=3.9


# Assignment 2.1 Median housing value prediction

The housing data can be downloaded from https://raw.githubusercontent.com/ageron/handson-ml/master/. The script has codes to download the data. We have modelled the median house value on given housing data. 

The following techniques have been used: 

 - Linear regression
 - Decision Tree
 - Random Forest

 ## Steps performed
 - We prepare and clean the data. We check and impute for missing values.
 - Features are generated and the variables are checked for correlation.
 - Multiple sampling techinuqies are evaluated. The data set is split into train and test.
 - All the above said modelling techniques are tried and evaluated. The final metric used to evaluate is mean squared error.

 ## To excute the scripts
 Assuming that we are in project root directory i.e mle-training
 ```
 cd mle-training
 ```
 - First Create conda environment : 
   ```
    conda env create -f env.yml 
    ```
 - Activate conda environment : 
   ```
   conda activate mle-dev
   ```
 - Project contains three modules available under src/main folder.

   1) ingest_data.py
   2) train.py
   3) score.py

 ### Execute the commands in following order to run individual scripts:
- To run ingest_data module : 
```
python src/main/ingest_data.py [--path [PATH]] [--log_level [LOG_LEVEL]] [--log_path [LOG_PATH]] [--no_console_log]
```
```
    optional arguments:   
    -h, --help            show this help message and exit
    --path [PATH]         Give the desired folder path to save the train and test datasets. If this argument is not provided, then datasets will be stored in default directory data/processed
    --log_level [LOG_LEVEL]
                            log level e.g DEBUG,ERROR,INFO,WARNING | default-DEBUG
    --log_path [LOG_PATH]
                            file path to store logs. If this argument is not specified, logs will not be stored in file.
    --no_console_log      Specify this arg to skip logs priniting in console. By default, logs will be displayed on console.
```
This command will save train and test datasets in data/processed folder of project root directory, by default if no arguments passed .

- To run train module : 
   ```
   python src/main/train.py [-i [INPUTPATH]] [-o [OUTPUTPATH]] [--log_level [LOG_LEVEL]] [--log_path [LOG_PATH]] [--no_console_log]
   ```
   ```
    optional arguments:
    -h,  --help          show this help message and exit
    -i [INPUTPATH], --inputPath [INPUTPATH]
                            Give the desired folder path to load the train datasets. If this argument is not provided, then train datasets will be loaded from default directory data/processed.
    -o [OUTPUTPATH], --outputPath [OUTPUTPATH]
                            Give the desired folder path to save model object. By default, saves model object in artifacts folder of project directory.
    --log_level [LOG_LEVEL]   log level e.g DEBUG,ERROR,INFO,WARNING | default-DEBUG
    --log_path [LOG_PATH]     file path to store logs. If this argument is not specified, logs will not be stored in file.
    --no_console_log          Specify this arg to skip logs priniting in console. By default, logs will be displayed on  console.
   ```
   Running this command will save trained model object "model.pkl" into artifacts folder of project root directory by default if no arguments passed .
 - To run score module : 
   ```
   python src/main/score.py [-m [MODELPATH]] [-d [DATAPATH]] [--log_level [LOG_LEVEL]] [--log_path [LOG_PATH]] [--no_console_log]
   ```
   ```
    optional arguments:
    -h, --help            show this help message and exit
    -m [MODELPATH], --modelPath [MODELPATH]
                            Give the desired folder path to load model. If this argument is not provided, then model object will be loaded from default directory artifacts.
    -d [DATAPATH], --dataPath [DATAPATH]
                            Give the desired folder path to load the dataset. If this argument is not provided, then test datasets will be loaded from default directory data/processed
    --log_level [LOG_LEVEL]
                            log level e.g DEBUG,ERROR,INFO,WARNING | default-DEBUG
    --log_path [LOG_PATH]
                            file path to store logs. If this argument is not specified, logs will not be stored in file.
    --no_console_log      Specify this arg to skip logs priniting in console. By default, logs will be displayed on  console.
   ```
Running this command will load the model and make predictions on test data.


## Testing
 - For Installation Tests, run command : 
 ```
 pytest tests/test_installation.py -v 
 ```
   This will run all the test cases required for correctly working of project. If sees any test failures, act wisely as per failure message.

 - For testing invidual modules, run command :
   1) Testing ingest_data module:
   ``` 
   pytest tests/unit_tests/test_ingest_data.py -v
   ```
   2) Testing train module:
   ``` 
   pytest tests/unit_tests/test_train.py -v
   ```
   3) Testing score module:
   ``` 
   pytest tests/unit_tests/test_score.py -v
   ```


- To run all tests at once, run command:
```
pytest -v
```
This will display summary of all test cases performed for each modules i.e ingest_data, train and score.
    

## Jupyter Notebooks
  Make sure, conda environment is activated.
  - To run jupyter notebooks, run command:
  ```
  jupyter notebook
  ```
This will open the jupyter notebook in the browser, if not, open the link from terminal.

Then, go to notebooks folder and open the notebook.

To run each cell by cell, hit  ``` Shift + Enter ```


## Command To run MLFlow UI
 To create a new MLFlow run, run command :
 ```
 python src/main/main.py
 ```
 This will run the main script that runs everything together under a single parent mlflow run-id. Each of the child tasks (i.e. data preparation, model training and scoring etc) will get their own mlflow run-id but run as child runs of the main run.

WE can create MLFlow runs using below commands as well:
* Below command creates a new conda environment that is mentioned in the `env.yml` file to run the python scripts.
```
  mlflow run . --entry-point [main | ingest-data | train | score] 
```
* Run the below command to run in the current active conda environment.
```
mlflow run . --env-manager=local --entry-point [main | ingest-data | train | score]
```

 To  view and track mlflow runs, run MLFlow UI with below command:
```
mlflow ui
```

It will genearate a link on local host (http://127.0.0.1:5000). Open this link in browser and see the logged parameters, metrics and artifacts.