jupiclass 1.0.0
Jupyter Notebook Image Classification assessment tool
License: BSD-3-Clause
Requires Python: >=2.7, !=3.0.*, !=3.1.*

<p align="center"><img src="./assets/logo.png"></p>

jupiclass is a **Jup**yter Notebook **I**mage **Cl**assification **ass**essment library. 

It performs the assessment of the development of an image classification model using Transfer Learning and Fine-Tuning training in a Jupyter Notebook, just requiring the user to set a [Python dictionary](#__ipynb). Also documents the model development by generating a model information JSON and a PDF containing information about the dataset, training, tests and interpretation questions to be performed on the Jupyter Notebook.

Please note that the library doesn't necessarily needs to be used with a Jupyter Notebook; if the `__ipynb` is filled, you can use it in any Python program.

This project was developed in the context of the [Computação na Escola](https://computacaonaescola.ufsc.br/) initiative, result of a collaboration of work by Gustavo de Castro Salvador, Prof.ª Dr.ª rer. nat. Christiane Gresse von Wangenheim, PMP and Prof. Dr. Jean Carlo Rossa Hauck.

# Use

## Install
```shell
$ pip install jupiclass fpdf PyPDF2
```

## Import
```python
from jupiclass import ImageClassification, evaluate, write_json, get_ninja_robot, PDFWriter

# Create new ImageClassification object with the __ipynb dictionary
jupic = ImageClassification(__ipynb)
# Evaluate the image classification
score = evaluate(jupic)
# Writes JSON with image classification data
write_json(jupic)
# Saves ninja robot image with a belt for the given score in './ninja_robot.png'
get_ninja_robot(score['total_score'])
# Writes PDF with image classification and interpretation answers
writer = PDFWriter(
    'logo.png', 'dataset_distribution.png', 
    'tl_confusion_matrix.png', 'tl_top_losses.png', 
    'ft_confusion_matrix.png', 'ft_top_losses.png')
writer.write()
```

# __ipynb
`ImageClassification` class expects a dictionary filled with image classification model training data. 

A simple example of a expected dictionary is the following:

> The example is of a tree classification model for six species: aroeira, jerivá, pitangueira, embaúba, mulungu and capororoca.

```python
__ipynb = {}
##### Model
__ipynb['model_categories'] = ['Aroeira','Jeriva','Pitangueira','Embauba', 'Mulungu', 'Capororoca']
__ipynb['model_correctly_labeled_images'] = 100
###### Dataset
__ipynb['dataset_categories_images'] = [{ 'Aroeira': 20 }, { 'Capororoca': 20 }, { 'Embauba': 20 }, { 'Jeriva': 20 }, { 'Mulungu': 20 }, { 'Pitangueira': 20 }]
__ipynb['dataset_total_images'] = 100
###### Transfer Learning
__ipynb['tl_models'] = ["resnet50", "resnet32"]
__ipynb['tl_epochs'] = [1, 2]
__ipynb['tl_learning_rates'] = [0e-1, 0e-2]
__ipynb['tl_trained'] = True
##### Accuracy - Transfer Learning
__ipynb['tl_accuracy_categories'] = [{ 'Aroeira': 0.9 }, { 'Capororoca': 0.9 }, { 'Embauba': 0.9 }, { 'Jeriva': 0.9 }, { 'Mulungu': 0.9 }, { 'Pitangueira': 0.9 }]
__ipynb['tl_accuracy_analysis'] = "Verdadeiro"
__ipynb['tl_accuracy_analysis_categories'] = ["Aroeira", "Capororoca", "Embauba", "Jeriva", "Mulungu", "Pitangueira"]
__ipynb['tl_accuracy_interpretation'] = "Falso"
##### Confusion matrix- Transfer Learning
__ipynb['tl_confusion_matrix_mislabeled_real'] = [{'Pitangueira': ['Capororoca', 'Aroeira', 'Embauba', 'Mulungu']}, {'Jeriva': []}, {'Capororoca': ['Jeriva', 'Aroeira']}, {'Mulungu': []}, {'Embauba': []}, {'Aroeira': ['Pitangueira', 'Embauba', 'Jeriva', 'Mulungu']}]_ipynb['tl_confusion_matrix_mislabeled'] = [{'Pitangueira': ['Aroeira', 'Capororoca', 'Embauba', 'Mulungu']}, {'Jeriva': []}, {'Capororoca': ['Aroeira', 'Jeriva']}, {'Mulungu': []}, {'Embauba': []}, {'Aroeira': ['Embauba', 'Jeriva', 'Mulungu', 'Pitangueira']}]
__ipynb['tl_confusion_matrix_interpretation'] = "Falso"
##### Fine-Tuning
__ipynb['ft_unfreezed'] = True
__ipynb['ft_learning_rate_found'] = True
__ipynb['ft_trained'] = True
##### Accuracy - Fine-Tuning
__ipynb['ft_accuracy_categories'] = [{ 'Aroeira': 0.1 }, { 'Capororoca': 0.2 }, { 'Embauba': 0.3 }, { 'Jeriva': 0.4 }, { 'Mulungu': 0.5 }, { 'Pitangueira': 0.6 }]
__ipynb['ft_accuracy_analysis'] = "Falso"
__ipynb['ft_accuracy_analysis_categories'] = ["Aroeira", "Capororoca", "Embauba", "Jeriva", "Mulungu", "Pitangueira"]
__ipynb['ft_accuracy_interpretation'] = "Falso"
##### Confusion matrix - Fine-Tuning
__ipynb['ft_confusion_matrix_mislabeled_real'] = [{'Pitangueira': ['Capororoca', 'Aroeira', 'Embauba', 'Mulungu']}, {'Jeriva': []}, {'Capororoca': ['Jeriva', 'Aroeira']}, {'Mulungu': []}, {'Embauba': []}, {'Aroeira': ['Pitangueira', 'Embauba', 'Jeriva', 'Mulungu']}]
__ipynb['ft_confusion_matrix_mislabeled'] = [{'Pitangueira': ['Capororoca', 'Aroeira', 'Embauba', 'Mulungu']}, {'Jeriva': []}, {'Capororoca': ['Jeriva', 'Aroeira']}, {'Mulungu': []}, {'Embauba': []}, {'Aroeira': ['Pitangueira', 'Embauba', 'Jeriva', 'Mulungu']}]
__ipynb['ft_confusion_matrix_interpretation'] = "Falso"
##### Performance
__ipynb['performance_tuning'] = "Sim, uma vez, mudando o que? Melhorou?"
__ipynb['performance_tuning_text'] = "Acurácia"
##### New objects
__ipynb['real_objects'] = ['Aroeira', 'Aroeira', 'Aroeira', 'Aroeira', 'Aroeira']
__ipynb['predicted_objects'] = ['Aroeira', 'Aroeira', 'Aroeira', 'Aroeira', 'Aroeira']
__ipynb['predicted_success_times'] = 5
__ipynb['predicted_success_interpretation'] = "Verdadeiro"
```

An example of a Jupyter Notebook with a image classification training using [fast.ai](https://fast.ai/), setting the `__ipynb` dictionary and using the `jupiclass` library as training assessment and documentation tool is available at [examples/Classificador_de_Imagens_de_Árvores_com_fast_ai.ipynb](/examples/Classificador_de_Imagens_de_Árvores_com_fast_ai.ipynb).

# PDF
For the PDF documentation there's a need for another extra dict variables to be set, as the example below:

```python
# Documentation PDF
__ipynb['model_name'] = "Modelo Teste"
__ipynb['model_date'] = "26/08/2021"
__ipynb['model_version'] = "v1.0.0"
__ipynb['model_task'] = "Classificar a espécie de árvore de uma imagem de árvore capturada de um aplicativo Android em relação a 6 categorias de árvores nativas/endêmicas de SC/Brasil. "
__ipynb['model_use_context'] = 'O modelo é utilizado como exemplo no contexto de ensino de na Educação Básica. Este modelo não foi treinado para ser utilizado em pesquisa na área de botânica.'
__ipynb['model_target_audience'] = 'Cidadãos (8+ anos), Foco em alunos do Ensino Médio'
__ipynb['model_risks'] = 'Risco de classificar erroneamente as espécies de árvores, porém se refere a classificação de árvores sem riscos à saúde dos usuários.'
__ipynb['model_task_type'] = 'Single-label classificação de imagens'
__ipynb['dataset_augmentation_size'] = "224x224 pixels"
__ipynb['dataset_augmentation_type'] = "rotate"
__ipynb['dataset_description'] = 'Conjunto de imagens de árvores (tipicamente a vista da árvore toda ou partes dentro do habitat natural (rua, praça,  parque etc.) capturada de um aplicativo Android'
__ipynb['dataset_origin'] = "Conjunto de dados de árvores disponibilizado pela CnE"
__ipynb['dataset_validation_percentage'] = 0.2
__ipynb['dataset_labeler_name'] = 'Por biólogos (conjunto de dados CnE)'
__ipynb['tl_batch_size'] = '200'
__ipynb['ft_learning_rate'] = '0.005'
__ipynb['ft_epoch'] = '35'
__ipynb['ethics_limitations'] = 'Esse modelo é limitado a somente 6 espécies nativas de árvores com um desempenho aceitável. Os resultados da classificação devem ser utilizados com cuidado sempre revisado por humanos.'
__ipynb['ethics_considerations'] = 'N/A'
__ipynb['author'] = 'C. Gresse von Wangenheim, R. M. Martins, A. Franz, G. Salvador, INCoD/INE/UFSC'
```

The Jupyter Notebook example available at [examples/Classificador_de_Imagens_de_Árvores_com_fast_ai.ipynb](/examples/Classificador_de_Imagens_de_Árvores_com_fast_ai.ipynb) also contains widgets to collect this data. There's also an example of a generated PDF at [examples/model_card.pdf](examples/model_card.pdf).

# API
The assessment is also available via HTTP API built with Flask. To build and execute the API Docker image, run:

```shell
$ make docker-build
$ make docker-run
```

The API has only one route, `POST /api/evaluate`. It accepts the JSON image classification data generated by `write_json`. An example of the POST body is available at [examples/request_body.json](/examples/request_body.json).

# Release
Releases are generated with [Poetry](https://python-poetry.org/). Generate a new version and publish the package to [PyPI](https://pypi.org/project/jupic-toolkit/):

> Please follow [Semantic Versioning](https://semver.org/).

```shell
$ make release_(patch|minor|major)
```