pycvtool 0.0.3
Deep learning tool for computer vision
License: MIT
Keywords: cvtool,computer vision,deep learning,object detection,instance segmentation,pose estimation,image classification

# CVTool

**This is a tool box for computer vision and deep learning, while providing [Flask Web API](#web-ui-application) for advanced applications.**

Note : The annotations must follow the [labelme](https://github.com/wkentaro/labelme) format.

The available model tools are as follows：

| MODEL                                                | TASK                  | REFERENCE                                                    |
| ---------------------------------------------------- | --------------------- | ------------------------------------------------------------ |
| [Torchvision](#torchvision-for-image-classification) | Image Classification  | [LINK](https://pytorch.org/vision/stable/models.html#classification) |
| [YOLOv7](#yolov7-for-object-detection)               | Object Detection      | [LINK](https://github.com/WongKinYiu/yolov7)                 |
| [UNet](#unet-for-semantic-segmentation)              | Semantic Segmentation | [LINK](https://github.com/xiaoyufenfei/Efficient-Segmentation-Networks) |
| [HRNet](#hrnet-for-semantic-segmentation)            | Semantic Segmentation | [LINK](https://github.com/CSAILVision/semantic-segmentation-pytorch) |
| [PointRend](#pointrend-for-instance-segmentation)    | Instance Segmentation | [LINK](https://github.com/ayoolaolafenwa/PixelLib)           |

## Requirements

- CUDA >= 11.8

- Python >= 3.7

- Windows Installation

  - Tested environment : CUDA==12.0, Python==3.8.10

  - Requirements : 

    ```bash
    pip install -r requirements/windows.txt
    ```

    ```
    pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
    ```

    ```shell
    pip install pycocotools-windows
    ```
    
    ```shell
    pip install flask
    ```

- Linux Installation

  - Tested environment : CUDA==12.0, Python==3.8.10

  - Requirements : 

    ```bash
    pip3 install -r requirements/linux.txt
    ```

## Dataset Preparation

Directory of the labelme dataset must contain all images and corresponding label json files, as shown below.

```markdown
├── dataset
│   ├── image0001.jpg
│   ├── image0001.json
│   ├── image0002.jpg
│   ├── image0002.json
│   ├── image0003.jpg
│   ├── image0003.json
│   ├──     .
│   ├──     .
│   ├──     .
│   ├── image9999.jpg
└── └── image9999.json
```

### YOLO format to Labelme format

If your dataset is in the format of [Yolo_mark](https://github.com/AlexeyAB/Yolo_mark), it can be converted to the [labelme](https://github.com/wkentaro/labelme) format by the script yolo2labelme.py. The method of use is as follows:

```shell
python cvtool/utils/yolo2labelme.py <label_dir> <output_dir> <classes>
```

- **label_dir** : path to a directory of yolo marks
- **output_dir** : path to a directory to save [labelme](https://github.com/wkentaro/labelme) annotations
- **classes** : path to a file with list of class names or a class string separated with comma, e.g., person,car,dog,cat
- note：This tool can still be used even if there are subfolders in your path, it will automatically search for all images and corresponding annotations json in the directory

You can also convert the format through script as follows,

```python
from cvtool.utils import yolo2labelme

yolo2labelme.convert(label_dir, output_dir, classes)
```

### MS COCO format to Labelme format

In addition, if you want to convert COCO's json format into Labelme, you can implement it as follows:

```shell
python cvtool/utils/coco2labelme.py <label_path> <image_dir> <output_dir>
```

- **label_path** : path to a JSON label file with [COCO](https://cocodataset.org/) format, i.e., instances_train2017.json
- **image_dir** : path to a root directory containing all images
- **output_dir** : path to a directory to save [labelme](https://github.com/wkentaro/labelme) annotations

You can also convert the format through script as follows,

```python
from cvtool.utils import coco2labelme

coco2labelme.convert(label_path, image_dir, output_dir)
```

### PASCAL VOC format to Labelme format

The following method can convert PASCAL VOC data into Labelme format:

```shell
python cvtool/utils/pascal2labelme.py <label_dir> <output_dir>
```

- **label_dir** : path to a directory of images and pascal poc labels <.xml>
- **output_dir** : path to a directory to save [labelme](https://github.com/wkentaro/labelme) annotations
- note：This tool can still be used even if there are subfolders in your path, it will automatically search for all images and corresponding annotations xml in the directory

You can also convert the format through script as follows,

```python
from cvtool.utils import pascal2labelme

pascal2labelme.convert(label_dir, output_dir)
```
### Mask image annotations to Labelme format
The following method can convert mask image annotations data into Labelme format:

```shell
python masks2labelme.py <mask_dir> <output_dir> <mask_mark> 
```

- **mask_dir** : path to a directory of original images and mask images <.jpg or .png>
- **output_dir** : path to a directory to save [labelme](https://github.com/wkentaro/labelme) annotations
- **mask_mark**： feature words of annotations data that can distinguish between the mask image and the original image on the path. (ex："_GT" or "annotations")
- note：This tool can still be used even if there are subfolders in your path, it will automatically search for all images and corresponding annotations mask in the directory

You can also convert the format through script as follows,

```python
from cvtool.utils import masks2labelme

masks2labelme.convert(mask_dir, output_dir, mask_mark)
```

### Examples for building datasets

1. Create dataset by the directory

```python
from cvtool.data import build_dataset
dataset = build_dataset("path/to/labelme/dataset/")
```

2. Split dataset into several subsets based on the given split ratios

```python
train_set, valid_set = dataset.random_split([0.8, 0.2], seed=21)
train_set, valid_set, test_set = dataset.random_split([0.7, 0.2, 0.1], seed=42)
```

3. Merge datasets

```python
dataset1 = build_dataset("path/to/the/first/labelme/dataset")
dataset2 = build_dataset("path/to/the/second/labelme/dataset")
# Append all data in dataset2 to dataset1
dataset1.merge(dataset2)
```

4. Flag dataset as the training set

```python
dataset.is_train = True
```

5. Dump dataset to a file that lists all image paths

```python
dataset.dump_list("train.txt")
```

6. View all object and class names

```python
print(dataset.object_names) # search all labels in `shapes`
print(dataset.class_names)  # search all flags in `flags`
```

7. Register dataset to a global variable `DATASETS`

```python
dataset.register("a-unique-dataset-name")
from cvtool.data import DATASETS # DATASETS is a global variable
dataset = DATASETS["a-unique-dataset-name"]
```

8. Iterate all images and labels

```python
for img, lbl in dataset:
    print(img, lbl)
```

9. Map class names, i.e., modify all stingray objects to be treated as `fish`

```python
dataset.map({
    "stingray": "fish", 
    "shark": "fish",
    "penguin": "bird",
    "puffin": "bird",
    "starfish": "cnidaria",
    "jellyfish": "cnidaria",
})
```

10. Dump and Load cache file

```python
from cvtool.data import build_dataset_from_cache

dataset.dump_cache("dataset.cache")
dataset = build_dataset_from_cache("dataset.cache", is_train=False)
```

## Torchvision for Image Classification

**Note:**

Depending on the version of torchvision installed, this classifier can support dozens of backbone networks.

You can build classifiers by calling the global variable `AVAILABLE_MODELS` to see the available backbone names.

```python
from cvtool.models.classification import AVAILABLE_MODELS
print(AVAILABLE_MODELS)
```

### Training Example

Dataset Preparation

You can split images into several folders as their categories as follows.

```
├── dataset
│   ├── class_name1
│   │   ├── image0001.jpg
│   │   ├── image0002.jpg
│   │   ├── image0003.jpg
│   ├──     .
│   ├──     .
│   ├── class_name2
│   │   ├── image0999.jpg
│   │   ├── image1000.jpg
│   │   ├── image1001.jpg
│   ├──     .
│   ├──     .
└── └──     .
```

The following method can convert image folders into into Labelme format as classification dataset

```shell
python cvtool/utils/dirs2labelme.py <base_dir> <output_dir>
```

- **base_dir** : path to the base directory of the images
- **output_dir** : path to a directory to save [labelme](https://github.com/wkentaro/labelme) annotations

```python
from cvtool.models.classification import efficientnet_b0
from cvtool.data import build_dataset

dataset = build_dataset("path/to/labelme/dataset/")
train_set, valid_set, test_set = dataset.random_split([0.7, 0.2, 0.1], seed=42)

trainer = efficientnet_b0.build_trainer()
trainer.cfg.learning_rate = 0.005 # learning rate (optional)
trainer.cfg.batch_size = 16 # training batch size (optional)
trainer.cfg.height = 512 # input height (optional)
trainer.cfg.width = 512 # input width (optional)
trainer.cfg.project = "output/" # path to save training results (optional)

# If you want to classify object boxes as ROIs, set trainer.cfg.roi = True
# You can use the pretrained weights initial model by setting trainer.cfg.weights = "weights.pth"
# You can resume training based on pretrained weights by setting trainer.cfg.resume = True

# start training
trainer.train(train_set, valid_set)
```

### Evaluation Example

```python
from cvtool.models.classification import efficientnet_b0
from cvtool.data import build_dataset

dataset = build_dataset("path/to/labelme/dataset/")
train_set, valid_set, test_set = dataset.random_split([0.7, 0.2, 0.1], seed=42)

evaluator = efficientnet_b0.build_evaluator()
evaluator.cfg.weights = "path/to/the/model_best.pth" # (required)
evaluator.cfg.cfg = "path/to/the/config.yaml" # (required)

# start evaluation
evaluator.evaluate(test_set)
```

### Test Example

```python
from cvtool.models.classification import efficientnet_b0

predictor = efficientnet_b0.build_predictor()
predictor.cfg.weights = "path/to/the/model_best.pth" # (required)
predictor.cfg.cfg = "path/to/the/config.yaml" # (required)
predictor.reset()

# start prediction
# The returned viz is the original image without any marks
(pred_name, prob), viz = predictor.predict("path/to/an/image.jpg", return_viz=True)

# If you want to predict ROI, slice the image before predicting, i.e.,
# x1, y1, x2, y2 = bbox
# (pred_name, prob), viz = predictor.predict(img[y1:y2, x1:x2])
```

## YOLOv7 for Object Detection

### Note: 

When using YOLOv7 in the Windows OS, in order to avoid repeated execution of threads, the code should be written as follows.

```python
if __name__ == "__main__":
    ### Code written here
```

or

```python
def main():
    ### Code written here

if __name__ == "__main__":
    main()
```

### Training Example (Resume, Transform)

```python
from cvtool.models.detection import yolov7
from cvtool.data import build_dataset
import os, cv2

dataset = build_dataset("path/to/labelme/dataset/")
train_set, valid_set, test_set = dataset.random_split([0.7, 0.2, 0.1], seed=42)

trainer = yolov7.build_trainer()
# def grayscale(img, lbl):
#     gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
#     return np.stack((gray,) * 3, -1), lbl

# 1. Resume training from pretrained weights
# trainer.cfg.weights = "path/to/last.pt"
# trainer.cfg.resume = True

# 2. transform images before forward pass
# trainer.cfg.transform_callback = grayscale

trainer.train(train_set, valid_set)
```

### Evaluation Example

```python
evaluator = yolov7.build_evaluator()

# the data was stored in the trainer.cfg.output_dir
evaluator.cfg.data = "path/to/data.yaml"
# the best weights were stored in the trainer.cfg.output_dir/weights/best.pt
evaluator.cfg.weights = "path/to/model/weights.pt"

# start evaluation
evaluator.evaluate(test_set)
```

### Test Example

```python
predictor = yolov7.build_predictor()

# the data was stored in the trainer.cfg.output_dir
predictor.cfg.data = "path/to/data.yaml"
# the best weights were stored in the trainer.cfg.output_dir/weights/best.pt
predictor.cfg.weights = "path/to/model/weights.pt"
# pretrained yolov7 weights for object detection (downloaded from https://github.com/WongKinYiu/yolov7/releases/download/v0.1/yolov7.pt)
predictor.cfg.weights = "path/to/cvtool/models/yolov7/weights/yolov7.pt"
predictor.reset()

# start prediction
detections, viz = predictor.predict("path/to/an/image.jpg", return_viz=True)
cv2.imwrite("detections.png", viz)
```

### YOLOv7 for Pose Estimation

```python
predictor = yolov7.build_predictor()
# assign the pretrained yolov7 weights for post estimation (downloaded from https://github.com/WongKinYiu/yolov7/releases/download/v0.1/yolov7-w6-pose.pt)
predictor.cfg.weights = "path/to/cvtool/models/yolov7/weights/yolov7-w6-pose.pt"
predictor.reset()

image_path = "path/to/an/image.jpg"

detections, viz = predictor.predict(image_path, return_viz=True)
cv2.imwrite("keypoints.png", viz)
```

## UNet for Semantic Segmentation

### Training Example

```python
from cvtool.models.segmentation import unet
from cvtool.data import build_dataset

dataset = build_dataset("path/to/labelme/dataset/")
train_set, valid_set, test_set = dataset.random_split([0.7, 0.2, 0.1], seed=42)

trainer = unet.build_trainer()

trainer.cfg.project = "path/to/save/model"
# total epochs for training
trainer.cfg.epochs = 250

trainer.train(train_set, valid_set)
```

### Evaluation Example

```python
evaluator = unet.build_evaluator()

# path to the pretrained weights
evaluator.cfg.weights = "path/to/the/pretrained/model_best.pth"
# path to the config file stored in the [project] folder
evaluator.cfg.cfg = "path/to/the/config.yaml"

# start evaluation
metrics = evaluator.evaluate(test_set)
```

### Test Example

```python
predictor = unet.build_predictor()

# path to the pretrained weights
predictor.cfg.weights = "path/to/the/pretrained/model_best.pth"
# path to the config file stored in the [project] folder
predictor.cfg.cfg = "path/to/the/config.yaml"
predictor.reset()

# start prediction
outputs, viz = predictor.predict("path/to/an/image", return_viz=True)
cv2.imwrite("segmentation.png", viz)
```
## HRNet for Semantic Segmentation

### Note: 
When using hrnet segmentation you need to set annotations mode, "ori_mask" or "fill_poly".
- "fill_poly"：Use the polygon list in the json file to fill to generate a mask image.

  **Note** : In "fill_poly" mode, if your labelme polygon list is generate by Contours detect (like masks2labelme.py). It might have some loss in detect process, so in this mode the final annotations may not be equal to the original mask annotations.  If not necessary, use "ori_mask" mode.

- "ori_mask" : When you original annotations are mask images, and you don't need to change original annotations, please use this mode.
If you set "ori_mask" mode, please put directory of the original dataset and labelme dataset in the same dir, and use "_Labelme" to distinguish between the original dataset and the dataset converted to labelme(labelme format as described above), detailed structure as shown below.
```
├── all_dataset_dir
│   ├── dataset
│   │   ├── annotations
│   |   |   ├──image0001.jpg
│   |   |   ├──image0002.jpg
│   |   |   | .
│   |   |   | .
│   │   ├── images (Non-essential)
│   |   |   ├──image0001.jpg
│   |   |   ├──image0002.jpg
│   │   |   | .
│   │   |   | .
│   ├── dataset_Labelme
│   │   ├── image0001.jpg
│   │   ├── image0001.json
│   │   ├── image0002.jpg
│   │   ├── image0002.json
│   │   |     .
│   │   |     .
└── └──       .
```

### Training Example

```python
from cvtool.models.segmentation import hrnet
from cvtool.data import build_dataset

dataset = build_dataset("path/to/labelme/dataset/")
train_set, valid_set, test_set = dataset.random_split([0.7, 0.2, 0.1], seed=42)

trainer = hrnet.build_trainer()
# Parameter configuration files, you can find more model mode configuration files in the config folder
trainer.cfg.cfg ="./cvtool/models/hrnet/utils/config/ade20k-hrnetv2.yaml"
# If you want to resume the weight you train before, just set start_epoch to last time epoch end number

# set 'ori_mask' or 'fill_poly' for annotations mode
trainer.cfg.mode = 'ori_mask'
            
trainer.train(train_set, valid_set)
```
### Evaluation Example

```python
evaluator = hrnet.build_evaluator()

# path/to/your/weights/folder/XXX.pth, XXX.pth can select best.pth or last.pth. Please ignore encoder_and decoder_ word system will automatically capture them.(required)
predictor.cfg.weights = "path/to/your/weights/folder/best.pth"

# path to the config file stored in the [project] folder
evaluator.cfg.cfg = "path/to/the/config.yaml"

# start evaluation
evaluator.evaluate(test_set)
```

### Test Example

```python
predictor = hrnet.build_predictor()

# path/to/your/weights/folder/XXX.pth, XXX.pth can select best.pth or last.pth. Please ignore encoder_and decoder_ word system will automatically capture them.(required)
predictor.cfg.weights = "path/to/your/weights/folder/best.pth"
# path to the config file stored in the [project] folder
predictor.cfg.cfg = "path/to/the/config.yaml"

# predictor.cfg.class_names_csv =  './cvtool/models/hrnet/utils/data/object150_info.csv'

# start prediction
# result will save in TEST.result path which set by config.yaml
predictor.reset()
segm, viz = predictor.predict("path/to/an/image", return_viz=True)
```
- **class_names_csv**： if in cfg your class_name is named by id_num, you can use a csv to replace it. sample csv is in './cvtool/models/hrnet/utils/data/object150_info.csv'

## PointRend for Instance Segmentation

### Training Example

```python
from cvtool.models.detection import pointrend
from cvtool.data import build_dataset
import os, cv2

dataset = build_dataset("path/to/labelme/dataset/")
train_set, valid_set, test_set = dataset.random_split([0.7, 0.2, 0.1], seed=42)

trainer = pointrend.build_trainer()

# path to save results
trainer.cfg.project = "path/to/save/model"
# total epochs for training
trainer.cfg.epochs = 250
# resume from the last checkpoint
trainer.cfg.resume = False

# start training
trainer.train(train_set, valid_set)
```

### Evaluation Example

```python
evaluator = pointrend.build_evaluator()

# path to the pretrained weights
evaluator.cfg.weights = "path/to/the/pretrained/model_final.pth"
# path to the config file stored in the [project] folder
evaluator.cfg.cfg = "path/to/the/config.yaml"

# start evaluation
metrics = evaluator.evaluate(test_set)
```

### Test Example

```python
predictor = pointrend.build_predictor()

# path to the pretrained weights
evaluator.cfg.weights = "path/to/the/pretrained/model_final.pth"
# path to the config file stored in the [project] folder
evaluator.cfg.cfg = "path/to/the/config.yaml"
predictor.reset()

# start prediction
outputs, viz = predictor.predict("path/to/an/image")
cv2.imwrite("detections.png", viz)
```

## Deployment

*This section mainly describes the functions or objects that may be used when deploying*

### Training in a thread

```python
# This method does not affect the execution of the main program when training
from cvtool.utils.thread import StoppableThread

training_thread = StoppableThread(target=trainer.train, args=(train_set, valid_set), daemon=True)
training_thread.start()

# Do something in the main program
# progress = trainer.get_progress()  # Get the current training progress (0.0 ~ 1.0)
# training_thread.stop()			# Stop training
```

### Setup

Run ```python setup.py`` to compile cvtool.

Tested environment :

- Linux : Python==3.8.10, Cython==0.29.34
- Windows : Python==3.7.9, Cython==0.29.33 on Visual Studio 2019 Community

## Change Logs

2023.04.11：Upload YOLOv7 engines

2023.04.14：Modify some YOLOv7 default configurations

2023.04.23：Update YOLOv7 for Pose Estimation

2023.04.27：Add annotation converter from YOLO mark to labelme format

2023.05.09：Indicate data and weights location in README

2023.05.16：Update the dataset cache dumping and loading method

2023.05.17 : Upload PointRend engines

2023.05.26 : Update the method of model training in a thread

2023.06.01 : Upload the engine and instructions of the classifiers

2023.06.20 : Upload the engine and instructions of the UNet. Add descriptions while printing the engine configs.

2023.07.10 : Upload CVTool Web UI. Update requirements for web UI.

2023.07.20 : Change blueprint import method in the app.

2023.07.31 : Update blueprint method for UI design. Add descriptions for custom pipeline.

2023.08.21 : Move the instructions related to Web UI to cvtool/app/README.md

2023.09.18 : Upload HRNet engines for semantic segmentation

2023.09.28 : Upload setup.py for compiling