Face Mask Detection With ResNet50 and SVM + Decision Tree
Tags: machinelearning, computervision, resnet
Published: 2022-04-15 13:38:20

![Face Mask Detection](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w57dp9r8aobnde3vna9c.png)

Welcome, this post is a quick explanation on how I build mask detection using [ResNet50 ](https://pytorch.org/hub/nvidia_deeplearningexamples_resnet50)as feature extractor and then use [Support Vector Machine (SVM)](http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html) + [Decision Tree](https://scikit-learn.org/stable/modules/tree.html#classification) with [stacking ensemble method](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.StackingClassifier.html) as classifier.

As tribute to fellow researcher, this app was based on research paper with title "[A hybrid deep transfer learning model with machine learning methods for face mask detection in the era of the COVID-19 pandemic](https://www.sciencedirect.com/science/article/pii/S0263224120308289)" written by Mohamed Loey, et al.

Table of contents:
* [Dataset Retrieval](#dataset-retrieval)
* [Preprocessing](#preprocessing)
* [Feature Extraction](#feature-extraction)
* [Split Dataset](#split-dataset)
* [Define Model Classifier](#define-model-classifier)
* [Tuning Model](#tuning-model)
* [Create Final Model](#create-final-model)
* [Deploy Real App](#deploy-real-app)


### Dataset Retrieval <a name="dataset-retrieval"></a>
This application uses a dataset from Kaggle. This dataset contains 853 images belonging to the 3 classes, as well as their bounding boxes in the PASCAL VOC format. The classes are _with_mask_, _without_mask_, and _mask_weared_incorrect_. For some reason, I only use the _with_mask_ and _without_mask_ labels. Check out this image sample below.

![Sample 1 Dataset](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rt59tyvz73z0xtzod8gf.png)
![Sample 2 Dataset](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/281s5i78rfopxn5s7st0.png)

You can access this dataset via this url below.
[https://www.kaggle.com/datasets/andrewmvd/face-mask-detection](https://www.kaggle.com/datasets/andrewmvd/face-mask-detection)

### Preprocessing <a name="preprocessing"></a>
Preprocessing can be achieved by cropping face area based on bounding box information. First, read all xml file and image file from dataset folder.
```python
import os

img_names = []
xml_names = []
for dirname, _, filenames in os.walk('./face-mask-detection'):
  for filename in filenames:
    if os.path.join(dirname, filename)[-3:] != "xml":
      img_names.append(filename)
    else:
      xml_names.append(filename)

print(len(img_names), "images")
```

Then crop all images by its bounding box and read the label.
```python
import xmltodict
from matplotlib import pyplot as plt
from skimage.io import imread

path_annotations = "face-mask-detection/annotations/"
path_images = "face-mask-detection/images/"

class_names = ['with_mask', 'without_mask']
images = []
target = []

def crop_bounding_box(img, bnd):
  x1, y1, x2, y2 = list(map(int, bnd.values()))
  _img = img.copy()
  _img = _img[y1:y2, x1:x2]
  _img = _img[:,:,:3]
  return _img

for img_name in img_names[:]:
  with open(path_annotations+img_name[:-4]+".xml") as fd:
    doc = xmltodict.parse(fd.read())
  
  img = imread(path_images+img_name)
  temp = doc["annotation"]["object"]
  if type(temp) == list:
    for i in range(len(temp)):
      if temp[i]["name"] not in class_names:
        continue
      images.append(crop_bounding_box(img, temp[i]["bndbox"]))
      target.append(temp[i]["name"])
  else:
    if temp["name"] not in class_names:
        continue
    images.append(crop_bounding_box(img, temp["bndbox"]))
    target.append(temp["name"])
```

Based on labels, this dataset consists of 3232 with mask faces and 717 without mask faces.
![Chart relationship between faces and label](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ve8svltiyfbzo5hrr0it.png)

This preprocessing also contains resize and normalization steps for ImageNet.
```python
import torch

from torchvision import transforms

# Define preprocessing
preprocess = transforms.Compose([
  transforms.ToPILImage(),
  transforms.Resize((128, 128)),
  transforms.ToTensor(),
  transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),
])

# Apply preprocess
image_tensor = torch.stack([preprocess(image) for image in images])
image_tensor.shape
```

### Feature Extraction <a name="feature-extraction"></a>
Feature extraction is needed to gather information from images using spatial operations to extract something that represents a label. In this application, I use ResNet50 as a feature extractor. The last layer of ResNet, which is a fully connected layer with 1.000 neurons, needs to be deleted.
```python
from torchvision import models

# Download model
resnet = models.resnet50(pretrained=True)
resnet = torch.nn.Sequential(*(list(resnet.children())[:-1]))
```

To freeze and keep the convolutional part of ResNet50 fixed, I need to set `requires_grad` to `False`.
```python
for param in resnet.parameters():
    param.requires_grad = False
```

I also need to call `eval()` to set ResNet50's batch normalization to disabled. Which will interfere with model accuracy and make sure ResNet50 only acts as a feature extractor.
```python
resnet.eval()
```

Last step apply ResNet50 to extract feature. Then ResNet will return a vector with 2048 features for each image.
```python
import numpy as np

result = np.empty((len(image_tensor), 2048))
for i, data in enumerate(image_tensor):
  output = resnet(data.unsqueeze(0))
  output = torch.flatten(output, 1)
  result[i] = output[0].numpy()
```

### Split Dataset <a name="split-dataset"></a>
To prevent the model from overfitting, I needed to split the data into 70% train data and 30% test data. Train data will be used to train the model and test data will be used to test or validate the model.
```python
from sklearn.model_selection import train_test_split

X, y = result, np.array(target)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

print("Training data\n", np.asarray(np.unique(y_train, return_counts=True)).T)
print("Test data\n", np.asarray(np.unique(y_test, return_counts=True)).T)
```

### Define Model Classifier <a name="define-model-classifier"></a>
As I have teased before, the proposed model is a stacking classifier (ensemble method) that will use SVM and decision tree as weak learners. Logistic regression will be the final estimator. In short definition, ensemble methods are techniques that **create multiple models and then combine them to produce improved results**. Ensemble methods usually produce more accurate solutions than a single model would.
```python
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression

clf = StackingClassifier(
    estimators=[('svm', SVC(random_state=42)),
                ('tree', DecisionTreeClassifier(random_state=42))],
    final_estimator=LogisticRegression(random_state=42),
    n_jobs=-1)
```

### Tuning Model <a name="tuning-model"></a>
Tuning is the process of maximizing a model's performance without overfitting or creating too high of a variance. In machine learning, this is accomplished by selecting appropriate "hyperparameters". You can define your own tuning method what ever you want. But here is mine.
```python
from sklearn.model_selection import GridSearchCV

param_grid = {
    'svm__C': [1.6, 1.7, 1.8],
    'svm__kernel': ['rbf'],
    'tree__criterion': ['entropy'],
    'tree__max_depth': [9, 10, 11],
    'final_estimator__C': [1.3, 1.4, 1.5]
}

grid = GridSearchCV(
    estimator=clf,
    param_grid=param_grid,
    scoring='accuracy',
    n_jobs=-1)

grid.fit(X_train, y_train)

print('Best parameters: %s' % grid.best_params_)
print('Accuracy: %.2f' % grid.best_score_)
```

Based on the tuning process, the best hyperparameters are:
```text
Best parameters: {'final_estimator__C': 1.3, 'svm__C': 1.6, 'svm__kernel': 'rbf', 'tree__criterion': 'entropy', 'tree__max_depth': 11}
Accuracy: 0.98
```

### Create Final Model <a name="create-final-model"></a>
Finally, I can create a final model with the best hyperparameters. I hope this model will not overfit.
```python
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

final_clf = StackingClassifier(
    estimators=[('svm', SVC(C=1.6, kernel='rbf', random_state=42)),
                ('tree', DecisionTreeClassifier(criterion='entropy', max_depth=11, random_state=42))],
    final_estimator=LogisticRegression(C=1.3, random_state=42),
    n_jobs=-1)

final_clf.fit(X_train, y_train)
y_pred = final_clf.predict(X_test)

print('Accuracy score : ', accuracy_score(y_test, y_pred))
print('Precision score : ', precision_score(y_test, y_pred, average='weighted'))
print('Recall score : ', recall_score(y_test, y_pred, average='weighted'))
print('F1 score : ', f1_score(y_test, y_pred, average='weighted'))
```

Then I test the model with test data based on accuracy, precision, recall, and f1 score. The result are:
```text
Accuracy score :  0.9721518987341772
Precision score :  0.9719379890530496
Recall score :  0.9721518987341772
F1 score :  0.9717932606523529
```

**Looks pretty good!** Check out this confusion matrix. If it's biased, please comment 😁.
![Confusion Matrix](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2i5yse3mzfvv4f58cnto.png)

### Deploy Real App <a name="deploy-real-app"></a>
This step is not required. But if you are interested, you must export the model first. Only the stacking classifier model, which was trained before. So you can load again in another program.
```python
import pickle

pkl_filename = 'face_mask_detection.pkl'
with open(pkl_filename, 'wb') as file:
  pickle.dump(final_clf, file)
```

This process might be simple, but first you need to check out this diagram below.
![Flowchart Program](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/re2q7k8xan2n05xtgcpr.png)

Important thing to remember is you need to implement your own face detection model and crop it. For my example of program, check out my [Github Repository](https://github.com/myxzlpltk/face-mask-detection).