High on accuracy but low on ROC score?
Tags: machinelearning, datascience, analytics
Published: 2021-03-20 07:08:48

**Why does the model yield to high accuracy but low ROC score?**

Sometimes, there is an unequal distribution of classes in the dataset in such a manner that the rare class constitutes to very small amount of data. And usually, the classification solution is expected to predict the rare class. Due to the imbalance in class, most commonly used classification algorithms do not perform very well, scoring low on precision and recall. The algorithms are also tilted towards predicting the majority class since their loss functions try to perform well in calculating error rate without taking into account the distribution of the feature.

**How to handle an imbalanced dataset?**

With an imbalanced dataset in hand, the developer is haunted with questions such as; how should the score of the precision / recall be improved? Several years of research on this as aspect has brought forth a few methods in handling such a dataset. Which is the best classification algorithm that can be fit to an imbalanced dataset? There is no concrete answer to this. What are the factors that affect the performance of an imbalanced dataset? Lack of correlation between the features and low variance play the key factors in low performance. So how should the dataset be handled?

**Sampling Technique**

Whenever there is an imbalance in data, the line of understanding is that the data is not a true representation of the population / reality. If this is so then, the next step towards resolving this issue is to collect more representative data. It’s not always possible to collect more real data. Sampling techniques can be applied to such a dataset to bridge the disproportion in classes. Let’s us see the various techniques, their merits & demerits and when they should be used. Sampling refers to a technique designed to balance a class distribution that is skewed.

**Undersampling, Oversampling, SMOTE, Bagging With Random Undersampling**

*Undersampling* is technique in which the data belonging to the majority class is removed from the training data, reducing the skew and balancing the distribution to 1:1 or 1:2. Although its an effective method, the data points are randomly deleted without any concern of how important they might be in decision making. This technique has been improvised by using heuristic learning models in undersampling which are able to identify redundant data points for deletion or do not mark useful data to be removed.

*Oversampling* refers to duplicating data points from the minority class and adding them to the dataset, replacing the original dataset. This technique can be effective when the model is influenced by skewed distribution due to duplicate data points of the class. It can be greatly effective in the training dataset but may perform very poorly with unknown dataset. It can lead to high chance of overfitting due to the redundant copies in the minority class. This in turn can increase the computational cost while fitting the data and can result in poor performance on the test dataset.

*SMOTE* is another sampling technique which uses the oversampling duplicate technique by synthesizing new samples of data, also known as Synthetic Minority Oversampling Technique.
SMOTE works by drawing newer samples close to the feature space. SMOTE first selects random data point a from the minority class. Then it finds its k-nearest neighbour b. Then a synthetic data instance is created using the convex combination of the chosen data point and its k-nearest neighbour. The approach is effective since new synthetic data points are created which are close to selected one in the feature space. The SMOTE class acts like a data transform object from scikit-learn library. It must be defined and imported, fit on a dataset, then applied to create a new transformed version of the dataset.

For the complete article please read
[https://ranjani-rajamani.medium.com/high-on-accuracy-but-low-on-roc-score-a40f2053b6c4]