A short brief about Classification
Tags: python, machinelearning, classification, datascience
Published: 2019-08-13 14:06:40

---
title: A short brief about Classification
published: true
description: 
tags: python,machinelearning,classification,datascience
---

![image from http://blog.ss8.com/optimizing-breach-classification-detection-machine-learning/](https://thepracticaldev.s3.amazonaws.com/i/7brl707yigrhno91vc6d.jpg)

# What is Classification?
This is the first question I did when I heard the term Classification. The definition says it is, fundamentally, a model to predict **labels**. It brought me a new question. What are **labels**? Well, in a **dataset** for a classification models we will find **features** and **labels**, where a feature is a column used as an input data and the label is the value we want to predict.
So, when we know what value we want to predict with a Machine Learning model we have a Classification Problem.

# Testing Classification Models
Let's get a NBA Log dataset. The goal is to predict if a player will last longer than 5 years in league. This data contains a target column, the TARGET_5Yrs column can be 0 (< 5 years) or 1 (>= 5 years). As we know our target (label), we can say for sure this is a Classification problem.

This dataset can be found [here](https://data.world/exercises/logistic-regression-exercise-1).

## Requirements
Here are the libraries we will use in this example.

```python
import os

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from pandas.plotting import scatter_matrix
from sklearn import model_selection
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC

# define the dataset path
DATASET_PATH = os.path.join("datasets")
SEED = 7
```

## Loading the dataset
The first thing to do is load the dataset, let's use Pandas to do it and check how data is available.

```python
# create a function to load the dataset
def load_nba_data(dataset_path=DATASET_PATH):
    csv_path = os.path.join(dataset_path, "nba_logreg.csv")
    return pd.read_csv(csv_path)

# load the dataset
nba_data = load_nba_data()
# replace NaN fields with 0
nba_data.fillna(0, inplace=True)
# show 10 first rows
nba_data.head(10)
```

<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Name</th>
      <th>GP</th>
      <th>MIN</th>
      <th>PTS</th>
      <th>FGM</th>
      <th>FGA</th>
      <th>FG%</th>
      <th>3P Made</th>
      <th>3PA</th>
      <th>3P%</th>
      <th>...</th>
      <th>FTA</th>
      <th>FT%</th>
      <th>OREB</th>
      <th>DREB</th>
      <th>REB</th>
      <th>AST</th>
      <th>STL</th>
      <th>BLK</th>
      <th>TOV</th>
      <th>TARGET_5Yrs</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Brandon Ingram</td>
      <td>36</td>
      <td>27.4</td>
      <td>7.4</td>
      <td>2.6</td>
      <td>7.6</td>
      <td>34.7</td>
      <td>0.5</td>
      <td>2.1</td>
      <td>25.0</td>
      <td>...</td>
      <td>2.3</td>
      <td>69.9</td>
      <td>0.7</td>
      <td>3.4</td>
      <td>4.1</td>
      <td>1.9</td>
      <td>0.4</td>
      <td>0.4</td>
      <td>1.3</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>Andrew Harrison</td>
      <td>35</td>
      <td>26.9</td>
      <td>7.2</td>
      <td>2.0</td>
      <td>6.7</td>
      <td>29.6</td>
      <td>0.7</td>
      <td>2.8</td>
      <td>23.5</td>
      <td>...</td>
      <td>3.4</td>
      <td>76.5</td>
      <td>0.5</td>
      <td>2.0</td>
      <td>2.4</td>
      <td>3.7</td>
      <td>1.1</td>
      <td>0.5</td>
      <td>1.6</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>2</th>
      <td>JaKarr Sampson</td>
      <td>74</td>
      <td>15.3</td>
      <td>5.2</td>
      <td>2.0</td>
      <td>4.7</td>
      <td>42.2</td>
      <td>0.4</td>
      <td>1.7</td>
      <td>24.4</td>
      <td>...</td>
      <td>1.3</td>
      <td>67.0</td>
      <td>0.5</td>
      <td>1.7</td>
      <td>2.2</td>
      <td>1.0</td>
      <td>0.5</td>
      <td>0.3</td>
      <td>1.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Malik Sealy</td>
      <td>58</td>
      <td>11.6</td>
      <td>5.7</td>
      <td>2.3</td>
      <td>5.5</td>
      <td>42.6</td>
      <td>0.1</td>
      <td>0.5</td>
      <td>22.6</td>
      <td>...</td>
      <td>1.3</td>
      <td>68.9</td>
      <td>1.0</td>
      <td>0.9</td>
      <td>1.9</td>
      <td>0.8</td>
      <td>0.6</td>
      <td>0.1</td>
      <td>1.0</td>
      <td>1.0</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Matt Geiger</td>
      <td>48</td>
      <td>11.5</td>
      <td>4.5</td>
      <td>1.6</td>
      <td>3.0</td>
      <td>52.4</td>
      <td>0.0</td>
      <td>0.1</td>
      <td>0.0</td>
      <td>...</td>
      <td>1.9</td>
      <td>67.4</td>
      <td>1.0</td>
      <td>1.5</td>
      <td>2.5</td>
      <td>0.3</td>
      <td>0.3</td>
      <td>0.4</td>
      <td>0.8</td>
      <td>1.0</td>
    </tr>
    <tr>
      <th>5</th>
      <td>Tony Bennett</td>
      <td>75</td>
      <td>11.4</td>
      <td>3.7</td>
      <td>1.5</td>
      <td>3.5</td>
      <td>42.3</td>
      <td>0.3</td>
      <td>1.1</td>
      <td>32.5</td>
      <td>...</td>
      <td>0.5</td>
      <td>73.2</td>
      <td>0.2</td>
      <td>0.7</td>
      <td>0.8</td>
      <td>1.8</td>
      <td>0.4</td>
      <td>0.0</td>
      <td>0.7</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>6</th>
      <td>Don MacLean</td>
      <td>62</td>
      <td>10.9</td>
      <td>6.6</td>
      <td>2.5</td>
      <td>5.8</td>
      <td>43.5</td>
      <td>0.0</td>
      <td>0.1</td>
      <td>50.0</td>
      <td>...</td>
      <td>1.8</td>
      <td>81.1</td>
      <td>0.5</td>
      <td>1.4</td>
      <td>2.0</td>
      <td>0.6</td>
      <td>0.2</td>
      <td>0.1</td>
      <td>0.7</td>
      <td>1.0</td>
    </tr>
    <tr>
      <th>7</th>
      <td>Tracy Murray</td>
      <td>48</td>
      <td>10.3</td>
      <td>5.7</td>
      <td>2.3</td>
      <td>5.4</td>
      <td>41.5</td>
      <td>0.4</td>
      <td>1.5</td>
      <td>30.0</td>
      <td>...</td>
      <td>0.8</td>
      <td>87.5</td>
      <td>0.8</td>
      <td>0.9</td>
      <td>1.7</td>
      <td>0.2</td>
      <td>0.2</td>
      <td>0.1</td>
      <td>0.7</td>
      <td>1.0</td>
    </tr>
    <tr>
      <th>8</th>
      <td>Duane Cooper</td>
      <td>65</td>
      <td>9.9</td>
      <td>2.4</td>
      <td>1.0</td>
      <td>2.4</td>
      <td>39.2</td>
      <td>0.1</td>
      <td>0.5</td>
      <td>23.3</td>
      <td>...</td>
      <td>0.5</td>
      <td>71.4</td>
      <td>0.2</td>
      <td>0.6</td>
      <td>0.8</td>
      <td>2.3</td>
      <td>0.3</td>
      <td>0.0</td>
      <td>1.1</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>9</th>
      <td>Dave Johnson</td>
      <td>42</td>
      <td>8.5</td>
      <td>3.7</td>
      <td>1.4</td>
      <td>3.5</td>
      <td>38.3</td>
      <td>0.1</td>
      <td>0.3</td>
      <td>21.4</td>
      <td>...</td>
      <td>1.4</td>
      <td>67.8</td>
      <td>0.4</td>
      <td>0.7</td>
      <td>1.1</td>
      <td>0.3</td>
      <td>0.2</td>
      <td>0.0</td>
      <td>0.7</td>
      <td>0.0</td>
    </tr>
  </tbody>
</table>
<p>10 rows × 21 columns</p>
</div>

Here we have a small sample of our dataset. Let's discard the Name and the TARGET_5Yrs columns, all the others are the features of every player, these will tell us if the player will last longer than 5 years in a league or not. The TARGET_5Yrs has the answer for every combination of features.

Let's check a quick description of our dataset with the info() function.

```python
# first let's remove the uneeded Name column, 'cause it's not relevant for this experiment
nba_data = nba_data.drop('Name', 1)
nba_data.info()
```

```python
# first let's remove the uneeded Name column, 'cause it's not relevant for this experiment
nba_data = nba_data.drop('Name', 1)
nba_data.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 1340 entries, 0 to 1339
    Data columns (total 20 columns):
    GP             1340 non-null int64
    MIN            1340 non-null float64
    PTS            1340 non-null float64
    FGM            1340 non-null float64
    FGA            1340 non-null float64
    FG%            1340 non-null float64
    3P Made        1340 non-null float64
    3PA            1340 non-null float64
    3P%            1340 non-null float64
    FTM            1340 non-null float64
    FTA            1340 non-null float64
    FT%            1340 non-null float64
    OREB           1340 non-null float64
    DREB           1340 non-null float64
    REB            1340 non-null float64
    AST            1340 non-null float64
    STL            1340 non-null float64
    BLK            1340 non-null float64
    TOV            1340 non-null float64
    TARGET_5Yrs    1340 non-null float64
    dtypes: float64(19), int64(1)
    memory usage: 209.5 KB

We can also check some statistics information on our dataset with the `describe` function.

```python
nba_data.describe()
```

<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>GP</th>
      <th>MIN</th>
      <th>PTS</th>
      <th>FGM</th>
      <th>FGA</th>
      <th>FG%</th>
      <th>3P Made</th>
      <th>3PA</th>
      <th>3P%</th>
      <th>FTM</th>
      <th>FTA</th>
      <th>FT%</th>
      <th>OREB</th>
      <th>DREB</th>
      <th>REB</th>
      <th>AST</th>
      <th>STL</th>
      <th>BLK</th>
      <th>TOV</th>
      <th>TARGET_5Yrs</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>count</th>
      <td>1340.000000</td>
      <td>1340.000000</td>
      <td>1340.000000</td>
      <td>1340.000000</td>
      <td>1340.000000</td>
      <td>1340.000000</td>
      <td>1340.000000</td>
      <td>1340.000000</td>
      <td>1340.000000</td>
      <td>1340.000000</td>
      <td>1340.000000</td>
      <td>1340.000000</td>
      <td>1340.000000</td>
      <td>1340.000000</td>
      <td>1340.000000</td>
      <td>1340.000000</td>
      <td>1340.000000</td>
      <td>1340.000000</td>
      <td>1340.000000</td>
      <td>1340.000000</td>
    </tr>
    <tr>
      <th>mean</th>
      <td>60.414179</td>
      <td>17.624627</td>
      <td>6.801493</td>
      <td>2.629104</td>
      <td>5.885299</td>
      <td>44.169403</td>
      <td>0.247612</td>
      <td>0.779179</td>
      <td>19.149627</td>
      <td>1.297687</td>
      <td>1.821940</td>
      <td>70.300299</td>
      <td>1.009403</td>
      <td>2.025746</td>
      <td>3.034478</td>
      <td>1.550522</td>
      <td>0.618507</td>
      <td>0.368582</td>
      <td>1.193582</td>
      <td>0.620149</td>
    </tr>
    <tr>
      <th>std</th>
      <td>17.433992</td>
      <td>8.307964</td>
      <td>4.357545</td>
      <td>1.683555</td>
      <td>3.593488</td>
      <td>6.137679</td>
      <td>0.383688</td>
      <td>1.061847</td>
      <td>16.051861</td>
      <td>0.987246</td>
      <td>1.322984</td>
      <td>10.578479</td>
      <td>0.777119</td>
      <td>1.360008</td>
      <td>2.057774</td>
      <td>1.471169</td>
      <td>0.409759</td>
      <td>0.429049</td>
      <td>0.722541</td>
      <td>0.485531</td>
    </tr>
    <tr>
      <th>min</th>
      <td>11.000000</td>
      <td>3.100000</td>
      <td>0.700000</td>
      <td>0.300000</td>
      <td>0.800000</td>
      <td>23.800000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.200000</td>
      <td>0.300000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.100000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>25%</th>
      <td>47.000000</td>
      <td>10.875000</td>
      <td>3.700000</td>
      <td>1.400000</td>
      <td>3.300000</td>
      <td>40.200000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.000000</td>
      <td>0.600000</td>
      <td>0.900000</td>
      <td>64.700000</td>
      <td>0.400000</td>
      <td>1.000000</td>
      <td>1.500000</td>
      <td>0.600000</td>
      <td>0.300000</td>
      <td>0.100000</td>
      <td>0.700000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>50%</th>
      <td>63.000000</td>
      <td>16.100000</td>
      <td>5.550000</td>
      <td>2.100000</td>
      <td>4.800000</td>
      <td>44.100000</td>
      <td>0.100000</td>
      <td>0.300000</td>
      <td>22.200000</td>
      <td>1.000000</td>
      <td>1.500000</td>
      <td>71.250000</td>
      <td>0.800000</td>
      <td>1.700000</td>
      <td>2.500000</td>
      <td>1.100000</td>
      <td>0.500000</td>
      <td>0.200000</td>
      <td>1.000000</td>
      <td>1.000000</td>
    </tr>
    <tr>
      <th>75%</th>
      <td>77.000000</td>
      <td>22.900000</td>
      <td>8.800000</td>
      <td>3.400000</td>
      <td>7.500000</td>
      <td>47.900000</td>
      <td>0.400000</td>
      <td>1.200000</td>
      <td>32.500000</td>
      <td>1.600000</td>
      <td>2.300000</td>
      <td>77.600000</td>
      <td>1.400000</td>
      <td>2.600000</td>
      <td>4.000000</td>
      <td>2.000000</td>
      <td>0.800000</td>
      <td>0.500000</td>
      <td>1.500000</td>
      <td>1.000000</td>
    </tr>
    <tr>
      <th>max</th>
      <td>82.000000</td>
      <td>40.900000</td>
      <td>28.200000</td>
      <td>10.200000</td>
      <td>19.800000</td>
      <td>73.700000</td>
      <td>2.300000</td>
      <td>6.500000</td>
      <td>100.000000</td>
      <td>7.700000</td>
      <td>10.200000</td>
      <td>100.000000</td>
      <td>5.300000</td>
      <td>9.600000</td>
      <td>13.900000</td>
      <td>10.600000</td>
      <td>2.500000</td>
      <td>3.900000</td>
      <td>4.400000</td>
      <td>1.000000</td>
    </tr>
  </tbody>
</table>
</div>

Let's take a look at our target is distributed over the dataset.

```python
nba_data.groupby('TARGET_5Yrs').size()
```


    TARGET_5Yrs
    0.0    509
    1.0    831
    dtype: int64


In resume, we have 1340 objects in our dataset where 509 will not last longer than 5 years in league and the others 831 will.

## Machine Learning Models Evaluation

As we saw in the beginning of this post, this is a **Classification Problem**. We will create some models with different ML algorithms and check their accuracy.

### Spliting Data
Let's split our dataset into two new datasets. We will use 80% of the dataset to train our classification models and 20% of it to perform the validation.

```python
data = nba_data.values
# data = np.array(data)

# now let's separate the features columns from the target column
X = data[:, 0:19]
Y = data[:, 19]

# as said before,  we will use 20% of the dataset for validation
validation_size = 0.20

# split the data into traning and testing
X_train, X_test, Y_train, Y_test = model_selection.train_test_split(X, Y, test_size=validation_size, random_state=SEED)
```

Now that we have our training and testing set, we are going to create an array with the models we want to evaluate. We will use each model with the default settings.

```python
models = []
models.append(('LR', LogisticRegression(solver='liblinear', multi_class='ovr')))
models.append(('LDA', LinearDiscriminantAnalysis()))
models.append(('KNN', KNeighborsClassifier()))
models.append(('CART', DecisionTreeClassifier()))
models.append(('NB', GaussianNB()))
models.append(('SVM', SVC(gamma='auto')))
```

To evaluate the models we will user K-Fold cross-validation and measure the accuracy for each model. This techinique randomly splits the training set into K distincts subsets (*folds*), then it trains and evaluates the model K times picking a different fold for every evaliation. The result will be an array with the K evaluation scores. For this example we will a Cross-Validation using the **StratifiedKFold** from **SKLearn**. We will use the Mean of the accuracies of each model to determinate which one has the best results.

```python
scoring = 'accuracy'
models_results = []
for name, model in models:
    results = []
    skfolds = model_selection.StratifiedKFold(n_splits=10, random_state=SEED)
    for train_index, test_index in skfolds.split(X_train, Y_train):
        X_train_folds = X_train[train_index]
        Y_train_folds = (Y_train[train_index])
        X_test_folds = X_train[test_index]
        Y_test_folds = (Y_train[test_index])
        
        model.fit(X_train_folds, Y_train_folds)
        pred = model.predict(X_test_folds)
        correct = sum(pred == Y_test_folds)
        results.append(correct / len(pred))
    models_results.append((name, results))
    

names = []
scores = []
# the snippet bellow calculates the mean of the accuracies
for name, results in models_results:
    mean = np.array(results).mean()
    std = np.array(results).std()
    print("Model: %s, Accuracy Mean: %f (%f)" % (name, mean, std))
    names.append(name)
    scores.append(results)
```

    Model: LR, Accuracy Mean: 0.705244 (0.026186)
    Model: LDA, Accuracy Mean: 0.706205 (0.027503)
    Model: KNN, Accuracy Mean: 0.674429 (0.026029)
    Model: CART, Accuracy Mean: 0.634372 (0.047236)
    Model: NB, Accuracy Mean: 0.632433 (0.040794)
    Model: SVM, Accuracy Mean: 0.619384 (0.021099)

The results above show us that the Linear Discriminant Analysis has the best accuracy score among the models we tested. The boxplot below shows the accuracy scores spread accross each fold.

```python
fig = plt.figure()
fig.suptitle('Models Comparison')
ax = fig.add_subplot(111)
plt.boxplot(scores)
ax.set_xticklabels(names)
plt.show()
```

![Plot01](https://thepracticaldev.s3.amazonaws.com/i/k4qoc4pc0zencsw0ww02.png)

## Making Predictions

Now we will check the accuracy of the LDA model by making some predictions with the validation set we've prepared before. To do so, we will create an instance of the model and use the method `predict`.

```python
model = LinearDiscriminantAnalysis()
model.fit(X_train, Y_train)
predictions = model.predict(X_test)
print("Accuracy: {}".format(accuracy_score(Y_test, predictions)))
```

    Accuracy: 0.6902985074626866

We can also check the Confusion Matrix for this model

```python
print(confusion_matrix(Y_test, predictions))
```

    [[ 52  45]
     [ 38 133]]

Each row in a confusion matrix represents an **actual target** and each column represents a **predicted target**. The first row of this matrix contains the **true negatives** and the **false positives**. Which means that 52 samples were correctly classified and 45 were wrongly classified. The second row shows us the **false negatives** and the **true positives**, wich means that 38 samples were wrongly classified and 133 were classified correctly.

The confusion matrix provides a lot of information, but if you want to get a more concise metrics you can use the `classification_report` function of Scikit-Learn. It will provide the **precision**, **recall** and **f1-score** metrics.

```python
print(classification_report(Y_test, predictions))
```

                  precision    recall  f1-score   support
    
             0.0       0.58      0.54      0.56        97
             1.0       0.75      0.78      0.76       171
    
        accuracy                           0.69       268
       macro avg       0.66      0.66      0.66       268
    weighted avg       0.69      0.69      0.69       268

The accuracy of the positive predictions is called **precision**. It's defined by the formula: TP/(TP + FP), where TP is the number of **True Positives** and FP is the number of **False Positives**. This metric is tipically used along the **recall** which is the true positive rate - the ratio of positive instances that are correctly detected by the model. It's equation is: TP / (TP + FN) where FN is the **False Negatives**.

## Conclusion
This is a short brief about Classification with Python and Scikit-Learning. There is a lot more to cover, we can improve our models results by normalizing the data for example. There's also others metrics to cover. But the firts steps into Machine Learning world can be done with this tutorial. Hope you enjoy it!!

You can access the notebook for this example [here](https://github.com/thnery/machine-learning/blob/develop/classification/classification_001.ipynb).

## References
* Your First Machine Learning Project in Python Step-By-Step - https://machinelearningmastery.com/machine-learning-in-python-step-by-step/

* Géron, A. (2017). Hands-on machine learning with Scikit-Learn and TensorFlow: Concepts, tools, and techniques to build intelligent systems.