Understanding the Confusion Matrix
Tags: machinelearning, python, deeplearning
Published: 2019-04-22 21:02:35

---
title: Understanding the Confusion Matrix
published: true
description: A short explanation of the confusion matrix and how it can be used to evaluate the performance of a machine learning classification model
tags: machine learning, ML, Python, deep learning,
---

![visual](https://thepracticaldev.s3.amazonaws.com/i/5s8xw56ad21s6skkmjuh.png)

A confusion matrix is a table that describes the performance of a classifier/classification model. It contains information about the **actual and prediction classifications** done by the classifier and this information is used to evaluate the performance of the classifier.

Note that the confusion matrix is only used for classification tasks, and as such cannot be used in regression models or other non-classification models.

Before we go on, let's look at some terms.

- Classifier: A classifier is basically an algorithm that uses "knowledge" gotten from training data to map input data to a particular category or class. Classifiers are either binary classifiers or multi-class/multi-categorical/multi-label/multi-output classifiers.
- Training and test data: When building a classification model/classifier, datasets are split into **training data** and **test data** which have associated labels. A label is an expected output which is the category or class data belongs to.
- Actual classifications: This is the expected output (labels) on the data.
- Prediction classifications: This is the output given by the classifier for a particular input data.

**An example**: Let's say we have built a classifier to categorize an input image of a car as either a sedan or not, and we have an image in our dataset that has been labeled as a non-sedan but the classification model classifies as a sedan.
In this scenario, the actual classification is **non-sedan** while the prediction classification is **sedan**.

### Types of Confusion Matrices

There are two types of confusion matrices:

- 2-class confusion matrix
- Multi-class confusion matrix

### 2-Class Confusion Matrix

A 2-class as the name implies is a confusion matrix that describes the performance of a binary classification model. A 2-class matrix for the **sedan** classifier I described earlier can be visualized as such:

 

![visual](https://thepracticaldev.s3.amazonaws.com/i/x5jengc1mjseexyl0r1b.png)

In this visualization, we have two sections which have been outlined. We have the **predicted** classifications section which contains two subsections for each of the classes and the **actual** classifications section which has two subsections for each of the classes.

If this is your first time seeing a confusion matrix, I know you must be wondering what all the variables in the table represent. It is quite simple actually, I will explain as simply as I can, but before I do it is important to know that these variables represent a number of predictions.

##### The variable a

The variable **a** falls under the **Non-sedan** sub-section in both the **Actual** and **Predicted** classification sections. This means **a** predictions were made that correctly classified an image of a non-sedan [as a non-sedan].

##### The variable b

The variable **b** falls under the **Non-sedan** sub-section in the **Actual** classification section and under the **Sedan** sub-section in the **Predicted** classification section. This means **b** predictions were made that incorrectly classified an image of a non-sedan as a sedan.

##### The variable c

The variable **c** falls under the **Sedan** sub-section in the **Actual** classification section and under the **Non-sedan** sub-section in the **Predicted** classification section. This means **c** predictions were made that incorrectly classified an image of a sedan as a non-sedan.

##### The variable d

The variable **d** falls under the **Sedan** sub-section in both the **Actual** and **Predicted** classification sections. This means **d** predictions were made that correctly classified an image of a sedan [as a sedan].

Easy peasy lemon squeezy. (I hope? 😅)



##### But wait, we're not done yet.......

Now we have our confusion matrix for our **sedan** classifier, but how does this help us ascertain our classifier's performance/efficiency?
To ascertain the performance of a classifier using the confusion matrix and the data it contains, there are some standard metrics that we can calculate [for] using the data(variables) in the confusion matrix.

##### Accuracy

Accuracy in a 2-Class confusion matrix is the ratio of the total number of correct predictions to the total number of predictions. 
From our confusion matrix, we can see that **a** and **d** predictions were made that correctly classified the input image and **b** and **c** predictions were made that incorrectly classified the input image.

Therefore, accuracy can be calculated as: 

​	**Accuracy** = (**a + d**) / (**a + b + c + d**)

​	Where, **a + d** is the total number of correct predictions and **a + b + c + d** is the total number of predictions made.

##### True positives, True negatives, False positives and False negative

With relation to our classifier and confusion matrix:

**True positives (TP)** are the number of predictions where an image of a sedan is correctly classified [as a sedan]. 
From our confusion matrix, the variable **d** is also the **TP**.

**True negatives (TN)** are the number of predictions where an image of a non-sedan is correctly classified [as a non-sedan]. 
From our confusion matrix, the variable **a** is also our **TN**.

**False positives (FP)** are the number of predictions where an image of a non-sedan is incorrectly classified as a sedan. 
From our confusion matrix, the variable **b** is also our **FP**.

**False negatives (FN)** are the number of predictions where an image of a sedan is incorrectly classified as a non-sedan. 
From our confusion matrix, the variable **c** is also our **FN**.

##### True Positive Rate

The true positive rate is a ratio of the **true positives** to the sum of the **true positives** and **false negatives**. It shows how often the classifier classifies an image of a sedan as a sedan.

Therefore, the true positive rate can be calculated as:

​	**True Positive Rate** = **d** / (**c + d**)
​	Where **d** is **TP** and **c** is **FN**

True positive rate is also known as **recall** or **sensitivity**

##### False Positive Rate

The false positive rate is a ratio of the **false positives** to the sum of the **true negatives** and **false positives**. It shows how often the classifier classifies an image of a non-sedan as a sedan.

Therefore, the false positive rate can be calculated as:

​	**False Positive Rate** = **b** / (**a + b**)
​	Where **a** is **TN** and b is **FP**

##### True Negative Rate

The true negative rate is a ratio of the **true negatives** to the sum of the **true negatives** and **false positives**. It shows how often the classifier classifies an image of a non-sedan as a non-sedan.

Therefore, the false positive rate can be calculated as:

​	**True Negative Rate** = **a** / (**a + b**)
​	Where **a** is **TN** and b is **FP**

The true negative rate is also known as **specificity**.

##### False Negative Rate

The false negative rate is a ratio of the **false negatives** to the sum of the **false negatives** and **true positives**. It shows how often the classifier classifies an image of a sedan as a non-sedan.

Therefore, the false positive rate can be calculated as:

​	**False Negative Rate** = **c** / (**c + d**)
​	Where **d** is **TP** and **c** is **FN**

##### Precision

The precision is a ratio of the **true positives** to the sum of the **true positives** and **false positives**. It shows how often the classifier classifies an input image as a sedan and it turns out to be correct.

It is calculated as:

​	**Precision** = **d** / **(b + d)**
​	Where **d** is **TP** and **b** is **FP**

### An Example

Suppose we have the image below as the confusion matrix for our classifier, we can use the metrics defined above to evaluate its performance.

 ![sample](https://thepracticaldev.s3.amazonaws.com/i/29erno9l9601wqb9zbqs.png)



From the confusion matrix, we can see that:

> - 4252 predictions were made that correctly classified a non-sedan [as a non-sedan].
>   Therefore, our variable **a** and the **True Negative (TN)** is 4252.
> - 875 predictions were made that incorrectly classified a non-sedan as a sedan.
>   Therefore, our variable **b** and the **False Positive (FP)** is 875.
> - 421 predictions were made that incorrectly classified a sedan as a non-sedan.
>   Therefore, our variable **c** and the **False Negative (FN)** is 421.
> - 4706 predictions were made that correctly classified a sedan [as a sedan].
>   Therefore, our variable **d** and the **True Positive (TP)** is 4706

​	Using the data we've "extracted", we can calculate the aforementioned metrics and ascertain the performance of the classifier. We can already tell that the classifier performs well since the number of correct predictions is greater than the number of incorrect predictions.

##### Accuracy

​	Accuracy =  **(a + d)** / **(a + b + c + d)**
​			= **(4252 + 4706)** / **(4252 + 875 + 421 + 4706)**
​			= **(8958)** / **(10254)**
​			= **0.8736102984201287**
​	Accuracy = **0.87**

​	Therefore the classifier has an accuracy of 0.87 which is 87%

##### True Positive Rate

​	TPR = **TP / (TP + FN)**
​		= **4706** / **(4706 + 421)**
​		= **4706** / **5127**
​		= **0.917885703140238**
​	TPR = **0.92**

​	Therefore the classifier has a True Positive Rate of 0.92 which is 92%

##### False Positive Rate

​	FPR = **FP / (FP + TN)**
​		= **875** /  **(875 + 4252)**
​		= **875** / **5127**
​		= **0.1706651062999805**
​	FPR = **0.17**

​	Therefore the classifier has a False Positive Rate of 0.17 which is 17%

##### True Negative Rate

​	TNR = **TN / (TN + FP)**
​		= **4252 / (4252 + 875)**
​		= **4252 / 5127**
​		= **0.8293348937000195**
​	TNR = **0.83**

​	Therefore the classifier has a True Negative Rate of 0.83 which is 83%

##### False Negative Rate

​	FNR = **FN / (FN + TP)**
​		= **421 / (421 + 4706)**
​		= **421 / 5127**
​		= **0.082114296859762**
​	FNR = **0.08**

​	Therefore the classifier has a False Negative Rate of 0.08 which is 8%

##### Precision

​	Precision = **TP / (TP + FP)**
​			 = **4706 / (4706 + 875)**
​			 = **4706 / 5581**
​			 = **0.8293348937000195**
​	Precision = **0.83**

​	Therefore the classifier has a Precision of 0.83 which is 83%

### How to generate a confusion matrix using Python

```python
import matplotlib.pylab as plt
import itertools
import numpy as np
from sklearn.metrics import confusion_matrix

def plot_confusion_matrix(cm, classes,normalize=False):
    plt.figure(figsize = (5,5))
    plt.imshow(cm, interpolation='nearest', cmap=plt.cm.Blues)
    plt.title('Confusion matrix')
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=90)
    plt.yticks(tick_marks, classes)
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]

    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, cm[i, j],
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")
    plt.tight_layout()
    plt.ylabel('Actual')
    plt.xlabel('Predicted')
    
dict_characters = {0: 'Non-sedan', 1: 'Sedan'}

y_pred = model.predict(test_data)
y_pred_classes = np.argmax(y_pred, axis=1)
y_true = np.argmax(test_labels, axis=1)

confusion_mat = confusion_matrix(y_true, y_pred_classes)
plot_confusion_matrix(confusion_mat, classes = list(dict_characters.values()))
```

To generate a confusion matrix, we utilize numpy, matplotlib.pylab to visualize the matrix, the confusion_matrix function from the sklearn.metrics package to generate the confusion matrix, and itertools for looping/iteration.

First, we define a function **plot_confusion_matrix** that takes the generated confusion matrix and expected/possible classes as arguments and the uses matplotlib.pylab to visualize the confusion matrix.

In the snippet, we assume we have already have our trained model and training and test data with associated labels. 

**dict_characters** is a dictionary of the two possible classes, in our case, "**non-sedan**" and "**sedan**".
**y_pred** is a numpy array of predictions done by the classifier on the test data
**model** is our trained classifier/algorithm
**test_data** is our test data
**y_pred_classes** is a numpy array of indices relative to **y_pred** which is the array of predictions done by the classifier on the test data.
**y_true** is a numpy array of indices relative to the actual/correct labels of the test_data.
**test_labels** is a list of labels of the test data.

Using the above, we use the **confusion_matrix** function from **sklearn.metrics** to generate the confusion matrix, passing in the correct values (**y_true**) and the estimated values returned by the classifier (**y_pred_classes**) and we store the generated confusion matrix in a variable **confusion_mat**.

We then pass the confusion matrix (**confusion_mat**) and a list of the values of our possible classes (**dict_characters**) as arguments to the **plot_confusion_matrix** function which then visualizes the confusion matrix.


In my next post, I [hopefully] would be writing on the multi-class confusion matrix.