saliencytools 0.34
A collection of metrics for comparing saliency maps
Requires Python: >=3.10

# Saliency Metrics

![Tests](https://github.com/valevalerio/saliencytools/actions/workflows/test.yml/badge.svg)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![PyPI version](https://img.shields.io/pypi/v/saliencytools)](https://pypi.org/project/saliencytools/)
[![Documentation Status](https://img.shields.io/website?down_color=red&up_color=44cc11&url=https://valevalerio.github.io/saliencytools&label=Documentation)](https://valevalerio.github.io/saliencytools/)
<!-- [![Documentation Status](https://img.shields.io/website?url=https://valevalerio.github.io/saliencytools/)](https://vvalerio.github.io/saliencytools/) -->
### This module is a work in progress and is not yet complete.
This package is a modular package that implements various metrics for comparing saliency maps generated by explanation methods. To ensure fair comparisons, metrics should be computed on the same saliency map and corresponding ground truth map.

**Saliency Metrics** is a Python package that implements various metrics for comparing saliency maps generated by explanation methods. To ensure fair comparisons, metrics should be computed on the same saliency map and corresponding ground truth map.
The package includes the following metrics:
- **SSIM (Structural Similarity Index)**: A perceptual metric that quantifies the similarity between two images. It considers changes in structural information, luminance, and contrast.
- **PSNR (Peak Signal-to-Noise Ratio)**: A metric that measures the ratio between the maximum possible power of a signal and the power of corrupting noise. It is often used to assess the quality of reconstructed images.
- **EMD (Earth Mover's Distance)**: A metric that measures the distance between two probability distributions over a region D. It is often used in computer vision and image retrieval tasks.

```tutorial.ipynb``` is an original way used to check and test the different metrics. 
# Installation

```pip install saliencytools```

# Usage

```python
from saliencytools import ssim, psnr, emd # and more

import numpy as np
import matplotlib.pyplot as plt



# create a random saliency map
saliency_map = np.random.rand(28*28).reshape(28, 28)- 0.5
# create a random ground truth map
ground_truth_map = np.random.rand(28*28).reshape(28, 28)- 0.5
# create a random binary mask

# use all the metrics to compare the saliency map with the ground truth map
for metric in [ssim, psnr, emd]:
    
    print(f"{metric.__name__}: {metric(saliency_map, ground_truth_map)}")
```
# Validating the metrics
The ```tutorial.ipynb``` file is a Jupyter notebook that showcases the validation of the metrics. It includes examples of how to use the package, as well as visualizations of the results. 

The notebook implements a _KNN-like_ classification task using the MNIST dataset. It uses the ```saliencytools``` package to compute the distance between the test images and _k_ prototypes of each class. 

The _k_ prototypes (_k_=5 in the notebook) are choosen random, and the distance is computed using all the metrics implemented in the package.

Some preprocessing is explored to check different ways to compute the distance.
```C``` stands for Clipping the saliency maps in -1, 1.
```N``` stands for Normalization of the saliency maps in 0, 1.
```S``` stands for the application of the sobel filter to the saliency maps, so that the edges are more influencing.


![F1_scores](https://valevalerio.github.io/saliencytools/_static/heatmap.png)

<!-- add another image but this time visible
They are both visible. What changes is where they redirect to.
I had to put the image in the _static folder under the doc/source/_static folder.
By the use of the workflow of sphinx-gallery, the image is copied to the _static folder of the gh-pages branch.
The image is visible in the README.md file, not anywhere in the documentation.
![F1_scores](https://github.com/valevalerio/saliencytools/blob/gh-pages/_static/heatmap.png)

-->


From the above images we can see that the F1 score changes depending on the metric used and the preprocessing applied. **In this case** the best metric is the Sign Agreement ratio (both without preprocessing and by clipping the saliency maps in -1, 1).
The SSIM metric is the second best, without any preprocessing.
For instance, the Jaccard metric is not good without preprocessing.

## Metrics to add

from https://ar5iv.labs.arxiv.org/html/1604.03605

- [] Area under ROC Curve 	AUC 	
- [] Shuffled AUC 	sAUC 	
- [] Normalized Scanpath Saliency 	NSS 	
- [] Kullback-Leibler divergence 	KL 	[23, 49, 68, 88]
- [] Information Gain 	IG 	[45, 46] 

# Why Saliencytools?
This package is **needed** since other alternatives are not stable, not maintained or trustable. They are not specific for images, they don't exploit ```numpy``` or ```scipy```, and they are not modular. The documentation of some is clear, yet the the examples are not complete. 

Open Sources alternatives to this package are:

- distancia https://github.com/ym001/distancia
    - images have to be converted to lists
    - there are errors in the code
    - there is no homogenization of the metrics
    - documentation is nice (https://distancia.readthedocs.io/en/latest/)
- saliency-metrics https://github.com/sandylaker/saliency-metrics
    - project abandoned (last commit 2022) 
    - documentaiton incomplete https://saliency-metrics.readthedocs.io/en/latest/index.html

<!--
Other resoureces:
- https://pypi.org/project/saliency/


https://doi.org/10.1364/JOSAA.31.000532

-->

# Further reading
- Interpretable Machine Learning Book https://christophm.github.io/interpretable-ml-book/pixel-attribution.html