cfscores 0.0.12
Calculation of counterfactual point scores

# Counterfactual Scoring Metrics

This algorithm is intended to evaluate counterfactual points in relation to the study point (which the counterfactual was created) and also to the dataset.

# HOW TO USE

## Install
````
pip install cfscores
````
## Create Analyser
```
from cfscores import CFAnalysis
analysis = CFAnalysis(x, cf, label_name, y, cf_y, dataset, cat_feats)

# Where:
# x => Original point (only support one point)
# cf => Counterfactual point(s)
# label_name => Name the label
# y => Category of the original point
# cf_y => Category(ies) of the counterfactual(s) point(s)
# dataset => Dataset of points
# cat_feats => List of categorical features names (features are treated as continuous by default)
```

## Run Analysis
```
from cfscores import CFAnalysis
analysis = CFAnalysis(x, cf, label_name, y, cf_y, dataset, cat_feats)

# From the created analysis you can run the metrics calculation
result = analysis.generate_metrics(["validity", "dice_proximity", "langel_proximity", "sparsity",
                                    "diversity", "im1", "im2"])
# The result then, will be a dict object with all scores specified
```

## To calculate just validity and sparsity you will need:
````
#   * Pandas DataFrame/Series with ONE original point (x) => x
#   * Pandas DataFrame/Series with one/several counterfactual points => cf
#   * Label feature name => label_name
#   [IF FEATURE IS SEPARATED FROM POINTS]
#   * Pandas DataFrame/Series with y point => y
#   * Pandas DataFrame/Series with cf_y point(s) => cf_y

# Case 1 - Feature in the x and y
from cfscores import CFAnalysis
analysis = CFAnalysis(x, cf, label_name)
# Assign analysis results to variable results
results = analysis.generate_metrics()

# Case 2 - Feature separeted from x and y
from cfscores import CFAnalysis
analysis = CFAnalysis(x, cf, label_name, y, cf_y)
# Assign analysis results to variable results
results = analysis.generate_metrics()
````

## To calculate all available scores you will need:
````
#   * Pandas DataFrame/Series with ONE original point (x) => x
#   * Pandas DataFrame/Series with one/several counterfactual points => cf
#   * Label feature name => label_name
#   * Pandas DataFrame with the dataset => dataset
#   [IF HAS CATEGORICAL FEATURES]
#   * List with names (in string) of the categorical features => cat_feats
#   [IF FEATURE IS SEPARATED FROM POINTS]
#   * Pandas DataFrame/Series with y point => y
#   * Pandas DataFrame/Series with cf_y point(s) => cf_y

# Case 1 - Dataset but without categorical features
from cfscores import CFAnalysis
analysis = CFAnalysis(x, cf, label_name, y, cf_y, dataset)
# Assign analysis results to variable results
results = analysis.generate_metrics()

# Case 2 - Dataset with categorical features
from cfscores import CFAnalysis
analysis = CFAnalysis(x, cf, label_name, y, cf_y, dataset, cat_feats)
# Assign analysis results to variable results
results = analysis.generate_metrics()
````

## To calculate a specific score (e.g. validity):
```
# Generate a analysis object like before
from cfscores import CFAnalysis
analysis = CFAnalysis(x, cf, label_name, y, cf_y, dataset, cat_feats)
# Choose a metric by passing all required metrics as string in a list
results = analysis.generate_metrics(["validity"])
```


## Supported Metrics

The following metrics are calculated:

### Validity
#### Definition
<p align="justify">
This is the simplest metric and just compare if the counterfactual(s) are indeed classified as counterfactual (a class different from the evaluated class)
</p>

#### Formal definition
<p align="center">
<img src="https://render.githubusercontent.com/render/math?math=\LARGE I(f({\bf x}) \neq f({\bf c})))">
</p>

### Proximity
<p align="justify">
This is a metric to assess the distance from the original point and the counterfactual. There are several ways to calculate this metric, this algorithm offers the following implementations:
</p>

#### > DiCE implementation
#### Definition
<p align="justify">
This implementation follows DiCE article [1] and treats continuous and categorical variables differently. For continuous variables the Watcher et al. [2] calculation, using the median absolute deviation (<img src="https://render.githubusercontent.com/render/math?math=MAD">) is used dividing the l1 distance between each continuous feature (<img src="https://render.githubusercontent.com/render/math?math=|{\bf c}^p - {\bf x}^p|">). For categorical features a simple check is used and it's assigned an arbitrary value of 1 when a categorical variable does not match the original evaluated point.
</p>

#### Formal definition
<p align="center">
<img src="https://render.githubusercontent.com/render/math?math=\LARGE\text{Continuous-Proximity} = dist\_cont({\bf c}, {\bf x})">
</br>
<img src="https://render.githubusercontent.com/render/math?math=\LARGE \text{Categorical-Proximity} = dist\_cat({\bf c}, {\bf x})">
</br>
</br>
Where:
</br>
</br>
<img src="https://render.githubusercontent.com/render/math?math=\LARGE dist\_cont({\bf c}, {\bf x}) = \frac{1}{d_{cont}} \sum_{p=1}^{d_{cont}} \frac{|c^p-x^p|}{MAD_p}">
</br>
<img src="https://render.githubusercontent.com/render/math?math=\LARGE dist\_cat({\bf c}, {\bf x}) = \frac{1}{d_{cat}} \sum_{p=1}^{d_{cat}}I(c^p \neq x^p)">
</p>

#### > Laugel's implementation
<p align="justify">
This is another metric based on Laugel et al. [3] article where it uses Local Outiler Factor [4] score (LOF) with k=1. It provides an idea if the counterfactual point is close to the the other points from the dataset with the same classification. Therefore, although it's also a proximity metric, it has a different sense of DiCE's, which is more related to the distance of the counterfactual with the evaluation point.
</p>

#### Formal definition
<p align="center">
<img src="https://render.githubusercontent.com/render/math?math=\LARGE \text{proximity-score}({\bf c}) = \frac{dist({\bf c},  CP)}{\underset{f({\bf x}_v)=y'}{min}dist({CP, \bf x}_v)}">
</br>
</br>
Where:
</br>
</br>
<img src="https://render.githubusercontent.com/render/math?math=\LARGE CP = \underset{f({\bf x}_v)=y'}{min}({\bf x}_v, {\bf c})">
</p>

### Sparsity
<p align="justify">
The sparsity is a metric related to how many variables changed compared with the original point. Therefore, it can be simply calculated by the number of features that were modified in the counterfactual generation process.
</p>

### Interpretability
<p align="justify">
Interpretability is not a straightforward metric as it is related to how humans can interpret the counterfactual. However, Looveren et al. [5] described some metrics that can be used to assess the counterfactual's interpretability. They all are based on autoencoders and relate the output counterfactual with the original dataset points that have the same classification.

There are two interpretability metrics, <img src="https://render.githubusercontent.com/render/math?math=IM1"> and <img src="https://render.githubusercontent.com/render/math?math=IM2">. The former is related on how different is the counterfactual if compared with the original class autoencoder representation (<img src="https://render.githubusercontent.com/render/math?math=AE_{y'}">) and the counterfactual class representation (<img src="https://render.githubusercontent.com/render/math?math=AE_y">). Lower <img src="https://render.githubusercontent.com/render/math?math=IM1"> scores mean the counterfactual is better represented by the counterfactual class autoencoder, therefore, it better interpreted as being of the counterfactual set.
On the other hand, <img src="https://render.githubusercontent.com/render/math?math=IM2"> compares the counterfactual class autoencoder (<img src="https://render.githubusercontent.com/render/math?math=AE_{y'}">) with the entire dataset encoder (<img src="https://render.githubusercontent.com/render/math?math=AE">), in this case, lower scores mean the counterfactual is well represented both by the general dataset points and the specific points from the counterfactual class. The metrics used here were slightly modified from the original article as out data treatment is different from them.
</p>

#### Formal definition
<p align="center">
<img src="https://render.githubusercontent.com/render/math?math=\LARGE IM1 = \frac{||{\bf c} - AE_{y'}({\bf c})||_2^2}{||{\bf c} - AE_{y}({\bf c})||_2^2}">
</br>
<img src="https://render.githubusercontent.com/render/math?math=\LARGE IM2 = ||AE_y({\bf c}) - AE({\bf c})||_2^2">
</p>

### Connectedness (NOT IMPLEMENTED YET)
<p align="justify">
This metric was defined by Laugel [3] and it's identifies if a continuous path exists between the counterfactual and at least one point from the counterfactual class. This score aims to find counterfactuals that are generated by artifacts on the classification algorithm. These artifacts are regions that are classified as one specific class but does not have any training point related to it. This metric can be calculated by using DBSCAN with the ϵ and minPts values equal to 2.
</p>

### Reasonability (NOT IMPLEMENTED YET)
<p align="justify">
Reasonability is a metric defined by Le et al [6] where it evaluates if the counterfactual is informative and if it follows the dataset domain. 

An informative counterfactual is one which few correlated features are suggested, for example, education level is highly correlated with a person's age, this is because a degree takes time to be completed. Therefore, a counterfactual suggesting a person to get a degree and increase its age may give some "useless" information, as the mere fact of getting a degree will take some time.

However, it must be noted that although the case above have two highly correlated features, the mathematical validity of the counterfactual may depend of the alteration of both features. In other words, if just one feature is modified (education level) a counterfactual may not be generated.

#### Formal definition
</p>
<p align="center">
<img src="https://render.githubusercontent.com/render/math?math=\LARGE M\text{reasonability} = M\text{domain} * M\text{informative}">
</br>
</br>
Where:
</br>
</br>
<img src="https://render.githubusercontent.com/render/math?math=\LARGE M\text{domain} = \frac{1}{d} \sum_{p=1}^d I(c^p \in \text{DOM}({\bf X}))">
</br>
<img src="https://render.githubusercontent.com/render/math?math=\LARGE M\text{informative} = (1 - \frac{1}{d}  \sum^d_{p=1}\mathscr{I}(c^p))">
</br>
</p>

### Diversity
<p align="justify">
This is another metric introduced by DiCE [1] and is able to determine the diversity of a set of multiple (<img src="https://render.githubusercontent.com/render/math?math=k">) counterfactuals. It measures the distance between the same feature in different counterfactual explanations then, a large value mean the counterfactual set has high diversity, while low values may indicate low diversity.
</p>

#### Formal definition
<p align="center">
<img src="https://render.githubusercontent.com/render/math?math=\LARGE Diversity = \frac{1}{\text{COMB}^2_k}\sum_{i=1}^{k-1}\sum_{j=i%2B1}^{k}dist({\bf c}_i, {\bf c}_j)">
</br>
</br>
Where:
</br>
</br>
<img src="https://render.githubusercontent.com/render/math?math=\LARGE dist(\bullet)=dist\_cont(\bullet) %2B dist\_cat(\bullet)">
</p>

## References
1 - Mothilal, Ramaravind K and Sharma, Amit and Tan, Chenhao. *Explaining machine learning classifiers through diverse counterfactual explanations*. https://arxiv.org/abs/1905.07697

2 - Wachter, Sandra and Mittelstadt, Brent and Russell, Chris. *Counterfactual explanations without opening the black box: Automated decisions and the GDPR*. https://arxiv.org/abs/1711.00399

3 - Laugel, Thibault and Lesot, Marie-Jeanne and Marsala, Christophe and Detyniecki, Marcin. *Issues with post-hoc counterfactual explanations: a discussion*. https://arxiv.org/abs/1906.04774

4 - Markus M. Breunig , Hans-Peter Kriegel , Raymond T. Ng , Jörg Sander. *LOF: identifying density-based local outliers*. https://dl.acm.org/doi/10.1145/342009.335388

5 - Van Looveren, Arnaud and Klaise, Janis. *Interpretable counterfactual explanations guided by prototypes*. https://arxiv.org/abs/1907.02584

6 - Le, Thai and Wang, Suhang and Lee, Dongwon. *GRACE: Generating Concise and Informative Contrastive Sample to Explain Neural Network Model's Prediction*. https://arxiv.org/abs/1911.02042