zhiminy/EvalEng huggingface dataset:
Repository: understandable-machine-intelligence-lab/Quantus
Issue: attribution localisation metric should consider positive attributions only
Root cause: Algorithmic Error

### Description
According to the paper (https://arxiv.org/pdf/1910.09840.pdf) attribution localisation should be computed only from the positive values in the attribution map. The implementation in quantus considers negative values also in the computation.

### Minimum acceptance criteria
- [ ] Update implementation to only consider positive attribution values
- [ ] Update documentation of AttributionLocalisation to correctly state this
- [ ] Potentially update test cases, in case above changes break any of them
- [ ] I think the warning w.r.t. inside_attribution_ration > 1.0 can also be removed then
- review pls @annahedstroem 


The current implementation set `abs=True`

and the current docstring says
```python
"""
Attribution Localization implements the ratio of positive attributions within the target to the overall
attribution. High scores are desired, as it means, that the positively attributed pixels belong to the
targeted object class.
"""
```
which for me is sufficient. Can we close this issue? @leanderweber
Repository: understandable-machine-intelligence-lab/Quantus
Issue: make expected input shape for segmentation masks more forgiving
Root cause: Validation Gap

### Description of the problem
The accepted input shapes for segmentation masks (s_batch) seem very strict currently. E.g, for data of the shape (32, 3, 224, 224), only segmentation masks of exactly shape (32, 1, 224, 224) are accepted. I believe quantus could handle some additional shape matching to increase usability. I.e., shapes such as (32, 224, 224) or (32, 224, 224, 1) should also be acceptable, as they could easily be converted. 

### Minimum acceptance criteria
- [ ] allow for more input shape freedom (within reason) for segmentation masks
- [ ] write additional test cases for these shapes
- pls review @annahedstroem
Repository: understandable-machine-intelligence-lab/Quantus
Issue: docstring for RelevanceMassAccuracy metric does not match implementation

### Description
The docstring for the RelevanceMassAccuracy metric states "The Relevance Mass Accuracy computes the ratio of positive attributions inside the bounding box to the sum of overall positive attributions". However, both in the implementation within quantus, as well as in the corresponding paper (https://www.sciencedirect.com/science/article/pii/S1566253521002335), both positive and negative attributions are considered.

### Minimum acceptance criteria
- [x] Update the docstring to reflect the implementation of the metric.
- pls review @annahedstroem


Done in commit https://github.com/understandable-machine-intelligence-lab/Quantus/pull/256/commits/efe19d8bbdfdd1c55b71cfafd78043064d8d8695
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Metrics support for tabular data
Root cause: Unimplemented Feature Gap

### Description of the problem
- Support tabular data for different categories of explanation quality such as Robustness, Faithfulness 
- Lay out the technical requirements of the different categories:
  - [] Faithfulness
  - [] Robustness
  - [] Localisation
  - [] Complexity
  - [] Randomisation

### Description of a solution
- Define minimal examples for perturbation given different dtypes
- Provide `data_availability` desc for all metrics if cannot support
- Warnings

### Minimum acceptance criteria
- New tests added
- All tests 


Randomisation and Complexity metrics already have support for tabular data, it is demonstrated in [Tutorial_Getting_Started_with_Tabular_Data.ipynb](https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/tutorials/Tutorial_Getting_Started_with_Tabular_Data.ipynb).

Hi @annahedstroem, @dilyabareeva, @aaarrti 

Are there any updates about this task? I was doing some investigation using Tabular data as a proxy measure before going to NLP and I could benefit from this 


Can you confirm whether Randomization and Complexity are the only metrics that support tabular data, or are there any others that also do?

> Can you confirm whether Randomization and Complexity are the only metrics that support tabular data, or are there any others that also do?

Hi @boltzmann-brain, 

you can check which data domain metric supports with: 

```python
import quantus

quantus.metrics.<MetricClass>.data_applicability
```

If the return value contains `<DataType.TABULAR: 'tabular'>`, the metric was verified to support tabular data.
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Implement six metrics from Disagreements paper by Krishna et al., 2022
Root cause: Unimplemented Feature Gap

### Description of the problem
- There are 6 ranking metrics presented in the paper: The Disagreement Problem in Explainable Machine Learning:
A Practitioner’s Perspective https://arxiv.org/pdf/2202.01602.pdf 

### Description of a solution
- Let's implement them under a new category of "ranking metrics"

### Minimum acceptance criteria
- New tests implemented and passed
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Mock model for return_nan_when_prediction_changes tests
Root cause: Validation Gap

### Description
- The tests for `return_nan_when_prediction_changes` parameter rely on high noise amplitudes, however if the predictions actually change is still not deterministic, and occasionally the tests will fail because of it.

### Steps to reproduce the behavior
- Mock models outputs for tests with `return_nan_when_prediction_changes=True`


![image](https://user-images.githubusercontent.com/44274784/214260862-7bc7e07d-4fe6-45f8-b8c9-c555a2f10153.png)


Hi @aaarrti 

What is meant by "however if the predictions actually change is still not deterministic, and occasionally the tests will fail because of it."? Do you mean that it is hard to predict if a model sometimes doesn't return the same predicted class after perturbation? **Why will a test fail because of this?**

Thank you!

> Hi @aaarrti
> 
> What is meant by "however if the predictions actually change is still not deterministic, and occasionally the tests will fail because of it."? Do you mean that it is hard to predict if a model sometimes doesn't return the same predicted class after perturbation? **Why will a test fail because of this?**
> 
> Thank you!

It means, if this test was run, e.g., 100 times, it will fail once or twice

@annahedstroem here is an example of "unstable test" https://github.com/understandable-machine-intelligence-lab/Quantus/actions/runs/4519087543/jobs/7959357854?pr=240
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Possible error in the application of aggregate_func in the 'Metric' class found in the 'base.py' script, especialy while using numpy methods.
Root cause: Documentation Deficiency

![Screenshot from 2023-01-10 01-02-41](https://user-images.githubusercontent.com/75955459/211433060-1c7fc51a-ad5a-4cc6-b953-bc133a661ae3.png)

Please confirm whether the mean should only be aggregated over the axis=0 direction, i.e along the number of steps (Fraction of pixels flipped)

Essentially the line 235 would be replaced by line 239 in the attached reference picture.

Thanks,
Shreyas Gururaj.

@Shreyas-Gururaj Thanks for your feedback! Could you provide more detail about the metric you were trying to use, and maybe a code snippet that would help us to recreate the error? That would be very helpful.

@dilyabareeva I was trying to use pixel flipping metric, with the 'explain_func' argument set to 'generate_zennit_explanation' and passing my own LRP composite and canonizer to the 'explain_func_kwargs' argument. The code snippet to reproduce this error would be a very large snippet. 

In short the aggregate function np.mean applied in the metric class returns a numpy array with a single element, instead of returning one element for each step of the pixel flipping. So, my guess was to take the np.mean along the direction of each individual steps accross each batch (i.e axis=0 direction).

@Shreyas-Gururaj Quantus provides flexibility in terms of aggregate function choice. For your case, I would suggest passing `aggregate_func=lambda x: numpy.mean(x, axis=0)` to the metric. Changing the base class to include axis=0 as you suggest might not make sense for all the metrics. Hope this helps.

@dilyabareeva Hi, Thanks for the update. Yes, that might break other running codes. I would make the necessary changes.
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Build an explain wrapper around innvestigate 2.0
Root cause: Unimplemented Feature Gap

### Description of the problem
- Innvestigate is a popular library for explanation methods https://github.com/albermax/innvestigate

### Description of a solution
- Build a wrapper around it
- Related files: `explain_func.py`, `constants.py`
- New `pytest`s

### Minimum acceptance criteria
- Write additional tests
- @mentions of the person that is apt to review these changes e.g., @annahedstroem
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Multi-label classification support
Root cause: Unimplemented Feature Gap

### Description of the problem
- afaik so far, Quantus assumes categorical encoding of `y_batch`, where each input data-point is assigned to exactly 1 label.

### Description of a solution
- Investigate possibilities to support multi-label classification.
- Add tutorial showcasing multi-label classification.


Is this issue still relevant? Currently, I'm working on my master thesis in XAI and MLC. For the evaluation I've forked Quantus and started to implement some very basic MLC support.
My current design choices are strongly based on my own use-case, but if you are interested we could discuss your requirements to make these changes public.

I can keep you updated on my progress and write a more detailed description of my approach if you want.
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Global and metric-wise config instead of `__init__` args/kwargs
Root cause: Architectural Constraint

### Description of the problem
- Each metric has quite many parameters passed to `__init__`. Most of them are shared among all metrics, and play the role of configuration.

### Description of a solution
- E.g., PRNG seed can be a global, library config value.
- E.g., `abs`, `normalise`, `return_aggregate`, `disable_warnings`, `display_progressbar` could be defined in config class, e.g., 
```
class MetricConfig(NamedTuple)
abs: bool
normalise: bool
return_aggreagte: bool
disable_warnings: bool
display_progressbar: bool
```
obviously, some metrics will have their own specialized config classes.
- This could reduce visual complexity of code and misconfiguration caused by an overwhelming number of args and kwargs.
- Alternativelly, we could use [pydantic](https://github.com/pydantic/pydantic) or [ml_collections](https://github.com/google/ml_collections) instead of named tuples.
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Export function signatures for user
Root cause: Documentation Deficiency

### Description of the problem
- Quantus is designed with a goal of being highly customizable. At this time, however, if user want to define his own, e.g.,  `similarity_func`, they need to read into the source code.

### Description of a solution
- Define type aliases for commonly used functions, e.g, `SimilarityFn = Callable[[np.ndarray, np.ndarray], np.ndarray]`
- Export type aliases at top level of library
- As a bonus, we could have more precise type hints in metrics, without overwhelmingly long type annotations.
- Ideally, could be done together with https://github.com/understandable-machine-intelligence-lab/Quantus/issues/70
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Move example models to `tests/`
GitHub labels: good first issue

refactoring

### Description of the problem
- Not really a problem
- Example models in https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/quantus/helpers/model/models.py are not required for the library to function, 
neither are they part of user-facing API.

### Description of a solution
- Move https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/quantus/helpers/model/models.py to `tests`, so they are ignored during lib distribution.
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Quantus.nlp support for custom models
Root cause: Unimplemented Feature Gap

### Description of the problem
- (https://github.com/understandable-machine-intelligence-lab/Quantus/pull/224) is designed with a goal to support models from Hugging Face Hub 🤗  because it is currently the most popular model sharing service. Ideally, user should also be able 
to provide his model, implementing `TextClassifier`, this is however not tested.

### Description of a solution
- Ensure Quantus compatibility with non HF models.
- Should be done after https://github.com/understandable-machine-intelligence-lab/Quantus/pull/224 is closed.
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Investigate support for NLP tasks
Root cause: Unimplemented Feature Gap

### Description of the problem
- First iteration of NLP support focuses on text-classification https://github.com/understandable-machine-intelligence-lab/Quantus/issues/103
- There is much more to NLP than  text-classification, e.g., here is an overview of tasks https://medium.com/@miranthaj/25-nlp-tasks-at-a-glance-52e3fdff32e2.

### Description of a solution
- Investigate how explanation for other NLP tasks could look like, and how could we benchmark them.
- Quite a few of them could be reduced to classification, though.
- Should be started only after https://github.com/understandable-machine-intelligence-lab/Quantus/pull/224 is closed.
Repository: understandable-machine-intelligence-lab/Quantus
Issue: plot() missing 1 required positional argument: 'plot_func'
Root cause: Validation Gap

Getting error here 
# Plot example!
pixel_flipping.plot(y_batch=y_batch, scores=scores)

message:
TypeError: plot() missing 1 required positional argument: 'plot_func'

Thanks a lot for this! Is this run from a notebook, which one?

Will investigate.

It seems plot_func is a required parameter of this function, I solved the issue by passing plot_func=None into it. 
<img width="967" alt="image" src="https://user-images.githubusercontent.com/101536791/234200097-d5fe1399-17f1-45a5-b4a6-b5680b17856e.png">


Awesome, I'll update that is the source code.

See commit https://github.com/understandable-machine-intelligence-lab/Quantus/pull/256/commits/cc69cc05430b50a3c5b05533caf93f8259e0cf6d.
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Missing module in 'quantus.helpers.utils'
Root cause: Configuration Error

A error just popped up which is required for torch models when I want to evaluate XAI methods:
----------------------------------------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-21-41b8706a9d11>](https://localhost:8080/#) in <cell line: 8>()
      6 
      7 # Call the metric instance to produce scores.
----> 8 results = {method: selectivity(model=model, 
      9                                x_batch=x_batch,
     10                                y_batch=y_batch,

[/usr/local/lib/python3.9/dist-packages/quantus/helpers/model/pytorch_model.py](https://localhost:8080/#) in shape_input(self, x, shape, channel_first, batched)
    125         # Set channel order according to expected input of model.
    126         if self.channel_first:
--> 127             return utils.make_channel_first(x, channel_first)
    128         raise ValueError("Channel first order expected for a torch model.")
    129 

AttributeError: module 'quantus.helpers.utils' has no attribute 'make_channel_first'
----------------------------------------------------------------------------------------------------------

I also checked utils module, but this module already has 'make_channel_first' function. IDK why this error happens?

Can anyone propose a solution?

Hi, thank you for letting us know about this!

Can you give some more context around this call? Is it from a notebook, in that case, which one?

The function `make_channel_first` do exist in `quantus.helpers.utils`.

Thanks!


I meet the same issue in https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/tutorials/Tutorial_ImageNet_Example_All_Metrics.ipynb when use ROAD
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Explanation pre- and post-processing pipeline
Root cause: Unimplemented Feature Gap

### Description of the problem
- attribution preprocessing before metrics should be available as separate modules which could be pipelined, like torchvision.transform or sklearn Pipeline. 
- At the moment, attribution preprocessing, such as normalise or interpolation, and even - attribution computation like softmax, are currently performed under-the-hood in each metric.
- is a follow-up for https://github.com/understandable-machine-intelligence-lab/Quantus/issues/186
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Implement custom plot function
Root cause: Documentation Deficiency

Hi! I want to plot the results of some Quantus metrics with different types of graphs for easy comparison when researching, how can I implement a custom plot function which passed as plot_func into the metric.plot() function? I made an effort to research about Quantus but didn't find any example or document about it

Hey! Here we have a short guide to how to contribute to Quantus.

https://quantus.readthedocs.io/en/latest/docs_dev/CONTRIBUTING.html

If you want to add a plotting function, please do so under `quantus/helpers/plotting.py`

Hope it helps, otherwise feel free to ask questions.
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Huggingface pip dependency issue Python 3.8
Root cause: Environment Incompatibility

### Description
- Huggingface package

### Steps to reproduce the behavior
- If installing quantus with conda Python 3.8, this package issue appear:
![image](https://github.com/understandable-machine-intelligence-lab/Quantus/assets/29717021/6ed49e57-c208-43ac-acf0-a093eb1374ac)

### Minimum acceptance criteria
- Specify what is necessary for the issue to be closed.
- @mentions of the person that is apt to review these changes e.g., @annahedstroem
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Write a short guide on how to use Quantus for different datasets (readthedocs.org)

### Description of the problem
- Write a short guide on how to use Quantus for different datasets (readthedocs.org), under `Getting Started`
- https://quantus.readthedocs.io/en/latest/

### Minimum acceptance criteria
- Specify what is necessary for the issue to be closed.
- @mentions of the person that is apt to review these changes e.g., @annahedstroem
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Some minor mistakes in the documentation

### Description
Thanks for the great project foremost! I went through the ReadMe of the project to use it for an XAI method I am working on. While working my way through, I encountered some small spelling mistakes and some errors in the provided example code. Nothing groundbreaking, but I guess others who will be using the ReadMe as a first guidance might encounter the same problems. 

I fixed the things I found in a local branch. Easiest would be if I could receive sufficient rights to push the branch and create corresponding pull request. Otherwise, I can post my changes here to the issue.

### ~~Steps to reproduce the behavior~~ Some examples
[here](https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/README.md?plain=1#L238):
 ```python
torchvision.datasets.MNIST(root='./sample_data', download=True, transforms=transforms.Compose([transforms.ToTensor()]))
```
 raises an error because the parameter `transforms` does not exist. Probably `transform`.

or

[here](https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/README.md?plain=1#L269): 2nd variable probably should be `a_batch_integrated`
```python
a_batch_saliency = load("path/to/precomputed/saliency/explanations")
a_batch_saliency = load("path/to/precomputed/intgrad/explanations")
```
and some minor ones

### Minimum acceptance criteria
- the small mistakes fixed


Hi @EricKolibacz, 
thanks for pointing out the issue! The fix should be live very soon. 
For the future, if you want to contribute on a similar issue, please create a fork of the repository, then you would be allowed to open a pull request (from your fork into the main repository)!

Hi @aaarrti 

Alright, next time I will do so :+1: Thanks for creating the PR.

The other minor mistakes I found in the documentation:

- [This important path](https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/README.md?plain=1#L320) did not work for me. I fixed it by modifying it to this
```python
metric = quantus.MaxSensitivity(nr_samples=10,
                                lower_bound=0.2,
                                norm_numerator=quantus.functions.norm_func.fro_norm,
                                norm_denominator=quantus.functions.norm_func.fro_norm,
                                perturb_func=quantus.uniform_noise,
                                similarity_func=quantus.difference)
```

- A missing `:` at [the end of the function's output type hint](https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/README.md?plain=1#L362).

- And newer `torch` version does not support [this syntax](https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/README.md?plain=1#L242) anymore, so:
```python
x_batch, y_batch = iter(test_loader).next() # next(iter(test_loader)) for newer versions of torch
```
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Different SHA256 checksums for GitHub release and PyPi release 
Root cause: Architectural Constraint

### Description
The .whl file uploaded to GitHub release, and the one on PyPI have different sha256 sums. Probably because, they were built on different hosts (mb event with different python versions). This will cause pip to see those two as different files. 

### Suggested solution
We need to upload the same file to GitHub and to PyPI. I'd suggest the following pipeline:
![Quantus_release_pipeline drawio (1)](https://github.com/understandable-machine-intelligence-lab/Quantus/assets/44274784/a94d7ea7-73d1-4b20-8123-bd7637eedffc)



### Steps to reproduce the behavior
```
# Github release
openssl sha256 quantus-0.4.0-py3-none-any.whl
# PyPi release
openssl sha256 quantus-0.4.0-py3-none-any\ \(1\).whl
```
### Minimum acceptance criteria
- The `.whl` in GitHub release, and in PyPi must be the same.
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Investigate shape and data type type-hints.

### Description of the problem
- We all love type hints. However, they mostly look the following way in Quantus:
```
def func(*args, **kwargs) -> np.ndarray: ...
```
- This is correct, but what would be much more helpful is to indicate expected shapes and data type.
I recently discovered a library, which does this https://github.com/google/jaxtyping. Despite the name `jaxtyping`, it does not require `jax` or `jaxlib` installation, and can be used with Numpy, TensorFlow and Torch as well.
- Let's investigate if we could make Quantus API more descriptive this way.
- Bonus point, it should also work with static type-checkers, and could help us catch shape or data type issues during linting stage.