zhiminy/EvalEng huggingface dataset:
Repository: understandable-machine-intelligence-lab/Quantus
Issue: minor typo in assert_attributions_order function
Root cause: Validation Gap

### Description
Thanks for the great project!
I have tried to work with `region_perturbation` metric, but this function does not accept order="lerf".
I think there is a typo in the function for assertion:
https://github.com/understandable-machine-intelligence-lab/Quantus/blob/f94853b653700b8f499d203118664edf8ad8bef3/quantus/helpers/asserts.py#L109-L126
`region_perturbation` metric only accepts either "lerf" or "morf" for order, so "lerf" instead of "lorf" should be used in the above function.

### Steps to reproduce the behavior
```
import torch
import numpy as np
from quantus.metrics.faithfulness.region_perturbation import RegionPerturbation

class TestModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.classifier = torch.nn.Linear(2,1)
    def forward(self, x):
        cls = self.classifier(x)
        return cls

model = TestModel()
model.eval()

metric = RegionPerturbation(order="lerf")
metric_results = metric(model=model, x_batch=np.random.rand(2,2), y_batch=np.array([0,1]), a_batch=np.random.rand(2,2))
```

### Minimum acceptance criteria
- "lorf" -> "lerf" in https://github.com/understandable-machine-intelligence-lab/Quantus/blob/f94853b653700b8f499d203118664edf8ad8bef3/quantus/helpers/asserts.py#L109


Thanks so much for highlighting this issue. We will fix this typo in upcoming PR.



Thank you very much!
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Getting started tutorial needs to specify explain_fun for MaxSensitivity metric
Root cause: Documentation Deficiency

### Description
The getting started tutorial doesn't run as expected when passing pre-calculated explanations/attribution to the MaxSensitivity metric.

### Error's source code
```
metric = quantus.MaxSensitivity(nr_samples=10,
  lower_bound=0.2,
  norm_numerator=quantus.fro_norm,
  norm_denominator=quantus.fro_norm,
  perturb_func=quantus.uniform_noise,
  similarity_func=quantus.difference)

scores = metric(
    model=model,
    x_batch=x_batch,
    y_batch=y_batch,
    a_batch=a_batch_saliency,
    device=device
)
```
When calculating the scores the MaxSensitivity gives the following error:
```
AssertionError: Make sure 'explain_func' is a Callable that takes model, inputs, targets and **kwargs as arguments.
```
Which is expected as the metric requires the ```explain_fun``` in order to evaluate the perturbed inputs.

### Minimum acceptance criteria
- Clearly specify in the tutorial what arguments are required by the metric.


Hi @smachar can you please link which tutorial you are referring to?

Cause the basic getting started tutorial (https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/tutorials/Tutorial_Getting_Started.ipynb) looks different from the code snippet you've created.


![image](https://github.com/understandable-machine-intelligence-lab/Quantus/assets/29717021/f9e7554e-893f-4b0d-a34c-a53ae38cdab1)


Here [https://github.com/understandable-machine-intelligence-lab/Quantus]( https://github.com/understandable-machine-intelligence-lab/Quantus ), in the main page **Step 3. Evaluate with Quantus**
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Pre-trained model loading API
Root cause: Unimplemented Feature Gap

### Description of the problem
- Quantus provides examples of simple models to get started with, e.g. https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/quantus/helpers/model/models.py#L128
- It is however up to user to download the weights from GitHub or event train them

### Description of a solution
- Provide API similar to `torchvision` or `keras.application`, e.g.
```
import quantus

model = quantus.sample_models.LeNet3D(weights="MNITS")
```
Which allows the user to directly execute examples.

- Some weights are already out there, but probably we will need to train a few simple models, store in GitHub.
- Weights, that can be downloaded from raw GitHub content under the hood and cached in local file system.
- Provide clear description in `readthedocs`, which weights for which models are available.
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Create transparency with respect to the metric’s individual data and model applicability
Root cause: Unimplemented Feature Gap

### Description of the problem
Create transparency with respect to the metric’s individual data and modelapplicability
- Add metric class markers to make it easier for users to know where they are applicable

### Implementation
- For each metric class add
- Model applicability
- Data applicability
- Name
- Add a user-friendly function to get all metrics associated with one datatype

### Minimum acceptance criteria
- Specify what is necessary for the issue to be closed.
- @mentions of the person that is apt to review these changes e.g., @annahedstroem


Done in PR #279
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Wanna get perturbed inputs from robustness metrics
Root cause: Unimplemented Feature Gap

It would be great to get perturbed inputs and their saliency maps as we use robustness metrics, such as LocalLipschitzEstimate. 
It is really easy to implement, just optionally return x_perturbed too.
This feature will create significant benefits when we produce qualitative results.
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Inconsistent implementation of get_random_layer_generator for Torch and TF
Root cause: Interface Contract Mismatch

### Description
- Here https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/quantus/helpers/model/tf_model.py#L272 for Keras model we shuffle each layer weights.
- Here https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/quantus/helpers/model/pytorch_model.py#L213 for torch model we reset module parameters (to default ones.)


### Minimum acceptance criteria
- Include both weight modification strategies for both framework's models.
- Keep the default version uniform.
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Wrong implementation in max_sensitivity.py
Root cause: Algorithmic Error

### Description
- Insert a clear and concise description of what the bug is.
In max_sensitivity.py line 386, the denominator use a=x_batch[instance_id]. but It seems that it should be a_batch[instance_id], as the  original code https://github.com/chihkuanyeh/saliency_evaluation/blob/master/infid_sen_utils.py, line 308 use normlization of explanation.


### Steps to reproduce the behavior
- Insert a description of the steps one needs to take to be able to reproduce the issue.
  - Step 1
  - Step 2
  - Step 3
- Insert a clear and concise description of what you expected to happen.
- If applicable, add screenshots to help explain your problem.
- Add any other context about the problem here. For example, set-up to run example.

### Minimum acceptance criteria
- Specify what is necessary for the issue to be closed.
- @mentions of the person that is apt to review these changes e.g., @annahedstroem


Fixed in PR https://github.com/understandable-machine-intelligence-lab/Quantus/pull/289.
Repository: understandable-machine-intelligence-lab/Quantus
Issue: toml missing
Root cause: Validation Gap

### Description
- I now can't install in colab. Stack trace:
~~~bash
[/usr/local/lib/python3.10/dist-packages/quantus/__init__.py](https://localhost:8080/#) in <module>
      7 # Set the correct version.
      8 import toml
----> 9 package = toml.load("pyproject.toml")
     10 __version__ = package["project"]["version"]
     11 

[/usr/local/lib/python3.10/dist-packages/toml/decoder.py](https://localhost:8080/#) in load(f, _dict, decoder)
    131 
    132     if _ispath(f):
--> 133         with io.open(_getpath(f), encoding='utf-8') as ffile:
    134             return loads(ffile.read(), _dict, decoder)
    135     elif isinstance(f, list):

FileNotFoundError: [Errno 2] No such file or directory: 'pyproject.toml'
~~~


### Steps to reproduce the behavior
- Insert a description of the steps one needs to take to be able to reproduce the issue.
  - Step 1 Go to Colab
  - Step 2 Install from wheel or from git directly
  - Step 3 import quantus


### Minimum acceptance criteria
-successful import 


Thanks for highlighting. Fixed in https://github.com/understandable-machine-intelligence-lab/Quantus/releases/tag/v0.4.3.
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Pointing game need a tolerance margin option
Root cause: Unimplemented Feature Gap

### Description of the problem
- Insert a clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
- Is your feature request related to a problem? Please describe.
- Add any other context about the problem here. For example, set-up to run the example.
Pointing game in original papar(Top-down Neural Attention by Excitation Backprop) expand ground truth region by a tolerance margin of 15 pixel, to compare different resolution saliency map methods fairly. 


### Description of a solution
Please refer the code from torchray:
https://github.com/facebookresearch/TorchRay/blob/master/torchray/benchmark/pointing_game.py

### Minimum acceptance criteria
- Specify what is necessary for the issue to be closed.
- @mentions of the person that is apt to review these changes e.g., @annahedstroem
Repository: understandable-machine-intelligence-lab/Quantus
Issue: numpy version requirements
Root cause: External Dependency Breakage

### Description
- when importing quantus, I got the following error: TypeError: 'numpy._DTypeMeta' object is not subscriptable'

### Steps to reproduce the behavior
- numpy version 1.21.5
- import quantus

### Minimum acceptance criteria
- after updating numpy to 1.26.0 the error was gone
- add the required numpy version to the docs


Hi @sltzgs, 
could you please post a full stacktrace of the exception? It would also help if you could provide a code snippet, for which it occurred, and describe your environment. 
As for the environment, it could be useful for us to know all the packages you have installed (e.g., `pip freeze > requirements_old.txt`), your OS and python distributions.
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Investigate vectorizing metric computation
Root cause: Unimplemented Feature Gap

### Description of the problem
- A lot of metrics do computations on a single instance. 
- They often could be computed in batches. 

### Description of a solution
Apply vectorization (or batch computations) for 
- Completeness
- Complexity

Investigate which other metrics can be batched/vectorized.

### Minimum acceptance criteria
- Metric outputs did not change.


A typical instance-wise metric is currently implemented like this way:
```python
return [self.evaluate_instance(x, a) for x, a in zip(x_batch, a_batch)]
```
we want to get rid of these `for-loops`.
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Adding more explantation methods
Root cause: Documentation Deficiency

This is more of a question I checked 
if I want to add LayerIntegratedGradients from captum or any other explanation method (for example pytorch forecasting library has built in attribution methods)
It seems to be that I should edit [quantus/helpers/constants.py] [quantus/functions/explanation_func.py] as long as I return a np.ndarray of same shape as inputs. Is there anything else or anywhere else I should also consider when adding these methods ?

Hey @youssefmecky96 thanks so much for your questions!

Yes, with the addition you describe, the following files are expecting an edit:
- [ ] `constants.py`
- [ ] `explanation_func.py`

but also, importantly add a test to see that it returns the correct shape and values in the correct places: 
- [ ] `test_explanation_func.py`

Also, make sure the captum arguments are included.

Otherwise, I'd look for the most similar captum method existing in the `quantus.explain` and start coding from there. 

Let me know if you have any other questions!
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Py37 end of life
Root cause: Unimplemented Feature Gap

### Description
- Python 3.7 has been already retired https://devguide.python.org/versions/, we probably should remove `py37` from the CI.


includes:
1. removal in CI/CD
2. removal in pyproject.toml
3. removal of 3.7 specific imports in scripts in repo
4. README.md update / announcement
Repository: understandable-machine-intelligence-lab/Quantus
Issue: InputInvariance for CNN
Root cause: Validation Gap

I'm trying to evaluate different explainability methods using the InputInvariance metric. I have faced some issues:
1- If there is a sequential container in pytorch implementation the first layer needs to be specified more precisely. something like : `module = modules[1][0]` at [here](https://github.com/understandable-machine-intelligence-lab/Quantus/blob/72a519bbcece68214edd8febe9a50de8d63314e2/quantus/helpers/model/pytorch_model.py#L344C1-L345C1).
2- I can't understand why torch.unique() has been used [here](https://github.com/understandable-machine-intelligence-lab/Quantus/blob/72a519bbcece68214edd8febe9a50de8d63314e2/quantus/helpers/model/pytorch_model.py#L352) and [here](https://github.com/understandable-machine-intelligence-lab/Quantus/blob/72a519bbcece68214edd8febe9a50de8d63314e2/quantus/helpers/model/pytorch_model.py#L356).
3- I expect the model's output to remain the same when you pass the shifted input from the modified model.

### Minimum acceptance criteria
- The output of the original input passed through the original network is the same as the shifted input passed through the modified network
@annahedstroem
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Infer_channel_first called even if channel_first is set
Root cause: Validation Gap

### Description
infer_channel_first is always called, even if channel_first has been specified.

### Steps to reproduce the behavior
Consider a model with a number of channels greater than H and W, infer_channel_first will fail and suggest
to set manually channel_first. But even if set it will fail and raise the error.
```python
import quantus
import torch
from torch import nn

model = nn.Sequential(
    nn.Conv2d(in_channels=156, out_channels=16, kernel_size=(3, 3), stride=1),
    nn.ReLU(),
    nn.Flatten(),
)
x = torch.randn(1, 156, 32, 32)
# make a classifier out of the model
in_dim = model(x).shape[1]
classifier = nn.Sequential(model, nn.Linear(in_dim, 10))
target = torch.tensor([0])
```
Then 
```python
attr = quantus.explain(
    model=classifier,
    inputs=x,
    targets=target,
    method="IntegratedGradients",
)
```
Raises 
```python
ValueError: Ambiguous input shape. Cannot infer channel-first/channel-last order. Try setting the `channel_first` argument
```
but
```python
attr = quantus.explain(
    model=classifier,
    inputs=x,
    targets=target,
    method="IntegratedGradients",
    channel_first=True,
)
```
raises the same error. 
The channel_first argument is present in kwargs but using kwargs.get will evaluate the default value so if it can't be inferred it cannot be overridden.

Proposed fix: 
replace as follows where needed (l. 480 of [explaination_func](https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/quantus/functions/explanation_func.py) for instance)
```python
    # channel_first = kwargs.get("channel_first", infer_channel_first(inputs)) 
    chanel_first = kwargs["channel_first"] if "channel_first" in kwargs else infer_channel_first(inputs)
``` 

### Minimum acceptance criteria
- Get the channel_first value if present in kwargs without call to infer_channel_first.
- @annahedstroem

Related note: 
not as problematic but in the use 
```python
kwargs.get("baseline", torch.zeros_like(inputs))
```
A tensor of zeros of the size of inputs will be created even if a baseline is provided.

Thanks @sergedurand for highlighting this issue and providing a suggestion for a solution. 

See PR: https://github.com/understandable-machine-intelligence-lab/Quantus/pull/305.

The PR https://github.com/understandable-machine-intelligence-lab/Quantus/pull/305 is merged with main. Please check if the solution works for you @sergedurand so we can close the issue!

Yes it does, thanks for the fixes!
Repository: understandable-machine-intelligence-lab/Quantus
Issue: tf-explain does not work with various data types, make more explicit to users
Root cause: Validation Gap

### Description
-tf-explain does not work with various data types, make more explicit to users

### Steps to reproduce the behavior
- e.g., with help of exceptions, warnings etc
- including investigate why certain tests with too few dimensions may pass e.g., see MPRT test

### Minimum acceptance criteria
- Specify what is necessary for the issue to be closed.
- @mentions of the person that is apt to review these changes e.g., @annahedstroem
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Refactor quantus.evaluate() and expand docstring with examples

### Description of the problem
- Make it clearer what XAI methods options are available

### Description of a solution
TBD

### Minimum acceptance criteria
- Specify what is necessary for the issue to be closed.
- @mentions of the person that is apt to review these changes e.g., @annahedstroem


@sebastian-lapuschkin please add here in case you imagined some specific functionality etc. but I will look at refactoring this function in the coming days

Fixed in PR #314, see release: https://github.com/understandable-machine-intelligence-lab/Quantus/releases/tag/v0.5.1.
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Make SmoothMPRT faster for torch (see implementation)
Root cause: Unimplemented Feature Gap

### Description of the problem
- Make SmoothMPRT faster with torch

### Description of a solution
- test and call this function within SmoothMPRT metric
```python
    def explain_smooth_batch_torch(
        self,
        model: ModelInterface,
        x_batch: np.ndarray,
        y_batch: np.ndarray,
        std: float,
        **kwargs,
    ) -> np.ndarray:
        """
        Compute explanations, normalise and take absolute (if was configured so during metric initialization.)
        This method should primarily be used if you need to generate additional explanation
        in metrics body. It encapsulates typical for Quantus pre- and postprocessing approach.
        It will do few things:
            - call model.shape_input (if ModelInterface instance was provided)
            - unwrap model (if ModelInterface instance was provided)
            - call explain_func
            - expand attribution channel

        Parameters
        -------
        model:
            A model that is subject to explanation.
        x_batch:
            A np.ndarray which contains the input data that are explained.
        y_batch:
            A np.ndarray which contains the output labels that are explained.
        std : float
            Standard deviation of the Gaussian noise.
        kwargs: optional, dict
            List of hyperparameters.

        Returns
        -------
        a_batch:
            Batch of explanations ready to be evaluated.
        """
        if not isinstance(x_batch, torch.Tensor):
            x_batch = torch.Tensor(x_batch).to(self.device)

        if not isinstance(y_batch, torch.Tensor):
            y_batch = torch.as_tensor(y_batch).to(self.device)

        a_batch_smooth = torch.zeros_like(x_batch)
        for n in range(self.nr_samples):
            # the last epsilon is defined as zero to compute the true output,
            # and have SmoothGrad w/ n_iter = 1 === gradient
            if n == self.nr_samples - 1:
                epsilon = torch.zeros_like(x_batch)
            else:
                epsilon = torch.randn_like(x_batch) * std

            a_batch = quantus.explain(model, x_batch + epsilon, y_batch, **kwargs)

            if a_batch_smooth is None:
                a_batch_smooth = a_batch / self.nr_samples
            else:
                a_batch_smooth += a_batch / self.nr_samples

        return a_batch_smooth
```
### Minimum acceptance criteria
- Specify what is necessary for the issue to be closed.
- @mentions of the person that is apt to review these changes e.g., @annahedstroem
Repository: understandable-machine-intelligence-lab/Quantus
Issue: Add new "compressibility" (complexity) metric
Root cause: Unimplemented Feature Gap

### Description of the feature
- Add "compressibility" (complexity) metric, i.e., by measuring the file size of a jpeg (or otherwise) compressed attribution map
- [Paper](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9369420): Section IV.B, around the last table in the section and footnote 1

### Minimum acceptance criteria
- Specify what is necessary for the issue to be closed.
- @mentions of the person that is apt to review these changes e.g., @annahedstroem