Repository: understandable-machine-intelligence-lab/Quantus
Issue: Error on certain XAI metrics for tabular data 
Root cause: Validation Gap

### Description
Some XAI metrics throw errors on precomputed integrated gradients explanations on tabular data.  

### Steps to reproduce the behavior
- The model used is a two layer neural network (each with 40 neurons and ReLU) in tensorflow keras.  
- The dataset is tabular. Dimension of X_test is (4977, 77), Y_test is (4977,) and attributions_test is (4977, 77).
- attributions_test consists of integrated gradients using INNvestigate library. 
- Call infidelity using the following:
```
scores_infidelity = quantus.Infidelity(
                   return_aggregate=True,
                   disable_warnings=False,
                   display_progressbar=True,
                   abs=True, normalise=True
                   )(model=model,
                     x_batch=X_test,
                     y_batch=Y_test,
                     a_batch=attributions_test)
```
- Call faithfulness correlation using the following:
```
scores_faithfulnesscorrelation = quantus.FaithfulnessCorrelation(
                   return_aggregate=True,
                   disable_warnings=False,
                   display_progressbar=True,
                   abs=True, normalise=True)(model=model,
                     x_batch=X_test,
                     y_batch=Y_test,
                     a_batch=attributions_test)
```

### Error Trace
- Infidelity
```
File ~/anaconda3/lib/python3.11/site-packages/quantus/metrics/faithfulness/infidelity.py:269, in Infidelity.__call__(self, model, x_batch, y_batch, a_batch, s_batch, channel_first, explain_func, explain_func_kwargs, model_predict_kwargs, softmax, device, batch_size, **kwargs)
    180 def __call__(
    181     self,
    182     model,
   (...)
    194     **kwargs,
    195 ) -> List[float]:
    196     """
    197     This implementation represents the main logic of the metric and makes the class object callable.
    198     It completes instance-wise evaluation of explanations (a_batch) with respect to input data (x_batch),
   (...)
    267         >> scores = metric(model=model, x_batch=x_batch, y_batch=y_batch, a_batch=a_batch_saliency)
    268     """
--> 269     return super().__call__(
    270         model=model,
    271         x_batch=x_batch,
    272         y_batch=y_batch,
    273         a_batch=a_batch,
    274         s_batch=s_batch,
    275         custom_batch=None,
    276         channel_first=channel_first,
    277         explain_func=explain_func,
    278         explain_func_kwargs=explain_func_kwargs,
    279         model_predict_kwargs=model_predict_kwargs,
    280         softmax=softmax,
    281         device=device,
    282         batch_size=batch_size,
    283         **kwargs,
    284     )

File ~/anaconda3/lib/python3.11/site-packages/quantus/metrics/base.py:290, in Metric.__call__(self, model, x_batch, y_batch, a_batch, s_batch, channel_first, explain_func, explain_func_kwargs, model_predict_kwargs, softmax, device, batch_size, custom_batch, **kwargs)
    288 for d_ix, data_batch in enumerate(batch_generator):
    289     data_batch = self.batch_preprocess(data_batch)
--> 290     result = self.evaluate_batch(**data_batch)
    291     self.evaluation_scores.extend(result)
    293 # Call post-processing.

File ~/anaconda3/lib/python3.11/site-packages/quantus/metrics/faithfulness/infidelity.py:433, in Infidelity.evaluate_batch(self, model, x_batch, y_batch, a_batch, **kwargs)
    402 def evaluate_batch(
    403     self,
    404     model: ModelInterface,
   (...)
    408     **kwargs,
    409 ) -> List[float]:
    410     """
    411     This method performs XAI evaluation on a single batch of explanations.
    412     For more information on the specific logic, we refer the metric’s initialisation docstring.
   (...)
    430         The evaluation results.
    431     """
--> 433     return [
    434         self.evaluate_instance(model=model, x=x, y=y, a=a)
    435         for x, y, a in zip(x_batch, y_batch, a_batch)
    436     ]

File ~/anaconda3/lib/python3.11/site-packages/quantus/metrics/faithfulness/infidelity.py:434, in <listcomp>(.0)
    402 def evaluate_batch(
    403     self,
    404     model: ModelInterface,
   (...)
    408     **kwargs,
    409 ) -> List[float]:
    410     """
    411     This method performs XAI evaluation on a single batch of explanations.
    412     For more information on the specific logic, we refer the metric’s initialisation docstring.
   (...)
    430         The evaluation results.
    431     """
    433     return [
--> 434         self.evaluate_instance(model=model, x=x, y=y, a=a)
    435         for x, y, a in zip(x_batch, y_batch, a_batch)
    436     ]

File ~/anaconda3/lib/python3.11/site-packages/quantus/metrics/faithfulness/infidelity.py:324, in Infidelity.evaluate_instance(self, model, x, y, a)
    320 sub_results = []
    322 for patch_size in self.perturb_patch_sizes:
    323     pred_deltas = np.zeros(
--> 324         (int(a.shape[1] / patch_size), int(a.shape[2] / patch_size))
    325     )
    326     a_sums = np.zeros(
    327         (int(a.shape[1] / patch_size), int(a.shape[2] / patch_size))
    328     )
    329     x_perturbed = x.copy()

IndexError: tuple index out of range
```
- Faithfulness Correlation
```
File ~/anaconda3/lib/python3.11/site-packages/quantus/metrics/faithfulness/faithfulness_correlation.py:262, in FaithfulnessCorrelation.__call__(self, model, x_batch, y_batch, a_batch, s_batch, channel_first, explain_func, explain_func_kwargs, model_predict_kwargs, softmax, device, batch_size, custom_batch, **kwargs)
    169 def __call__(
    170     self,
    171     model,
   (...)
    184     **kwargs,
    185 ) -> List[float]:
    186     """
    187     This implementation represents the main logic of the metric and makes the class object callable.
    188     It completes instance-wise evaluation of explanations (a_batch) with respect to input data (x_batch),
   (...)
    260         >> scores = metric(model=model, x_batch=x_batch, y_batch=y_batch, a_batch=a_batch_saliency)
    261     """
--> 262     return super().__call__(
    263         model=model,
    264         x_batch=x_batch,
    265         y_batch=y_batch,
    266         a_batch=a_batch,
    267         s_batch=s_batch,
    268         custom_batch=custom_batch,
    269         channel_first=channel_first,
    270         explain_func=explain_func,
    271         explain_func_kwargs=explain_func_kwargs,
    272         softmax=softmax,
    273         device=device,
    274         model_predict_kwargs=model_predict_kwargs,
    275         batch_size=batch_size,
    276         **kwargs,
    277     )

File ~/anaconda3/lib/python3.11/site-packages/quantus/metrics/base.py:266, in Metric.__call__(self, model, x_batch, y_batch, a_batch, s_batch, channel_first, explain_func, explain_func_kwargs, model_predict_kwargs, softmax, device, batch_size, custom_batch, **kwargs)
    263 warn.deprecation_warnings(kwargs)
    264 warn.check_kwargs(kwargs)
--> 266 data: Dict[str, Any] = self.general_preprocess(
    267     model=model,
    268     x_batch=x_batch,
    269     y_batch=y_batch,
    270     a_batch=a_batch,
    271     s_batch=s_batch,
    272     custom_batch=custom_batch,
    273     channel_first=channel_first,
    274     explain_func=explain_func,
    275     explain_func_kwargs=explain_func_kwargs,
    276     model_predict_kwargs=model_predict_kwargs,
    277     softmax=softmax,
    278     device=device,
    279 )
    281 # Create generator for generating batches.
    282 batch_generator = self.generate_batches(
    283     data=data,
    284     batch_size=batch_size,
    285 )

File ~/anaconda3/lib/python3.11/site-packages/quantus/metrics/base.py:473, in Metric.general_preprocess(self, model, x_batch, y_batch, a_batch, s_batch, channel_first, explain_func, explain_func_kwargs, model_predict_kwargs, softmax, device, custom_batch)
    463 data = {
    464     "model": model,
    465     "x_batch": x_batch,
   (...)
    469     "custom_batch": custom_batch,
    470 }
    472 # Call custom pre-processing from inheriting class.
--> 473 custom_preprocess_dict = self.custom_preprocess(**data)
    474 # Save data coming from custom preprocess to data dict.
    475 if custom_preprocess_dict is not None:

File ~/anaconda3/lib/python3.11/site-packages/quantus/metrics/faithfulness/faithfulness_correlation.py:356, in FaithfulnessCorrelation.custom_preprocess(self, x_batch, **kwargs)
    339 """
    340 Implementation of custom_preprocess_batch.
    341 
   (...)
    353     returning a custom preprocess batch (custom_preprocess_batch).
    354 """
    355 # Asserts.
--> 356 asserts.assert_value_smaller_than_input_size(
    357     x=x_batch, value=self.subset_size, value_name="subset_size"
    358 )

File ~/anaconda3/lib/python3.11/site-packages/quantus/helpers/asserts.py:290, in assert_value_smaller_than_input_size(x, value, value_name)
    273 """
    274 Checks if value is smaller than input size, assumes batch and channel first dimension.
    275 
   (...)
    287 None
    288 """
    289 if value >= np.prod(x.shape[2:]):
--> 290     raise ValueError(
    291         f"'{value_name}' must be smaller than input size."
    292         f" [{value} >= {np.prod(x.shape[2:])}]"
    293     )

ValueError: 'subset_size' must be smaller than input size. [224 >= 1.0]
```
### Minimum acceptance criteria
- Any suggestions on how to resolve this issue will be helpful. Thank you.
- @annahedstroem


Hi @sallana22

As can be found in the logic of the infidelity metric, it "patches the input" meaning that it expects input data. I think the line:
```python
data_applicability = {DataType.IMAGE, DataType.TIMESERIES, DataType.TABULAR}
```
may have caused the issue, as our implementation of the infidelity metric does not support tabular data. This is removed in the following commit: https://github.com/understandable-machine-intelligence-lab/Quantus/commit/0ef57b62dc6e202deb4d77e12009d61781c7eb50.

The second error appeared as we assumed that the input also had a channel, the docstring includes __"Checks if value is smaller than input size, assumes batch and channel first dimension."__

I have made changes (see PR https://github.com/understandable-machine-intelligence-lab/Quantus/pull/326) and will merge to main branch when the tests branch. Just pull from the main and then check that it is working for you.



Hi @annahedstroem,

Thanks for the quick resolution! I will exclude infidelity analysis on the tabular dataset. 

For faithfulness correlation, your changes work but hit an error on asserts.py, line 295. Adding an elif on line 295 as below solves the problem:
```
if len(x.shape) == 2:
        if value >= np.prod(x.shape[1:]):
            raise ValueError(
                f"'{value_name}' must be smaller than input size."
                f" [{value} >= {np.prod(x.shape[1:])}]"
            )
elif value >= np.prod(x.shape[2:]):    
        raise ValueError(
            f"'{value_name}' must be smaller than input size."
            f" [{value} >= {np.prod(x.shape[2:])}]"
)
```
The metric evaluation, however, is computationally slow on the complete test set. Evaluating on 10% of records (approx. 500) takes about 30 mins on a 2GHz quad-core cpu.