Repository: Giskard-AI/giskard-oss
Issue: Test output in range - indexing error while using slicing function for classification problem
Root cause: Validation Gap

https://github.com/Giskard-AI/giskard/blob/4f88d8095adf3d87d5be4900b37dad87d3950c7b/giskard/testing/tests/statistic.py#L86C1-L164C114

Hello, it seems that while using a slicing function for this test, in a classification problem, there is a mismatch between the index of the sliced dataframe and the index of the predictions. 

For instance, this is the index of the sliced dataframe:
<img width="401" alt="image" src="https://github.com/user-attachments/assets/b41c1ee6-97b4-4a2e-8660-7ded966133ba">

You can see that it contains 4372 rows, and that the rows keep having the index they had in the non-sliced dataframe.

On the other hand, here is the index of output (the predictions for current slice, for the chosen classification label):
<img width="237" alt="image" src="https://github.com/user-attachments/assets/b3574453-dea5-4d2d-9468-fd2eacb4ffdd">

This mismatch causes this line to fail:
`passed_idx = dataset.df.loc[(output <= max_range) & (output >= min_range)].index.values`

In particular, with an indexing error.

A simple fix could be to run this:
`dataset.df.reset_index(inplace=True)`
Right before running the failing line. I already tried that and it seems to work.

Thank you!





Hello @AndreaBosi96,
I'll investigate this possible issue, thanks for reporting it.
Could you provide the full code you used to find out this bug?

Hello @henchaves, thanks for your quick reply.

I prepared a very simple mock case that you can run to investigate the behavior I described in the issue.

```
# Import libraries
from giskard import Dataset, Model, slicing_function, Suite, testing
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline

# Constant
RS = 42

# Sample dataset with some numerical and categorial columns (and binary target)
np.random.seed(RS)  # For reproducibility
data = {
    'feature1': np.random.rand(100),
    'feature2': np.random.rand(100) * 100,
    'feature3': np.random.rand(100) * 50,
    'feature4': np.random.randint(1, 10, 100),
    'sex': np.random.choice(['Male', 'Female'], 100),
    'category': np.random.choice(['A', 'B'], 100),
    'target': np.random.choice(["0", "1"], 100)
}
df = pd.DataFrame(data)

# Let's prepare to train a super simple model
X = df.drop('target', axis=1)
y = df['target']

# Split the dataset into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=RS)

# Prepare model
preprocessor = ColumnTransformer(
    transformers=[
        ('num', 'passthrough', ['feature1', 'feature2', 'feature3', 'feature4']),
        ('cat', OneHotEncoder(), ['sex', 'category'])
    ])

model = Pipeline(steps=[
    ('preprocessor', preprocessor),
    ('classifier', RandomForestClassifier(n_estimators=100, random_state=RS))
])

# Train the model
model.fit(X_train, y_train)

# Now we can use this model for Giskard tests
# Let's wrap the dataset and the model
gsk_dataset = giskard_dataset=Dataset(
    df=df,
    target="target",
    cat_columns=["sex", "category"],
)
gsk_model = Model(
    model=model.predict_proba,
    model_type="classification",
    name="test",
    classification_labels=["0", "1"],
    features=['feature1', 'feature2', 'feature3', 'feature4', 'sex', 'category']
)

# Simple slicing function
@slicing_function()
def slice_sex(row: pd.Series, category: str):
    return row["sex"] == category

# Example suite with the test of the issue
suite = Suite().add_test(
    testing.test_output_in_range(
        classification_label="1",  
        min_range=0.5, 
        max_range=1,  
        threshold=0.15,
        slicing_function=slice_sex(category="Male")
    ),
    test_id = 'Test output in range'
)

# This should fail (or still show some indexing error trace in the logs)
test_results = suite.run(
    model=gsk_model, 
    dataset=gsk_dataset,
)
```

Even if the code runs, you should be able to check the logs and see an error trace raised when running the test.
I think this is the important line of the error trace:

```
.....\site-packages\giskard\testing\tests\statistic.py", line 153, in test_output_in_range
passed_idx = dataset.df.loc[(output <= max_range) & (output >= min_range)].index.values
```

In the specific test code, the slicing function is applied to the dataset, but the remaining rows still have the original index that they had in the "whole" dataframe. On the other hand, the predictions computed only on the slice will have a normal progressive index. This mismatch of index causes the error, since we try to slice a dataframe with a condition on another one (that has same length, but different index).

Let me know if I can be of further assistance, thank you!

Thanks @AndreaBosi96 

We have been able to identify and fix the bug. As we want to keep the output dataset indexes intact for debugging purpose, the solution was to reindex the mask.

A new version with the bug fix should be released by the end of the week.